Annotation-based Configuration for BMUnit in Byteman 2.2.0

If you have not yet used BMUnit . . .

The BMUnit package integrates Byteman with the popular Java unit and integration test automation tools, JUnit and TestNG. BMUnit allows you to automatically install the Byteman agent and upload/unload Byteman rules into/from your test JVM using a few simple annotations on test classes or methods. If you have not already used BMUnit then take a look at the Byteman fault injection tutorial. It provides a complete worked example which demonstrates how easy it is to use BMUnit and Byteman to engineer reliable, repeatable tests. The example code is a simple, thread-parallel application which would be very difficult to test without using Byteman. It provides a great example of the power of fault injection as a test methodology.

Configuring BMUnit via system properties

Up until now the environment in which both BMUnit and its underlying Byteman agent execute your tests has only been configurable via system property settings. Not only is this a clumsy and error-prone control model, it also allows only one configuration to be defined which gets used for all tests.

Switching on BMUnit trace

For example, if you want to trace operations of BMUnit, watching it upload and unload Byteman rules or checking that it has indeed installed the Byteman agent into your test JVM you would modify your maven pom as follows

1:      <systemProperties>  
2:        <property>  
3:          <name>org.jboss.byteman.contrib.bmunit.verbose</name>  
4:          <value>true</value>  
5:        </property>  
6:      </systemProperties>  

After applying this change to the pom in the junit directory of the tutorial you will see BMUnit trace output as the tests are executed

 [adinn@zenade byteman-tutorial2]$ mvn -P junit test  
 /usr/lib/jvm/java-1.7.0  
 [INFO] Scanning for projects...  
  . . .  
 -------------------------------------------------------  
  T E S T S  
 -------------------------------------------------------  
 Running org.my.BytemanJUnitTests  
 BMUNit : loading agent id = 31122  
 byteman jar is /home/adinn/.m2/repository/org/jboss/byteman/byteman/2.1.4/byteman-2.1.4.jar  
 BMUNit : loading file script = target/test-classes/trace.btm  
 testPipeLine:  
 #INPUT  
 hello world!  
 . . .  
 #END  
 entered run for org.my.pipeline.impl.CharSequenceSink  
 . . .  
 exited run for org.my.pipeline.impl.CharSequenceSink  
 #OUTPUT  
 hello mum!  
 . . .  
 #END  
 BMUNit : unloading file script = target/test-classes/trace.btm  
 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.443 sec  
 . . .   

As you can see, BMunit tells you it is loading the Byteman agent into the Java process with pid 31122 and then prints the location of the Byteman jar. It reports that it is uploading a rule script before running the first test and then tells you when it unloads the test script.

Switching on Byteman verbose trace

For more detailed trace you could also enable the Byteman agent's verbose trace

 [adinn@zenade byteman-tutorial2]$ mvn -P junit test  
 /usr/lib/jvm/java-1.7.0  
 [INFO] Scanning for projects...  
 . . .  
 Running org.my.BytemanJUnitTests  
 BMUNit : loading agent id = 5025  
 byteman jar is /ssd/home/adinn/.m2/repository/org/jboss/byteman/byteman/2.1.4/byteman-2.1.4.jar  
 TransformListener() : accepting requests on localhost:9091  
 BMUNit : loading file script = target/test-classes/trace.btm  
 TransformListener() : handling connection on port 9091  
 testPipeLine:  
 org.jboss.byteman.agent.Transformer : possible trigger for rule trace input characters in class org.my.pipeline.impl.CharSequenceSource  
 RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.my.pipeline.impl.CharSequenceSource.<init>(java.lang.CharSequence) void for rule trace input characters  
 org.jboss.byteman.agent.Transformer : inserted trigger for trace input characters in class org.my.pipeline.impl.CharSequenceSource  
 org.jboss.byteman.agent.Transformer : possible trigger for rule trace Thread.run entry in class org.my.pipeline.core.SourceProcessor  
 RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.my.pipeline.core.SourceProcessor.run() void for rule trace Thread.run entry  
 org.jboss.byteman.agent.Transformer : inserted trigger for trace Thread.run entry in class org.my.pipeline.core.SourceProcessor  
 org.jboss.byteman.agent.Transformer : possible trigger for rule trace Thread.run exit in class org.my.pipeline.core.SourceProcessor  
 RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.my.pipeline.core.SourceProcessor.run() void for rule trace Thread.run exit  
 RuleTriggerMethodAdapter.injectTriggerPoint : inserting trigger into org.my.pipeline.core.SourceProcessor.run() void for rule trace Thread.run exit  
 org.jboss.byteman.agent.Transformer : inserted trigger for trace Thread.run exit in class org.my.pipeline.core.SourceProcessor  
 Rule.execute called for trace input characters_0  
 HelperManager.install for helper class org.jboss.byteman.rule.helper.Helper  
 calling activated() for helper class org.jboss.byteman.rule.helper.Helper  
 Default helper activated  
 calling installed(trace input characters) for helper classorg.jboss.byteman.rule.helper.Helper  
 Installed rule using default helper : trace input characters  
 trace input characters execute  
 #INPUT  
 hello world!  
 . . .   

This time the agent's trace messages show it opening a server socket and listening for requests. After the BMUnit trace for the first file load you can see the agent's socket listener handling the connection for the upload. Next, as various rule target classes get loaded, the agent prints messages showing rule code being injected into the target methods. The inject messages are followed by tracing of an execution event for the first rule and the associated "activated" and "installed" helper lifecycle events. Once lifecycle management has completed the rule finally  gets executed and the test continues, printing its output.

Other configuration choices

There are a variety of other system properties which can be used to configure different aspects of how BMUnit and Byteman operate. For example, it is possible to reset the hostname and port used by the agent listener when it waits for rules to be uploaded. These same settings are also used by BMUNit to dispatch upload and unload requests. You can also set the default directory in which to look for rule scripts, disable automatic loading of the agent and so on.

Better configuration via annotations

This configuration capability is very useful but it suffers from a few drawbacks. As mentioned above, it just provides a single global setting for all tests in the test execution. So, even though you might only be interested in tracing what BMUnit or the Byteman agent is doing when executing a specific failing test you have enable tracing for all tests and then wade through the output to find the relevant printout.
A second problem is that configuring the relevant properties is error prone. It is very easy to mis-spell the long property names employed by Byteman and BMUnit. Also, unlike the annotations used to mark up BMUnit tests these properties get configured in the pom (or ant build) file, separate from the tests themselves.

The @BMUNitConfig annotation

The latest version of BMUnit (2.2.0) implements a new configuration annotation, @BMUnitConfig which addresses all these problems. Annotation attributes can be used to specify all the different BMUnit and Byteman behaviours currently managed using system properties. An @BMUnitConfig annotation can be attached to a test class in order to specify the configuration for all tests in that class. It can also be attached to a test method, overriding the class level settings just for the duration of that specific test. The same annotation works with both JUnit and TestNG.
So, for example you could modify the JUnit example test class executed in the above example as follows

1:  @RunWith(BMUnitRunner.class)  
2:  @BMUnitConfig(bmunitVerbose=true, agentPort="99999")  
3:  @BMScript(value="trace", dir="target/test-classes")  
4:  public class BytemanJUnitTests  
5:  {  
6:    @Test  
7:    @BMUnitConfig(bmunitVerbose=false, debug=true)  
8:    public void testPipeline() throws Exception  
9:    {  
10:    . . .  

Class level configuration

The initial, class-level annotation configures a different agent port to the default. So, when the test runs and the agent gets installed it will listen on port 99999. BMUnit will also use the same port to upload and unload rules specified via  @BMRule or @BMScript annotations.
The class annotation also sets the bmunitVerbose attribute to true. This has the same effect as configuring the system property which controls BMUnit trace. So, when the test is run BMUnit will trace auto-loading of the agent and rule upload/unload.

Method level configuration

The annotation on test method testPipeline() resets the bmunitVerbose attribute to false, disabling BMUnit trace during execution of this specific test. It also sets the debug attribute to true, enabling Byteman debug trace.
Byteman debug trace is normally disabled, which means that calls to built-in method debug don't print any output. Switching on debug like this is very useful when you are trying to understand what your rules are doing. You can add debug messages to all your rules but you will still only  see debug messages associated with the rules fired during this test.
Notice that the annotation fields have much simpler names than the corresponding system properties. Between your IDE and the javac compiler any misspellings will be discovered very quickly.

What else can I configure?

Full details of the BMUnitConfig annotation attributes and what happens when you set the them are provided in the README file distributed with the BMUnit sources.