Byteman 1.5.0 JUnit Integration Makes Fault Injection Simple

Byteman 1.5.0 has just been released and the major new feature is a really nice user contributed package, BMUnit, which provides integration with JUnit. It makes fault injection testing and execution tracing unbelievably simple.

BMUnit provides annotations which you use to identify the Byteman rules needed for each of your JUnit tests. Annotations attached to a test class specify rules which apply to all test methods. Annotations attached to a method identify rules which only apply for that specific test.

The BMUnit test runner ensures the Byteman agent is installed in the JUnit JVM before any testing starts. It automatically loads all required rules into the JVM before running a test and unloads them when they are no longer needed. All you need to do to use BMUnit is is attach an @RunWith(BMUnitRunner.class) annotation to your test class.

Here's a simple example to show you how quick and easy BMUnit is to use

@RunWith(BMUnitRunner.class)
public class FileOpenTest {
  @BMRule(name="throw security exception",
    targetClass="FileInputStream",
    targetMethod="<init>(File)",
    condition="$1.getName().equals\"badname.txt\")"
    action="throw new SecurityException(\"bad name!\")")
  @Test
  public void testSecurityException() {
    MyTestClass myTestObj = new MyTestClass();
    myTestObj.processFile("badname.txt");
  }
  . . .
}

In this example a single rule is provided inline in the test program itself, using a @BMRule annotation attached to test method testSecurityException. The rule is triggered on entry to the constructor FileInputStream(File). The condition restricts the rule to firing only when the File argument has name "badname.txt" (which we have cunningly arranged to be the name supplied in the test call). When the rule does fire the action causes a SecurityException to be thrown from the constructor.

So, when this unit test is run the runner class, BMUnitRunner, will install the agent into the test JVM then load the rule defined in the annotation into the agent database, injecting it into the constructor method. When a file open is attempted under the call to myTestObj.processFile() the rule is triggered, ensuring that the test class, MyTestClass, receives a SecurityException.

Clearly, it would be more interesting if we could be sure that the security exception was handled correctly. A few more rules can be used to check that the correct handler code has been called or, if not, force a test failure. In this case we will place the rules in a separate script to avoid adding too much clutter to the test class. First we change the annotation to @BMScript, pointing the BMUnit test runner at a rule script file.

@RunWith(BMUnitRunner.class)
public class FileOpenTest {
  @BMScript(value="security-exception", dir="scripts")
  @Test
  public void testSecurityException() {
    MyTestClass myTestObj = new MyTestClass();
    myTestObj.processFile("badname.txt");
  }
  . . .
}

The BMScript annotation identifies the name and location of the script file. Locations with a non-absolute path are interpreted relative to the working directory of the test JVM. The script file name is looked for using either a ".btm" or ".txt" extension. So, let's assume we have placed the following rules in a file called "security-exception.btm" located in subdirectory "scripts".

RULE throw security exception 
CLASS FileInputStream 
METHOD <init>(File)", 
IF "$1.getName().equals\"badname.txt\" 
DO clear("handled"); 
   throw new SecurityException("bad name!") 
ENDRULE 
RULE track handler call, 
CLASS MyTestClass 
METHOD handle(SecurityException) 
IF true 
DO flag("handled") 
ENDRULE 
RULE ensure handler call 
CLASS MyTestClass 
METHOD processFile(File) 
AT RETURN 
IF !flagged("handled") 
DO throw new RuntimeException("unhandled security exception") 
ENDRULE 

The first rule is essentially the same as the one we had in the @BMRule annotation. The only difference is that it clears any flag associated with keyword "handled". The second rule is triggered when method handle(SecurityException) of class MyTestClass gets called. It fires unconditionally, setting a flag labelled by the keyword string "handled". We will assume that calling this method is enough to guarantee that the test has passed. The third rule is triggered when method processFile(File) returns normally. If the "handled" flag has not been set it throws an exception, causing the unit test to fail. If processFile(File) returns abnormally this rule will be bypassed but the exception will be detected by JUnit, also indicating a failure.

Of course, we could have used the BMScript and BMRule annotations in combination, defining, say, the first rule inline and the second and third rule in the script. We can also specify more than one rule or more than one script by using the annotations @BMRules and @BMScripts. The values of these annotations are an array of, respectively, @BMRule or @BMScript annotations. For example

@RunWith(BMUnitRunner.class)
public class FileOpenTest {
  @BMRules( {
    @BMRule(name = "throw security exception", ...)
    @BMRule(name = "track handler call", ...)
  })
  @Test
  public void testSecurityException() {
    MyTestClass myTestObj = new MyTestClass();
    myTestObj.processFile("badname.txt");
  }
  . . .
}

Full details of how to use the package are included in the README located in the contrib/bmunit subdirectory of the 1.5.0 release.