BMUNit does not always work with TestNG test suites which configure test classes by package and/or group

When using TestNG to run tests it is possible to select the tests to be run in several different ways:

• by listing individual test classes

• by specifying test group ids

• by specifying packages to which tests must belong

It has been noticed that BMUnit does not always work properly when the last two configuration methods are used. The test rules will be loaded, injected and triggered but the rule code fails to execute.

The failure arises because some of the Byteman classes in the byteman agent jar (byteman.jar) are being loaded by the system classloader before the agent is installed. When class BMUnit is loaded it adds the byteman agent jar to the bootstrap class path in order to be able to inject into JVM classes. This means that some of the Byteman class references occur relative to the system loader and some to the bootstrap loader, the critical one being class Rule which appears in both loaders. The agent listener class (which is resolved against the bootstrap class loader) installs rules into a static hashmap defined by class Rule as resolved against the bootstrap class loader. Meanwhile the injected trigger code resolves its classes via the class loader of the class it is injected into, which means that its rule lookup uses the empty static hashmap defined by class Rule resolved against the system class loader.

The problem arises when the test classes include a helper class in the package specified in the package specification which needs to resolve against the Byteman classes, in particular when it specialises class Helper. The latter includes reference to class Rule in some of its method signatures. TestNG loads classes in the specified package (into the system class loader), including the test helper class, before running the tests and, as a consequence, when resolving its super class (Helper) is also
loads class Rule. This happens before the BMUnitRunner class is loaded and hence before BMUnit gets a chance to autoload the agent and ensure that the byteman jar is in the bootstrap classpath.

This problem can be avoided (worked around) by ensuring that helper classes are not included in the package specified in the TestNG configuration. A full solution would ensure that the Byteman agent is loaded before TestNG loads the required test classes. However, the latter does not appear to be simple given what hooks TestNG provides. This JIRA is being left open to report the problem and its workaround and just in case such a full solution can be identified or determined not to be possible.