We were using TestNG as the testing framework and we had some issues while trying, for instance, to override the @BeforeSuite or @BeforeClass methods. After trying some workarounds we ended up asking Aslak for help/advice. He told us that before/after in TestNG doesn't play nice with Arquillian since it does not give them the control they need. TestNG doesn't go through phases, like before/test/after, those phases only occur if the user has defined them (they work like callbacks); consequently if the user doesn't define a given annotation method, the SPI is never called.

Another issue is that despite TestNG docs stating that tests do not run in parallel:

If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.

The truth is that if we do not define dependencies between test classes we end up with ClassCastException and ClassNotFoundException errors. Meaning that the same WAR is being deployed to the container before the previous one has been undeployed.

We do think that this is important information that could in some situations/projects invalidate the choice of TestNG as the testing framework. Thereof, it makes sense adding it to the Arquillian documentation. We asked Aslak for his opinion, and he has agreed.

Relevant #jbosstesting channel logs: http://echelog.com/logs/browse/jbosstesting/1336514400