Intermittent failures in client side tests that use remote EJB

There are lots of intermittent failures in tests that use remote EJB, and they all fail with the same failure message (EJBCLIENT000025: No EJB receiver available for handling...).

An example of this is here: http://brontes.lab.eng.brq.redhat.com/viewLog.html?buildId=61476&buildTypeId=WFPR

The issue is that there is no guard to make sure that the 'module availability' message for the deployment has been processed before the test starts (there is a guard for this on initial connection, with a hard coded 5 second timeout, however the test suite will re-use connections between tests).

At the moment I don't really have any idea of a good way to fix this.

This failure does not affect end user code because user code does usually perform a deployment then make a call (although it could affect end user Arquillian tests).

This issue can be reproduced 100% of the time by adding a sleep into org.jboss.ejb.client.remoting.ModuleAvailabilityMessageHandler#processMessage, then running the test suite. The first test that uses the remote EJB will pass (because the client waits for the initial module availability message), but all subsequent tests that attempt to use it will fail.