Updated README: https://github.com/lfryc/aerogear-js-integration/blob/AGJS-211-rework-integration-tests/README.md

There is a sample output on Travis:
https://travis-ci.org/lfryc/aerogear-js/builds/37499742

lholmqui@redhat.com could you please test current state of AGJS-211-rework-integration-tests branches on Mac? I don't have any available now.

The tests are now pretty stable,

but simplepush tests fail for some reason (on Chrome as well).
I guess there is an issue with API navigator.push.register incompatibility.

• fix MQTT tests (as Luke reported, they didn't run properly) - solved, forgot to install overlay configuration (conf.tar.gz)

-Got this issue:- (solved by switching from targz to extract-tarball)

Running "ci:vertx" (ci) task
Running "download:vertx" (download) task
Fatal error: Didn't get expected byte count
expect: 6343774
actual: 6344657

It's issue with untargz module.

Things that I still need to address:

• -remove old dependencies, configs and tasks-
• -fix MQTT tests (as Luke reported, they didn't run properly)-
• -package uber-JAR for SPS and configure its tests/tasks- (AGSMPLPUSH-67)
• prepare grunt-external-daemon for contribution to upstream
• testing cross-platform
• -testing on Travis-
• -npm install needs to be invoked in aerogear-js-integration before we can run tests-
• rename test tasks to show what they test, not what container they run on
• appropriate README notes

cool, i will take a look

Just noticed the task ci (e.g. ci:activemq) don't stop on failure - this is actually nice side effect.

We could run all integration tests and then report aggragated result - for all tests.

I've revamped the tests in this way so far:

• implemented "download" multi-task (could be exposed as grunt module)
  • leaving aside its name, it does actually much more: 1. downloads distribution, 2. validates checksum, 3. extracts, 4. extracts configuration file (overlay), 5. renames to the stable name (e.g. runtimes/apache-activemq instead of apache-activemq-5.8.0)
• ability to kill the process running on background
  • each need to use different approach
    • activemq runs as daemon, is needs to be killed by running 'activemq stop' command
    • vertx runs as process, can be killed simply by issuing SIGTERM
  • when grunt ends prematurily (by killing its process or by test failure) the server is stopped gracefully (that requires grunt:0.4.1 as highter versions contains a bug/limitation)
  • we are able to stop the server after test by executing :stop subtask
• tests rewritten to Karma
  • that wasn't intentional first, but Karma has another advantage - it uses its own server and bypasses browser security (CORS) so testing is much easier
    • as connect can't be killed gracefully after the test (at least not simple way)
  • Karma has one downside though, it can't load HTML file with QUnit script (at least not vanilla Karma), because it expects files array in the config

I believe this approach has certain advantages:

• configuration for downloads is way simpler (declarative)
  • all you need is declaring http urls
  • it is also smart: installs new runtime just when needed
  • because it uses temporary folder for extraction we know exactly when the distribution is ready to be moved to its position (process-kill-proof solution)
• running tasks is faster and more stable (in theory, the time will tell us)
  • its faster because it waits for certain log in process' stdout (as oppose to waiting hard-coded delay time)
  • its more stable because thanks to using grunt-external-daemon the processes are handled correctly
• it can be simply debugged
  • grunt ... --verbose --debug
• the only what aerogear-js needs to do is make sure aerogear-js-integration is cloned and then run grunt ci:activemq ci:vertx

Some disadvantages:

• none
  • just kidding, here it is:
• using forked version of grunt-external-daemon
  • the original didn't handle kill processes and didn't support a way of stopping that activemq requires as a daemon process
  • but we should contribute this improvement back
  • the project doesn't use cleaneast project ever and I made it even bigger mess, some re-structure would be nice after agreement with an original author

The code is here:

aerogear-js: https://github.com/lfryc/aerogear-js/tree/AGJS-211-rework-integration-tests

aerogear-js-integration: https://github.com/lfryc/aerogear-js-integration/tree/AGJS-211-rework-integration-tests

Some polishing is still needed as it is still rather proof of concept.
Old configs and dependencies need to be removed.

Some commands to play with after clones:

• grunt ci:vertx ci:activemq
• grunt ci:vertx --verbose --debug