Details

    • Type: Bug Bug
    • Status: Closed Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Rejected
    • Affects Version/s: EAP 6.0.0 DR 13, EAP 6.0.0 ER 2, EAP 6.0.0 ER 3_Beta1
    • Fix Version/s: EAP No Release
    • Component/s: System
    • Security Level: Public (Everyone can see)
    • Labels:
      None
    • Docs QE Status:
      NEW
    • Similar Issues:
      Show 6 results 

      Description

      07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
      07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
      07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
      07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
      07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
      07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
      07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
      07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
      07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
      07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
      07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
      07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
      07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"

      ... and then nothing happened anymore.

      Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

      The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

      https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
      This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

      For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

      If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

      02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

        Activity

        Hide
        Brian Stansberry
        added a comment -

        James Perkins is looking further into this.

        Show
        Brian Stansberry
        added a comment - James Perkins is looking further into this.
        Hide
        James Perkins
        added a comment -

        It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled.

        The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as:

        class StreamGobbler extends Thread {
            private final InputStream inputStream;
        
            StreamGobbler(final InputStream inputStream) {
                this.inputStream = inputStream;
            }
        
            static void startFor(final Process process) {
                new StreamGobbler(process.getErrorStream()).start();
                new StreamGobbler(process.getInputStream()).start();
            }
        
            @Override
            public void run() {
                try {
                    final BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
                    while (reader.readLine() != null){}
                } catch (IOException ioe) {
                    ioe.printStackTrace();
                }
            }
        }
        

        Then in the groovy code you just need to execute it:

        p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG)
        StreamGobbler.startFor(p)
        

        That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.

        Show
        James Perkins
        added a comment - It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled. The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as: class StreamGobbler extends Thread { private final InputStream inputStream; StreamGobbler( final InputStream inputStream) { this .inputStream = inputStream; } static void startFor( final Process process) { new StreamGobbler(process.getErrorStream()).start(); new StreamGobbler(process.getInputStream()).start(); } @Override public void run() { try { final BufferedReader reader = new BufferedReader( new InputStreamReader(inputStream)); while (reader.readLine() != null ){} } catch (IOException ioe) { ioe.printStackTrace(); } } } Then in the groovy code you just need to execute it: p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG) StreamGobbler.startFor(p) That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.
        Hide
        James Perkins
        added a comment -

        Bug appears to be in the test case and not the server code.

        Show
        James Perkins
        added a comment - Bug appears to be in the test case and not the server code.
        Hide
        Jan Martiska
        added a comment -

        Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn

        Show
        Jan Martiska
        added a comment - Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn
        Hide
        Paul Gier
        added a comment -

        Moving out of TBD.

        Show
        Paul Gier
        added a comment - Moving out of TBD.

          People

          • Assignee:
            James Perkins
            Reporter:
            Jan Martiska
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: