Details

    • Type: Bug Bug
    • Status: Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Rejected
    • Affects Version/s: EAP 6.0.0 DR 13, EAP 6.0.0 ER 2, EAP 6.0.0 ER 3_Beta1
    • Fix Version/s: EAP No Release
    • Component/s: System
    • Labels:
      None
    • Docs QE Status:
      NEW
    • Similar Issues:
      Show 6 results 

      Description

      07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
      07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
      07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
      07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
      07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
      07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
      07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
      07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
      07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
      07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
      07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
      07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
      07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"

      ... and then nothing happened anymore.

      Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

      The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

      https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
      This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

      For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

      If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

      02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

        Gliffy Diagrams

          Activity

          Jan Martiska created issue -
          Jan Martiska made changes -
          Field Original Value New Value
          Link This issue relates to JBQA-5917 [ JBQA-5917 ]
          Jan Martiska made changes -
          Fix Version/s EAP 6.0.0 ER 1 [ 12318305 ]
          Hide
          Brian Stansberry added a comment -

          Please attach or email to me jaxrpc-samples-swa.war. I want to experiment using the deployment this test used.

          Show
          Brian Stansberry added a comment - Please attach or email to me jaxrpc-samples-swa.war. I want to experiment using the deployment this test used.
          Hide
          Brian Stansberry added a comment -

          I just deployed and undeployed an app a few hundred times without incident (using upstream master), so I'll definitely need the jaxrpc-samples-swa.war to try and reproduce.

          Show
          Brian Stansberry added a comment - I just deployed and undeployed an app a few hundred times without incident (using upstream master), so I'll definitely need the jaxrpc-samples-swa.war to try and reproduce.
          Hide
          Rajesh Rajasekaran added a comment -

          Attaching war file used in the test

          Show
          Rajesh Rajasekaran added a comment - Attaching war file used in the test
          Rajesh Rajasekaran made changes -
          Attachment jaxrpc-samples-swa.war [ 12351665 ]
          Hide
          Jan Martiska added a comment -

          If it works in upstream master, you should try it with EAP DR13, that is where I encountered the issue.

          Show
          Jan Martiska added a comment - If it works in upstream master, you should try it with EAP DR13, that is where I encountered the issue.
          Hide
          Paul Gier added a comment -

          Making this resolved because it works upstream. The fix should have been pulled in the last rebase.

          Show
          Paul Gier added a comment - Making this resolved because it works upstream. The fix should have been pulled in the last rebase.
          Paul Gier made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Done [ 1 ]
          Hide
          Brian Stansberry added a comment -

          Thanks Paul. I tried the attached war over the weekend and can confirm it was working in upstream (at least on my local machine.)

          If this comes up again, please try and get a thread dump from the server under test.

          Show
          Brian Stansberry added a comment - Thanks Paul. I tried the attached war over the weekend and can confirm it was working in upstream (at least on my local machine.) If this comes up again, please try and get a thread dump from the server under test.
          Hide
          Jan Martiska added a comment -

          I tried with the newly released ER1 and the issue has probably indeed been resolved. I will notify if I stumble upon it again.

          Show
          Jan Martiska added a comment - I tried with the newly released ER1 and the issue has probably indeed been resolved. I will notify if I stumble upon it again.
          Jan Martiska made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Jan Martiska added a comment -

          The issue persists in ER2 and ER3. Will create a reproducer.

          Show
          Jan Martiska added a comment - The issue persists in ER2 and ER3. Will create a reproducer.
          Jan Martiska made changes -
          Resolution Done [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Jan Martiska made changes -
          Fix Version/s TBD EAP 6 [ 12315096 ]
          Fix Version/s EAP 6.0.0 ER 1 [ 12318305 ]
          Description {quote}
          07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
          07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
          07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
          07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
          07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
          07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
          07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
          07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
          07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
          07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
          07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          {quote}

          ... and then nothing happened anymore.

          Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

          The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

          https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
          This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

          For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.
          {quote}
          07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
          07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
          07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
          07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
          07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
          07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
          07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
          07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
          07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
          07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
          07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          {quote}

          ... and then nothing happened anymore.

          Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

          The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

          https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
          This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

          For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

          If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

          02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


          Affects Version/s EAP 6.0.0 ER 3 [ 12319176 ]
          Affects Version/s EAP 6.0.0 ER 2 [ 12318374 ]
          Jan Martiska made changes -
          Description {quote}
          07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
          07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
          07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
          07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
          07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
          07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
          07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
          07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
          07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
          07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
          07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
          07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
          07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
          {quote}

          ... and then nothing happened anymore.

          Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

          The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

          https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
          This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

          For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

          If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

          02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


          {quote}
          07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
          07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
          07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
          07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
          07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
          07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
          07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
          07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
          07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
          07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
          07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
          07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
          07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
          {quote}

          ... and then nothing happened anymore.

          Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

          The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

          https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
          This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

          For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

          If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

          02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


          Hide
          Jan Martiska added a comment -

          Reproducer for this issue. Instructions are in file named HOW_TO_RUN_REPRODUCER ..

          Show
          Jan Martiska added a comment - Reproducer for this issue. Instructions are in file named HOW_TO_RUN_REPRODUCER ..
          Jan Martiska made changes -
          Attachment jbpapp8124-reproducer.zip [ 12352372 ]
          Hide
          Jan Martiska added a comment -

          Thread dump of the server under test, after this issue occurs.

          Show
          Jan Martiska added a comment - Thread dump of the server under test, after this issue occurs.
          Jan Martiska made changes -
          Attachment threaddump [ 12352373 ]
          Hide
          Brian Stansberry added a comment -

          Thanks for the details. I ran the reproducer on my laptop against upstream/master for 250 cycles without incident. It seems to essentially be doing what I did manually before and at about the same pace, so it doesn't surprise me that it works. I'll try on a different machine later.

          The thread dump has a big clue. My assumption is a second thread dump would show this thread to still be in the same state. It would be very unusual for a thread to be in org.jboss.as.webservices.service.EndpointService.start() long after deployment began if it wasn't stuck for some reason. If in fact this thread is not progressing, that is the root of the problem:

          Thread 13287: (state = IN_NATIVE)

          • java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0 (Interpreted frame)
          • java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=260 (Interpreted frame)
          • java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 (Interpreted frame)
          • java.io.BufferedOutputStream.write(int) @bci=13, line=78 (Compiled frame)
          • java.io.PrintStream.write(int) @bci=13, line=399 (Compiled frame)
          • org.jboss.logmanager.handlers.UncloseableOutputStream.write(int) @bci=5, line=39 (Compiled frame)
          • java.io.FilterOutputStream.write(int) @bci=5, line=60 (Compiled frame)
          • org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(int) @bci=4, line=59 (Compiled frame)
          • java.io.FilterOutputStream.write(byte[], int, int) @bci=41, line=108 (Compiled frame)
          • org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(byte[], int, int) @bci=11, line=87 (Interpreted frame)
          • sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Interpreted frame)
          • sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Interpreted frame)
          • sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Interpreted frame)
          • sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Interpreted frame)
          • java.io.OutputStreamWriter.flush() @bci=4, line=212 (Interpreted frame)
          • org.jboss.logmanager.handlers.WriterHandler.safeFlush(java.io.Flushable) @bci=5, line=162 (Interpreted frame)
          • org.jboss.logmanager.handlers.WriterHandler.flush() @bci=12, line=133 (Interpreted frame)
          • org.jboss.logmanager.ExtHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=8, line=92 (Interpreted frame)
          • org.jboss.logmanager.handlers.WriterHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=81, line=70 (Interpreted frame)
          • org.jboss.logmanager.ExtHandler.publish(java.util.logging.LogRecord) @bci=17, line=64 (Interpreted frame)
          • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=26, line=283 (Compiled frame)
          • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
          • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
          • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
          • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
          • org.jboss.logmanager.Logger.logRaw(org.jboss.logmanager.ExtLogRecord) @bci=81, line=649 (Interpreted frame)
          • org.jboss.logmanager.Logger.log(java.lang.String, java.util.logging.Level, java.lang.String, org.jboss.logmanager.ExtLogRecord$FormatStyle, java.lang.Object[], java.lang.Throwable) @bci=70, line=600 (Compiled frame)
          • org.jboss.logging.JBossLogManagerLogger.doLogf(org.jboss.logging.Logger$Level, java.lang.String, java.lang.String, java.lang.Object[], java.lang.Throwable) @bci=40, line=54 (Interpreted frame)
          • org.jboss.logging.Logger.logf(java.lang.String, org.jboss.logging.Logger$Level, java.lang.Throwable, java.lang.String, java.lang.Object) @bci=23, line=2096 (Interpreted frame)
          • org.jboss.as.webservices.WSLogger_$logger.starting(java.lang.Object) @bci=34, line=373 (Interpreted frame)
          • org.jboss.as.webservices.service.EndpointService.start(org.jboss.msc.service.StartContext) @bci=7, line=97 (Interpreted frame)
          • org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(org.jboss.msc.service.Service, org.jboss.msc.service.StartContext) @bci=13, line=1811 (Compiled frame)
          • org.jboss.msc.service.ServiceControllerImpl$StartTask.run() @bci=95, line=1746 (Compiled frame)
          • java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Compiled frame)
          • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Compiled frame)
          • java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

          So, the thread is blocking trying to write a log message to a file. Specifically, INFO [org.jboss.as.webservices] (MSC service thread 1-1) JBAS015539: Starting service jboss.ws.endpoint."jaxrpc-samples-swa.war".AttachmentJSE

          I believe some of the earlier reports showed progress stopping at different points in the deploy/undeploy cycle. That is, the last log message written before progress halted varied. But my expectation is in all cases the root issue was the same – something is preventing the writing of the log.

          Show
          Brian Stansberry added a comment - Thanks for the details. I ran the reproducer on my laptop against upstream/master for 250 cycles without incident. It seems to essentially be doing what I did manually before and at about the same pace, so it doesn't surprise me that it works. I'll try on a different machine later. The thread dump has a big clue. My assumption is a second thread dump would show this thread to still be in the same state. It would be very unusual for a thread to be in org.jboss.as.webservices.service.EndpointService.start() long after deployment began if it wasn't stuck for some reason. If in fact this thread is not progressing, that is the root of the problem: Thread 13287: (state = IN_NATIVE) java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0 (Interpreted frame) java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=260 (Interpreted frame) java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 (Interpreted frame) java.io.BufferedOutputStream.write(int) @bci=13, line=78 (Compiled frame) java.io.PrintStream.write(int) @bci=13, line=399 (Compiled frame) org.jboss.logmanager.handlers.UncloseableOutputStream.write(int) @bci=5, line=39 (Compiled frame) java.io.FilterOutputStream.write(int) @bci=5, line=60 (Compiled frame) org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(int) @bci=4, line=59 (Compiled frame) java.io.FilterOutputStream.write(byte[], int, int) @bci=41, line=108 (Compiled frame) org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(byte[], int, int) @bci=11, line=87 (Interpreted frame) sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Interpreted frame) sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Interpreted frame) sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Interpreted frame) sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Interpreted frame) java.io.OutputStreamWriter.flush() @bci=4, line=212 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.safeFlush(java.io.Flushable) @bci=5, line=162 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.flush() @bci=12, line=133 (Interpreted frame) org.jboss.logmanager.ExtHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=8, line=92 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=81, line=70 (Interpreted frame) org.jboss.logmanager.ExtHandler.publish(java.util.logging.LogRecord) @bci=17, line=64 (Interpreted frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=26, line=283 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.Logger.logRaw(org.jboss.logmanager.ExtLogRecord) @bci=81, line=649 (Interpreted frame) org.jboss.logmanager.Logger.log(java.lang.String, java.util.logging.Level, java.lang.String, org.jboss.logmanager.ExtLogRecord$FormatStyle, java.lang.Object[], java.lang.Throwable) @bci=70, line=600 (Compiled frame) org.jboss.logging.JBossLogManagerLogger.doLogf(org.jboss.logging.Logger$Level, java.lang.String, java.lang.String, java.lang.Object[], java.lang.Throwable) @bci=40, line=54 (Interpreted frame) org.jboss.logging.Logger.logf(java.lang.String, org.jboss.logging.Logger$Level, java.lang.Throwable, java.lang.String, java.lang.Object) @bci=23, line=2096 (Interpreted frame) org.jboss.as.webservices.WSLogger_$logger.starting(java.lang.Object) @bci=34, line=373 (Interpreted frame) org.jboss.as.webservices.service.EndpointService.start(org.jboss.msc.service.StartContext) @bci=7, line=97 (Interpreted frame) org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(org.jboss.msc.service.Service, org.jboss.msc.service.StartContext) @bci=13, line=1811 (Compiled frame) org.jboss.msc.service.ServiceControllerImpl$StartTask.run() @bci=95, line=1746 (Compiled frame) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Compiled frame) java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Compiled frame) java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) So, the thread is blocking trying to write a log message to a file. Specifically, INFO [org.jboss.as.webservices] (MSC service thread 1-1) JBAS015539: Starting service jboss.ws.endpoint."jaxrpc-samples-swa.war".AttachmentJSE I believe some of the earlier reports showed progress stopping at different points in the deploy/undeploy cycle. That is, the last log message written before progress halted varied. But my expectation is in all cases the root issue was the same – something is preventing the writing of the log.
          Hide
          Jan Martiska added a comment -

          This is really weird. I had another look at it and it really seems it freezes in the middle of FileOutputStream.writeBytes while writing the log, having obtained a lock when entering org.jboss.logmanager.handlers.WriterHandler.doPublish - and any subsequent attempt to deploy more files will be blocked waiting for this lock. It was questioned whether this can be an NFS issue, but it does the same on my local filesystem. Never seen anything like this

          Show
          Jan Martiska added a comment - This is really weird. I had another look at it and it really seems it freezes in the middle of FileOutputStream.writeBytes while writing the log, having obtained a lock when entering org.jboss.logmanager.handlers.WriterHandler.doPublish - and any subsequent attempt to deploy more files will be blocked waiting for this lock. It was questioned whether this can be an NFS issue, but it does the same on my local filesystem. Never seen anything like this
          Hide
          Jan Martiska added a comment -

          And yes, it happens in different phases of deployment, that is - last message written to the log varies.

          Show
          Jan Martiska added a comment - And yes, it happens in different phases of deployment, that is - last message written to the log varies.
          Rajesh Rajasekaran made changes -
          Labels eap6_need_triage
          Hide
          Rajesh Rajasekaran added a comment -

          This issue is repetitive with every quickstart application that we tried.
          https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Deployment-Soak-Tests/
          Interestingly for each quick start it freezes after a fixed number of iterations (different for different apps) as seen in the hudson console log.

          Show
          Rajesh Rajasekaran added a comment - This issue is repetitive with every quickstart application that we tried. https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Deployment-Soak-Tests/ Interestingly for each quick start it freezes after a fixed number of iterations (different for different apps) as seen in the hudson console log.
          Hide
          Jan Martiska added a comment - - edited

          Even if I disable all logging on the server completely, it still fails in the same method (writeBytes). I have no idea what file it is trying to write to. If I list open files ('lsof' command) after EAP freezes, it has no files open for writing, except for boot.log, which seems to be default for all logging if all other logging is disabled.

          Show
          Jan Martiska added a comment - - edited Even if I disable all logging on the server completely, it still fails in the same method (writeBytes). I have no idea what file it is trying to write to. If I list open files ('lsof' command) after EAP freezes, it has no files open for writing, except for boot.log, which seems to be default for all logging if all other logging is disabled.
          Hide
          Brian Stansberry added a comment -

          James Perkins is looking further into this.

          Show
          Brian Stansberry added a comment - James Perkins is looking further into this.
          Brian Stansberry made changes -
          Assignee Brian Stansberry [ brian.stansberry ] James Perkins [ jamezp ]
          Hide
          James Perkins added a comment -

          It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled.

          The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as:

          class StreamGobbler extends Thread {
              private final InputStream inputStream;
           
              StreamGobbler(final InputStream inputStream) {
                  this.inputStream = inputStream;
              }
           
              static void startFor(final Process process) {
                  new StreamGobbler(process.getErrorStream()).start();
                  new StreamGobbler(process.getInputStream()).start();
              }
           
              @Override
              public void run() {
                  try {
                      final BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
                      while (reader.readLine() != null){}
                  } catch (IOException ioe) {
                      ioe.printStackTrace();
                  }
              }
          }
          

          Then in the groovy code you just need to execute it:

          p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG)
          StreamGobbler.startFor(p)
          

          That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.

          Show
          James Perkins added a comment - It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled. The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as: class StreamGobbler extends Thread { private final InputStream inputStream;   StreamGobbler( final InputStream inputStream) { this .inputStream = inputStream; }   static void startFor( final Process process) { new StreamGobbler(process.getErrorStream()).start(); new StreamGobbler(process.getInputStream()).start(); }   @Override public void run() { try { final BufferedReader reader = new BufferedReader( new InputStreamReader(inputStream)); while (reader.readLine() != null ){} } catch (IOException ioe) { ioe.printStackTrace(); } } } Then in the groovy code you just need to execute it: p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG) StreamGobbler.startFor(p) That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.
          Hide
          James Perkins added a comment -

          Bug appears to be in the test case and not the server code.

          Show
          James Perkins added a comment - Bug appears to be in the test case and not the server code.
          James Perkins made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Rejected [ 2 ]
          Hide
          Jan Martiska added a comment -

          Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn

          Show
          Jan Martiska added a comment - Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn
          Jan Martiska made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Paul Gier made changes -
          Resolution Rejected [ 2 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          Paul Gier added a comment -

          Moving out of TBD.

          Show
          Paul Gier added a comment - Moving out of TBD.
          Paul Gier made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Labels eap6_need_triage
          Fix Version/s EAP 6 No Release [ 12318379 ]
          Fix Version/s TBD EAP 6 [ 12315096 ]
          Resolution Rejected [ 2 ]

            People

            • Assignee:
              James Perkins
              Reporter:
              Jan Martiska
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development