Details

    • Type: Bug Bug
    • Status: Closed Closed (View Workflow)
    • Priority: Critical Critical
    • Resolution: Rejected
    • Affects Version/s: EAP 6.0.0 DR 13, EAP 6.0.0 ER 2, EAP 6.0.0 ER 3_Beta1
    • Fix Version/s: EAP No Release
    • Component/s: System
    • Security Level: Public (Everyone can see)
    • Labels:
      None
    • Docs QE Status:
      NEW
    • Similar Issues:
      Show 6 results 

      Description

      07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
      07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
      07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
      07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
      07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
      07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
      07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
      07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
      07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
      07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
      07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
      07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
      07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"

      ... and then nothing happened anymore.

      Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

      The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

      https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
      This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

      For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

      If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

      02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.

        Activity

        Jan Martiska
        made changes -
        Field Original Value New Value
        Link This issue relates to JBQA-5917 [ JBQA-5917 ]
        Jan Martiska
        made changes -
        Fix Version/s EAP 6.0.0 ER 1 [ 12318305 ]
        Hide
        Brian Stansberry
        added a comment -

        Please attach or email to me jaxrpc-samples-swa.war. I want to experiment using the deployment this test used.

        Show
        Brian Stansberry
        added a comment - Please attach or email to me jaxrpc-samples-swa.war. I want to experiment using the deployment this test used.
        Hide
        Brian Stansberry
        added a comment -

        I just deployed and undeployed an app a few hundred times without incident (using upstream master), so I'll definitely need the jaxrpc-samples-swa.war to try and reproduce.

        Show
        Brian Stansberry
        added a comment - I just deployed and undeployed an app a few hundred times without incident (using upstream master), so I'll definitely need the jaxrpc-samples-swa.war to try and reproduce.
        Hide
        Rajesh Rajasekaran
        added a comment -

        Attaching war file used in the test

        Show
        Rajesh Rajasekaran
        added a comment - Attaching war file used in the test
        Rajesh Rajasekaran
        made changes -
        Attachment jaxrpc-samples-swa.war [ 12351665 ]
        Hide
        Jan Martiska
        added a comment -

        If it works in upstream master, you should try it with EAP DR13, that is where I encountered the issue.

        Show
        Jan Martiska
        added a comment - If it works in upstream master, you should try it with EAP DR13, that is where I encountered the issue.
        Hide
        Paul Gier
        added a comment -

        Making this resolved because it works upstream. The fix should have been pulled in the last rebase.

        Show
        Paul Gier
        added a comment - Making this resolved because it works upstream. The fix should have been pulled in the last rebase.
        Paul Gier
        made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Done [ 1 ]
        Hide
        Brian Stansberry
        added a comment -

        Thanks Paul. I tried the attached war over the weekend and can confirm it was working in upstream (at least on my local machine.)

        If this comes up again, please try and get a thread dump from the server under test.

        Show
        Brian Stansberry
        added a comment - Thanks Paul. I tried the attached war over the weekend and can confirm it was working in upstream (at least on my local machine.) If this comes up again, please try and get a thread dump from the server under test.
        Hide
        Jan Martiska
        added a comment -

        I tried with the newly released ER1 and the issue has probably indeed been resolved. I will notify if I stumble upon it again.

        Show
        Jan Martiska
        added a comment - I tried with the newly released ER1 and the issue has probably indeed been resolved. I will notify if I stumble upon it again.
        Jan Martiska
        made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Jan Martiska
        added a comment -

        The issue persists in ER2 and ER3. Will create a reproducer.

        Show
        Jan Martiska
        added a comment - The issue persists in ER2 and ER3. Will create a reproducer.
        Jan Martiska
        made changes -
        Resolution Done [ 1 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Jan Martiska
        made changes -
        Fix Version/s TBD EAP 6 [ 12315096 ]
        Fix Version/s EAP 6.0.0 ER 1 [ 12318305 ]
        Description {quote}
        07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
        07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
        07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
        07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
        07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
        07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
        07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
        07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
        07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
        07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
        07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        {quote}

        ... and then nothing happened anymore.

        Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

        The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

        https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
        This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

        For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.
        {quote}
        07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
        07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
        07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
        07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
        07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
        07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
        07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
        07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
        07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
        07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
        07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        {quote}

        ... and then nothing happened anymore.

        Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

        The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

        https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
        This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

        For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

        If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

        02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


        Affects Version/s EAP 6.0.0 ER 3 [ 12319176 ]
        Affects Version/s EAP 6.0.0 ER 2 [ 12318374 ]
        Jan Martiska
        made changes -
        Description {quote}
        07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:14,378 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:14,379 WARN [org.jboss.as.dependency.private] (MSC service thread 1-16) JBAS018567: Deployment "deployment.deployment.ear5066419734" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
        07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
        07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
        07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:16,140 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:16,141 WARN [org.jboss.as.dependency.private] (MSC service thread 1-12) JBAS018567: Deployment "deployment.deployment.ear287054427" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
        07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
        07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
        07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        07:33:17,860 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ejb3:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.ee:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.ws.spi:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jboss.as.security:main") which may be changed or removed in future versions without notice.
        07:33:17,861 WARN [org.jboss.as.dependency.private] (MSC service thread 1-14) JBAS018567: Deployment "deployment.deployment.ear829511841624" is using a private module ("org.jacorb:main") which may be changed or removed in future versions without notice.
        07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
        07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
        07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
        07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
        07:33:19,618 WARN [org.jboss.as.dependency.private] (MSC service thread 1-3) JBAS018567: Deployment "deployment.deployment.ear498231454" is using a private module ("org.jboss.iiop-client:main") which may be changed or removed in future versions without notice.
        {quote}

        ... and then nothing happened anymore.

        Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

        The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

        https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
        This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

        For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

        If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

        02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


        {quote}
        07:33:13,783 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015876: Starting deployment of "deployment.ear5066419734"
        07:33:14,395 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear5066419734"
        07:33:14,436 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment deployment.ear5066419734 in 22ms
        07:33:14,455 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear5066419734"
        07:33:15,471 INFO [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "deployment.ear287054427"
        07:33:16,157 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear287054427"
        07:33:16,197 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015877: Stopped deployment deployment.ear287054427 in 21ms
        07:33:16,263 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear287054427"
        07:33:17,278 INFO [org.jboss.as.server.deployment] (MSC service thread 1-10) JBAS015876: Starting deployment of "deployment.ear829511841624"
        07:33:17,876 INFO [org.jboss.as.server] (management-handler-threads - 201) JBAS018559: Deployed "deployment.ear829511841624"
        07:33:17,914 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment deployment.ear829511841624 in 20ms
        07:33:17,935 INFO [org.jboss.as.server] (management-handler-threads - 202) JBAS018558: Undeployed "deployment.ear829511841624"
        07:33:18,950 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) JBAS015876: Starting deployment of "deployment.ear498231454"
        {quote}

        ... and then nothing happened anymore.

        Deployment service freezes after multiple deployments and undeployments of the same file (the names in this log are different because it is always generated randomly). It happened on Sun JDK 6 (both 32bit and 64bit) and OpenJDK 6. Tried with different deployment files, from the most basic EJB3 jar to a 58 megabyte EAR containing 5 different applications.

        The server keeps running and doesn't complain about any memory issues, but trying to deploy anything will get you stuck. For example the 'deploy' operation in CLI will never return, only a "JBAS014900: Content added at location......." will appear in server.log (and not every time).

        https://hudson.qa.jboss.com/hudson/job/eap6-deployment-soak-test-memory-leaks-32bit/ - see the console logs of runs #13 to #19.
        This test runs a server and keeps deploying+undeploying an application while always waiting for a log in server.log that confirms successful deployment.

        For bigger files, this happens sooner - after about 12 deploy/undeploy operations, for smaller applications it happens after about 20.

        If you try to deploy a file by copying it into "deployments" directory, a .isdeploying marker will be created, and after a while this log appears in server.log:

        02:45:40,527 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.


        Hide
        Jan Martiska
        added a comment -

        Reproducer for this issue. Instructions are in file named HOW_TO_RUN_REPRODUCER ..

        Show
        Jan Martiska
        added a comment - Reproducer for this issue. Instructions are in file named HOW_TO_RUN_REPRODUCER ..
        Jan Martiska
        made changes -
        Attachment jbpapp8124-reproducer.zip [ 12352372 ]
        Hide
        Jan Martiska
        added a comment -

        Thread dump of the server under test, after this issue occurs.

        Show
        Jan Martiska
        added a comment - Thread dump of the server under test, after this issue occurs.
        Jan Martiska
        made changes -
        Attachment threaddump [ 12352373 ]
        Hide
        Brian Stansberry
        added a comment -

        Thanks for the details. I ran the reproducer on my laptop against upstream/master for 250 cycles without incident. It seems to essentially be doing what I did manually before and at about the same pace, so it doesn't surprise me that it works. I'll try on a different machine later.

        The thread dump has a big clue. My assumption is a second thread dump would show this thread to still be in the same state. It would be very unusual for a thread to be in org.jboss.as.webservices.service.EndpointService.start() long after deployment began if it wasn't stuck for some reason. If in fact this thread is not progressing, that is the root of the problem:

        Thread 13287: (state = IN_NATIVE)

        • java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0 (Interpreted frame)
        • java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=260 (Interpreted frame)
        • java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 (Interpreted frame)
        • java.io.BufferedOutputStream.write(int) @bci=13, line=78 (Compiled frame)
        • java.io.PrintStream.write(int) @bci=13, line=399 (Compiled frame)
        • org.jboss.logmanager.handlers.UncloseableOutputStream.write(int) @bci=5, line=39 (Compiled frame)
        • java.io.FilterOutputStream.write(int) @bci=5, line=60 (Compiled frame)
        • org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(int) @bci=4, line=59 (Compiled frame)
        • java.io.FilterOutputStream.write(byte[], int, int) @bci=41, line=108 (Compiled frame)
        • org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(byte[], int, int) @bci=11, line=87 (Interpreted frame)
        • sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Interpreted frame)
        • sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Interpreted frame)
        • sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Interpreted frame)
        • sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Interpreted frame)
        • java.io.OutputStreamWriter.flush() @bci=4, line=212 (Interpreted frame)
        • org.jboss.logmanager.handlers.WriterHandler.safeFlush(java.io.Flushable) @bci=5, line=162 (Interpreted frame)
        • org.jboss.logmanager.handlers.WriterHandler.flush() @bci=12, line=133 (Interpreted frame)
        • org.jboss.logmanager.ExtHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=8, line=92 (Interpreted frame)
        • org.jboss.logmanager.handlers.WriterHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=81, line=70 (Interpreted frame)
        • org.jboss.logmanager.ExtHandler.publish(java.util.logging.LogRecord) @bci=17, line=64 (Interpreted frame)
        • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=26, line=283 (Compiled frame)
        • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
        • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
        • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
        • org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame)
        • org.jboss.logmanager.Logger.logRaw(org.jboss.logmanager.ExtLogRecord) @bci=81, line=649 (Interpreted frame)
        • org.jboss.logmanager.Logger.log(java.lang.String, java.util.logging.Level, java.lang.String, org.jboss.logmanager.ExtLogRecord$FormatStyle, java.lang.Object[], java.lang.Throwable) @bci=70, line=600 (Compiled frame)
        • org.jboss.logging.JBossLogManagerLogger.doLogf(org.jboss.logging.Logger$Level, java.lang.String, java.lang.String, java.lang.Object[], java.lang.Throwable) @bci=40, line=54 (Interpreted frame)
        • org.jboss.logging.Logger.logf(java.lang.String, org.jboss.logging.Logger$Level, java.lang.Throwable, java.lang.String, java.lang.Object) @bci=23, line=2096 (Interpreted frame)
        • org.jboss.as.webservices.WSLogger_$logger.starting(java.lang.Object) @bci=34, line=373 (Interpreted frame)
        • org.jboss.as.webservices.service.EndpointService.start(org.jboss.msc.service.StartContext) @bci=7, line=97 (Interpreted frame)
        • org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(org.jboss.msc.service.Service, org.jboss.msc.service.StartContext) @bci=13, line=1811 (Compiled frame)
        • org.jboss.msc.service.ServiceControllerImpl$StartTask.run() @bci=95, line=1746 (Compiled frame)
        • java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Compiled frame)
        • java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Compiled frame)
        • java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

        So, the thread is blocking trying to write a log message to a file. Specifically, INFO [org.jboss.as.webservices] (MSC service thread 1-1) JBAS015539: Starting service jboss.ws.endpoint."jaxrpc-samples-swa.war".AttachmentJSE

        I believe some of the earlier reports showed progress stopping at different points in the deploy/undeploy cycle. That is, the last log message written before progress halted varied. But my expectation is in all cases the root issue was the same – something is preventing the writing of the log.

        Show
        Brian Stansberry
        added a comment - Thanks for the details. I ran the reproducer on my laptop against upstream/master for 250 cycles without incident. It seems to essentially be doing what I did manually before and at about the same pace, so it doesn't surprise me that it works. I'll try on a different machine later. The thread dump has a big clue. My assumption is a second thread dump would show this thread to still be in the same state. It would be very unusual for a thread to be in org.jboss.as.webservices.service.EndpointService.start() long after deployment began if it wasn't stuck for some reason. If in fact this thread is not progressing, that is the root of the problem: Thread 13287: (state = IN_NATIVE) java.io.FileOutputStream.writeBytes(byte[], int, int) @bci=0 (Interpreted frame) java.io.FileOutputStream.write(byte[], int, int) @bci=4, line=260 (Interpreted frame) java.io.BufferedOutputStream.flushBuffer() @bci=20, line=65 (Interpreted frame) java.io.BufferedOutputStream.write(int) @bci=13, line=78 (Compiled frame) java.io.PrintStream.write(int) @bci=13, line=399 (Compiled frame) org.jboss.logmanager.handlers.UncloseableOutputStream.write(int) @bci=5, line=39 (Compiled frame) java.io.FilterOutputStream.write(int) @bci=5, line=60 (Compiled frame) org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(int) @bci=4, line=59 (Compiled frame) java.io.FilterOutputStream.write(byte[], int, int) @bci=41, line=108 (Compiled frame) org.jboss.logmanager.handlers.UninterruptibleOutputStream.write(byte[], int, int) @bci=11, line=87 (Interpreted frame) sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=202 (Interpreted frame) sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=272 (Interpreted frame) sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=276 (Interpreted frame) sun.nio.cs.StreamEncoder.flush() @bci=12, line=122 (Interpreted frame) java.io.OutputStreamWriter.flush() @bci=4, line=212 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.safeFlush(java.io.Flushable) @bci=5, line=162 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.flush() @bci=12, line=133 (Interpreted frame) org.jboss.logmanager.ExtHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=8, line=92 (Interpreted frame) org.jboss.logmanager.handlers.WriterHandler.doPublish(org.jboss.logmanager.ExtLogRecord) @bci=81, line=70 (Interpreted frame) org.jboss.logmanager.ExtHandler.publish(java.util.logging.LogRecord) @bci=17, line=64 (Interpreted frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=26, line=283 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.LoggerNode.publish(org.jboss.logmanager.ExtLogRecord) @bci=63, line=291 (Compiled frame) org.jboss.logmanager.Logger.logRaw(org.jboss.logmanager.ExtLogRecord) @bci=81, line=649 (Interpreted frame) org.jboss.logmanager.Logger.log(java.lang.String, java.util.logging.Level, java.lang.String, org.jboss.logmanager.ExtLogRecord$FormatStyle, java.lang.Object[], java.lang.Throwable) @bci=70, line=600 (Compiled frame) org.jboss.logging.JBossLogManagerLogger.doLogf(org.jboss.logging.Logger$Level, java.lang.String, java.lang.String, java.lang.Object[], java.lang.Throwable) @bci=40, line=54 (Interpreted frame) org.jboss.logging.Logger.logf(java.lang.String, org.jboss.logging.Logger$Level, java.lang.Throwable, java.lang.String, java.lang.Object) @bci=23, line=2096 (Interpreted frame) org.jboss.as.webservices.WSLogger_$logger.starting(java.lang.Object) @bci=34, line=373 (Interpreted frame) org.jboss.as.webservices.service.EndpointService.start(org.jboss.msc.service.StartContext) @bci=7, line=97 (Interpreted frame) org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(org.jboss.msc.service.Service, org.jboss.msc.service.StartContext) @bci=13, line=1811 (Compiled frame) org.jboss.msc.service.ServiceControllerImpl$StartTask.run() @bci=95, line=1746 (Compiled frame) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Compiled frame) java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Compiled frame) java.lang.Thread.run() @bci=11, line=662 (Interpreted frame) So, the thread is blocking trying to write a log message to a file. Specifically, INFO [org.jboss.as.webservices] (MSC service thread 1-1) JBAS015539: Starting service jboss.ws.endpoint."jaxrpc-samples-swa.war".AttachmentJSE I believe some of the earlier reports showed progress stopping at different points in the deploy/undeploy cycle. That is, the last log message written before progress halted varied. But my expectation is in all cases the root issue was the same – something is preventing the writing of the log.
        Hide
        Jan Martiska
        added a comment -

        This is really weird. I had another look at it and it really seems it freezes in the middle of FileOutputStream.writeBytes while writing the log, having obtained a lock when entering org.jboss.logmanager.handlers.WriterHandler.doPublish - and any subsequent attempt to deploy more files will be blocked waiting for this lock. It was questioned whether this can be an NFS issue, but it does the same on my local filesystem. Never seen anything like this

        Show
        Jan Martiska
        added a comment - This is really weird. I had another look at it and it really seems it freezes in the middle of FileOutputStream.writeBytes while writing the log, having obtained a lock when entering org.jboss.logmanager.handlers.WriterHandler.doPublish - and any subsequent attempt to deploy more files will be blocked waiting for this lock. It was questioned whether this can be an NFS issue, but it does the same on my local filesystem. Never seen anything like this
        Hide
        Jan Martiska
        added a comment -

        And yes, it happens in different phases of deployment, that is - last message written to the log varies.

        Show
        Jan Martiska
        added a comment - And yes, it happens in different phases of deployment, that is - last message written to the log varies.
        Rajesh Rajasekaran
        made changes -
        Labels eap6_need_triage
        Hide
        Rajesh Rajasekaran
        added a comment -

        This issue is repetitive with every quickstart application that we tried.
        https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Deployment-Soak-Tests/
        Interestingly for each quick start it freezes after a fixed number of iterations (different for different apps) as seen in the hudson console log.

        Show
        Rajesh Rajasekaran
        added a comment - This issue is repetitive with every quickstart application that we tried. https://hudson.qa.jboss.com/hudson/view/EAP6/view/EAP6-Deployment-Soak-Tests/ Interestingly for each quick start it freezes after a fixed number of iterations (different for different apps) as seen in the hudson console log.
        Hide
        Jan Martiska
        added a comment - - edited

        Even if I disable all logging on the server completely, it still fails in the same method (writeBytes). I have no idea what file it is trying to write to. If I list open files ('lsof' command) after EAP freezes, it has no files open for writing, except for boot.log, which seems to be default for all logging if all other logging is disabled.

        Show
        Jan Martiska
        added a comment - - edited Even if I disable all logging on the server completely, it still fails in the same method (writeBytes). I have no idea what file it is trying to write to. If I list open files ('lsof' command) after EAP freezes, it has no files open for writing, except for boot.log, which seems to be default for all logging if all other logging is disabled.
        Hide
        Brian Stansberry
        added a comment -

        James Perkins is looking further into this.

        Show
        Brian Stansberry
        added a comment - James Perkins is looking further into this.
        Brian Stansberry
        made changes -
        Assignee Brian Stansberry [ brian.stansberry ] James Perkins [ jamezp ]
        Hide
        James Perkins
        added a comment -

        It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled.

        The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as:

        class StreamGobbler extends Thread {
            private final InputStream inputStream;
        
            StreamGobbler(final InputStream inputStream) {
                this.inputStream = inputStream;
            }
        
            static void startFor(final Process process) {
                new StreamGobbler(process.getErrorStream()).start();
                new StreamGobbler(process.getInputStream()).start();
            }
        
            @Override
            public void run() {
                try {
                    final BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream));
                    while (reader.readLine() != null){}
                } catch (IOException ioe) {
                    ioe.printStackTrace();
                }
            }
        }
        

        Then in the groovy code you just need to execute it:

        p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG)
        StreamGobbler.startFor(p)
        

        That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.

        Show
        James Perkins
        added a comment - It seems the issue lies in the test. The test creates a new process, but doesn't handle the standard output and error output from the process. This leads to the streams overflowing, but it's hidden since it isn't handled. The processes error stream and input stream need to be handled. This is usually done with a StreamGobbler of some sort. For a process it can be something as simple as: class StreamGobbler extends Thread { private final InputStream inputStream; StreamGobbler( final InputStream inputStream) { this .inputStream = inputStream; } static void startFor( final Process process) { new StreamGobbler(process.getErrorStream()).start(); new StreamGobbler(process.getInputStream()).start(); } @Override public void run() { try { final BufferedReader reader = new BufferedReader( new InputStreamReader(inputStream)); while (reader.readLine() != null ){} } catch (IOException ioe) { ioe.printStackTrace(); } } } Then in the groovy code you just need to execute it: p = EapCommonTools.runStandaloneEAP(jbossHome, STANDALONE_CONFIG) StreamGobbler.startFor(p) That will just ignore any output that has been written to the console from JBoss AS. You could alternatively print the streams out, but I think for the test case ignoring the output is probably sufficient.
        Hide
        James Perkins
        added a comment -

        Bug appears to be in the test case and not the server code.

        Show
        James Perkins
        added a comment - Bug appears to be in the test case and not the server code.
        James Perkins
        made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Rejected [ 2 ]
        Hide
        Jan Martiska
        added a comment -

        Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn

        Show
        Jan Martiska
        added a comment - Wow, that really worked. Hmm, what to say about this, it is quite embarrassing for me. Sorry for confusion and huge thanks to James/Brian/Rajesh for wasting time with it, I owe you! I still have much to learn
        Jan Martiska
        made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Paul Gier
        made changes -
        Resolution Rejected [ 2 ]
        Status Closed [ 6 ] Reopened [ 4 ]
        Hide
        Paul Gier
        added a comment -

        Moving out of TBD.

        Show
        Paul Gier
        added a comment - Moving out of TBD.
        Paul Gier
        made changes -
        Status Reopened [ 4 ] Closed [ 6 ]
        Labels eap6_need_triage
        Fix Version/s EAP 6 No Release [ 12318379 ]
        Fix Version/s TBD EAP 6 [ 12315096 ]
        Resolution Rejected [ 2 ]

          People

          • Assignee:
            James Perkins
            Reporter:
            Jan Martiska
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: