Application Server 7
  1. Application Server 7
  2. AS7-4260

Occurences of "Exception acquiring ownership of X (via SharedLocalYieldingClusterLockManager)"

    Details

    • Similar Issues:
      Show 10 results 

      Description

      There are often occurrences of "Exception acquiring ownership of X":

      During shutdown, using dist:

      13:09:28,290 WARN  [org.apache.catalina.session.ManagerBase] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) JBAS018080: Standard expiration of session qLQps+CQJxEi6pJTBSzis0g3 failed; switching to a brute force cleanup. Problem is JBAS018060: Exception acquiring ownership of qLQps+CQJxEi6pJTBSzis0g3
      

      During server crash, using repl:

      12:55:49,556 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (Incoming-2,null) JBAS010248: New cluster view for partition ejb: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@327579f6 delta: -1, merge: false)
      12:55:49,557 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,null) ISPN000094: Received new cluster view: [perf21/ejb|6] [perf21/ejb, perf20/ejb, perf18/ejb]
      12:56:31,694 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-perf20-10.16.90.58-8009-211) Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of MvwC8KGWTuKPjD1-JjLW7URZ
      	at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:528) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.ClusteredSession.expire(ClusteredSession.java:1260) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.ClusteredSession.isValid(ClusteredSession.java:1208) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.ClusteredSession.isValid(ClusteredSession.java:598) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.DistributableSessionManager.add(DistributableSessionManager.java:917) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1407) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:686) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:85) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.apache.catalina.connector.Request.doGetSession(Request.java:2618) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:841) [jbossweb-7.0.13.Final.jar:]
      	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:72) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
      	at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:58) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
      	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:31) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
      	at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:16) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
      	at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:134) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:445) [jbossweb-7.0.13.Final.jar:]
      	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]
      	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010223: Cannot acquire lock //default-host//clusterbench/MvwC8KGWTuKPjD1-JjLW7URZ from cluster
      	at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)
      	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:372)
      	at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:520) [jboss-as-web-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	... 23 more
      
      1. view-perf18.log
        2 kB
        Richard Achmatowicz
      2. view-perf19.log
        55 kB
        Richard Achmatowicz
      3. view-perf20.log
        10 kB
        Richard Achmatowicz
      4. view-perf21.log
        13 kB
        Richard Achmatowicz
      1. SessionTravel.gif
        12 kB

        Issue Links

          Activity

          Hide
          Richard Achmatowicz
          added a comment -

          Still looking into this one. What I can see happening is as follows:
          1. when a failover occurs, a new jvmRoute is being computed by looking up the cache to get the DistributionManager (ii) finding out which addresses own the session key (iii) use the JvmRouteRegistry to look up the new jvmRoute for the key, using the address of the key
          2. the jvmRoute is then adjusted and the request continues processing on the node which it arrived at
          3. when the request is returned, the load balancer then uses the new jvmRoute to process the request

          For example:

          11:49:46,418 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) checkJvmRoute(): check if need to re-route based on JvmRoute. Session id: mdbFou-te-0qzdaY1T6FXyaw.perf21 jvmRoute: perf18
          11:49:46,418 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) handleJvmRoute(): We have detected a failover with different jvmRoute. old one: perf21, new one: perf18. Will reset the session id.
          ...
          11:49:46,420 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) resetSessionId(): changed catalina session to= [mdbFou-te-0qzdaY1T6FXyaw.perf19] old one= [mdbFou-te-0qzdaY1T6FXyaw.perf21]
          

          In the above, the request arrives at perf18, the jvmRoute assigned is perf19 (based on key ownership), but the request processing is completed at perf18 until the request is returned. In processing at perf18, perf18 will have to take the global lock. If for some reason, the global lock were not being released, then the next time the request was processed by perf19, it would not be able to obtain the lock and the exception would occur. This would at least explain how two requests involving the same session were arising.

          Unfortunately, when I add TRACE logging to try to see more of what is going on, the exceptions do not occur.

          Show
          Richard Achmatowicz
          added a comment - Still looking into this one. What I can see happening is as follows: 1. when a failover occurs, a new jvmRoute is being computed by looking up the cache to get the DistributionManager (ii) finding out which addresses own the session key (iii) use the JvmRouteRegistry to look up the new jvmRoute for the key, using the address of the key 2. the jvmRoute is then adjusted and the request continues processing on the node which it arrived at 3. when the request is returned, the load balancer then uses the new jvmRoute to process the request For example: 11:49:46,418 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) checkJvmRoute(): check if need to re-route based on JvmRoute. Session id: mdbFou-te-0qzdaY1T6FXyaw.perf21 jvmRoute: perf18 11:49:46,418 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) handleJvmRoute(): We have detected a failover with different jvmRoute. old one: perf21, new one: perf18. Will reset the session id. ... 11:49:46,420 TRACE [org.jboss.as.web.session.JvmRouteValve] (ajp-perf18-10.16.90.54-8009-361) resetSessionId(): changed catalina session to= [mdbFou-te-0qzdaY1T6FXyaw.perf19] old one= [mdbFou-te-0qzdaY1T6FXyaw.perf21] In the above, the request arrives at perf18, the jvmRoute assigned is perf19 (based on key ownership), but the request processing is completed at perf18 until the request is returned. In processing at perf18, perf18 will have to take the global lock. If for some reason, the global lock were not being released, then the next time the request was processed by perf19, it would not be able to obtain the lock and the exception would occur. This would at least explain how two requests involving the same session were arising. Unfortunately, when I add TRACE logging to try to see more of what is going on, the exceptions do not occur.
          Hide
          Marc Grimme
          added a comment -

          Any news on that bug?
          Is it likely to be fixed with 7.1.3?

          Thanks for you help.

          Show
          Marc Grimme
          added a comment - Any news on that bug? Is it likely to be fixed with 7.1.3? Thanks for you help.
          Hide
          Maksym Gryevtsov
          added a comment -
          Show
          Maksym Gryevtsov
          added a comment - Will https://community.jboss.org/thread/201663?tstart=30 help? Thanks
          Hide
          vishal kumar
          added a comment -

          Hi,

          We are facing the exactly same issue, can you please help in finding out the route cause.

          We are having two jboss in standalone cluster , they are giving this particular issue while using the application.

          Thanks

          Show
          vishal kumar
          added a comment - Hi, We are facing the exactly same issue, can you please help in finding out the route cause. We are having two jboss in standalone cluster , they are giving this particular issue while using the application. Thanks
          Hide
          Tomaz Cerar
          added a comment -

          vishal kumar so upgrade to newer version that has this fixed.

          Show
          Tomaz Cerar
          added a comment - vishal kumar so upgrade to newer version that has this fixed.

            People

            • Assignee:
              Paul Ferraro
              Reporter:
              Radoslav Husar
            • Votes:
              6 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: