Uploaded image for project: 'JBoss Enterprise Application Platform 4 and 5'
  1. JBoss Enterprise Application Platform 4 and 5
  2. JBPAPP-4094

Don't call expire when processing remote invalidation of clustered session

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 4.3.0.GA_CP09, 4.2.0.GA_CP10
    • 4.2.0.GA_CP08, 4.3.0.GA_CP07, 4.2.0.GA_CP09, 4.3.0.GA_CP08
    • Clustering, System
    • None
    • Workaround Exists
    • Hide

      Remove FC from JGroups stack config If it's synchronous replication, or the data traffic is not very high

      Show
      Remove FC from JGroups stack config If it's synchronous replication, or the data traffic is not very high

    Description

      When JBC notifies JBossCacheManager that the root node for a session has been invalidated via a remote call, JBMC.processRemoteInvalidation() calls Session.expire on the local session (if there is one).
      This expire call is problematic, as internally it synchronizes on the session. Users experiencing problems with session affinity can experience deadlocks due to this:

      1) Thread T1 on node 1 modifies session and thus is replicating it, which synchronizes on session.
      2) T1 gets to JGroups FC which blocks waiting for credits from node 2.
      3) Thread T2 on node 2 concurrently expires the session or attempts to gravitate it (T2 could be a background processing thread cleaning old sessions, or a user request thread that invalidates the session or needs to gravitate it if buddy replication is used.) This activity by T2 results in a JBC remove() call being sent to the network
      4) The JGroups up handler thread UT on node 1 carries the JBC remove message from node 2. It gets to the processRemoteInvalidation call which attempts to synchronize on session, which blocks due to T1 holding the monitor from step 1.
      5) Node 2 sends FC credits to node 1. Those credits cannot be delivered, because the thread that would deliver them, UT, is blocking in step 4. Deadlock.

      Note that this deadlock situation doesn't occur with EAP 5 because there is more than one thread carrying messages up, and FC credits are sent OOB, so the credits in step 5 will arrive, the replication in step 1 will complete, and the UT thread in step 4 will then be able to acquire the session object monitor.

      The session.expire call includes flags that result in no notifications being sent to any listeners. I don't see any point to this expire call if there are no notifications. The other work expire does is removing content from JBC, but the remote invalidation that triggers all this is already removing that content. Just drop the session from the local session map. Perhaps flag the session so if there is a concurrency/sticky-session problem and a local request thread is handling the session it knows the session is invalid.

      Attachments

        Issue Links

          Activity

            People

              bstansbe@redhat.com Brian Stansberry
              bstansbe@redhat.com Brian Stansberry
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: