HotRod-based session manager can leave entry orphans in remote cache if no WF clients are active at the time of expiration

The hotrod-based session manager relies on a single mortal cache entry per session, and uses a client listener to react to expiration events in order to send session destroy events to the application, and to remove the dependent immortal cache entries from infinispan-server.

However, if all WildFly servers are shut down, no WildFly server will receive expiration events, and thus the dependent immortal cache entries are never removed. Additionally, no session destroy events will be triggered for these sessions.

To fix this, there are only so many options:

1. Add logic during session manager startup:
   • Detect the above condition
     • Because hotrod clients have no knowledge of each other, I can think of no other way to do this apart from using JGroups
   • If the server determines that there are no other EAP servers servicing the application (i.e. cluster size is 1), iterate over all immortal metadata entries in cache that lack a mortal metadata entry equivalent, and, for each, trigger a manual (client-side) expiration event. This must be done after registering the normal expiration client listener.
2. Revert back to managing session expiration within WildFly itself.
   • One of the design goals for offloading expiration to infinispan-server was to avoid the need for a JGroups cluster between WildFly servers. If we need one anyway (for option #1), why not just manage expiration within WildFly as we already do for infinispan-session-management? This has a couple of advantages:
     • In the current expiration mechanism, all WildFly servers will receive an expiration event. Consequently, all servers will contend for the same RemoteCache.remove(...) where only one will be successful. This contention prevents the expiration mechanism from scaling well.
     • If we use a JGroups cluster solely for the purpose of membership, we can distribute expiration responsibilities to all WildFly servers based on a consistent hash algorithm. This solves the problem of contention during expiration, since only one server will attempt to expire a session at a time; and redistributed session expiration responsibilities on view change.
   • Considering the above, it would be better to use the default distributed session manager using an invalidation-cache w/shared hotrod store.