Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-6665

clustering-web-infinispan: missing error handling when unmarshalling session

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 10.0.0.Final
    • Clustering
    • None
    • Hide
      1. build war archive from https://github.com/podlesh/unmarshall-error-war
        • mvn clean package
      2. deploy to wildfly AS (via console or just by copying to standalone/deployment)
        • tested on default configuration (just unpacked downloaded archive)
      3. start server (./bin/standalone.sh)
      4. open browser, http://localhost:8080/unmarshall-error/bad2.jsp
        • it should display message with session ID
      5. shutdown server (ctrl-C)
      6. start server again
      7. reload page (http://localhost:8080/unmarshall-error/bad2.jsp)
        • response is 500 Internal Error
      8. repeated request to this url (or any other under /unmarshall-error/ ) ends with the same error
        • as long as the request contains JSESSIONID cookie
      Show
      build war archive from https://github.com/podlesh/unmarshall-error-war mvn clean package deploy to wildfly AS (via console or just by copying to standalone/deployment) tested on default configuration (just unpacked downloaded archive) start server (./bin/standalone.sh) open browser, http://localhost:8080/unmarshall-error/bad2.jsp it should display message with session ID shutdown server (ctrl-C) start server again reload page ( http://localhost:8080/unmarshall-error/bad2.jsp ) response is 500 Internal Error repeated request to this url (or any other under /unmarshall-error/ ) ends with the same error as long as the request contains JSESSIONID cookie

      When unmarshalling of session throws unexpected exception, it is not handled correctly and is propagated to undertow worker, thus causing generic http 500 error to be shown to user. Problem is that each request tries to unmarshall session again and fails... making this session completely broken (for with no recovery until session cookie expires or is removed (ie browser closed).

      Some exceptions are expected and handled correctly by removing session and continuing (usually, new session is created). They are catched in MarshalledValueMarshaller.read, wrapped by InvalidSerializedFormException and then handled in CoarseSessionAttributesFactory.findValue. Expected exceptions are these:

      • java.lang.ClassNotFoundException
      • java.io.InvalidClassException
      • java.io.InvalidObjectException
        This list is sufficient in most of the time, but sometimes even jboss-marshalling-river throws some RuntimeException (not to mention IOException).

      Sample stacktrace (there are several possible variants):

      14:26:14,366 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.lang.IllegalArgumentException] while invoking method [public void org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.activated(org.infinispan.notifications.cachelistener.event.CacheEntryActivatedEvent)] on listener instance: org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager@25e2eac0
      	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:291)
      	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:21)
      	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
      	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1234)
      	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1229)
      	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1206)
      	at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryActivated(CacheNotifierImpl.java:522)
      	at org.infinispan.interceptors.ActivationInterceptor.sendNotification(ActivationInterceptor.java:33)
      	at org.infinispan.interceptors.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:392)
      	at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:362)
      	at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:183)
      	at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:137)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:133)
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:123)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
      	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71)
      	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      	at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:345)
      	at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:330)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76)
      	at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
      	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
      	at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411)
      	at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403)
      	at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:92)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:69)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:39)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:61)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:234)
      	at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:140)
      	at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:726)
      	at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:756)
      	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:69)
      	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.IllegalArgumentException: always failing!
      	at eu.lmc.experiment.FailingDeserialization2.readObject(FailingDeserialization2.java:36)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:307)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1637)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1285)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
      	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
      	at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
      	at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
      	at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:365)
      	at java.util.HashMap.readObject(HashMap.java:1396)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:307)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1637)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1285)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
      	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
      	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
      	at org.wildfly.clustering.marshalling.jboss.SimpleMarshalledValue.get(SimpleMarshalledValue.java:101)
      	at org.wildfly.clustering.marshalling.jboss.SimpleMarshalledValue.get(SimpleMarshalledValue.java:44)
      	at org.wildfly.clustering.marshalling.jboss.MarshalledValueMarshaller.read(MarshalledValueMarshaller.java:45)
      	at org.wildfly.clustering.marshalling.jboss.MarshalledValueMarshaller.read(MarshalledValueMarshaller.java:32)
      	at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionAttributesFactory.findValue(CoarseSessionAttributesFactory.java:84)
      	at org.wildfly.clustering.web.infinispan.session.coarse.CoarseSessionAttributesFactory.findValue(CoarseSessionAttributesFactory.java:48)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:63)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:40)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.lambda$activated$24(InfinispanSessionManager.java:300)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager$$Lambda$129/153353148.run(Unknown Source)
      	at org.wildfly.clustering.service.concurrent.StampedLockServiceExecutor.execute(StampedLockServiceExecutor.java:42)
      	at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.activated(InfinispanSessionManager.java:297)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
      	... 62 more
      Caused by: an exception which occurred:
      	in object of type eu.lmc.experiment.FailingDeserialization2
      	in object of type java.util.HashMap
      

      Note 1: application must be marked as "distributable" in web.xml or equivalent.

      Note 2: in 9.0.2, this bug is way more serious because handling of "expected" exceptions is completely broken and ends up in infinite recursion. So in addition to completely broken session, server logs is spammed by 40 megabytes of stacktraces per each request.

            pferraro@redhat.com Paul Ferraro
            podlesakk Kamil Podlešák (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: