Uploaded image for project: 'Red Hat Process Automation Manager'
  1. Red Hat Process Automation Manager
  2. RHPAM-172

ConcurrentModificationException in hibernate with SINGLETON deployments under load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 7.0.0.GA
    • 6.x.x
    • jBPM Core

      ConcurrentModificationException is thrown from business-central REST API under load

      16:36:12,786 WARN  [com.arjuna.ats.arjuna] (http-127.0.0.1:8080-24) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff7f000001:-7900c1fe:5a28eab5:2b017, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@34a6c813 >: javax.persistence.PersistenceException: error during managed flush
        at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1515) [hibernate-entitymanager-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1211)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:173)
        at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:245) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:71) [jbpm-human-task-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.persistence.jta.TransactionLockInterceptor.execute(TransactionLockInterceptor.java:73) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.persistence.jpa.OptimisticLockRetryInterceptor.execute(OptimisticLockRetryInterceptor.java:82) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) [jbpm-human-task-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.impl.command.CommandBasedTaskService.execute(CommandBasedTaskService.java:157) [jbpm-human-task-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.runtime.manager.impl.task.SynchronizedTaskService.execute(SynchronizedTaskService.java:851) [jbpm-runtime-manager-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.kie.services.impl.UserTaskServiceImpl.execute(UserTaskServiceImpl.java:952) [jbpm-kie-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.cdi.impl.UserTaskServiceCDIImpl$Proxy$_$$_WeldClientProxy.execute(UserTaskServiceCDIImpl$Proxy$_$$_WeldClientProxy.java) [jbpm-services-cdi-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean.doTaskOperation(ProcessRequestBean.java:410) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean.processCommand(ProcessRequestBean.java:194) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean$Proxy$_$$_WeldClientProxy.processCommand(ProcessRequestBean$Proxy$_$$_WeldClientProxy.java) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.util.ExecuteCommandUtil.restProcessJaxbCommandsRequest(ExecuteCommandUtil.java:61) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.rest.ExecuteResourceImpl.execute(ExecuteResourceImpl.java:60) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.rest.ExecuteResourceImpl$Proxy$_$$_WeldClientProxy.execute(ExecuteResourceImpl$Proxy$_$$_WeldClientProxy.java) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source) [:1.8.0_66]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:168) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:560) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:542) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:130) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.kie.remote.services.rest.jaxb.DynamicJaxbContextFilter.doFilter(DynamicJaxbContextFilter.java:67) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.BasicAuthSecurityFilter.doFilter(BasicAuthSecurityFilter.java:70) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.SecureHeadersFilter.doFilter(SecureHeadersFilter.java:69) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.SecurityIntegrationFilter.doFilter(SecurityIntegrationFilter.java:57) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:400) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
Caused by: java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) [rt.jar:1.8.0_66]
        at java.util.ArrayList$Itr.next(ArrayList.java:851) [rt.jar:1.8.0_66]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:386) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:303) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        ... 69 more

      Attached server.log.ConcurrentModificationException.zip (Please ignore lots of PermissionDeniedException... the load test is pretty rude).

      After that, some side effects are observed which caused unexpected process instance state (probably EntityManager related) so we need to reboot BPMS.

      I reproduced the issue with the conditions below:

      • 3 deployments in business-central "Deploy"->"Process Deployments"
      • All deployments are SINGLETON
      • Processes have some "isAsync = true" Tasks
      • Access with REST API
      • kie-server is not deployed (so don't need to worry about kie-server's executor)
      • org.kie.executor.pool.size=10
      • org.kie.executor.interval=1
      • 10 thread REST clients which start a process and complete a Task
      • Oracle 11g (I could reproduce with MySQL as well but was disturbed by more other issues like lock timeout. I also reproduced with H2 once but not reliable)

      Attached reproducer01.zip

        1. reproducer01.zip
          261 kB
        2. reproducer02.zip
          324 kB
        3. server.log.ConcurrentModificationException.zip
          1.20 MB

            swiderski.maciej Maciej Swiderski (Inactive)
            rhn-support-tkobayas Toshiya Kobayashi
            Marian Macik Marian Macik
            Marian Macik Marian Macik
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: