Uploaded image for project: 'JBoss BPMS Platform'
  1. JBoss BPMS Platform
  2. RHBPMS-5085

[GSS] (6.4.z) ConcurrentModificationException in hibernate with SINGLETON deployments under load

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 6.4.8
    • 6.4.7
    • jBPM Core
    • CR1

    Description

      ConcurrentModificationException is thrown from business-central REST API under load

      16:36:12,786 WARN  [com.arjuna.ats.arjuna] (http-127.0.0.1:8080-24) ARJUNA012125: TwoPhaseCoordinator.beforeCompletion - failed for SynchronizationImple< 0:ffff7f000001:-7900c1fe:5a28eab5:2b017, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@34a6c813 >: javax.persistence.PersistenceException: error during managed flush
        at org.hibernate.ejb.AbstractEntityManagerImpl$CallbackExceptionMapperImpl.mapManagedFlushFailure(AbstractEntityManagerImpl.java:1515) [hibernate-entitymanager-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:114) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization.beforeCompletion(RegisteredSynchronization.java:53) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.beforeCompletion(SynchronizationImple.java:76)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.beforeCompletion(TwoPhaseCoordinator.java:273)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:93)
        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1211)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.commit(ServerVMClientUserTransaction.java:173)
        at org.drools.persistence.jta.JtaTransactionManager.commit(JtaTransactionManager.java:245) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:71) [jbpm-human-task-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.persistence.jta.TransactionLockInterceptor.execute(TransactionLockInterceptor.java:73) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41) [drools-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.drools.persistence.jpa.OptimisticLockRetryInterceptor.execute(OptimisticLockRetryInterceptor.java:82) [drools-persistence-jpa-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40) [jbpm-human-task-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.task.impl.command.CommandBasedTaskService.execute(CommandBasedTaskService.java:157) [jbpm-human-task-core-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.runtime.manager.impl.task.SynchronizedTaskService.execute(SynchronizedTaskService.java:851) [jbpm-runtime-manager-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.kie.services.impl.UserTaskServiceImpl.execute(UserTaskServiceImpl.java:952) [jbpm-kie-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.jbpm.services.cdi.impl.UserTaskServiceCDIImpl$Proxy$_$$_WeldClientProxy.execute(UserTaskServiceCDIImpl$Proxy$_$$_WeldClientProxy.java) [jbpm-services-cdi-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean.doTaskOperation(ProcessRequestBean.java:410) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean.processCommand(ProcessRequestBean.java:194) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.cdi.ProcessRequestBean$Proxy$_$$_WeldClientProxy.processCommand(ProcessRequestBean$Proxy$_$$_WeldClientProxy.java) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.util.ExecuteCommandUtil.restProcessJaxbCommandsRequest(ExecuteCommandUtil.java:61) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.rest.ExecuteResourceImpl.execute(ExecuteResourceImpl.java:60) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.kie.remote.services.rest.ExecuteResourceImpl$Proxy$_$$_WeldClientProxy.execute(ExecuteResourceImpl$Proxy$_$$_WeldClientProxy.java) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source) [:1.8.0_66]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:168) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:560) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:542) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:130) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.15.Final-redhat-1.jar:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.kie.remote.services.rest.jaxb.DynamicJaxbContextFilter.doFilter(DynamicJaxbContextFilter.java:67) [kie-remote-services-6.5.0.Final-redhat-12.jar:6.5.0.Final-redhat-12]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.BasicAuthSecurityFilter.doFilter(BasicAuthSecurityFilter.java:70) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.SecureHeadersFilter.doFilter(SecureHeadersFilter.java:69) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.uberfire.ext.security.server.SecurityIntegrationFilter.doFilter(SecurityIntegrationFilter.java:57) [uberfire-servlet-security-0.9.0.Final-redhat-10.jar:0.9.0.Final-redhat-10]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:420) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.11.Final-redhat-1.jar:7.5.11.Final-redhat-1]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:400) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.19.Final-redhat-1.jar:7.5.19.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]
Caused by: java.util.ConcurrentModificationException
        at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901) [rt.jar:1.8.0_66]
        at java.util.ArrayList$Itr.next(ArrayList.java:851) [rt.jar:1.8.0_66]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:386) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:303) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:349) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1195) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallbackCoordinatorNonTrackingImpl.java:109) [hibernate-core-4.2.24.Final-redhat-1.jar:4.2.24.Final-redhat-1]
        ... 69 more

      Attached server.log.ConcurrentModificationException.zip (Please ignore lots of PermissionDeniedException... the load test is pretty rude).

      After that, some side effects are observed which caused unexpected process instance state (probably EntityManager related) so we need to reboot BPMS.

      I reproduced the issue with the conditions below:

      • 3 deployments in business-central "Deploy"->"Process Deployments"
      • All deployments are SINGLETON
      • Processes have some "isAsync = true" Tasks
      • Access with REST API
      • kie-server is not deployed (so don't need to worry about kie-server's executor)
      • org.kie.executor.pool.size=10
      • org.kie.executor.interval=1
      • 10 thread REST clients which start a process and complete a Task
      • Oracle 11g (I could reproduce with MySQL as well but was disturbed by more other issues like lock timeout. I also reproduced with H2 once but not reliable)

      Attached reproducer01.zip

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. RHPAM-172 ConcurrentModificationException in hibernate with SINGLETON deployments under load

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              swiderski.maciej Maciej Swiderski (Inactive)
              rhn-support-tkobayas Toshiya Kobayashi
              Marian Macik Marian Macik
              Marian Macik Marian Macik
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: