Uploaded image for project: 'JBoss BPMS Platform'
  1. JBoss BPMS Platform
  2. RHBPMS-4548

RuntimeDataService 'getTaskAssignedAsPotentialOwner' runs single threaded and therefore doesn't scale.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 6.4.2
    • 6.4.0
    • jBPM Core
    • None

    • Red Hat JBoss BPM Suite 6.4.0, PostgreSQL 9, RHEL 7, macOS 10.11.6

    • CR1
    • Hide

      1) Run a load test against the "getTaskAssignedAsPotentialOwner" query on the RuntimeDataResource.
      2) Observe that only a single connection (Active Connection Count) is used from the datasource connection pool on EAP.
      3) Observe that the (PostgreSQL) database only uses a single process on a single core (which it maxes out) as it only uses a single connection.

      Show
      1) Run a load test against the "getTaskAssignedAsPotentialOwner" query on the RuntimeDataResource. 2) Observe that only a single connection (Active Connection Count) is used from the datasource connection pool on EAP. 3) Observe that the (PostgreSQL) database only uses a single process on a single core (which it maxes out) as it only uses a single connection.

    Description

      The RESTful search "org.kie.server.remote.rest.jbpm.RuntimeDataResource.getTasksAssignedAsPotentialOwner" effectively runs single threaded due to a synchronized "execute" method in the TaskTransactionInterceptor.

      When running a performance test on that RESTful resource (100 threads, looping), we can see in the database stats (PostgreSQL on EAP 7) that only a single database connection is used. At the same time, a single PostgreSQL process is maxing out on one core. As PostgreSQL uses a OS process per connection, but the system only uses one connection, the database starts struggling. This dramatically decrease the throughput and performance of the system.

      We've created some thread dumps, and we can clearly see that all threads are blocking behind this one:

      "default task-62" #310 prio=5 os_prio=0 tid=0x00007feeb8010800 nid=0x3fd4 runnable [0x00007fee36ebe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
	- locked <0x0000000643860da8> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery(WrappedPreparedStatement.java:504)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:79)
	at org.hibernate.loader.Loader.getResultSet(Loader.java:2090)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1866)
	at org.hibernate.loader.Loader.doQuery(Loader.java:905)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:347)
	at org.hibernate.loader.Loader.doList(Loader.java:2578)
	at org.hibernate.loader.Loader.doList(Loader.java:2564)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2394)
	at org.hibernate.loader.Loader.list(Loader.java:2389)
	at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)
	at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)
	at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:198)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1230)
	at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
	at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268)
	at org.jbpm.services.task.persistence.JPATaskPersistenceContext.queryStringWithParameters(JPATaskPersistenceContext.java:575)
	at org.jbpm.services.task.persistence.JPATaskPersistenceContext.queryWithParametersInTransaction(JPATaskPersistenceContext.java:398)
	at org.jbpm.services.task.impl.TaskQueryServiceImpl.getTasksAssignedAsPotentialOwner(TaskQueryServiceImpl.java:368)
	at org.jbpm.services.task.commands.GetTaskAssignedAsPotentialOwnerCommand.execute(GetTaskAssignedAsPotentialOwnerCommand.java:85)
	at org.jbpm.services.task.commands.GetTaskAssignedAsPotentialOwnerCommand.execute(GetTaskAssignedAsPotentialOwnerCommand.java:31)
	at org.jbpm.services.task.commands.TaskCommandExecutorImpl$SelfExecutionCommandService.execute(TaskCommandExecutorImpl.java:65)
	at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41)
	at org.jbpm.services.task.persistence.TaskTransactionInterceptor.execute(TaskTransactionInterceptor.java:69)
	- locked <0x000000065b1cbc90> (a org.jbpm.services.task.persistence.TaskTransactionInterceptor)
	at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41)
	at org.drools.persistence.jta.TransactionLockInterceptor.execute(TransactionLockInterceptor.java:73)
	at org.drools.core.command.impl.AbstractInterceptor.executeNext(AbstractInterceptor.java:41)
	at org.drools.persistence.jpa.OptimisticLockRetryInterceptor.execute(OptimisticLockRetryInterceptor.java:82)
	at org.jbpm.services.task.commands.TaskCommandExecutorImpl.execute(TaskCommandExecutorImpl.java:40)
	at org.jbpm.services.task.impl.command.CommandBasedTaskService.getTasksAssignedAsPotentialOwner(CommandBasedTaskService.java:229)
	at org.jbpm.kie.services.impl.RuntimeDataServiceImpl.getTasksAssignedAsPotentialOwner(RuntimeDataServiceImpl.java:752)
	at org.kie.server.services.jbpm.RuntimeDataServiceBase.getTasksAssignedAsPotentialOwner(RuntimeDataServiceBase.java:392)
	at org.kie.server.remote.rest.jbpm.RuntimeDataResource.getTasksAssignedAsPotentialOwner(RuntimeDataResource.java:381)
	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:139)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
	at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)
	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
	at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)
	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:207)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:802)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
   Locked ownable synchronizers:
	- <0x00000007716d8b88> (a java.util.concurrent.ThreadPoolExecutor$Worker)
	- <0x00000006438cfec0> (a java.util.concurrent.locks.ReentrantLock$FairSync)

      at the "TaskTransactionInterceptor.execute" method which is synchronized: https://github.com/droolsjbpm/jbpm/blob/master/jbpm-human-task/jbpm-human-task-jpa/src/main/java/org/jbpm/services/task/persistence/TaskTransactionInterceptor.java#L61

      So effectively, this query runs single threaded and does not scale.

      Attachments

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. RHBPMS-4655 Cannot get tasks in the same transaction on WAS and WLS

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. CLOUD-1731 findTasksAssignedAsPotentialOwner doesn't work anymore with usernames on 6.4 if KIE_SERVER_BYPASS_AUTH_USER=true

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • New

          Activity

            People

              swiderski.maciej Maciej Swiderski (Inactive)
              rhn-gps-ddoyle Duncan Doyle (Inactive)
              Karel Suta Karel Suta
              Karel Suta Karel Suta
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: