FUSE Message Broker
  1. FUSE Message Broker
  2. MB-949

Speeding up consumer transactions with ActiveMQ

    Details

    • Type: Task Task
    • Status: Resolved Resolved
    • Priority: Major Major
    • Resolution: Done
    • Affects Version/s: 5.4.2-fuse-03-00, 5.5.0-fuse-00-00
    • Component/s: broker
    • Labels:
      None
    • Environment:
      Broker:
      ActiveMQ 5.5
      Java 6 (Sun JVM?)
      Sun Solaris ?

      Client:
      ActiveMQ 5.4.2 (client library)
      Camel 2.5.0
      Spring 2.5.6
      Websphere 6.1
      Java 1.5 (IBM JVM ?)
      Sun Solaris ?
    • Similar Issues:
      Show 10 results 

      Description

      A customer is having an issue with an ActiveMQ deployment that is blocking their production roll-out. The deployment that causes trouble consists of:

      • A producer application that reads many small messages from a large file and sends them to a queue. Each message is 2-3 kbytes long. The number of messages read from a file exceeds 100,000.
      • An ActiveMQ broker that hosts the queue that receives those messages. That queue stores messages persistently. The broker is configured to use an amqPersistenceAdapter.
      • The client is a Camel route inside a WebSphere container. The route performs business logic on each message received from the queue. The processing of a message takes roughly 1 second. The route uses Camel transactions to ensure complete processing of each message, and consumes messages from the queue in the transacted mode, 1 message per transaction.

      To ensure processing of large files within reasonable time frames, the consumer is configured to use 80 threads. Initially, the threads were using a connection pool with maxConnections=10 and maximumActive=120. The customer claims that such configuration resulted in overall processing rate of 60 messages per second. However, messages were getting stuck in the pooled sessions after about 70,000 messages were processed.

      I advised them to reduce the pool size to maxConnections=1 and maximumActive=80 (the number of threads) and set the pre-fetch limit to 1. That resolved the problem of stuck messages, but reduced the processing rate to about 13 messages per second. Thread dumps on the client show that most client threads are waiting for the broker to commit their transactions. Broker threads seem to be busy writing to disk. When they disable transactions, the processing rate goes back up, but they cannot do that in production.

      Is it possible to tune the broker or client configuration to speed up transaction commits?

        Activity

        Hide
        Gary Tully
        added a comment -

        There is a problem with concurrent message consumption in a transaction. More connections should give better through put when the disk is slow, but this is not the case due to a fix for another issue. I have reopened this issue to revisit that fix such that it won't impact message consumption

        reopened issue at apache: https://issues.apache.org/jira/browse/AMQ-2868

        On the stuck messages in spring with prefetch=1000, we should be able to configure spring to keep X concurrent consumers active such that messages do not get stuck. Torsten is working out that config as I type. If that is not possible, we may need to implement some workaround to make that possible.

        Short term, I think the solution will be to move away from prefetch=1 via an alternative fix to the missing messages.

        Show
        Gary Tully
        added a comment - There is a problem with concurrent message consumption in a transaction. More connections should give better through put when the disk is slow, but this is not the case due to a fix for another issue. I have reopened this issue to revisit that fix such that it won't impact message consumption reopened issue at apache: https://issues.apache.org/jira/browse/AMQ-2868 On the stuck messages in spring with prefetch=1000, we should be able to configure spring to keep X concurrent consumers active such that messages do not get stuck. Torsten is working out that config as I type. If that is not possible, we may need to implement some workaround to make that possible. Short term, I think the solution will be to move away from prefetch=1 via an alternative fix to the missing messages.
        Hide
        Gary Tully
        added a comment - - edited

        Note re above reopened issue, it only effects the KahaDB store, the AMQ store does not have this problem. Not true, for a given destination there is serialization of transaction writes in the AMQ store also.

        Show
        Gary Tully
        added a comment - - edited Note re above reopened issue, it only effects the KahaDB store, the AMQ store does not have this problem. Not true, for a given destination there is serialization of transaction writes in the AMQ store also.
        Hide
        Gary Tully
        added a comment - - edited

        On Spring and multiple connections.

        With CACHE_NONE (0), multiple connections will be used. With any other cache level, a single connection will be shared by a message listener container so that it can cache the sessions and consumers.
        So for spring with caching, which is good, you need many containers to get many connections in the mix.

        Show
        Gary Tully
        added a comment - - edited On Spring and multiple connections. With CACHE_NONE (0), multiple connections will be used. With any other cache level, a single connection will be shared by a message listener container so that it can cache the sessions and consumers. So for spring with caching, which is good, you need many containers to get many connections in the mix.
        Hide
        Gary Tully
        added a comment - - edited

        I think adding a second route, using a second activemq component is the best way to get concurrency. In this way, spring can manage the concurrent consumers and use a single connection with full caching. I did not find any stuck messages with the default prefetch (1000) 10 concurrent consumers getting through 100,000 messages. The camel context I used, based on the use case, is in svn : http://svn.apache.org/viewvc/activemq/trunk/activemq-camel/src/test/resources/org/apache/activemq/camel/transactedconsume.xml?view=co&revision=1162175&content-type=text%2Fplain&pathrev=1162175

        When we use CACHE_NONE, we don't get any consumer caching from the ActiveMQConnectionPool. I think add a consumer cache makes sense as in enhancement once the prefetch messages are available by matching users with consumers or doing round robin allocation and expiry of pooled consumers.

        To get better scale out performance with multiple concurrent consumer and connections, we need to resolve the synchronization around transaction completion. This requires a rethink of an existing bug fix but makes sense as the store is build for concurrency in this way. This fix would be in KahaDB only. It looks like the synchronization in the AMQ store has been there from inception so it may be too late to fix that.

        Show
        Gary Tully
        added a comment - - edited I think adding a second route, using a second activemq component is the best way to get concurrency. In this way, spring can manage the concurrent consumers and use a single connection with full caching. I did not find any stuck messages with the default prefetch (1000) 10 concurrent consumers getting through 100,000 messages. The camel context I used, based on the use case, is in svn : http://svn.apache.org/viewvc/activemq/trunk/activemq-camel/src/test/resources/org/apache/activemq/camel/transactedconsume.xml?view=co&revision=1162175&content-type=text%2Fplain&pathrev=1162175 When we use CACHE_NONE, we don't get any consumer caching from the ActiveMQConnectionPool. I think add a consumer cache makes sense as in enhancement once the prefetch messages are available by matching users with consumers or doing round robin allocation and expiry of pooled consumers. To get better scale out performance with multiple concurrent consumer and connections, we need to resolve the synchronization around transaction completion. This requires a rethink of an existing bug fix but makes sense as the store is build for concurrency in this way. This fix would be in KahaDB only. It looks like the synchronization in the AMQ store has been there from inception so it may be too late to fix that.
        Hide
        Gary Tully
        added a comment -

        Linking to the reworked apache issue https://issues.apache.org/jira/browse/AMQ-2868 - the reworked fix allows concurrent transactions to block on a batch journal update, which is what we want for maximum scalability.

        Show
        Gary Tully
        added a comment - Linking to the reworked apache issue https://issues.apache.org/jira/browse/AMQ-2868 - the reworked fix allows concurrent transactions to block on a batch journal update, which is what we want for maximum scalability.
        Hide
        Gary Tully
        added a comment -

        This fix will be in the 5.6 release

        Show
        Gary Tully
        added a comment - This fix will be in the 5.6 release
        Hide
        Gary Tully
        added a comment -

        roll this into the next 5.5.x release

        Show
        Gary Tully
        added a comment - roll this into the next 5.5.x release
        Hide
        Gary Tully
        added a comment -

        reopen pending merge to 5.5.x branch

        Show
        Gary Tully
        added a comment - reopen pending merge to 5.5.x branch
        Hide
        Gary Tully
        added a comment -

        fix merged

        Show
        Gary Tully
        added a comment - fix merged

          People

          • Assignee:
            Gary Tully
            Reporter:
            Stan Livitski
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: