Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-361

The lease-database-locker does not work properly if master/slave broker servers are out of sync with Database server

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • JBoss A-MQ 6.1
    • JBoss A-MQ 6.0
    • None
    • None

    Description

      I have an test scenario where I configured two brokers on two separated machines in a jdbc master/slave setup to connect to a backend database using lease-database-locker. So one broker is running on server A and the other broker is running on server B and the backend database is also running on the server B. I deliberately changed system time on one server to be 2 minutes out of sync with the other server to test behavior of the lease-database-locker.Here is the jdbcPersistentAdapter configuration I used:| <persistenceAdapter>|

      <jdbcPersistenceAdapter dataSource="#oracle-ds" lockKeepAlivePeriod="5000">
      <locker>
      <lease-database-locker lockAcquireSleepInterval="10000"/>
      </locker>
      </jdbcPersistenceAdapter>
      </persistenceAdapter>

      So I started the first broker and it was successfully connected to the backend database and became a master. Then I started the second broker. The expected behaviour is that the second broker would fail acquiring least-database-locker and became slave broker. But it would continue retrying to acquire the lease-database-locker. Here is where the problem arises. If the system time on second broker machine is 2 minutes behind the database server, it will be no problem since it will fail acquiring the lease-database-locker and become a slave broker. However, in case that the second broker machine is 2 minutes ahead of the database server, it will acquire the lease-database-locker straight away and also become a master broker. So we will have two master brokers at the same time and both are connected to the same backend database. If I shutdown any of the two brokers, the other will also be shutting down automatically.I then tried to configure "maxAllowableDiffFromDBTime" option on the lease-database-locker one both machine like:| <persistenceAdapter>|

      <jdbcPersistenceAdapter dataSource="#oracle-ds" lockKeepAlivePeriod="5000">
      <locker>
      <lease-database-locker lockAcquireSleepInterval="10000" maxAllowableDiffFromDBTime="120000"/>/>
      </locker>
      </jdbcPersistenceAdapter>
      </persistenceAdapter>

      But I still saw the similar behaviour althought I was getting some logging that telling me about DBTime dfference between current broker machine and the backend database machine like:2013-07-11 14:53:10,169 | INFO | Database adapter driver override recognized for :[oracle_jdbc_driver]- adapter: class org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter | org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
      2013-07-11 14:53:18,041 | INFO | jdbcBroker attempting to acquire exclusive lease to become the Master broker | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
      2013-07-11 14:53:18,157 | INFO | jdbcBroker diff from db: 120560, db time: 2013-07-11 14:51:17.597 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
      2013-07-11 14:53:20,401 | INFO | jdbcBroker, becoming the master on dataSource: org.apache.commons.dbcp.BasicDataSource@4523a389 | org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
      2013-07-11 14:53:21,059 | INFO | Apache ActiveMQ 5.8.0.redhat-60024 (jdbcBroker, ID:jluoMac-54845-1373550800422-0:1) is starting | org.apache.activemq.broker.BrokerService | main
      2013-07-11 14:53:21,904 | INFO | Listening for connections at: tcp://jluoMac:61616 | org.apache.activemq.transport.TransportServerThreadSupport | main
      2013-07-11 14:53:21,904 | INFO | Connector default Started | org.apache.activemq.broker.TransportConnector | main
      2013-07-11 14:53:21,907 | INFO | Apache ActiveMQ 5.8.0.redhat-60024 (jdbcBroker, ID:jluoMac-54845-1373550800422-0:1) started | org.apache.activemq.broker.BrokerService | main
      2013-07-11 14:53:21,907 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
      2013-07-11 14:54:15,491 | INFO | jdbcBroker, no longer able to keep the exclusive lock so giving up being a master | org.apache.activemq.broker.LockableServiceSupport | ActiveMQ Cleanup Timer
      2013-07-11 14:54:15,492 | INFO | Apache ActiveMQ 5.8.0.redhat-60024 (jdbcBroker, ID:jluoMac-54845-1373550800422-0:1) is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ Cleanup Timer
      2013-07-11 14:54:15,910 | INFO | Connector default Stopped | org.apache.activemq.broker.TransportConnector | ActiveMQ Cleanup TimerI also tried different value for the "maxAllowableDiffFromDBTime" option but it did not have any impact at all.

      Attachments

        Activity

          People

            gtully@redhat.com Gary Tully
            rhn-support-qluo Joe Luo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: