Uploaded image for project: 'FUSE Message Broker'
  1. FUSE Message Broker
  2. MB-1156

Add support for lease based lock to jdbc persistent adapter.


    • Type: Enhancement
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 5.5.1-fuse-07-11
    • Component/s: broker
    • Labels:
    • Environment:

      Fuse MB 5.5.x


      Add support for lease based lock to jdbc persistent adapter.

      The current transaction based locking mechanism works well for single broker but
      is difficult to configure correctly in a master/slave scenario.

      The two main problems are:

      1) You need to use a combination of an IOExceptionHandler and the lockKeepAlivePeriod
      in order to get the desired master/slave behavior, making the current solution difficult
      to configure.

      2) It would be preferable for a master broker to stay alive when it loses its connection to
      to the DB, but shutdown its transports and then go into retry mode to reacquire the lock.
      In reacquire mode, its effectively a slave with its transports shutdown.

      This is possible currently using an IOExceptionHandler, but the problem
      is once the broker refreshes its connection, it doesn't refresh the lock
      and so there are edge conditions where the slave can incorrectly acquire
      the lock after a failover (and mixed configs between master and slave are
      required to work around the issue).

      This enhancement is to add support for a lease based lock. This would allow
      us to simplify master/slave jdbc config.

      Master Behavior:

      When the first broker starts it acquires a lease for a time slice, renewing it periodically.

      When the master broker:
      + Terminates - it automatically releases the lease.

      + Crashes - the slave detects that the lease has expired and if it can
      acquire the lease it becomes the new master.

      + Network Glitch - once master detects the connection is gone (and so timeouts would
      be configured via broker thread or jdbc driver timeouts), it goes into retry mode. This
      would shutdown transport connectors until it can reacquire lease.


      In this manner, both graceful and ungraceful broker process terminations are detected. Also
      we don't have the transaction log overhead associated with the current mechanism.


      Lease based approach would add additional lease renewal traffic going between the broker and db. The slaves
      will need to periodically try and acquire the lease. The master will need to periodically renew
      its lease. Its not expected this would be significant.

      Solution will require clocks to be sync'd between master and slave for reliable operation. Would be nice
      if brokers where able to detect/protect against out of sync peers.


      1) lease_ping_time - amount of time between lease renewals

      2) lease_reap_time - if broker doesn't renew within lease_reap_time,
      the db lock is released and open to a new master. lease_reap_time should
      be larger than lease_ping_time.

      Nice to have:

      + activemq-admin command to tell you the current master
      + activemq-admin command to do a soft-failover (force expiration of current lease) ?

        Gliffy Diagrams


            Issue Links



                • Assignee:
                  garytully Gary Tully
                  davestanley Dave Stanley
                • Votes:
                  0 Vote for this issue
                  0 Start watching this issue


                  • Created: