Add support for lease based lock to jdbc persistent adapter.
The current transaction based locking mechanism works well for single broker but
is difficult to configure correctly in a master/slave scenario.
The two main problems are:
1) You need to use a combination of an IOExceptionHandler and the lockKeepAlivePeriod
in order to get the desired master/slave behavior, making the current solution difficult
2) It would be preferable for a master broker to stay alive when it loses its connection to
to the DB, but shutdown its transports and then go into retry mode to reacquire the lock.
In reacquire mode, its effectively a slave with its transports shutdown.
This is possible currently using an IOExceptionHandler, but the problem
is once the broker refreshes its connection, it doesn't refresh the lock
and so there are edge conditions where the slave can incorrectly acquire
the lock after a failover (and mixed configs between master and slave are
required to work around the issue).
This enhancement is to add support for a lease based lock. This would allow
us to simplify master/slave jdbc config.
When the first broker starts it acquires a lease for a time slice, renewing it periodically.
When the master broker:
+ Terminates - it automatically releases the lease.
+ Crashes - the slave detects that the lease has expired and if it can
acquire the lease it becomes the new master.
+ Network Glitch - once master detects the connection is gone (and so timeouts would
be configured via broker thread or jdbc driver timeouts), it goes into retry mode. This
would shutdown transport connectors until it can reacquire lease.
In this manner, both graceful and ungraceful broker process terminations are detected. Also
we don't have the transaction log overhead associated with the current mechanism.
Lease based approach would add additional lease renewal traffic going between the broker and db. The slaves
will need to periodically try and acquire the lease. The master will need to periodically renew
its lease. Its not expected this would be significant.
Solution will require clocks to be sync'd between master and slave for reliable operation. Would be nice
if brokers where able to detect/protect against out of sync peers.
1) lease_ping_time - amount of time between lease renewals
2) lease_reap_time - if broker doesn't renew within lease_reap_time,
the db lock is released and open to a new master. lease_reap_time should
be larger than lease_ping_time.
Nice to have:
+ activemq-admin command to tell you the current master
+ activemq-admin command to do a soft-failover (force expiration of current lease) ?