When the job scheduler starts its work for a time slice, one of things it does before executing any jobs is to acquire a time slice lock. The current code for obtaining the lock only allows for one scheduler node to hold the lock. Multiple nodes should be able to acquire the lock. There are a couple changes needed.
First, we need to change the CQL used for acquiring the lock to:
Secondly, it is also possible that the lock value is set to scheduling. This should be considered temporary, and the job scheduler should attempt to acquire the lock after a short delay.
Lastly, automated tests are needed for these scenarios.