This stems from
HWKMETRICS-477. It might be somewhat of a corner case, but JobExecutionTest.executeLotsOfJobs revealed it. There are a couple of problems with the changes made for HWKEMTRICS-477.
First, a different CQL statement is needed for acquiring the job execution lock. We want an exclusive lock for job execution.
Secondly, SchedulerImpl maintains an in-memory set of active jobs. If we have a scenario in which a job gets rescheduled for execution immediately because the job is behind schedule, there is a chance that another job scheduler thread will try to obtain the execution lock, which involves executing a heavyweight light weight transaction. We can check the in-memory set of active jobs and avoid the LWT in that scenario, which is the scenario that has been failing in executeLotsOfJobs.