Using synchronous replication with high number of concurrent clients doing put() operations over a shared set of keys, lock-acquisition timeouts occur when various thread pools (internal, jgroups oob) do not have appropriate size.
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [3 seconds] on key [key_00000000000003B4] for requestor [Thread[OOB-66,default,node03-12795,5,main]]! Lock held by [Thread[OOB-314,default,node03-12795,5,main]]
[org.infinispan.interceptors.InvocationContextInterceptor] (Stressor-1) ISPN000136: Execution error
org.infinispan.util.concurrent.TimeoutException: org.infinispan.util.concurrent.TimeoutException: Node node04-24454 timed out
This applies to both transactional and non-transactional configuration. The problem can be mitigated by increasing Infinispan's internal thread pool size (defined for remoteCommandsExecutor, blockingBoundedQueueThreadPool). In order to improve user experience either:
a) When needed, the size of the thread pool should be increased as the load increases
b) The default values should be high enough to handle even significant load (in terms of number of concurrent clients per node)
c) The documentation should describe how the end user should size the thread pools based on expected load on the system