Uploaded image for project: 'Keycloak'
  1. Keycloak
  2. KEYCLOAK-5910

Cross DC - Server does not start when other DC is not available

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 3.4.0.Final
    • Fix Version/s: 3.4.1.CR1
    • Component/s: None
    • Labels:
      None
    • Environment:

      Two DCs, one up and running, other completely down (in fact one ISPN server with relay2 configured, one Keycloak server with remote store configured)

    • Docs QE Status:
      NEW
    • QE Status:
      VERIFIED

      Description

      Upon start, the following exception is logged by Keycloak and Keycloak is terminated:

      ...
      2017-11-25 14:53:19,190 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (ServerService Thread Pool -- 60) Node name: ip-10-0-1-33, Site name: dc1
      2017-11-25 14:53:19,878 DEBUG [org.keycloak.cluster.infinispan.CrossDCAwareCacheFactory] (ServerService Thread Pool -- 60) RemoteStore is available. Cross-DC scenario will be used
      2017-11-25 14:53:29,958 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21] (ServerService Thread Pool -- 60) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
      javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
      2017-11-25 14:53:29,959 WARN  [org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory] (ServerService Thread Pool -- 60) Failed to write key 'cluster-start-time' and value '1511621598' in iteration '0' . Retrying: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=16 returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
      javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:350)
              at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:139)
              at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:125)
              at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
              at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
              at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:36)
              at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)
              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:340)
              at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:59)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.lambda$putIfAbsentWithRetries$0(InfinispanClusterProviderFactory.java:142)
              at org.keycloak.common.util.Retry.executeWithBackoff(Retry.java:82)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.putIfAbsentWithRetries(InfinispanClusterProviderFactory.java:135)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.initClusterStartupTime(InfinispanClusterProviderFactory.java:118)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.lazyInit(InfinispanClusterProviderFactory.java:97)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.create(InfinispanClusterProviderFactory.java:79)
              at org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory.create(InfinispanClusterProviderFactory.java:58)
              at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
              at org.keycloak.models.cache.infinispan.InfinispanCacheRealmProviderFactory.lazyInit(InfinispanCacheRealmProviderFactory.java:59)
              at org.keycloak.models.cache.infinispan.InfinispanCacheRealmProviderFactory.create(InfinispanCacheRealmProviderFactory.java:47)
              at org.keycloak.models.cache.infinispan.InfinispanCacheRealmProviderFactory.create(InfinispanCacheRealmProviderFactory.java:37)
              at org.keycloak.services.DefaultKeycloakSession.getProvider(DefaultKeycloakSession.java:163)
              at org.keycloak.services.DefaultKeycloakSession.getRealmProvider(DefaultKeycloakSession.java:78)
              at org.keycloak.services.DefaultKeycloakSession.realms(DefaultKeycloakSession.java:227)
              at org.keycloak.migration.MigrationModelManager.migrate(MigrationModelManager.java:76)
              at org.keycloak.services.resources.KeycloakApplication.migrateModel(KeycloakApplication.java:244)
              at org.keycloak.services.resources.KeycloakApplication.migrateAndBootstrap(KeycloakApplication.java:185)
              at org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:144)
              at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
              at org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:135)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
              at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
              at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2298)
              at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:340)
              at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:253)
              at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:120)
              at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
              at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
              at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
              at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
              at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:250)
              at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:133)
              at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:565)
              at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:536)
              at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
              at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
              at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:578)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
              at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      2017-11-25 14:53:29,997 DEBUG [org.keycloak.cluster.infinispan.InfinispanClusterProviderFactory] (ServerService Thread Pool -- 60) Loaded cluster startup time: Sat Nov 25 14:53:18 UTC 2017
      2017-11-25 14:53:29,999 DEBUG [org.keycloak.cluster.infinispan.InfinispanNotificationsManager] (ServerService Thread Pool -- 60) Added listener for infinispan cache: work
      2017-11-25 14:53:30,023 DEBUG [org.keycloak.cluster.infinispan.InfinispanNotificationsManager] (ServerService Thread Pool -- 60) Added listener for HotRod remoteStore cache: work
      2017-11-25 14:53:30,025 DEBUG [org.keycloak.models.cache.infinispan.InfinispanCacheRealmProviderFactory] (ServerService Thread Pool -- 60) Registered cluster listeners
      2017-11-25 14:53:32,102 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 60) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
      2017-11-25 14:53:55,996 INFO  [org.keycloak.connections.jpa.updater.liquibase.LiquibaseJpaUpdaterProvider] (ServerService Thread Pool -- 60) Initializing database schema. Using changelog META-INF/example-changelog.xml
      2017-11-25 14:53:56,219 INFO  [org.hibernate.jpa.internal.util.LogHelper] (ServerService Thread Pool -- 60) HHH000204: Processing PersistenceUnitInfo [
              name: keycloak-default
              ...]
      2017-11-25 14:53:56,449 INFO  [org.hibernate.Version] (ServerService Thread Pool -- 60) HHH000412: Hibernate Core {5.1.10.Final}
      2017-11-25 14:53:56,451 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 60) HHH000206: hibernate.properties not found
      2017-11-25 14:53:56,452 INFO  [org.hibernate.cfg.Environment] (ServerService Thread Pool -- 60) HHH000021: Bytecode provider name : javassist
      2017-11-25 14:53:56,480 INFO  [org.hibernate.annotations.common.Version] (ServerService Thread Pool -- 60) HCANN000001: Hibernate Commons Annotations {5.0.1.Final}
      2017-11-25 14:53:56,605 INFO  [org.hibernate.dialect.Dialect] (ServerService Thread Pool -- 60) HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect
      2017-11-25 14:53:56,640 INFO  [org.hibernate.envers.boot.internal.EnversServiceImpl] (ServerService Thread Pool -- 60) Envers integration enabled? : true
      2017-11-25 14:53:57,236 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 60) HV000001: Hibernate Validator 5.3.5.Final
      2017-11-25 14:53:58,389 INFO  [org.hibernate.hql.internal.QueryTranslatorFactoryInitiator] (ServerService Thread Pool -- 60) HHH000397: Using ASTQueryTranslatorFactory
      2017-11-25 14:53:59,233 INFO  [org.keycloak.services] (ServerService Thread Pool -- 60) KC-SERVICES0050: Initializing master realm
      2017-11-25 14:54:10,375 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21] (ServerService Thread Pool -- 60) ISPN004005: Error received from the server: org.infinispan.commons.CacheException: Could not commit implicit transaction
      javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
      2017-11-25 14:54:10,388 INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
      2017-11-25 14:54:10,390 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 60) MSC000001: Failed to start service jboss.undertow.deployment.default-server.default-host./auth: org.jboss.msc.service.StartException in service jboss.undertow.deployment.default-server.default-host./auth: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:84)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
              at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      Caused by: java.lang.RuntimeException: RESTEASY003325: Failed to construct public org.keycloak.services.resources.KeycloakApplication(javax.servlet.ServletContext,org.jboss.resteasy.core.Dispatcher)
              at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:162)
              at org.jboss.resteasy.spi.ResteasyProviderFactory.createProviderInstance(ResteasyProviderFactory.java:2298)
              at org.jboss.resteasy.spi.ResteasyDeployment.createApplication(ResteasyDeployment.java:340)
              at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployment.java:253)
              at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.init(ServletContainerDispatcher.java:120)
              at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.init(HttpServletDispatcher.java:36)
              at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:117)
              at org.wildfly.extension.undertow.security.RunAsLifecycleInterceptor.init(RunAsLifecycleInterceptor.java:78)
              at io.undertow.servlet.core.LifecyleInterceptorInvocation.proceed(LifecyleInterceptorInvocation.java:103)
              at io.undertow.servlet.core.ManagedServlet$DefaultInstanceStrategy.start(ManagedServlet.java:250)
              at io.undertow.servlet.core.ManagedServlet.createServlet(ManagedServlet.java:133)
              at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:565)
              at io.undertow.servlet.core.DeploymentManagerImpl$2.call(DeploymentManagerImpl.java:536)
              at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
              at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
              at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)
              at io.undertow.servlet.core.DeploymentManagerImpl.start(DeploymentManagerImpl.java:578)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
              at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
              ... 6 more
      Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=19 returned server error (status=0x85): org.infinispan.commons.CacheException: Could not commit implicit transaction
      javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:350)
              at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:139)
              at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:125)
              at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
              at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:56)
              at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:32)
              at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)
              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:328)
              at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:129)
              at org.keycloak.cluster.infinispan.InfinispanNotificationsManager.notify(InfinispanNotificationsManager.java:156)
              at org.keycloak.cluster.infinispan.InfinispanClusterProvider.notify(InfinispanClusterProvider.java:130)
              at org.keycloak.models.cache.infinispan.CacheManager.sendInvalidationEvents(CacheManager.java:206)
              at org.keycloak.models.cache.infinispan.RealmCacheSession.runInvalidations(RealmCacheSession.java:264)
              at org.keycloak.models.cache.infinispan.RealmCacheSession$2.commit(RealmCacheSession.java:326)
              at org.keycloak.services.DefaultKeycloakTransactionManager.commit(DefaultKeycloakTransactionManager.java:136)
              at org.keycloak.services.resources.KeycloakApplication.migrateAndBootstrap(KeycloakApplication.java:218)
              at org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:144)
              at org.keycloak.models.utils.KeycloakModelUtils.runJobInTransaction(KeycloakModelUtils.java:227)
              at org.keycloak.services.resources.KeycloakApplication.<init>(KeycloakApplication.java:135)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
              at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(ConstructorInjectorImpl.java:150)
              ... 28 more
       
      2017-11-25 14:54:10,448 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel ejb
      2017-11-25 14:54:10,448 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb
      2017-11-25 14:54:10,449 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel ejb
      2017-11-25 14:54:10,449 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel ejb
      2017-11-25 14:54:10,449 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel ejb
      2017-11-25 14:54:10,449 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb
      2017-11-25 14:54:10,452 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/KeycloakDS]
      2017-11-25 14:54:10,456 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow HTTPS listener https suspending
      2017-11-25 14:54:10,461 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow AJP listener ajp suspending
      2017-11-25 14:54:10,461 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 65) MODCLUSTER000002: Initiating mod_cluster shutdown
      2017-11-25 14:54:10,471 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 10.0.1.33:8443
      2017-11-25 14:54:10,473 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 10.0.1.33:8009
      2017-11-25 14:54:10,475 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
      2017-11-25 14:54:10,485 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0019: Host default-host stopping
      2017-11-25 14:54:10,495 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = mysql-connector-java
      2017-11-25 14:54:10,495 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
      2017-11-25 14:54:10,504 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
      2017-11-25 14:54:10,505 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 10.0.1.33:8080
      2017-11-25 14:54:10,510 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 1.4.18.Final stopping
      2017-11-25 14:54:10,511 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 69) WFLYCLINF0003: Stopped realms cache from keycloak container
      2017-11-25 14:54:10,512 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 73) WFLYCLINF0003: Stopped authorization cache from keycloak container
      2017-11-25 14:54:10,514 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0003: Stopped keys cache from keycloak container
      2017-11-25 14:54:10,515 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 67) WFLYCLINF0003: Stopped users cache from keycloak container
      2017-11-25 14:54:10,518 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0028: Stopped deployment keycloak-server.war (runtime-name: keycloak-server.war) in 115ms
      2017-11-25 14:54:10,526 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) WFLYCLINF0003: Stopped authenticationSessions cache from keycloak container
      2017-11-25 14:54:10,543 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 70) WFLYCLINF0003: Stopped offlineClientSessions cache from keycloak container
      2017-11-25 14:54:10,544 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) WFLYCLINF0003: Stopped sessions cache from keycloak container
      2017-11-25 14:54:10,545 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 74) WFLYCLINF0003: Stopped loginFailures cache from keycloak container
      2017-11-25 14:54:10,546 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 60) WFLYCLINF0003: Stopped offlineSessions cache from keycloak container
      2017-11-25 14:54:10,549 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 75) WFLYCLINF0003: Stopped actionTokens cache from keycloak container
      2017-11-25 14:54:10,550 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) WFLYCLINF0003: Stopped work cache from keycloak container
      2017-11-25 14:54:10,562 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 72) WFLYCLINF0003: Stopped clientSessions cache from keycloak container
      2017-11-25 14:54:10,577 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 76) WFLYCLINF0003: Stopped client-mappings cache from ejb container
      2017-11-25 14:54:10,584 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb
      2017-11-25 14:54:10,584 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb
      2017-11-25 14:54:10,586 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) WFLYCLINF0003: Stopped authorizationRevisions cache from keycloak container
      2017-11-25 14:54:10,601 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) WFLYCLINF0003: Stopped realmRevisions cache from keycloak container
      2017-11-25 14:54:10,605 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) WFLYCLINF0003: Stopped userRevisions cache from keycloak container
      2017-11-25 14:54:10,605 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel ejb
      2017-11-25 14:54:10,606 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel ejb
      2017-11-25 14:54:10,648 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0050: Keycloak 3.4.1.Final-SNAPSHOT (WildFly Core 3.0.8.Final) stopped in 247ms
      

      and this exception by Infinispan

      2017-11-25 14:53:20,040 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerHandler-1-1) ip-10-0-1-110: no route to dc-1: dropping message
      2017-11-25 14:53:30,044 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerHandler-1-1) ISPN000136: Error executing command PrepareCommand, writing keys [[B0x033e12636c757374..[21]]: The local cache work failed
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsentInternal(CacheImpl.java:1176)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1164)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1783)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:252)
              at org.infinispan.server.hotrod.CacheDecodeContext.putIfAbsent(CacheDecodeContext.scala:179)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:77)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
       
      2017-11-25 14:53:30,045 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (HotRodServerHandler-1-1) ISPN000097: Error while processing a prepare in a single-phase transaction: The local cache work failed to backup data to th
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsentInternal(CacheImpl.java:1176)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1164)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1783)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:252)
              at org.infinispan.server.hotrod.CacheDecodeContext.putIfAbsent(CacheDecodeContext.scala:179)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:77)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
       
      2017-11-25 14:53:30,047 WARN  [com.arjuna.ats.jta] (HotRodServerHandler-1-1) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a00016e:4145825b:5a197354:ed, node_name=1, branch_uid=0:ffff0
              at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsentInternal(CacheImpl.java:1176)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1164)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1783)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:252)
              at org.infinispan.server.hotrod.CacheDecodeContext.putIfAbsent(CacheDecodeContext.scala:179)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:77)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: The local cache work failed to backup data to the remote sites:
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              ... 22 more
       
      2017-11-25 14:53:30,049 WARN  [org.infinispan.cache.impl.CacheImpl] (HotRodServerHandler-1-1) ISPN000160: Could not complete injected transaction.: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsentInternal(CacheImpl.java:1176)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1164)
              at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1783)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.putIfAbsent(AbstractDelegatingAdvancedCache.java:252)
              at org.infinispan.server.hotrod.CacheDecodeContext.putIfAbsent(CacheDecodeContext.scala:179)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:77)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
              Suppressed: javax.transaction.xa.XAException
                      at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
                      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
                      at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
                      at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
                      at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
                      at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
                      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
                      at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
                      ... 15 more
              Caused by: The local cache work failed to backup data to the remote sites:
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
                      at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
                      at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
                      at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
                      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
                      at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
                      at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
                      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
                      ... 22 more
       
      2017-11-25 14:54:10,468 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerHandler-1-4) ISPN000136: Error executing command PrepareCommand, writing keys [[B0x033e243132336633..[39]]: The local cache work failed
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1134)
              at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1124)
              at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:543)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:237)
              at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:189)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:74)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
       
      2017-11-25 14:54:10,468 ERROR [org.infinispan.transaction.impl.TransactionCoordinator] (HotRodServerHandler-1-4) ISPN000097: Error while processing a prepare in a single-phase transaction: The local cache work failed to backup data to th
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1134)
              at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1124)
              at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:543)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:237)
              at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:189)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:74)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
       
      2017-11-25 14:54:10,469 WARN  [com.arjuna.ats.jta] (HotRodServerHandler-1-4) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a00016e:4145825b:5a197354:f2, node_name=1, branch_uid=0:ffff0
              at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
              at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
              at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
              at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
              at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
              at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1134)
              at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1124)
              at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:543)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:237)
              at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:189)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:74)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: The local cache work failed to backup data to the remote sites:
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
              at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
              at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
              at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
              at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
              at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
              at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
              at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
              at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
              at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
              at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
              at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
              at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
              ... 22 more
       
      2017-11-25 14:54:10,469 WARN  [org.infinispan.cache.impl.CacheImpl] (HotRodServerHandler-1-4) ISPN000160: Could not complete injected transaction.: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
              at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1212)
              at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
              at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89)
              at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1735)
              at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1692)
              at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1134)
              at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1124)
              at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:543)
              at org.infinispan.cache.impl.AbstractDelegatingAdvancedCache.put(AbstractDelegatingAdvancedCache.java:237)
              at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:189)
              at org.infinispan.server.hotrod.ContextHandler.realRead(ContextHandler.java:74)
              at org.infinispan.server.hotrod.ContextHandler.lambda$channelRead0$1(ContextHandler.java:57)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
              at java.lang.Thread.run(Thread.java:748)
              Suppressed: javax.transaction.xa.XAException
                      at org.infinispan.transaction.impl.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:213)
                      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:159)
                      at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:112)
                      at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:704)
                      at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2366)
                      at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1495)
                      at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
                      at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                      at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200)
                      ... 15 more
              Caused by: The local cache work failed to backup data to the remote sites:
      dc-1: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from dc-1 (sync, timeout=10000)
       
                      at org.infinispan.xsite.BackupSenderImpl.processFailedResponses(BackupSenderImpl.java:226)
                      at org.infinispan.xsite.BackupSenderImpl.processResponses(BackupSenderImpl.java:131)
                      at org.infinispan.interceptors.xsite.PessimisticBackupInterceptor.visitPrepareCommand(PessimisticBackupInterceptor.java:29)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:37)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:93)
                      at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:106)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:158)
                      at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:145)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:42)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:229)
                      at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:87)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
                      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
                      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:113)
                      at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:112)
                      at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:176)
                      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
                      at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:157)
                      ... 22 more
      
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                hmlnarik Hynek Mlnařík
                Reporter:
                hmlnarik Hynek Mlnařík
                Tester:
                Vlastislav Ramik
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: