Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-5335

Singleton deployment of EAR fails

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 10.0.0.CR5
    • 10.0.0.Beta2
    • Clustering
    • None
    • Hide

      1. Get two copies of EAP 7.0.0.DR9, no configuration modification needed.
      2. For both servers, copy cluterbench-ee7.ear (see attachments) to standalone/deployments folder.
      3. Start both servers with standalone-ha profile:

      Server 1 startup command
      ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1
      
      Server 2 startup command
      ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100
      

      4. Check the server logs.

      Show
      1. Get two copies of EAP 7.0.0.DR9, no configuration modification needed. 2. For both servers, copy cluterbench-ee7.ear (see attachments) to standalone/deployments folder. 3. Start both servers with standalone-ha profile: Server 1 startup command ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 Server 2 startup command ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100 4. Check the server logs.

    Description

      When trying to deploy our clustering test application "clusterbench" as a singleton deployment, server boot fails in an unrecoverable manner.

      To identify clusterbench-ee7.ear as a singleton deployment, I added singleton-deployment.xml deployment descriptor into deployment /META-INF/ folder. The modified clusterbench-ee7.ear is attached.
      The same issue occurs when trying with latest wildfly upstream/master bits.

      Server logs:

      Server 1
      [mvinkler@localhost jboss-eap-7.0_1]$ ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1
      =========================================================================
      
        JBoss Bootstrap Environment
      
        JBOSS_HOME: /usr/work/eap/tmp2/jboss-eap-7.0_1
      
        JAVA: /usr/work/java/jdk1.8.0_45/bin/java
      
        JAVA_OPTS:  -server -verbose:gc -Xloggc:"/usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
      
      =========================================================================
      
      14:49:40,976 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final
      14:49:41,156 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      14:49:41,204 INFO  [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) starting
      14:49:41,957 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 2) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
      14:49:41,962 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0015: Re-attempting failed deployment clusterbench-ee7.ear
      14:49:41,979 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 30) WFLYDR0001: Content added at location /usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/data/content/b4/8714b2fc6cba3b7ba7fac5e1100c85dc0a36ce/content
      14:49:42,052 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      14:49:42,073 INFO  [org.xnio] (MSC service thread 1-6) XNIO version 3.4.0.Beta1
      14:49:42,078 INFO  [org.xnio.nio] (MSC service thread 1-6) XNIO NIO Implementation Version 3.4.0.Beta1
      14:49:42,100 INFO  [org.jboss.remoting] (MSC service thread 1-6) JBoss Remoting version 4.0.9.Final
      14:49:42,107 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0001: Activating Infinispan subsystem.
      14:49:42,111 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
      14:49:42,116 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 36) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
      14:49:42,212 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 45) WFLYCLJG0001: Activating JGroups subsystem.
      14:49:42,231 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 48) WFLYJSF0007: Activated the following JSF Implementations: [main]
      14:49:42,233 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 60) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      14:49:42,245 INFO  [org.jboss.as.connector] (MSC service thread 1-5) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)
      14:49:42,248 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 51) WFLYNAM0001: Activating Naming Subsystem
      14:49:42,249 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0018: Started Driver service with driver-name = h2
      14:49:42,254 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 58) WFLYSEC0002: Activating Security Subsystem
      14:49:42,253 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 62) WFLYWS0002: Activating WebServices Extension
      14:49:42,264 INFO  [org.jboss.as.security] (MSC service thread 1-1) WFLYSEC0001: Current PicketBox version=4.9.3.Final
      14:49:42,310 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 1.3.0.Beta9 starting
      14:49:42,311 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0003: Undertow 1.3.0.Beta9 starting
      14:49:42,339 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
      14:49:42,339 INFO  [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
      14:49:42,419 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0014: Creating file handler for path '/usr/work/eap/tmp2/jboss-eap-7.0_1/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
      14:49:42,429 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0012: Started server default-server.
      14:49:42,431 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0018: Host default-host starting
      14:49:42,485 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0006: Undertow HTTP listener default listening on /127.0.0.1:8080
      14:49:42,487 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow AJP listener ajp listening on /127.0.0.1:8009
      14:49:42,490 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final
      14:49:42,511 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
      14:49:42,597 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      14:49:42,641 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /usr/work/eap/tmp2/jboss-eap-7.0_1/standalone/deployments
      14:49:42,647 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear")
      14:49:42,723 INFO  [org.jboss.ws.common.management] (MSC service thread 1-2) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.1.0.Beta1
      14:49:42,726 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war")
      14:49:42,726 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar")
      14:49:42,727 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war")
      14:49:42,847 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-7) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
      14:49:42,852 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-2) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
      14:49:43,066 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      14:49:43,066 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      14:49:43,067 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      14:49:43,067 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      14:49:46,454 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups channel server
      14:49:46,464 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view for channel server: [node1|0] (1) [node1]
      14:49:46,467 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Channel server local address is node1, physical addresses are [127.0.0.1:55200]
      14:49:46,471 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Infinite Darkness' 8.0.0.CR1
      14:49:46,595 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started default cache from server container
      14:49:46,625 INFO  [org.wildfly.clustering.server] (ServerService Thread Pool -- 64) WFLYCLSV0003: node1 elected as the singleton provider of the jboss.deployment.unit."clusterbench-ee7.ear".FIRST_MODULE_USE service
      14:49:48,948 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,node1) ISPN000094: Received new cluster view for channel server: [node1|1] (2) [node1, node2]
      14:49:49,477 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t1) ISPN000310: Starting cluster-wide rebalance for cache default, topology CacheTopology{id=1, rebalanceId=1, currentCH=ReplicatedConsistentHash{ns = 60, owners = (1)[node1: 60]}, pendingCH=ReplicatedConsistentHash{ns = 60, owners = (2)[node1: 30, node2: 30]}, unionCH=null, actualMembers=[node1, node2]}
      14:54:42,636 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[
          ("core-service" => "management"),
          ("management-interface" => "http-interface")
      ]'
      14:54:42,695 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-passivating.war) in 57ms
      14:54:42,695 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-default.war) in 57ms
      14:54:42,695 INFO  [org.infinispan.CLUSTER] (remote-thread--p2-t1) ISPN000336: Finished cluster-wide rebalance for cache default, topology id = 1
      14:54:42,698 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0003: Stopped default cache from server container
      14:54:42,699 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-ejb.jar) in 61ms
      14:54:42,701 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000080: Disconnecting JGroups channel server
      14:54:42,702 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000082: Stopping the RpcDispatcher for channel server
      14:54:42,707 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0028: Stopped deployment clusterbench-ee7.ear (runtime-name: clusterbench-ee7.ear) in 69ms
      14:54:42,716 ERROR [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0022: Deploy of deployment "clusterbench-ee7.ear" was rolled back with no failure message
      14:54:42,718 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0008: Undertow HTTP listener default suspending
      14:54:42,721 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 39) MODCLUSTER000002: Initiating mod_cluster shutdown
      14:54:42,719 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0008: Undertow AJP listener ajp suspending
      14:54:42,722 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) WFLYUT0007: Undertow HTTP listener default stopped, was bound to /127.0.0.1:8080
      14:54:42,723 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
      14:54:42,722 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to /127.0.0.1:8009
      14:54:42,724 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
      14:54:42,728 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0004: Undertow 1.3.0.Beta9 stopping
      14:54:42,728 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
      14:54:42,747 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
      14:54:42,752 INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
      14:54:42,765 INFO  [org.jboss.as] (MSC service thread 1-4) WFLYSRV0050: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) stopped in 9ms
      
      Server 2
      [mvinkler@localhost jboss-eap-7.0_2]$ ./bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.socket.binding.port-offset=100
      =========================================================================
      
        JBoss Bootstrap Environment
      
        JBOSS_HOME: /usr/work/eap/tmp2/jboss-eap-7.0_2
      
        JAVA: /usr/work/java/jdk1.8.0_45/bin/java
      
        JAVA_OPTS:  -server -verbose:gc -Xloggc:"/usr/work/eap/tmp2/jboss-eap-7.0_2/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms64m -Xmx512m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
      
      =========================================================================
      
      14:49:46,623 INFO  [org.jboss.modules] (main) JBoss Modules version 1.4.3.Final
      14:49:46,835 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      14:49:46,883 INFO  [org.jboss.as] (MSC service thread 1-5) WFLYSRV0049: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) starting
      14:49:47,735 INFO  [org.jboss.as.controller.management-deprecated] (ServerService Thread Pool -- 30) WFLYCTL0028: Attribute 'enabled' in the resource at address '/subsystem=datasources/data-source=ExampleDS' is deprecated, and may be removed in future version. See the attribute description in the output of the read-resource-description operation to learn more about the deprecation.
      14:49:47,851 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      14:49:47,866 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.4.0.Beta1
      14:49:47,874 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.4.0.Beta1
      14:49:47,908 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.9.Final
      14:49:47,909 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0001: Activating Infinispan subsystem.
      14:49:47,916 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 40) WFLYIO001: Worker 'default' has auto-configured to 16 core threads with 128 task threads based on your 8 available processors
      14:49:47,997 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 45) WFLYCLJG0001: Activating JGroups subsystem.
      14:49:47,999 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 36) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
      14:49:48,016 INFO  [org.jboss.as.connector] (MSC service thread 1-5) WFLYJCA0009: Starting JCA Subsystem (IronJacamar 1.2.5.Final)
      14:49:48,024 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-5) WFLYJCA0018: Started Driver service with driver-name = h2
      14:49:48,027 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 51) WFLYNAM0001: Activating Naming Subsystem
      14:49:48,028 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 60) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      14:49:48,034 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 48) WFLYJSF0007: Activated the following JSF Implementations: [main]
      14:49:48,059 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 62) WFLYWS0002: Activating WebServices Extension
      14:49:48,079 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 58) WFLYSEC0002: Activating Security Subsystem
      14:49:48,085 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0003: Undertow 1.3.0.Beta9 starting
      14:49:48,088 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0003: Undertow 1.3.0.Beta9 starting
      14:49:48,111 INFO  [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=4.9.3.Final
      14:49:48,123 INFO  [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service
      14:49:48,123 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-8) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
      14:49:48,188 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 61) WFLYUT0014: Creating file handler for path '/usr/work/eap/tmp2/jboss-eap-7.0_2/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
      14:49:48,191 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0012: Started server default-server.
      14:49:48,192 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0018: Host default-host starting
      14:49:48,236 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow AJP listener ajp listening on /127.0.0.1:8109
      14:49:48,240 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTP listener default listening on /127.0.0.1:8180
      14:49:48,243 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000001: Initializing mod_cluster version 1.3.1.Final
      14:49:48,256 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 64) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
      14:49:48,358 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
      14:49:48,423 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /usr/work/eap/tmp2/jboss-eap-7.0_2/standalone/deployments
      14:49:48,434 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0027: Starting deployment of "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear")
      14:49:48,515 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war")
      14:49:48,515 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar")
      14:49:48,515 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war")
      14:49:48,530 INFO  [org.jboss.ws.common.management] (MSC service thread 1-3) JBWS022052: Starting JBoss Web Services - Stack CXF Server 5.1.0.Beta1
      14:49:48,661 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-2) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
      14:49:48,664 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-4) WFLYSRV0018: Deployment "deployment.clusterbench-ee7.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
      14:49:48,875 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      14:49:48,875 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      14:49:48,876 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      14:49:48,876 WARNING [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      14:49:49,291 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups channel server
      14:49:49,299 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view for channel server: [node1|1] (2) [node1, node2]
      14:49:49,302 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Channel server local address is node2, physical addresses are [127.0.0.1:55300]
      14:49:49,345 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Infinite Darkness' 8.0.0.CR1
      14:50:49,523 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 64) MSC000001: Failed to start service jboss.infinispan.server.default: org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
              at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
              at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:839)
              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:629)
              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:580)
              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445)
              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:464)
              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455)
              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
              at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:111)
              at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
              at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
              ... 4 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node2
              at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:225)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:497)
              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
              ... 19 more
      
      14:50:49,526 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      14:50:49,529 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([
          ("subsystem" => "infinispan"),
          ("cache-container" => "server"),
          ("replicated-cache" => "default")
      ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.infinispan.server.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node2"}}
      14:50:49,557 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 37) WFLYSRV0010: Deployed "clusterbench-ee7.ear" (runtime-name : "clusterbench-ee7.ear")
      14:50:49,559 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
      WFLYCTL0186:   Services which failed to start:      service jboss.infinispan.server.default: org.jboss.msc.service.StartException in service jboss.infinispan.server.default: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      
      14:50:49,642 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:10090/management
      14:50:49,643 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:10090
      14:50:49,643 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: EAP 7.0.0.Alpha1 (WildFly Core 2.0.0.Beta4) started (with errors) in 63233ms - Started 321 of 583 services (7 services failed or missing dependencies, 363 services are lazy, passive or on-demand)
      14:51:49,528 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      14:52:49,529 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      14:53:49,531 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (transport-thread--p4-t2) ISPN000209: Failed to retrieve transactions for segments [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59] of cache default from node node1: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node1
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$16/514373550.apply(Unknown Source)
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
              at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
              at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      14:54:42,713 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,ee,node2) ISPN000094: Received new cluster view for channel server: [node2|2] (1) [node2]
      

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              mvinkler1@redhat.com Michal Vinkler
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: