Loading...

Details

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: None
Affects Version/s: None
Component/s: OpenShift
Labels:

Target Release:

7.backlog.GA

SFDC Cases Counter:
SFDC Cases Links:

Description

I'm running a set of tests on a s390x testing image: registry-proxy.engineering.redhat.com/rh-osbs/jboss-eap-7-eap72-openj9-11-openshift-rhel8:openjdk-11-rhel8-containers-candidate-76008-20191024211400.

I'm seeing long delays in EAP startup, that cause readiness/liveness probe fails. The delay seems to occur at a time when the JVM is being initialized, so this could be related to OpenJ9. There's also a suspicious warning in the logs: JVMJ9VM085W Malformed option: '-Xlog:gc*:file=/opt/eap/standalone/log/gc.log:time'.

The delay is ~3 minutes long, then EAP gets terminated, then waits about 1 minute longer, then prints all the startup text and quits because it got SIGTERM. Search for the ERROR *** JBossAS wrapper process (1) received TERM signal *** message to see where the delay occurred.

[root@slave-11 openshift-tests]# oc logs --previous jgroups-1-dvbcc
WARN Configuration of an embedded messaging broker within the appserver is enabled but is not recommended. Support for such a configuration will be removed in a future release.
INFO If you are not configuring messaging destinations, to disable configuring an embedded messaging broker set the DISABLE_EMBEDDED_JMS_BROKER environment variable to true.
INFO Service account has sufficient permissions to view pods in kubernetes (HTTP 200). Clustering will be available.
INFO Configuring JGroups discovery protocol to openshift.KUBE_PING
INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT.
WARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted.
Using PicketBox SSL configuration.
WARN Missing SSO_URL. Unable to properly configure SSO-enabled applications
INFO Access log is disabled, ignoring configuration.
INFO Running jboss-eap-7/eap72-openj9-11-openshift-rhel8 image, version 7.2
-XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError
ERROR *** JBossAS wrapper process (1) received TERM signal ***
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/eap

  JAVA: /usr/lib/jvm/jre-11-openj9/bin/java

  JAVA_OPTS:  -server -Xlog:gc*:file=/opt/eap/standalone/log/gc.log:time,uptimemillis:filecount=5,filesize=3M -Xms1303m -Xmx1303m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.logmanager,jdk.nashorn.api,com.sun.crypto.provider -Djava.awt.headless=true -javaagent:/opt/jboss/container/jolokia/jolokia.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -Xbootclasspath/a:/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-7.2.4.CP/org/jboss/logmanager/main/jboss-logmanager-2.1.14.Final-redhat-00001.jar:/opt/eap/modules/system/layers/base/org/glassfish/javax/json/main/javax.json-1.1.2.redhat-1.jar:/opt/eap/modules/system/layers/base/javax/json/api/main/javax.json-api-1.1.2.redhat-1.jar:/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-7.2.4.CP/org/wildfly/common/main/wildfly-common-1.5.1.Final-redhat-00001.jar -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dsun.util.logging.disableCallerCheck=true -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom    --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED --add-modules=java.se

=========================================================================

JVMJ9VM085W Malformed option: '-Xlog:gc*:file=/opt/eap/standalone/log/gc.log:time'
Nov 11, 2019 12:50:37 PM java.lang.System$LoggerFinder lambda$getLoggerFinder$0
WARNING: Failed to instantiate LoggerFinder provider; Using default.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jolokia.util.ClassUtil (file:/opt/jboss/container/jolokia/jolokia-jvm-1.6.2.redhat-00002-agent.jar) to constructor sun.security.x509.X500Name(java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String,java.lang.String)
WARNING: Please consider reporting this to the maintainers of org.jolokia.util.ClassUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
12:50:37,666 INFO  [org.jboss.modules] (main) JBoss Modules version 1.8.8.Final-redhat-00001
12:50:38,283 INFO  [org.jboss.msc] (main) JBoss MSC version 1.4.8.Final-redhat-00001
12:50:38,291 INFO  [org.jboss.threads] (main) JBoss Threads version 2.3.2.Final-redhat-1
12:50:38,487 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: JBoss EAP 7.2.4.GA (WildFly Core 6.0.16.Final-redhat-00002) starting
12:50:38,489 DEBUG [org.jboss.as.config] (MSC service thread 1-1) Configured system properties:
	[Standalone] =
	awt.toolkit = sun.awt.X11.XToolkit
	com.ibm.cpu.endian = big
	com.ibm.jcl.checkClassPath =
	com.ibm.oti.configuration = scar
	com.ibm.oti.shared.enabled = false
	com.ibm.oti.vm.bootstrap.library.path = /usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib
	com.ibm.oti.vm.library.version = 29
	com.ibm.system.agent.path = /usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib
	com.ibm.util.extralibs.properties =
	com.ibm.vm.bitmode = 64
	file.encoding = ANSI_X3.4-1968
	file.encoding.pkg = sun.io
	file.separator = /
	ibm.signalhandling.rs = false
	ibm.signalhandling.sigchain = true
	ibm.signalhandling.sigint = true
	ibm.system.encoding = ANSI_X3.4-1968
	java.awt.fonts =
	java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
	java.awt.headless = true
	java.awt.printerjob = sun.print.PSPrinterJob
	java.class.path = /opt/eap/jboss-modules.jar:/opt/jboss/container/jolokia/jolokia.jar
	java.class.version = 55.0
	java.compiler = j9jit29
	java.fullversion = 11.0.5+10
JRE 11 Linux s390x-64-Bit Compressed References 20191016_379 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 2a7af5674b based on jdk-11.0.5+10
	java.home = /usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x
	java.io.tmpdir = /tmp
	java.library.path = /usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib:/usr/lib64:/usr/lib
	java.net.preferIPv4Stack = true
	java.runtime.name = OpenJDK Runtime Environment
	java.runtime.version = 11.0.5+10
	java.security.egd = file:/dev/./urandom
	java.specification.name = Java Platform API Specification
	java.specification.vendor = Oracle Corporation
	java.specification.version = 11
	java.util.logging.manager = org.jboss.logmanager.LogManager
	java.vendor = Eclipse OpenJ9
	java.vendor.url = http://www.eclipse.org/openj9
	java.vendor.version = AdoptOpenJDK
	java.version = 11.0.5
	java.version.date = 2019-10-15
	java.vm.info = JRE 11 Linux s390x-64-Bit Compressed References 20191016_379 (JIT enabled, AOT enabled)
OpenJ9   - 77c1cf708
OMR      - 20db4fbc
JCL      - 2a7af5674b based on jdk-11.0.5+10
	java.vm.name = Eclipse OpenJ9 VM
	java.vm.specification.name = Java Virtual Machine Specification
	java.vm.specification.vendor = Oracle Corporation
	java.vm.specification.version = 11
	java.vm.vendor = Eclipse OpenJ9
	java.vm.version = openj9-0.17.0
	javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
	jboss.bind.address = 10.129.0.30
	jboss.bind.address.management = 127.0.0.1
	jboss.bind.address.private = 10.129.0.30
	jboss.home.dir = /opt/eap
	jboss.host.name = jgroups-1-dvbcc
	jboss.messaging.cluster.password = <redacted>
	jboss.messaging.host = 10.129.0.30
	jboss.modules.dir = /opt/eap/modules
	jboss.modules.system.pkgs = org.jboss.logmanager,jdk.nashorn.api,com.sun.crypto.provider
	jboss.node.name = jgroups-1-dvbcc
	jboss.qualified.host.name = jgroups-1-dvbcc
	jboss.server.base.dir = /opt/eap/standalone
	jboss.server.config.dir = /opt/eap/standalone/configuration
	jboss.server.data.dir = /opt/eap/standalone/data
	jboss.server.deploy.dir = /opt/eap/standalone/data/content
	jboss.server.log.dir = /opt/eap/standalone/log
	jboss.server.name = jgroups-1-dvbcc
	jboss.server.persist.config = true
	jboss.server.temp.dir = /opt/eap/standalone/tmp
	jdk.extensions.name = Extensions for OpenJDK for Eclipse OpenJ9
	jdk.extensions.version = 11.0.5.0
	line.separator =

	logging.configuration = file:/opt/eap/standalone/configuration/logging.properties
	module.path = /opt/eap/modules
	org.jboss.boot.log.file = /opt/eap/standalone/log/server.log
	org.jboss.resolver.warning = true
	os.arch = s390x
	os.encoding = ANSI_X3.4-1968
	os.name = Linux
	os.version = 4.18.0-80.11.2.el8_0.s390x
	path.separator = :
	sun.arch.data.model = 64
	sun.boot.library.path = /usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib
	sun.cpu.endian = big
	sun.io.unicode.encoding = UnicodeBig
	sun.java.command = /opt/eap/jboss-modules.jar -mp /opt/eap/modules org.jboss.as.standalone -Djboss.home.dir=/opt/eap -Djboss.server.base.dir=/opt/eap/standalone -c standalone-openshift.xml -bmanagement 127.0.0.1 -b 10.129.0.30 -bprivate 10.129.0.30 -Djboss.node.name=jgroups-1-dvbcc -Djboss.messaging.host=10.129.0.30 -Djboss.messaging.cluster.password=HORNETQ_CLUSTER_PASSWORD
	sun.java.launcher = SUN_STANDARD
	sun.java.launcher.pid = 702
	sun.jnu.encoding = ANSI_X3.4-1968
	sun.util.logging.disableCallerCheck = true
	user.country = US
	user.dir = /home/jboss
	user.home = /home/jboss
	user.language = en
	user.name = jboss
	user.timezone = Etc/UTC
	user.variant =
12:50:38,490 DEBUG [org.jboss.as.config] (MSC service thread 1-1) VM Arguments: -Xoptionsfile=/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/options.default -Xlockword:mode=default,noLockword=java/lang/String,noLockword=java/util/MapEntry,noLockword=java/util/HashMap$Entry,noLockword=org/apache/harmony/luni/util/ModifiedMap$Entry,noLockword=java/util/Hashtable$Entry,noLockword=java/lang/invoke/MethodType,noLockword=java/lang/invoke/MethodHandle,noLockword=java/lang/invoke/CollectHandle,noLockword=java/lang/invoke/ConstructorHandle,noLockword=java/lang/invoke/ConvertHandle,noLockword=java/lang/invoke/ArgumentConversionHandle,noLockword=java/lang/invoke/AsTypeHandle,noLockword=java/lang/invoke/ExplicitCastHandle,noLockword=java/lang/invoke/FilterReturnHandle,noLockword=java/lang/invoke/DirectHandle,noLockword=java/lang/invoke/ReceiverBoundHandle,noLockword=java/lang/invoke/DynamicInvokerHandle,noLockword=java/lang/invoke/FieldHandle,noLockword=java/lang/invoke/FieldGetterHandle,noLockword=java/lang/invoke/FieldSetterHandle,noLockword=java/lang/invoke/StaticFieldGetterHandle,noLockword=java/lang/invoke/StaticFieldSetterHandle,noLockword=java/lang/invoke/IndirectHandle,noLockword=java/lang/invoke/InterfaceHandle,noLockword=java/lang/invoke/VirtualHandle,noLockword=java/lang/invoke/PrimitiveHandle,noLockword=java/lang/invoke/InvokeExactHandle,noLockword=java/lang/invoke/InvokeGenericHandle,noLockword=java/lang/invoke/VarargsCollectorHandle,noLockword=java/lang/invoke/ThunkTuple -Xjcl:jclse29 -Dcom.ibm.oti.vm.bootstrap.library.path=/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib -Dsun.boot.library.path=/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib -Djava.library.path=/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib/compressedrefs:/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x/lib:/usr/lib64:/usr/lib -Djava.home=/usr/lib/jvm/java-11-openj9-11.0.5.10.0.17.0-1.el8.s390x -Duser.dir=/home/jboss -D[Standalone] -Xlog:gc*:file=/opt/eap/standalone/log/gc.log:time -Xms1303m -Xmx1303m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.logmanager,jdk.nashorn.api,com.sun.crypto.provider -Djava.awt.headless=true -javaagent:/opt/jboss/container/jolokia/jolokia.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -Xbootclasspath/a:/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-7.2.4.CP/org/jboss/logmanager/main/jboss-logmanager-2.1.14.Final-redhat-00001.jar:/opt/eap/modules/system/layers/base/org/glassfish/javax/json/main/javax.json-1.1.2.redhat-1.jar:/opt/eap/modules/system/layers/base/javax/json/api/main/javax.json-api-1.1.2.redhat-1.jar:/opt/eap/modules/system/layers/base/.overlays/layer-base-jboss-eap-7.2.4.CP/org/wildfly/common/main/wildfly-common-1.5.1.Final-redhat-00001.jar -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dsun.util.logging.disableCallerCheck=true -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom --add-exports=java.base/sun.nio.ch=ALL-UNNAMED --add-exports=jdk.unsupported/sun.misc=ALL-UNNAMED --add-exports=jdk.unsupported/sun.reflect=ALL-UNNAMED --add-modules=java.se -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties -Djava.class.path=/opt/eap/jboss-modules.jar -Dsun.java.command=/opt/eap/jboss-modules.jar -mp /opt/eap/modules org.jboss.as.standalone -Djboss.home.dir=/opt/eap -Djboss.server.base.dir=/opt/eap/standalone -c standalone-openshift.xml -bmanagement 127.0.0.1 -b 10.129.0.30 -bprivate 10.129.0.30 -Djboss.node.name=jgroups-1-dvbcc -Djboss.messaging.host=10.129.0.30 -Djboss.messaging.cluster.password=HORNETQ_CLUSTER_PASSWORD -Dsun.java.launcher=SUN_STANDARD -Dsun.java.launcher.pid=702
12:50:39,951 INFO  [org.wildfly.security] (ServerService Thread Pool -- 29) ELY00001: WildFly Elytron version 1.6.4.Final-redhat-00002
12:50:40,741 INFO  [stdout] (JolokiaStart) I> No access restrictor found, access to any MBean is allowed
12:50:40,825 INFO  [stdout] (JolokiaStart) Jolokia: Agent started with URL https://10.129.0.30:8778/jolokia/
12:50:41,707 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 28) WFLYDR0001: Content added at location /opt/eap/standalone/data/content/f5/57c1134e5395ed3dba006a831c000dafde75e4/content
12:50:42,060 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
12:50:42,066 WARN  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0035: No security realm or http server authentication defined for http management service; all access will be unrestricted.
12:50:42,077 INFO  [org.xnio] (MSC service thread 1-2) XNIO version 3.7.3.Final-redhat-00001
12:50:42,083 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.7.3.Final-redhat-00001
12:50:42,107 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 5.0.14.SP1-redhat-00001
12:50:42,241 INFO  [org.jboss.as.jaxrs] (ServerService Thread Pool -- 51) WFLYRS0016: RESTEasy version 3.6.1.SP6-redhat-00001
12:50:42,274 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 50) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
12:50:42,277 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 63) WFLYNAM0001: Activating Naming Subsystem
12:50:42,288 INFO  [org.wildfly.extension.microprofile.config.smallrye._private] (ServerService Thread Pool -- 60) WFLYCONF0001: Activating WildFly MicroProfile Config Subsystem
12:50:42,326 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 74) WFLYWS0002: Activating WebServices Extension
12:50:42,326 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 69) WFLYSEC0002: Activating Security Subsystem
12:50:42,336 INFO  [org.jboss.as.security] (MSC service thread 1-6) WFLYSEC0001: Current PicketBox version=5.0.3.Final-redhat-00004
12:50:42,342 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 49) WFLYCLINF0001: Activating Infinispan subsystem.
12:50:42,388 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 54) WFLYCLJG0001: Activating JGroups subsystem. JGroups version 4.0.20
12:50:42,417 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
12:50:42,441 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 57) WFLYJSF0007: Activated the following JSF Implementations: [main]
12:50:42,470 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0003: Undertow 2.0.25.SP1-redhat-00001 starting
12:50:42,542 INFO  [org.wildfly.extension.microprofile.health.smallrye] (ServerService Thread Pool -- 61) WFLYHEALTH0001: Activating Eclipse MicroProfile Health Subsystem
12:50:42,544 INFO  [org.jboss.as.connector] (MSC service thread 1-1) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.4.17.Final-redhat-00001)
12:50:42,608 INFO  [org.jboss.as.naming] (MSC service thread 1-5) WFLYNAM0003: Starting Naming Service
12:50:42,629 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 42) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
12:50:42,636 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-8) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
12:50:42,641 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-8) WFLYJCA0018: Started Driver service with driver-name = h2
12:50:42,778 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
12:50:42,903 WARN  [org.jboss.as.remoting] (MSC service thread 1-6) ****** All authentication is ANONYMOUS for org.jboss.as.remoting.RemotingHttpUpgradeService
12:50:42,906 INFO  [org.jboss.as.ejb3] (MSC service thread 1-8) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 16 (per class), which is derived from the number of CPUs on this host.
12:50:42,909 INFO  [org.jboss.as.ejb3] (MSC service thread 1-3) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 64 (per class), which is derived from thread worker pool sizing.
12:50:43,043 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0012: Started server default-server.
12:50:43,056 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) WFLYUT0018: Host default-host starting
12:50:43,062 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
12:50:43,063 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 73) WFLYUT0014: Creating file handler for path '/opt/eap/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
12:50:43,075 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow AJP listener ajp listening on 0.0.0.0:8009
12:50:43,146 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 76) MODCLUSTER000001: Initializing mod_cluster version 1.4.0.Final-redhat-1
12:50:43,181 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 76) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
12:50:43,267 INFO  [org.jboss.as.ejb3] (MSC service thread 1-5) WFLYEJB0493: EJB subsystem suspension complete
12:50:43,492 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-7) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
12:50:43,629 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-8) WFLYMSGAMQ0075: AIO wasn't located on this platform, it will fall back to using pure Java NIO. Your platform is Linux, install LibAIO to enable the AIO journal and achieve optimal performance.
12:50:43,650 INFO  [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: JBoss EAP cumulative patch ID is: jboss-eap-7.2.4.CP, one-off patches include: eap-724-jbeap-17715, eap-724-jbeap-17583
12:50:43,686 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/eap/standalone/deployments
12:50:43,694 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0027: Starting deployment of "ROOT.war" (runtime-name: "ROOT.war")
12:50:43,735 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=/opt/eap/standalone/data/activemq/journal,bindingsDirectory=/opt/eap/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/eap/standalone/data/activemq/largemessages,pagingDirectory=/opt/eap/standalone/data/activemq/paging)
12:50:43,753 INFO  [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022052: Starting JBossWS 5.2.4.Final-redhat-00001 (Apache CXF 3.2.9.redhat-00002)
12:50:43,768 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221013: Using NIO Journal
12:50:43,857 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 683,147,264
12:50:43,875 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
12:50:43,875 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
12:50:43,973 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221034: Waiting indefinitely to obtain live lock
12:50:43,974 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221035: Live Server Obtained live lock
12:50:44,185 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-6) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
12:50:44,186 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-2) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
12:50:44,185 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-8) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor-throughput acceptor
12:50:44,186 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-7) WFLYMSGAMQ0016: Registered HTTP upgrade for activemq-remoting protocol handled by http-acceptor acceptor
12:50:44,378 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221007: Server is now live
12:50:44,378 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 76) AMQ221001: Apache ActiveMQ Artemis Message Broker version 2.9.0.redhat-00005 [default, nodeID=df9e7eea-0481-11ea-8222-0a580a81001e]
12:50:44,394 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-1) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
12:50:44,458 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-2) WFLYJCA0007: Registered connection factory java:/JmsXA
12:50:44,460 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
12:50:44,507 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 76) WFLYMSGAMQ0002: Bound messaging object to jndi name java:jboss/exported/jms/RemoteConnectionFactory
12:50:44,510 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 79) WFLYMSGAMQ0002: Bound messaging object to jndi name java:/ConnectionFactory
12:50:44,561 INFO  [org.apache.activemq.artemis.ra] (MSC service thread 1-2) AMQ151007: Resource adaptor started
12:50:44,562 INFO  [org.jboss.as.connector.services.resourceadapters.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-2) IJ020002: Deployed: file://RaActivatoractivemq-ra
12:50:44,563 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-5) WFLYJCA0002: Bound JCA ConnectionFactory [java:/JmsXA]
12:50:44,563 INFO  [org.wildfly.extension.messaging-activemq] (MSC service thread 1-7) WFLYMSGAMQ0002: Bound messaging object to jndi name java:jboss/DefaultJMSConnectionFactory
12:50:44,583 INFO  [org.jgroups.protocols.openshift.KUBE_PING] (ServerService Thread Pool -- 77) namespace [ibm-tests-full-5] set; clustering enabled

Events show the expected liveness/readiness probe fails and pod termination:

6m        6m        1         jgroups-1-dvbcc.15d61c90855ba863    Pod                                     Normal    Scheduled   default-scheduler                                    Successfully assigned ibm-tests-full-5/jgroups-1-dvbcc to master-1.ocp-s390x-test-ad4607.redhat.com
4m        6m        3         jgroups-1-dvbcc.15d61c92653e18cc    Pod       spec.containers{jgroups}      Normal    Pulling     kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Pulling image "image-registry.openshift-image-registry.svc:5000/ibm-tests-full-5-builds/ha-servlet-counter-jboss@sha256:cf1876d108c55cc55f19a5dff484a7e9364c679c8991a0e6dbea2e5f38f12f45"
5m        6m        2         jgroups-1-dvbcc.15d61c9269042a5a    Pod       spec.containers{jgroups}      Normal    Pulled      kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Successfully pulled image "image-registry.openshift-image-registry.svc:5000/ibm-tests-full-5-builds/ha-servlet-counter-jboss@sha256:cf1876d108c55cc55f19a5dff484a7e9364c679c8991a0e6dbea2e5f38f12f45"
5m        6m        2         jgroups-1-dvbcc.15d61c927ee09d8d    Pod       spec.containers{jgroups}      Normal    Created     kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Created container jgroups
5m        6m        2         jgroups-1-dvbcc.15d61c9281b63c19    Pod       spec.containers{jgroups}      Normal    Started     kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Started container jgroups
6m        6m        1         jgroups-1-dvbcc.15d61c934d5c4c76    Pod       spec.containers{jgroups}      Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Readiness probe failed: {
6m        6m        1         jgroups-1-dvbcc.15d61c94bd7bc829   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Liveness probe failed: {
6m        6m        1         jgroups-1-dvbcc.15d61c959a47f824   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Readiness probe failed: {
6m        6m        1         jgroups-1-dvbcc.15d61c970e9279b6   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Liveness probe failed: {
5m        5m        1         jgroups-1-dvbcc.15d61c97e84d1766   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Readiness probe failed: {
5m        5m        1         jgroups-1-dvbcc.15d61c99626fdfa6   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Liveness probe failed: {
4m        5m        2         jgroups-1-dvbcc.15d61c99627fef2d   Pod       spec.containers{jgroups}   Normal    Killing     kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Container jgroups failed liveness probe, will be restarted
5m        5m        1         jgroups-1-dvbcc.15d61ca07a72ba9d   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Readiness probe errored: rpc error: code = Unknown desc = container is not created or running
5m        5m        1         jgroups-1-dvbcc.15d61ca1556662b8   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Readiness probe failed: {
5m        5m        1         jgroups-1-dvbcc.15d61ca2b5b53030   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   Liveness probe failed: {
1m        5m        24        jgroups-1-dvbcc.15d61ca390e6f436   Pod       spec.containers{jgroups}   Warning   Unhealthy   kubelet, master-1.ocp-s390x-test-ad4607.redhat.com   (combined from similar events): Liveness probe failed: {
6m        6m        1         jgroups-1.15d61c90851ef08c   ReplicationController             Normal    SuccessfulCreate              replication-controller        Created pod: jgroups-1-dvbcc

Attachments

Issue Links

blocks

JBEAP-18095 Provide an EAP Openshift image for IBM Z series

Pull Request Sent

is related to

JBEAP-17656 (7.3.z) '-Xlog:gc' option is not supported on OpenJDK11 + OpenJ9

Verified

CLOUD-3450 Update s390 image builds to 0.18.0 release of OpenJ9 and drop JAVA_TOOL_OPTIONS workaround for java startup times

New

Long startup time on OpenJ9 image causes readiness probe fail

Details

Description

Attachments

Issue Links

Activity

People

Dates