-
Bug
-
Resolution: Won't Do
-
Critical
-
None
-
7.0.0.DR9
-
None
Some of our tests for large domain sometimes fail with error: "Writes closed". It happens only with some higher number of servers per host.
Failed response: { "outcome" => "failed", "result" => { "step-1" => { "outcome" => "failed", "rolled-back" => true }, "step-2" => { "outcome" => "failed", "rolled-back" => true }, "step-3" => { "outcome" => "failed", "rolled-back" => true } }, "failure-description" => {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => { "server-group-A" => {"host" => { "perf31" => { "perf31-server-3" => "Writes closed", "perf31-server-5" => "Writes closed" }, "perf33" => {"perf33-server-3" => "Writes closed"} }}, "server-group-B" => {"host" => {"perf32" => { "perf32-server-2" => "Writes closed", "perf32-server-6" => "Writes closed" }}} }}}, "rolled-back" => true, "server-groups" => { "server-group-A" => {"host" => { "perf31" => { "perf31-server-1" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf31-server-3" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => "Writes closed", "rolled-back" => true }}, "perf31-server-5" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => "Writes closed", "rolled-back" => true }}, "perf31-server-7" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf31-server-9" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} }, "perf32" => { "perf32-server-1" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf32-server-3" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => undefined, "rolled-back" => true }}, "perf32-server-5" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf32-server-7" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf32-server-9" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} }, "perf33" => { "perf33-server-1" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-3" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => "Writes closed", "rolled-back" => true }}, "perf33-server-5" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-7" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-9" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} } }}, "server-group-B" => {"host" => { "perf31" => { "perf31-server-10" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf31-server-2" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => undefined, "rolled-back" => true }}, "perf31-server-4" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => undefined, "rolled-back" => true }}, "perf31-server-6" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf31-server-8" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} }, "perf32" => { "perf32-server-10" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf32-server-2" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => "Writes closed", "rolled-back" => true }}, "perf32-server-4" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => undefined, "rolled-back" => true }}, "perf32-server-6" => {"response" => { "outcome" => "failed", "result" => undefined, "failure-description" => "Writes closed", "rolled-back" => true }}, "perf32-server-8" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} }, "perf33" => { "perf33-server-10" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-2" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-4" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-6" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }}, "perf33-server-8" => {"response" => { "outcome" => "failed", "result" => {"step-1" => { "outcome" => "failed", "rolled-back" => true }}, "failure-description" => "WFLYCTL0063: Composite operation was rolled back", "rolled-back" => true }} } }} } }
Stacktrace from one of slaves:
2015-09-08 06:47:23,049 DEBUG [org.jboss.as.protocol] (Host Controller Service Threads - 124) active-op (1367836977) failed org.jboss.as.controller.remote.TransactionalProtocolClientImpl$ExecuteRequestContext@541c0a2: org.jboss.remoting3.NotOpenException: Writes closed at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:117) at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:306) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$1.writeMessage(AbstractMessageHandler.java:214) at org.jboss.as.protocol.mgmt.AbstractManagementRequest.sendRequest(AbstractManagementRequest.java:51) at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest.sendRequestInternal(TransactionalProtocolClientImpl.java:290) at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest.access$500(TransactionalProtocolClientImpl.java:242) at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest$1.execute(TransactionalProtocolClientImpl.java:264) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$1$1.doExecute(AbstractMessageHandler.java:176) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320)
I know this isn't detailed bug report, I didn't find easy way to reproduce it, other than running large domain tests. Tests are modified tests from https://github.com/wildfly/wildfly/blob/master/testsuite/domain/src/test/java/org/jboss/as/test/integration/domain/suites/DeploymentManagementTestCase.java .
I can provide other logs, outputs ...., just let me know which log categories would help.
Also I tried to play a bit with remoting's HEARTBEAT interval (through subsystem configuration), but that doesn't help.
- clones
-
WFCORE-965 "Writes closed" when deploying application in domain
- Resolved