Details
-
Bug
-
Resolution: Won't Do
-
Critical
-
None
-
7.0.0.DR9
-
None
Description
Some of our tests for large domain sometimes fail with error: "Writes closed". It happens only with some higher number of servers per host.
Failed response:
{
"outcome" => "failed",
"result" => {
"step-1" => {
"outcome" => "failed",
"rolled-back" => true
},
"step-2" => {
"outcome" => "failed",
"rolled-back" => true
},
"step-3" => {
"outcome" => "failed",
"rolled-back" => true
}
},
"failure-description" => {"WFLYDC0074: Operation failed or was rolled back on all servers. Server failures:" => {"server-group" => {
"server-group-A" => {"host" => {
"perf31" => {
"perf31-server-3" => "Writes closed",
"perf31-server-5" => "Writes closed"
},
"perf33" => {"perf33-server-3" => "Writes closed"}
}},
"server-group-B" => {"host" => {"perf32" => {
"perf32-server-2" => "Writes closed",
"perf32-server-6" => "Writes closed"
}}}
}}},
"rolled-back" => true,
"server-groups" => {
"server-group-A" => {"host" => {
"perf31" => {
"perf31-server-1" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf31-server-3" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => "Writes closed",
"rolled-back" => true
}},
"perf31-server-5" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => "Writes closed",
"rolled-back" => true
}},
"perf31-server-7" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf31-server-9" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
},
"perf32" => {
"perf32-server-1" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf32-server-3" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => undefined,
"rolled-back" => true
}},
"perf32-server-5" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf32-server-7" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf32-server-9" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
},
"perf33" => {
"perf33-server-1" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-3" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => "Writes closed",
"rolled-back" => true
}},
"perf33-server-5" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-7" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-9" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
}
}},
"server-group-B" => {"host" => {
"perf31" => {
"perf31-server-10" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf31-server-2" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => undefined,
"rolled-back" => true
}},
"perf31-server-4" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => undefined,
"rolled-back" => true
}},
"perf31-server-6" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf31-server-8" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
},
"perf32" => {
"perf32-server-10" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf32-server-2" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => "Writes closed",
"rolled-back" => true
}},
"perf32-server-4" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => undefined,
"rolled-back" => true
}},
"perf32-server-6" => {"response" => {
"outcome" => "failed",
"result" => undefined,
"failure-description" => "Writes closed",
"rolled-back" => true
}},
"perf32-server-8" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
},
"perf33" => {
"perf33-server-10" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-2" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-4" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-6" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}},
"perf33-server-8" => {"response" => {
"outcome" => "failed",
"result" => {"step-1" => {
"outcome" => "failed",
"rolled-back" => true
}},
"failure-description" => "WFLYCTL0063: Composite operation was rolled back",
"rolled-back" => true
}}
}
}}
}
}
Stacktrace from one of slaves:
2015-09-08 06:47:23,049 DEBUG [org.jboss.as.protocol] (Host Controller Service Threads - 124) active-op (1367836977) failed org.jboss.as.controller.remote.TransactionalProtocolClientImpl$ExecuteRequestContext@541c0a2: org.jboss.remoting3.NotOpenException: Writes closed
at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:117)
at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:306)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$1.writeMessage(AbstractMessageHandler.java:214)
at org.jboss.as.protocol.mgmt.AbstractManagementRequest.sendRequest(AbstractManagementRequest.java:51)
at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest.sendRequestInternal(TransactionalProtocolClientImpl.java:290)
at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest.access$500(TransactionalProtocolClientImpl.java:242)
at org.jboss.as.controller.remote.TransactionalProtocolClientImpl$CompleteTxRequest$1.execute(TransactionalProtocolClientImpl.java:264)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$1$1.doExecute(AbstractMessageHandler.java:176)
at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
I know this isn't detailed bug report, I didn't find easy way to reproduce it, other than running large domain tests. Tests are modified tests from https://github.com/wildfly/wildfly/blob/master/testsuite/domain/src/test/java/org/jboss/as/test/integration/domain/suites/DeploymentManagementTestCase.java .
I can provide other logs, outputs ...., just let me know which log categories would help.
Also I tried to play a bit with remoting's HEARTBEAT interval (through subsystem configuration), but that doesn't help.
Attachments
Issue Links
- clones
-
-
- Resolved
-
