Uploaded image for project: 'Application Server 7'
  1. Application Server 7
  2. AS7-4981

Server instance cannot start after host-contoroller and server instance processes are killed at once in domain mode

    XMLWordPrintable

Details

    • Hide

      Steps to reproduce:
      1. Start server instance in domain mode
      For instance,

      • master node: running "domain-controller" only
      • slave node: running "host-controller" and "server-one"

      2. Kill the slave "host-contoroller" and "server-one" instance processes at once by "kill -SIGKILL" command

       $kill -SIGKILL <host-controller-java-pid> <server-one-java-pid>

      3. Try to stop and start "server-one" by CLI

      $./jboss-cli.sh
[disconnected /] connect master:9999
[domain@master:9999 /] /host=slave/server-config=server-one:stop 
{
    "outcome" => "success",
    "result" => "STOPPING"
}
[domain@master:9999 /] /host=slave/server-config=server-one:start
{
    "outcome" => "failed",
    "result" => undefined,
    "failure-description" => "JBAS014749: Operation handler failed: null",
    "rolled-back" => true
}
      Show
      Steps to reproduce: 1. Start server instance in domain mode For instance, master node: running "domain-controller" only slave node: running "host-controller" and "server-one" 2. Kill the slave "host-contoroller" and "server-one" instance processes at once by "kill -SIGKILL" command $kill -SIGKILL <host-controller-java-pid> <server-one-java-pid> 3. Try to stop and start "server-one" by CLI $./jboss-cli.sh [disconnected /] connect master:9999 [domain@master:9999 /] /host=slave/server-config=server-one:stop { "outcome" => "success" , "result" => "STOPPING" } [domain@master:9999 /] /host=slave/server-config=server-one:start { "outcome" => "failed" , "result" => undefined, "failure-description" => "JBAS014749: Operation handler failed: null " , "rolled-back" => true }

    Description

      When host-contoroller and server instance processes are killed at once in domain mode, the host-controller process is auto-restarted successfully by process-controller but the server instance process is not auto-restarted with the following ERROR in process-controller.log:

      process-controller.log
      16:24:47,599 INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012010: Process 'Host Controller' finished with an exit status of 137
16:24:47,601 INFO  [org.jboss.as.process-controller.server] (reaper for Host Controller) JBAS012020: Waiting 1 seconds until trying to restart process Host Controller. 
16:24:47,624 INFO  [org.jboss.as.process.Server:server-one.status] (reaper for Server:server-one) JBAS012010: Process 'Server:server-one' finished with an exit status of 137
16:24:48,602 INFO  [org.jboss.as.process.Host Controller.status] (reaper for Host Controller) JBAS012017: Starting process 'Host Controller'
16:24:57,088 ERROR [org.jboss.as.process.Server:server-one.status] (ProcessController-threads - 4) JBAS012007: Failed to send reconnect message to process 'Server:server-one' input stream: java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method) [rt.jar:1.6.0_24]
        at java.io.FileOutputStream.write(FileOutputStream.java:297) [rt.jar:1.6.0_24]
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) [rt.jar:1.6.0_24]
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) [rt.jar:1.6.0_24]
        at org.jboss.as.process.ManagedProcess.reconnect(ManagedProcess.java:170) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.ProcessController.sendReconnectProcess(ProcessController.java:336) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.ProcessControllerServerHandler$InitMessageHandler$ConnectedMessageHandler.handleMessage(ProcessControllerServerHandler.java:224) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.protocol.ConnectionImpl.safeHandleMessage(ConnectionImpl.java:269) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.process.protocol.ConnectionImpl$1$1.run(ConnectionImpl.java:223) [jboss-as-process-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT] 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]
        at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]

      After the above happens, server instance proccess cannot start. When I try to invoke stop and start operation by cli (For instance, /host=slave/server-config=server-one:stop and /host=slave/server-config=server-one:start), the following ERROR is thrown in host-controller.log:

      host-controller.log
      16:26:02,501 INFO  [org.jboss.as.host.controller] (domain-connection-threads - 1) JBAS010923: Stopping server server-one
16:26:11,211 ERROR [org.jboss.as.controller.management-operation] (domain-connection-threads - 1) JBAS014612: Operation ("start") failed - address: ([
    ("host" => "slave"),
    ("server-config" => "server-one")
]): java.lang.IllegalStateException
        at org.jboss.as.host.controller.ManagedServer.start(ManagedServer.java:192) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.ServerInventoryImpl.startServer(ServerInventoryImpl.java:166) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.DomainModelControllerService$DelegatingServerInventory.startServer(DomainModelControllerService.java:557) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.host.controller.operations.ServerStartHandler$1.execute(ServerStartHandler.java:80) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]        
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.executeDirect(OperationCoordinatorStepHandler.java:161) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.OperationCoordinatorStepHandler.execute(OperationCoordinatorStepHandler.java:112) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.domain.controller.operations.coordination.PrepareStepHandler.execute(PrepareStepHandler.java:85) [jboss-as-host-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:397) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:284) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:126) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:111) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler.doExecute(TransactionalProtocolOperationHandler.java:112) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ExecuteRequestHandler$1.execute(TransactionalProtocolOperationHandler.java:95) [jboss-as-controller-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:286) [jboss-as-protocol-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:491) [jboss-as-protocol-7.1.3.Final-SNAPSHOT.jar:7.1.3.Final-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.6.0_24]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.6.0_24]
        at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_24]
        at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA.jar:2.0.0.GA]

      Attachments

        Issue Links

          is incorporated by

          Bug - A problem that impairs or prevents the functions of the product. AS7-5662 Cannot start a managed server anymore if it's failed and the host controller is reloaded/restarted

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            People

              emuckenhuber_jira Emanuel Muckenhuber (Inactive)
              rhn-support-mmiura Masafumi Miura
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: