Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-4136

Booting a slave HC fails if the content repository entry for a rollout plan is not present

    Details

    • Steps to Reproduce:
      Hide

      Originally reported reproducer:

      • Create a server-group containing two jvms spanned across two hosts.
      • Shutdown the HostController on one of the hosts
      • Create a new rollout-plan that targets the server-group.
      • Attempt to bring up the HostController that was shutdown.

      This is what I did:

      • Unzip the appserver in two different dirs, 'master' and 'slave'
      • In master:
        • bin/jboss-cli.sh
        • embed-host-controller --host-config=host-master.xml
        • rollout-plan add --name=test --content= {rollout main-server-group(rolling-to-servers=true) rollback-across-groups}
        • quit
        • bin/domain.sh --host-config=host-master.xml
      • In 'slave'
        • bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.0 -Djboss.domain.management.port=19999

      Booting the slave fails

      Show
      Originally reported reproducer: Create a server-group containing two jvms spanned across two hosts. Shutdown the HostController on one of the hosts Create a new rollout-plan that targets the server-group. Attempt to bring up the HostController that was shutdown. This is what I did: Unzip the appserver in two different dirs, 'master' and 'slave' In master: bin/jboss-cli.sh embed-host-controller --host-config=host-master.xml rollout-plan add --name=test --content= {rollout main-server-group(rolling-to-servers=true) rollback-across-groups} quit bin/domain.sh --host-config=host-master.xml In 'slave' bin/domain.sh --host-config=host-slave.xml -Djboss.domain.master.address=127.0.0.0 -Djboss.domain.management.port=19999 Booting the slave fails
    • Workaround:
      Workaround Exists
    • Workaround Description:
      Hide

      Add a 'ignored-resources' element to the domain-controller/remote element in the slave's host xml file to ignore management-client-content resources. For example:

          <domain-controller>
              <remote security-realm="ManagementRealm">
      	    <ignored-resources type="management-client-content" wildcard="true"/>
                  <discovery-options>
                      <static-discovery name="primary" protocol="${jboss.domain.master.protocol:remote}" host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}"/>
                  </discovery-options>
              </remote>
          </domain-controller>
      

      This allows the slave to boot. However it removes the ability for that slave to use pre-stored rollout plans when deciding how to rollout changes targeted at it to its servers (i.e. operations targeted at addresses in the /host=slave resource tree.) However, most operations involving rolling plans are targeted at the DC, which will have access to the rollout plans.

      Show
      Add a 'ignored-resources' element to the domain-controller/remote element in the slave's host xml file to ignore management-client-content resources. For example: <domain-controller> <remote security-realm= "ManagementRealm" > <ignored-resources type= "management-client-content" wildcard= " true " /> <discovery-options> < static -discovery name= "primary" protocol= "${jboss.domain.master.protocol:remote}" host= "${jboss.domain.master.address}" port= "${jboss.domain.master.port:9999}" /> </discovery-options> </remote> </domain-controller> This allows the slave to boot. However it removes the ability for that slave to use pre-stored rollout plans when deciding how to rollout changes targeted at it to its servers (i.e. operations targeted at addresses in the /host=slave resource tree.) However, most operations involving rolling plans are targeted at the DC, which will have access to the rollout plans.

      Description

      Slave HC fails to boot if it connects to a DC that has a rollout-plan configured when the slave was shut down.

      See Steps to Reproduce.

      Failure is:

      [Host Controller] 14:51:04,208 ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 4) WFLYCTL0013: Operation ("add") failed - address: ([("management-client-content" => "rollout-plans")]): org.jboss.as.management.client.content.ContentStorageException: java.io.FileNotFoundException: /Users/bstansberry/tmp/support/02184219/slave/domain/data/content/2d/56893cc4da1c67a79f4845a57ae90c04a969b4/content (No such file or directory)
      [Host Controller] 	at org.jboss.as.management.client.content.ManagedDMRContentTypeResource.loadContent(ManagedDMRContentTypeResource.java:306)
      [Host Controller] 	at org.jboss.as.management.client.content.ManagedDMRContentTypeResource.<init>(ManagedDMRContentTypeResource.java:91)
      [Host Controller] 	at org.jboss.as.management.client.content.ManagedDMRContentTypeAddHandler.execute(ManagedDMRContentTypeAddHandler.java:58)
      [Host Controller] 	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:982)
      [Host Controller] 	at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:726)
      [Host Controller] 	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
      [Host Controller] 	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
      [Host Controller] 	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:418)
      [Host Controller] 	at org.jboss.as.controller.AbstractControllerService.internalExecute(AbstractControllerService.java:490)
      [Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService.access$1100(DomainModelControllerService.java:200)
      [Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.lambda$apply$0(DomainModelControllerService.java:1372)
      [Host Controller] 	at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:63)
      [Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1372)
      [Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor$1.apply(DomainModelControllerService.java:1369)
      [Host Controller] 	at org.jboss.as.host.controller.SecurityActions$Execution$1.execute(SecurityActions.java:77)
      [Host Controller] 	at org.jboss.as.host.controller.SecurityActions.privilegedExecution(SecurityActions.java:47)
      [Host Controller] 	at org.jboss.as.host.controller.DomainModelControllerService$InternalExecutor.execute(DomainModelControllerService.java:1375)
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnectionService.applyRemoteDomainModel(RemoteDomainConnectionService.java:603)
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnectionService.access$1200(RemoteDomainConnectionService.java:138)
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnectionService$3.applyDomainModel(RemoteDomainConnectionService.java:546)
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection.applyDomainModel(RemoteDomainConnection.java:325)
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection$RegisterSubsystemsRequest$1.execute(RemoteDomainConnection.java:468)
      [Host Controller] 	at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$1.doExecute(ManagementRequestContextImpl.java:70)
      [Host Controller] 	at org.jboss.as.protocol.mgmt.ManagementRequestContextImpl$AsyncTaskRunner.run(ManagementRequestContextImpl.java:160)
      [Host Controller] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      [Host Controller] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      [Host Controller] 	at java.lang.Thread.run(Thread.java:748)
      [Host Controller] 	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      [Host Controller] Caused by: java.io.FileNotFoundException: /Users/bstansberry/tmp/support/02184219/slave/domain/data/content/2d/56893cc4da1c67a79f4845a57ae90c04a969b4/content (No such file or directory)
      [Host Controller] 	at java.io.FileInputStream.open0(Native Method)
      [Host Controller] 	at java.io.FileInputStream.open(FileInputStream.java:195)
      [Host Controller] 	at java.io.FileInputStream.<init>(FileInputStream.java:138)
      [Host Controller] 	at org.jboss.vfs.spi.RootFileSystem.openInputStream(RootFileSystem.java:51)
      [Host Controller] 	at org.jboss.vfs.VirtualFile.openStream(VirtualFile.java:318)
      [Host Controller] 	at org.jboss.as.management.client.content.ManagedDMRContentTypeResource.loadContent(ManagedDMRContentTypeResource.java:292)
      [Host Controller] 	... 27 more
      [Host Controller] 
      [Host Controller] 14:51:04,212 ERROR [org.jboss.as.host.controller] (Host Controller Service Threads - 4) WFLYHC0143: Failed to apply domain-wide configuration from master host controller. Operation outcome: failed. Failure description {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-552" => "WFLYCTL0158: Operation handler failed: org.jboss.as.management.client.content.ContentStorageException: java.io.FileNotFoundException: /Users/bstansberry/tmp/support/02184219/slave/domain/data/content/2d/56893cc4da1c67a79f4845a57ae90c04a969b4/content (No such file or directory)"}}
      [Host Controller] 14:51:04,218 WARN  [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0001: Could not connect to remote domain controller remote://127.0.0.1:9999: 1-$-
      [Host Controller] 	at org.jboss.as.host.controller.RemoteDomainConnection$RegisterSubsystemsRequest$1.execute(RemoteDomainConnection.java:472)
      ...
      [Host Controller] 14:51:04,219 WARN  [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0147: No domain controller discovery options remain.
      [Host Controller] 14:51:04,220 ERROR [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0002: Could not connect to master. Error was: java.lang.IllegalStateException: WFLYHC0120: Tried all domain controller discovery option(s) but unable to connect
      [Host Controller] 14:51:04,221 FATAL [org.jboss.as.host.controller] (Controller Boot Thread) WFLYHC0178: Aborting with exit code 99
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  brian.stansberry Brian Stansberry
                  Reporter:
                  brian.stansberry Brian Stansberry
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: