Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-5022

Authentication Error when Adding Containers to Ensemble

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: jboss-fuse-6.2.1
    • Fix Version/s: None
    • Component/s: Fabric8 v1
    • Labels:
      None
    • Environment:

      Fuse 6.2.1 GA
      RHEL 6 / 7
      OpenJDK 7.x

    • Steps to Reproduce:
      Hide

      Note: These directions represent the steps followed to create an ensemble with an external git repo. I have seen this issue pop up without an external repo, as well, but the issue reproduces faithfully in this scenario.

      1. Install 3 instances of Fuse 6.2.1
      
      2. Configure etc/io.fabric8.datastore.cfg on each instance to hold the authentication info for external git (workaround for https://issues.jboss.org/browse/ENTESB-4939 - original reproducer used a locally available gitlab instance)
      
      3. Configure etc/users.properties to give the admin user the same authentication tokens as the git user (user/password) - may not be necessary, but this is how it was configured in my environment (this may not be necessary to reproduce the issue, but was configured this way in the original reproducer e.g. change "admin=admin,admin... to <gituser>=<gitpasswd>,admin.. in users.properties before first container boot.)
      
      4. Create a fabric from the first node with: fabric:create --wait-for-provisioning --external-git-url http://gitlab/gituser/ensemble.git --external-git-user <gituser> --external-git-password <gitpassword> --zookeeper-password <gitpassword>
      
      5. Install fabric-zookeeper-utils: fabric:profile-edit --feature fabric-zookeeper-commands/0.0.0 default
      
      6. From the second two nodes, join the first: fabric:join --zookeeper-password <gitpassword> node0:2181 node1,  fabric:join --zookeeper-password <gitpassword> node0:2181 node2.  I had to restart these containers from once to several times to get them to finish provisioning.
      
      7. From the original fabric container, execute: ensemble-add node1 node2 and answer yes to the warning prompt.
      
      8. Operation fails with: "Error executing command: io.fabric8.api.FabricAuthenticationException: java.lang.SecurityException: Authentication failed" and a corresponding stacktrace in the log.
      
      9. Verify that node /fabric/authentication/containers/root (where root is the nodename for the first ensemble container) is missing with zk:get /fabric/authentication/containers/root.  You should see something like "Error executing command: KeeperErrorCode = NoNode for /fabric/authentication/containers/root"
      
      10. Insert the missing nodes with: 
      zk:create /fabric/authentication/containers
      zk:create /fabric/authentication/containers/root
      zk:set /fabric/authentication/containers ""
      zk:set /fabric/authentication/containers/root ""
      
      11. Retry the ensemble-add command.
      
      12. Ensemble expansion now succeeds
      
      Show
      Note: These directions represent the steps followed to create an ensemble with an external git repo. I have seen this issue pop up without an external repo, as well, but the issue reproduces faithfully in this scenario. 1. Install 3 instances of Fuse 6.2.1 2. Configure etc/io.fabric8.datastore.cfg on each instance to hold the authentication info for external git (workaround for https: //issues.jboss.org/browse/ENTESB-4939 - original reproducer used a locally available gitlab instance) 3. Configure etc/users.properties to give the admin user the same authentication tokens as the git user (user/password) - may not be necessary, but this is how it was configured in my environment ( this may not be necessary to reproduce the issue, but was configured this way in the original reproducer e.g. change "admin=admin,admin... to <gituser>=<gitpasswd>,admin.. in users.properties before first container boot.) 4. Create a fabric from the first node with: fabric:create --wait- for -provisioning --external-git-url http: //gitlab/gituser/ensemble.git --external-git-user <gituser> --external-git-password <gitpassword> --zookeeper-password <gitpassword> 5. Install fabric-zookeeper-utils: fabric:profile-edit --feature fabric-zookeeper-commands/0.0.0 default 6. From the second two nodes, join the first: fabric:join --zookeeper-password <gitpassword> node0:2181 node1, fabric:join --zookeeper-password <gitpassword> node0:2181 node2. I had to restart these containers from once to several times to get them to finish provisioning. 7. From the original fabric container, execute: ensemble-add node1 node2 and answer yes to the warning prompt. 8. Operation fails with: "Error executing command: io.fabric8.api.FabricAuthenticationException: java.lang.SecurityException: Authentication failed" and a corresponding stacktrace in the log. 9. Verify that node /fabric/authentication/containers/root (where root is the nodename for the first ensemble container) is missing with zk:get /fabric/authentication/containers/root. You should see something like "Error executing command: KeeperErrorCode = NoNode for /fabric/authentication/containers/root" 10. Insert the missing nodes with: zk:create /fabric/authentication/containers zk:create /fabric/authentication/containers/root zk:set /fabric/authentication/containers "" zk:set /fabric/authentication/containers/root "" 11. Retry the ensemble-add command. 12. Ensemble expansion now succeeds
    • Workaround Description:
      Hide

      Included in reproducer steps.

      Show
      Included in reproducer steps.

      Description

      When starting with three karaf containers (say, node0, node1, and node2) and creating a fabric from on of the three (say node0), joining from the other two, then executing ensemble add <container 2> <container 3> from the first container, the user is met with

      Error executing command: io.fabric8.api.FabricAuthenticationException: java.lang.SecurityException: Authentication failed
      

      Debugging though, the auth failure seems coming from the createConnector method of ContainerTemplate. The source of the issue seems to be a missing node /fabric/authentication/containers/root in the zookeeper registry causing null to be returned as the container auth password by ZooKeeperUtils.getContainerTokens(CuratorFramework). Inserting an empty node at that location with:

      zk:create /fabric/authentication/containers
      zk:create /fabric/authentication/containers/root
      zk:set /fabric/authentication/containers ""
      zk:set /fabric/authentication/containers/root ""
      

      where root represents the name of the first ensemble container seems to be enough to resolve the issue.

      In the original case, the customer was using an external git repository, but this issue seems to be encountered with the normal internal repo, as well.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  sonicaaaa Paolo Antinori
                  Reporter:
                  hawkinsds Duane Hawkins
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: