Uploaded image for project: 'Fabric8'
  1. Fabric8
  2. FABRIC-45

provide a file <-> ZK Registry bridge so we can watch for changes in a file system and apply any updates to the config area of the ZK registry

    Details

    • Type: Enhancement
    • Status: Resolved
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.1-old
    • Component/s: None
    • Labels:
      None
    • Similar Issues:
      Show 10 results 

      Description

      e.g. so folks could use a file system (then use svn / git / whatever to version control it) then mirror the configuration to various environments.

      Then folks could either just run a command now and again on a machine (e.g. on a svn/git commit hook) to check out the configuration as files, iterate through it and compare it to ZK and apply any changes).

      We could then mirror contents of ZK as a tree of property files (or JSON/XML docs) using folders for folders in ZK etc.

      Folks could then easily install a configuration in ZK from a file; or do the heroku style model where fabrics can be tied to a branch in a SCM; then the SCM machine can replicate changes to the ZK fabric registries; without the ZK servers having to communicate with unreliable SCM servers etc.

        Gliffy Diagrams

          Activity

          Hide
          jastrachan James Strachan added a comment -

          I guess we probably want to allow the other way; if a change has been made in ZK we may wish to replicate it to the source code again (e.g. if folks use a UI or command line shell to update the ZK registry).

          So we could maybe do things the other way too - checkout from the source code, then overwrite the source code with the current values from ZK; then folks could have a process to commit the changes.

          This would then catch & version changes in the ZK config.

          Care would be needed to avoid having any time based changes the config; to avoid infinite loops of change in ZK -> source change -> ZK change -> ... etc

          Show
          jastrachan James Strachan added a comment - I guess we probably want to allow the other way; if a change has been made in ZK we may wish to replicate it to the source code again (e.g. if folks use a UI or command line shell to update the ZK registry). So we could maybe do things the other way too - checkout from the source code, then overwrite the source code with the current values from ZK; then folks could have a process to commit the changes. This would then catch & version changes in the ZK config. Care would be needed to avoid having any time based changes the config; to avoid infinite loops of change in ZK -> source change -> ZK change -> ... etc
          Hide
          stlewis Stan Lewis added a comment - - edited

          Looks like zk:export and zk:import could allow this, right now you can do zk:export, copy paste the output to some file, make changes and then pass the URI of that file to zk:import, but they seem incomplete, for example if I pass a properties file to import I can't specify what path it should go in, so all the keys wind up at the root of the ZK config. Do you think though that if the import/export commands were more user friendly this would fit the bill? The problem with trying to do a live sync between the file system and ZK I can see is how to determine who has the definitive version of the config at a given time.

          Show
          stlewis Stan Lewis added a comment - - edited Looks like zk:export and zk:import could allow this, right now you can do zk:export, copy paste the output to some file, make changes and then pass the URI of that file to zk:import, but they seem incomplete, for example if I pass a properties file to import I can't specify what path it should go in, so all the keys wind up at the root of the ZK config. Do you think though that if the import/export commands were more user friendly this would fit the bill? The problem with trying to do a live sync between the file system and ZK I can see is how to determine who has the definitive version of the config at a given time.
          Hide
          jastrachan James Strachan added a comment -

          So maybe we start with a recursive import and export; where we can import from a directory to a given path in the config - or export from a path in ZK to a directory?

          Once we've recursive import/exports, we should be able to build a 'if it changes in git/svn, update ZK' little heroku style example, where folks can hack files in source control then update ZK on a push.

          i.e. the git on commit handler would checkout git and do a recursive ZK import.

          Folks could even modify this so that only certain parts of the tree are imported from SCM etc.

          Extra bonus points - we could maybe have a .fabricignore file like a .gitignore where it lists the ZK paths to ignore when exporting maybe? On import we could maybe delete stuff not in ZK (unless the paths are in the .fabricignore) - so if you delete something in the file system, it'll be deleted in ZK.

          Then using the .fabricignore files; we can define 'holes' in the ZK tree where we don't import/export (or delete in ZK).

          The only worry with this I can see is if folks decide to setup a heroku style git commit hook to update ZK; if folks also setup a 'on ZK change export to file system' thingy; then automatically do a commit & push; we could end up with a ZK <-> git infinite loop. Maybe some code review may be required to move the 'runtime changes' branch into the 'heroku deploy branch' in git? I guess folks can figure out (outside of our fabric java code) how best they want to deal with this stuff.

          Or to say that another way - if we can provide flexible recursive import & export - with an easy way to ignore bits of the tree or certain named entries with wildcards etc - folks can build whatever file system / SCM / operational / workflow mechanism they wish to maintaining the fabric config.

          Show
          jastrachan James Strachan added a comment - So maybe we start with a recursive import and export; where we can import from a directory to a given path in the config - or export from a path in ZK to a directory? Once we've recursive import/exports, we should be able to build a 'if it changes in git/svn, update ZK' little heroku style example, where folks can hack files in source control then update ZK on a push. i.e. the git on commit handler would checkout git and do a recursive ZK import. Folks could even modify this so that only certain parts of the tree are imported from SCM etc. Extra bonus points - we could maybe have a .fabricignore file like a .gitignore where it lists the ZK paths to ignore when exporting maybe? On import we could maybe delete stuff not in ZK (unless the paths are in the .fabricignore) - so if you delete something in the file system, it'll be deleted in ZK. Then using the .fabricignore files; we can define 'holes' in the ZK tree where we don't import/export (or delete in ZK). The only worry with this I can see is if folks decide to setup a heroku style git commit hook to update ZK; if folks also setup a 'on ZK change export to file system' thingy; then automatically do a commit & push; we could end up with a ZK <-> git infinite loop. Maybe some code review may be required to move the 'runtime changes' branch into the 'heroku deploy branch' in git? I guess folks can figure out (outside of our fabric java code) how best they want to deal with this stuff. Or to say that another way - if we can provide flexible recursive import & export - with an easy way to ignore bits of the tree or certain named entries with wildcards etc - folks can build whatever file system / SCM / operational / workflow mechanism they wish to maintaining the fabric config.
          Hide
          stlewis Stan Lewis added a comment - - edited

          Okay, have improved export/import and also the "list" command.

          zk:export by default will export the whole ZK tree into an "export" directory in the PWD of karaf. You can change this by specifying some other target as the argument for this command. You can specify -p/ --path to export just a subset of the ZK tree. Finally you can specify one or more -f arguments with a regex (uses java.util.regex stuff) to filter what paths are exported. The format of the exported filesystem is kinda like sysfs, where the file is the config setting name and the contents of the file is the value. I had to append ".cfg" on everything because you can have a path have a value and have child paths. Oh also you can specify -d/ --delete and the export operation will wipe out the target directory first and re-create.

          zk:import will now by default import a filesystem tree like what's created with zk:export. You can filter what's imported with one or more -f options. You can specify a target location in the ZK tree with -t/ --target. You can specify -props and supply a URL to a .properties file and the command will import that properties file (this was the old behavior), though of course now you can put those properties anywhere in the tree using -t. Also you can specify -d (for dangerous and anything not in the imported tree that's in zookeeper will be deleted.

          zk:list now defaults to listing the root of the hierarchy, I also added a -d/ --display option to also show values, so you can see your whole ZK tree via "zk:list -r -d"

          Show
          stlewis Stan Lewis added a comment - - edited Okay, have improved export/import and also the "list" command. zk:export by default will export the whole ZK tree into an "export" directory in the PWD of karaf. You can change this by specifying some other target as the argument for this command. You can specify -p/ --path to export just a subset of the ZK tree. Finally you can specify one or more -f arguments with a regex (uses java.util.regex stuff) to filter what paths are exported. The format of the exported filesystem is kinda like sysfs, where the file is the config setting name and the contents of the file is the value. I had to append ".cfg" on everything because you can have a path have a value and have child paths. Oh also you can specify -d/ --delete and the export operation will wipe out the target directory first and re-create. zk:import will now by default import a filesystem tree like what's created with zk:export. You can filter what's imported with one or more -f options. You can specify a target location in the ZK tree with -t/ --target. You can specify -props and supply a URL to a .properties file and the command will import that properties file (this was the old behavior), though of course now you can put those properties anywhere in the tree using -t. Also you can specify -d (for dangerous and anything not in the imported tree that's in zookeeper will be deleted. zk:list now defaults to listing the root of the hierarchy, I also added a -d/ --display option to also show values, so you can see your whole ZK tree via "zk:list -r -d"
          Hide
          jastrachan James Strachan added a comment -

          Great stuff! Will have a play!

          Show
          jastrachan James Strachan added a comment - Great stuff! Will have a play!
          Hide
          stlewis Stan Lewis added a comment -

          Thanks! Was also thinking a --dry-run option would be good too for testing that an import/export with filters does what you want.

          Show
          stlewis Stan Lewis added a comment - Thanks! Was also thinking a --dry-run option would be good too for testing that an import/export with filters does what you want.
          Hide
          stlewis Stan Lewis added a comment -

          Added support for a .fabricignore file as well as a .fabricinclude file. The import/export command will check Karaf's PWD for both .fabricignore and .fabricinclude and add the contents (one regex per line) to the list of regex's each path is checked against. So if you create a .fabricignore file containing:

          ^/zookeeper\S+
          ^/zookeeper
          ^/fabric/registry\S+
          ^/fabric/registry

          And do a zk:export -d you shouldn't see /fabric/registry or /zookeeper in the filesystem. Looks like I have to double-check that zk:import is working properly.

          Show
          stlewis Stan Lewis added a comment - Added support for a .fabricignore file as well as a .fabricinclude file. The import/export command will check Karaf's PWD for both .fabricignore and .fabricinclude and add the contents (one regex per line) to the list of regex's each path is checked against. So if you create a .fabricignore file containing: ^/zookeeper\S+ ^/zookeeper ^/fabric/registry\S+ ^/fabric/registry And do a zk:export -d you shouldn't see /fabric/registry or /zookeeper in the filesystem. Looks like I have to double-check that zk:import is working properly.
          Hide
          jastrachan James Strachan added a comment -

          Awesome!!!

          Show
          jastrachan James Strachan added a comment - Awesome!!!

            People

            • Assignee:
              stlewis Stan Lewis
              Reporter:
              jastrachan James Strachan
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development