Infinispan
  1. Infinispan
  2. ISPN-3318

Migrate data from one cache store to another

    Details

    • Similar Issues:
      Show 10 results 

      Description

      Find a generic way to transfer data from one cache store to another, which could involve different Infinispan versions. This is handy to migrate file cache store based users to single file cache store (ISPN-2806).

      Ideally, this should be added as a recipe for rolling upgrades.

        Activity

        Hide
        Galder Zamarreño
        added a comment -

        Interesting idea from Randall:

        Let's look at this from a user's perspective. Is it possible that the new file store *replaces* the old one, so that the user doesn't (really) need to know the difference?

        How similar are the configuration properties of each? Might it be possible to:
        rename and deprecate the existing one (e.g., "org.infinispan.loaders.file.LegacyFileCacheStore"), changing it to 'package' or 'protected' (iff the new one is stable enough that the old one should definitely not be used anymore),
        rename the new one to "org.infinispan.loaders.file.FileCacheStore" (so that existing configurations work without change),
        have some code in the new one detect the old file structure and automatically run a migration to convert from the old file structure to the new format (and possibly move the old files into a subdirectory)

        If this is possible, this is by far the best option for users since it provides a simple (perhaps even transparent) migration path with little if any changes required to configurations, but at the same time it allows the developers to get what they want: a new FCS that replaces the old one (which goes away).

        Show
        Galder Zamarreño
        added a comment - Interesting idea from Randall: Let's look at this from a user's perspective. Is it possible that the new file store * replaces * the old one, so that the user doesn't (really) need to know the difference? How similar are the configuration properties of each? Might it be possible to: rename and deprecate the existing one (e.g., "org.infinispan.loaders.file.LegacyFileCacheStore"), changing it to 'package' or 'protected' (iff the new one is stable enough that the old one should definitely not be used anymore), rename the new one to "org.infinispan.loaders.file.FileCacheStore" (so that existing configurations work without change), have some code in the new one detect the old file structure and automatically run a migration to convert from the old file structure to the new format (and possibly move the old files into a subdirectory) If this is possible, this is by far the best option for users since it provides a simple (perhaps even transparent) migration path with little if any changes required to configurations, but at the same time it allows the developers to get what they want: a new FCS that replaces the old one (which goes away).
        Hide
        Galder Zamarreño
        added a comment -
        Show
        Galder Zamarreño
        added a comment - Check also Mircea's suggestion in http://lists.jboss.org/pipermail/infinispan-dev/2013-July/013479.html
        Hide
        Mircea Markus
        added a comment -

        We need to keep the old file cache store around for the backward compatibility (JDG requirement).
        We'd also need a migration tool for migrating other cache stores (e.g. LevelDB cache store from certain users) so +1 for the migration tool.

        Show
        Mircea Markus
        added a comment - We need to keep the old file cache store around for the backward compatibility (JDG requirement). We'd also need a migration tool for migrating other cache stores (e.g. LevelDB cache store from certain users) so +1 for the migration tool.
        Hide
        Galder Zamarreño
        added a comment -

        We need to keep the old file cache store around for the backward compatibility (JDG requirement).

        Mircea, Randall's suggestion is compatible with what you're saying. The idea is that anyone that would use the FCS with Infinispan 6.0+, if FCS would be able to detect old FCS data structure, it could migrate it on the fly to the new single file based structure. That is backwards compatible and keeps the old file cache store code around, but transforms its inner details.

        We'd also need a migration tool for migrating other cache stores (e.g. LevelDB cache store from certain users) so +1 for the migration tool.

        Indeed, that might be needed, but for the specific case of the new FCS upgrade, you might not even need that, which is even better. If people want to migrate data from cache store X to cache store Y on a whim, then yes, you need a migration tool.

        Disclaimer: I have not investigated yet the feasibility of Randall's approach.

        Show
        Galder Zamarreño
        added a comment - We need to keep the old file cache store around for the backward compatibility (JDG requirement). Mircea, Randall's suggestion is compatible with what you're saying. The idea is that anyone that would use the FCS with Infinispan 6.0+, if FCS would be able to detect old FCS data structure, it could migrate it on the fly to the new single file based structure. That is backwards compatible and keeps the old file cache store code around, but transforms its inner details. We'd also need a migration tool for migrating other cache stores (e.g. LevelDB cache store from certain users) so +1 for the migration tool. Indeed, that might be needed, but for the specific case of the new FCS upgrade, you might not even need that, which is even better. If people want to migrate data from cache store X to cache store Y on a whim, then yes, you need a migration tool. Disclaimer: I have not investigated yet the feasibility of Randall's approach.
        Hide
        Mircea Markus
        added a comment -

        Mircea, Randall's suggestion is compatible with what you're saying. The idea is that anyone that would use the FCS with Infinispan 6.0+, if FCS would be able to detect old FCS data structure, it could migrate it on the fly to the new single file based structure.

        fair point. The new SFCS implementation has a particularity though: keeps all the keys in memory. This might have an impact on the system that uses it, so if you go this way it might still make sense to have an explicit param to enable migration so that existing users ack the change.
        Personally I'm still biased towards the migration tool: I don't like the tools that are too sharp and explicit migration is not something people wouldn't expect in this situation.

        Show
        Mircea Markus
        added a comment - Mircea, Randall's suggestion is compatible with what you're saying. The idea is that anyone that would use the FCS with Infinispan 6.0+, if FCS would be able to detect old FCS data structure, it could migrate it on the fly to the new single file based structure. fair point. The new SFCS implementation has a particularity though: keeps all the keys in memory. This might have an impact on the system that uses it, so if you go this way it might still make sense to have an explicit param to enable migration so that existing users ack the change. Personally I'm still biased towards the migration tool: I don't like the tools that are too sharp and explicit migration is not something people wouldn't expect in this situation.
        Hide
        Galder Zamarreño
        added a comment -

        Given that apart from FCS to SFCS migration, there's been a format change in how data is stored in the cache stores (as a result new Cache Store API), a generic way is needed to migrate data from one cache store to another. This could potentially be done within the rolling upgrade infrastructure...

        Show
        Galder Zamarreño
        added a comment - Given that apart from FCS to SFCS migration, there's been a format change in how data is stored in the cache stores (as a result new Cache Store API), a generic way is needed to migrate data from one cache store to another. This could potentially be done within the rolling upgrade infrastructure...
        Hide
        Michal Linhard
        added a comment -

        Hi Galder,

        your commit dbf63e2d4b21aaf8a7a5eda1dca1f88c72fc9070
        concretely: git diff 43f82b7 dbf63e2 core/src/main/java/org/infinispan/util/logging/Log.java (on current master)

        made the infinispan-cachestore-rest build unfunctional
        http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-qe-build-infinispan60-snapshot-auto/149/console

        [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project infinispan-cachestore-rest: Compilation failure: Compilation failure:
        [ERROR] Only one message with the same format parameters is allowed.
        [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore-rest/src/main/java/org/infinispan/persistence/rest/logging/Log.java:[60,19] Only one message with the same format parameters is allowed.
        [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore-rest/src/main/java/org/infinispan/persistence/rest/logging/Log.java:[64,9] Only one message with the same format parameters is allowed.
        [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore-rest/src/main/java/org/infinispan/persistence/rest/logging/Log.java: Only one message with the same format parameters is allowed.
        

        could you please fix this in the rest cachestore as well ? (it blocks building of infinispan-server for perf regression tests)

        Show
        Michal Linhard
        added a comment - Hi Galder, your commit dbf63e2d4b21aaf8a7a5eda1dca1f88c72fc9070 concretely: git diff 43f82b7 dbf63e2 core/src/main/java/org/infinispan/util/logging/Log.java (on current master) made the infinispan-cachestore-rest build unfunctional http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jdg-qe-build-infinispan60-snapshot-auto/149/console [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile ( default -compile) on project infinispan-cachestore- rest : Compilation failure: Compilation failure: [ERROR] Only one message with the same format parameters is allowed. [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore- rest /src/main/java/org/infinispan/persistence/ rest /logging/Log.java:[60,19] Only one message with the same format parameters is allowed. [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore- rest /src/main/java/org/infinispan/persistence/ rest /logging/Log.java:[64,9] Only one message with the same format parameters is allowed. [ERROR] /home_local/workspace/jdg-qe-build-infinispan60-snapshot-auto/infinispan-cachestore- rest /src/main/java/org/infinispan/persistence/ rest /logging/Log.java: Only one message with the same format parameters is allowed. could you please fix this in the rest cachestore as well ? (it blocks building of infinispan-server for perf regression tests)

          People

          • Assignee:
            Galder Zamarreño
            Reporter:
            Galder Zamarreño
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: