Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-3431

Lost version info during state transfer causes overwrite of newer data that joining node has read from store

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Rejected
    • Affects Version/s: 6.0.0.Alpha3
    • Fix Version/s: 6.0.0.CR1
    • Component/s: State Transfer
    • Labels:
    • Environment:

      Description

      When state transfer sends data to newly joining node no version information is provided despite the fact that this information is available to the cluster (for example, loaded from store). When newly joining node has loaded from its store newer version of data than cluster has then this newer version of data is overwritten because no version information is provided during state transfer.

      Scenario:

      1.	Start node1, node2, node3
      2.	Put {A=>A1} from node1
      3.	Put {B=>B1} from node2
      4.	Put {C=>C1} from node3
      5.	Gracefully shutdown node1 saving the data
      6.	Gracefully shutdown node2 saving the data
      7.	Put {C=>C2} from node3
      8.	Gracefully shutdown node3 saving the data
      9.	Start node1 loading the data
      10.	Start node2 loading the data
      11.	Start node3 loading the data
      

      Loaded from store on node3 - version information shows its newer than nodes 1 and 2 have:

      org.infinispan.interceptors.CallInterceptor
      Executing command: PutKeyValueCommand{key=C, value=C2, flags=[CACHE_MODE_LOCAL, SKIP_LOCKING, SKIP_CACHE_STORE, SKIP_INDEXING, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES], putIfAbsent=false, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=8, version=2}}, successful=true, ignorePreviousValue=false}.
      

      During state transfer older value C1 is received with version information being null:

      org.infinispan.statetransfer.StateTransferInterceptor
      handleNonTxWriteCommand for command PutKeyValueCommand{key=C, value=C1, flags=[CACHE_MODE_LOCAL, SKIP_REMOTE_LOOKUP, PUT_FOR_STATE_TRANSFER, SKIP_SHARED_CACHE_STORE, SKIP_OWNERSHIP_CHECK, IGNORE_RETURN_VALUES, SKIP_XSITE_BACKUP], putIfAbsent=false, metadata=EmbeddedMetadata{version=null}, successful=true, ignorePreviousValue=false}
      

      As a result newer value is replaced with older:

      org.infinispan.interceptors.EntryWrappingInterceptor
      About to commit entry ClusteredRepeatableReadEntry(323f265b){key=C, value=C1, oldValue=C2, isCreated=false, isChanged=true, isRemoved=false, isValid=true, skipRemoteGet=true, metadata=EmbeddedMetadata{version=SimpleClusteredVersion{topologyId=8, version=2}}}
      

      As far as i can see, any key on any node that it didnt written but has received during resync has version=null after loading from store.

      Config:

      <?xml version="1.0" encoding="UTF-8"?>
      
      <!--
      
         This is just a very simplistic example configuration file.  For more information, please see
         http://docs.jboss.org/infinispan/5.3/configdocs/infinispan-config-5.3.html
      
      -->
      
      <infinispan
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="urn:infinispan:config:6.0 http://docs.jboss.org/infinispan/schemas/infinispan-config-6.0.xsd"
          xmlns="urn:infinispan:config:6.0">
      
          <global>
              <globalJmxStatistics enabled="true" jmxDomain="Infinispan" />
              <transport>
                  <properties>
                      <property name="configurationFile" value="jgroups.xml" />
                  </properties>
              </transport>
          </global>
      
          <namedCache name="routing_table">
              <clustering mode="REPL">
                  <stateTransfer awaitInitialTransfer="true" fetchInMemoryState="true"/>
                  <sync replTimeout="1000"/>
                  <!--async useReplQueue="true" replQueueMaxElements="100" /-->
              </clustering>
              <loaders passivation="false" shared="false" preload="true">
                  <singleFileStore 
                      location="target/routing_table${infinispan.store_name:not_specified}/">
                  </singleFileStore>
              </loaders>
              <versioning enabled="true" versioningScheme="SIMPLE"/>
              <transaction
                  transactionMode="TRANSACTIONAL"
                  autoCommit="true"
                  transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
                  lockingMode="OPTIMISTIC"
                  />
              <locking
                  writeSkewCheck="true"
                  isolationLevel="REPEATABLE_READ"/>
          </namedCache>
      
      </infinispan>
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  pruivo Pedro Ruivo
                  Reporter:
                  nikolaymartynov Nikolay Martynov
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: