Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7682

DistributionManager's cache topology updated in wrong order

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      StateTransferInterceptor sets command topology according to StateTransferManager.getCacheTopology() which goes to StateConsumerImpl, but EntryWrappingInterceptor routes according to DistributionManager.getCacheTopology(). StateConsumer's topology is set in onTopologyUpdate before DistributionManager's, and therefore a command can have the current topology set, but EWI does not wrap the entry and some assertions in BaseDistributionInterceptor.remoteGet fails.

      This is a regression brought in ISPN-7400.

        1. NonTxPutIfAbsentDuringLeaveStressTest.testNodeLeavingDuringPutIfAbsent_master_5.log.gz
          156 kB

            [ISPN-7682] DistributionManager's cache topology updated in wrong order

            Dan Berindei (Inactive) added a comment - - edited

            Saw this cause a failure in NonTxPutIfAbsentDuringLeaveStressTest.

            18:59:01,119 WARN  (ForkThread-763,NonTxPutIfAbsentDuringLeaveStressTest:[]) [NonTxPutIfAbsentDuringLeaveStressTest] Exiting fork callable due to exception
org.infinispan.commons.CacheException: java.lang.IllegalStateException
	at org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:144) ~[classes/:?]
	at org.infinispan.interceptors.impl.InvocationContextInterceptor.access$000(InvocationContextInterceptor.java:44) ~[classes/:?]
	at org.infinispan.interceptors.impl.InvocationContextInterceptor$1.apply(InvocationContextInterceptor.java:61) ~[classes/:?]
	at org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21) ~[classes/:?]
	at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:67) ~[classes/:?]
	at org.infinispan.interceptors.InvocationStage.andExceptionally(InvocationStage.java:34) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:132) ~[classes/:?]
	at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:97) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) ~[classes/:?]
	at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[classes/:?]
	at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) ~[classes/:?]
	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) ~[classes/:?]
	at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:528) ~[classes/:?]
	at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:521) ~[classes/:?]
	at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) ~[classes/:?]
	at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:640) ~[classes/:?]
	at org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest$1.doPut(NonTxPutIfAbsentDuringLeaveStressTest.java:81) ~[test-classes/:?]
	at org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest$1.call(NonTxPutIfAbsentDuringLeaveStressTest.java:70) ~[test-classes/:?]
	at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:543) [test-classes/:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.lang.IllegalStateException
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.remoteGet(BaseDistributionInterceptor.java:163) ~[classes/:?]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleMissingEntryOnLocalRead(BaseDistributionInterceptor.java:858) ~[classes/:?]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.onEntryMiss(BaseDistributionInterceptor.java:853) ~[classes/:?]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitGetCommand(BaseDistributionInterceptor.java:848) ~[classes/:?]
	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitGetKeyValueCommand(BaseDistributionInterceptor.java:865) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102) ~[classes/:?]
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:217) ~[classes/:?]
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:205) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) ~[classes/:?]
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataReadCommand(NonTransactionalLockingInterceptor.java:32) ~[classes/:?]
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:110) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) ~[classes/:?]
	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.updateAndInvokeNextRead(BaseStateTransferInterceptor.java:196) ~[classes/:?]
	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:191) ~[classes/:?]
	at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:174) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) ~[classes/:?]
	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:94) ~[classes/:?]
	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83) ~[classes/:?]
	at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?]
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) ~[classes/:?]
	... 18 more

            Dan Berindei (Inactive) added a comment - - edited Saw this cause a failure in NonTxPutIfAbsentDuringLeaveStressTest . 18:59:01,119 WARN (ForkThread-763,NonTxPutIfAbsentDuringLeaveStressTest:[]) [NonTxPutIfAbsentDuringLeaveStressTest] Exiting fork callable due to exception org.infinispan.commons.CacheException: java.lang.IllegalStateException at org.infinispan.interceptors.impl.InvocationContextInterceptor.rethrowException(InvocationContextInterceptor.java:144) ~[classes/:?] at org.infinispan.interceptors.impl.InvocationContextInterceptor.access$000(InvocationContextInterceptor.java:44) ~[classes/:?] at org.infinispan.interceptors.impl.InvocationContextInterceptor$1.apply(InvocationContextInterceptor.java:61) ~[classes/:?] at org.infinispan.interceptors.InvocationExceptionFunction.apply(InvocationExceptionFunction.java:21) ~[classes/:?] at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.addCallback(SimpleAsyncInvocationStage.java:67) ~[classes/:?] at org.infinispan.interceptors.InvocationStage.andExceptionally(InvocationStage.java:34) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:132) ~[classes/:?] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:97) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) ~[classes/:?] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) ~[classes/:?] at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:106) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) ~[classes/:?] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248) ~[classes/:?] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:528) ~[classes/:?] at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:521) ~[classes/:?] at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:348) ~[classes/:?] at org.infinispan.cache.impl.EncoderCache.get(EncoderCache.java:640) ~[classes/:?] at org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest$1.doPut(NonTxPutIfAbsentDuringLeaveStressTest.java:81) ~[test-classes/:?] at org.infinispan.distribution.rehash.NonTxPutIfAbsentDuringLeaveStressTest$1.call(NonTxPutIfAbsentDuringLeaveStressTest.java:70) ~[test-classes/:?] at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:543) [test-classes/:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: java.lang.IllegalStateException at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.remoteGet(BaseDistributionInterceptor.java:163) ~[classes/:?] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleMissingEntryOnLocalRead(BaseDistributionInterceptor.java:858) ~[classes/:?] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.onEntryMiss(BaseDistributionInterceptor.java:853) ~[classes/:?] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitGetCommand(BaseDistributionInterceptor.java:848) ~[classes/:?] at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.visitGetKeyValueCommand(BaseDistributionInterceptor.java:865) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102) ~[classes/:?] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:217) ~[classes/:?] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:205) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) ~[classes/:?] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataReadCommand(NonTransactionalLockingInterceptor.java:32) ~[classes/:?] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:110) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) ~[classes/:?] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.updateAndInvokeNextRead(BaseStateTransferInterceptor.java:196) ~[classes/:?] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:191) ~[classes/:?] at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:174) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) ~[classes/:?] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:94) ~[classes/:?] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:83) ~[classes/:?] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:38) ~[classes/:?] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) ~[classes/:?] ... 18 more

            Dan Berindei (Inactive) added a comment -

            I agree, there's no "proper order" to update all these topology fields. Initially I wanted to use a single field, but I think I hit some problems in the test suite and went back to having a separate field in StateConsumerImpl. I'll see if I can merge the TriangleOrderManager one, too.

            Dan Berindei (Inactive) added a comment - I agree, there's no "proper order" to update all these topology fields. Initially I wanted to use a single field, but I think I hit some problems in the test suite and went back to having a separate field in StateConsumerImpl . I'll see if I can merge the TriangleOrderManager one, too.

            Radim Vansa (Inactive) added a comment -

            It would probably be safer if we had single volatile field to set & read - not three in DM, STM and TriangleOrderManager.

            Radim Vansa (Inactive) added a comment - It would probably be safer if we had single volatile field to set & read - not three in DM, STM and TriangleOrderManager.

              dberinde@redhat.com Dan Berindei (Inactive)
              rvansa1@redhat.com Radim Vansa (Inactive)
              Archiver:
              rhn-support-adongare Amol Dongare

                Created:
                Updated:
                Resolved:
                Archived: