[ISPN-7779] State transfer does not work with protobuf encoded entities

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 8.2.7.Final, 9.1.0.Final, 9.0.3.Final
Affects Version/s: 9.0.0.Final
Component/s: Remote Querying
Labels:
None

Git Pull Request:
https://github.com/infinispan/infinispan/pull/5172, https://github.com/infinispan/infinispan/pull/5173, https://github.com/infinispan/infinispan/pull/5176
Bugzilla References:
https://bugzilla.redhat.com/show_bug.cgi?id=1458500, https://bugzilla.redhat.com/show_bug.cgi?id=1459157

I am currently testing remote deployment of JBoss Data Grid 7.1 with a 3-node replicated cluster.

Items stored in the replicated cache are protobuf encoded and indexed. I have enabled global-state persistence on each of the cluster nodes to ensure that the protobuf schema cache is not purged on startup. However, when state-transfer occurs to a node joining the cluster, the transfer triggers an array of exceptions as shown below.

If i then kill the nodes transferring state to this erroring new joiner, the new joiner node continues to start-up and only at that point is this log message show

15:54:19,991 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4)                                    
DGISPN0001: Started ___protobuf_metadata cache from clustered container

It seems as through the protobuf metadata cache starts after the configured caches which means that during their startup they do not have access to the persisted schema

State Transfer Exception

2017-04-28 15:03:08,036 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (stateTransferExecutor-thread--p7-t14) ISPN000136: Error executing command PutKeyValueCommand, writing keys [[B0x4A0E4C45494C4757..[16]]: org.hibernate.search.bridge.BridgeException: Exception while calling bridge#set
	entity class: org.infinispan.query.remote.impl.indexing.ProtobufValueWrapper
	field bridge: org.infinispan.query.remote.impl.indexing.ProtobufValueWrapperFieldBridge@44f36069
	at org.hibernate.search.bridge.util.impl.ContextualExceptionBridgeHelper.buildBridgeException(ContextualExceptionBridgeHelper.java:103)
	at org.hibernate.search.bridge.util.impl.ContextualExceptionBridgeHelper$OneWayConversionContextImpl.set(ContextualExceptionBridgeHelper.java:137)
	at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.buildDocumentFieldForClassBridges(DocumentBuilderIndexedEntity.java:890)
	at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.buildDocumentFields(DocumentBuilderIndexedEntity.java:454)
	at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.getDocument(DocumentBuilderIndexedEntity.java:391)
	at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.createUpdateWork(DocumentBuilderIndexedEntity.java:301)
	at org.hibernate.search.engine.spi.DocumentBuilderIndexedEntity.addWorkToQueue(DocumentBuilderIndexedEntity.java:243)
	at org.hibernate.search.engine.impl.WorkPlan$PerEntityWork.enqueueLuceneWork(WorkPlan.java:520)
	at org.hibernate.search.engine.impl.WorkPlan$PerClassWork.enqueueLuceneWork(WorkPlan.java:282)
	at org.hibernate.search.engine.impl.WorkPlan.getPlannedLuceneWork(WorkPlan.java:154)
	at org.hibernate.search.backend.impl.WorkQueue.prepareWorkPlan(WorkQueue.java:114)
	at org.hibernate.search.backend.impl.BatchedQueueingProcessor.prepareWorks(BatchedQueueingProcessor.java:57)
	at org.hibernate.search.backend.impl.PerTransactionWorker.performWork(PerTransactionWorker.java:86)
	at org.infinispan.query.backend.QueryInterceptor.performSearchWorks(QueryInterceptor.java:241)
	at org.infinispan.query.backend.QueryInterceptor.performSearchWork(QueryInterceptor.java:235)
	at org.infinispan.query.backend.QueryInterceptor.updateIndexes(QueryInterceptor.java:228)
	at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:457)
	at org.infinispan.query.backend.QueryInterceptor.lambda$visitPutKeyValueCommand$0(QueryInterceptor.java:154)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
	at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:154)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:397)
	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:340)
	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:123)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:164)
	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:153)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
	at org.infinispan.interceptors.impl.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:49)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:77)
	at org.infinispan.interceptors.compat.BaseTypeConverterInterceptor.visitPutKeyValueCommand(BaseTypeConverterInterceptor.java:82)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
	at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
	at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
	at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
	at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:282)
	at org.infinispan.statetransfer.StateConsumerImpl.doApplyState(StateConsumerImpl.java:621)
	at org.infinispan.statetransfer.StateConsumerImpl.applyChunk(StateConsumerImpl.java:589)
	at org.infinispan.statetransfer.StateConsumerImpl.access$000(StateConsumerImpl.java:93)
	at org.infinispan.statetransfer.StateConsumerImpl$1.run(StateConsumerImpl.java:552)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Message descriptor not found : "******** I HAVE MASKED THIS VALUE *********"
	at org.infinispan.protostream.impl.SerializationContextImpl.getMessageDescriptor(SerializationContextImpl.java:166)
	at org.infinispan.query.remote.impl.indexing.WrappedMessageTagHandler.onTag(WrappedMessageTagHandler.java:65)
	at org.infinispan.protostream.ProtobufParser.parseMessage(ProtobufParser.java:75)
	at org.infinispan.protostream.ProtobufParser.parseInternal(ProtobufParser.java:57)
	at org.infinispan.protostream.ProtobufParser.parse(ProtobufParser.java:44)
	at org.infinispan.query.remote.impl.indexing.ProtobufValueWrapperFieldBridge.decodeAndIndex(ProtobufValueWrapperFieldBridge.java:58)
	at org.infinispan.query.remote.impl.indexing.ProtobufValueWrapperFieldBridge.set(ProtobufValueWrapperFieldBridge.java:46)
	at org.hibernate.search.bridge.util.impl.ContextualExceptionBridgeHelper$OneWayConversionContextImpl.set(ContextualExceptionBridgeHelper.java:134)
	... 56 more

is cloned by

JDG-1014 State transfer does not work with protobuf encoded entities

Closed

is related to

ISPN-7946 Create test which checks that protobuf metadata cache is started once any ordinary cache is started

Closed

Adrian Nistor (Inactive) added a comment - 2017/06/08 6:11 AM

Thanks gfernand@redhat.com for merging this in 9.0.x and master. I've also prepared a PR for 8.2.x. Could you please handle it and mark this issue as resolved? Thanks!

Adrian Nistor (Inactive) added a comment - 2017/06/08 6:11 AM Thanks gfernand@redhat.com for merging this in 9.0.x and master. I've also prepared a PR for 8.2.x. Could you please handle it and mark this issue as resolved? Thanks!

RH Bugzilla Integration added a comment - 2017/06/04 1:27 AM

Adrian Nistor <anistor@redhat.com> changed the Status of bug 1458500 from NEW to POST

RH Bugzilla Integration added a comment - 2017/06/04 1:27 AM Adrian Nistor <anistor@redhat.com> changed the Status of bug 1458500 from NEW to POST

Adrian Nistor (Inactive) added a comment - 2017/06/01 6:25 AM

Currently there is no way to declare/configure the start order of caches. But it could be enforced in code by calling EmbeddedCacheManager.getCache("___protobuf_metadata") at the right time. I'm trying to see if I can include this in org.infinispan.query.remote.impl.LifecycleManager. I vaguely remember this was done before but I don't see that code there anymore.

Adrian Nistor (Inactive) added a comment - 2017/06/01 6:25 AM Currently there is no way to declare/configure the start order of caches. But it could be enforced in code by calling EmbeddedCacheManager.getCache("___protobuf_metadata") at the right time. I'm trying to see if I can include this in org.infinispan.query.remote.impl.LifecycleManager. I vaguely remember this was done before but I don't see that code there anymore.

AHMAD EL ABIARY (Inactive) added a comment - 2017/06/01 4:34 AM

Any update on this, this is potentially a blocking issue for us?

AHMAD EL ABIARY (Inactive) added a comment - 2017/06/01 4:34 AM Any update on this, this is potentially a blocking issue for us?

AHMAD EL ABIARY (Inactive) added a comment - 2017/04/28 7:10 PM

This seems to be an issue with the order in which the caches are started on the node state is being transferred to. When i change the cache name to ___price_cache, this issue is resolved, this is because the __protobuf_metadata cache starts before it. is there anyway to enforce the order in which caches are started?

AHMAD EL ABIARY (Inactive) added a comment - 2017/04/28 7:10 PM This seems to be an issue with the order in which the caches are started on the node state is being transferred to. When i change the cache name to ___price_cache, this issue is resolved, this is because the __protobuf_metadata cache starts before it. is there anyway to enforce the order in which caches are started?

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

Collapse comment: Adrian Nistor (Inactive) added a comment - 2017/06/08 6:11 AM

Expand comment: Adrian Nistor (Inactive) added a comment - 2017/06/08 6:11 AM

Collapse comment: RH Bugzilla Integration added a comment - 2017/06/04 1:27 AM

Expand comment: RH Bugzilla Integration added a comment - 2017/06/04 1:27 AM

Collapse comment: Adrian Nistor (Inactive) added a comment - 2017/06/01 6:25 AM

Expand comment: Adrian Nistor (Inactive) added a comment - 2017/06/01 6:25 AM

Collapse comment: AHMAD EL ABIARY (Inactive) added a comment - 2017/06/01 4:34 AM

Expand comment: AHMAD EL ABIARY (Inactive) added a comment - 2017/06/01 4:34 AM

Collapse comment: AHMAD EL ABIARY (Inactive) added a comment - 2017/04/28 7:10 PM

Expand comment: AHMAD EL ABIARY (Inactive) added a comment - 2017/04/28 7:10 PM

People

Dates

PagerDuty