Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-2842

Failure in data insertion into 4 node topology aware cluster with DIST_SYNC mode & enabled InfinispanIndexManager

    XMLWordPrintable

Details

    Description

      We have tests which run queries on TopologyAware distributed cache nodes (2 nodes) with enabled indexing and infinispan directory_provider.

      I decided to see what will happen if we will start 4 such nodes, and found out that in this case when trying during the data put into the cache on the first node, the following exception appears:

      2013-02-21 14:27:55,958 ERROR [InvocationContextInterceptor] (OOB-2,ISPN,NodeD-4638(test3)) ISPN000136: Execution error
      org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager person
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:230)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:102)
      	at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:414)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildIncrementalSearchFactory(SearchFactoryBuilder.java:169)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:149)
      	at org.hibernate.search.impl.MutableSearchFactory.addClasses(MutableSearchFactory.java:194)
      	at org.infinispan.query.backend.QueryInterceptor.enableClassesIncrementally(QueryInterceptor.java:225)
      	at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:250)
      	at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:426)
      	at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:128)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:84)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:472)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
      	at org.jgroups.JChannel.up(JChannel.java:719)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1007)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:183)
      	at org.jgroups.protocols.FC.up(FC.java:479)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
      	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:787)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:411)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:609)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:371)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1267)
      	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1398)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      	at java.lang.Thread.run(Thread.java:722)
      Caused by: org.hibernate.search.SearchException: Unable to open Lucene IndexReader for IndexManager person
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:125)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.initialize(SharingBufferReaderProvider.java:103)
      	at org.hibernate.search.indexes.impl.CommonPropertiesParse.createDirectoryBasedReaderProvider(CommonPropertiesParse.java:164)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createIndexReader(DirectoryBasedIndexManager.java:228)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:105)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
      	... 58 more
      Caused by: java.io.IOException: Read past EOF
      	at org.infinispan.lucene.impl.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:76)
      	at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
      	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:314)
      	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)
      	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
      	at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
      	at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
      	at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
      	at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:143)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:217)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:120)
      	... 63 more
      2013-02-21 14:27:55,962 ERROR [InboundInvocationHandlerImpl] (OOB-2,ISPN,NodeD-4638(test3)) Exception executing command
      org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager person
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:230)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:102)
      	at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:414)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildIncrementalSearchFactory(SearchFactoryBuilder.java:169)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:149)
      	at org.hibernate.search.impl.MutableSearchFactory.addClasses(MutableSearchFactory.java:194)
      	at org.infinispan.query.backend.QueryInterceptor.enableClassesIncrementally(QueryInterceptor.java:225)
      	at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:250)
      	at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:426)
      	at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:128)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:84)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:472)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
      	at org.jgroups.JChannel.up(JChannel.java:719)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1007)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:183)
      	at org.jgroups.protocols.FC.up(FC.java:479)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
      	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:787)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:411)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:609)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:371)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1267)
      	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1398)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      	at java.lang.Thread.run(Thread.java:722)
      Caused by: org.hibernate.search.SearchException: Unable to open Lucene IndexReader for IndexManager person
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:125)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.initialize(SharingBufferReaderProvider.java:103)
      	at org.hibernate.search.indexes.impl.CommonPropertiesParse.createDirectoryBasedReaderProvider(CommonPropertiesParse.java:164)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createIndexReader(DirectoryBasedIndexManager.java:228)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:105)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
      	... 58 more
      Caused by: java.io.IOException: Read past EOF
      	at org.infinispan.lucene.impl.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:76)
      	at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
      	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:314)
      	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)
      	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
      	at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
      	at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
      	at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
      	at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:143)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:217)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:120)
      	... 63 more
      2013-02-21 14:27:55,981 ERROR [InvocationContextInterceptor] (main) ISPN000136: Execution error
      org.infinispan.remoting.RemoteException: ISPN000217: Received exception from NodeD-4638(test3), see cause for remote stack trace
      	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:72)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:310)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:241)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:236)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:231)
      	at org.infinispan.interceptors.distribution.NonTxConcurrentDistributionInterceptor.handleLocalWrite(NonTxConcurrentDistributionInterceptor.java:104)
      	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.handleWriteCommand(NonTxDistributionInterceptor.java:94)
      	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:68)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:275)
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:166)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:127)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:82)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      	at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1162)
      	at org.infinispan.CacheImpl.putInternal(CacheImpl.java:760)
      	at org.infinispan.CacheImpl.put(CacheImpl.java:754)
      	at org.infinispan.CacheImpl.put(CacheImpl.java:748)
      	at org.infinispan.CacheSupport.put(CacheSupport.java:53)
      	at org.infinispan.query.blackbox.TopologyAware4NodeClusteredCacheTest.prepareTestData(TopologyAware4NodeClusteredCacheTest.java:101)
      	at org.infinispan.query.blackbox.TopologyAware4NodeClusteredCacheTest.testSimple(TopologyAware4NodeClusteredCacheTest.java:107)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:601)
      	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
      	at org.testng.internal.Invoker.invokeMethod(Invoker.java:714)
      	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
      	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
      	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
      	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
      	at org.testng.TestRunner.privateRun(TestRunner.java:767)
      	at org.testng.TestRunner.run(TestRunner.java:617)
      	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
      	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
      	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
      	at org.testng.SuiteRunner.run(SuiteRunner.java:240)
      	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
      	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
      	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1198)
      	at org.testng.TestNG.runSuitesLocally(TestNG.java:1123)
      	at org.testng.TestNG.run(TestNG.java:1031)
      	at org.testng.remote.RemoteTestNG.run(RemoteTestNG.java:111)
      	at org.testng.remote.RemoteTestNG.initAndRun(RemoteTestNG.java:204)
      	at org.testng.remote.RemoteTestNG.main(RemoteTestNG.java:175)
      	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:111)
      Caused by: org.hibernate.search.SearchException: HSEARCH000103: Unable to initialize IndexManager person
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:230)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:102)
      	at org.hibernate.search.spi.SearchFactoryBuilder.initDocumentBuilders(SearchFactoryBuilder.java:414)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildIncrementalSearchFactory(SearchFactoryBuilder.java:169)
      	at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:149)
      	at org.hibernate.search.impl.MutableSearchFactory.addClasses(MutableSearchFactory.java:194)
      	at org.infinispan.query.backend.QueryInterceptor.enableClassesIncrementally(QueryInterceptor.java:225)
      	at org.infinispan.query.backend.QueryInterceptor.updateKnownTypesIfNeeded(QueryInterceptor.java:250)
      	at org.infinispan.query.backend.QueryInterceptor.processPutKeyValueCommand(QueryInterceptor.java:426)
      	at org.infinispan.query.backend.QueryInterceptor.visitPutKeyValueCommand(QueryInterceptor.java:128)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:84)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:122)
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:86)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:247)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:220)
      	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:472)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
      	at org.jgroups.JChannel.up(JChannel.java:719)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1007)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:183)
      	at org.jgroups.protocols.FC.up(FC.java:479)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
      	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:787)
      	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:411)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:609)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:371)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1267)
      	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1398)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      	at java.lang.Thread.run(Thread.java:722)
      Caused by: org.hibernate.search.SearchException: Unable to open Lucene IndexReader for IndexManager person
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:125)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.initialize(SharingBufferReaderProvider.java:103)
      	at org.hibernate.search.indexes.impl.CommonPropertiesParse.createDirectoryBasedReaderProvider(CommonPropertiesParse.java:164)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.createIndexReader(DirectoryBasedIndexManager.java:228)
      	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:105)
      	at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:227)
      	... 58 more
      Caused by: java.io.IOException: Read past EOF
      	at org.infinispan.lucene.impl.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:76)
      	at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)
      	at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:314)
      	at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)
      	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)
      	at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)
      	at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
      	at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)
      	at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)
      	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)
      	at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)
      	at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferReaderProvider.java:143)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<init>(SharingBufferReaderProvider.java:217)
      	at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferReaderProvider.java:120)
      	... 63 more
      

      Please note, that the exception appears after the first put. So the code is the following:

            List caches = new ArrayList();
            ConfigurationBuilder builder = AbstractCacheTest.getDefaultClusteredCacheConfig(CacheMode.DIST_SYNC, false);
            builder.indexing().enable().indexLocalOnly(true);
            builder.indexing().addProperty("hibernate.search.default.indexmanager", "org.infinispan.query.indexmanager.InfinispanIndexManager")
                  .addProperty("default.directory_provider", "infinispan")
                  .addProperty("hibernate.search.default.exclusive_index_use", "false")
                  .addProperty("lucene_version", "LUCENE_36");
      
            for(int i = 0; i < 4; i++) {
               GlobalConfigurationBuilder globalConfigurationBuilder = GlobalConfigurationBuilder
                     .defaultClusteredBuilder();
               globalConfigurationBuilder.transport().machineId("a" + i).rackId("b" + i).siteId("test" + i);
      
               EmbeddedCacheManager cm1 = TestCacheManagerFactory.createClusteredCacheManager(
                     globalConfigurationBuilder, builder);
      
               caches.add(cm1.getCache());
            }
      
            for(Object cache : caches) {
               cacheManagers.add(((Cache) cache).getCacheManager());
            }
      
            waitForClusterToForm();
            cache1 = (Cache<String, Person>) caches.get(0);
            cache2 = (Cache<String, Person>) caches.get(1);
            cache3 = (Cache<String, Person>) caches.get(2);
      
            Person person1 = initialize(); //person1 initialization
            Person person2 = initialize(); //person1 initialization
      
            cache1.put(key1, person1);
            cache1.put(key2, person2);  //<----- The failure takes place on this line
      

      I also tried to increase the numOwners from 2 to 4, but it doesn't help. This issue appears when creating the NON_TRANSACTIONAL cache.

      I tried to create the same configuration but with TRANSACTION enabled, in this case the issue ISPN-2815 appears.
      Also, I have tried the same configuration but with REPL_SYNC mode, and the execution passed.

      Another note: in case of 3 nodes -> 2 num owners, the test passes, but starting from 4 nodes - this exception appears.

      I've tried to set the number of nodes to 2 + I've set the value for numOwners to 1: again the same failure appears.

      Attachments

        Activity

          People

            gfernand@redhat.com Gustavo Fernandes (Inactive)
            amanukya@redhat.com Anna Manukyan
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: