Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-10272

RocskDbStore BatchWrite org.rocksdb.RocksDBException: unknown WriteBatch tag

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 10.0.0.Beta4, 9.4.15.Final
    • 9.4.14.Final
    • None
    • None
    • DataGrid Sprint #29
    • Hide

      This can be only reproduced with the application code (unable to produce with synthetic test) with multiple caches (C1,C2,C3) configured with rocksdb store persistence like below with explicit transactions and the loops runs every seconds with 100s of records

      Thread 1

      loop
      tx begin
      create new records in C 1
      tx commit

      Thread 2

      loop
      tx begin
      query records in C 1
      create new records in C 2
      update queried records C 1
      tx commit

      Thread3

      loop
      tx begin
      query records in C2
      create new rcords in C3
      update records in C1
      tx commit

      Show
      This can be only reproduced with the application code (unable to produce with synthetic test) with multiple caches (C1,C2,C3) configured with rocksdb store persistence like below with explicit transactions and the loops runs every seconds with 100s of records Thread 1 loop tx begin create new records in C 1 tx commit Thread 2 loop tx begin query records in C 1 create new records in C 2 update queried records C 1 tx commit Thread3 loop tx begin query records in C2 create new rcords in C3 update records in C1 tx commit

    Description

      Issue

      RocksDBStore is not closing WriteBatch native handles and under heavy write with multiple databases, following error is continuously throwing and database never recovers (following error is from infinispan 9.2.5-Final but i believe same issue exists in master , see https://github.com/infinispan/infinispan/blob/1bcfe5823e7957060080741f7d06f343c3447699/persistence/rocksdb/src/main/java/org/infinispan/persistence/rocksdb/RocksDBStore.java#L756)

      Caused by: org.rocksdb.RocksDBException: unknown WriteBatch tag
at org.rocksdb.RocksDB.write0(Native Method) 
at org.rocksdb.RocksDB.write(RocksDB.java:602)
at org.infinispan.persistence.rocksdb.RocksDBStore.writeBatch(RocksDBStore.java:393)
at org.infinispan.persistence.rocksdb.RocksDBStore.writeBatch(RocksDBStore.java:374)
at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$writeBatchToAllNonTxStores$12

      I was also getting other errors from this list, for example "bad WriteBatch Delete"https://github.com/facebook/rocksdb/blob/cae22c53fbad071be8aa3a8543415383b4dfaef4/db/write_batch.cc#L302

      Fix
      private void writeBatch(WriteBatch batch) throws InterruptedException, RocksDBException {
    semaphore.acquire();
    try {
        if (stopped)
            throw new PersistenceException("RocksDB is stopped");

        db.write(dataWriteOptions(), batch);
    } finally {
        semaphore.release();
        batch.close();
    }
}

      Attachments

        Activity

          People

            wburns@redhat.com Will Burns
            jestan Jestan Nirojan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: