Details
-
Bug
-
Resolution: Done
-
Major
-
13.0.0.Final
-
None
Description
During the log appender step of SIFS it will guarantee ordering during the write operation. However, there is an extra step before this to facilitate clear pausing the appender. Unfortunately, due to this extra hand off there is a small window where if a clear comes in and the index update is not done before the clear submits to the index you can get into a case where the clear removes the data file but the index update is queued for later.
write thread -> writes to log appender
write thread -> about to update index
clear thread -> pauses log appender
clear thread -> removes data files
clear thread -> submits index sync requests
write thread -> submits to index (which is pending the index clear)
clear thread -> finishes
writes thread -> now updates the index which points to a file that no longer exists.
Code to reproduce
public void testClearDuringWrite() throws ExecutionException, InterruptedException, TimeoutException { WaitDelegatingNonBlockingStore store = TestingUtil.getFirstStoreWait(cache(0, cacheName)); LogAppender logAppender = TestingUtil.extractField(store.delegate(), "logAppender"); CheckPoint checkPoint = new CheckPoint(); checkPoint.triggerForever(Mocks.AFTER_RELEASE); TemporaryTable original = Mocks.blockingFieldMock(checkPoint, TemporaryTable.class, logAppender, LogAppender.class, "temporaryTable", (stubber, temporaryTable) -> stubber.when(temporaryTable).set(anyInt(), any(), anyInt(), anyInt())); // Use fork as the index update in the table is blocked Future<Object> future = fork(() -> cache(0, cacheName).put("some-value", "1")); checkPoint.awaitStrict(Mocks.BEFORE_INVOCATION, 10, TimeUnit.SECONDS); Exceptions.expectException(TimeoutException.class, () -> future.get(10, TimeUnit.MILLISECONDS)); // Put the original table back so our clear request can work TestingUtil.replaceField(original, "temporaryTable", logAppender, LogAppender.class); store.clear().toCompletableFuture().get(10, TimeUnit.SECONDS); checkPoint.triggerForever(Mocks.BEFORE_RELEASE); future.get(10, TimeUnit.SECONDS); // This line will fail with Error reading header from 0:0 | 0 assertEquals(0, cache(0, cacheName).size()); }
