The SoftIndexFileStore appends to a log file and waits for that append to complete before returning. The index of that write however is done asynchronously with no back pressure applied. This means it can severely lag behind requests. We should try to reduce memory pressure and cpu usage as much as possible to reduce this. If we cannot lower this, we may need to add some type of back pressure awareness to slow down requests in turn to avoid OOM issues.