One of the challenges with implementing counters is handling resets which would happen with a server restart for example. Let's consider some sample data to see how we will deal with this.
|Time||Reported Count||Total Count||Rate (each T delta is 1 for simplicity)|
The reported count is the value that gets persisted at each T n. There is a reset at T 3. We know this because the total count is less than the value at T 2. The count reported at T 4 is 15, which is the total since the reset. The total count (since T 0) at T 4 is actually 45. We calculate the total count after the reset at time T n by adding the value to the last total count known before the reset, which in this case is 30.
We need to keep track of the last reset value so that we can still calculate rates efficiently in the event of resets. One possibility is to store it in metrics_idx either as a new column or as a tag. This could work since GenerateRate, the rate calculation job, queries metrics_idx for counter metrics.