This is a follow up to
HWKMETRICS-612. Having APIs to delete metrics is necessary but not sufficient because we cannot guarantee that clients will use the APIs. The goal here is to prevent situations like HWKMETRICS-606. We can use a simple heuristic to decide if a metric is no longer active. For example, if it has been at least N days since we received any data points for a metrics and if the metric has no data points, then the metric is inactive and it can be deleted.
As per the suggestion from Stefan Negrea we will need a new index table to track which metrics are active. We will update it whenever we store data points for a metric. The schema will look something like:
We do not need to store millisecond timestamps. It should be sufficient to store just the day in which case we could use 4 or even 2 bytes instead of an 8 byte timestamp.
We will have a scheduled job that will check this table to find inactive metrics. When we have determined that a metric can be deleted, we will first have to query metrics_idx to get the set of tags so we know which partitions in metrics_tags_idx to update. Lastly, we need to remove the metric from metrics_activity_idx.