MetricsService.createMetric(Metric<?> metric) does several things including,
- Ensure there are no metric naming conflicts
- If a metric with the same name already exists an exception is thrown
- Add metric meta data to indexes which includes,
- retention settings
- Generate and schedule rate calculation job for counter metrics
createMetric involves a number of writes. If any of those writes fail, the onError method of the returned Observable is invoked to let any subscribers know that the operation failed.
The problem is that we are left in an inconsistent state with respect to the metric and cannot get back to a consistent state. Unless it is the initial write to the metrics_idx table that fails, retrying the operation will result in a MetricAlreadyExistsException. There needs to be a way to get back to a consistent state such that all of the aforementioned actions are effectively carried out.