When we insert data points for a metric, two writes happen. We perform one write to the data table for each data point. Then we also do one write to the metrics_idx table. metrics_idx is an index table that we use to query for metrics in the system. By updating as we do metrics do not have to be explicitly created up front. This convenience of course comes with a performance cost.
When a client submits an HTTP request (using the REST API) to insert data points the request typically contains data for multiple metrics. The HTTP response is not sent until all data points have been inserted and until the metrics_idx updates are finished. The more metrics there are in a request, the more index updates we have to do. While writes in Cassandra are very fast, there is still overhead.
Clients typically collect and write metric data points on a fixed schedule. Given that, we can scale back on the frequency of index updates. Furthermore, there is no need to perform the index updates as part of the HTTP request. It can be done out of band. This will improve response times since we would only be waiting for data point writes to complete before sending the response.
We could maintain an in-memory buffer of metric ids. When data points are stored the buffer gets updated. We run a background job that periodically flushes the buffer, i.e., updates the metrics_idx table. The frequency of flushes as well as the buffer size should be configurable.