HWKMETRICS-367, we reduced the value of gc_grace_seconds for the data table from the default of 10 days to 1 day. For a single node Cassandra cluster, gac_grace_seconds on every table can safely be set to zero. This would be an optimization for single node installations because disk space for deleted data will be reclaimed much sooner. When the deletes are flushed to disk and then when compaction runs, the deleted data should be purged right away.
We can easily determine when the Cassandra cluster expands beyond a single node by registering an event listener with the driver. The listener will get notified whenever nodes are added to or removed from the cluster.
If multiple hawkular metrics instances are running, then we would wind up executing the ALTER TABLE statement multiple times. This should be ok as schema change statement do not incur a lot of overhead.
At start up we can check what the current value of gc_grace_seconds is for each table. We also query the driver to find out the cluster size. Based on the cluster size and the current value, we make any changes accordingly. We then also execute the same logic in our listener code.