Providing JMX metrics via jolokia to Prometheus does not work well with complex types. We are providing several metric sets - engine stats, worker pool stats, requests, etc. as beans which needs to be flattened. Simple parameters are ok to the metrics methods, but simple results are required.
We need to make sure that at least statistics are available on:
- number of plans waiting to be processed - this will be a primary metric for determining horizontal scale out
- amount /%of buffer memory used - this will be a primary metric for determining if pods need to be allocated with more memory/disk
Ideally alerting could be available on percent utilization - but we could also add a metric such as out of disk count to provide a hard count of exceptional conditions.
It's assumed that cpu utilization will be monitored from the pod itself - which will be a primary metric for determining if pods should be allocated with more cpu resources.