This is the issue for Operator folks.

Define and implementation of:

• Grafana dashboards
• Prometheus rules

Datasources:

• System components exposing metrics. Seems that for now, only system-sidekiq component exports metrics.
• kube-state-metrics (https://issues.redhat.com/browse/THREESCALE-4679)
• aggregated health endpoints metrics. (https://issues.redhat.com/browse/THREESCALE-4681)

System guys have the domain knowledge to define useful dashboards and effective prometheus rules. Ops teams can help you with the syntax.
I will provide an environment to easily run the loop: update -> check on dashboards and prometheus rules.

Setup your development environment with latest monitoring tools enabled: https://docs.google.com/document/d/1ogntOEYyi39E2SPp9DiHl-og4p_Ti3PPiEb4tCkZzv8/edit#