Alerts could be triggered in response to:

• Events (e.g. certain policy violations, critical errors, ...)
• Metric-related event (e.g. response time has dropped below X)

Alerts could be things such as:

• Email
• SMS
• Push notification?
• Simple in-UI list (github style 'inbox') with dismissable elements