Custom Alerts in ROSA 4.11.x

Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects . Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts.

Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled off and on using the OpenShift Cluster Manager (OCM).

Prerequisites

• AWS CLI
• A Red Hat OpenShift for AWS (ROSA) cluster 4.11.0 or higher

Create Environment Variables

Configure User Workload Monitoring to include AlertManager

1. Edit the user workload config to include AlertManager

   Note: If you have other modifications to this config, you will need to hand edit the resource rather than brute forcing it like below.

2. Verify that a new Alert Manager instance is defined

3. If you want non-admin users to be able to define alerts in their own namespaces you can run the following.

4. Create a Slack webhook integration in your Slack workspace. Create environment variables for it.

5. Update the Alert Manager Configuration file

   This will create a basic AlertManager configuration to send alerts to your slack channel. If you have an existing configuration you’ll need to edit it, not brute force paste like shown.

Create an Example Alert

1. Create a Namespace for your custom alert

2. Verify it works by creating a Prometheus Rule that will fire off an alert

3. Log into your OpenShift Console and switch to the Developer view. Select the Observe item from the menu on the left and then click Alerts. Make your your Project is set to custom-alert

   

4. Check the Alert was sent to Slack

   

What about Cluster Alerts ?

By default, cluster alerts are only sent to Red Hat SRE and you cannot modify the cluster alert receivers. However you can make a copy of any of the alerts that you wish to see in your own alerting namespace which will then get picked up by the user workload alert-manager.

1. Allow cross-project alerting for your project (i.e custom-alert) by adding namespacesWithoutLabelEnforcement to the user-workload-monitoring-config ConfigMap above.

2. Download prometheus rules from the openshift-monitoring namespace into a file

3. Search and replaced “namespace”: “openshift-monitoring”, with your namespace i.e “namespace”: “custom-alert”

4. Add prometheus rules to your project

5. Check the Alerts in the custom-alert Project in the OpenShift Console, and you’ll now see the Watchdog and other cluster alerts.

   

6. You should also have received an alert for the Watchdog alert in your Slack channel.