Cloud Experts Documentation

Custom Alerts in ROSA 4.11.x

This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.

Starting with OpenShift 4.11 it is possible to manage alerting rules for user-defined projects . Similarly, in ROSA clusters the OpenShift Administrator can enable a second AlertManager instance in the user workload monitoring namespace which can be used to create such alerts.

Note: Currently this is not a managed feature of ROSA. Such an implementation may get overwritten if the User Workload Monitoring functionality is toggled off and on using the OpenShift Cluster Manager (OCM). We


Create Environment Variables

Configure User Workload Monitoring to include AlertManager

  1. Edit the user workload config to include AlertManager

    Note: If you have other modifications to this config, you will need to hand edit the resource rather than brute forcing it like below.

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: ConfigMap
      name: user-workload-monitoring-config
      namespace: openshift-user-workload-monitoring
      config.yaml: |
          enabled: true
          enableAlertmanagerConfig: true
  2. Verify that a new Alert Manager instance is defined

    oc -n openshift-user-workload-monitoring get alertmanager
    NAME            VERSION   REPLICAS   AGE
    user-workload   0.24.0    2          2m
  3. If you want non-admin users to be able to define alerts in their own namespaces you can run the following.

    oc -n <namespace> adm policy add-role-to-user alert-routing-edit <user>
  4. Create a Slack webhook integration in your Slack workspace. Create environment variables for it.

  5. Update the Alert Manager Configuration file

    This will create a basic AlertManager configuration to send alerts to your slack channel. If you have an existing configuration you’ll need to edit it, not brute force paste like shown.

    cat << EOF | oc apply -f -
    apiVersion: v1
    kind: Secret
      name: alertmanager-user-workload
      namespace: openshift-user-workload-monitoring
      alertmanager.yaml: |
          slack_api_url: "${SLACK_API_URL}"
          receiver: Default
          group_by: [alertname]
          - name: Default
              - channel: ${SLACK_CHANNEL}
                send_resolved: true

Create an Example Alert

  1. Create a Namespace for your custom alert

    oc create namespace custom-alert
  2. Verify it works by creating a Prometheus Rule that will fire off an alert

    cat << EOF | oc apply -f -
    kind: PrometheusRule
      name: prometheus-example-rules
      namespace: custom-alert
      - name: example.rules
        - alert: ExampleAlert
          expr: vector(1)
  3. Log into your OpenShift Console and switch to the Developer view. Select the Observe item from the menu on the left and then click Alerts. Make your your Project is set to custom-alert

    Screenshot of Alert in OCP Console
  4. Check the Alert was sent to Slack

    Screenshot of Alert in Slack

What about Cluster Alerts ?

By default, cluster alerts are only sent to Red Hat SRE and you cannot modify the cluster alert receivers. However you can make a copy of any of the alerts that you wish to see in your own alerting namespace which will then get picked up by the user workload alert-manager.

  1. Duplicate a subset of the Cluster Monitoring Prometheus Rules

    oc -n openshift-monitoring get prometheusrule \
      cluster-monitoring-operator-prometheus-rules -o json | \
      jq '.metadata.namespace = "custom-alert"' | \
      kubectl create -f -
  2. Check the Alerts in the custom-alert Project in the OpenShift Console, and you’ll now see the Watchdog and other cluster alerts.

    Screenshot of cluster alerts in custom project
  3. You should also have received an alert for the Watchdog alert in your Slack channel.

    Screenshot of watchdog alert in slack

Interested in contributing to these docs?

Collaboration drives progress. Help improve our documentation The Red Hat Way.

Red Hat logo LinkedIn YouTube Facebook Twitter



Try, buy & sell


About Red Hat

We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Subscribe to our newsletter, Red Hat Shares

Sign up now
© 2023 Red Hat, Inc.