Configuring Cluster Observability Operator (COO) in ARO and Enabling remote writing of metrics to Azure Monitor Workspace

The Cluster Observability Operator (COO) is an optional OpenShift Container Platform Operator that enables administrators to create standalone monitoring stacks that are independently configurable for use by different services and users.

Deploying COO helps you address monitoring requirements that are hard to achieve using the default monitoring stack. COO is ideal for users who need high customizability, scalability, and long-term data retention, especially in complex, multi-tenant enterprise environments.

This guide will walk users through an example of how to use the COO to set up a highly available Prometheus instance that persists metrics, and enables remote writes of metrics to an Azure Monitor Prometheus

Prerequisites

1. You have access to the cluster as a cluster-admin
2. Follow the OpenShift documentation for Installing the Cluster Observability operator

Set up Azure Monitor Workspace/Grafana

Create Azure Monitor Workspace

Note: In this tutorial, all of the Azure resources will be created in the same resource group as the cluster

1. Follow the Microsoft documentation to Create an Azure Monitor Workspace

Create and Link Azure Managed Grafana

1. Follow the Microsoft documentation to Create an Azure Managed Grafana Workspace
2. Proceed to link the Grafana instance to your Azure Monitor workspace following the Microsoft documentation Link a grafana workspace

Create a Service Principal for use with Azure Monitor Workspace

Give the Service Principal permisssions to publish metrics to your Azure Monitor Workspace

1. Go to the Azure portal. On the resource menu for your Azure Monitor workspace, select Overview. For Data collection rule, select the link.

   

2. On the resource menu for the data collection rule, select Access control (IAM). Select + Add, and then select Add role assignment

   

3. Select the Monitoring Metrics Publisher role, and then select Next

   

4. Select User, group, or service principal, and then choose +select members. Select the Service principal name that you created in the previous step, and then choose Select.

   

5. To complete the role assignment, select Review + assign.

Configuring the Cluster Observability Operator to monitor services

1. Create a project for COO monitoring resources

2. Create a secret for COO prometheus to use to ship metrics to Azure Monitor prometheus

3. Create 2 additonal projects and label them. These will be used to deploy 2 sample applications

Deploy Sample app and ServiceMonitor in namespace test1

1. Deploy a sample application in project test1

2. Verify that the pod is running

3. Create a ServiceMonitor object to specify how the service created above is to be monitored

Deploy Sample app and ServiceMonitor in namespace test2

1. Deploy sample app using CRD below

2. Verify that the pod is running

3. Create a ServiceMonitor object to specify how the service created above is to be monitored

Create a Cluster Observability Operator MonitoringStack

1. Use the CRD below to create the COO MonitoringStack(the equivalent of a Prometheus stack) in the centralized-monitoring project. The MonitoringStack will be used to monitor the applications created above. The configuration creates a single Prometheus replica, retains metrics for 5 days, adds persistent storage to the Prometheus instance, and enables the remote writing of metrics to your Azure Monitor workspace.

   Replace {{INGESTION-URL}} value below with the Metrics ingestion endpoint from the Overview page for the Azure Monitor workspace.

2. Confirm your Prometheus instance is running successfully by running the following command:

   Example output

   Note that 2 alertmanager instances are also created, which you can configure to set up custom alerting. We will not be covering that in this guide.

To disable alertmanager, set the following in the spec: section of the COO MonitoringStack CRD

Validating the Monitoring Stack

Generate Metrics

1. Expose your applications by creating routes

2. Access the routes from your browser or terminal to generate metrics

   Run the command below to get the route url

   Use the HOST/PORT output to access the applications and generate metrics

3. Execute a query on the Prometheus pod to return the total HTTP request metric

   Example output

View Metrics in Azure Monitor

1. Navigate to your Azure Monitor Worskspace and run the following PromQL query in the Prometheus explorer

   Example output

   

View Metrics in Grafana

1. In the Azure portal, navigate to your Azure Grafana workspace. In the Overview tab’s Essentials section, select the Endpoint URL to access your Grafana instance. Single sign-on via Microsoft Entra ID has been configured for you automatically

2. Go to the Explore tab to begin viewing metrics in grafana