ROSA - Federating Metrics to AWS Prometheus

Federating Metrics from ROSA is a bit tricky as the cluster metrics require pulling from its /federated endpoint while the user workload metrics require using the prometheus remoteWrite configuration.

This guide will walk you through using the MOBB Helm Chart to deploy the necessary agents to federate the metrics into AWS Prometheus and then use Grafana to visualize those metrics.

As a bonus it will set up a CloudWatch datasource to view any metrics or logs you have in Cloud Watch.

Make sure to use a region where Amazon Prometheus service is supported

Prerequisites

• A ROSA HCP cluster
• aws CLI
• jq

Set up environment

1. Create environment variables

2. Create namespace

Deploy Operators

1. Add the MOBB chart repository to your Helm

2. Update your repositories

3. Use the mobb/operatorhub chart to deploy the needed operators

4. Wait for the Operator to install

   You should see the following after a few minutes

Deploy and Configure the AWS Sigv4 Proxy and the Grafana Agent

1. Create a Policy for access to AWS CloudWatch

2. Apply the Policy

3. Create a Trust Policy

4. Create Role for AWS Prometheus and CloudWatch

5. Attach the Policies to the Role

   Note: this policy is very permissive, you may want to restrict access for production use cases.

6. Create an AWS Prometheus Workspace

7. Deploy AWS Prometheus Proxy Helm Chart

8. Configure remoteWrite for user workloads

Verify Metrics are being collected

1. Access Grafana and check for metrics

2. Browse to the URL provided in the above command and log in with your OpenShift Credentials

3. Enable Admin by hitting sign in and user admin and password

4. Browse to /datasources and verify that cloudwatch and prometheus are present

   If not, you may have hit a race condition that can be fixed by running the following then trying again

5. Browse to /dashboards and select the aws-prometheus-proxy -> NodeExporter / Use Method / Cluster dashboard

   

Cleanup

1. Delete the aws-prometheus-proxy Helm Release

2. Delete the custom-metrics-operators Helm Release

3. Delete the custom-metrics namespace

4. Detach AWS Role Policies

5. Delete the custom Cloud Watch Policy

6. Delete the AWS Prometheus Role

7. Delete AWS Prometheus Workspace