ROSA - Federating Metrics to AWS Prometheus

Federating Metrics from ROSA is a bit tricky as the cluster metrics require pulling from its /federated endpoint while the user workload metrics require using the prometheus remoteWrite configuration.

This guide will walk you through using the MOBB Helm Chart to deploy the necessary agents to federate the metrics into AWS Prometheus and then use Grafana to visualize those metrics.

As a bonus it will set up a CloudWatch datasource to view any metrics or logs you have in Cloud Watch.

Prerequisites

• A ROSA HCP cluster
• aws CLI
• jq
• AWS account permissions to create IAM policies/roles, AMP workspaces, and (optionally) CloudWatch read access
• Cluster administrator access (cluster-admin or equivalent) to configure cluster monitoring and user workload monitoring

Set up environment

1. Create environment variables

2. Create namespace

Deploy Operators

1. Add the MOBB chart repository to your Helm

2. Update your repositories

3. Use the mobb/operatorhub chart to deploy the needed operators

4. Wait for the Grafana Operator to install

   You should see the following after a few minutes

Deploy and configure AWS SigV4 proxy and Grafana Alloy

1. Create a Policy for access to AWS CloudWatch

2. Apply the Policy

3. Create a Trust Policy

4. Create Role for AWS Prometheus and CloudWatch

5. Attach the Policies to the Role

   Note: this policy is very permissive, you may want to restrict access for production use cases.

6. Create an AWS Prometheus Workspace

7. Deploy AWS Prometheus Proxy Helm Chart

8. Enable monitoring for user-defined projects if it is not already enabled. The procedure is documented in the Monitoring collection for your OpenShift version; for reference, see Enabling monitoring for user-defined projects

9. Configure remoteWrite for user workloads

Verify Metrics are being collected

1. Access Grafana and check for metrics

2. Browse to the URL provided in the above command and log in with your OpenShift Credentials

3. Enable Admin by hitting sign in and user admin and password

4. Browse to /datasources and verify that cloudwatch and prometheus are present

   If not, you may have hit a race condition that can be fixed by running the following then trying again

5. Browse to /dashboards and select the aws-prometheus-proxy -> NodeExporter / Use Method / Cluster dashboard

   

Cleanup

1. Delete the aws-prometheus-proxy Helm Release

2. Delete the custom-metrics-operators Helm Release

3. Delete the project namespace

4. Detach AWS Role Policies

5. Delete the custom Cloud Watch Policy

6. Delete the AWS Prometheus Role

7. Delete AWS Prometheus Workspace