In this article, I will demonstrate how to monitor Ansible Automation Platform(AAP) running on OpenShift, using user-workload-monitoring with Prometheus and Grafana.
This article uses the following versions:
- OpenShift v4.13.1
- Grafana Operator v5.4.1
- Ansible Automation Platform v2.4
I won't cover the installation of the Ansible Automation Platform here.
About this article
This article is aimed at users who want a more centralized view of the main usage metrics of the Ansible Automation Platform and identify possible situations of concern. It covers resources such as Grafana, datasources, dashboards, Prometheus, and ServiceMonitors to collect data dynamically.
Prerequisites
- User with the cluster-admin cluster role
- OpenShift 4.12 or +
- Grafana Operator
- User-Defined Projects enabled
Procedure
Use the following steps to begin monitoring Ansible Automation Platform(AAP) using Prometheus and Grafana.
Enable user-defined projects
Execute this command to add `enableUserWorkload: true` under `data/config.yaml`:
$ oc -n openshift-monitoring patch configmap cluster-monitoring-config -p '{"data":{"config.yaml":"enableUserWorkload: true"}}'
`Validate that the prometheus and thanos-ruler pods were created in the openshift-user-workload-monitoring project:
$ oc get pods -n openshift-user-workload-monitoring
NAME READY STATUS RESTARTS AGE
grafana-deployment-6847648746-4mbn9 1/1 Running 0 95m
grafana-operator-controller-manager-7f74d54f44-58pwk 1/1 Running 0 6h55m
prometheus-operator-cf59f9bdc-t7nvm 2/2 Running 0 7h6m
prometheus-user-workload-0 6/6 Running 0 7h6m
prometheus-user-workload-1 6/6 Running 0 7h6m
thanos-ruler-user-workload-0 4/4 Running 0 7h6m
thanos-ruler-user-workload-1 4/4 Running 0 7h6m
Install Grafana Operator
Using the WebConsole, in the left side menu, select OperatorHub, then search for Grafana Operator in the search field.
Make sure to change the project context to openshift-user-workload-monitoring at the top.
Click on the operator and click on Install.
Use the following settings:
- In Update Channel, select v5.
- In Installation Mode, select A specific namespace on the cluster and choose openshift-user-workload-monitoring below.
- In Update approval, select Automatic.
- Click Install.
Now, create a service account and assign permission to read metrics. Use the following commands:
$ oc project openshift-user-workload-monitoring
$ oc create sa grafana-sa
$ oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-sa
Collect the grafana-sa serviceaccount token and create a secret for the Grafana instance:
$ SECRET=`oc -n openshift-user-workload-monitoring describe sa grafana-sa | awk '/Tokens/{ print $2 }'`
$ TOKEN=`oc -n openshift-user-workload-monitoring get secret $SECRET --template='{{ .data.token | base64decode }}'`
$ cat <<EOF > grafana-secret-creds.yaml
kind: Secret
apiVersion: v1
metadata:
name: credentials
namespace: openshift-user-workload-monitoring
stringData:
GF_SECURITY_ADMIN_PASSWORD: grafana <------ Set the password you want to authenticate with Grafana
GF_SECURITY_ADMIN_USER: root <------ Set the desired user to authenticate in Grafana
PROMETHEUS_TOKEN: '${TOKEN}' <------ This variable will receive the token collected above
type: Opaque
EOF
$ oc create -f grafana-secret-creds.yaml
Next, create the Grafana instance. It will read the credentials defined in the previously created secret, as seen below:
$ cat <<EOF > grafana-instance.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
name: grafana
labels:
dashboards: "grafana"
folders: "grafana"
spec:
deployment:
spec:
template:
spec:
containers:
- name: grafana
env:
- name: GF_SECURITY_ADMIN_USER
valueFrom:
secretKeyRef:
key: GF_SECURITY_ADMIN_USER
name: credentials
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: GF_SECURITY_ADMIN_PASSWORD
name: credentials
config:
auth:
disable_login_form: "false"
disable_signout_menu: "true"
auth.anonymous:
enabled: "false"
log:
level: warn
mode: console
EOF
Apply and validate the created Instance:
$ oc -n openshift-user-workload-monitoring create -f grafana-instance.yaml
$ oc -n openshift-user-workload-monitoring get pods -l app=grafana
NAME READY STATUS RESTARTS AGE
grafana-deployment-c4959687c-7vg9d 1/1 Running 0 6m24s
Expose the grafana service using an edge-type route. Use the service called grafana-service. Here's an example:
$ oc -n openshift-user-workload-monitoring get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana-operator-operator-metrics-service ClusterIP 172.30.37.111 <none> 8443/TCP 7h1m
grafana-service ClusterIP 172.30.244.194 <none> 3000/TCP 6h16m
prometheus-operated ClusterIP None <none> 9090/TCP,10901/TCP 7h12m
prometheus-operator ClusterIP None <none> 8443/TCP 7h12m
prometheus-user-workload ClusterIP 172.30.159.129 <none> 9091/TCP,9092/TCP,10902/TCP 7h12m
prometheus-user-workload-thanos-sidecar ClusterIP None <none> 10902/TCP 7h12m
thanos-ruler ClusterIP 172.30.213.131 <none> 9091/TCP,9092/TCP,10901/TCP 7h12m
thanos-ruler-operated ClusterIP None <none> 10902/TCP,10901/TCP 7h12m
$ oc -n openshift-user-workload-monitoring create route edge grafana --service=grafana-service --insecure-policy=Redirect
Display the route exposed to Grafana:
$ oc -n openshift-user-workload-monitoring get route grafana -o jsonpath='{.spec.host}'
It's time to create the Grafana Datasource, which will connect to thanos-querier in the openshift-monitoring project and will use the grafana-sa serviceaccount token that is stored in secret credentials.
$ cat <<EOF > grafana-datasource.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
name: grafana-ds
namespace: openshift-user-workload-monitoring
spec:
valuesFrom:
- targetPath: "secureJsonData.httpHeaderValue1"
valueFrom:
secretKeyRef:
name: "credentials"
key: "PROMETHEUS_TOKEN"
instanceSelector:
matchLabels:
dashboards: "grafana"
datasource:
name: Prometheus
type: prometheus
access: proxy
url: https://thanos-querier.openshift-monitoring.svc:9091
isDefault: true
jsonData:
"tlsSkipVerify": true
"timeInterval": "5s"
httpHeaderName1: 'Authorization'
secureJsonData:
"httpHeaderValue1": "Bearer ${PROMETHEUS_TOKEN}"
editable: true
EOF
Apply and validate the created Datasource:
$ oc -n openshift-user-workload-monitoring create -f grafana-datasource.yaml
$ oc -n openshift-user-workload-monitoring get GrafanaDatasource
NAME NO MATCHING INSTANCES LAST RESYNC AGE
grafana-ds 119s 3d23h
To validate the created datasource using Grafana Console, use the edge route created previously and access it via a browser. Authenticate using the username and password added in secret credentials.
Once authenticated, click Configuration > Data sources.
Creating User in Ansible Automation Platform
Access the AAP console and create a user for monitoring. Click on Users > Add in the left side menu to do this.
To generate the token, authenticate to AAP using the created user and then click on Users > select the name of the created user > Token > Add.
Define a description and scope as read and click Save. A popup will be displayed with the token; copy and save it.
Creating Prometheus ServiceMonitor
Create a ServiceMonitor to collect metrics from AAP and export them through the Prometheus and Thanos Querier.
First, create a secret to store the bearer token previously collected in AAP with the user aap-metrics. Here is the command:
$ oc create secret generic aap-monitor-creds --from-literal=token={{ YOUR AAP BEARER TOKEN }} -n aap
Next, create ServiceMonitor, which will discover the AAP service and collect the metrics in the path /api/v2/metrics.
$ cat <<EOF > svc-monitor-aap.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: aap-monitor
namespace: aap
spec:
endpoints:
- interval: 30s
scrapeTimeout: 10s
honor_labels: true
path: /api/v2/metrics/
port: http
scheme: http
bearerTokenSecret:
key: token
name: aap-monitor-creds <------ Secret previously created with our Bearer Token
namespaceSelector:
matchNames:
- aap
selector:
matchLabels:
app.kubernetes.io/component: automationcontroller
EOF
Finally, apply and validate the created ServiceMonitor using the following two commands:
$ oc create -f svc-monitor-aap.yaml
$ oc get servicemonitor -n aap
NAME AGE
aap-monitor 31m
To validate using the WebConsole, in the left side menu, click on Targets in the Observe Session, and in Filter, select User.
While still in the Observe section, click on Metrics. You will identify whether the AAP metrics are arriving correctly. Use any metric starting with awx_, such as awx_instance_info.
Creating Grafana Dashboard
Create a Grafana dashboard, which will fetch the JSON externally from GitHub.
$ cat <<EOF > grafana-dashboard-aap.yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
name: grafana-dashboard-aap
labels:
app: grafana
spec:
instanceSelector:
matchLabels:
dashboards: grafana
folder: "AAP"
url: https://raw.githubusercontent.com/leoaaraujo/aap-dashboard/main/aap-dash.json
EOF
Next, apply and validate the created Grafana dashboard:
$ oc -n openshift-user-workload-monitoring create -f grafana-dashboard-aap.yaml
$ oc -n openshift-user-workload-monitoring get grafanadashboard
NAME NO MATCHING INSTANCES LAST RESYNC AGE
grafana-dashboard-aap 3s 145m
Viewing the Dashboard
Access Grafana, and in the left side menu, click on Dashboards and then on Browse.
It will display a folder named AAP and the dashboard AAP - Metrics. Click on the dashboard.
The dashboard looks like this:
Wrap up
I demonstrated creating monitoring for the Ansible Automation Platform using User-Defined Projects from the OpenShift Monitoring stack. I used a Grafana Dashboard to visualize usage metrics and statistics, such as subscription information, playbook metrics, users, and resource consumption within OpenShift.
References
For more details and other configurations, start with the reference documents below:
About the author
More like this
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit