Adding infrastructure nodes to an ARO cluster

This document shows how to set up infrastructure nodes in an ARO cluster and move infrastructure related workloads to them. This can help with larger clusters that have resource contention between user workloads and infrastructure workloads such as Prometheus.

Important note: Infrastructure nodes are billed at the same rates as your existing ARO worker nodes.

You can find the original (and more detailed) document describing the process for a self-managed OpenShift Container Platform cluster here

Prerequisites

• Azure Red Hat OpenShift cluster
• Helm CLI

Create Infra Nodes

We’ll use the MOBB Helm Chart for adding ARO machinesets which parameters for creating infra nodes, it looks up an existing machineset to collect cluster specific settings and then creates a new machineset specific for infra nodes with the same settings. The chart used to default to infra nodes up to version 0.2.0 from and including version 0.2.0 you need to specify the roles, labels and taints explicitly.

1. Add the MOBB chart repository to your Helm

2. Update your repositories

3. Install the mobb/aro-machinesets Chart with parameters to create infra nodes

   Create a values.yaml file like this:

4. Wait for the new nodes to be available

Moving Infra workloads

Ingress

You may choose this for any additional Ingress controllers you may have in the cluster, however if you application has very high Ingress resource requirements it may make sense to allow them to spread across the worker nodes, or even a dedicated MachineSet.

1. Set the nodePlacement on the ingresscontroller to node-role.kubernetes.io/infra and increase the replicas to match the number of infra nodes

2. Check the Ingress Controller Operator is starting pods on the new infra nodes

Registry

1. Set the nodePlacement on the registry to node-role.kubernetes.io/infra

2. Check the Registry Operator is starting pods on the new infra nodes

Cluster Monitoring

1. Configure the cluster monitoring stack to use the infra nodes

   Note: This will override any other customizations to the cluster monitoring stack, so you may want to merge your existing customizations into this before running the command.

2. Check the OpenShift Monitoring Operator is starting pods on the new infra nodes

   some Pods like prometheus-operator will remain on master nodes.