Optimizing Costs with ROSA - Scheduled Cluster Scaling

One of the key benefits of Red Hat OpenShift Service on AWS (ROSA) is its ability to scale efficiently, ensuring you only pay for the resources you actually need. While ROSA includes autoscaling features that adjust cluster size based on demand, you can further optimize costs by scheduling scaling during off-peak hours when the cluster isn’t heavily used. This helps reduce expenses without impacting performance.

In this guide, we’ll show you how to schedule scaling in ROSA, allowing your cluster to automatically adjust its size based on a predefined schedule. You’ll learn how to schedule scale-downs during periods of low activity and scale-ups when additional resources are required, ensuring both cost efficiency and optimal performance.

ROSA clusters consist of multiple machine pools, each containing worker nodes. For example, each availability zone has its own machine pool, allowing for granular control over the cluster’s worker node configuration. Since each machine pool can scale independently, you can fine-tune resource allocation based on workload demands.

This guide will walk you through scaling a single machine pool, a process that can be repeated for additional machine pools you wish to schedule scaling for.

Prerequisites

The following CLIs need to be installed and logged in:

• oc CLI
• rosa CLI
• aws CLI
• podman CLI
• jq

Prepare the Environment

This guide utilizes several environment variables that will be referenced throughout.

Set cluster and AWS environment variables

Retrieve the list of machine pool names:

Example output:

For this guide, we will use the first machine pool workers-0.

Set the machine pool you want to scale and the minimum and maximum number of replicas as environment variables:

ROSA Service Account

Using Red Hat’s Hybrid Cloud Console, you can create service accounts for automation tasks such as DevOps processes or, in this case, scheduling cluster scaling. One of the key benefits of service accounts in ROSA is the ability to restrict access to only what’s necessary.

For scaling machine pools up and down, the service account we create will be granted access solely to machine pools, ensuring a least-privilege approach while keeping the cluster secure.

To get a service account, start by logging into Red Hat’s Hybrid Cloud Console .

Click on the gear button and then click Create Service Account.

Give your service account a name and a short description.

On the next page, you will see the service account credentials. Make sure to copy these down somewhere safe, this is the only time you will be able to see these credentials.

Next, we will create a group that will have permissions to adjust machine pool size. On the left-hand navigation, click on Groups and then Create group.

Next, enter a name and description for the group.

Next, filter and search for “machine”, then select the OCM Machine Pool Editor role. This will ensure that the service account, which we will add in the next step, has permissions limited to interacting only with machine pools.

Click Next to skip the add members dialog and advance to the add service accounts dialog. On the Add service accounts dialog, select the service account you created in the previous step.

Finally, on the last step, click Submit and this will create the group for you.

Export the client ID and client secret variables:

Create AWS Policy

One of the standout features of ROSA is its integration with AWS STS, enabling fine-grained access control to AWS resources. By leveraging IAM Roles for Service Accounts (IRSA), we can extend this access control to applications running within the cluster, allowing them to securely interact with AWS resources while following least-privilege principles.

1. Create an AWS IAM trust policy:

2. Create a role for the scheduler:

3. Attach the policy to the role:

Create OpenShift CronJob to Schedule Scaling

To schedule scaling of the worker nodes, we will create 2 CronJobs which will control when the cluster will be scheduled to be scaled up and scaled down. The CronJob will leverage an image that contains the ROSA CLI to adjust the machine pool sizes.

1. Create a new OpenShift project:

2. Create a service account that the CronJob will run under (note that it uses IRSA and the role we just created):

3. Create a secret that contains ROSA credentials for the service account you created previously:

4. Create a ConfigMap for the scaling up settings:

5. Create a ConfigMap for the scaling down settings:

6. Build and push a container image:

   This guide will use the internal image registry in the cluster, but any image repository will work.

   Patch the registry so we can push images to it:

   Retrieve the image registry URL:

   Build a container image with the ROSA CLI:

7. Create a CronJob to scale up the cluster on a schedule:

   For testing purposes, the CronJob is scheduled to run every 5 minutes. Adjust the schedule accordingly for your production use case.

8. Create a CronJob to scale down the cluster on a schedule:

   For testing purposes, the CronJob is scheduled to run every 5 minutes. Adjust the schedule accordingly for your production use case. If a scaling job fails and the pod logs report that the “specified username does not exist,” the cluster is likely still reconciling permissions. This sync issue typically resolves itself automatically within an hour, so allow some time for the system to update before retrying or modifying your configuration.

9. Finally, sit back and watch the machine pools scale on the schedule you configured. To watch machine pools scaling up and down, run this command: