Scaling to multicluster from the command line with ClusterPools

Introduction

In the last article Bring your own fleet with Red Hat Advanced Cluster Management a for Kubernetes auto-import and automation tools, Gurney Buchanan and I discussed on how to auto-import a cluster into the Red Hat Advanced Cluster Management.

Today I'm writing about the Hive ClusterPool and how the cm-cli can help you to manage clusters in your Red Hat Advanced Cluster Management or when using the multicluster engine for Kubernetes operator.

Red Hat Advanced Cluster Management and the multicluster engine include the Hive project, which includes a ClusterPool feature. The ClusterPool feature allows you to pre-propovision a number of clusters and request them from the pool on demand. This drastically reduces the waiting time for a developer or a CICD process to get a fully operational cluster. While the clusters are in the queue, they are in hibernation mode, which reduces the cost. The end-user can request a cluster from the pool by creating a cluster claim toward a certain pool.

The cluster manager command line interface (cm-cli) provides a number of commands allowing the developer to easily manage clusters from different pools and different Hive installation clusters. The cm-cli is not a mandatary command line interface to manage ClusterPool, as everything can be achieved by using oc command line interface and creating a different custom resource, but the cm-cli simplifies developer day-to-day work.

In our organization, we have a shared cluster with Red Hat Advanced Cluster Management and a Hive installation. Through RBAC we secure this cluster in order to give each team their own namespace and ClusterPools. A team can see and access all clusters they created, but can not see the clusters that are created by other teams. Gurney Buchanan will explain in another article how to setup the RBAC to achieve such security.

cm-cli installation

The cm-cli can be used as a stand-alone command line or as an oc plugin. You can install it from Krew, or by downloading the binary from the cm-cli repository, or by simply cloning the repo and run make build (golang 1.17 minimum). Check here for more details.

Prerequisites

If you want to execute the following commands as an exercise, you will need to have an AWS account, as commands are creating resources on AWS in order to provision a cluster.

You will also need a cluster with Red Hat Advanced Cluster Management or the multicluster engine for Kubernetes operator deployed. You can create such cluster by using the cm create cluster --values <yaml_file>. Run cm create cluster -h for more information.

ClusterPoolHost concept

A ClusterPool host (cph) is cm-cli configuration and that is saved on a local computer. It defines the API URL of the cluster where Hive is installed, the namespace where the cluster claims are created, and a user group. Each user is part of one or more user group. It also keeps track on the current ClusterPool host that you are working on.

cm-cli relies on the ClusterPool host to take actions on the ClusterPools and cluster claims.

1. To create a ClusterPool host, you have to run the following command:

cm create cph my-cph --api-server <cluster_api_url> --console <cluster_console_url> --namespace <cluster_claim_namespace> --group <User_group>

For example, while logged in on your Red Hat Advanced Cluster Management server you can run:

cm create cph my-cph  --api-server  https://api.my-acm.my-domain:6443  --console https://console-openshift-console.apps.my-acm.my-domain --namespace default --group system:cluster-admins

This will generate on the local computer a configuration giving access to other cm-cli commands in the context of this ClusterPool host. You can have one or multiple ClusterPool host configurations allowing the end-user to jump from one environment to another to get cluster provisioned.

If you have multiple ClusterPool hosts because you want to managed multiple Hive clusters, or on the same Hive cluster you want to access different namespaces, you can switch between the ClusterPool host by running the command:

cm set cph <cluster_pool_host_name>

2. Run the cm get cph command to display all ClusterPool hosts that you created. The ACTIVE column displays what is active.

Create a Hive ClusterPool

A Hive ClusterPool defines the caracteristic of the clusters in that pool. Each time the pool needs to be refilled, Hive reads the ClusterPool configuration to create new clusters. Discarded clusters are deleted and not reinserted in the pool.

Generally you create a ClusterPool once in a while, for example if there is a new version of Red Hat OpenShift or if you want to define a ClusterPool for each cluster sizes (small/medium/large).

The Hive ClusterPool uses an imageSet to deploy Red Hat Advanced Cluster Management and you have to define that imageSet on your cluster, but the cm-cli allows you to simply give an OpenShift image and it will create the imageSet for you.

The cm-cli provides a simplified way to create a ClusterPool with a limited number of parameters in a single .yaml file to create a ClusterPool. (run cm create clusterpool -h to get the template).

You can reference an clusterImageSet custom reference of an ocpImage from the Red Hat repository located at https://quay.io/repository/openshift-release-dev/ocp-release?tab=tags.

For example for a clusterpool on AWS:

1. Create a file .yaml with the following content. See the configuration of ClusterPool to create single node clusters on AWS:

clusterPool:
  size: 2 #Number of cluster in the pool
  master: 
    replicas: 1 #Number of master node, default 3
  worker:
    replicas:  0  #Number of worker node, default 3
  cloud: aws # clouds values can be aws, azure, gcp, openstack, vsphere
  ocpImage: quay.io/openshift-release-dev/ocp-release:4.9.20-x86_64
  imagePullSecret: |- 
    <your_image_pool_secret>
  sshPublicKey: |-
    <your_public_ssh_key>
  aws:
    baseDnsDomain: mycompany.com # baseDomain of your cluster (ie: mycompany.com)
    awsAccessKeyID: <your_aws_access_key_id>
    awsSecretAccessKeyID: <your_aws_secret_access_key_id>
    region: us-east-1 # Region (ie: us-east-1)
    master:
      type: m5.2xlarge
      rootVolume: # see below the default values 
        iops: 4000
        size: 100
        type: gp2
    worker:
      type: m5.2xlarge
      rootVolume: # see below the default values
        iops: 2000
        size: 100
        type: gp2

2. Run the following command to create the ClusterPool:

cm create clusterpool aws-4-9-sno --values <file_name>

Note: cm-cli allows you to create ClusterPool not only for AWS but also for Azure and Google cloud.

You can check the ClusterPool status by running the following command:

cm get clusterpool
CLUSTER_POOL_HOST   CLUSTER_POOL   SIZE   READY   STANDBY   ACTUAL_SIZE
my-cph              aws-4-9-sno    2      0       <none>    2

Interestingly, cm-cli also offers a way to generate the .yaml file from an existing ClusterPool and doing so you can modify the generate .yaml file and create a new ClusterPool with new parameters.

cm get config cp <your_existing_clusterpool> --output-file <my-clusterpool_yaml>

You can also scale up and down your ClusterPool with the command cm scale clusterpool --size <clusterpool_size>.

Request a cluster from a Hive ClusterPool

After you set up your ClusterPool host and you have a ClusterPool, you can request a cluster from the pool by creating a clusterclaim.

1. List the available ClusterPools on your current ClusterPool host. Run the following command:

cm get cp
CLUSTER_POOL_HOST   CLUSTER_POOL      SIZE   READY   STANDBY   ACTUAL_SIZE
my-cph              aws-4-9-sno       2      2       <none>    2

See that two clusters are ready in the pool, which has a total size of two. So the pool is full.

2. Request a cluster from that pool. Run the following command to create a cluster claim:

cm create clusterclaim aws-4-9-sno my-cluster

If a cluster is ready in the ClusterPool, your cluster will be ready to use after few minutes, otherwise you will need to wait that the cluster gets provisioned by your cloud provider.

cm create cc aws-4-9-sno my-cluster  
clusterclaim my-cluster created
(0/60) clusterclaim my-cluster is not running
(1/60) clusterclaim my-cluster is not running
(2/60) clusterclaim my-cluster is not running
(3/60) clusterclaim my-cluster is not running
(4/60) clusterclaim my-cluster is not running
(5/60) clusterclaim my-cluster is not running
(6/60) clusterclaim my-cluster is not running
(7/60) clusterclaim my-cluster is running with id aws-4-9-sno-prwg6

To login on that cluster, you can run the following command:

cm use clusterclaim my-cluster
oc cluster-info

To get the credential of your cluster run the following command:

cm get clusterclaim my-cluster --creds

To open the Openshift console for that cluster and get the credentials to access it, run the following command:

cm console clusterclaim my-cluster --creds

This will automatically open your default browser on the Openshift console home page and display the credentials to login.

Save money with hibernate

You can setup a schedule to hibernate all clusters in your namespace. If you don't, then at any time you can hibernate or wake up your cluster by running the following command:

cm hibernate clusterclaim my-cluster

Then, run the following:

cm run clusterclaim my-cluster

Add the cluster into Red Hat Advanced Cluster Management a or Multicluster engine

If you didn't install Red Hat Advanced Cluster Management or a Multicluster Engine on that cluster, you can bring it under the management of either by running the following commands.

1. Log in on your ClusterPool host with the following command:

cm use clusterpoolhost my-cph

2. Run the following command to import:

cm attach clusterclaim my-cluster

3. List the cluster ready for management by running the following command:

cm get clusters

4. Finally, you can remove that cluster from the management by running the following command:

cm detach clusterlaim my-cluster

Delete a cluster

To delete the cluster, run cm delete clusterclaim my-cluster.

Conclusion

The cm-cli helps you to not only interact in a simplied way with the Red Hat Advanced Cluster Management or the Multicluster Engine platform for Kubernetes, but it also provides you an easy way to manage the cluster lifecycle, and increases the productivity of your team.