Assisted Installer On-Premise Deep Dive

October 6, 2021Karim Latouche

Introduction

In this series of blog posts, we will demonstrate how Infrastructure as a code becomes a reality with OpenShift Assisted Installer onprem. This post will leverage kvm to show how to use Assisted Installer to deploy OpenShift, but the concepts here can extend to bare metal or vSphere deployments just as easily.

Lab Preparation

In this lab we will simulate Bare metal nodes with KVM VMs. Terraform will be used to orchestrate this virtual infrastructure. A minimum of 256Gb of Ram and 500Gb SSD drive is recommended. The scripts and install steps below are based around the use of a Centos 8 machine as your host machine. In order to have everything set and all the bits installed, run the following commands:

git clone https://github.com/latouchek/assisted-installer-deepdive.git
cd assisted-installer-deepdive
cp -r terraform /opt/
cd scripts
sh prepare-kvm-host.sh

The script creates a dedicated ocp network. It is mandatory to have a DNS and a static DHCP server on that network. A dnsmasq.conf template is provided in assisted-installer-deepdive/config/ with mac adresses matching the OCP VMs that we will deploy later. It can be run on the host or on a dedicated VM/container.

Part I : Deploying the OpenShift Assisted Installer service on premise

1. Get the bits and build the service

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
setenforce 0
dnf install -y @container-tools
dnf group install "Development Tools" -y
dnf -y install python3-pip socat make tmux git jq crun
git clone https://github.com/openshift/assisted-service
cd assisted-service
IP=192.167.124.1
AI_URL=http://$IP:8090

Modify onprem-environment and Makefile to set proper URL and port forwarding

sed -i "s@SERVICE_BASE_URL=.*@SERVICE_BASE_URL=$AI_URL@" onprem-environment
sed -i "s/5432,8000,8090,8080/5432:5432 -p 8000:8000 -p 8090:8090 -p 8080:8080/" Makefile
make deploy-onprem
.
.
.

If everything went well, we should see 4 containers running inside a pod

[root@kvm-host ~]podman ps
CONTAINER ID  IMAGE                                          COMMAND               CREATED             STATUS                 PORTS                                                                                           NAMES
a940818185cb  k8s.gcr.io/pause:3.5                                                 3 minutes ago       Up 2 minutes ago       0.0.0.0:5432->5432/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp  59b56cb07140-infra
d94a46c8b515  quay.io/ocpmetal/postgresql-12-centos7:latest  run-postgresql        2 minutes ago       Up 2 minutes ago       0.0.0.0:5432->5432/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp  db
8c0e90d8c4fa  quay.io/ocpmetal/ocp-metal-ui:latest           /opt/bitnami/scri...  About a minute ago  Up About a minute ago  0.0.0.0:5432->5432/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp  ui
e98627cdc5f8  quay.io/ocpmetal/assisted-service:latest       /assisted-service     42 seconds ago      Up 43 seconds ago      0.0.0.0:5432->5432/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:8090->8090/tcp  installer

[root@kvm-host ~] podman pod ps
POD ID        NAME                STATUS      CREATED        INFRA ID      # OF CONTAINERS
59b56cb07140  assisted-installer  Running     4 minutes ago  a940818185cb  4

API should be accessible at http://192.167.124.1:8090 and GUI at http://192.167.124.1:8080/

API documentation can be found here

2. How does it work

In order to provision a cluster the following process must be followed:

Create a new OpenShift cluster definition in a json file
Register the new cluster by presenting the definition data to the API
Create a discovery boot media the nodes will boot from in order to be introspected and validated
Assign roles to introspected nodes and complete the cluster definition
Trigger the deployment

Part II : Using the Assisted Installer API

In this part we will show how to deploy a 5 nodes OCP cluster y following the steps we mentioned above. Even though this lab is purely cli based it is recommended to have the UI on sight to understand the whole process.

1. Deploy our first cluster with AI API

Create a cluster definition file

export CLUSTER_SSHKEY=$(cat ~/.ssh/id_ed25519.pub)
export PULL_SECRET=$(cat pull-secret.txt | jq -R .)
cat << EOF > ./deployment-multinodes.json
{
  "kind": "Cluster",
  "name": "ocpd",  
  "openshift_version": "4.8",
  "ocp_release_image": "quay.io/openshift-release-dev/ocp-release:4.8.5-x86_64",
  "base_dns_domain": "lab.local",
  "hyperthreading": "all",
  "ingress_vip": "192.167.124.8",
  "schedulable_masters": false,
  "high_availability_mode": "Full",
  "user_managed_networking": false,
  "platform": {
    "type": "baremetal"
   },
  "cluster_networks": [
    {
      "cidr": "10.128.0.0/14",
      "host_prefix": 23
    }
  ],
  "service_networks": [
    {
      "cidr": "172.31.0.0/16"
    }
  ],
  "machine_networks": [
    {
      "cidr": "192.167.124.0/24"
    }
  ],
  "network_type": "OVNKubernetes",
  "additional_ntp_source": "ntp1.hetzner.de",
  "vip_dhcp_allocation": false,      
  "ssh_public_key": "$CLUSTER_SSHKEY",
  "pull_secret": $PULL_SECRET
}
EOF

high_availability_mode and schedulable_masters parameters let you decide what type of cluster you want to install. Here is how to set those parameters:

3 nodes clusters: "high_availability_mode": "Full" and "schedulable_masters": true
3+ nodes clusters: "high_availability_mode": "Full" and "schedulable_masters": false
Single Node: "high_availability_mode": "None"

You can choose if you want to handle loadbalancing in house or leave it to OCP by setting user_managed_networking to true. In both case, DHCP and DNS server are mandatory (Only DNS in the case of a static IP deployment).

Use deployment-multinodes.json to register the new cluster

AI_URL='http://192.167.124.1:8090'
curl -s -X POST "$AI_URL/api/assisted-install/v1/clusters" \
   -d @./deployment-multinodes.json --header "Content-Type: application/json" | jq .

Check cluster is registered Once the cluster definition has been sent to an the API we should be able to retrieve its unique id

  CLUSTER_ID=$(curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].id')
  [root@kvm-host ~] echo $CLUSTER_ID
  43b9c2f0-218e-4e76-8889-938fd52d6290

Check the new cluster status

  [root@kvm-host ~] curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].status'
pending-for-input

When registering a cluster, the assisted installer runs a series of validation tests to assess if the cluster is ready to be deployed. 'pending-for-input' tells us we need to take some actions. Let's take a look at validations_info:

 curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].validations_info'|jq .

We can see below that the installer is waiting for the hosts . Before building the hosts, we need to create the Discovery ISO.

  {
    "id": "sufficient-masters-count",
    "status": "failure",
    "message": "Clusters must have exactly 3 dedicated masters. Please either add hosts, or disable the worker host"
    }
      ],

  {
    "id": "cluster-cidr-defined",
    "status": "success",
    "message": "The Cluster Network CIDR is defined."
  },

Build the discovery boot ISO The discovery boot ISO is a live CoreOS image that the nodes will boot from. Once booted an introspection will be performed by the discovery agent and data sent to the assisted service. If the node passes the validation tests its status_info will be "Host is ready to be installed". We need some extra parameters to be injected into the ISO . To do so, we create a data file as described bellow:
```
  cat << EOF > ./discovery-iso-params.json
 {
  "ssh_public_key": "$CLUSTER_SSHKEY",
   "pull_secret": $PULL_SECRET,
   "image_type": "full-iso"
  }
 EOF
```
ISO is now ready to be built! Let's make the API call! As you can see we use the data file so pull-secret and ssh public key are injected into the live ISO.
```
 curl -s -X POST "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \  
-d @discovery-iso-params.json \
--header "Content-Type: application/json" \
| jq '.'
```
In real world we would need to present this ISO to our hosts so they can boot from it. Because we are using KVM, we are going to download the ISO in the libvirt images directory and later create the VMs
```
curl \
  -L "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/image" \
  -o /var/lib/libvirt/images/default/discovery_image_ocpd.iso
```

Start the nodes and the discovery process In this lab, BM nodes are virtual and need to be provisioned first. A Terraform file is provided and will build 3 Masters, 4 workers. All the VMS are using the previously generated ISO to boot. Run the following commands inside the Terraform folder

[root@kvm-host terraform-ocp4-cluster-ai] terraform init ; terraform apply -auto-approve
 Apply complete! Resources: 24 added, 0 changed, 0 destroyed.

[root@kvm-host terraform-ocp4-cluster-ai] virsh list --all
 Id   Name               State
-----------------------------------
 59   ocp4-master3       running
 60   ocp4-master1       running
 61   ocp4-master2       running
 -    ocp4-worker1       shut off
 -    ocp4-worker1-ht    shut off
 -    ocp4-worker2       shut off
 -    ocp4-worker3       shut off

Only the master nodes will start for now. Wait 1 mn for them to be discovered and check validations_info

 curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" \
 -H "accept: application/json" \
 -H "get_unregistered_clusters: false"| jq -r '.[].progress'

   ........
   "hosts-data": [
     {
       "id": "all-hosts-are-ready-to-install",
       "status": "success",
       "message": "All hosts in the cluster are ready to install."
     },
     {
       "id": "sufficient-masters-count",
       "status": "success",
       "message": "The cluster has a sufficient number of master candidates."
     }
     .........

Our hosts have been validated and are ready to be installed. Let's take a closer look at the discovery data.

Retrieve the discovery hosts data with an API call

[root@kvm-host ~]curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" \
-H "accept: application/json" \
-H "get_unregistered_clusters: false"| jq -r '.[].hosts'

{
"checked_in_at": "2021-09-15T22:57:25.484Z",
"cluster_id": "71db492e-207e-47eb-af7b-c7c716c7e09d",
"connectivity": "{\"remote_hosts\":[{\"host_id\":\"2121a000-d27e-4596-a408-6813d3114caf\",\"l2_connectivity\":[{\"outgoing_ip_address\":\"192.167.124.12\",\"outgoing_nic\":\"ens3\",\"remote_ip_address\":\"192.167.124.13\",\"remote_mac\":\"aa:bb:cc:11:42:11\",\"successful\":true}],\"l3_connectivity\":[{\"average_rtt_ms\":0.304,\"outgoing_nic\":\"ens3\",\"remote_ip_address\":\"192.167.124.13\",\"successful\":true}]},{\"host_id\":\"84083091-8c0c-470b-a157-d002dbeed785\",\"l2_connectivity\":[{\"outgoing_ip_address\":\"192.167.124.12\",\"outgoing_nic\":\"ens3\",\"remote_ip_address\":\"192.167.124.14\",\"remote_mac\":\"aa:bb:cc:11:42:12\",\"successful\":true}],\"l3_connectivity\":[{\"average_rtt_ms\":0.237,\"outgoing_nic\":\"ens3\",\"remote_ip_address\":\"192.167.124.14\",\"successful\":true}]}]}",
"created_at": "2021-09-15T19:23:23.614Z",
"discovery_agent_version": "latest",
"domain_name_resolutions": "{\"resolutions\":[{\"domain_name\":\"api.ocpd.lab.local\",\"ipv4_addresses\":[\"192.167.124.7\"],\"ipv6_addresses\":[]},{\"domain_name\":\"api-int.ocpd.lab.local\",\"ipv4_addresses\":[],\"ipv6_addresses\":[]},{\"domain_name\":\"console-openshift-console.apps.ocpd.lab.local\",\"ipv4_addresses\":[\"192.167.124.8\"],\"ipv6_addresses\":[]},{\"domain_name\":\"validateNoWildcardDNS.ocpd.lab.local\",\"ipv4_addresses\":[],\"ipv6_addresses\":[]}]}",
"href": "/api/assisted-install/v2/infra-envs/71db492e-207e-47eb-af7b-c7c716c7e09d/hosts/fa89d7cd-c2d9-4f26-bd78-155647a32b04",
"id": "fa89d7cd-c2d9-4f26-bd78-155647a32b04",
"infra_env_id": "71db492e-207e-47eb-af7b-c7c716c7e09d",
"installation_disk_id": "/dev/disk/by-path/pci-0000:00:05.0",
"installation_disk_path": "/dev/vda",
"inventory": "{\"bmc_address\":\"0.0.0.0\",\"bmc_v6address\":\"::/0\",\"boot\":{\"current_boot_mode\":\"bios\"},\"cpu\":{\"architecture\":\"x86_64\",\"count\":12,\"flags\":[\"fpu\",\"vme\",\"de\",\"pse\",\"tsc\",\"msr\",\"pae\",\"mce\",\"cx8\",\"apic\",\"sep\",\"mtrr\",\"pge\",\"mca\",\"cmov\",\"pat\",\"pse36\",\"clflush\",\"mmx\",\"fxsr\",\"sse\",\"sse2\",\"ss\",\"syscall\",\"nx\",\"pdpe1gb\",\"rdtscp\",\"lm\",\"constant_tsc\",\"arch_perfmon\",\"rep_good\",\"nopl\",\"xtopology\",\"cpuid\",\"tsc_known_freq\",\"pni\",\"pclmulqdq\",\"vmx\",\"ssse3\",\"fma\",\"cx16\",\"pdcm\",\"pcid\",\"sse4_1\",\"sse4_2\",\"x2apic\",\"movbe\",\"popcnt\",\"tsc_deadline_timer\",\"aes\",\"xsave\",\"avx\",\"f16c\",\"rdrand\",\"hypervisor\",\"lahf_lm\",\"abm\",\"cpuid_fault\",\"invpcid_single\",\"pti\",\"ssbd\",\"ibrs\",\"ibpb\",\"stibp\",\"tpr_shadow\",\"vnmi\",\"flexpriority\",\"ept\",\"vpid\",\"ept_ad\",\"fsgsbase\",\"tsc_adjust\",\"bmi1\",\"avx2\",\"smep\",\"bmi2\",\"erms\",\"invpcid\",\"xsaveopt\",\"arat\",\"umip\",\"md_clear\",\"arch_capabilities\"],\"frequency\":3491.914,\"model_name\":\"Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz\"},\"disks\":[

  {\"bootable\":true,\"by_path\":\"/dev/disk/by-path/pci-0000:00:01.1-ata-1\",\"drive_type\":\"ODD\",\"hctl\":\"0:0:0:0\",\"id\":\"/dev/


"progress": {
  "current_stage": "",
.
},
"progress_stages": null,
"role": "auto-assign",

"user_name": "admin",
"validations_info": "{\"hardware\":[{\"id\":\"has-inventory\",\"status\":\"success\",\"message\":\"Valid inventory exists for the host\"},{\"id\":\"has-min-cpu-cores\",\"status\":\"success\",\"message\":\"Sufficient CPU cores\"},{\"id\":\"has-min-memory\",\"status\":\"success\",\"message\":\"Sufficient minimum RAM\"},{\"id\":\"has-min-valid-disks\",\"status\":\"success\",\"message\":\"Sufficient disk capacity\"},{\"id\":\"has-cpu-cores-for-role\",\"status\":\"success\",\"message\":\"Sufficient CPU cores for role auto-assign\"},{\"id\":\"has-memory-for-role\",\"status\":\"success\",\"message\":\"Sufficient RAM for role auto-assign\"},{\"id\":\"hostname-unique\",\"status\":\"success\",\"message\":\"Hostname ocp4-master0.ocpd.lab.local is unique in cluster\"},{\"id\":\"hostname-valid\",\"status\":\"success\",\"message\":\"Hostname ocp4-master0.ocpd.lab.local is allowed\"},{\"id\":\"valid-platform\",\"status\":\"success\",\"message\":\"Platform KVM is allowed\"},
.............................................................................
{\"id\":\"sufficient-installation-disk-speed\",\"status\":\"success\",\"message\":\"Speed of installation disk has not yet been measured\"},{\"id\":\"compatible-with-cluster-platform\",\"status\":\"success\",\"message\":\"Host is compatible with cluster platform \"message\":\"lso is disabled\"},{\"id\":\"ocs-requirements-satisfied\",\"status\":\"success\",\"message\":\"ocs is disabled\"}]}"
}

This is a truncated version of the full ouput as it contains quite a lot of informations. Basically the agent provides all hardware info to the assisted service so it can have a precise inventory of the host hardware and eventually validate the nodes. To get more info about validation and hardware inventory, you can use these 2 one liners

curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" \
-H "get_unregistered_clusters: false"| jq -r '.[].validations_info'|jq .

curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" \
-H "get_unregistered_clusters: false"| jq -r '.[].hosts[].inventory'|jq -r .

One important point to notice is that each hosts gets its own id after this process. We can extract these with the following call:

[root@kvm-host ~] curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true"\
-H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].id'

2121a000-d27e-4596-a408-6813d3114caf
84083091-8c0c-470b-a157-d002dbeed785
fa89d7cd-c2d9-4f26-bd78-155647a32b04

Assign role to discovered Nodes

After validation, each node gets the 'auto-assign' role. We can check with this API call:

[root@kvm-host ~]curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept:    application/json" -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].role'
auto-assign
auto-assign
auto-assign

If you want something a bit more predictable, you can assign roles based on nodes id. Since only our master nodes have been discovered, we will assign them the master role:

 for i in `curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true"\
   -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].id'| awk 'NR>0' |awk '{print $1;}'`
 do curl -X PATCH "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"hosts_roles\": [ { \"id\": \"$i\", \"role\": \"master\" } ]}"
 done

Check the result:

[root@kvm-host ~]curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true"\
-H "accept: application/json" \
 -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].role'
master
master
master

Add workers, complete configuration and trigger the installation

It's now time to start our workers. The same discovery process will take place and the new nodes will get the auto-assign role. Because a cluster cannot have more than 3 masters, we are sure auto-assign=worker this time. Because we set vip_dhcp_allocation to false in the cluster definition file, we need to set api_vip parameter before we can trigger the installation.
```
  curl -X PATCH "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID" \
  -H "accept: application/json"\
  -H "Content-Type: application/json" -d "{ \"api_vip\": \"192.167.124.7\"}"
```
And finally start installation:
```
  curl -X PATCH "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" -d "{ \"api_vip\": \"192.167.124.7\"}"
```
During the installation process, disks will be written and nodes will reboot. One of the masters will also play the bootstrap role until the control plane is ready then the installation will continue as usual.

Monitoring the installation progress We can closely monitor the nodes states during the installation process:

[root@kvm-host ~]curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true"\
 -H "accept: application/json" \
 -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].progress'
{
  "current_stage": "Writing image to disk",
  "installation_percentage": 42,
  "progress_info": "92%",
  "stage_started_at": "2021-09-16T15:56:39.275Z",
  "stage_updated_at": "2021-09-16T15:57:31.215Z"
}
{
  "current_stage": "Writing image to disk",
  "installation_percentage": 42,
  "progress_info": "93%",
  "stage_started_at": "2021-09-16T15:56:38.290Z",
  "stage_updated_at": "2021-09-16T15:57:31.217Z"
}
{
  "current_stage": "Writing image to disk",
  "installation_percentage": 30,
  "progress_info": "92%",
  "stage_started_at": "2021-09-16T15:56:38.698Z",
  "stage_updated_at": "2021-09-16T15:57:31.218Z"
}
{
  "current_stage": "Waiting for control plane",
  "installation_percentage": 44,
  "stage_started_at": "2021-09-16T15:56:32.053Z",
  "stage_updated_at": "2021-09-16T15:56:32.053Z"
}
{
  "current_stage": "Waiting for control plane",
  "installation_percentage": 44,
  "stage_started_at": "2021-09-16T15:56:42.398Z",
  "stage_updated_at": "2021-09-16T15:56:42.398Z"
}

To monitor the whole installation progress:

[root@kvm-host ~]curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" \
-H "accept: application/json" \
-H "get_unregistered_clusters: false"| jq -r '.[].progress'
{
 "finalizing_stage_percentage": 100,
 "installing_stage_percentage": 100,
 "preparing_for_installation_stage_percentage": 100,
 "total_percentage": 100
}

Retrieve kubeconfig and credentials

[root@kvm-host ~] curl -X GET "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID/downloads/kubeconfig" \
-H "accept: application/octet-stream" > .kube/config
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                               Dload  Upload   Total   Spent    Left  Speed
100 12104  100 12104    0     0  2955k      0 --:--:-- --:--:-- --:--:-- 2955k
[root@kvm-host ~]oc get nodes
NAME                          STATUS   ROLES    AGE    VERSION
ocp4-master0.ocpd.lab.local   Ready    master   119m   v1.21.1+9807387
ocp4-master1.ocpd.lab.local   Ready    master   134m   v1.21.1+9807387
ocp4-master2.ocpd.lab.local   Ready    master   134m   v1.21.1+9807387
ocp4-worker0.ocpd.lab.local   Ready    worker   119m   v1.21.1+9807387
ocp4-worker1.ocpd.lab.local   Ready    worker   119m   v1.21.1+9807387

[root@kvm-host ~] curl -X GET "$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID/credentials" \
-H "accept: application/json" |jq -r .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   132  100   132    0     0  44000      0 --:--:-- --:--:-- --:--:-- 44000
{
  "console_url": "https://console-openshift-console.apps.ocpd.lab.local",
  "password": "8Tepe-uxF7Q-ztHg5-yoKPQ",
  "username": "kubeadmin"
}

Part III : Day 2 Operations

Adding worker nodes

In order to add extra workers to an existing cluster the following process must be followed:

Create a new 'AddHost cluster'

This creates a new OpenShift cluster definition for adding nodes to our existing OCP cluster. We have to manually generate the new cluster id and name and make sure the api_vip_dnsname matches the existing cluster.

###Generate id for addhost cluster####

NCLUSTER_ID=$(curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].id'| tr b c)
##### creating addhost cluster
echo $NCLUSTER_ID

curl -X POST "http://192.167.124.1:8090/api/assisted-install/v1/add_hosts_clusters" \
  -H "accept: application/json" -H "Content-Type: application/json" \
  -d "{ \"id\": \"$NCLUSTER_ID\", \"name\": \"ocp2\", \"api_vip_dnsname\": \"api.ocpd.lab.local\", \"openshift_version\": \"4.8\"}"

Create a new discovery boot media the new nodes will boot from in order to be introspected and validated

####Patch new cluster to add pullsecret####

cat << EOF > ./new-params.json
 {
   "ssh_public_key": "$CLUSTER_SSHKEY",
   "pull_secret": $PULL_SECRET
 }
 EOF
 curl -s -X PATCH "$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID"   -d @new-params.json   --header "Content-Type: application/json"   | jq '.'

 ####create and download new ISO ####
 curl -s -X POST "$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID/downloads/image" \
   -d @new-params.json --header "Content-Type: application/json" \
   | jq '.'

 curl -L "$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID/downloads/image" \
 -o /var/lib/libvirt/images/default/discovery_image_ocpd2.iso

Before starting the extra workers, make sure they boot from the new created ISO

virsh change-media --domain ocp4-worker1-ht hda \
                   --current --update \
                   --source /var/lib/libvirt/images/default/discovery_image_ocpd2.iso
virsh start --domain ocp4-worker1-ht

Let's take a closer look at both clusters

curl -s -X GET   -H "Content-Type: application/json"   "$AI_URL/api/assisted-install/v1/clusters"   | jq -r .[].id

Output should look like this:

58fb589e-2f8b-44ee-b056-08499ba7ddd5  <-- UUID of the existing cluster
58fc589e-2f8c-44ee-c056-08499ca7ddd5  <-- UUID of the AddHost cluster

Use the API to get more details:

[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID"   | jq -r .kind
AddHostsCluster
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID"   | jq -r .kind
Cluster
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID"   | jq -r .status
adding-hosts
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID"   | jq -r .status
installed
####Check nodes for each clusters####
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$CLUSTER_ID"   | jq -r .hosts[].requested_hostname
ocp4-master1.ocpd.lab.local
ocp4-master2.ocpd.lab.local
ocp4-worker0.ocpd.lab.local
ocp4-worker1.ocpd.lab.local
ocp4-master0.ocpd.lab.local
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID"   | jq -r .hosts[].requested_hostname
ocp4-worker1-ht.ocpd.lab.local
[root@kvm-host ~] curl -s -X GET   --header "Content-Type: application/json"   \
"$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID"   | jq -r .hosts[].role
auto-assign

We see above that everything is working as expected:

The new cluster is in adding-hosts state
The existing cluster is in installed state
The new worker has been discovered and has been given the right role

Start the new node installation:

curl -X POST "$AI_URL/api/assisted-install/v1/clusters/$NCLUSTER_ID/actions/install_hosts" \
-H "accept: application/json" | jq '.'

As soon the installation begin,the new node will get the worker roles

[root@kvm-host ~] curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" -H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r '.[].hosts[].role'
master
master
worker
worker
master
worker

Wait for the new worker to reboot and check pending CSRs

[root@kvm-host ~] oc get csr|grep Pending
csr-5jrm7 5m55s   kubernetes.io/kube-apiserver-client-kubelet   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper         <none>              Pending

###Approve all CSR###
[root@kvm-host ~] for csr in $(oc -n openshift-machine-api get csr | awk '/Pending/ {print $1}'); do oc adm certificate approve $csr;done
certificatesigningrequest.certificates.k8s.io/csr-5jrm7 approved

We should now see the new node:

[root@kvm-host ~] oc get nodes
NAME                             STATUS     ROLES    AGE   VERSION
ocp4-master0.ocpd.lab.local      Ready      master   59m   v1.22.0-rc.0+75ee307
ocp4-master1.ocpd.lab.local      Ready      master   40m   v1.22.0-rc.0+75ee307
ocp4-master2.ocpd.lab.local      Ready      master   59m   v1.22.0-rc.0+75ee307
ocp4-worker0.ocpd.lab.local      Ready      worker   42m   v1.22.0-rc.0+75ee307
ocp4-worker1-ht.ocpd.lab.local   NotReady   worker   48s   v1.22.0-rc.0+75ee307
ocp4-worker1.ocpd.lab.local      Ready      worker   42m   v1.22.0-rc.0+75ee307

After a few minutes we should see:

[root@kvm-host ~] oc get nodes
NAME                             STATUS   ROLES    AGE     VERSION
ocp4-master0.ocpd.lab.local      Ready    master   62m     v1.22.0-rc.0+75ee307
ocp4-master1.ocpd.lab.local      Ready    master   44m     v1.22.0-rc.0+75ee307
ocp4-master2.ocpd.lab.local      Ready    master   62m     v1.22.0-rc.0+75ee307
ocp4-worker0.ocpd.lab.local      Ready    worker   45m     v1.22.0-rc.0+75ee307
ocp4-worker1-ht.ocpd.lab.local   Ready    worker   3m54s   v1.22.0-rc.0+75ee307
ocp4-worker1.ocpd.lab.local      Ready    worker   45m     v1.22.0-rc.0+75ee307

Check with AI API

[root@kvm-host ~] curl -s -X GET "$AI_URL/api/assisted-install/v2/clusters?with_hosts=true" \
-H "accept: application/json" -H "get_unregistered_clusters: false"| jq -r .[].hosts[].status
installed
installed
installed
installed
installed
added-to-existing-cluster

We successfully added an extra worker.

Thank you for reading

References

About the author

Karim Latouche

Browse by channel

Explore all channels

Platform products

Try & buy

Featured cloud services

By category

By organization type

By customer

Featured

Topics

Articles

More to explore

For customers

For partners

About us

Open source

Company details

Communities

Recommendations

Select a language

Select a language

Assisted Installer On-Premise Deep Dive

Introduction

Lab Preparation

Part I : Deploying the OpenShift Assisted Installer service on premise

1. Get the bits and build the service

2. How does it work

Part II : Using the Assisted Installer API

1. Deploy our first cluster with AI API

Part III : Day 2 Operations

Adding worker nodes

Thank you for reading

References

About the author

Karim Latouche

More like this

Browse by channel

Products

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links