How to perform an IPI installation of OpenShift on vSphere with static IP addresses

Deploying OpenShift on vSphere is quick and simple when performing an IPI installation. However, IPI installations require using Dynamic Host Configuration Protocol (DHCP). Many organizations do not allow the use of DHCP on their networks. Additionally, use cases such as external load balancers are difficult to manage with DHCP-assigned addresses. OpenShift 4.14 allows static IP addresses with Tech Preview IPI installations.

Overview

Static IP addresses are provisioned at install time by configuring install-config.yaml. After installation, additional machines may be scaled in to the cluster with static IPs by providing IP addresses in the machine API machine specs or using a controller that assigns IP addresses to the systems.

Prerequisites

• A network administrator has allocated IP addresses for the bootstrap, control plane, compute nodes, API VIP, and ingress VIP.
• install-config.yaml is configured with a static IPs bootstrap, control plane, compute nodes, API VIP, and ingress VIP.
• install-config.yaml is configured with the TechPreviewNoUpgrade feature set.

Note: Allocating an API and ingress VIP is optional if the cluster is being installed with an external load balancer.

Sample install-config.yaml file:

featureSet: TechPreviewNoUpgrade
baseDomain: somelab.net
compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    vsphere: {}
  replicas: 3
controlPlane:
  architecture: amd64
  hyperthreading: Enabled
  name: master
  platform:
    vsphere: {}
  replicas: 3
...  
platform:
  vsphere:
    apiVIPs:
      - 192.168.0.100
    ingressVIPs:
      - 192.168.0.101
    hosts:
    - role: bootstrap (0)
      networkDevice: 
        ipAddrs: (1)
        - 192.168.0.4/24
        gateway: 192.168.0.1 (2)
        nameservers:
        - 192.168.0.1 (3)
    - role: control-plane
      networkDevice:
        ipAddrs:
        - 192.168.0.5/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1
    - role: control-plane
      networkDevice:
        ipAddrs:
        - 192.168.0.6/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1
    - role: control-plane
      networkDevice:
        ipAddrs:
        - 192.168.0.7/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1
    - role: compute
      networkDevice:
        ipAddrs:
        - 192.168.0.8/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1
    - role: compute
      networkDevice:
        ipAddrs:
        - 192.168.0.9/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1
    - role: compute
      networkDevice:
        ipAddrs:
        - 192.168.0.10/24
        gateway: 192.168.0.1
        nameservers:
        - 192.168.0.1

These annotations apply to the settings in the file above:

• (0) - The role of the network definition. Valid values are bootstrap, control-plane, or compute.
• (1) - IPv4 and/or IPv6 addresses to be applied to the network interface.
• (2) - The default gateway to be applied to the network interface.
• (3) - Up to three nameservers to be used for DNS resolution.

Note: The number of control-plane and compute network definitions must match the master and worker machine pool replicas, respectively. 

In addition, you must provide one bootstrap network definition.

Machine scaling with static IP addresses

There are two options for scaling in new machines with static IPs:

1. Create a new machine resource with static IP configuration set in the machine YAML.
2. Scale up new machines with a machineset.

Scaling individual machines

1. Create a machine resource YAML that includes the required network definition.

apiVersion: machine.openshift.io/v1beta1
kind: Machine
metadata:
  creationTimestamp: null
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id> 
    machine.openshift.io/cluster-api-machine-role: <role> 
    machine.openshift.io/cluster-api-machine-type: <role> 
    machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
  name: <infrastructure_id>-<role>
  namespace: openshift-machine-api
spec:
  lifecycleHooks: {}
  metadata: {}
  providerSpec:
    value:
      apiVersion: machine.openshift.io/v1beta1
      credentialsSecret:
        name: vsphere-cloud-credentials
      diskGiB: 120
      kind: VSphereMachineProviderSpec
      memoryMiB: 8192
      metadata:
        creationTimestamp: null
      network:
        devices:
        - gateway: 192.168.0.1 (1)
          ipAddrs:
          - 192.168.0.8/24 (2)
          nameservers: (3)
          - 192.168.0.1
          networkName: qe-segment-204
      numCPUs: 4
      numCoresPerSocket: 2
      snapshot: ""
      template: <vm_template_name>
      userDataSecret:
        name: worker-user-data
      workspace:
        datacenter: <vcenter_datacenter_name> 
        datastore: <vcenter_datastore_name> 
        folder: <vcenter_vm_folder_path> 
        resourcepool: <vsphere_resource_pool> 
        server: <vcenter_server_ip> 
status: {}

The following annotations apply to the file above:

• (1) - The default gateway to be applied to the network interface.
• (2) - IPv4 and/or IPv6 addresses to be applied to the network interface.
• (3) - Up to three nameservers to be used for DNS resolution.

Run the following command to implement the YAML file:

oc create -f new-machine.yaml

Scaling with machinesets

Machinesets may be used to scale machines with static IPs. However, the machine API machine controller does not directly interface with an IP management API. Rather, when a machineset is configured to request a static IP, the machine API machine controller will create an IPAddressClaim in the openshift-machine-api namespace.

Note: Organizations may use any number of IP management services. To enable the use of these services, the OpenShift administrator is responsible for handling the creation and binding of IPAddressClaims. As such, automation capable of interfacing with the IP management service and the OpenShift API must be implemented.

In Kubernetes, controllers typically handle this type of reconciliation. It is recommended that a controller be written that can interface with the IP management service and the OpenShift API. The example at github.com/rvanderp3/machine-ipam-controller implements such a controller.

The following section describes how IPAddresses can be bound with a series of oc commands.

Binding IPAddressClaims

When an IPAddressClaim is created in association with a machine, a virtual machine associated with that machine will not be created until outstanding IPAddressClaim(s) are bound.

Here is an overview of machine API machine controller reconciliation as related to IPAddressClaims:

IPAddressClaims may be bound manually by using oc invocations or creating a customer controller. The diagram below describes how to reconcile IPAddressClaims.

Here's an overview of IPAddressClaim binding:

1. Configure a machineset with an IPPool reference. The IPPool reference is included in IPAddressClaims created by the machine API.

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  annotations:
    machine.openshift.io/memoryMb: "8192"
    machine.openshift.io/vCPU: "4"
  labels:
    machine.openshift.io/cluster-api-cluster: <infrastructure_id>
  name: <infrastructure_id>-<role>
  namespace: openshift-machine-api
spec:
  replicas: 0
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: <infrastructure_id>
      machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
  template:
    metadata:
      labels:
        ipam: "true"
        machine.openshift.io/cluster-api-cluster: <infrastructure_id>
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: <infrastructure_id>-<role>
    spec:
      lifecycleHooks: {}
      metadata: {}
      providerSpec:
        value:
          apiVersion: machine.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 120
          kind: VSphereMachineProviderSpec
          memoryMiB: 8192
          metadata: {}
          network:
            devices:
            - addressesFromPools: (1)
              - group: ipamcontroller.openshift.io
                name: static-ci-pool
                resource: IPPool
              nameservers:
              - "192.168.0.1" (2)
              networkName: qe-segment-204
          numCPUs: 4
          numCoresPerSocket: 2
          snapshot: ""
          template: cluster-dev-9n5wg-rhcos-generated-region-generated-zone
          userDataSecret:
            name: worker-user-data
          workspace:
            datacenter: datacenter
            datastore: /datacenter/datastore/vsanDatastore
            folder: /datacenter/vm/cluster-dev-9n5wg
            resourcePool: /datacenter/host/cluster/Resources
            server: test-vcenter.com

Note the following references in the file above:

• (1) - A reference to an IPPool. An IPPool reference may refer to a registered CRD or could simply be a reference that is recognized by the administrator or controller handling IPAddressClaims. IPPools allow a machineset to specify a configuration to be used by the administrator or controller when allocating an IP Address.
• (2) - IPAddress resources do not define nameservers. As such, at least one nameserver should be provided.

2. Scale up a machine:

oc scale machineset <infrastructure_id>-<role> -n openshift-machine-api --replicas=1

3. After scaling up a machine, an IPAddressClaim will be created by the machine API controller. To check for IPAddressClaims:

$ oc get ipaddressclaims.ipam.cluster.x-k8s.io -n openshift-machine-api
NAME                                           POOL NAME        POOL KIND
cluster-dev-9n5wg-worker-0-m7529-claim-0-0   static-ci-pool   IPPool
cluster-dev-9n5wg-worker-0-wdqkt-claim-0-0   static-ci-pool   IPPool

Below is an example of the structure of an IPAddressClaim:

kind: IPAddressClaim
metadata:
  finalizers:
  - machine.openshift.io/ip-claim-protection
  name: cluster-dev-9n5wg-worker-0-m7529-claim-0-0
  namespace: openshift-machine-api
spec:
  poolRef:
    apiGroup: ipamcontroller.openshift.io
    kind: IPPool
    name: static-ci-pool
status: {}

4. Allocate an IP address and create an IPAddress to bind to the IPAddressClaim.

Below is an example of the structure of an IPAddress:

apiVersion: ipam.cluster.x-k8s.io/v1alpha1
kind: IPAddress
metadata:
  name: cluster-dev-9n5wg-worker-0-m7529-ipaddress-0-0
  namespace: openshift-machine-api
spec:
  address: 192.168.0.129
  claimRef: (1)
    name: cluster-dev-9n5wg-worker-0-m7529-claim-0-0
  gateway: 192.168.0.1
  poolRef: (2)
    apiGroup: ipamcontroller.openshift.io
    kind: IPPool
    name: static-ci-pool
  prefix: 23

• (1) - The name of the associated IPAddressClaim
• (2) - The reference to the IPPool associated with the IPAddressClaim

oc create -f ipaddress.yaml

5. Update the IPAddressClaim status with a reference to the IPAddress. Once an IPAddress has been created, the IPAddressClaim must be updated to include a reference to the IPAddress.

To do this with the oc client:

oc --type=merge patch IPAddressClaim cluster-dev-9n5wg-worker-0-m7529-claim-0-0 -p='{"status":{"addressRef": {"name": "cluster-dev-9n5wg-worker-0-m7529-ipaddress-0-0}}}' -n openshift-machine-api --subresource=status

Wrap up

By enabling static IP addresses for vSphere IPI installations, administrators now have more control over how OpenShift nodes are deployed. We are interested in hearing your thoughts on this feature. Please let us know in the comments.