Faster Onboarding for New Infrastructure Providers with Red Hat OpenShift’s External Platform

OpenShift users have many options when it comes to deploying a Red Hat OpenShift cluster. Whether installing on-premises, in the cloud, or to your own hybrid configuration, OpenShift has integrations to help your journey. Today we are pleased to announce a new platform type to enable partners to achieve varying levels of integration not previously possible in OpenShift: the external platform.

What is the External Platform?

The external platform is a new configuration for OpenShift which allows deep infrastructure customization. Whereas integrated platforms such as AWS, GCP, or OpenStack install components specific to those providers, the external platform provides partners the opportunity to run their own components for infrastructure management. What this means is adding cloud provider components to OpenShift is becoming easier.

Previously, implementing new platform types in OpenShift required the addition of components directly to the source code and release artifacts. With the previous approach, the integration efforts would take an entire release cycle or more to complete. With the external platform, this process no longer requires any additions to the OpenShift source code, and partners have control of their own components for release and lifecycle management. So, the external platform provides partners a self-service approach for adding infrastructure components.

How Does External Platform Make Integration Easier?

When installing OpenShift on one of the included integrated cloud providers, such as Azure, Nutanix, and vSphere, infrastructure related components are installed automatically by the OpenShift tooling. These components can include Cloud Controller Managers, Container Storage Interface drivers, and Machine API providers, which perform provider-specific actions to enable specific infrastructure behavior for Kubernetes. Using the external platform informs OpenShift that the infrastructure layer components may be replaced, thus OpenShift should make the necessary adjustments for those new components to provide that functionality.

In some respects, the external platform cluster looks similar to the agnostic (a.k.a. none) platform cluster in that no infrastructure related components are installed by default. The difference is in the details. When you install to an external platform, OpenShift provides configuration options which allow the user to inform OpenShift about the components that will be installed. The configuration options are then expressed in the cluster in the form of changes to the internal behaviors of core infrastructure operators. For example, when you configure an external platform cluster to utilize cloud controller managers, OpenShift will know to deploy the kubelets with the necessary flags to interact with those managers.

Red Hat OpenShift on Oracle Cloud Infrastructure

An example of how we’re using external platform is enabling OpenShift on Oracle Cloud Infrastructure (OCI). In partnership with Oracle, we’ve made it easier for our customers to install OpenShift on OCI using the external platform to deliver a solution to enable our joint customers to build best of breed cloud native solutions and to accelerate their application modernization journey.

Enabling cloud native Oracle Cloud Infrastructure storage

To understand how we’ve made it easier to install OpenShift on OCI using the external platform, you need to understand how container storage interface drivers (CSI) work, and how they interact with regional and zonal information on Kubernetes nodes. The CSI standard has been available in Kubernetes since the 1.13 release. CSI drivers helped push the Kubernetes community forward by enabling cloud storage interfaces to be added as a “day 2” operation to most clusters. This means that cloud storage providers can now create and manage their own drivers without the need to merge code directly into the Kubernetes core.

When operating a CSI driver, node and persistent volume resources in Kubernetes must be labeled with the zone and region where they physically exist to provideproper scheduling on infrastructures that use geographical awareness. The labeling operation is usually performed by the cloud controller manager (CCM). Operating a CCM within a Kubernetes cluster requires some knowledge about the deployment, and the privileges to run that controller such that it can read and modify node objects. On OpenShift, CCMs are managed by an operator to configure permissions and monitor the running controllers for failures.

Although CSI drivers can be deployed on OpenShift through the operator catalog or by user interaction, the same cannot be said about CCMs. There are changes to the core Kubernetes command line flags which must be expressed before the system will acknowledge the CCMs. The external platform feature tells OpenShift to recognize partner CCMs and configure Kubernetes appropriately. Utilizing the CCM feature of the external platform, Oracle is able to deploy OpenShift on OCI using their own CCM, thus enabling the proper operation of their CSI driver as well. The end result is that clusters deployed on OCI using the external platform are able to access the full benefits of the native OCI storage.

Getting Started with External Platform

The process for creating an external platform cluster is similar to the instructions for Installing a cluster on any platform. To begin, create the necessary infrastructure for installing OpenShift as described in the product documentation.

Before running the “openshift-install” command, manually create the installation configuration file. After creating this file, update the following fields to indicate the external platform:

platform:
  external:
    platformName: "providerName"

The “platformName” field needs to be a string to identify the provider platform – for example “OCI” on Oracle Cloud Infrastructure.

Next you will need to create the installation manifests. Creating these manifests provides an opportunity to enable or disable support for cloud controller managers. Create the manifests by running the following command:

openshift-install create manifests

This command will consume the “install-config.yaml” file and create two directories: “manifests”, and “openshift”. In the “manifests” directory you will find a file named “cluster-infrastructure-02-config.yml” which contains the configuration information for the cluster.

If you are planning to run your own CCM, you will need to modify the “cluster-infrastructure-02-config.yml” to contain the following fields:

status: 
  platformStatus:
    external:
     cloudControllerManager:
      state: External
  type: External

If you do not plan to run your own CCM, the fields should look like this:

status: 
  platformStatus:
    external:
     cloudControllerManager:
      state: None
  type: External

After modifying the “cluster-infrastructure-02-config.yml”, you are nearly ready to run the installation. Note that if you have enabled the use of CCMs, you will also need to manage the deployment of those pods within OpenShift. You can do this by adding a manifest to the “manifests” directory before running the installation. Once you have added any extra manifests you will need, you are ready to run the installation.

You can confirm the external platform installation has succeeded by querying the infrastructure  configuration object in your new cluster.

oc get infrastructure/config -o yaml

What’s Next?

The external platform was initially released in OpenShift 4.13 and OKD 4.13. OpenShift 4.14 adds the capability to add a partner’s cloud controller manager when the external platform is configured. We are actively documenting how to use this new platform type and will provide more examples of how it can be utilized. More continuous integration testing will be added to exercise this platform type across a wide variety of infrastructures.

Through the external platform, we invite and encourage partners, cloud providers, and platform operators to explore the possibility of OpenShift cluster installations that utilize infrastructure-aware components without the need for deep integrations.