OpenShift Update Service: Update Manager for Your Cluster

Background

Red Hat’s OpenShift Container Platform update service is the hosted service located behind public APIs that provides over-the-air updates to the OpenShift Container Platform, including its underlying Red Hat Enterprise Linux CoreOS (RHCOS) operating system. It performs this task by providing upgrade graph information to the cluster that shows the recommended versions you can safely update to.

For environments with internet accessibility, OpenShift clusters routinely connect to the hosted service to report the health of the cluster and check for updates. In the event an update is available, administrators are notified in the console and can quickly apply it with the push of a button. This process involves reporting cluster parameters to the hosted service, which in turn, provides an upgrade graph specific to the cluster containing a list of recommended update versions that can safely be applied.

For clusters in a restricted network, there’s no way to access these public APIs to obtain this information, so administrators must determine what the next recommended update version is by manually querying the hosted service. Only once this is known can the specific update payload be mirrored into its environment to enable the update. This approach isn’t particularly difficult for maintaining a few clusters, but can be laborious for environments with a fleet of mixed-version clusters. Selecting the next recommended update version for a group of clusters can be a complex task.

Introducing the OpenShift Update Service

In order to get the same user experience when applying cluster updates in restricted network environments, Red Hat is making available the tools necessary to deploy the update service on-premise. This new service is called the OpenShift Update Service, and the initial 1.0 release is now available as a developer preview that can be deployed on OpenShift 4. It’s designed to provide a seamless, UI-driven upgrade experience for clusters residing inside a restricted network.

Update Graph

The OpenShift Update Service is built on the Cincinnati protocol, which uses a directed acyclic graph (DAG) to represent valid updates. Each node in the graph is a release payload, and each edge is a valid transition. The service is responsible for iterating over the set of releases and building a DAG that can be queried.

Release payloads contain a small JSON-formatted metadata document which declares information about the update payload. An example of this metadata document can be seen here:

$ oc adm release info -o json quay.io/openshift-release-dev/ocp-release:4.5.4-x86_64 | jq .metadata
{
  "kind": "cincinnati-metadata-v0",
  "version": "4.5.4",
  "previous": [
    "4.4.12",
    "4.4.13",
    "4.4.14",
    "4.4.15",
    "4.5.1",
    "4.5.2",
    "4.5.3"
  ],
  "metadata": {
    "description": "",
    "url": "https://access.redhat.com/errata/RHBA-2020:3028"
  }
}

If you would like to visualize the directed acyclic graph, a simple script (leveraging Graphviz DOT) is available for rendering an image from the JSON data:

$ curl https://raw.githubusercontent.com/openshift/cincinnati/master/hack/graph.sh > 
  ./graph.sh
$ chmod 755 graph.sh

$ curl -sH 'Accept:application/json' 
  'https://api.openshift.com/api/upgrades_info/v1/graph?channel=fast-4.5' | 
  ./graph.sh | dot -Tpng >graph-fast-4.5.png

Normally, upgrade graph information is pulled directly from Red Hat’s hosted service. However, the OpenShift Update Service hands over control of the upgrade graph data to the administrator. This allows the admin to control how the upgrade graph is built through edge pruning and which update payloads are found in the local container registry.

How It Works

The OpenShift Update Service works by building an upgrade graph from the metadata found inside the OpenShift release payloads hosted in the local container registry. Upgrade graph data is then made available to OpenShift clusters by configuring the Cluster Version Operator (CVO) to point to the on-premise OpenShift Update Service endpoint. Running this service in the restricted network allows updates to quickly be visible to the cluster once the release payload has been mirrored to local container registry. Administrators can then easily update clusters with the click of a button in the web console or from the CLI.

The OpenShift Update Service is comprised of two components:

1. Graph Builder: Fetches OpenShift release payload information (primary metadata) from any container registry (compatible with Docker registry V2 API) and builds a DAG representing valid upgrade edges.
2. Policy Engine: Applies cluster-specific filters to return customized available update graphs.

Deploying the OpenShift Update Service

To start using the OpenShift Update Service, you will need to ensure you have mirrored at least two OpenShift update release payloads (without multiple nodes connected by edges, there’s no benefit to running a local service) to your local container image registry. To perform this task, you can follow the process outlined in the OpenShift product documentation for mirroring OpenShift release payloads.

Graph data is typically fetched directly from the upgrade graph data repository. In environments where an internet connection is unavailable, loading this information from an init container is another way to make the graph data available to the OpenShift Update Service. The role of the init container is to provide a local copy of the graph data, and during pod initialization, the init container copies the data to a volume accessible by the service.

An example of how to build an init container can be found within the operator repository. In the example, the image takes a tarball of the upgrade graph data repository. When the init container runs, it untars the data to ‘/var/lib/cincinnati/graph-data’ for the OpenShift Update Service to use:

FROM registry.access.redhat.com/ubi8/ubi:8.1

RUN curl -L -o cincinnati-graph-data.tar.gz https://github.com/openshift/cincinnati-graph-data/archive/master.tar.gz

CMD exec /bin/bash -c "tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati/graph-data/ --strip-components=1"

Build and push the image to your own repository:

$ podman build -f ./dev/Dockerfile -t registry.example.com/graph-data-image:latest
$ podman push registry.example.com/graph-data-image:latest

The OpenShift Update Service operator will need to be obtained from Red Hat’s Operator Catalog before it can be deployed. This can be done by following the process in the OpenShift product documentation for mirroring content and configuring OLM and OperatorHub to use a custom Operator catalog image hosted in the restricted network.

Now deploy the OpenShift Update Service with two configuration changes:

1. Set the ‘repository’ parameter to point at your local container image registry where OpenShift release payload information can be scraped.
2. Set the ‘graphDataImage’ parameter to the location where you pushed the graph data init container.

$ oc edit cincinnati

apiVersion: cincinnati.openshift.io/v1beta1
kind: Cincinnati
metadata:
  name: example
spec:
  replicas: 1
  registry: "registry.example.com"
  repository: "openshift-release-dev/ocp-release"
  graphDataImage: "registry.example.com/graph-data-image:latest"

Once the service has been deployed and an ingress route has been associated with it, you will need to configure the Cluster Version Operator (CVO) on each OpenShift cluster before it will pull graph data from the on-premise service. This can be done by editing the ‘spec.upstream’ URL of the ClusterVersion CRD with the on-premise OpenShift Update Service route:

$ oc patch clusterversion version --type='json' -p='[{"op": "replace", "path": "/spec/upstream", "value": "https://osus.example.com/api/upgrades_info/v1/graph"}]'

The resulting change should look like this:

$ oc get clusterversion version -o yaml | grep ' upstream:' -A 3 -B 3
spec:
  channel: stable-4.5
  clusterID: 5fc2ade1-c2f3-4e2b-aaf5-eb89da975347
  upstream: https://osus.example.com/api/upgrades_info/v1/graph
status:
  availableUpdates:
  - force: false

Now that you have the OpenShift Update Service running on-premise, update notifications will appear in the OpenShift web console whenever an upgrade is made available and can be applied to the cluster with just the click of a button!

What’s Next

The OpenShift Update Service is planned for GA in OpenShift 4.6. Red Hat is also working on publishing containerized graph data to provide administrators with up to date information as graph edges and channels change.

In the next installment, we’re going to show how the user experience improvement scales in multicluster environments. When combined with the Red Hat Advanced Cluster Manager (ACM) product, upgrades can be triggered across the fleet with the click of a button.

Log any issues with the OpenShift Update Service here.

References:

https://github.com/openshift/cincinnati

https://github.com/openshift/cincinnati-operator
https://docs.openshift.com/container-platform/4.5

https://www.redhat.com/en/technologies/management/advanced-cluster-management