Improved UX Scales: OpenShift Update Service with Advanced Cluster Management

In a restricted network environment, updating OpenShift is a different experience because the Cluster Version Operator (CVO) normally uses an internet connection to get directions for upgrades. Without an internet connection, the CVO relies on the user to direct the cluster upgrade by gathering graph data and running some command line operations. This works fine when tracking a few clusters, but it becomes more tedious as more clusters are added and as versions drift apart.

The recent release of the OpenShift Update Service unlocks the ability for admins to take control over the cluster’s update path from the UI. An admin can launch the OpenShift Update Service (OSUS) from the UI, point the OpenShift Update Service at a registry with mirrored OpenShift releases, configure the Cluster Version Operator (CVO) to reach the OSUS route, and voilà. This greatly improves user experience for clusters in restricted networks, however horizontally scaling clusters into the 100s will still be challenging to update, as 100s of UI visits start to add up.

The Advanced Cluster Management for Kubernetes is a multi cluster management solution that unifies cluster management into a single UI. Combining the OpenShift Update Service with Advanced Cluster Management provides administrators the tools necessary to horizontally scale clusters while maintaining a smooth management experience.

Setup the OpenShift Update Service

The Advanced Cluster Management uses a management cluster (Hub Cluster) and managed cluster(s) (Spoke Clusters). There needs to be a place to run the OpenShift Update Service, and the Hub Cluster is a good fit because it will be responsible for overseeing the management operations of the Spoke Clusters. In addition, the Hub and Spoke clusters can connect to one another, so the OpenShift Update Service will advertise an endpoint from the Hub and all the Spoke Cluster CVOs will gather graph data from the endpoint.

1. Mirror OpenShift releases to the Hub container image registry
2. Build the graph data init container and publish it in the Hub container image registry
   

   FROM registry.access.redhat.com/ubi8/ubi:8.1
   
   RUN curl -L -o cincinnati-graph-data.tar.gz https://github.com/openshift/cincinnati-graph-data/archive/master.tar.gz
   
   CMD exec /bin/bash -c "tar xvzf cincinnati-graph-data.tar.gz -C /var/lib/cincinnati/graph-data/ --strip-components=1"

   $ podman build -f ./dev/Dockerfile -t registry.example.com/graph-data-image:latest
   
   $ podman push registry.example.com/graph-data-image:latest

3. Locate the OpenShift Update Service in the UI and start the operator on the Hub Cluster
   1. Set the ‘repository’ parameter to point at the container image registry on the Hub Cluster
   2. Set the ‘graphDataImage’ parameter to the location where you pushed the graph data init container
      

      apiVersion: cincinnati.openshift.io/v1beta1
      kind: Cincinnati
      metadata:
        name: example
        namespace: example-namespace
      spec:
        replicas: 1
        registry: "hub.registry.example.com"
        repository: "openshift-release-dev/ocp-release"
        graphDataImage: "hub.registry.example.com/graph-data-image:latest"

       

4. Extract the route to pass to the spoke clusters:

   ROUTE_URI="$(oc -n example-namespace get route example-policy-engine-route -o jsonpath=https://'{.spec.host}'/api/upgrades_info/v1/graph)"

   1. Note: You may also need to provide a custom default certificate on the spoke clusters, so the Cluster Version Operator doesn’t fail on the route’s TLS termination.
5. Poll until the operand service is ready:

   curl --header 'Accept:application/json' "${ROUTE_URI}"

Configure Spoke Clusters

Now that the OpenShift Update Service is installed and configured on the Hub, configure the Spoke clusters to contact the OpenShift Update Service endpoint for graph data. ACM provides the ability to apply changes across multiple clusters with Policies and PlacementRules:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
 name: policy-graph-container
 namespace: default
 annotations:
   policy.open-cluster-management.io/standards: NIST-CSF
   policy.open-cluster-management.io/categories: PR.IP Information Protection Processes and Procedures
   policy.open-cluster-management.io/controls: PR.IP-1 Baseline configuration
spec:
 remediationAction: enforce
 disabled: false
 policy-templates:
   - objectDefinition:
       apiVersion: policy.open-cluster-management.io/v1
       kind: ConfigurationPolicy
       metadata:
         name: role-policy-musthave
       spec:
         remediationAction: enforce
         namespaceSelector:
           exclude: ["kube-*"]
           include: ["default"]
         object-templates:
           - complianceType: musthave
             objectDefinition:
               apiVersion: config.openshift.io/v1
               kind: ClusterVersion
               metadata:
                 name: version
               spec:
                 channel: stable-4.4
                 clusterID: exmaple_cluster_id
                 upstream: https://hub.updateservice.com/api/upgrades_info/v1/graph

apiVersion: policy.open-cluster-management.io/v1
kind: PlacementBinding
metadata:
 name: binding-graph-container
 namespace: default
placementRef:
 name: placement-policy-graph-container
 kind: PlacementRule
 apiGroup: apps.open-cluster-management.io
subjects:
- name: policy-graph-container
 kind: Policy
 apiGroup: policy.open-cluster-management.io

apiVersion: apps.open-cluster-management.io/v1
kind: PlacementRule
metadata:
 name: placement-policy-graph-container
 namespace: default
spec:
 clusterConditions:
 - status: "True"
   type: ManagedClusterConditionAvailable
 clusterSelector:
   matchExpressions:
     - {key: name, operator: In, values: ["spoke-cluster-name"]}

Check the Cluster Version Operator’s spec on a Spoke Cluster to verify the change:

$ oc get clusterversion version -o yaml | grep ' upstream:' -A 3 -B 3
spec:
  channel: stable-4.5
  clusterID: 5fc2ade1-c2f3-4e2b-aaf5-eb89da975347
  upstream: https://hub.updateservice.com/api/upgrades_info/v1/graph
status:
  availableUpdates:
  - force: false

Navigate to the Advance Cluster Management UI instance and your spoke clusters will show if there’s an Upgrade available. Use the checkbox to select multiple clusters to upgrade at once or upgrade them all individually:

Improved UX Scales

The improved user experience that the OpenShift Update Service provides when combined with Advanced Cluster Management make it so cluster updates in restricted networks can easily be triggered across the fleet with the click of a button. Administrators no longer need to run command line operations across multiple clusters or potentially lose track of cluster versions; everything can be UI driven, in one location, simplifying the overall upgrade experience.