Background

OpenShift makes it easy to manage Kubernetes clusters in internet-connected environments using well-known registries. However, preparation for and maintenance of clusters reliant on mirror registries can be challenging. This task may not be tricky when managing proof of concept or lab deployments. However, it can be a heavy load in production when performing upgrades and other lifecycle operations of operator and platform components. The number of steps involved, lesser-known dependencies, and disparate configurations can lead to errors and frustration. Another challenge stems from the amount of disk space required to perform most of these tasks. With granular ways to filter or focus on newer data only, the size of the downloaded data can be massive and will only grow with time.

Introducing oc-mirror

Red Hat is making a utility to streamline the management of different content types, including OpenShift releases, operator catalogs, and user workloads. It is a plugin to the OpenShift CLI  called oc-mirror. oc-mirror is designed to provide users of OpenShift environments reliant on mirror registries with an experience comparable to internet-connected clusters and is now available as a technical preview for use with OpenShift 4 clusters starting at version 4.10.

How it Works

The oc-mirror plugin processes a configuration file called an imageset-config that defines a mirror registries’ contents. The two main outputs of oc-mirror are a mirror registry payload, called an imageset, and a metadata file used to store information about previously generated Imagests.

The plugin can run in either stateless or stateful mode. In stateless mode, oc-mirror will not retain the metadata for each imageset creation. However, we recommend running oc-mirror with a storage backend defined within the configuration file to ensure OpenShift release and operator upgradeability. The ability to calculate and maintain upgrade paths for OpenShift releases and operators are the features that constitute oc-mirror`s lifecycle management capabilities. The capability includes intelligently mirroring OpenShift releases and operators while maintaining upgrade paths from all previous component versions within a mirror registry. All release and operator component versions within a mirror registry should have an upgrade path to any higher version of that component within the mirror registry.

Now that we have established how to enable lifecycle management, we can dive into what those features entail. Three main components of oc-mirror lifecycle management are the following:

  • Operator Catalog Updates: To update operators, an update graph has to be maintained to ensure operator versions are updated in sequence. For these graphs to be maintained, the tools will determine what has already been mirrored compared to what is currently specified in terms of the desired (newer) version in the imageset configuration file. The operator catalog information from the previous run is recorded and compared with the current operator catalog to find the delta. This process ensures the imageset contains the new operator packages and all required intermediate releases to successfully update to the latest version of each selected operator.
  • OpenShift Cluster updates: Like with operator catalogs, oc-mirror during differential operations will ensure that an upgrade path is available to newer OpenShift release versions. The oc-mirror plugin records previously downloaded OpenShift release payloads to determine the required versions to complete the upgrade path to the requested version. 
  • Image Layer Differential: The oc-mirror plugin design eliminates the duplication of shared image layers between mirror synchronizations, allowing shared layers to be transferred to the mirror registry only one time. This process keeps the size of each generated Imageset as small as possible.

How to Use

This tutorial will show you how to create a configuration to pull an operator from the Red Hat OpenShift 4.9 operator catalog, mirror it to disk, and publish it to a remote container image registry.

The first step when using `oc-mirror` is to discover available content. This task will help to compose the imageset configuration in the next step. Below are some examples of how content is found for operator catalogs and OpenShift release payloads.

- Find the available catalogs for the target version:

oc-mirror list operators --catalogs --version=4.9

- Find the available packages within the selected catalog:

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9

- Find channels for the selected package:

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9 --package=nfd

- Find package versions within the selected channel:

oc-mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.9 --package=nfd --channel=stable

- Find OCP releases by major/minor version:

oc-mirror list releases --version=4.9

The next step is to create the Imageset configured with all the information gathered. An example configuration is below:

#imageset-config.yaml
apiVersion: mirror.openshift.io/v1alpha1
kind: ImageSetConfiguration
storageConfig:
local:
  path: metadata
mirror:
operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.9
    headsOnly: false
    packages:
      - name: rhacs-operator
        channels:
        - name: latest
        versions:
        - '3.67.0'

An in-depth description of the configuration specification is located here.

The following oc-mirror command will mirror the sources provided in the configuration to disk in the output-di directory:

oc-mirror --config imageset-config.yaml file://output-dir

Once the imageset creation finishes, transfer it to the disconnected environment.

The following command will unpack the Imageset archive and publish it to the target directory. In this example, we are using a local registry with TLS disabled.

oc-mirror --from ./output-dir/mirror_seq1_000000.tar docker://localhost:5011 --dest-skip-tls

Now that we have published our first Imageset, we can go back to the beginning of the process and create a differential update.

The configuration below is an example of an updated configuration file with an added package for the operator catalog.

#imageset-config.yaml
apiVersion:mirror.openshift.io/v1alpha1
kind: ImageSetConfiguration
storageConfig:
local:
  path: metadata
mirror:
operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.9
    headsOnly: false
    packages:
      - name: rhacs-operator
        channels:
        - name: latest
        versions:
        - '3.67.0'
# Added package below
      - name: submariner

 We can create and publish the Imageset as we did before. Notice the name of the archive now has the number two. This value will increment with each run if the `storageConfig` is set in the `imageset-config.yaml`. 

oc-mirror --config imageset-config.yaml file://output-dir
oc-mirror --from ./output-dir/mirror_seq2_000000.tar docker://localhost:5011 --dest-skip-tls

Like this, the desired mirror content can be declared in the imageset configuration file statically while the mirror jobs are executed regularly, for example as part of a cron job. This way, the mirror can be kept up to date in an automated fashion.

What's Next

oc-mirror is planned for GA with OpenShift 4.11. Planned features include usability enhancements like container image registry pruning and direct stream support for a mirror to mirror transaction.

Ready to learn more? Try out oc mirror by downloading it from the Red Hat Hybrid Cloud Console.  If you are interested in the implementation and in contributing, check out the git repository.


About the author

Jennifer Power joined Red Hat in 2021 as a Solution Architect for the North America Public Sector. She brings over eight years of experience working in IT services for the public sector and has a Bachelor’s degree in Computer Science from Old Dominion University. Jennifer is passionate about learning new technologies and is focused on contributing to the open source community to benefit the public sector and regulated industries.

Read full bio