Introducing Mirror Registry for Red Hat OpenShift

The Red Hat Quay team and Red Hat OpenShift team are excited to announce the general availability of the mirror registry for Red Hat OpenShift. This is a small, streamlined container image registry that provides users the ability to bootstrap their first OpenShift cluster in an offline network environment where no image registry is present. Look for a demo below.

Supporting disconnected installations

In some environments where Red Hat OpenShift is deployed, no direct network access is available to the internet. Fortunately, Red Hat OpenShift supports such disconnected environments in various flavors, ranging from situations where internet access is available but only via an HTTP(S) proxy, to setups where there is no routed connection to the public services like registry.redhat.io or registry.access.redhat.com, all the way to environments that have a physical air-gap between systems with and without internet connection and rely on physical, portable media for data transfer.

In the latter two scenarios a private container image registry is required to be used with the oc CLI utility to mirror Red Hat OpenShift core images and OpenShift operator catalogs. This usually looks something like the below:

Sometimes however, such a private registry is not yet available, because there is an intent to run it on top of Red Hat OpenShift, for example Red Hat Quay via the Quay Operator. To break the dependency loop between Red Hat OpenShift and container registries users have resorted to running a small, custom private registry on a separate node in the disconnected / air-gapped environment and using it to host the initial image mirror. From there, the first cluster would be installed which runs the production Quay registry. A more complete mirror would be created on this registry and used in turn, to install other clusters in the disconnected environment.

The mirror registry for Red Hat OpenShift now provides a supported option to run this small bootstrap registry for the first cluster and greatly simplifies the deployment of the registry on a separate Red Hat Enterprise Linux (RHEL) 8 host. This results in the following setup:

The above illustration is based on internet connectivity on the RHEL system that hosts the mirror registry, an often encountered setup where a bastion host has direct internet access whereas Red Hat OpenShift clusters do not. Users with an air-gap in their network environment can still use the same setup while relying on the existing functionality of the oc CLI to first mirror content from public Red Hat registries to portable storage as explained in the product documentation.

Standing on the shoulders of giants

The mirror registry for Red Hat OpenShift is built on top of the Red Hat Quay technology. It deploys a small, minimal Quay configuration on a single node that leverages a local database and local filesystem storage. It is deployed with a fully self-contained installer, that has no dependencies on internet connectivity and thus, can also be used behind an air-gapped network. Internally the installer is making use of proven tools like podman and Ansible to carry out the installation on a local or remote system.

The install procedure has been reduced to a single command for both installation and uninstallation.

In its simplest form it looks like this:

$ ./mirror-registry install --quayHostname $(hostname -f)

This would deploy the mirror registry on the local machine using the default settings, which create a container registry using the system’s hostname in long form with self-signed TLS certificates for the HTTPS endpoint of the registry and image layer blobs stored at /etc/quay-install. Credentials for initial users will be auto-generated and reported when the install completes.

Further customization can be done with regards to the storage location, TLS certificates and the initial users passwords and the entire deployment can also target a different host, like in the following example:

$ ./mirror-registry install --quayHostname mirror.example.org \
                            --sslKey tls.key                  \
                            --targetHostname internal.mirror  \
                            --quayRoot /var/mirror-registry   \
                            --initPassword changeme

This is a more realistic example as users with internal deployments will likely want to leverage an internal certificate authority. When this installation completes, a minimal Red Hat Quay deployment will be running at https://<quayHostname>:8443 with the UI being available, an initial user called “init” pre-created and the registry ready to be used. The registry itself is run in containerized form which can be seen when listing running podman pods:

$ sudo podman pod ps
POD ID        NAME        STATUS      CREATED     INFRA ID      # OF CONTAINERS
f6ef36b6f539  quay-pod    Running     3 days ago  5410dc4127aa  4

Podman pods are a way of grouping running containers into a common management construct, much like pods on Kubernetes. This way the database for Quay and the Quay application container are run in a shared networking namespace. The running containers can also be inspected using podman ps.

The installer also configures the registry to auto-start using systemd units:

$ sudo systemctl list-units --type service | grep quay
quay-app.service                       loaded active running Quay Container
quay-postgres.service                  loaded active running PostgreSQL Podman Container for Quay
quay-redis.service                     loaded active running Redis Podman Container for Quay

None of these components need additional configuration or maintenance. The entire construct can also be removed with a single command:

$ ./mirror-registry uninstall

…or…

$ ./mirror-registry uninstall --quayRoot /opt/quay

… if the mirror registry was installed in a non-default location (e.g. /opt/quay)

…or…

$ ./mirror-registry uninstall --targetHostname internal.mirror

…if the mirror registry was deployed on a remote machine.

As soon as the mirror registry is running, the oc CLI utility can be used to start mirroring content that is required for installing Red Hat OpenShift, either from local disk storage or from public Red Hat registries into the mirror registry. See below for an example of a OpenShift 4.9.13 release being downloaded into the mirror registry for Red Hat OpenShift:

A full walk-through of installing and using the mirror registry to mirror an OpenShift release can be found in this demo.

Requirements and support scope

To run the mirror registry for Red Hat OpenShift a single RHEL 8 (x86_64) system with 2 vCPUs and 8GB of RAM, podman greater or equal version 3.3 and sufficient disk space will be required. While mirroring a single OpenShift Container Platform release takes up around 12GB of storage, mirroring the official Red Hat OpenShift Operators catalog (containing Red Hat OpenShift Service Mesh, Pipelines, GitOps and others) will consume more than 350GB for the whole catalog. This can be somewhat reduced by selecting only specific operators to be mirrored which is explained in the product documentation.

The intended purpose is solely the process of mirroring Red Hat OpenShift images, whether it’s core images or operator catalogs or the default ImageStreams or the OpenShift CLI tools. Therefore the advanced features of Quay like image scanning, image mirroring, clustering or geo-replication are not available. Users may also use it to store a limited set of custom monitoring agents. Storage of custom applications or other images used in production clusters is explicitly not supported. This is not due to technical limitations or size constraints but more so due to the limited availability of the mirror registry.

By the nature of a single host running the registry, the availability of the mirror registry for Red Hat OpenShift is tied to the availability of this host. It should be noted that clusters that are connected to the mirror registry will suffer operational degradation when the registry is down or images inside are not available for other reasons. This degradation is due to the fact that the underlying Kubernetes platform requires container images to be present to perform a cold boot (starting the Kubelet), adding additional nodes or putting nodes into maintenance and thus draining the containers they are running to the rest of the cluster. If those remaining cluster nodes do not have the required images in their local caches, they will attempt to reach the mirror registry to run them. If the mirror registry is unavailable for some reason this will fail and the workloads in question cannot be scheduled, effectively causing downtime for workloads deployed from the image registry.

Therefore the mirror registry should only serve images that can easily be redownloaded should the storage be unrecoverable. The mirror registry for Red Hat OpenShift should ideally only be used to serve a single, central infrastructure cluster, that runs no business applications but instead a central, highly available registry, like Red Hat Quay, which serves the rest of the disconnected cluster fleet and is in itself running across multiple nodes.

Users might be able to mitigate some of these single deployment characteristics and increase uptime of the mirror registry by employing additional infrastructure-level redundancy like highly-available block storage, virtual machine live-migration and automatic restarts should the underlying host fail.  

For use cases where a registry with high fault-tolerance without a single point of failure is required even before the first cluster is running, the fully featured Red Hat Quay product should be used and deployed on virtual machines behind a load balancer. Red Hat Quay is also already part of Red Hat OpenShift Platform Plus.

How to get it

The mirror registry for Red Hat OpenShift can be downloaded at no additional cost from the OpenShift downloads page. It is supported as part of a Red Hat OpenShift support subscription, regardless of the flavor of OpenShift or the underlying infrastructure. The mirror registry for Red Hat OpenShift supports all currently supported versions of OpenShift across all architectures. The ability to update an existing deployment of the mirror registry will be made available soon with the next release.