Red Hat SummitRed Hat blog
Introduction
Containers, whether running in a system like Red Hat OpenShift, or on a container host like Red Hat Enterprise Linux (RHEL), are instantiated from container images. Generally, rather than directly run base images for a Linux distribution, we build custom images by taking these base images and adding extra files, configuration, etc. on top.
Keeping these custom images up to date as bug fixes and security patches are released is an ongoing process that requires careful thought and system design.
In this article, we will discuss how we detect when a custom image we have created needs updating. These techniques are generally applicable to most container image build scenarios, but we are going to focus on the RHEL and Universal Base Image (UBI) container images, which allows us to take advantage of some extra options because they are hosted by Red Hat.
We will not discuss how to detect whether an image, taken as-is from a container registry, needs to be updated and restarted. This sort of facility should be provided by container tools, as outlined in the Red Hat Enable sysadmin article How to use auto-updates and rollbacks in Podman.
What is the Red Hat Universal Base Image (UBI)?
Red Hat Universal Base Images (UBIs) are Open Container Initiative (OCI)-compliant container base operating system images with complementary runtime languages and packages that are freely redistributable. They are built from portions of RHEL. UBI images can be obtained from the Red Hat container catalog and be built and deployed anywhere.
What makes the UBI different from a RHEL base image?
The UBI offers a number of base images, based on RHEL bits. We can enhance them with a set of packages available from a special UBI package repository. So long as any derived images stay within these rules (UBI base image, packages from UBI repository), they are freely redistributable (unlike those based on RHEL base images), and we can run them on any Linux container host.
That said, in that situation, they would be unsupported by Red Hat. To take advantage of Red Hat's commercial support, we need to run these images on a container host from Red Hat, either RHEL or OpenShift.
If we need a package from the wider RHEL ecosystem, such as MySQL or PostgreSQL, these are not available from the UBI repositories. Rather, we will need the RHEL repositories, and at this point, the image comes under the RHEL terms and conditions. Thus, it must only be built and run on a RHEL or OpenShift system.
How do I know if a base image has been updated?
First, a 101 note: If we are using a common pattern to create our container images, such as bringing in the UBI as a base image, enhancing it with some corporate standards, and then applying an application-specific customization, we should almost certainly create an intermediary image specification.
This can reduce both container image storage and load on build servers. Additionally, we can start to apply good practices around versioning, governance, and others, which are the same good practices already applied for operating system maintenance, to these intermediary images.
When it comes to maintaining these custom images, we need to detect when the “base image” has been updated. This can be done in a variety of ways, but it should be no surprise that automation is the most reliable and scalable option.
Manually checking for updates and re-running builds
No human wants to do this, and even if they did, we could not trust that they were checking in a timely manner, correctly identifying rebuild triggers, or correctly issuing rebuild instructions.
These are jobs for a robot, so let's pretend we never even saw the option.
Email notification from Red Hat Errata repositories
Red Hat customers can receive email notifications of security updates, bug fixes, and enhancements, also known as errata.
This needs careful filtering to avoid it turning into a firehose. To adequately filter, we would either need to invest engineering time to automate it, or use human analysis (see above). Thus, the best we can do is wire a Continuous Integration/Continuous Delivery (CI/CD) system to the email notification, and kick off all rebuilds when we receive an email.
This seems like a very blunt instrument indeed. We need something a bit more targeted.
Polling
We can identify base image changes by polling a container image registry for changes. When we detect a change, we can issue a rebuild.
We mostly build custom images using a CI/CD pipeline. So this update case is the same as our usual image build process. What we are talking about is detecting when to build, and kicking that off.
The great news is that most CI/CD systems have the ability to run jobs periodically. Usually these can be set to daily, hourly, etc. intervals, whatever is appropriate for your needs. Similarly, we can trigger OpenShift Pipelines (Tekton) using an OpenShift cron job.
The first step in our polling pipeline will be to grab the latest upstream image and detect whether there has been a change that warrants a rebuild of our custom image. If no change is detected, the pipeline will finish.
If a change is detected, then the custom image should be rebuilt, appropriately tagged, and uploaded to the internal corporate registry.
From here, the tools that are running instances of the custom image (OpenShift, podman, etc.) can detect the new image and restart the container.
Image streams on OpenShift
Image streams on the OpenShift platform can be used to store images and to initiate OpenShift build and deployment operations. An image stream can import images from an external registry, such that teams can combine the new image identification process described above with a simple way to get the new image into the image stream. In essence, if the skopeo inspect process identifies a new image, then the command below can be used to add the image to the OpenShift image stream:
oc import-image <imagestreamtag> --from=<image> --confirm
The presence of the new image in the image stream can trigger a rebuild process on OpenShift, and it can trigger a redeployment, too. The command shown below can be used to configure an automatic re-deployment of an application (and hence use the new image) when an image change occurs:
oc set triggers deployment/<deployment-name> --from-image <image-stream> -c <container>
The use of image streams for triggering build and deployment processes should not be overlooked. It is very easy for teams to quickly jump to external build and deployment processes, because that is what has been done for years. However, there are some excellent capabilities built into the OpenShift platform, particularly for the fast moving development environment when teams need a chain of operations that will rebuild the right assets at the right time.
What if we are not using OpenShift image streams?
We need to take advantage of tools such as skopeo to examine the images in the external container registry. The `skopeo inspect` command is particularly useful as it performs a metadata examination of a container image stored in a registry without actually pulling any images. For example, the command shown below will list all current image tags associated with the image:
skopeo inspect docker://registry.access.redhat.com/ubi8
To detect a change, we need detailed version information about the container images that we are using. The upstream base UBI images all have a specific version identifier in a label called ‘url’ as shown below:
skopeo inspect docker://registry.access.redhat.com/ubi8:latest
{
"Name": "registry.access.redhat.com/ubi8",
"Digest": "sha256:f85554c06bf8f4d712a25d4c0373491f7c7437c578a434c8111c6dfce738559a",
"RepoTags": [
"8.4-213",
"8.4-211",
<all tags available are listed here>
"latest"
],
"Created": "2022-05-03T08:39:27.737951Z",
"Labels": {
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754",
So we can see from the information above that the current ‘latest’ upstream image is actually version 8.6-754.
Our container build pipeline can easily grab this version identifier and make sure that it is backed into our new container image as a label, using (suggested) the identifier ‘madeFrom’.
Many container images in a registry have a moving ‘latest’ tag (such as shown above) that is always applied to the most recent image. If we adopt the same principle in our custom image, then the same skopeo inspect command could be used to find the following:
skopeo inspect docker://private-registry/custom-image-java:latest
{
"Name": "registry.access.redhat.com/ubi8",
"RepoTags": [
"1.0",
"1.1",
<all tags available are listed here>
"latest"
],
"Labels": {
"madeFrom": "8.6-752",
As shown above, our latest custom image was built from a slightly older version than the current latest upstream image and so a rebuild is required.
What else do I need to think about?
If a new image is found, and we need to copy the image to a private registry, then the `skopeo copy` command can be used to perform that task. Storing a separate copy of the base image enables an organization to perform immediate vulnerability analysis prior to making any changes or additions of content.
Some organizations will want to take a more cautious approach to the ingestion of new container images, at least in the infancy of a process. In these cases, our pipelines do not need to immediately rebuild images. Instead, pipelines could add a human task to approve a rebuild, perhaps by integrating with an external platform. With OpenShift Pipelines, the process could look like this:
- Execute an hourly cron task that initiates a Tekton process via a webhook.
- The Tekton process uses Skopeo to check the remote repository and identify any new container image tags.
- If a new tag is found, the Tekton process will raise a ServiceNOW ticket including details of the new image.
- An appropriate individual reviews and approves the ServiceNOW ticket.
- Approval of the ServiceNOW ticket triggers a second Tekton pipeline, pulling the container image to a local registry.
- The appearance of the image in the local registry will trigger an image rebuild process (described below in the section on Red Hat Quay).
We could augment the above process with image vulnerability scanning at appropriate stages, as described here.
Notifications
Polling works, but it is hardly the most efficient or timely way of determining if there are changes to a base image. If we are not careful, then the image registry may start to throttle our use. Ideally, we would like to be told when something is new. For this, notification mechanisms, like webhooks, are the best solution. Ideally, the image repository should tell us.
Sadly not all image repositories contain functionality like this. An exception is Red Hat Quay.
Red Hat Quay
If we use Red Hat Quay to store images for our company, including keeping a cached version of base images we use, then we can use this to monitor upstream registries by using Quay repository mirroring. In this scenario, Quay acts as both the monitor and the trigger.
We can configure Quay repository mirroring to synchronize at a specified interval, allowing some repositories to be checked on a more aggressive schedule than others. A filter can be used to identify a specific subset of tags; this allows us to keep the number of images under control.
Once we have a mirrored repository, we can instruct Quay to fire off a webhook notification when there is a new version of a container image. This could target any CI/CD system that supports webhook triggers. OpenShift Pipelines has this facility, covered in Mark’s 2020 article Guide to OpenShift Pipelines Part 6 - Triggering Pipeline Execution From GitHub.
What about when we add extras from external sources?
The vast majority of images derived from the UBI (and from any base image) will customize the base image. These could include the addition of runtime languages (for example, Python3), servers (for example, httpd) or other files. Though these will often take the form of YUM packages downloaded from a repository, they could also be brought in from another third-party location.
Red Hat does not push new UBI base images when optional packages (within the YUM repositories, but not included in the base image) are updated, so we need to monitor the repositories we use, too.
Hypothetically, if we were able to get a generic "repository updated" notification/hook, or a specific "package updated" notification/hook, we would need to apply some fairly significant engineering work to build, maintain, and test (!) a system to detect if the specific packages that an image depended on were updated.
With all this engineering effort, we might be able to cover all the external sources, and set up notifications for all of them. But we can never be sure we have received a notification, even if it was sent.
Thus, the safest, and conceptually simplest, solution is to simply rebuild our derived images and see if there are any changes. We can do this on a schedule.
Happily, this covers the general case: both YUM repositories, and any other files brought in from third-party locations. Sadly, it does come at the cost of "unnecessarily" building these images when no changes are available.
Fortunately, Red Hat *does* provide a set of customized container images that include commonly used software stacks. These are designed to help avoid this periodic rebuild in a good proportion of cases.
Downstream Red Hat container images
Red Hat keeps more than the base RHEL and UBI images in its registry. The Red Hat container registry contains customized images for many commonly used software stacks.
Application Stream container images
These are a set of images built from the base UBI image and augmented with packages from the UBI YUM repositories. They cover the most common use cases for the UBI, including but not limited to:
- NGINX (for example, ubi8/nginx-*)
- Apache HTTPD (for example, ubi8-httpd-24)
- Node.js (for example, ubi8/nodejs-*)
- Python (for example, ubi8/python-*)
- OpenJDK (for example, ubi8/openjdk-*)
Red Hat pushes new versions of these images when either the base image, or packages included in the image, are updated. Rather than recreating them ourselves, these images should be used when possible as the basis for our custom images. If used, all the methods for UBI-based detection and rebuilds apply here as well.
Red Hat middleware
Similarly, for products like Red Hat JBoss EAP, Red Hat provides images that take the UBI base image and add the middleware packages, configuration variables, etc. for getting the software up and running in a containerized manner. We should use these as-is whenever possible. If we have to customize them, it is likely rebuilding the custom image will be an inexpensive exercise.
Conclusion
We can spend a lot of time creating polling and notification architectures for determining if base images, packages, and other external dependencies we need for building our custom images have changed.
In reality, though, apart from in the simplest of scenarios, we cannot comprehensively cover all the possible sources of change with asynchronous notifications, or even metadata-oriented polling. Nor can we be sure we have not missed an edge case if we write filtering logic to reduce unnecessary rebuilds, or be entirely confident we have received a notification sent from an external source.
The real solution lies in a hybrid approach. We should:
- Use downstream container images, whether provided by Red Hat or internally, to minimize the cost of container rebuilds.
- Unconditionally, periodically rebuild our custom images on a set schedule to be sure we catch all the edge cases.
- Consider reducing the periodic rebuild frequency (and load on our infrastructure) by performing cheap metadata-based polling or notification-triggered rebuilds. We would need to be confident that our engineering effort here covers enough of the potential sources of change to warrant a reduction in scheduled rebuilds.
Links
- E-book: Red Hat Universal Base Images (UBI)
- Search: Red Hat certified container images
