Some OpenShift administrators asked for more flexibility for RHEL CoreOS customization in specific contexts. The RHCOS image layering feature is generally available in OpenShift 4.13.

CoreOS-Layering is making containers bootable in OCP, which allows customers to manage their OS images through containers. Containers have become an industry standard, and many tools have been developed around them, including pipelines, security checkers, smoke tests, deliverability, and more. Therefore, bootable containers bring many benefits and ease of maintenance.

We can now create a customized base image using a different kernel or any particular software and consume it in a given cluster.

RHCOS image layering allows you to easily extend the functionality of your base image by layering additional images onto the base image. This layering does not modify the base image, and it creates a custom layered image that includes all base functionality and add-ons to specific nodes in the cluster.

Different methods of injecting kernel modules into a cluster

  • Day 0: Create a custom RHCOS image for deployment, including the kernel module.
  • Day 1/2 MCO: Use the Machine Config Operator on day2 (as shown in this demo). It can also be used as day1 by adding a MachineConfig manifests to the cluster installation process (not shown in this demo).
  • Day 2 KMM: Use the Kernel Module Management Operator.

Note: Future recommendations are subject to change as enhancements and features are added.

Red Hat support for third-party components

Red Hat support model

For more details about the support model, please check the support-model for using third-party components such as kernel modules.


RHCOS layering is in tech preview in OCP 4.12 and fully supported in OCP 4.13+.

We will use the driver-toolkit as a base image to build the simple-kmod kernel module and then add the .ko to a new layer on top of the RHEL-CoreOS image that runs on the node.

Get the correct driver-toolkit image

The driver-toolkit contains the kernel package and headers to build kernel modules for a specific kernel. We must make sure we use the correct digest of the driver-toolkit image to fit our specific node.

Let's get the driver-toolkit image for our 4.12.18 cluster:

$ oc adm release info --image-for=driver-toolkit


For any other version/arch, you can use:

$ oc adm release info<cluste version>-x86_64 --image-for=driver-toolkit
$ oc adm release info<cluster version>-aarch64 --image-for=driver-toolkit


Get the correct rhel-coreos-8 image

Make sure to use the same image that runs on the nodes as the base image of the driver-container image for minimal side effects. We only want to add the new .ko file (and required configuration files if such exists) to the node. Nothing else should change.

Note: OCP 4.13 uses rhel-coreos-9 while OCP 4.12 uses rhel-coreos-8. Be sure to use the correct --image-for flag based on your cluster version.

Let's get the rhel-coreos image for our cluster version.

$ oc adm release info --image-for=rhel-coreos-8


Get the correct kernel-version

We also need the correct kernel version to build the driver-container.

After reboot, the host kernel will be the kernel RPM installed on the new image—the rhel-coreos-8 image that we are using as the last layer. The correct way to get the kernel version is from the image itself.

$ podman run -it \
  rpm -qa | grep kernel


If you don't have access to the image, make sure you are using your pull-secret with Podman.

We can see the kernel RPM in the image is 4.18.0-372.53.1.el8_6.x86_64.

Build the container image

We will use the following Dockerfile:


FROM ${DTK_IMAGE} as builder

WORKDIR /build/

RUN git clone

WORKDIR /build/simple-kmod

RUN make all install KVER=${KERNEL_VERSION}


COPY --from=builder /etc/driver-toolkit-release.json /etc/
COPY --from=builder /lib/modules/${KERNEL_VERSION}/*.ko /usr/lib/modules/${KERNEL_VERSION}/

# This is needed to autoload the module at boot time.
RUN depmod -a "${KERNEL_VERSION}" && echo simple_kmod > /etc/modules-load.d/simple_kmod.conf


For simplicity, let's define some environment variables to hold the previous results:

$ export
$ export
$ export KERNEL_VERSION=4.18.0-372.53.1.el8_6.x86_64


Now, we can build the container image.

$ podman build \
  --build-arg DTK_IMAGE=${DTK_IMAGE} \
  --build-arg RHCOS_IMAGE=${RHCOS_IMAGE} \
  -t .
[1/2] STEP 1/6: FROM AS builder
--> Using cache 0a1ec1eb4b3eb2accba3475b2e347ac81c0f01b607d25c5e81c0d01471092d21
--> 0a1ec1eb4b3
[1/2] STEP 3/6: WORKDIR /build/
--> Using cache 02fd2db894dbf7f2c59b7dbabf64c2114eb283ddd230163fb9bd8df892045e02
--> 02fd2db894d
[1/2] STEP 4/6: RUN git clone
--> Using cache 419fda4f52482891200333bfa165ab40d66c40bd4180be1c207dd48ba6fa7d27
--> 419fda4f524
[1/2] STEP 5/6: WORKDIR /build/simple-kmod
--> Using cache 80d50b672deb18ba0f3876e32d487eaaa5832765c87f724c079130576c8cfd6d
--> 80d50b672de
[1/2] STEP 6/6: RUN make all install KVER=4.18.0-372.53.1.el8_6.x86_64
make -C /lib/modules/4.18.0-372.53.1.el8_6.x86_64/build M=/build/simple-kmod EXTRA_CFLAGS=-DKMODVER=\\\"e852852\\\" modules
make[1]: Entering directory '/usr/src/kernels/4.18.0-372.53.1.el8_6.x86_64'
 CC [M] /build/simple-kmod/simple-kmod.o
 CC [M] /build/simple-kmod/simple-procfs-kmod.o
 Building modules, stage 2.
 MODPOST 2 modules
 CC /build/simple-kmod/simple-kmod.mod.o
 LD [M] /build/simple-kmod/simple-kmod.ko
 CC /build/simple-kmod/simple-procfs-kmod.mod.o
 LD [M] /build/simple-kmod/simple-procfs-kmod.ko
make[1]: Leaving directory '/usr/src/kernels/4.18.0-372.53.1.el8_6.x86_64'
gcc -o spkut ./simple-procfs-kmod-userspace-tool.c
install -v -m 755 spkut /bin/
'spkut' -> '/bin/spkut'
install -v -m 755 -d /lib/modules/4.18.0-372.53.1.el8_6.x86_64/
install -v -m 644 simple-kmod.ko /lib/modules/4.18.0-372.53.1.el8_6.x86_64/simple-kmod.ko
'simple-kmod.ko' -> '/lib/modules/4.18.0-372.53.1.el8_6.x86_64/simple-kmod.ko'
install -v -m 644 simple-procfs-kmod.ko /lib/modules/4.18.0-372.53.1.el8_6.x86_64/simple-procfs-kmod.ko
'simple-procfs-kmod.ko' -> '/lib/modules/4.18.0-372.53.1.el8_6.x86_64/simple-procfs-kmod.ko'
depmod -F /lib/modules/4.18.0-372.53.1.el8_6.x86_64/ 4.18.0-372.53.1.el8_6.x86_64
--> 26dfedc51be
[2/2] STEP 1/5: FROM
--> 2879805a870
[2/2] STEP 3/5: COPY --from=builder /etc/driver-toolkit-release.json /etc/
--> 8dea5bb2bd8
[2/2] STEP 4/5: COPY --from=builder /lib/modules/4.18.0-372.53.1.el8_6.x86_64/*.ko /usr/lib/modules/4.18.0-372.53.1.el8_6.x86_64/
--> 6249995c281
[2/2] STEP 5/5: RUN depmod -a 4.18.0-372.53.1.el8_6.x86_64 && echo simple_kmod > /etc/modules-load.d/simple_kmod.conf
[2/2] COMMIT
--> 612c8c8a54f
Successfully tagged


Push the image to some image registry, so it is accessible by the cluster:

$ podman push


Make sure the image is public, or it has the correct pull-secret to pull the image. The pull secret in the cluster is:

$ oc get secret/pull-secret -n openshift-config


For more information, check the how to update the global pull-secret doc.

Load the kernel module into the nodes

First, create a MachineConfig file:

kind: MachineConfig
 labels: worker
 name: 99-ybettan-external-image


Make sure to change the osImageURL section in the MachineConfig YAML to the image you just built, and then apply it:

$ oc apply -f machineconfig.yaml


The Machine Config Operator will update worker nodes one by one with the new image specified in osImageURL, which contains the kernel module.

Although some optimizations are in place to apply live-updates, nodes will reboot in some cases.

This process may take some time, and you can track the progress using this command:

$ watch oc get nodes


Watch for all worker nodes to display Ready again.

Here is a snapshot of the node updating process:

ip-10-0-146-108.ec2.internal Ready control-plane,master 39m v1.25.8+37a9a08
ip-10-0-147-43.ec2.internal Ready,SchedulingDisabled worker 32m v1.25.8+37a9a08
ip-10-0-166-128.ec2.internal Ready control-plane,master 39m v1.25.8+37a9a08
ip-10-0-178-9.ec2.internal Ready worker 30m v1.25.8+37a9a08
ip-10-0-226-15.ec2.internal Ready control-plane,master 39m v1.25.8+37a9a08
ip-10-0-228-190.ec2.internal Ready worker 29m v1.25.8+37a9a08


For some additional info regarding MCO in case it doesn't update, the nodes check the machine-config-pull-status.

Validate that the kernel module was loaded

Start by validating that the node is indeed running a customer image:

$ oc debug node/ip-10-0-147-43.ec2.internal -- chroot host/ rpm-ostree status
Starting pod/ip-10-0-147-43.ec2.internal-debug ...
To use host binaries, run `chroot /host`
State: idle
                 Digest: sha256:9894d87312b03e04a01371864dd30b56d223f37d7b808ee08f4054eef92f5033
                  Version: 412.86.202305161131-0 (2023-06-04T10:55:24Z)

Removing debug pod ...


Next, make sure that the kernel-module was loaded successfully using this command:

$ oc debug node/ip-10-0-147-43.ec2.internal -- chroot host/ lsmod | grep simple_kmod
Starting pod/ip-10-0-147-43ec2internal-debug ...
To use host binaries, run `chroot /host`
simple_kmod 16384 0

Removing debug pod ...


Alternative Solution

For kernel-modules installation on an OCP cluster, we can also use the dedicated kernel-module-management-operator, which is used for day2 kernel-modules installation use cases and some day1 use cases as well.



Red Hat CoreOS, openshift, drive-toolkit

< Back to the blog