Today is a beautiful day. Just like most. I am in a good mood and have been thinking about ways to help the users of my OpenShift cluster. I think I have come up with a way to make all our lives a little easier. And I have been told that it is the little things that mean a lot.

When the users create virtual machines using OpenShift Virtualization, they have to perform a bunch of extra tasks to get it ready for use in our shop. These are tasks like installing packages that don't come in the automatically supplied boot source images, configuring users and authentication to use our corporate servers as well as automatically mounting some NFS file shares.

It would be great if I could supply the users with an image that did all of this. It would be even better if I could make the cluster automatically consume any new images I create. Well, guess what? OpenShift and OpenShift Virtualization provides a way to do this.

Virtual machine templates can point to various sources for the retrieval of the disk images. These could be PVCs, DataVolumes or images on a webserver. There are also a few other sources, one of which is called a DataSource. The great thing about a DataSource is that you can populate it using an ImageStream that points to a container in a registry. A cron job can be defined in OpenShift Virtualization that will check if there is a new, updated image and automatically pull it.

A process like this means that any newly created virtual machines will be based off the latest image available. Automation for the win!

A little about the environment

The corporate environment provides a Red Hat Identity Manager (IdM) server that not only provides user account and authentication information, but also provides automount maps for user's home directories. It would be great to have all this configured on a boot source to prevent the users from needing to do so after deployment.

This should reduce the load on the network since packages will not need to be downloaded each time a VM is created. It will also save users both time and possibly frustration if they encounter issues. And probably most importantly, it will give me some peace and tranquility because I wont need to be fighting user fires.

The IdM server has a hostenroller user configured that has limited permissions. The user does have the permission to add new servers to the domain and will be used when customizing the image.

Preparing a custom boot source image.

Let's create a custom image that will do what we need. We can start with a Red Hat Enterprise Linux 8 (RHEL8) operating system for this custom image. We will customize the image as follows.

  • Update the packages on the image

  • Install a few extra packages

    • nfs-utils
    • ipa-client
    • realmd
    • bind-utils
    • wget
    • jq
  • Customize users and authentication to use a centralized IdM system.

  • Create an automount for an NFS mountpoint that will contain the users home directories.

We do need an image to start with. This can be a disk image taken from an existing system, one we create from scratch, a cloud-ready image (an image that is in a QCOW2 or raw format), or an image already in a container. An example containerized image is available to pull from Red Hats registry.redhat.io.

Let's use the image already in a container. It should be smaller than either of the other two image types. And after the image is extracted, the remaining process is the same for most other images.

Please note that my laptop currently runs Fedora 35 and this laptop will be used to customize the disk image. The qemu-img and guestfs-tools packages have been installed. These packages provide the commands needed to customize the image. See the man pages for the qemu-img, virt-customize, and virt-builder commands for more information.

Extracting an image from an existing container

To extract the disk image from within the container, we first need to pull the image.

$ podman pull registry.redhat.io/rhel8/rhel-guest-image:latest

Trying to pull registry.redhat.io/rhel8/rhel-guest-image:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4f4fb700ef54 skipped: already exists
Copying blob 624b05f45c88 done
Copying config 44b84a629b done
Writing manifest to image destination
Storing signatures
44b84a629bd9764bbdc41cab3ab8621b41db95e1292cd030f18bbd83e46f2694

Now that the image is in the local registry, it can be saved to a tarball on the local computer.

$ podman save -o custom.tar registry.redhat.io/rhel8/rhel-guest-image:latest 

Copying blob 5f70bf18a086 done
Copying blob 8a25dad6d941 done
Copying config 44b84a629b done
Writing manifest to image destination
Storing signatures

The tarball contains multiple files. One file should be another tarball that contains the disk image. It should be the largest file in the image. We can use sort and tail to help us find and extract it.

$ tar tvf custom.tar | sort  -k3 -n | tail -1 | sed 's/^.* //'

8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar


$ tar xvf custom.tar 8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar

8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar


$ ls

8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar custom.tar

There is a QCOW2 image in this file. This is the disk image we need. Let's find and extract the image.

$ tar tvf 8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar | sed -n 's/^.* \(.*\.qcow2\)/\1/p'

disk/rhel-guest-image-8.6-1959.x86_64.qcow2


$ tar xvf 8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar --strip-components 1 disk/rhel-guest-image-8.6-1959.x86_64.qcow2

disk/rhel-guest-image-8.6-1959.x86_64.qcow2


$ ls

8a25dad6d941f51ef1d28617a1332ae826402d1b36b7e80df30571db06cd4a95.tar
custom.tar
rhel-guest-image-8.6-1959.x86_64.qcow2

Later on, we will need to tag a container image with a unique identifier. Let's base the identifier off the image's OS and the current date and time.

Let's verify the OS version. We can use the virt-cat command to view the /etc/redhat-release file inside the image.

$ virt-cat -a rhel-guest-image-8.6-1959.x86_64.qcow2 /etc/redhat-release

Red Hat Enterprise Linux release 8.6 (Ootpa)

The image is using RHEL 8.6. Let's generate our identifier using the date command.

$ date +8.6-%Y%j.%H%M

8.6-2022193.1244

We will use the identifier 8.6-2022193.1244 to identify this image.

Now that we have the image, we can begin customizing it.

Customizing the image

As I said earlier, the remaining process should be the same for most QCOW2 images.

We will use the virt-customize command to do the customizations to the disk image. It accepts commands as options that instruct it what to do. See the man pages for virt-customize command for details on how to use the command.

virt-customize has and option, --commands-from-file, that instructs it to pull its commands from a file. Using this option allows us to easily change the commands passed to virt-customize. This is useful for testing. This should also help prevent syntax errors and it make it easier to create updated images in the future.

The commands are the same as those given on the command line, without the preceding double dash (--).

Registering the image for package installation

There are some packages we need and some others that are just nice to have. Before we can install the packages, the operating system in the image needs to know where to get them. They can be first placed into the image along with any dependencies or they can use the normal method used by the operating system.

A Fedora system should not need any extra configuration since its package repositories are public. But the RHEL9 image needs to subscribe to the Red Hat Subscription service. It only needs to subscribe temporarily. Just long enough to install the packages. There are a few command options needed to do this.

Let's start creating a command file that contains the commands. We will call it customized-rhel.cmds. The file will be built on until we have a complete file containing all the customizations.

Register system and install packages

First let's register the os and attach a subscription pool to the image. The following pulls the password and pool id from files. The permissions on the files are set so that only the user creating the image can view and edit them. This is for security purposes only. Its better to put these in a file with limited access than as plain text in the configuration file.

# Register the image to Red Hat Subscription Manager(RHSM)
sm-credentials nada:file:rhsm-nada.pswd
sm-register
sm-attach file:rhsm-rhel.pool

Let's make sure the image has the latest packages in it.

# Update the packages
update

Install the packages that we need and some we just want.

# Install some useful and needed packages
install wget,bind-utils,ipa-client,nfs-utils,realmd,jq

After the packages are installed, we can remove the subscriptions and unregister the system.

# Unregister the image from RHSM
sm-remove
sm-unregister

Domain configuration

We need to add the system to the corporate domain and setup the user authentication and remote home directory mounts. If we do this to the image, then all systems deployed from the image will appear as the same system to the domain servers. This is not what we want.

We will configure the image so that it will finish its domain configuration after it boots the first time.

Cloud-init is present in the images we use and can be used to do various configuration changes after the system boots such as creating local user accounts, configuring the system for package installation, and running custom scripts. It is an excellent way to configure the system. It is also the default configuration service used with cloud based images in the OpenShift Cluster.

However, we will use the virt-customize command to stage the changes. It is already being used to install packages and do other configuration to the image. This will keep the related commands and configurations in a single file and easier for us to manage.

Virt-customize has commands that allow us to run scripts or commands on first boot. We will use this to register the VM after it is created. This helps ensure each VM shows as its own system in the domain servers.

# Make sure the hostname is set to a FQDN so the VM can join the realm.
firstboot-command [[ $(hostname --domain) ]] || nmcli general hostname $(hostname --short).pearl.vlan212.example.org

# Join the VM to the realm and configure automount for the users home directories.
firstboot-command echo enrollmeplease | realm join --verbose --user hostenroller idm.vlan212.example.org

firstboot-command ipa-client-automount --unattended

The password for the hostenroller user is enrollmeplease and is passed as plain text in the above command. This is not ideal, but will suffice for this blog post.

Image identification

Let's create a file on the system that will indicate the version of the image. This can be useful later to identify issues with images or to help a user troubleshoot. We will use the identifier created earlier.

We will create the /etc/corporate-release file.

# Create a file to identify the image
firstboot-command echo "Corporate RHEL Server Image version 8.6-2022193.1244" > /etc/corporate-release

Sudo

It would be nice to allow members of the admin-pearl-vm domain group to have root access on the server. Appending a line to the /etc/sudoers files will allow this.

# Configure sudoers to allow users of the admin-pearl-vm group to use sudo.
append-line /etc/sudoers:%admin-pearl-vm ALL=(ALL) NOPASSWD: ALL

SELinux

Since RHEL uses SELinux, the use_nfs_home_dirs SELinux boolean must be enabled to allow the users home directories to be NFS mounts.

# Configure SELinux to allow users home directories to be nfs mounts and then relabel the system
run-command setsebool -P use_nfs_home_dirs 1

This image has SELinux enabled so the contexts for files must be correct. Since we configured the image using virt-customize, some files may have incorrect context. The selinux-relabel command will relabel the files in the image. If this process cannot be done immediately, then it will be done on first boot of the image.

selinux-relabel

Applying the customizations

After we put all the above together in a file, the file should look like this:

customized-rhel.cmds:

# Register the image to Red Hat Subscription Manager(RHSM)
sm-credentials nada:file:rhsm-nada.pswd
sm-register
sm-attach file:rhsm-rhel.pool

# Update the packages
update

# Install some useful and needed packages
install wget,bind-utils,ipa-client,nfs-utils,realmd,jq

# Unregister the image from RHSM
sm-remove
sm-unregister

# Make sure the hostname is set to a FQDN so the VM can join the realm.
firstboot-command [[ $(hostname --domain) ]] || nmcli general hostname $(hostname --short).pearl.vlan212.example.org

# Join the VM to the realm and configure automount for the users home directories.
firstboot-command echo enrollmeplease | realm join --verbose --user hostenroller idm.vlan212.example.org

firstboot-command ipa-client-automount --unattended

# Create a file to identify the image
firstboot-command echo "Corporate RHEL Server Image version 8.6-2022193.1244" > /etc/corporate-release

# Configure sudoers to allow users of the admin-pearl-vm group to use sudo.
append-line /etc/sudoers:%admin-pearl-vm ALL=(ALL) NOPASSWD: ALL

# Configure SELinux to allow users home directories to be nfs mounts and then relabel the system
run-command setsebool -P use_nfs_home_dirs 1

selinux-relabel

We need to make sure the image is writable and then we can customize the image. We specify the image type and the command file when invoking the command. We also attach the image to be customized.

$ chmod +w rhel-guest-image-8.6-1959.x86_64.qcow2


$ virt-customize --format qcow2 --commands-from-file customized-rhel.cmds -a rhel-guest-image-8.6-1959.x86_64.qcow2

[ 0.0] Examining the guest ...
[ 4.9] Setting a random seed
[ 5.0] Setting the machine ID in /etc/machine-id
[ 5.0] Registering with subscription-manager
[ 13.5] Attaching to the pool 8a85f99f7d76f2e8017d96c057c523d3
[ 38.9] Updating packages
[ 98.5] Installing packages: wget bind-utils ipa-client nfs-utils realmd jq
[ 129.1] Removing all the subscriptions
[ 131.9] Unregistering with subscription-manager
[ 132.8] Installing firstboot command: [[ $(hostname --domain) ]] || nmcli general hostname $(hostname --short).pearl.vlan212.example.org
[ 132.8] Installing firstboot command: echo changeme | realm join --verbose --user hostenroller idm.vlan212.example.org
[ 132.9] Installing firstboot command: ipa-client-automount --unattended
[ 132.9] Installing firstboot command: echo "Corporate RHEL Server Image version 8.6-2022193.1244" > /etc/corporate-release
[ 132.9] Appending line to /etc/sudoers
[ 132.9] Running: setsebool -P use_nfs_home_dirs 1
[ 133.8] SELinux relabelling
[ 149.1] Finishing off

After the image is customized, we should make sure it is clean of any unique information that would get passed onto the VMs. This would be things like the machine ID, nic information, identifiers for RHSM, and other information.

$ virt-sysprep -a rhel-guest-image-8.6-1959.x86_64.qcow2

[ 0.0] Examining the guest ...
[ 2.4] Performing "abrt-data" ...
[ 2.4] Performing "backup-files" ...
[ 2.7] Performing "bash-history" ...
[... Truncated for Brevity ...]
[ 3.0] Performing "yum-uuid" ...
[ 3.0] Performing "customize" ...
[ 3.0] Setting a random seed
[ 3.1] Setting the machine ID in /etc/machine-id
[ 3.1] Performing "lvm-uuids" ...

The image should now be customized and clean. Let's look at the details of the disk image using the qemu-img command.

$ qemu-img info rhel-guest-image-8.6-1959.x86_64.qcow2

image: rhel-guest-image-8.6-1959.x86_64.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.52 GiB
cluster_size: 65536
Format specific information:
compat: 0.10
compression type: zlib
refcount bits: 16

In the above, you can see the actual size of the disk image (disk size: 1.52 GiB) and the virtual size of the disk image (virtual size: 10 GiB 10737418240 bytes)

It would be great to get this image as small as possible. Let's run virt-sparsify against the image to see if we can get the image smaller. This command will write out a new image file for us called custom-corp-image-* the end of the name will be a unique integer based upon the current time and date.

$ virt-sparsify --compress rhel-guest-image-8.6-1959.x86_64.qcow2 custom-corp-image-8.6-2022193.1244.qcow2

[ 0.0] Create overlay file in /tmp to protect source disk
[ 0.0] Examine source disk
[ 2.3] Fill free space in /dev/sda2 with zero
[ 2.5] Fill free space in /dev/sda3 with zero
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 7.2] Copy to destination and make sparse
[ 85.1] Sparsify operation completed with no errors.
virt-sparsify: Before deleting the old disk, carefully check that the
target disk boots and works correctly.


$ qemu-img info custom-corp-image-8.6-2022193.1244.qcow2

image: custom-corp-image-8.6-2022193.1244.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.03 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false

Containerize image and push to registry

Now that the image is customized, it can be containerized and uploaded to a registry. This is a lot easier than extracting the image from a container. We will use podman to create the container.

We need to create a Containerfile that instructs podman on how to create the container. The Containerfile only needs two lines. The first line tells podman that we are creating an empty container from scratch. The second line instructs podman to add the customized image into the container.

rhel-custom.container:

FROM scratch
ADD custom-corp-image-8.6-2022193.1244.qcow2 /disk/

Now we can build the image. We run the podman build command and specify the Containerfile and the registry/tag for the local registry. We can exclude the registry portion and localhost will be automatically prepended. Once the image builds, it will be placed in the local repository on the system it was built on.

Let's tag the image with 8.6-2022193.1244.

$ podman build -f rhel-custom.container -t corp-server-rhel:8.6-2022193.1244

STEP 1/2: FROM scratch
STEP 2/2: ADD custom-corp-image-8.6-2022193.1244.qcow2 /disk/
COMMIT corp-server-rhel:8.6-2022193.1244
--> c3ce5a98029
Successfully tagged localhost/corp-server-rhel:8.6-2022193.1244
c3ce5a98029687e53d718bf72363907ec5c0e97ad8c473bdca8bd7bb8cc6f228


$ podman images

REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/corp-server-rhel 8.6-2022193.1244 c3ce5a980296 47 seconds ago 1.12 GB

The image is now containerized. We can push the image to either an internal or external registry. For this blog post, we can use a repository of mine on quay.io since it is only temporary. Normally we would probably want to use a private or internal registry for this. We will use the joherr/corp-server-rhel repository.

We will push the container to the repository using the same tag used locally.

$ podman push localhost/corp-server-rhel:8.6-2022193.1244 quay.io/joherr/corp-server-rhel:8.6-2022193.1244

Getting image source signatures
Copying blob 1d7afcd7f7a6 done
Copying config c3ce5a9802 done
Writing manifest to image destination
Storing signatures

Pushing the same container to the registry but with a different tag will add the tag to the image. The registry knows the image already exists and does not push it a second time.

We do this for the latest tag. Using the tag called latest will allow us easily update the images pulled by the cluster by simply changing which container is tagged with latest.

$ podman push localhost/corp-server-rhel:8.6-2022193.1244 quay.io/joherr/corp-server-rhel:latest

Getting image source signatures
Copying blob 7345eb3b2972 skipped: already exists
Copying config 4720f6884d done
Writing manifest to image destination
Storing signatures

We can see the image has been uploaded to the quay.io repository and has both tags. Looking to the right of the Manifest column shows a line connecting the two tags. This indicates they are the same container. 

8.6aslatest

Create an ImageStream

We have a containerized image uploaded to a registry. Now we need to tell the cluster about it. We will create an image stream, or reference, for the container using the following yaml file.

Here are some important fields in the file.

  • metadata.name
    • This is the name of the ImageStream
  • metadata.namespace
    • This is the namespace the ImageStream is in.
  • spec.tags.from.name
    • This points to the image in the repository
  • spec.tags.name
    • The name of the tag (not the tag of the container image)

is-corp-svr-rhel.yaml:

apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
labels:
app: kubevirt-hyperconverged
app.kubernetes.io/component: compute
app.kubernetes.io/managed-by: hco-operator
app.kubernetes.io/part-of: hyperconverged-cluster
app.kubernetes.io/version: 4.10.1
name: is-corp-svr-rhel
namespace: openshift-virtualization-os-images
spec:
lookupPolicy:
local: false
tags:
- name: latest
annotations: null
from:
kind: DockerImage
name: quay.io/joherr/corp-server-rhel:latest
importPolicy:
scheduled: true
referencePolicy:
type: Source

Applying the file creates the ImageStream.

$ oc apply -f is-corp-svr-rhel.yaml 

imagestream.image.openshift.io/is-corp-svr-rhel created

After the ImageStream is created, it should find the latest image and indicate the images sha256 checksum in its status.

$ oc -n openshift-virtualization-os-images get imagestream is-corp-svr-rhel -o json   | jq -r '.status.tags[] | [.tag, .items[0].image] | @tsv'

latest sha256:72047088c6576035aa634902c5d51274634b306a91f3dcd71e8b3583ac3099e8

This sha256 checksum should match the one in the Quay repository. This can be checked by selecting SHA256 under the MANIFEST column for the tag.

sha256

Populate the DataSource

The cluster is now aware of the container. Let's tell the cluster to use the image to populate a DataSource. The following patch to the deployed HyperConverged tells the deployment to populate a DataSource using the ImageStream.

There is currently a default StorageClass defined in the cluster. The default StorageClass can be updated by changing the class.kubernetes.io/is-default-class annotation.

Set the annotation to false to unset the current default StorageClass.

oc patch storageclass <default_storage_class> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Set the annotation to true to set a new default StorageClass.

oc patch storageclass <storage_class> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Some important fields in the file are:

  • spec.dataImportCronTemplates.metadata.name
    • This is the name of the dataImportCronTemplate
  • spec.dataImportCronTemplates.spec.schedule
    • This is the schedule for checking for new images and populating the DataSource with them. This is in standard cron format. For this blog post, I am setting it to run every 5 minutes. But for a production system, the time between runs should probably be a lot longer.
  • spec.dataImportCronTemplates.spec.template.spec.source.registry.imageStream
    • This is the name of the image stream to check.
  • spec.dataImportCronTemplates.spec.template.spec.storage.resources.requests.storage
    • This is the amount of storage that should be reserved for the DataSource. This should be larger than the virtual size of the disk image as shown by qemu-img. The size reported by qemu-img was 10 GiB, we will make this 12 GiB just to be safe.
  • spec.dataImportCronTemplates.spec.managedDataSource
    • This is the name of the DataSource to create from the ImageStream.

patch-hyperconverged.yaml

spec:
dataImportCronTemplates:
- metadata:
name: cron-corp-svr-rhel
spec:
schedule: "*/5 * * * *"
template:
spec:
source:
registry:
imageStream: is-corp-svr-rhel
pullMethod: node
storage:
resources:
requests:
storage: 12Gi
managedDataSource: corprhels
retentionPolicy: "None"

An annotation of cdi.kubevirt.io/storage.bind.immediate.requested: "true" is needed if the StorageClass used has a volumeBindingMode of WaitForFirstConsumer.

Let's get the name of the hyperconverged deployment. We need this to apply the patch.

$ oc get -n openshift-cnv hyperconverged

NAME AGE
kubevirt-hyperconverged 11d

Let's patch the hyperconverged with the above file.

$ oc -n openshift-cnv patch hyperconverged kubevirt-hyperconverged \
--type merge --patch-file patch-hyperconverged.yaml

hyperconverged.hco.kubevirt.io/kubevirt-hyperconverged patched

We can verify the patch was successful by looking at the hyperconverged.

$ oc get -n openshift-cnv hyperconverged -o json \
| jq '.items[].spec.dataImportCronTemplates'

[
{
"metadata": {
"annotations": {
"cdi.kubevirt.io/storage.bind.immediate.requested": "true"
},
"name": "cron-corp-svr-rhel"
},
"spec": {
"managedDataSource": "corprhels",
"retentionPolicy": "None",
"schedule": "*/5 * * * *",
"template": {
"spec": {
"source": {
"registry": {
"imageStream": "is-corp-svr-rhel",
"pullMethod": "node"
}
},
"storage": {
"resources": {
"requests": {
"storage": "12Gi"
}
}
}
}
}
}
}
]

The DataImportCron controller creates a PVC. After several minutes, the DataSource should be populated and ready to use.

$ oc -n openshift-virtualization-os-images get datasource corprhels -o json | jq '.spec, .status'
{
"source": {
"pvc": {
"name": "corprhels-8ca6cde7de25",
"namespace": "openshift-virtualization-os-images"
}
}
}
{
"conditions": [
{
"lastHeartbeatTime": "2022-07-12T21:57:15Z",
"lastTransitionTime": "2022-07-12T21:57:15Z",
"message": "DataSource is ready to be consumed",
"reason": "Ready",
"status": "True",
"type": "Ready"
}
]
}

That is all there is to creating a DataSource that automatically updates. Based upon the cron schedule, the DataSource will update automatically whenever the container image changes.

Create a template

Let's create a template that will use the new DataSource. All we need to do is add a dataVolumeTemplates section under objects.spec in the template.

The following is all that is needed.

    dataVolumeTemplates:
- apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
name: ${NAME}
spec:
sourceRef:
kind: DataSource
name: ${DATA_SOURCE_NAME}
namespace: ${DATA_SOURCE_NAMESPACE}
storage:
resources:
requests:
storage: 12Gi

Please notice that we are using variables for some of the settings. These are set to defaults at the end of the template's yaml file and allows these variables to be more easily changed from command line.

Here are the variable settings. The names and descriptions of the variables are self explanatory:

- description: VM name
from: rhel9corp-[a-z0-9]{16}
generate: expression
name: NAME
- description: Name of the DataSource to clone
name: DATA_SOURCE_NAME
value: rhel9corp
- description: Namespace of the DataSource
name: DATA_SOURCE_NAMESPACE
value: openshift-virtualization-os-images
- description: Size of virtual disk
name: DISK
value: 12Gi

We will use the following template file. The objects.spec.dataVolumeTemplates.spec.storage.resources.requests.storage must be at least as large as the size reserved for the DataSource.

Note: The annotations and labels are not required. Adding this small set of labels and annotations allows the template to not only show up in the User Interface (UI), but also provide some meaningful information to be displayed in the UI.

As of the publishing of this blog, even though the VM can be created using the UI - the configurations done after selecting the Customize virtual machine button will fail. Only the name, project, and exposing SSH settings work in the UI. But the template does show up in the UI and will allow the users to create a working VM that is created identical to the template's defaults.

The labels and annotations do not affect the use of the template from the command line.

corp-server-rhel.yaml:

apiVersion: template.openshift.io/v1
kind: Template
metadata:
name: corp-server-rhel
namespace: templates-corporate
annotations:
description: This is a server configured with corporate access
openshift.io/display-name: Corporate RHEL Server
template.kubevirt.io/provider-support-level: Corporate
iconClass: icon-rhel
template.kubevirt.io/provider: Corporate IT
template.kubevirt.ui/parent-support-level: Full
labels:
template.kubevirt.io/type: base
objects:
- apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
labels:
app: ${NAME}
name: ${NAME}
spec:
dataVolumeTemplates:
- metadata:
name: ${NAME}-rootdisk
spec:
sourceRef:
kind: DataSource
name: ${DATA_SOURCE_NAME}
namespace: ${DATA_SOURCE_NAMESPACE}
storage:
resources:
requests:
storage: ${DISK}
running: false
template:
metadata:
annotations:
vm.kubevirt.io/os: rhel
vm.kubevirt.io/workload: server
labels:
kubevirt.io/domain: ${NAME}
spec:
domain:
cpu:
cores: 1
sockets: 1
threads: 1
devices:
disks:
- disk:
bus: virtio
name: cloudinitdisk
- bootOrder: 1
disk:
bus: virtio
name: rootdisk
gpus: []
hostDevices: []
interfaces:
- masquerade: {}
model: virtio
name: default
networkInterfaceMultiqueue: true
rng: {}
machine:
type: pc-q35-rhel8.4.0
resources:
requests:
memory: 2Gi
evictionStrategy: LiveMigrate
hostname: ${NAME}
networks:
- name: default
pod: {}
terminationGracePeriodSeconds: 180
volumes:
- cloudInitNoCloud:
userData: |
#cloud-config
user: cloud-user
password: '${CLOUD_USER_PASSWORD}'
chpasswd:
expire: false
name: cloudinitdisk
- dataVolume:
name: ${NAME}-rootdisk
name: rootdisk
parameters:
- description: Name for the new VM
from: corp-rh-[a-z0-9]{14}
generate: expression
name: NAME
required: true
- description: Randomized password for the cloud-init user
from: '[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}'
generate: expression
name: CLOUD_USER_PASSWORD
- description: Name of the DataSource to clone
name: DATA_SOURCE_NAME
value: corprhels
- description: Namespace of the DataSource
name: DATA_SOURCE_NAMESPACE
value: openshift-virtualization-os-images
- description: Size of virtual disk
name: DISK
value: 12Gi

Let's create the template.

$ oc apply -f corp-server-rhel.yaml 

template.template.openshift.io/corp-server-rhel created

We can see that the VM shows up in both the command line and UI.

$ oc -n templates-corporate get templates

NAME DESCRIPTION PARAMETERS OBJECTS
corp-server-rhel This is a server configured with corporate access 4 (2 generated) 1

In the UI, the template shows that Corporate IT is the provider, but the Boot source is provided by Red Hat. This seems to appear because the label for the template type is set to base. Bug 2094074

template-available

Testing the template and image

Now let's create a VM from the template. We will use the UI first to show that it can be used to create a VM from the template. These VMs will be created by the jcarp user in the imauser project namespace.

Note: All cluster users have been granted view access to the templates-corporate project namespace. Without this access, they would not be able to see the templates in the templates-corporate project namespace.

$ oc create rolebinding -n templates-corporate corporate-view --clusterrole=view --group=system:authenticated

rolebinding.rbac.authorization.k8s.io/corporate-view created

When we go to create a VM, we can see our new Corporate RHEL Server template available. We need to make sure we are creating a VM in the imauser project namespace as shown in the upper left corner of the following picture. Under the Select a Template section, we need to select the namespace we stored the corporate templates in.

vm-from-template

After we select the template, we can configure the basics for the VM. Remember we cannot currently customize the VM other than choosing the project namespace, VM name, and SSH keys. Selecting the Customize virtual machine button will result in failures on the following pages. So we select Create virtual machine and the VM should get created.

vm-create

After the VM starts, the details page shows a command that can be used to connect to the VM using ssh. This command works if using an image with cloud-init installed. If it does not work for a reason like the default user is incorrect, below it shows the port that you can connect to using ssh.

Let's check if things are working as expected. First we will log in using the user frank. This user is a member of the admin-pearl-vm group and has sudo priviledges.

$ ssh frank@console-openshift-console.apps.pearl.example.org -p 31766
(frank@console-openshift-console.apps.pearl.example.org) Password:

Last login: Thu Jul 14 10:55:28 2022 from 10.128.0.1

We can see the VM is created from the custom image we created.

[frank@corprhels-artistic-pheasant ~]$ cat /etc/redhat-release 

Red Hat Enterprise Linux release 8.6 (Ootpa)


[frank@corprhels-artistic-pheasant ~]$ cat /etc/corporate-release

Corporate RHEL Server Image version 8.6-2022193.1244

The user's home directory is an NFS mount.

[frank@corprhels-artistic-pheasant ~]$ pwd

/home/frank


[frank@corprhels-artistic-pheasant ~]$ mount | sed -n "s/\(.*$USER\).*/\1/p"

nfs.vlan212.example.org:/exports/home/frank on /home/frank

The disk created for the VM is 12 GiB. The second disk listed is the cloud-init disk mounted by OpenShift Virtualization.

[frank@corprhels-artistic-pheasant ~]$ sudo -i

[frank@corprhels-artistic-pheasant ~]# df -h | grep '/$'

/dev/vda3 12G 2.4G 9.6G 20% /


[frank@corprhels-artistic-pheasant ~]# fdisk -l | grep '^Disk /'

Disk /dev/vda: 12 GiB, 12884901888 bytes, 25165824 sectors
Disk /dev/vdb: 1 MiB, 1048576 bytes, 2048 sectors

Next, we log in as the holly user. This user is not a member of any group with sudo privileges.

$ ssh holly@console-openshift-console.apps.pearl.example.org -p 31766 -i ~/.ssh/id_rsa_cloud-user 
(holly@console-openshift-console.apps.pearl.example.org) Password:

Last login: Thu Jul 14 10:56:04 2022 from 10.130.0.1

The user's home directory is a remote NFS mount.

[holly@corprhels-artistic-pheasant ~]$ mount | sed -n "s/\(.*$USER\).*/\1/p"

nfs.vlan212.example.org:/exports/home/holly on /home/holly

Let's create a file in the user directory. We will use this to verify the NFS mounted home directories are working when we create an updated image.

[holly@corprhels-artistic-pheasant ~]$ date > hollys.file

[holly@corprhels-artistic-pheasant ~]$ cat hollys.file

Thu Jul 14 11:23:09 EDT 2022

The holly user should not be able to use sudo since the user is not a member of the admin-pearl-vm group.

[holly@corprhels-artistic-pheasant ~]$ sudo -i

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

[sudo] password for holly:
holly is not allowed to run sudo on corprhels-artistic-pheasant. This incident will be reported.

Updating the DataSource

With the configuration above, updating the existing DataSource is as simple as uploading a new container to the repository and tagging it as latest.

Since the steps are identical, they will not be shown again. Instead, I will download the RHEL9 container image, customize it, and push it to the repository and tag it as latest.

So take a break while I will do some magic to the RHEL9 image.

[... Intermission ...]

I hope you enjoyed your break.

An updated image based on RHEL9 has been pushed to the registry. The image is tagged as 9.0-2022195.1144 and latest.

Let's get the sha256 checksum for it and verify the two tags are the same image.

$ skopeo inspect docker://quay.io/joherr/corp-server-rhel:9.0-2022195.1144 | jq .Digest

"sha256:ea880f8b3e01661f0d2252660870ded8c193907dd5602cb52bc4902ffd026304"


$ skopeo inspect docker://quay.io/joherr/corp-server-rhel:latest | jq .Digest

"sha256:ea880f8b3e01661f0d2252660870ded8c193907dd5602cb52bc4902ffd026304"

After a few minutes, the ImageStream updates.

$ oc -n openshift-virtualization-os-images get imagestream is-corp-svr-rhel -o json   | jq -r '.status.tags[] | [.tag, .items[0].image] | @tsv'

latest sha256:ea880f8b3e01661f0d2252660870ded8c193907dd5602cb52bc4902ffd026304

Then the DataSource gets updated with a new pvc after the cron runs. Do not use the date fields to determine if the boot source changes. Instead, watch for the pvc name to change. See bz# 2108339

$ oc -n openshift-virtualization-os-images get datasource corprhels -o json | jq .spec.source

{
"pvc": {
"name": "corprhels-72047088c657",
"namespace": "openshift-virtualization-os-images"
}
}

Testing the update

Let's create and start a VM to verify the DataSource used by the template updated correctly. We will process the template from the command line and increase its disk size to 20 GiB.

$ oc -n templates-corporate process corp-server-rhel DISK=20Gi | oc apply -f -

virtualmachine.kubevirt.io/corp-rh-g8tgsm8w2n4igh created


$ virtctl start corp-rh-g8tgsm8w2n4igh

VM corp-rh-g8tgsm8w2n4igh was scheduled to start

When creating the VM from the command line, we need to manually create the service that exposes the ssh port outside the cluster. This allows us to connect to the VM remotely.

$ virtctl expose vmi corp-rh-g8tgsm8w2n4igh --port 22 --name corp-rh-g8tgsm8w2n4igh-ssh-service --type NodePort

Service corp-rh-g8tgsm8w2n4igh-ssh-service successfully exposed for vmi corp-rh-g8tgsm8w2n4igh

Once the Service is exposed, we need to get the port that is created for inbound ssh connections. The output below shows the exposed port is 32479.

$ oc get service corp-rh-g8tgsm8w2n4igh-ssh-service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
corp-rh-g8tgsm8w2n4igh-ssh-service NodePort 172.30.98.237 <none> 22:32479/TCP 13s

Let's login to the new VM and verify which image it was created from. We will login using the user frank since the user has sudo privileges.

$ ssh frank@console-openshift-console.apps.pearl.example.org -p 32479
(frank@console-openshift-console.apps.pearl.example.org) Password:

Checking the version shows the VM was created using the new updated image.

[frank@corp-rh-g8tgsm8w2n4igh ~]$ cat /etc/redhat-release 

Red Hat Enterprise Linux release 9.0 (Plow)


[frank@corp-rh-g8tgsm8w2n4igh ~]$ cat /etc/corporate-release

Corporate RHEL Server Image version 9.0-2022195.1144

When we processed the template, we told it to increase the disk size to 20GiB. Checking the disk and partition size shows the disk is now 20 GiB. But it also shows that the last partition on the disk and its filesystem were both expanded to cosume the extra disk space. Isn't that a really nice feature.

# The disk disk is 20 GiB as we requested
[frank@corp-rh-g8tgsm8w2n4igh ~]$ sudo fdisk -l | grep '^Disk /'

Disk /dev/vda: 1 MiB, 1048576 bytes, 2048 sectors
Disk /dev/vdb: 20 GiB, 21474836480 bytes, 41943040 sectors


# The last partition on the disk was automatically resized to consume the entire 20 GiB
[root@corp-rh-g8tgsm8w2n4igh ~]# df -h | grep '/$'

/dev/vdb4 20G 1.4G 18G 8% /

Now let's login as the user holly and see if we can see the file created earlier.

$ ssh holly@console-openshift-console.apps.pearl.example.org -p 32479
(holly@console-openshift-console.apps.pearl.example.org) Password:

The following shows the file exists and has the correct information. We know that the NFS mounted home directories are working.

[holly@corp-rh-g8tgsm8w2n4igh ~]$ ls

hollys.file


[holly@corp-rh-g8tgsm8w2n4igh ~]$ cat hollys.file

Thu Jul 14 11:23:09 EDT 2022

Final Thoughts

Now anytime a new image is created it will automatically update after it's pushed to the registry. With a little more work, we should be able to automate the entire process. But that's for another day.

I hope you have enjoyed reading this blog as much as I have enjoyed writing it. Hopefully you learned something from reading it. I know I have learned something from writing it.