Updating a Boot Source Image

At some point, images used to create virtual machines (VM) in any virtualization environment will need to be updated for a variety of reasons. This could be a newer version of the operating system (OS), new or updated software, or possibly just configuration changes within the OS itself. This is true for OpenShift Virtualization on the Red Hat OpenShift Container Platform (RHOCP) as well.

In OpenShift Virtualization, Boot Source images are used to provide a consistent OS image to create VMs. Using Boot Source images allows a VM to be more easily created since the storage information will automatically populate based upon the image.

This blog post shows a method that can be used to update an existing Boot Source image that is used to install Red Hat Enterprise Linux 8 (RHEL). The web console will be used to perform most of the work. However, the command line equivalent will also be shown. The Persistent Volume Claim (PVC) and Data Volume (DV) for the Boot Source image are called rhel8 and are stored in the openshift-virtualization-os-images project namespace. This Boot Source image will be updated from a RHEL 8.1 image to a RHEL 8.3 image.

The Environment

The environment used in this blog post consists of five nodes running as an OpenShift Container Platform (OCP 4.6) cluster. The cluster consists of three master nodes running as virtual machines. The two compute nodes in the cluster are running on two physical servers.

The Boot Source image for the Red Hat Enterprise Linux 8 OS is called rhel8. The image currently exists and is an image of the RHEL8.1 OS.

The procedures discussed in this blog post require OpenShift 4.6 or greater and that the OpenShift Virtualization operator be installed. Once the operator is installed, OpenShift Virtualization must be deployed and storage must be configured. See the OpenShift Virtualization documentation on how to install and configure these.

This cluster is using the Trident storage provider to connect to a NetApp storage device.

The Image

The new Boot Source image can be customized before it is uploaded or it can be customized after it is uploaded. Customizing the image before it is uploaded can be done using the virt-sysprep command. This command can also generalize an image by removing SSH keys, network configuration, and subscription information as well as inject configuration, update packages and run scripts as well as many other things. Please see the man page on the virt-sysprep command for all its capabilities.

The virt-sysprep tool works on offline Linux images before they are uploaded into the OpenShift cluster. At times it is more desirable to use an image that is already in the cluster for the Boot Source image. The existing Boot Source image may just need some packages updated or a configuration file changed. Or maybe there is an image in the cluster that was used to test the changes the Boot Source image needs.

Customizing an existing image in the cluster is quite easy. If the image to be customized is an existing Boot Source image, simply create a VM from the image and make the desired changes. If the image to be used is not an existing Boot Source image, create a new VM using the wizard and have it clone the existing PVC you wish to use as the Boot Source image.

Using the virt-sysprep command and customizing the image is beyond the scope of this blog post. We will simply download a newer version of the RHEL 8 operating system that already contains the needed drivers. But we will step through the processes that would allow you to update an image after it is uploaded. This will be mentioned, but not discussed.

The Boot Source images are stored in the openshift-virtualization-os-images project namespace and are called a specific name. These templated names can be displayed using the following command. The output displays a list of template names followed by the Boot Source name for each OS in the system templates. The following output has been truncated, but the RHEL 8 OS uses rhel8 for its Boot Source name.

$ oc -n openshift get template \
    -l template.kubevirt.io/type=base \
    -o jsonpath='{range .items[*]}{@.metadata.name}{"\t"}{@.parameters[?(@.name=="SRC_PVC_NAME")].value}{"\n"}'

[[..Output Truncated..]]
rhel8-desktop-large-v0.11.3	rhel8
rhel8-desktop-medium-v0.11.3	rhel8
rhel8-desktop-small-v0.11.3	rhel8
rhel8-desktop-tiny-v0.11.3	rhel8
[[..Output Truncated..]]

The Process

During the following process, there is some time that the web interface will not show that a Boot Source exists for the RHEL 8 OS. This will limit the users ability to create VMs during that time. The process used here was done to limit this impact to the user as much as possible.

As a note, any VMs in the process of being created from the rhel8 Boot Source image should not be affected. If the cloning process has already been started then the system will complete the cloning process before it removes the existing Boot Source image. If the cloning process has not started then it will start after the new Boot Source image has been uploaded.Upload the Image

Upload the Image

The new image can be uploaded directly to the openshift-virtualization-os-images project namespace if no changes are needed to it after uploading. If the VM needs to be customized after being uploaded, it should be uploaded to a different project namespace. Even though we will not be customizing the image after it is uploaded, we will still upload the image to a different project namespace called images-workspace.

This allows a VM to be created using the uploaded image. Once the VM is created, it can be powered up and the image can be customized. The VM can then be deleted, making sure to not delete the VMs disks, and the PV can be used as the new Boot Source image.

We will be uploading a RHEL 8.3 image that will replace the current RHEL 8.1 Boot Source image called rhel8.

Image Size

It is important to know the size of the uncompressed disk image. This uncompressed size, plus a little extra for overhead, needs to be reserved on the storage device. This is because the image will be expanded when written to the PV. The size can be checked by running the qemu-img info image.qcow2 command and looking at the virtual size field.

$ qemu-img info rhel-8.3-x86_64-kvm.qcow2 

image: rhel-8.3-x86_64-kvm.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 1.23 GiB
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16

The output shows the uncompressed image size will be 10 GiB. We will add a little extra just in case, so we will reserve 11 GiB for the uploaded image since it is an easy number to use.

Create Project Namespace

This step can be skipped or another namespace can be used. Make sure to avoid using the openshift-virtualization-os-images project namespace.

Navigate to [Home ⮞ Projects] in the web interface and select [Create Project].

Give the project the [Name] images-workspace.

To create the namespace using the command line, use the oc create namespace command.

$ oc create namespace images-workspace

namespace/images-workspace created

Upload

Upload the image to the images-workspace project namespace.

Using the web interface, navigate to [Storage ⮞ Persistent Volume Claims] and select [Create Persistent Volume Claim ⮞ With Data Upload Form]

Complete the form by selecting the local image to upload and use rhel8.3 for the Persistent Volume Claim Name. Make sure that [Attach this data to a Virtual Machine Operating System] is not checked and specify the [Size] using the value calculated earlier. Also make sure the Project listed at the top is set to images-workspace.

Select [Upload] to start the upload process. The cluster uses a self-signed certificate by default and you may be prompted to trust it.

To upload the image to the images-workspace namespace using the command line, use the virtctl image-upload command.

$ virtctl image-upload dv rhel8.3 \
    --image-path rhel-8.3-x86_64-kvm.qcow2 \
    --size 11Gi \
    --namespace images-workspace \
    --insecure

The command uploads the image to a DataVolume (DV). This is ok, since we ultimately just need the PV.

The --insecure option is needed if the CA server used to sign the certificate of the cluster is not configured on the local system. This is usually the case since the cluster uses a self signed certificate.

Customize the Image

If the uploaded image needs to be customized, now is the time to do it. It is beyond the scope of this blog post to discuss customizations. But here are some brief guidelines on how to do it.

• Create a new VM in the images-workspace project namespace using the uploaded image or a clone of the uploaded image as the VMs disk.

• Power on the VM and make any customizations desired, such as updating or installing packages. Install the guest-agent as well.

• Power off the VM.

• Remove the VM. Make sure to uncheck the [Delete Disks] checkbox in the Delete Virtual Machine dialog box.

Update the Boot Source Image

We need the rhel8 PVC to point to the PV that was created when the rhel8.3 image was uploaded and modified. Unfortunately we cannot simply modify the current rhel8 PVC to point to the updated rhel8.3 PV. Instead, we will need to remove the DV, PVC, and PV associated with the current Boot Source image. We will then create a new PVC called rhel8 in the openshift-virtualization-os-images project namespace and have it bind to the updated rhel8.3 PV. We do not need to create a DV. This binding of the PV is possible because PVs exist in all project namespaces and are not specific to a project namespace.

To help make sure the PVC we will create will bind to the existing PV, we need to configure the PV to point to a PVC called rhel8 in the openshift-virtualization-os-images project namespace and we need to also tell the new PVC we are creating that it needs to bind to a specific PV. If both of these are done correctly, the PVC should bind to the PV quickly and cause less of an impact to the users.

Patch the PV

For the new PV that was created when the image was uploaded and/or customized, the spec.claimRef.name must be changed to rhel8, the spec.claimRef.namespace must be changed to openshift-virtualization-os-images, and the spec.claimRef.uid must be cleared. The spec.persistentVolumeReclaimPolicy should be changed to Retain. This allows the PVC to be removed without the PV getting removed as well.

This change is done, using an administrative user account, by editing the YAML configuration for the PV. Navigate to [Storage ⮞ Persistent Volumes] and select [Edit Persistent Volume] from the kebab menu to the right of the new PV.

Locate the settings in the YAML file and change each appropriately. Make sure to select Save the new configuration.

Once this change is made, the PV will show that its claim is rhel8. Even though it is shown this way, it is not yet bound to the rhel8 PVC. The PVC will show a Status of Lost, but this is fine as it will be removed later.

The PV can also be patched from the command line using the oc patch pv command.

$ oc patch pv pvc-23f39fa6-86f4-49d4-a896-3c17b3bc5b3c \
  -p '{"spec": {
     "persistentVolumeReclaimPolicy":"Retain",
     "claimRef":{
         "name":"rhel8",
         "namespace":"openshift-virtualization-os-images",
         "uid":""
     }}}'

persistentvolume/pvc-23f39fa6-86f4-49d4-a896-3c17b3bc5b3c patched

Recreate the PVC

Once a PVC is created, its configuration is immutable and cannot be modified. Because of this, the existing rhel8 PVC for the Boot Source image needs to be removed and a new one created. The web interface does not currently have GUI configuration for the fields needed, but a YAML file describing the new PVC can be entered in the web interface.

The following YAML file can be used to create the new PVC. The fields metadata.name and metadata.namespace must be set to rhel8 and openshift-virtualization-os-images, respectively. spec.resources.requests.storage is set to the size of the PV, it is 11Gi in this case. spec.volumeName must be specified and point to the name of the updated PV.

Once this YAML is applied, it will create a new PVC that will quickly bind to the PV.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rhel8
  namespace: openshift-virtualization-os-images
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 11Gi
  volumeName: pvc-23f39fa6-86f4-49d4-a896-3c17b3bc5b3c

Before this PVC can be created, the existing DV (if it exists) and PVC must be removed. If a DV exists, remove it and the PVC will be removed automatically. Removing a PVC that was created by a DV will cause the DV to try and recreate the PVC. If there is no DV associated with the rhel8 PVC, then just remove the PVC.

To remove the DV and PVC using the web interface, navigate to [Storage ⮞ Persistent Volume Claims] and select the kebab menu to the right of the rhel8 PVC. Select [Delete Persistent Volume Claim] to remove the PVC and any DV that might have created it.

To remove the DV and PVC from the command line, use the oc delete dv command. If no DV exists for the PVC, then delete the PVC using the oc delete pvc command.

$ oc delete dv rhel8 -n openshift-virtualization-os-images

Once the DV and PVC are removed, create the new PVC. To create the PVC using the web interface, navigate to [Storage ⮞ Persistent Volume Claims ⮞ Create Persistent Volume Claim ⮞ With Form].

Select the [Edit YAML] link, paste the YAML into the editor and select [Create].

To create the PVC from the command line, place the YAML into a file and use the oc create command to create the PVC.

$ oc create -f pvc-rhel8-bind-to-pv.yaml

Patch the PVs Reclaim Policy

The PVs spec.persistentVolumeReclaimPolicy should be set back to Delete. This will make sure it is removed when the PVC is deleted and help prevent orphaned PVs from consuming storage.

As before, Navigate to [Storage ⮞ Persistent Volumes] and select [Edit Persistent Volume] from the kebab menu to the right of the PV. Locate the spec.persistentVolumeReclaimPolicy setting and set it to Delete. Make sure to select save the new configuration.

To patch the PVs configuration using the command line, use the oc patch command.

$ oc patch pv pvc-23f39fa6-86f4-49d4-a896-3c17b3bc5b3c \
  -p '{"spec": {"persistentVolumeReclaimPolicy":"Retain"}}'

persistentvolume/pvc-23f39fa6-86f4-49d4-a896-3c17b3bc5b3c patched

Delete the DV and PVC

As before, remove the PVC for the image that was uploaded by navigate to [Storage ⮞ Persistent Volume Claims] and selecting the kebab menu to the right of the PVC. Select [Delete Persistent Volume Claim]. This will remove the PVC and any DV associated with it.

The PVC and DV can also be removed from the command line. Removing the DV will also remove the PVC. Please note that if the PVC is removed first, the DV will try to recreate it.

$ oc delete dv rhel8.3 -n images-workspace

Conclusion

The Boot Source image should now be updated. Any VMs that were created using the rhel8 Boot Source image but had not yet cloned the image should start cloning the new image. If any new VMs created during the update process do not automatically start the cloning process, simply removing the PVC associated with the VM should cause the PVC to be recreated and the cloning process to start.