Deploying Openshift Virtualization on ROSA with NetApp FSx storage.

OpenShift Virtualization is a feature of OpenShift that allows you to run virtual machines alongside your containers. This is useful for running legacy applications that can’t be containerized, or for running applications that require special hardware or software that isn’t available in a container.

In this tutorial, I’ll show you how to deploy OpenShift Virtualization on Red Hat OpenShift on AWS (ROSA) using the AWS NetApp FSx service (specifically NFS, not ISCSI or SAN) to provide resilience and live migration. I’ll show you how to create a ROSA cluster, deploy the OpenShift Virtualization operator, deploy the NetApp Trident Operator and create a virtual machine.

If you’re planning to deploy OpenShift Virtualization in a production environment, you should follow the official documentation and best practices.

Pre-requisites

• A ROSA Cluster (see Deploying ROSA HCP with Terraform )
• An AWS account with permissions to create FSx for ONTAP
• The git binary installed on your machine. You can download it from the git website .
• The virtctl binary installed on your machine. You can follow installation instructions on the OpenShift website .

Note: This guide re-uses environment variables from the Deploying ROSA HCP with Terraform guide. If you have an existing cluster, you’ll need to set them appropriately for the cluster.

Prepare the Environment

1. Run this these commands to set some environment variables to use throughout (Terraform commands need to be run in the directory you ran Terraform)

2. Create a bare metal machine pool

   Note: bare metal machines are not cheap, so be warned!

Provision FSx for ONTAP

1. Change to a directory to clone the FSx for ONTAP CloudFormation template

2. Create the CloudFormation Stack

Note: As of this writing, you may need to open the ./FSxONTAP.yaml file and edit the TridentIAMPolicy name if there are multiple Cloudformation Stack instances deployed, because this stack attempts to create an IAM policy with a hard-coded name and duplicate policy names are not allowed via the AWS API. The conflict is in the YAML file around line 299 and looks as follows: ManagedPolicyName: 'TridentIAMPolicy'. Until this is corrected in the upstream Git repo, the Cloudformation Stack cannot be run multiple times.

Note: Make sure generated FSX_ADMIN_PASS and SVM_ADMIN_PASS passwords are per Fsx password policy , otherwise following clodformation will fail.

• Fsx CloudFormation:

This can take some time, so we can go ahead and deploy the OpenShift Virtualization Operator while we wait.

Deploy the OpenShift Virtualization Operator

1. Deploy the OpenShift Virtualization Operator

2. If you want to see the progress of the operator you can log into the OpenShift Console (hint run oc whoami --show-console to get the URL)

   

3. Create an OpenShift Virtualization operand

   Note: this is all defaults, so will not support a lot of the more advanced features you might want such as live migration.

4. New “Virtualization” Section in the OpenShift Console

   Once the operator is installed you should see a new “Virtualization” section in the OpenShift Console (you may be prompted to refresh the page)

   

5. Close the popup window and click the “Download virtctl” button to download the virtctl binary.

Install and Configure the Trident CSI driver

1. Verify the cloudformation stack is complete

2. Get the FSx ONTAP filesystem id

3. Get the FSx Management and NFS LIFs

4. Add the NetApp Helm Repository

5. Install the Trident CSI driver

6. Make sure the trident pods are running

7. Create a secret containing the SVM credentials

8. Create a BackendConfig for the FSx ONTAP

Create a Virtual Machine

1. Create a project and a secret containing your public SSH key

2. Create a VM

3. Watch for the VM to be ready

4. SSH into the VM

   Note: Be sure you have a compatible virtctl binary otherwise this command may fail if you installed an incompatible version (e.g. brew install virtctl). Proper installation instructions will always be located in the OpenShift documentation ( https://docs.openshift.com/container-platform/4.15/virt/getting_started/virt-using-the-cli-tools.html) .

   Output:

5. Check what node the VM is deployed on

6. Live migrate the VM

7. Wait a moment, and check the node again

Congratulations! You now have a virtual machine running on OpenShift Virtualization on ROSA, and you’ve successfully live migrated it between hosts.

Cleanup

1. Delete the VM

2. Uninstall the Trident CSI driver

3. Delete the FSx Storage Volumes (except for “SVM1_root” volume)

4. Wait until the volumes are deleted

5. Delete the FSx for ONTAP stack

6. Wait for the stack to be deleted

7. Delete the ROSA HCP Cluster

   If you used the Terraform from the Deploying ROSA HCP with Terraform guide, you can run the following command to delete the cluster from inside the terraform repository: