Backup and Restore for Azure Red Hat OpenShift using OpenShift API for Data Protection

This guide outlines how to implement OpenShift API for Data Protection (OADP) for comprehensive backup and recovery for Azure Red Hat OpenShift (ARO) clusters using a storage account.

Overview of OADP

OADP provides robust disaster recovery solution, covering OpenShift applications, application-related cluster resources, persistent volumes. OADP is also capable of backing up both containerized applications and virtual machines (VMs). However,it is important to note that etcd and Operators are not covered under OADP’s disaster recovery capabilities.OADP support is provided to customer workload namespaces, and cluster scope resources.

OADP includes a built-in Data Mover feature that allows you to move Container Storage Interface (CSI) volume snapshots to a remote object store, such as Azure Blob Storage. This enables you to restore stateful applications from the remote store in the event of a cluster failure, accidental deletion, or data corruption. The Data Mover uses Kopia as the uploader mechanism to read snapshot data and write it to the repository

For additional information about OADP refer to documentation

Prerequisites checklist

Before starting, ensure you have:

• An ARO 4.14 cluster with cluster-admin access
• Configure EntraID for authentication
• Azure CLI
• OpenShift CLI
• Azure subscription with permissions to create storage accounts and resource groups

Step 1: Preparing Azure Resources

You must create the necessary Azure infrastructure to store the backup data. This involves setting up a dedicated resource group, a storage account, and a service principal with the correct permissions

1.1 Create environment variables

A resource group will contain the storage account, and the storage account will house a container for your backup files

1.2 Create resource group for storage account

1.3 Create Azure storage account

1.4 Create container (using Azure EntraID auth)

1.5 Create service principal

A service principal is required for OADP to securely access Azure resources without using personal credentials. It needs permissions to access both the storage account and the resources required for volume snapshots.

Create service principal with appropriate permissions to access both storage and snapshot resources

1.6 Get service principal values

1.7 Validating client ID and tenant ID values

1.8 Assign additional permissions for Storage account and ARO resource group for snapshot location

Step 2: Install the OADP Operator

The OADP Operator automates the management of Velero, plugins, and custom resources required for backup and restore operations. You can install it from OperatorHub in the OpenShift web console.

• Log in to your ARO cluster as a user with cluster-admin privileges.
• Navigate to OperatorHub and search for the OADP Operator.
• Install the operator in the openshift-adp namespace.
• Verify the installation by checking the operator pods and the ClusterServiceVersion (CSV) status. It may take a few minutes for the operator to be ready.

2.1 Connect to your ARO cluster.

Reference link

oc login $apiServer –username kubeadmin –password $kubevar

2.2 Install OADP operator 1.4 via OperatorHub

You must be logged in as a user with cluster-admin privileges. You can install operator via webconsole

Install operator in openshift-adp namespace

2.3 Verify operator installation

Wait for operator to be ready (may take 2-3 minutes)

Check operator pods

Example output:

Step 3: Configure OADP with Data Mover

After installation, you must configure OADP by creating a DataProtectionApplication (DPA) custom resource. This CR defines the backup storage location, volume snapshot location, and enables the Data Mover feature

3.1 Create cloud credentials secret

This will store secret of the service principal credentials, allowing OADP to authenticate with Azure

3.2 Create DataProtectionApplication (DPA)

This resource configures Velero and its plugins.

• backupLocations: Defines the Azure blob container as the destination for backups.
• snapshotLocations: Configures the provider for creating volume snapshots. For Azure, the CSI snapshot is stored within Azure’s disk infrastructure, not the blob container.
• defaultPlugins: Enables the openshift, csi, and azure plugins for full functionality.
• defaultSnapshotMoveData: true: Enables the Data Mover feature by default for all CSI snapshots.
• nodeAgent.enable: true: Deploys the node-agent daemonset, which is required for data movement operations

After applying the DPA, wait for it to reconcile and for all OADP-related pods (Velero, node-agent) to be running

Note: Important Considerations for Azure GovCloud and Private Endpoints While this procedure works with standard Azure, there are specific configurations required for other environments, such as Azure GovCloud or when using a private endpoint.

By default, OADP and Velero assume the standard public cloud endpoint. However, in specialized environments, this endpoint is different. OADP is flexible enough to handle this, but you must explicitly define the correct URI in manifest

What to Do:

You need to specify the correct URI for your storage account in the storageAccountURI field within the backupLocations section of your DPA manifest.

Example for Azure GovCloud:

For users operating in Azure GovCloud, the storageAccountURI will be different. Your spec block should be configured as shown below, using the .usgovcloudapi.net domain:

3.3 Wait for DPA to be ready

Check DPA status (wait for Reconcile Succeeded)

Check all OADP pods are running

Example output:

Step 4: Verify OADP Installation

To verify a successful installation of the OADP operator, you must check the status of the operator and its related resources in the OpenShift cluster. The verification can be performed using either the OpenShift web console or the oc command-line tool.

Confirm that OADP is correctly configured by checking the status of the BackupStorageLocation and VolumeSnapshotLocation resources

4.1 Check Backup storage location

Verify the backup storage location is available

Example output:

Describe the location and check that the Phase is “Available”

Example output:

4.2 Check Volume snapshot location

An Azure Disk CSI snapshot is stored within Azure’s disk infrastructure, not in blob storage bucket.The snapshotLocation is used to tell Velero where to interact with the underlying storage system’s snapshot APIs.

Verify the volume snapshot location is created

Example output:

4.3 Check details for volume snapshot

Step 5: Create test database for validation of backup and restore

To validate the backup and restore functionality, deploy a stateful application with persistent storage. The following example uses a PostgreSQL database with a PersistentVolumeClaim

5.1 Create test namespace

5.2 Apply the manifest to create a Deployment, PVC, and Service for PostgreSQL

5.3 Wait for the database pod to be ready

5.4 Add test data to the database to verify data integrity after restore

Step 6: Create and validate backup

Now, create a Backup custom resource to back up the test application’s namespace

6.1 Create backup resource

6.2 Monitor backup progress

Get the name of the latest backup

Check backup status of the backup object

For backups using the Data Mover, a DataUpload CR is created. You can monitor its status.phase field, which will transition from InProgress to Completed upon successful transfer of snapshot data to the remote object store

Describe the backup to check its phase and any errors

(Advanced) Check the data upload status for Data Mover backups

6.3 Verify backup in Azure

You can also verify that backup files have been created in your Azure Storage container

Note you need to have Storage Blob Reader Role assigned to container within storage account

Navigation Steps for setting IAM role for storage account

1. Go to Azure Portal - portal.azure.com
2. In the top search bar, type: aroprojectbackups (your storage account name)
3. Click on your storage account when it appears
4. In the left sidebar, navigate to Data storage → Containers
5. Click on project-backups container
6. Once inside the container, in the left sidebar click Access Control (IAM)
7. Click on the Role assignments tab
8. Look for “Storage Blob Data Reader” or “Storage Blob Data Contributor” roles

Verification of backup

1. From the same container (aro-project)
2. Click Overview in the left sidebar (or click “aro-project” in the breadcrumb)
3. You should see folders like:
   • backups (since you used prefix: backups in your DPA)
   • Inside might be: backup files, kopia data, etc.

What to Verify

• In the IAM view: Verify your user or service principal has “Storage Blob Data Reader” role
• In the container view: Look for a backups folder (this is your prefix from the DPA configuration)

The backup should show Phase: Completed before files appear in Azure.(refer 6.2)

Troubleshooting

If you don’t see the backup files yet, check if your backup completed successfully:

Step 7: Test restore (Optional Validation)

To simulate a disaster recovery scenario, delete the test project and then restore it from the backup

7.1 Delete the test project

7.2 Restore from backup

Create a Restore resource, referencing the backup name

7.3 Verify restore

Monitor the restore progress:

• Check the status of the Restore object.
• When restoring CSI volumes, a DataDownload CR is created. You can monitor its status.phase to track the data transfer from the object store back to the new volume.

Get name of latest restore

Check the restore status

(Advanced) Check the data download status

After the restore completes, verify the application pods are running

Verify that the test data exists in the restored database

Step 8: Configure Backup Schedules (Optional)

OADP allows you to create scheduled backups using a Schedule custom resource, which uses standard cron syntax. This example creates a daily backup at 1:00 AM with a 7-day retention period (ttl)

8.1 Create daily backup schedule

Step 9: Cleanup Steps

9.1 Delete OpenShift Resources

9.2 Delete Azure Service Principal and Role Assignments

9.3. Delete Storage Container and Storage Account

9.4. Verify Cleanup

Additional Note

Please note that this cleanup process DOES NOT delete your cluster nor the resource group of your cluster.