Back up and restore application persistent data with Red Hat Advanced Cluster Management for Kubernetes policies

Introduction

Red Hat Advanced Cluster Management for Kubernetes (RHACM) offers the ability to manage fleets of Kubernetes and OpenShift clusters by defining two main types of clusters: 

• Hub clusters: The main clusters with RHACM installed. It is used to create, manage, and monitor the managed clusters and deploy applications and workloads to these clusters. 
• Managed clusters: The clusters managed by the hub clusters. Applications and workloads run on the managed clusters.

RHACM provides a business continuity scenario for the hub cluster using the backup and restore component, but this only covers disaster recovery scenarios for the hub cluster content. The following question is the challenge:

If the managed cluster goes offline or the application fails, how do you provide a disaster recovery scenario for stateful applications running on managed clusters?

Read this blog to answer this question and learn to build a business continuity solution for stateful applications running on managed clusters using RHACM policies.

Back up and restore application data with RHACM policies

The policies discussed in this blog provide backup and restore support for stateful applications running on managed clusters or the hub cluster by using Velero to back up and restore the application data. Install Velero by using the OpenShift APIs for Data Protection (OADP) operator on each of the managed clusters where the applications are running.

This blog explains how to install the backup and restore policies on the hub cluster and place these policies on the managed clusters for them to receive backup or restore application data.

In the RHACM policy collection from the community folder, three RHACM application backup policies are available, grouped in two policy sets. Find the description in the List of PolicySets section from the ACM Persistent data backup and restore, which list descriptions of the policy sets containing policies.

The policy sets and policies appear on the hub cluster as shown in the following images:

Prerequisites

• RHACM Operator version 2.7 or newer must be installed on your hub cluster.
• The MultiClusterHub resource must be created and display the status of Running.

Note: The MultiClusterHub resource is automatically created when you install RHACM Operator version 2.7 or newer.

• The RHACM cluster backup and restore operator is not required to be enabled if you just plan to use the backup policies to backup application data. The cluster backup and restore operator only backs up hub cluster content.

Back up and restore the Pacman application

You can use the backup and restore policies to manage the Pacman sample application data. You need a hub cluster managing two clusters: demo-managed-0 and demo-managed-1. The demo-managed-0 cluster is where the Pacman application is deployed and running, and the demo-managed-1 cluster is where the Pacman application is restored.

See the following overview of steps to back up and restore the application data:

1. Create the Pacman sample on the hub cluster using an application subscription and place it on the demo-managed-0 managed cluster. After creating the sample, you can play the application running on the demo-managed-0 cluster before saving the data to store it in the MongoDB database for the Pacman application.
2. Use the backup policy to back up the Pacman application data running on the demo-managed-0 cluster.
3. Install the restore policy on the demo-managed-1 cluster to restore the data from the application backed up on the demo-managed-0 cluster. 

The following sections explain each of the previous steps in more detail.

Create Pacman on the hub cluster and deploy on demo-managed-0

Create the Pacman sample on the hub cluster using an application subscription and place it on the demo-managed-0 cluster using a Placement resource. 

To create and place the application from the hub cluster using the application subscription, navigate to the Applications page from the RHACM console to create an application subscription. See the following image:

Choose the Git repository option and select the Git URL for the Pacman sample, then select demo-managed-0 for the Placement rule.

After creating the application, verify it exists on the demo-managed-0 cluster. See the following image:

Access the Pacman application using the Route resource installed on the demo-managed-0 cluster. See the following image:

Play the Pacman application and save the data. You can use the saved data to validate that the persistent data is backed up and restored on demo-managed-1 at a later time. 

See the following images:

Backup Pacman on demo-managed-0

After playing the Pacman game, back up the data. You can use the backup policy to create a backup schedule on the demo-managed-0 cluster by completing the following steps:

1. Install the acm-app-backup policy set on the hub cluster by applying all policy resources as described in the Apply policies on the hub section.

2. Install the hdr-app-configmap ConfigMap to set the connection to the backup storage and configure the policy:

    2.1 Update the dpa.aws.backup.cloud.credentials property and set your storage credentials where you want to save the backup. 

    2.2 Update the dpa.spec property and set the storage configuration. The following configuration example displays the dpa.spec property. It uses restic for data backup and an Amazon Web Services (AWS) storage bucket named vb-velero-backup.

Note: The backup is saved in the pacman-test folder in this bucket. If you want to use PV Snapshots instead of restic, use the hdr-app-configmap ConfigMap from the pv-snap folder. Read the Usage Considerations section from the policy-collection repository to learn more about which option to use.

    2.3. Update and set the backup.nsToBackup property to back up pacman-ns, which is the namespace name where you created the Pacman application on the hub cluster. See the following example: backup.nsToBackup: '["pacman-ns"]'

    2.4. Change the backup.prefix property to backup.prefix: pacman. This is used as a prefix for the Velero backup names.

    2.5. For all other ConfigMap properties, you can keep the default values unless you want to change them. Read the Policies input data using hdr-app-configmap section to understand what each property does.

 The following file is an example of a dpa.spec configuration using AWS storage to back up the data. If you use a different type of storage, update the data.spec to match your configuration. Refer to Velero supported storage providers to learn about all of the configurable parameters of other storage providers:

    dpa.spec: "{
  \"backupLocations\": [
    {
      \"velero\": {
        \"config\": {
          \"profile\": \"default\",
          \"region\": \"us-east-1\"
        },
        \"credential\": {
          \"key\": \"cloud\",
          \"name\": \"cloud-credentials\"
        },
        \"default\": true,
        \"objectStorage\": {
          \"bucket\": \"vb-velero-backup\",
          \"prefix\": \"pacman-test\"
        },
        \"provider\": \"aws\"
      }
    }
  ],
  \"configuration\": {
    \"restic\": {
      \"enable\": true
    },
    \"velero\": {
      \"defaultPlugins\": [
        \"openshift\",
        \"aws\"
      ],
      \"podConfig\": {
        \"resourceAllocations\": {
          \"limits\": {
            \"cpu\": \"2\",
            \"memory\": \"1Gi\"
          },
          \"requests\": {
            \"cpu\": \"500m\",
            \"memory\": \"128Mi\"
          }
        }
      }
    }
  }
}"

3. Now that the acm-app-backup policy set is installed on the hub cluster, place the policy set on the demo-managed-0 cluster by adding the acm-pv-dr=backup label to the demo-managed-0 cluster, so that the backup PolicySet resource is placed on this cluster.

4. Notice that the acm-app-backup policy set is successfully installed on demo-managed-0 in the following image:

5. Verify that the oadp-hdr-app-install install policy and the oadp-hdr-app-backup backup policy are successfully placed on the demo-managed-0 cluster, as shown in the following image. Make sure there are no violations on the oadp-hdr-app-install and oadp-hdr-app-backup policies that are installed by the acm-app-backup policy set:

6. Notice the backup named pacman-restic-demo-managed-0-20230504185419, shown by the check-backup-completed template message in the previous image. This is the last backup generated by the backup schedule and can be used to restore the Pacman application on the demo-managed-1 cluster. 

Note: By default, the backupSchedule parameter in the hdr-app-configmap ConfigMap is set to run at the beginning of every hour (backupSchedule: 0 */1 * * *). It keeps the backups for two days (backupTTL: 240h0m0s). You can update the cron job and the backup expiration time to fit your own scenario.

Restore Pacman on demo-managed-1

To restore the Pacman application on the demo-managed-1 cluster, complete the following steps:

1. Make sure the acm-app-restore policy set is installed on the hub cluster in the same namespace as the previous acm-app-backup policy set. The acm-app-restore policy set should already be installed by step 2 in the previous section. The connection to the storage location was set by the hdr-app-configmap ConfigMap and was also created in step 2 of the previous section.

2. Add the acm-app-restore=pacman-restic-demo-managed-0-20230504185419 label to the demo-managed-1 cluster so that the acm-app-restore policy set is placed on this cluster. The name of the backup on your hub cluster is different than the one previously specified. Use the name of the backup as it appears on your hub cluster. See the following image:

3. As a result of the acm-app-restore=pacman-restic-demo-managed-0-20230504185419 label being added to the cluster, the content of the acm-app-restore policy set is applied to the demo-managed-1 cluster. See the following image:

4. Verify that the oadp-hdr-app-install install policy and oadp-hdr-app-restore restore policy are successfully placed on the demo-managed-1 cluster. There should be no violations on the oadp-hdr-app-install and oadp-hdr-app-restore policies installed by the acm-app-restore policy set, as shown in the following image. Verify that the check-restore-completed template has no violations, which indicates that the restore operation for the pacman-restic-demo-managed-0-20230504185419 backup is complete and there are no errors to report:

5. Verify that the Pacman policy route is now available on the demo-managed-1 cluster and that you can view the previously saved data when accessing the application. See the following image:

Usage considerations

• When the policies are deleted from the hub cluster, the pruneObjectBehavior: DeleteIfCreated option is used in the backup and restore policies to clean up resources created by the backup and restore policies. This allows the backup components installed on the managed clusters to be cleaned up when the owning policies are deleted from the hub cluster. Note that the Velero Backup data is not deleted from the backup storage location when the Backup policy is deleted because the policy does not own the backup resource. The Velero Backup resource is owned by the Velero Schedule resource, which is created by the backup policy. The Velero Restore resource is deleted from the managed cluster if the restore policy is removed from the hub cluster, but the resources restored by this resource on the managed cluster are not affected by the delete operation.
• Both the oadp-hdr-app-backup and oadp-hdr-app-restore policies are set to depend on the oadp-hdr-app-install policy; therefore, the policies do not start installing until the oadp-hdr-app-install policy installed in the same namespace is deployed without violations. This verifies that the OADP operator is properly installed and configured on the managed cluster before trying to back up or restore resources. 
• Complete the following steps if you want to install the backup policies on the hub cluster and want to use the same OADP operator installation for both the RHACM cluster backup and restore operator and the application backup policies:

1. Enable the cluster-backup operator on the hub cluster by editing the MultiClusterHub resource and setting the cluster-backup option to true. This installs the RHACM cluster backup and restore operator and the OADP operator in the open-cluster-management-backup namespace. 

2. Install the backup policies as shown in this blog, and set the backupNS=open-cluster-management-backup parameter when configuring the policy settings by using the hdr-app-configmap ConfigMap.

Note: If you use the open-cluster-management-backup namespace for the backupNS hdr-app-configmap property, and the cluster-backup option is set to false on the MultiClusterHub, MultiClusterHub reconciles and deletes the open-cluster-management-backup namespace.

• Read the Usage considerations section from the policy-collection repository to understand what backup setup is suitable for you, based on your cluster and your application configurations. 

Create multiple backup configurations

Read the Create multiple backup configurations section to understand how to back up numerous applications in separate backups, or use a different backup configuration for a subset of clusters.

Conclusion

This blog showed you how to build a business continuity solution for stateful applications running on managed clusters using RHACM policies. It provided examples on how to use the RHACM policy sets, acm-app-backup and acm-app-restore to back up and then restore the Pacman application data.

References

• Backup and Restore Policies for application persistent data
• OpenShift APIs for Data Protection (OADP) operator
• RHACM Hub Backup and Restore - documentation
• Backup and Restore Hub Clusters - blog
• Pacman sample