ROSA HCP Disaster Recovery with ACM and OpenShift GitOps

This guide demonstrates how to set up an active/passive disaster recovery pattern for applications running on ROSA HCP clusters using Red Hat Advanced Cluster Management (ACM) and OpenShift GitOps (ArgoCD). ACM handles cluster health monitoring, while ArgoCD deploys the application to the active cluster.

The pattern works as follows:

• The guide first deploys the application to the primary cluster, records the EFS path mapping, pre-stages static DR PVs, and then enables the warm DR application
• After that point, ArgoCD maintains the application on both primary and warm DR
• ACM monitors cluster health via klusterlet heartbeats
• A Placement resource determines the current active/failover target
• DNS cutover follows the PlacementDecision
• Applications are not moved during failover; both are already maintained by ArgoCD

Prerequisites

Before starting this guide, complete the Application Disaster Recovery on ROSA HCP guide. That guide sets up:

• EFS CSI Driver on both clusters
• S3 Cross-Region Replication for application data and backup buckets
• EFS replication from the primary to the DR Region

You need the environment variables from that guide still set in your shell:

• PRIMARY_CLUSTER_NAME, DR_CLUSTER_NAME
• PRIMARY_REGION, DR_REGION
• APP_BUCKET_PRIMARY, APP_BUCKET_DR
• APP_S3_ROLE_ARN_PRIMARY, APP_S3_ROLE_ARN_DR
• PRIMARY_EFS, DR_EFS

If you are starting a new shell session, re-run the environment variable steps from the DR infrastructure guide .

In addition to the shared infrastructure, this guide requires:

• A third ROSA HCP cluster for the ACM hub ($CLUSTER_ACM)
• AWS CLI, oc CLI, and rosa CLI configured
• For DNS-based failover:
  • A Route 53 public hosted zone
  • A custom domain or hostname where you are allowed to create DNS records
  • certbot
  • The certbot-dns-route53 plugin

If you follow the DNS failover or custom certificate workflow, verify the Route 53 Certbot plugin before starting those steps:

Environment Variables

The following variables carry over from the DR infrastructure guide : PRIMARY_CLUSTER_NAME, DR_CLUSTER_NAME, PRIMARY_REGION, DR_REGION, APP_BUCKET_PRIMARY, APP_BUCKET_DR, APP_S3_ROLE_ARN_PRIMARY, APP_S3_ROLE_ARN_DR, PRIMARY_EFS, DR_EFS, AWS_ACCOUNT_ID. Set the additional variables needed for this guide:

Update S3 IRSA Trust Policies

The DR infrastructure guide creates S3 IRSA roles with trust policies scoped to the dr-demo namespace. This guide deploys the application in the ${NAMESPACE} namespace, so the trust policies must be updated to allow service accounts from both namespaces:

Log into the ACM Hub Cluster

All resources in this guide are created on the ACM hub cluster unless otherwise noted.

Install ACM on the Hub Cluster

Install the Red Hat Advanced Cluster Management operator. Create the namespace, OperatorGroup, and Subscription:

Wait for the operator CSV to reach Succeeded:

Create the MultiClusterHub to deploy ACM components:

Wait for the MultiClusterHub to be ready. This can take several minutes:

Wait until the output shows Running.

Import Managed Clusters into ACM

Import each regional cluster so ACM can monitor and manage them.

Import the Primary Cluster

Create the ManagedCluster resource:

Log in to the primary cluster and get a token:

Log back in to the ACM hub, create the namespace, then create the auto-import secret:

Wait for the cluster to be imported and available:

Wait until AVAILABLE shows True:

Import the DR Cluster

Create the ManagedCluster resource:

Log in to the DR cluster and get a token:

Log back in to the ACM hub, create the namespace, then create the auto-import secret:

Wait for the cluster to be imported and available:

Verify both clusters are imported:

Create a ManagedClusterSet

Group the regional clusters into a ManagedClusterSet so they can be referenced as a single pool for placement decisions.

Before creating shared hub resources, verify that the names derived from ACM_PREFIX do not already exist:

Create the ManagedClusterSet:

Add only the primary cluster to the set. The DR cluster is added after the initial failover Placement selects the primary cluster:

Install OpenShift GitOps on the Hub

Install the OpenShift GitOps operator which provides ArgoCD:

Wait for the operator to install:

Wait until the PHASE shows Succeeded.

Grant the ArgoCD service accounts cluster-admin privileges. The application controller needs this to deploy resources to managed clusters, and the ApplicationSet controller needs it to read cluster secrets:

Get the ArgoCD admin password:

ArgoCD Health During Cluster Outages

ArgoCD uses the ACM-generated cluster secrets created by the application-manager addon. Those secrets connect through ACM’s cluster-proxy. During a managed-cluster worker outage, use ManagedClusterConditionAvailable and the failover PlacementDecision as the authoritative failover signals. ArgoCD sync and health remain useful for validating reachable clusters, but this guide does not depend on ArgoCD marking the failed primary application as Degraded.

Bind the ManagedClusterSet to GitOps

Create a ManagedClusterSetBinding in the openshift-gitops namespace to allow ArgoCD to use the cluster set:

Configure ACM Placement for Failover

Create the Placement that monitors cluster health. This Placement selects exactly one cluster from the ${CLUSTERSET_NAME} set. Because only the primary cluster was added to the set before this Placement is created, the initial decision is deterministic. After the DR cluster is added, the Steady prioritizer keeps the decision on the primary cluster while it remains healthy.

Wait until the initial PlacementDecision selects the primary cluster:

Now add the DR cluster to the set and verify the decision remains on the primary cluster:

The tolerationSeconds value is not the total failover time. Failover occurs after ACM detects the worker outage, updates ManagedClusterConditionAvailable, adds the unreachable or unavailable taint, the toleration period expires, and Placement reconciles the decision.

Register Managed Clusters with ArgoCD

Enable the application-manager addon on both managed clusters. This addon creates the ArgoCD cluster secrets that allow ArgoCD to deploy to managed clusters via the ACM cluster-proxy:

Wait for the addons to become available:

Wait until both clusters show True in the AVAILABLE column.

Create a Placement to select all clusters in the DR cluster set. This Placement is used only by the GitOpsCluster to register both clusters as ArgoCD deployment targets:

Create the GitOpsCluster resource:

Verify the clusters appear as ArgoCD cluster secrets:

You should see secrets for both managed clusters with the cluster set label:

The application-manager addon copies ManagedCluster labels to the cluster secrets, including the cluster.open-cluster-management.io/clusterset label.

Keep ACM-Managed Cluster Secrets

The application-manager addon creates ArgoCD cluster secrets for the managed clusters. These secrets are managed by the ACM gitopscluster controller and use ACM’s cluster-proxy URL in data.server. Do not patch the generated *-application-manager-cluster-secret objects to direct managed-cluster API endpoints; controller-owned fields can be reconciled back to the cluster-proxy URL.

This guide uses ManagedClusterConditionAvailable as the authoritative cluster outage signal and the failover PlacementDecision as the authoritative active/failover target. ArgoCD sync and health are used to validate application state when a target cluster is reachable.

Tune Lease Duration for Faster Failover Detection

By default, ACM checks the klusterlet heartbeat lease every 5 minutes. For a faster demo, reduce the lease duration to 10 seconds on both managed clusters.

Obtain a TLS Certificate (Optional)

If you want to serve the application on a custom domain with a valid TLS certificate, obtain one using Let’s Encrypt with the certbot-dns-route53 plugin. Certbot uses your AWS credentials to create a temporary TXT record in Route 53 for domain validation.

Important: Replace your-email@example.com with your actual email address.

Set the certificate directory:

Create the Primary ArgoCD Application

The ApplicationSet maintains the Phoenix application on both clusters after DR storage is pre-staged. Placement does not decide where the ApplicationSet deploys; Placement determines the active/failover target that DNS should follow.

Create the ApplicationSet with only the primary cluster first. This lets the primary PVCs dynamically provision EFS access points so you can record their root paths before any DR PVCs exist.

The YAML is built in segments to cleanly embed the multi-line PEM certificate and key. The first heredoc writes everything up to certificate: |, then sed appends the indented PEM content directly from the cert files, and a final heredoc closes the YAML.

Verify the primary Application is synced and healthy:

Wait until the primary application shows Synced and Healthy:

Log in to the primary cluster and verify the PVCs are bound before recording the EFS mapping:

Prepare DR Cluster for EFS Data Continuity

When ArgoCD dynamically provisions a PVC, the EFS CSI driver creates a new access point with a fresh subdirectory. The replicated data from the primary EFS lives under the original primary subdirectories. To ensure the warm DR application sees the replicated data, pre-create static PersistentVolumes on the DR cluster before the DR PVCs exist. These PVs use claimRef pre-binding so the DR PVCs bind to the intended replicated paths instead of dynamically provisioning empty directories.

The demo application uses 3 EFS-backed PVCs:

• shared-flight-data – shared volume mounted by the dashboard and flight recorder
• flight-data-flight-recorder-0 – StatefulSet replica 0
• flight-data-flight-recorder-1 – StatefulSet replica 1

Record the PVC-to-path mapping

Log in to the primary cluster, then map each PVC to its EFS access point path. The PV volumeHandle format is <efs-id>::<access-point-id>, and each access point has a root directory path where the PVC’s data is stored:

Pre-stage static PersistentVolumes on the DR cluster

Log in to the DR cluster, then create static PVs with claimRef pre-binding. The claimRef reserves each PV for a specific PVC so that when ArgoCD deploys the application, the PVCs bind to these PVs instead of dynamically provisioning new access points:

Log back in to the ACM hub cluster.

Enable the Warm DR Application

After the DR static PVs exist, update the ApplicationSet to include the DR cluster. The ApplicationSet now maintains both applications, but Placement remains the source of truth for the active/failover target.

Verify that both Applications are synced and healthy:

Log in to the DR cluster and verify the DR PVCs bound to the pre-staged static PVs:

Expected volumes:

• shared-flight-data -> ${ACM_PREFIX}-dr-shared-flight-data
• flight-data-flight-recorder-0 -> ${ACM_PREFIX}-dr-flight-data-0
• flight-data-flight-recorder-1 -> ${ACM_PREFIX}-dr-flight-data-1

Set Up DNS

Create a Route 53 A record pointing to the router of the active cluster.

Get the router IP for each cluster by extracting the apps domain from the console URL:

Create the DNS record pointing to the primary cluster:

Verify the application is accessible:

Failover Test

Simulate a region failure by stopping the worker instances on the primary cluster.

Delete EFS replication to promote the DR replica to read-write:

Disable auto-repair on the primary cluster’s machine pools so ROSA does not replace the stopped workers, then stop the instances:

Log back in to the ACM hub cluster. Watch for ACM to detect the failure and for Placement to move the active decision to the DR cluster. With the tuned lease duration (10s) and toleration (30s), failover time is approximately ACM detection latency plus the 30-second toleration period and controller reconciliation:

Wait until the primary cluster shows Available: Unknown or False and the PlacementDecision shows only the DR cluster. ArgoCD should continue to report the reachable DR application as Healthy; the primary ArgoCD Application health is not the authoritative outage signal because ArgoCD reaches managed clusters through ACM’s cluster-proxy.

Because the application is already maintained on both clusters after DR storage pre-stage, no ArgoCD redeployment is needed. The DR cluster’s application is already running; DNS is switched to the cluster selected by the PlacementDecision.

Switch DNS to the DR cluster:

Flush local DNS cache and verify:

Failback

Failing back is a manual process. The Steady prioritizer in the health-monitoring Placement can keep the selection on the DR cluster after the primary recovers, preventing unnecessary flip-flopping.

Start the primary worker instances:

Wait for the primary cluster to rejoin ACM and for the nodes to be replaced. This typically takes 3-5 minutes:

Wait until the primary cluster shows Available: True and ${APP_NAME_PRIMARY} shows Synced and Healthy.

Re-establish EFS replication from primary to DR. Before allowing production traffic to return to the primary cluster, verify that the application is healthy and that any required DR-side EFS and S3 data has been reconciled. Application health alone is not sufficient to determine that a stateful workload is ready for failback.

Re-establish EFS replication so it is in place for future failovers. First, disable the overwrite protection that AWS enables on the replica after replication is deleted:

Switch DNS back to the primary cluster:

Flush local DNS cache and verify:

Failover Timeline Summary

Production Considerations

• Resource overhead: After DR storage is pre-staged, the application runs on both clusters. For resource-intensive applications, consider whether the cost of running on both clusters is acceptable. The trade-off is faster failover (DNS-only, no deployment delay) versus higher steady-state resource consumption.
• EFS path mapping: Record and maintain the PVC-to-EFS access point path mapping as part of your DR runbook. In a real disaster, the primary cluster API might not be available to query. Update this mapping whenever PVCs are recreated.
• Data reconciliation before failback: Both EFS and S3 replication are one-directional (primary to DR). Data written during failover must be manually synced or merged back to the primary before re-establishing replication. See the Disaster Recovery with OADP on ROSA HCP guide for detailed failback data reconciliation steps.
• ACM hub availability: The ACM hub is a single point of failure for failover detection. In production, deploy the hub with high availability or consider an active-passive hub configuration.
• DNS automation: Replace the manual DNS switch with Route 53 health checks and failover routing policies for fully automated DR.
• Lease duration tuning: The 10-second lease used in this guide is aggressive. For production, balance detection speed against the risk of false positives from transient network issues. A 60-second lease is a reasonable starting point.
• EFS mount targets: Ensure the DR cluster has EFS mount targets in all worker subnets before a disaster occurs. Creating mount targets during a failover adds delay to the recovery process.

Cleanup

From the ACM hub cluster, delete the ApplicationSet and hub-side namespace:

The ApplicationSet deletion removes the ArgoCD Applications, which prunes resources on the managed clusters. If a managed cluster was unreachable during cleanup, log in to that cluster and delete the namespace manually:

From the DR cluster, delete the static PVs:

Log back in to the ACM hub cluster.

Delete the Placements and GitOpsCluster:

Delete the ManagedClusterSetBinding and ManagedClusterSet:

Detach the managed clusters:

Delete the DNS record. The record value may point to either cluster depending on whether failback was completed:

Shared Infrastructure

After cleaning up the ACM-specific resources above, remove the shared DR infrastructure (EFS, S3, IAM, EFS CSI Driver) by following the Cleanup section in the parent guide.