Red Hat blog

How to Backup and Restore ACM with OADP and MinIO

June 13, 2022Leonardo Araujo

This article will demonstrate how to backup and restore Red Hat ACM 2.5 using the OADP solution and the MinIO from the object store.

First, let's understand what are the tools that we will use in this article:

About

Red Hat Advanced Cluster Management (ACM), offers end-to-end visibility and control for managing your cluster and application lifecycle. Among other features, it ensures security and compliance for your entire Kubernetes domain across multiple data centers and public clouds.
OpenShift API for Data Protection (OADP) enables backup, restore, and disaster recovery of applications on an OpenShift cluster. Data that can be protected with OADP include Kubernetes resource objects, persistent volumes, and internal images.
MinIO, is an open source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality.

This article will not cover the installation of ACM.

Prerequisites

Red Hat OpenShift Container Platform with ACM
Server rhel 8
- 8 vcpu
- 16GB ram
- 50GB disk of OS
- 50GB disk of DATA
- podman

Install MinIO Server

Install podman on server:

[root@minio-server ~]# dnf install @container-tools -y

Let's configure our mount point with an LVM volume, we will use the sdb disk as follows:

[root@minio-server ~]# lsblk
NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                      8:0    0   50G  0 disk 
├─sda1                   8:1    0  600M  0 part /boot/efi
├─sda2                   8:2    0    1G  0 part /boot
└─sda3                   8:3    0 48.4G  0 part 
  ├─rhel_isolated-root 253:0    0 43.4G  0 lvm  /
  └─rhel_isolated-swap 253:1    0    5G  0 lvm  [SWAP]
sdb                      8:16   0   50G  0 disk 
sr0                     11:0    1 1024M  0 rom

Let's configure our pv, vg, and lv per the following commands:

[root@minio-server ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

[root@minio-server ~]# vgcreate vg_minio-data /dev/sdb
  Volume group "vg_minio-data" successfully created

[root@minio-server ~]# lvcreate -n lv_minio-data -l 100%FREE vg_minio-data
  Logical volume "lv_minio-data" created.

Now let's format our lvm as xfs using the mkfs.xfs command:

[root@minio-server ~]# mkfs.xfs /dev/mapper/vg_minio--data-lv_minio--data
meta-data=/dev/mapper/vg_minio--data-lv_minio--data isize=512    agcount=4, agsize=3276544 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=13106176, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=6399, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Now let's create the directory that we will use to make our mount point:

[root@minio-server ~]# mkdir -pv /minio/data
mkdir: created directory '/minio'
mkdir: created directory '/minio/data'

Before performing the mount, let's adjust the contexts of this directory; using the restorecon command, run the following command:

[root@minio-server ~]# restorecon -RFvv /minio
Relabeled /minio from unconfined_u:object_r:default_t:s0 to system_u:object_r:default_t:s0
Relabeled /minio/data from system_u:object_r:unlabeled_t:s0 to system_u:object_r:default_t:s0

Let's also install tuned and apply a configuration profile to improve our performance:

[root@minio-server ~]# dnf install tuned.noarch tuned-utils.noarch tuned-utils-systemtap.noarch -y

To list available profiles, run the following command:

[root@minio-server ~]# tuned-adm list
Available profiles:
- accelerator-performance     - Throughput performance based tuning with disabled higher latency STOP states
- balanced                    - General non-specialized tuned profile
- desktop                     - Optimize for the desktop use-case
- hpc-compute                 - Optimize for HPC compute workloads
- intel-sst                   - Configure for Intel Speed Select Base Frequency
- latency-performance         - Optimize for deterministic performance at the cost of increased power consumption
- network-latency             - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance
- network-throughput          - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks
- optimize-serial-console     - Optimize for serial console use.
- powersave                   - Optimize for low power consumption
- throughput-performance      - Broadly applicable tuning that provides excellent performance across a variety of common server workloads
- virtual-guest               - Optimize for running inside a virtual guest
- virtual-host                - Optimize for running KVM guests
Current active profile: balanced

Let's define the throughput-performance profile with the following command:

[root@minio-server ~]# tuned-adm profile throughput-performance

[root@minio-server ~]# tuned-adm active
Current active profile: throughput-performance

Now let's mount our /minio/data directory. For that, add the following line in your /etc/fstab:

[root@minio-server ~]# tail -n2 /etc/fstab 
# MINIO DATA
/dev/mapper/vg_minio--data-lv_minio--data /minio/data xfs defaults,noatime,nodiratime 1 2

Now to mount, run the following command:

[root@minio-server ~]# mount -v /minio/data
mount: /dev/mapper/vg_minio--data-lv_minio--data mounted on /minio/data.

Now let's create the mini user and define its home:

[root@minio-server ~]# useradd -s /sbin/nologin -d /minio minio
[root@minio-server ~]# chown -R minio:minio /minio

Now let's run our MinIO container:

podman run \
  --detach \
  -p 9000:9000 \                           <---- MinIO API Port
  -p 9001:9001 \                           <---- MinIO Console Port
  --user $(id -u minio):$(id -g minio) \   
  --name minio-server \
  -e "MINIO_ROOT_USER=miniouseradmin" \    <---- MinIO User API & Console
  -e "MINIO_ROOT_PASSWORD=miniouserpass" \ <---- MinIO Pass API & Console
  -v /minio/data:/data:z \                 <---- Data Volume Mount
  quay.io/minio/minio server /data --console-address ":9001"

Change the value of `minio_root_user` and `minio_root_password`.

Now we can visualize our minio running by running the following command:

[root@minio-server ~]# podman ps
CONTAINER ID  IMAGE                       COMMAND               CREATED         STATUS             PORTS                             NAMES
d69929bc4701  quay.io/minio/minio:latest  server /data --co...  11 seconds ago  Up 11 seconds ago  0.0.0.0:9000-9001->9000-9001/tcp  minio-server

Now, to configure our container as a service managed by systemd, let's do the following: create the following directory structure:

[root@minio-server ~]# mkdir -pv .config/systemd/user
mkdir: created directory '.config'
mkdir: created directory '.config/systemd'
mkdir: created directory '.config/systemd/user'

[root@minio-server ~]# cd ~/.config/systemd/user/

With the following command, our file service will be generated, to control the start, stop, restart, and status of our container:

[root@minio-server user]# podman generate systemd --name minio-server --files --new 
/root/.config/systemd/user/container-minio-server.service

[root@minio-server user]# systemctl --user daemon-reload

Now let's enable our service so it can start along with the OS and validate the status of our service:

[root@minio-server user]# systemctl --user enable container-minio-server.service 
Created symlink /root/.config/systemd/user/multi-user.target.wants/container-minio-server.service → /root/.config/systemd/user/container-minio-server.service.
Unit /root/.config/systemd/user/container-minio-server.service is added as a dependency to a non-existent unit multi-user.target.
Created symlink /root/.config/systemd/user/default.target.wants/container-minio-server.service → /root/.config/systemd/user/container-minio-server.service.


[root@minio-server user]# systemctl --user status container-minio-server.service 
● container-minio-server.service - Podman container-minio-server.service
   Loaded: loaded (/root/.config/systemd/user/container-minio-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-05-05 17:23:20 -03; 15s ago
     Docs: man:podman-generate-systemd(1)
  Process: 31407 ExecStartPre=/bin/rm -f /run/user/0/container-minio-server.service.ctr-id (code=exited, status=0/SUCCESS)
 Main PID: 31532 (conmon)
    Tasks: 2 (limit: 101098)
   Memory: 2.2M
   CGroup: /user.slice/user-0.slice/user@0.service/container-minio-server.service
           └─31532 /usr/bin/conmon --api-version 1 -c b9e648d667dd599898f70e4eb6ad280e7a81c9bd06885841a88c9c890ca08491 -u b9e648d667dd599898f70e4eb6ad280e7a81c9bd06885841a88c9c890ca08491 -r /usr/bin/runc -b /va>
May 05 17:23:19 minio-server systemd[4546]: Starting Podman container-minio-server.service...
May 05 17:23:20 minio-server podman[31409]: d69929bc470164c82bc802069f7d1f31c3023bcdf48c9bdabbc1c4a1cb8d9f4f
May 05 17:23:20 minio-server systemd[4546]: Started Podman container-minio-server.service.
May 05 17:23:20 minio-server podman[31409]: b9e648d667dd599898f70e4eb6ad280e7a81c9bd06885841a88c9c890ca08491

Now let's adjust the firewalld and authorize access through our ports 9000 and 9001. For that, execute the following commands:

[root@minio-server user]# firewall-cmd --state
running
[root@minio-server user]# firewall-cmd --get-default-zone
public
[root@minio-server user]# firewall-cmd --zone=public --add-port=9000-9001/tcp --permanent
success
[root@minio-server user]# firewall-cmd --reload
success

Access the server ip on port 9001 (console); to authenticate, use the user and pass defined in the execution of the container:

After accessing the console, on the left-side menu, click Buckets and then on the Create Bucket button, set the name and click Create Bucket again.

After creating our bucket, we go to OpenShift to install the OADP.

Configuring OpenShift Advanced Cluster Management and OADP

Now that we have our OADP installed and configured, let's configure our ACM.

In the multiclusterhub we need to enable the backup. For that, let's run the following command to modify the component cluster-backup to enabled: true

[root@bastion OADP]# oc project open-cluster-management

[root@bastion OADP]# oc get multiclusterhub multiclusterhub -o yaml | grep "cluster-backup" -B1
    - enabled: false
      name: cluster-backup

[root@bastion OADP]# oc patch MultiClusterHub multiclusterhub \
> -n open-cluster-management --type=json \
> -p='[{"op": "add", "path": "/spec/overrides/components/-","value":{"name":"cluster-backup","enabled":true}}]'
multiclusterhub.operator.open-cluster-management.io/multiclusterhub patched

[root@bastion OADP]# oc get multiclusterhub multiclusterhub -o yaml | grep "cluster-backup" -B1
    - enabled: true
      name: cluster-backup

After we enable the cluster-backup parameter, we can see that a component called cluster-backup-chart-sub appears as Installed successfully.

- This component has just installed the operator OADP in the open-cluster-management-backup namespace, maximizing configuration steps.

[root@bastion OADP]# oc get multiclusterhub multiclusterhub -o yaml | grep "cluster-backup-chart-sub" -A4 -B2
status:
  components:
    cluster-backup-chart-sub:
      lastTransitionTime: "2022-06-23T19:10:23Z"
      reason: InstallSuccessful
      status: "True"
      type: Deployed

Now let's create our credential secret, informing our MinIO username and password:

cat <<EOF > credentials
[default]
aws_access_key_id=miniouseradmin
aws_secret_access_key=miniouserpass
EOF

[root@bastion OADP]# cat credentials
[default]
aws_access_key_id=miniouseradmin
aws_secret_access_key=miniouserpass

To create the secret, run the following command, remembering that the name of the secret must be cloud-credentials:

[root@bastion OADP]# oc create secret generic cloud-credentials -n open-cluster-management-backup --from-file cloud=credentials
secret/cloud-credentials created

[root@bastion OADP]# oc -n open-cluster-management-backup extract secret/cloud-credentials --to=-
# cloud
[default]
aws_access_key_id=miniouseradmin
aws_secret_access_key=miniouserpass

⚠️ For greater security, create a dedicated user for this action through the MinIO console.

Let's now create our DataProtectionApplication instance. For that, run the following command:

cat <<EOF > oadp-instance.yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: oadp-minio
  namespace: open-cluster-management-backup
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: minio
          s3ForcePathStyle: 'true'
          s3Url: 'http://10.36.8.155:9000'
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: acm-backup
          prefix: velero
        provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
        - openshift
        - aws
        - kubevirt
  snapshotLocations:
    - velero:
        config:
          profile: default
          region: minio
        provider: aws
EOF

[root@bastion OADP]# oc -n open-cluster-management-backup create -f oadp-instance.yaml 
dataprotectionapplication.oadp.openshift.io/oadp-minio created

[root@bastion OADP]# oc -n open-cluster-management-backup get DataProtectionApplication oadp-minio
NAME         AGE
oadp-minio   12s

Now let's create our BackupSchedule

cat <<EOF > backup-schedule-acm.yaml
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: BackupSchedule
metadata:
  name: schedule-acm
spec:
  maxBackups: 3                 # Maximum number of backups after which old backups should be removed
  veleroSchedule: 0 */6 * * *   # Create a backup every 6 hours
  veleroTtl: 72h                # Deletes scheduled backups after 72h
EOF

[root@bastion OADP]# oc -n open-cluster-management-backup create -f backup-schedule-acm.yaml
backupschedule.cluster.open-cluster-management.io/schedule-acm created

[root@bastion OADP]# oc get BackupSchedule -n open-cluster-management-backup
NAME           PHASE     MESSAGE
schedule-acm   Enabled   Velero schedules are enabled

Once we create our BackupSchedule, the following Schedules will automatically be created: acm-credentials, acm-managed-clusters and acm-resources

[root@bastion OADP]# oc get Schedule -n open-cluster-management-backup
NAME                               AGE
acm-credentials-cluster-schedule   113s
acm-credentials-hive-schedule      113s
acm-credentials-schedule           114s
acm-managed-clusters-schedule      113s
acm-resources-generic-schedule     113s
acm-resources-schedule             113s
acm-validation-policy-schedule     113s

Description of each schedule

Credentials Backup
- This backup schedule contains three backup files for Hive, Red Hat Advanced Cluster Management, and user-created credentials.
Managed Clusters Backup
- This schedule contains only resources that activate the managed cluster connection to the hub cluster, where the backup is restored.
Resources Backup
- This backup schedule contains one backup for the Red Hat Advanced Cluster Management resources and one for generic resources. These resources use the following label, cluster.open-cluster-management.io/backup.

After a few minutes, our first backup completes successfully.

[root@bastion OADP]# oc get Backup -n open-cluster-management-backup
NAME                                              AGE
acm-credentials-cluster-schedule-20220624192134   12m
acm-credentials-hive-schedule-20220624192134      12m
acm-credentials-schedule-20220624192134           12m
acm-managed-clusters-schedule-20220624192134      12m
acm-resources-generic-schedule-20220624192134     12m
acm-resources-schedule-20220624192134             12m
acm-validation-policy-schedule-20220624192134     12m

We can also view our backup data directly from the MinIO console:

By clicking the Browse button, we can browse the created directories and files:

Data Loss Simulation

Now let's simulate the removal of some objects: Credentials, ClusterSet and Application. After deleting these objects, let's restore using our most recent backup. Let's delete the following objects:
- Credentials: rhocmcreds
- Applications: teste-acm-deploy
- ClusterSet: clusters-rhocm

Before:

After:

Restore Objects

First, let's understand the possible options for completing the following fields:

veleroManagedClustersBackupName
- latest <----- Restore the most recent Backup
- skip <----- Ignore the job, taking no action
- backup_name <----- Restore the specific backup name
veleroCredentialsBackupName
- latest
- skip
- specific backup name
veleroResourcesBackupName
- latest
- skip
- specific backup name

Let's now recover the deleted objects. For that, we'll perform the following steps below

cat <<EOF > acm-cluster-restore.yaml
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Restore
metadata:
  name: restore-acm
spec:
  cleanupBeforeRestore: CleanupRestored       <---- Clean up only resources created by a previous acm restore
  veleroManagedClustersBackupName: latest     <---- Restore our latest acm-managed-clusters-schedule
  veleroCredentialsBackupName: latest         <---- Restore our latest acm-credentials-schedule 
  veleroResourcesBackupName: latest           <---- Restore our latest acm-resources-schedule 
EOF

[root@bastion OADP]# oc create -f acm-cluster-restore.yaml

After creating and applying the restore yaml, let's run the following commands to view our restore job:

[root@bastion OADP]# oc get restores -n open-cluster-management-backup
NAME          PHASE     MESSAGE
restore-acm   Started   Prepare to restore, cleaning up resources


[root@bastion OADP]# oc get restores.velero.io -n open-cluster-management-backup
NAME                                                       AGE
restore-acm-acm-credentials-cluster-schedule-20220624192134   2m36s
restore-acm-acm-credentials-hive-schedule-20220624192134      2m36s
restore-acm-acm-credentials-schedule-20220624192134           2m36s
restore-acm-acm-managed-clusters-schedule-20220624192134      2m36s
restore-acm-acm-resources-generic-schedule-20220624192134     2m36s
restore-acm-acm-resources-schedule-20220624192134             2m36s

Follow through the following command, until the message appears like this example:

[root@bastion OADP]# oc get restores -n open-cluster-management-backup
NAME          PHASE                MESSAGE
restore-acm   FinishedWithErrors   Velero restores have run to completion but encountered 1+ errors

In this example, our restore was executed with failure, because with all backup resources created, there is no cluster, such as acm-credentials-cluster-schedule.

To view the restore jobs that have failed or partially failed status, we will execute the following commands:

# To list all restore jobs

[root@bastion OADP]# oc -n open-cluster-management-backup exec $(oc get pod -l app.kubernetes.io/instance=oadp-minio -o name) -c velero -- ./velero restore get
NAME                                                          BACKUP                                            STATUS            STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
restore-acm-acm-credentials-cluster-schedule-20220624192134   acm-credentials-cluster-schedule-20220624192134   PartiallyFailed   2022-06-24 20:16:48 +0000 UTC   2022-06-24 20:16:49 +0000 UTC   1        0          2022-06-24 20:16:44 +0000 UTC   <none>
restore-acm-acm-credentials-hive-schedule-20220624192134      acm-credentials-hive-schedule-20220624192134      PartiallyFailed   2022-06-24 20:16:47 +0000 UTC   2022-06-24 20:16:48 +0000 UTC   1        0          2022-06-24 20:16:44 +0000 UTC   <none>
restore-acm-acm-credentials-schedule-20220624192134           acm-credentials-schedule-20220624192134           Completed         2022-06-24 20:16:44 +0000 UTC   2022-06-24 20:16:45 +0000 UTC   0        0          2022-06-24 20:16:44 +0000 UTC   <none>
restore-acm-acm-managed-clusters-schedule-20220624192134      acm-managed-clusters-schedule-20220624192134      PartiallyFailed   2022-06-24 20:16:47 +0000 UTC   2022-06-24 20:16:47 +0000 UTC   1        0          2022-06-24 20:16:44 +0000 UTC   <none>
restore-acm-acm-resources-generic-schedule-20220624192134     acm-resources-generic-schedule-20220624192134     PartiallyFailed   2022-06-24 20:16:45 +0000 UTC   2022-06-24 20:16:45 +0000 UTC   1        0          2022-06-24 20:16:44 +0000 UTC   <none>
restore-acm-acm-resources-schedule-20220624192134             acm-resources-schedule-20220624192134             Completed         2022-06-24 20:16:45 +0000 UTC   2022-06-24 20:16:47 +0000 UTC   0        0          2022-06-24 20:16:44 +0000 UTC   <none>


# To see logs of restore job

[root@bastion OADP]# oc -n open-cluster-management-backup exec $(oc get pod -l app.kubernetes.io/instance=oadp-minio -o name) -c velero -- ./velero restore logs restore-acm-acm-credentials-cluster-schedule-20220624192134
time="2022-06-24T20:16:48Z" level=info msg="starting restore" logSource="pkg/controller/restore_controller.go:465" restore=open-cluster-management-backup/restore-acm-acm-credentials-cluster-schedule-20220624192134
time="2022-06-24T20:16:48Z" level=info msg="Starting restore of backup open-cluster-management-backup/acm-credentials-cluster-schedule-20220624192134" logSource="pkg/restore/restore.go:387" restore=open-cluster-management-backup/restore-acm-acm-credentials-cluster-schedule-20220624192134
time="2022-06-24T20:16:48Z" level=info msg="restore completed" logSource="pkg/controller/restore_controller.go:480" restore=open-cluster-management-backup/restore-acm-acm-credentials-cluster-schedule-20220624192134

# To visualize description of restore job

[root@bastion managedcluster]# oc -n open-cluster-management-backup exec $(oc get pod -l app.kubernetes.io/instance=oadp-minio -o name) -c velero -- ./velero restore describe restore-acm-acm-credentials-cluster-schedule-20220624192134
Name:         restore-acm-acm-credentials-cluster-schedule-20220624192134
Namespace:    open-cluster-management-backup
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs restore-acm-acm-credentials-cluster-schedule-20220624192134' for more information)

Started:    2022-06-24 20:16:48 +0000 UTC
Completed:  2022-06-24 20:16:49 +0000 UTC

Errors:
  Velero:   error parsing backup contents: directory "resources" does not exist
  Cluster:    <none>
  Namespaces: <none>

Backup:  acm-credentials-cluster-schedule-20220624192134

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Preserve Service NodePorts:  auto

This way we can troubleshoot our restore job.

We can check that the restore worked correctly by viewing the creation time in Credentials and Applications:

After performing the restore successfully, we can see in MinIO that a new directory has been created and inside we have the backups that were used in the restore:

References

About the author

Leonardo Araujo

Read full bio

Platform products

Try & buy

Featured cloud services

By category

By organization type

By customer

Services

Training & certification

Featured

Topics

Articles

More to explore

For customers

For partners

About us

Open source

Company details

Communities

Recommendations

Select a language

Select a language

How to Backup and Restore ACM with OADP and MinIO

This article will demonstrate how to backup and restore Red Hat ACM 2.5 using the OADP solution and the MinIO from the object store.

About

Prerequisites

Install MinIO Server

Configuring OpenShift Advanced Cluster Management and OADP

Description of each schedule

Data Loss Simulation

Restore Objects

References

About the author

Leonardo Araujo

Products

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links