Velero Backup and Restore of an Application Using gp2 StorageClass on ROSA

While Red Hat and Amazon Web Services (AWS) manage the ROSA service, the customer has certain responsibilities, including managing their data and their applications.

In a situation where the customer accidentally removes their application on OpenShift, there is no way to recover the application data unless it’s been backed up by the customer. In this post we will demonstrate how to back up an  application project using Velero Backup. The OADP (OpenShift API for Data Protection) operator will be used to set up and install Velero on the OpenShift platform.

Steps to test the backup and recovery of a WordPress application running on ROSA:

1. Deploy a ROSA cluster
2. Install the OADP Operator
3. Create credentials secret
4. Install Velero using OADP
5. Deploy a WordPress application
6. Create a Backup
7. Delete the application
8. Restore from the backup 

Install ROSA

Install the ROSA cluster by following the documentation.

Install the OADP Operator

Install the OADP Operator from OperatorHub. After selecting the defaults and clicking on Install, the OADP Operator will be installed in the oadp-operator project (namespace) as shown in the screenshot below.

Create credentials secret

Create a secret for the AWS credentials, which is used by Velero to access the Velero S3 bucket in your account. Use the “oc create secret” command as shown below to create the secret, where cloud-credentials.txt is the name of the file where the credentials are saved. Make sure that the cloud-credentials.txt file follows the AWS credentials template.

oc create secret generic cloud-credentials --namespace oadp-operator --from-file cloud=cloud-credentials.txt

Install Velero using OADP

Next we will create a Velero instance as follows from the CLI

First, create a file called velero.yaml with your Velero configuration. A sample velero.yaml file is included below:

apiVersion: konveyor.openshift.io/v1alpha1
kind: Velero
metadata:
  name: example-velero
  namespace: oadp-operator
spec:
  olm_managed: true
  backup_storage_locations:
        - config:
            profile: "default"
            region: us-west-2
          credentials_secret_ref:
            name: cloud-credentials
            namespace: oadp-operator
          name: default
          object_storage:
            bucket: managed-velero-backups-1dc95bf4-3830-4231-904a-7782bd14f79e
            prefix: "velero"
          provider: aws
  default_velero_plugins:
        - aws
        - openshift
  enable_restic: true
  volume_snapshot_locations:
        - config:
            profile: default
            region: us-west-2
          name: default
          provider: aws

Then create that file by running the following command:

oc apply -f velero.yaml

Check to see if the Velero pods are up and running.

oc get pods -n oadp-operator

The output will look similar to this:

NAME                                             READY   STATUS        RESTARTS   AGE
oadp-default-aws-registry-774b459f89-c7977   1/1         Running   0              19s
oadp-operator-57985ccd67-mndhb                   1/1         Running   0              22h
restic-cws5x                                     1/1         Running   0              21s
restic-dpj29                                     1/1         Running   0              21s
restic-xw8n2                                     1/1         Running   0              21s
velero-655b4bbf95-jssmk                          1/1         Running   0              22s

Deploy a WordPress application

Deploy the WordPress application from https://kubernetes.io/docs/tutorials/stateful-application/mysql-wordpress-persistent-volume/ 

If the wordpress-mysql pod does not start, apply the fix mentioned here.

Update application files mysql-deployment.yaml and wordpress-deployment.yaml to use the gp2 StorageClass

mysql-deployment.yaml:
apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
        app: wordpress
spec:
  ports:
        - port: 3306
  selector:
        app: wordpress
        tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
        app: wordpress
spec:
  accessModes:
        - ReadWriteOnce
  storageClassName: gp2
  resources:
        requests:
          storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
        app: wordpress
spec:
  selector:
        matchLabels:
          app: wordpress
          tier: mysql
  strategy:
        type: Recreate
  template:
        metadata:
          labels:
            app: wordpress
            tier: mysql
        spec:
          containers:
          - image: mysql:5.6
            name: mysql
            env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            ports:
            - containerPort: 3306
              name: mysql
            volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
          volumes:
          - name: mysql-persistent-storage
            persistentVolumeClaim:
              claimName: mysql-pv-claim
wordpress-deployment.yaml:
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
        app: wordpress
spec:
  ports:
        - port: 80
  selector:
        app: wordpress
        tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
        app: wordpress
spec:
  accessModes:
        - ReadWriteOnce
  storageClassName: gp2
  resources:
        requests:
          storage: 20Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: wordpress
  labels:
        app: wordpress
spec:
  selector:
        matchLabels:
          app: wordpress
          tier: frontend
  strategy:
        type: Recreate
  template:
        metadata:
          labels:
            app: wordpress
            tier: frontend
        spec:
          containers:
          - image: wordpress:4.8-apache
            name: wordpress
            env:
            - name: WORDPRESS_DB_HOST
              value: wordpress-mysql
            - name: WORDPRESS_DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-pass
                  key: password
            ports:
            - containerPort: 80
              name: wordpress
            volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
          volumes:
          - name: wordpress-persistent-storage
            persistentVolumeClaim:
              claimName: wp-pv-claim
kustomization.yaml:
secretGenerator:
- name: mysql-pass
  literals:
  - password=test123
resources:
  - mysql-deployment.yaml
  - wordpress-deployment.yaml

Install the application using “oc apply -k ./” into the wordpress project, and expose the route for the service using “oc expose svc/wordpress”.

oc get pods,pvc,svc,route -n wordpress
NAME                                      READY   STATUS        RESTARTS   AGE
pod/wordpress-6c77879668-vxkxt            1/1         Running   0              11h
pod/wordpress-mysql-8599ff598-psbsf   1/1         Running   0              11h

NAME                                       STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound        pvc-0ce99737-ae50-4784-a2ee-7ea0d2776d9b   20Gi           RWO                gp2                11h
persistentvolumeclaim/wp-pv-claim          Bound        pvc-6241249b-2eab-4051-8bdb-948253c8c1fd   20Gi           RWO                gp2                11h

NAME                          TYPE               CLUSTER-IP          EXTERNAL-IP                                                                   PORT(S)            AGE
service/wordpress             LoadBalancer   172.30.183.22   ad67d6c27c3434d41805205b38f2ecc4-1422739893.us-west-2.elb.amazonaws.com   80:30599/TCP   22h
service/wordpress-mysql   ClusterIP          None                <none>                                                                        3306/TCP           22h

NAME                                     HOST/PORT                                                       PATH   SERVICES        PORT   TERMINATION   WILDCARD
route.route.openshift.io/wordpress   wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com              wordpress   80                       None

Enter the URL wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com in a web browser and configure WordPress. Then create your first blog entry for testing.

Create a Backup

The backup.yaml file below instructs Velero to backup the namespace wordpress where our application is running.  The backup will be named backup-1.

apiVersion: velero.io/v1
kind: Backup
metadata:
  namespace: oadp-operator
  name: backup-1
spec:
  includedNamespaces:
        - wordpress

From within the wordpress namespace run the following command:

oc apply -f backup.yaml

To determine when the backup has completed,  check the Status Phase to see if it is marked as Completed

oc get backup -n oadp-operator
NAME           AGE
backup-1   12h

Check the status of the backup as follows

oc describe backup -n oadp-operator
…
…
…
Status:
  Completion Timestamp:  2021-09-01T18:26:10Z
  Expiration:                2021-10-01T18:25:26Z
  Format Version:            1.1.0
  Phase:                     Completed
  Progress:
        Items Backed Up:  82
        Total Items:          82
  Start Timestamp:        2021-09-01T18:25:26Z
  Version:                1
Events:                   <none>

Delete the application

To test that the backup was indeed successful, we will simulate a failure by deleting the wordpress project

oc delete project wordpress
project.project.openshift.io "wordpress" deleted

Quick check to see if the project has indeed been deleted

oc get pods,pvc -n wordpress
No resources found in wordpress namespace.

Restore from the backup 

Let restore the backup using Velero as follows using the restore.yaml shown below. Here we are calling our restore restore-1, and under the spec we specify the name of the backup that we want to use ie. backup-1. We specifically call out the from backup-1 restore the namespace wordpress.

apiVersion: velero.io/v1
kind: Restore
metadata:
  namespace: oadp-operator
  name: restore-1
spec:
  backupName: backup-1
  includeNamespaces:
        - wordpress

To restore the project run the following command:

oc apply -f restore.yaml
restore.velero.io/restore-1 created

Check if the restore can be seen under the oadp-operator namespace.

oc get restore -n oadp-operator
NAME            AGE
restore-1   33s

Check that restore status has indeed changed to Completed using the oc describe restore restore-5 -n oadp-operator command.

oc describe restore restore-1 -n oadp-operator
…
…
…
Status:
  Completion Timestamp:  2021-09-02T17:14:47Z
  Phase:                     Completed
  Progress:
        Items Restored:  52
        Total Items:         52
  Start Timestamp:   2021-09-02T17:14:05Z
  Warnings:              19
Events:                  <none>

Check if the pods, pvc, service,route names are seen under the wordpress namespace.

oc get pods,pvc,svc -n wordpress

NAME                                      READY   STATUS        RESTARTS   AGE
pod/wordpress-6c77879668-vxkxt            1/1         Running   0              15m
pod/wordpress-mysql-8599ff598-psbsf   1/1         Running   0              15m

NAME                                       STATUS   VOLUME                                         CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/mysql-pv-claim   Bound        pvc-8e6a38ba-79cd-46fa-ac23-be43fc32c751   20Gi           RWO                gp2                15m
persistentvolumeclaim/wp-pv-claim          Bound        pvc-ae827f20-92b1-4ca5-84e1-657d3cf61166   20Gi           RWO                gp2                15m

NAME                          TYPE               CLUSTER-IP          EXTERNAL-IP                                                                   PORT(S)            AGE
service/wordpress             LoadBalancer   172.30.146.20   a1e4531ad6fd44fefa57469c96db579d-1493697719.us-west-2.elb.amazonaws.com   80:30711/TCP   15m
service/wordpress-mysql   ClusterIP          None                <none>                                                                        3306/TCP           15m

NAME                                     HOST/PORT                                                       PATH   SERVICES        PORT   TERMINATION   WILDCARD
route.route.openshift.io/wordpress   wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com              wordpress   80                       None

We can now access the WordPress application using the URL wordpress-wordpress.apps.mytest.wgms.p1.openshiftapps.com As we can see below that the blog post that we had before we deleted the project has been restored.

Scheduling Backups:

Now that we have seen how to backup and restore our application using Velero, we will now look into scheduling the backup using the Schedule API available in the OADP Operator.

As shown below we have created a schedule called test-schedule. For this post we have scheduled a backup at 5 minutes (*/5 * * * *). Under the Template enter the VolumeSnapshotLocations as “default”, IncludedNamespaces as “wordpress”, and StorageLocation as “default”

This will schedule the backup from the OADP Operator as shown below.

Under the Backup tab of the OADP Operator we can see that the backups are being taken at the scheduled time.

Conclusion

In this post we saw how to backup and restore an application running in a ROSA cluster. We also looked into scheduling backups of our application namespace using the OADP Operator.

Please look into the OADP Operator github location for any new features being added to it.