This article is focused on guiding the creation of automatic backups for ETCD in OCP 4.x clusters. This activity is of paramount importance for a successful disaster recovery.
This solution has been tested from versions 4.7 onwards.
You may be curious how ETCD automated backups can assist in the recovery of one or more Master Nodes Cluster on OpenShift 4. Below I will demonstrate what necessary resources you will need to create automatic backups using CronJob from OpenShift.
Let’s go into what matters. First, you must create the following features:
- Namespace
- Service Account
- Cluster Role
- Cluster Role Binding
- Set Privileges for Service Account
- CronJob
Namespace
Specific namespaces must be created for running ETCD backup pods. In the CronJob section, I will explain the pods that will be created to perform the backup in more detail. An example of setting this up is in the following command:
$ oc new-project ocp-etcd-backup --description "Openshift Backup Automation Tool" --display-name "Backup ETCD Automation"
Service Account
You must create a service account in the namespace dedicated to backup ETCD. This service account will be responsible for performing backup commands for the master nodes:
---
kind: ServiceAccount
apiVersion: v1
metadata:
name: openshift-backup
namespace: ocp-etcd-backup
labels:
app: openshift-backup
---
$ oc apply -f sa-etcd-bkp.yml
Cluster Role
As a security measure, the service account created earlier can not have excessive permissions on the cluster, so you must create a Cluster Role with specific permissions for running the backup.
This Cluster Role has specific permissions to specific resources and verbs, such as:
- Resource: Nodes
- Verbs: Get e List
- Resource: pods and pods/log
- Verbs: Get, List, Create, Delete, and Watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cluster-etcd-backup
rules:
- apiGroups: [""]
resources:
- "nodes"
verbs: ["get", "list"]
- apiGroups: [""]
resources:
- "pods"
- "pods/log"
verbs: ["get", "list", "create", "delete", "watch"]
---
$ oc apply -f cluster-role-etcd-bkp.yml
In this link you can find more information about RBAC permissions.
Cluster Role Binding
After creating the Service Account and Cluster Role, you need to create the link between the two resources:
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: openshift-backup
labels:
app: openshift-backup
subjects:
- kind: ServiceAccount
name: openshift-backup
namespace: ocp-etcd-backup
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-etcd-backup
---
$ oc apply -f cluster-role-binding-etcd-bkp.yml
Service Account With Special Privileges
You will need to set special privileges to the service account running the directory creation commands and backup execution of ETCD (The aforementioned commands are executed with sudo, so you will need the service account to have special privileges):
$ oc adm policy add-scc-to-user privileged -z openshift-backup
In the above command, we are using the -z parameter for the SCC (Secure Content Context.) "Privileged" is applied to the service account.
CronJob
Finally, you need to create CronJob.
In CronJob, we will be using the OpenShift client image to create the backup and debug pods.
In CronJob, the backup will be created in /home/core/backup. If this directory does not exist, it will be created automatically.
The CronJob will also delete the backups older than 1 minute, avoiding unnecessary use of disk resources of Master Nodes.
To create CronJob, do this:
---
kind: CronJob
apiVersion: batch/v1
metadata:
name: openshift-backup
namespace: ocp-etcd-backup
labels:
app: openshift-backup
spec:
schedule: "56 23 * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 5
failedJobsHistoryLimit: 5
jobTemplate:
metadata:
labels:
app: openshift-backup
spec:
backoffLimit: 0
template:
metadata:
labels:
app: openshift-backup
spec:
containers:
- name: backup
image: "registry.redhat.io/openshift4/ose-cli"
command:
- "/bin/bash"
- "-c"
- oc get no -l node-role.kubernetes.io/master --no-headers -o name | xargs -I {} -- oc debug {} --to-namespace=ocp-etcd-backup -- bash -c 'chroot /host sudo -E /usr/local/bin/cluster-backup.sh /home/core/backup/ && chroot /host sudo -E find /home/core/backup/ -type f -mmin +"1" -delete'
restartPolicy: "Never"
terminationGracePeriodSeconds: 30
activeDeadlineSeconds: 500
dnsPolicy: "ClusterFirst"
serviceAccountName: "openshift-backup"
serviceAccount: "openshift-backup"
---
$ oc apply -f cronjob-etcd-bkp.yml
After creating CronJob, you can force the execution for validation with the command:
$ oc create job backup --from=cronjob/openshift-backup
During the execution of CronJob, the automatically created pods will debug on nodes for backup execution.
With these commands, you can check the pods that were created for a backup running:
$ oc get pods -n ocp-etcd-backup
NAME READY STATUS RESTARTS AGE
pod/backup-jcn87 1/1 Running 0 40s
pod/zmaciel-f9fbb-master-0-debug 1/1 Running 0 18s
$ oc get pods -n ocp-etcd-backup
NAME READY STATUS RESTARTS AGE
pod/backup-jcn87 1/1 Running 0 56s
pod/zmaciel-f9fbb-master-1-debug 1/1 Running 0 14s
$ oc get pods -n ocp-etcd-backup
NAME READY STATUS RESTARTS AGE
pod/backup-jcn87 1/1 Running 0 67s
pod/zmaciel-f9fbb-master-2-debug 0/1 Running 0 6s
These pods will be automatically excluded from the end of the backup execution.
If CronJob ran successfully, you will have a result similar to the images below.
Backup Pod, Job, and CronJob
Backups Created on Master Nodes
Final Thoughts
So, now you can see how easy it is to create backups of the ETCD through the OpenShift Platform.
About the author
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit