References:
GitOps is the way to do Kubernetes or OpenShift cluster management through declarative management of API manifests (yaml files) rather than configuration drifting.
There are many posts on OpenShift blog talking about Vault Integration:
- Managing Secrets on OpenShift - Vault Integration
- Vault Integration Using Kubernetes Authentication Method
- Integrating Vault with Legacy Applications
- Integrating Hashicorp Vault in OpenShift 4
But there are other options and possibilities out there in the community, so in this post, we will explore a couple of options for secret management the GitOps way:
The code and scripts referred in this blog can be found at:
https://github.com/rahmed-rh/oc4-learn/tree/master/secret-managment
Using bitnami-labs Sealed Secrets
The idea is to Encrypt your Secret into a Sealed Secret, which is safe to store even in public repos as it is encrypted.
So your normal GitOps workflow is not impacted, you will store Sealed Secret yaml files rather than the standard secret yaml files. Sealed Secrets Controller is responsible for doing the conversion for you.
Understand Sealed Secrets
Sealed Secrets is composed of two parts:
- A cluster-side controller / operator
- A client-side utility: kubeseal
Installation
I’m using a bash script to download the controller file from bitnami repo. Then using kustomize, I modify some security context attributes to run on ocp. Then it downloads kubeseal utility:
sealed-secrets/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1 |
To deploy the controller, run the deploy script:
$ ./00_build.sh
What is kubeseal
The kubeseal utility uses asymmetric crypto to encrypt Secrets that only the controller can decrypt. The key certificate (public key portion) is used for sealing Secrets, and needs to be available wherever kubeseal is going to be used.
Kubeseal will fetch the certificate from the controller at runtime, but public key certificate can also be download for offline usage by:
kubeseal --fetch-cert >mycert.pem
Use it offline with:
kubeseal --cert mycert.pem
Create an Encrypted Sealed Secret
Encrypted Sealed Secret cannot allow users to read a Sealed Secret meant for a namespace they wouldn’t have access to and does not just push a copy of it in a namespace where they can read secrets.
Sealed Secrets thus behaves as if each namespace had its own independent encryption key and thus once you seal a Secret for a namespace, it cannot be moved into another namespace and decrypted there. For more information about scopes, please check documentation.
By default, kubeseal will use current context configured in ~/.kube/config.
This will include server connection and active project you can display it by running:
$ oc config current-context
To create the Sealed Secret named “test-secret”:
$ oc create secret generic test-secret --from-literal=dummykey1=supersecret --from-literal=dummykey2=topsecret --dry-run -o yaml >test-secret.yaml $ cat test-secret.yaml |kubeseal --controller-namespace sealed-secrets -o yaml --scope strict > sealedtest-secret.yaml |
Then apply the Sealed Secret:,
$ oc apply -f sealedtest-secret.yaml
Note that the actual Secret is created automatically:
$ oc describe secret/test-secret |
Update an Encrypted Sealed Secret
To update the Sealed Secret “test-secret,” reate the Secret data with the new values, then use “--merge-into” to update the existing Sealed Secret:
$ oc create secret generic test-secret --from-literal=dummykey1=supersecret --from-literal=dummykey2=topsecret --from-literal=dummykey3=new-secret --dry-run -o yaml >test-secret.yaml $ cat test-secret.yaml |kubeseal --controller-namespace sealed-secrets -o yaml --scope strict --merge-into sealedtest-secret.yaml |
Then apply the Sealed Secret:
$ oc apply -f sealedtest-secret.yaml |
Using GoDaddy Kubernetes External Secrets
The idea is to use external Secret management systems, like AWS Secrets Manager or HashiCorp Vault, to securely add Secrets in Kubernetes. The controller is responsible to fetch Secret data from external providers and add it to namespace
Understand External Secrets
An External Secret is composed of a cluster controller, which is responsible to fetch Secret data from external providers and adds it to namespace so the Pods can use it.
You define your external provider configuration through External Secrets custom resource:
source https://github.com/godaddy/kubernetes-external-secrets/raw/master/architecture.png
Installation
I’m using a bash script to clone the whole repo as the installer using helm charts, and I converted it to template files and used kustomize to modify some resource files to remove helm-related stuff.
You will need helm client to be installed on the machine, follow this link to Install Helm.
In case you changed the bash variable "InstanaceName," you will need to modify the resource name in all patch yaml files (helm generate the name based on instance name):
export InstanaceName=dev (1)
(1) The helm instance name
For example, go to patch file and change the name:
external-secrets/base/clusterrole-patch.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1 |
(1) Name should match helm instance name:
external-secrets/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1 |
The bash script configures the environment variable needed for each back-end external service, through kustomization overlays.
For example, for Ali Baba, it modifies the needed environment variable through the overlay patch file:
external-secrets/overlays/alibaba/deployment-patch.yaml
apiVersion: apps/v1 |
And in the deploy bash script, it uses this specific overlay:
oc apply -k overlays/alibaba/ (1)
(1) Will deploy the controller with the needed Deployment environment variables for Ali Baba cloud
To deploy the controller, run the deploy script:
$ ./00_build.sh
What Are External Secrets?
An External Secret is the Custom Resource definition that will be created after the installation. It defines the back-end external service that will be used to retrieve the Secret data:
apiVersion: kubernetes-client.io/v1 |
Create an External Secret
- Create secret by using the aliyun-cli command:
aliyun kms CreateSecret --SecretName db_cred --SecretData "{\"username\": \"test\", \"password\": \"123456\"}" --VersionId v001 |
- Create the CR yaml file:
external-secrets/demo-external-secret.yaml
apiVersion: kubernetes-client.io/v1 |
(1) (optional) Specify role to assume using provided access key ID and access key secret when retrieving the data.
(2) Match the same Secret name you created using aliyun-cli.
(3) version of the secret, ACSCurrent means the latest
- Apply the CR file:
oc apply -f demo-external-secret.yaml
That's all folks, please let me know your feedback!
About the author
More like this
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit