Installing the HashiCorp Vault Secret CSI Driver

The HashiCorp Vault Secret CSI Driver allows you to access secrets stored in HashiCorp Vault as Kubernetes Volumes.

Prerequisites

1. An OpenShift Cluster (ROSA, ARO, OSD, and OCP 4.x all work)
2. oc
3. helm v3

Installing the Kubernetes Secret Store CSI

1. Create an OpenShift Project to deploy the CSI into

   oc new-project k8s-secrets-store-csi
   

2. Set SecurityContextConstraints to allow the CSI driver to run (otherwise the DaemonSet will not be able to create Pods)

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver
   

3. Add the Secrets Store CSI Driver to your Helm Repositories

   helm repo add secrets-store-csi-driver \
     https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   

4. Update your Helm Repositories

   helm repo update
   

5. Install the secrets store csi driver

   helm install -n k8s-secrets-store-csi csi-secrets-store \
     secrets-store-csi-driver/secrets-store-csi-driver \
     --version v1.3.2 \
     --set "linux.providersDir=/var/run/secrets-store-csi-providers"
   

6. Check that the Daemonsets is running

   oc -n k8s-secrets-store-csi get pods -l "app=secrets-store-csi-driver"
   

   You should see the following

   NAME                                               READY   STATUS    RESTARTS   AGE
   csi-secrets-store-secrets-store-csi-driver-cl7dv   3/3     Running   0          57s
   csi-secrets-store-secrets-store-csi-driver-gbz27   3/3     Running   0          57s
   

7. Add pod security profile label for CSI Driver

   This is required starting in OpenShift v4.13

   oc label csidriver/secrets-store.csi.k8s.io security.openshift.io/csi-ephemeral-volume-profile=restricted
   

Install HashiCorp Vault with CSI driver enabled

1. Add the HashiCorp Helm Repository

   helm repo add hashicorp https://helm.releases.hashicorp.com
   

2. Update your Helm Repositories

   helm repo update
   

3. Create a namespace for Vault

   oc new-project hashicorp-vault
   

4. Create a SCC for the CSI driver

   oc adm policy add-scc-to-user privileged \
     system:serviceaccount:hashicorp-vault:vault-csi-provider
   

5. Create a values file for Helm to use

   cat << EOF > values.yaml
   global:
     openshift: true
   csi:
     enabled: true
     daemonSet:
       providersDir: /var/run/secrets-store-csi-providers
   injector:
     enabled: false
   server:
     image:
       repository: "registry.connect.redhat.com/hashicorp/vault"
       tag: "1.8.0-ubi"
     dev:
       enabled: true
   EOF
   

6. Install Hashicorp Vault with CSI enabled

   helm install -n hashicorp-vault vault \
     hashicorp/vault --values values.yaml
   

7. Patch the CSI daemonset

   Currently the CSI has a bug in its manifest which we need to patch

   oc patch daemonset vault-csi-provider --type='json' \
       -p='[{"op": "add", "path": "/spec/template/spec/containers/0/securityContext", "value": {"privileged": true} }]'
   

Configure Hashicorp Vault

1. Get a bash prompt inside the Vault pod

   oc exec -it vault-0 -- bash
   

2. Create a Secret in Vault

   vault kv put secret/db-pass password="hunter2"
   

3. Configure Vault to use Kubernetes Auth

   vault auth enable kubernetes
   

4. Check your Cluster’s token issuer in another terminal

   oc get authentication.config cluster \
     -o json | jq -r .spec.serviceAccountIssuer
   

5. Configure Kubernetes auth method

   If the issuer here does not match the above, update it.

   vault write auth/kubernetes/config \
   issuer="https://kubernetes.default.svc.cluster.local" \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
   

6. Create a policy for our app

   vault policy write internal-app - <<EOF
   path "secret/data/db-pass" {
     capabilities = ["read"]
   }
   EOF
   

7. Create an auth role to access it

   vault write auth/kubernetes/role/database \
     bound_service_account_names=webapp-sa \
     bound_service_account_namespaces=default \
     policies=internal-app \
     ttl=20m
   

8. exit from the vault-0 pod

   exit
   

Deploy a sample application

1. Create a SecretProviderClass in the default namespace

   cat <<EOF | oc apply -f -
   apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
   kind: SecretProviderClass
   metadata:
     name: vault-database
     namespace: default
   spec:
     provider: vault
     parameters:
       vaultAddress: "http://vault.hashicorp-vault:8200"
       roleName: "database"
       objects: |
         - objectName: "db-password"
           secretPath: "secret/data/db-pass"
           secretKey: "password"
   EOF
   

2. Create a service account webapp-sa

   oc create serviceaccount -n default webapp-sa
   

3. Create a Pod to use the secret

   cat << EOF | oc apply -f -
   kind: Pod
   apiVersion: v1
   metadata:
     name: webapp
     namespace: default
   spec:
     serviceAccountName: webapp-sa
     containers:
     - image: jweissig/app:0.0.1
       name: webapp
       volumeMounts:
       - name: secrets-store-inline
         mountPath: "/mnt/secrets-store"
         readOnly: true
     volumes:
       - name: secrets-store-inline
         csi:
           driver: secrets-store.csi.k8s.io
           readOnly: true
           volumeAttributes:
             secretProviderClass: "vault-database"
   EOF
   

4. Check the Pod has the secret

   oc -n default exec webapp \
     -- cat /mnt/secrets-store/db-password
   

   The output should match

   hunter2
   

Uninstall HashiCorp Vault with CSI driver enabled

1. Delete the pod and

   oc delete -n default pod webapp
   oc delete -n default secretproviderclass vault-database
   oc delete -n default serviceaccount webapp-sa
   

2. Delete the Hashicorp Vault Helm

   helm delete -n hashicorp-vault vault
   

3. Delete the SCC for Hashicorp Vault

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:hashicorp-vault:vault-csi-provider
   

4. Delete the Hashicorp vault project

   oc delete project hashicorp-vault
   

Uninstalling the Kubernetes Secret Store CSI

1. Delete the secrets store csi driver

   helm delete -n k8s-secrets-store-csi csi-secrets-store
   

2. Delete the SecurityContextConstraints

   oc adm policy remove-scc-from-user privileged \
     system:serviceaccount:k8s-secrets-store-csi:secrets-store-csi-driver