When companies talk about security, they are referring to preventing data loss and securely automating and integrating applications.

That cannot be done without knowing who is doing what to which assets, and that is where identity management, like HashiCorp Vault, comes in. The “who” in the equation becomes very important.

Properly issued certificates enable end-to-end security through a trusted chain of identities.

As with most security objectives, there is usually tension between the requirement to make things secure and trying to get the actual work done. The art here is to balance the two conflicting requirements, one way to reduce the burden on the developer is to automate as much as possible.

In this blog, we will illustrate how OpenShift together with Cert Manager and HashiCorp Vault can be used to achieve an automated and reproducible process to increase the security of applications.

From the developer's point of view, this automated approach is easy to use and is also instrumented so that we know what is going on and can take appropriate action if it fails.

Certificate Authority

The purpose of a Certificate authority (CA) is to validate and issue certificates. A Certificate Authority may be a third-party entity or organization that runs its own provider to issue digital certificates.

An intermediate certificate authority is a CA signed by a superior CA (for example, a root CA or another Intermediate CA) and signs CAs (for example, another intermediate or subordinate CA).

If an Intermediate CA exists, it is positioned within the middle of a trust chain between the trust anchor, or root, and the subscriber certificate that is issuing subordinate CAs. So not use a root CA directly?

Typically, the root CA does not sign server or client certificates directly. The root CA is used only to create one or more intermediate CAs. Using an intermediate CA is primarily for security purposes and the root CA is hosted elsewhere in a secure place; offline, and used as infrequently as possible.

So, it is better to not expose it within target environments and to instead issue a shorter-lived intermediate CA. Using intermediate CA also aligns with industry best practices.

CA Hierarchy

In large organizations, it may be ideal to delegate responsibility for issuing certificates to different certificate authorities for granular security controls appropriate to each CA.

For example, the number of certificates may be too large for a single CA to effectively track the certificates it has issued; or each departmental unit may have different policies and rules, such as validity periods; or it may be important to differentiate certificates for internal or external communication.

The X.509 standard includes a template for setting up a hierarchy of CAs:

Installation

Cert Manager

Cert Manager is a tool for Kubernetes and OpenShift that automates certificate management in cloud-native environments.

It builds on top of these platforms to provide X.509 certificates and issuers as first-class resource types.

It provides easy-to-use tools to manage certificates including a “certificates as a service” for securely enabling developers and applications working within a cluster and a standardized API for interacting with multiple certificate authorities (CAs). This gives security teams the confidence to allow developers to manage certificates in a self-service fashion.

Various integrations are available including ACME (Let’s Encrypt), HashiCorp Vault, Venafi, and self-signed and internal certificate authorities. In addition, extension points can be added to support custom, internal or otherwise unsupported CAs.

To install Cert Manager Operator within an OpenShift Container Platform environment, log into the web console as a user with the cluster-admin role. As of this writing, the Cert Manager Operator requires version 4.10 or higher.

1. Click Operators → OperatorHub.

2. Type the name of the Operator into the filter box and select “cert-manager Operator for Red Hat OpenShift”.

3. Click Install.

4. On the Install Operator page, select installation options.

   4.1 in the Update Channel section, select tech-preview.

   4.2. The Cert Manager operator is installed in the openshift-cert-manager-operator namespace.

   4.3. Click Install and wait until the Operator is installed.

5. Click Operators → Installed Operators to verify that the Operator installed successfully.

Create the CA Chain

Let’s start from scratch and simulate the creation of our own certificate authority and building the CA hierarchy.

We are going to create the root CA certificate-key pair using ​​OpenSSL program.

First, navigate to a directory for which the certificates will be created.

export CERT_ROOT=$(pwd)

Define the directory structure:

mkdir -p ${CERT_ROOT}/{root,intermediate}

Generate the CA Private Key:

cd ${CERT_ROOT}/root/

openssl genrsa -out ca.key 2048

touch index.txt
echo 1000 > serial
mkdir -p newcerts

Define the openssl.cnf file:

cat <<EOF > openssl.cnf
[ ca ]
default_ca = CA_default

[ CA_default ]
# Directory and file locations.
dir               = ${CERT_ROOT}/root
certs             = \$dir/certs
crl_dir           = \$dir/crl
new_certs_dir     = \$dir/newcerts
database          = \$dir/index.txt
serial            = \$dir/serial
RANDFILE          = \$dir/private/.rand

# The root key and root certificate.
private_key       = \$dir/ca.key
certificate       = \$dir/ca.crt

# For certificate revocation lists.
crlnumber         = \$dir/crlnumber
crl               = \$dir/crl/ca.crl
crl_extensions    = crl_ext
default_crl_days  = 30

# SHA-1 is deprecated, so use SHA-2 instead.
default_md        = sha256

name_opt          = ca_default
cert_opt          = ca_default
default_days      = 375
preserve          = no

policy            = policy_strict

[ policy_strict ]
# The root CA should only sign intermediate certificates that match.
countryName               = match
stateOrProvinceName       = optional
organizationName          = optional
organizationalUnitName    = optional
commonName                = supplied
emailAddress              = optional

[ v3_intermediate_ca ]
# Extensions for a typical intermediate CA.
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true, pathlen:1
keyUsage = critical, digitalSignature, cRLSign, keyCertSign

[req_distinguished_name]
countryName = CH
countryName = Country Name
countryName_default = CH
stateOrProvinceName = State or Province Name
stateOrProvinceName_default = ZH
localityName= Locality Name
localityName_default = Zurich
organizationName= Organization Name
organizationName_default = Red Hat
commonName= Company Name
commonName_default = company.io
commonName_max = 64

[req]
distinguished_name = req_distinguished_name
[ v3_ca ]
basicConstraints = critical,CA:TRUE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
EOF

Generate the certificate:

openssl req -x509 -new -nodes -key ca.key -sha256 -days 1024 -out ca.crt -extensions v3_ca -config openssl.cnf
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----

Country Name [CH]:

State or Province Name [ZH]:

Locality Name [Zurich]:

Organization Name [Red Hat]:

Company Name [company.io]:

As shown in the output above, the value defined in the openssl.cnf configuration file includes a req_distinguished_name entry which is used as the default set of values when generating the certificate. The default values can be used or a user defined set of values can be provided .

Now with the root CA, we can start with the second step of the chain: the intermediate CA.

Generate the Intermediate CA Private Key:

cd ../intermediate

openssl genrsa -out ca.key 2048

Generate the Certificate Siging Request:

openssl req -new -sha256 -key ca.key -out ca.csr

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) []:CH
State or Province Name (full name) []:ZH
Locality Name (eg, city) []:Zurich
Organization Name (eg, company) []:Red Hat
Organizational Unit Name (eg, section) []:RH
Common Name (eg, fully qualified host name) []:int.company.io
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:

Make sure the Country Name and the Common Name are defined, because the policy ( policy_strict ) entry on the openssl.cnf requires a matching_countryName_ and a defined commonName.

Create the intermediate certificate:

openssl ca -config ../root/openssl.cnf -extensions v3_intermediate_ca -days 365 -notext -md sha256 -in ca.csr -out ca.crt

...

Certificate is to be certified until May 12 12:52:52 2023 GMT (365 days)

Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y

Write out database with 1 new entries
Data Base Updated

Our intermediate CA is now ready to be used.

Issuer with Cert Manager

The first thing you will need to configure after you have installed cert-manager is for an Issuer to be created, which can be then used to issue certificates.

Issuers are Kubernetes’ resources that represent Certificate Authorities (CAs) that are able to generate signed certificates by honoring Certificate Signing Requests.

The simplest issuer type is the CA, which references the Kubernetes TLS Secret containing a ca-key pair.

Generate SSL Certificates for Vault using Cert Manager

Before Vault can be installed, certificates must be provisioned within a newly created namespace.

First, define the namespace where we want to install Vault:

oc new-project hashicorp

From the intermediate directory, let’s create the Kubernetes Secret containing the certificate previously generated

oc create secret tls intermediate --cert=${CERT_ROOT}/intermediate/ca.crt --key=${CERT_ROOT}/intermediate/ca.key -n hashicorp

After the secret is created, we can apply a cert-manager Issuer CR of type CA to the cluster:

cat <<EOF | oc apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: int-ca-issuer
spec:
  ca:
    secretName: intermediate
EOF

Let’s check the Issuer status to confirm it was successfully created:

oc get issuer int-ca-issuer

NAME          READY     AGE

int-ca-issuer True      5s

NOTE:This specific Cert-Manager Issuer containing the Intermediate certificate authority is strictly used to sign the certificate of Vault only. No other applications will request certificates.

It is time to create a Certificate CR for HashiCorp Vault. First, let’s define some convenient variables:

export BASE_DOMAIN=$(oc get dns cluster -o jsonpath='{.spec.baseDomain}')
export VAULT_HELM_RELEASE=vault
export VAULT_ROUTE=${VAULT_HELM_RELEASE}.apps.$BASE_DOMAIN
export VAULT_ADDR=https://${VAULT_ROUTE}
export VAULT_SERVICE=${VAULT_HELM_RELEASE}-active.hashicorp.svc

Deploy the certificate:

cat <<EOF|oc apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: vault-certs
spec:
  secretName: vault-certs
  issuerRef:
    name: int-ca-issuer
    kind: Issuer
  dnsNames: 
  - ${VAULT_ROUTE}
  # Service Active FQDN
  - ${VAULT_SERVICE}
  organization:
  - company.io
EOF

NOTE: The generated certificate must be valid for both the Vault active service and the Vault Route.

Once cert-manager detects the creation of the Certificate CR, an SSL certificate will be generated on behalf of the internal intermediate certificate authority and save it as a Kubernetes TLS secret.

oc get secret vault-certs

NAME        TYPE              DATA  AGE

vault-certs kubernetes.io/tls 3     12s

NOTE: This certificate will be used by the HashiCorp pods for securing the Vault API port 8200. When the Vault nodes will join the cluster, they will need to make an API request to the Vault active node (through the vault-active service). However, later on, when the Vault cluster is set up, the nodes will communicate through Cluster port 8201, which is secured by a certificate generated by the Vault active node internally.

HashiCorp Vault

HashiCorp Vault is an identity-based secret and encryption management system.

A secret is anything you want to tightly control access to, such as API encryption keys, passwords, or certificates.

Vault provides encryption services that are gated by authentication and authorization methods. Using Vault’s UI, CLI, or HTTP API, access to secrets and other sensitive data can be securely stored and managed, tightly controlled (restricted), and auditable.

With the certificate now in pace, it is time to deploy and configure Vault.

The official way to install Hashicorp Vault is to use the Vault Helm Chart.

In production environments, it is deployed in a high-availability manner. HashiCorp Vault needs an underlying storage backend to store the data, and it can rely on its integrated storage, which uses the RAFT consensus algorithm.

When deploying to OpenShift, the integrated storage is a convenient solution for storing data, as this removes the burden of managing an additional storage component for storing Vault’s data.

1. Create a working directory for the Vault installation.

mkdir -p vault
cd vault/

2. Configure Helm Repository:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

3. Define the Helmvalues.yaml that will configure a Highly Available Vault environment with Raft storage:

cat <<EOF > values.yaml
global:
  tlsDisable: false
  openshift: true
injector:
  image:
    repository: "registry.connect.redhat.com/hashicorp/vault-k8s"
    tag: "0.14.2-ubi"
  agentImage:
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.9.6-ubi"
ui:
  enabled: true
server:
  image:
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.9.6-ubi"
  route:
    enabled: true
    host:
  extraEnvironmentVars:
    VAULT_CACERT: "/etc/vault-tls/vault-certs/ca.crt"
    VAULT_TLS_SERVER_NAME:
  standalone:
    enabled: false
  auditStorage:
    enabled: true
    size: 15Gi
  extraVolumes:
    - type: "secret"
      name: "vault-certs"
      path: "/etc/vault-tls"
  ha:
    enabled: true
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/etc/vault-tls/vault-certs/tls.crt"
          tls_key_file = "/etc/vault-tls/vault-certs/tls.key"
          tls_client_ca_file = "/etc/vault-tls/vault-certs/ca.crt"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-active.hashicorp.svc:8200"
            leader_ca_cert_file = "/etc/vault-tls/vault-certs/ca.crt"
          }
        }
        log_level = "debug"
        service_registration "kubernetes" {}
  service:
    enabled: true
EOF

3. Install the Vault Helm chart using the previously configured values:

helm install vault hashicorp/vault -f values.yaml \
    --set server.route.host=$VAULT_ROUTE \
    --set server.extraEnvironmentVars.VAULT_TLS_SERVER_NAME=$VAULT_ROUTE \
    --wait \
    -n hashicorp

4. Initialize Vault and save the generated key and token:

oc -n hashicorp exec -ti vault-0 -- vault operator init -key-threshold=1 -key-shares=1

Unseal Key 1: 7tbxdHjNqLsCAS16b0ac92jb+uvXEVSPwFZyf2Ln8Gk=

Initial Root Token: s.lSHpKvhYhjy5xwR0wtkXEk6H

5. Unseal all the Vault instances because they starts in asealed state:

oc -n hashicorp exec -ti vault-0 -- vault operator unseal

Unseal Key (will be hidden):

oc -n hashicorp exec -ti vault-1 -- vault operator unseal

Unseal Key (will be hidden):

oc -n hashicorp exec -ti vault-2 -- vault operator unseal

Unseal Key (will be hidden):

6. Now that the vault has been sealed, check the Raft storage has one leader and two followers:

oc -n hashicorp rsh vault-0

vault login

Token (will be hidden):

vault operator raft list-peers

vault operator raft list-peers
Node       Address                        State       Voter
----       -------                        -----       -----
vault-0    vault-0.vault-internal:8201    leader      true
vault-1    vault-1.vault-internal:8201    follower    true
vault-2    vault-2.vault-internal:8201    follower    true

7. Verify the access from Vault UI. On OpenShift Console Click Networking → Routes and click the vault route.

8. To authenticate use the root token generated before by the initialize command.

Integrate Vault with OpenShift

At this point, Vault is ready to be integrated into the OpenShift platform.

Kubernetes Auth Method

The Kubernetes authentication method can be used to authenticate with Vault using a Kubernetes service account token. The token for a pod’s service account is automatically mounted within a pod at /var/run/secrets/kubernetes.io/serviceaccount/token and is sent to Vault for authentication.

Vault, like all pods in Kubernetes, is configured with a service account that has permissions to access the TokenReview API. This service account can then be used to make authenticated calls to Kubernetes to verify tokens of the service accounts of pods that want to connect to Vault to get secrets.

Configuring Vault for OpenShift Operator-based

To support a variety of Vault configuration workflows and conventions, we will leverage the Vault config operator to automate the setup and configuration.

The advantage of a Vault Config Operator is that we can now configure Vault by creating a set of custom resources (CRs), which in turn can be packaged as Helm charts or kustomize manifests and managed in a GitOps fashion. So, configuring Vault is no longer an imperative action, but is now declarative. Read more about the background and capabilities of the Vault Config Operator on this blog.

Vault Config Operator

To install the Vault Config Operator, return to OpenShift Container Platform web console as a user with the cluster-admin role.

1. Click Operators → OperatorHub.

2. Type the name of the Operator into the filter box and select Vault Config Operator:

3. Click Install.

4. On the Install Operator page, select installation options.

   4.1 in the Update Channel section, select alpha.

   4.2. The Vault config operator is installed in the vault-config-operator namespace.

   4.3. Click Install. Wait until the Operator is installed.

5. Click Operators → Installed Operators to verify that the Operator installed successfully.

Connect to Vault.

The connection to Vault can be initialized with Vault's standard environment variables that are applied to the Vault Config Operator pod. See the OLM documentation on how to pass environment variables via a Subscription. The variables that are read at client initialization are listed here.

For the operator to trust the certificates presented by Vault, the recommended approach is to mount a Secret or ConfigMap containing the certificate as described here, and then configure the corresponding variables to reference the file locations in the path mounted into the container.

The configurations that are applied against Vault by the operator need to authenticate via a Kubernetes Authentication. To facilitate authenticating with Vault by the Operator, a root Kubernetes authentication mount point and role are required.

1. Download and install the Vault client on your local computer by following the instructions here.

2. Login to Vault using the root password that was displayed earlier when initializing Vault:

vault login -tls-skip-verify

Token (will be hidden):

Success! You are now authenticated.

3. Create an admin policy:

cat <<EOF > ./policy.hcl
path "/*" {
  capabilities = ["create", "read", "update", "delete", "list","sudo"]
}
EOF

vault policy -tls-skip-verify write vault-admin ./policy.hcl

NOTE: this policy is intentionally broad to allow testing anything in Vault. In a real life scenario this policy would be scoped down.

4. Switch to the Vault Config Operator project which was automatically created when the operator was installed:

oc project vault-config-operator

5. Enable the Kubernetes auth method by first obtaining details related to the Kubernetes API including the certificate:

JWT_SECRET=$(oc get sa controller-manager -o jsonpath='{.secrets}' | jq '.\[] | select(.name|test("token-")).name')
JWT=$(oc sa get-token controller-manager)
KUBERNETES_HOST=https://kubernetes.default.svc:443

oc extract configmap/kube-root-ca.crt -n vault-config-operator

vault auth enable -tls-skip-verify kubernetes

vault write -tls-skip-verify auth/kubernetes/config token_reviewer_jwt=$JWT kubernetes_host=$KUBERNETES_HOST kubernetes_ca_cert=@./ca.crt

6. Create a Role and assign it to the policy previously creted

vault write -tls-skip-verify auth/kubernetes/role/vault-admin bound_service_account_names=controller-manager bound_service_account_namespaces=vault-config-operator policies=vault-admin ttl=1h

Verify the recently created kubernetes auth method from Vault UI → Access → AuthMethods.

7. Create a configmap that contains our intermediate CA:

oc create configmap int-ca --from-file=${CERT_ROOT}/intermediate/ca.crt -n vault-config-operator

8. Patch the subscription to include the connection to our Vault instance:

cat <<EOF > patch.yaml
spec:
  config:
    env:
    - name: VAULT_ADDR
      value: https://vault-active.hashicorp.svc:8200
    - name: VAULT_CACERT
      value: /vault-ca/ca.crt
    - name: VAULT_TOKEN
      valueFrom:
        secretKeyRef:
          name: $JWT_SECRET
          key: token
    volumes:
    - name: vault-ca
      configMap:
        name: int-ca
    volumeMounts:
    - mountPath: /vault-ca
      name: vault-ca
EOF

oc patch subscription vault-config-operator --type=merge --patch-file patch.yaml -n vault-config-operator

The patch of the subscription updates the Deployment of the vault-config-operator with the new configuration as show in the picture below.

9. Add the token review role for the controller-manager Service Account for which the Vault Config Operator use:

oc adm policy add-cluster-role-to-user system:auth-delegator -z controller-manager

Vault PKI Secrets Engine.

The Vault PKI secrets engine generates dynamic X.509 certificates, without requiring all of the manual actions.

Vault's built-in authentication and authorization mechanisms provide the necessary verification functionality.

As seen in the diagram above, there are several steps to enable Vault to be a certificate manager in OpenShift. These steps are detailed below:

Vault:

1. Enable the Kubernetes Auth Engine in Vault.

2. Authorize the Vault SA in k8s to Token Review.

3. Enable the PKI secrets engine.

4. Configure the PKI role in Vault.

5. Configure the PKI policy in Vault.

6. Authorize/Bind issuer SA to use (policy) the PKI role.

Cert-Manager:

1. Create the Vault Issuer in-app namespace with issuer SA.

2. cert-manager validates the credentials of the issuer against Vault.

Thanks to the Vault Config Operator and cert-manager, define the relative customer resource and let them do the job for you.

It is time to create the CA chain hierarchy with an offline root CA and online intermediate CAs in Vault for each application namespace.

Having a dedicated intermediate CA per organization or team can increase security and gain greater control over the chain of trust in your ecosystem, allowing you to trust only certificates issued by your trust model.

1. Create an Intermediate SecretEngineMount:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: SecretEngineMount
metadata:
  name: intermediate
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  type: pki
  path: pki
  config:
    # 1 Year
    maxLeaseTTL: "8760h"
EOF

Verify from Vault UI → Secrets the PKI Secret Engine created.

2. Configure an intermediate PKISecretEngineConfig:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PKISecretEngineConfig
metadata:
  name: intermediate
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  path: pki/intermediate
  commonName: vault.int.company.io
  TTL: "8760h"
  type: intermediate
  privateKeyType: exported
  country: CH
  province: ZH
  locality: Zurich
  organization: Red Hat
  maxPathLength: 1
  issuingCertificates:
  - https://${VAULT_ROUTE}/v1/pki/intermediate/ca
  crlDistributionPoints:
  - https://${VAULT_ROUTE}/v1/pki/intermediate/crl"
EOF

Note: PKISecretEngineConfig stays in error status until the signed certificate has been provided.

Waiting spec.externalSignSecret with signed intermediate certificate.

3. Sign the CSR with the company root CA:

oc extract secret/intermediate --keys=csr

openssl ca -config ${CERT_ROOT}/root/openssl.cnf -extensions v3_intermediate_ca -days 365 -notext -md sha256 -in csr -out tls.crt

4. Create the secret with the signed intermediate certificate:

oc create secret generic signed-intermediate --from-file=tls.crt

5. Patch the PKISecretEngineConfig with the new signed-intermediate secret.

cat <<EOF > patch-pki.yaml
spec:
  externalSignSecret:
    name: signed-intermediate
EOF

oc patch pkisecretengineconfig intermediate --type=merge --patch-file patch-pki.yaml -n vault-config-operator

Verify from Vault UI → Secret → pki/intermediate the signed certificate.

At this point, it is time to configure the PKI for the application namespace, for this example we configure the team-one namespace.

We walk through each step in the process; nevertheless, these steps can be automated in the future by leveraging the Namespace Configuration Operator.

1. Create AuthEngineMount to define an authentication engine endpoint:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: AuthEngineMount
metadata:
  name: team-one
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  type: kubernetes
  path: app-kubernetes
EOF

Verify from Vault UI → Access → AuthMethods.

2. Create KubernentesAuthEngineConfig to configure the auth engine mount to point to a specific Kubernetes master API endpoint:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: KubernetesAuthEngineConfig
metadata:
  name: team-one
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  tokenReviewerServiceAccount:
    name: controller-manager
  path: app-kubernetes
EOF

3. Create the KubernetesAuthEngineRole, which configures all of the default service accounts in the application namespace:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: KubernetesAuthEngineRole
metadata:
  name: team-one
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  path: app-kubernetes/team-one
  policies:
  - team-one-pki-engine
  targetServiceAccounts:
  - default
  targetNamespaces:
    targetNamespaces:
    - team-one
EOF

Verify from Vault UI → Access → app-kubernetes/team-one → Roles.

4. Define the policy to give the right access to the PKI engine:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: Policy
metadata:
  name: team-one-pki-engine
spec:
  authentication:
    path: kubernetes
    role: vault-admin
    serviceAccount:
      name: controller-manager
  policy: |
    # query existing mounts
    path "/sys/mounts" {
      capabilities = [ "list", "read"]
      allowed_parameters = {
        "type" = ["pki"]
        "*"   = []
      }
    }

    # mount pki secret engines
    path "/sys/mounts/app-pki/team-one*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }

    # tune
    path "/sys/mounts/app-pki/team-one/tune" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }

    # internal sign pki
    path "pki/intermediate/root/sign-intermediate" {
      capabilities = ["create"]
    }

    # pki 
    path "app-pki/team-one*" {
      capabilities = ["create", "read", "update", "delete", "list"]
    }
EOF

5. Now, create the team-one application namespace that can leverage the resources that we previously configured:

oc new-project team-one

6. Create the SecretEngineMount of type PKI in the application namespace team-one:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: SecretEngineMount
metadata:
  name: team-one
spec:
  authentication:
    path: app-kubernetes/team-one
    role: team-one
  type: pki
  path: app-pki
  config:
    # 1 Year
    maxLeaseTTL: "8760h"
EOF

Verify from Vault UI → Secrets

7. Generate the intermediate certificate signed by the internal Vault pki/intermediate CA:

cat <<EOF|oc apply -f
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PKISecretEngineConfig
metadata:
  name: team-one
spec:
  authentication:
    path: app-kubernetes/team-one
    role: team-one
  path: app-pki/team-one
  commonName: team-one.vault.int.company.io
  TTL: "8760h"
  type: intermediate
  privateKeyType: exported
  internalSign:
    name: pki/intermediate
  issuingCertificates:
  - https://${VAULT_ROUTE}/v1/app-pki/team-one/ca
  crlDistributionPoints:
  - https://${VAULT_ROUTE}/v1/app-pki/team-one/crl"
EOF

Verify from Vault UI → Secret → app-pki/team-one the signed certificate.

9. Configure the PKI role:

cat <<EOF|oc apply -f -
apiVersion: redhatcop.redhat.io/v1alpha1
kind: PKISecretEngineRole
metadata:
  name: team-one
spec:
  authentication:
    path: app-kubernetes/team-one
    role: team-one
  path: app-pki/team-one
  allowedDomains:
   - team-one.vault.int.company.io
   - team-one.svc
   - "*-team-one.apps.${BASE_DOMAIN}"
  allowSubdomains: true
  allowedOtherSans: "*"
  allowGlobDomains: true
  allowedURISans:
  - "*-team-one.apps.${BASE_DOMAIN}"
  maxTTL: "8760h"
EOF

Verify from Vault UI → Secret → app-pki/team-one → Roles.

Deploy sample application

To demonstrate the end-to-end functionality, a demonstration application can be deployed consisting of two simple Quarkus client/server services that leverage mutual TLS communication through certificates provided by cert-manager and Hashicorp Vault integration.

Requesting certificate from HashiCorp Vault PKI provider

As we can see in the above diagram, there are several steps to request certificates from Hashicorp Vault that can be consumed by applications.

On OpenShift:

1. The cert-manager Issuer is the interface between certificates and HashiCorp Vault. We defined the path where certificates will be created and Kubernetes authentication to access PKI team-one role. Let’s create an issuer at the namespace level, which is the best way to isolate certificates. So, it is not possible to issue certificates from an Issuer in a different namespace:

   1.1 Get HashiCorp Vault CA Bundle and team-one default service account token.

       export CA_BUNDLE=$(oc get secret vault-certs -n hashicorp -o json | jq -r '.data."ca.crt"')
   
       export DEFAULT_SECRET=$(oc get sa default -n team-one -o json | jq -r '.secrets[0].name')

   1.2 Create cert-manager issuer.

   cat <<EOF| oc apply -f -
   apiVersion: cert-manager.io/v1
   kind: Issuer
   metadata:
     name: team-one-issuer-vault
     namespace: team-one
   spec:
     vault:
       path: app-pki/team-one/sign/team-one
       server: https://vault-active.hashicorp.svc:8200
       caBundle: $CA_BUNDLE
       auth:
         kubernetes:
           role: team-one
           mountPath: /v1/auth/app-kubernetes/team-one
           secretRef:
             key: token
             name: $DEFAULT_SECRET
   EOF

As we can observe in the code sample above, authentication is performed using namespace default service account, which is specified as secretRef in the issuer authentication section.

This is configured in Vault UI → Access → Auth Methods → app-kubernetes/team-one → Roles → team-one as follows:

We have developed a demo Java application which you can find in the following repository:

Mutual TLS code sample

Since the Issuer is already created, we can deploy the full example performing next steps:

1. Git clone repository https://github.com/openlab-red/hashicorp-vault-for-openshift.git
2. Connect to your OCP cluster.
3. Access repository folder examples/quarkus-mtls-example/
4. Build and deploy the application stack:

   mvn oc:build oc:resource oc:apply -Pprod

Now, what is happening, behind the scenes, during the application deployment.

2. The deployment creates a certificate that is watched by the cert-manager controller. To support the demo application, there will be two certificates needed in order to enable mutual TLS authentication between client and server.

The Client Certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: client
  namespace: team-one
spec:
  commonName: client.team-one.vault.int.company.io
  dnsNames:
  - client.team-one.vault.int.company.io
  - client.team-one.svc
  issuerRef:
    name: team-one-issuer-vault
  keystores:
    pkcs12:
      create: true
      passwordSecretRef:
        key: password
        name: client-keystore-pass
  secretName: client

The Server Certificate.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: server
  namespace: team-one
spec:
  commonName: server.team-one.vault.int.company.io
  dnsNames:
  - server.team-one.vault.int.company.io
  - server.team-one.svc
  issuerRef:
    name: team-one-issuer-vault
  keystores:
    pkcs12:
      create: true
      passwordSecretRef:
        key: password
        name: server-keystore-pass
  secretName: server

3. The cert-manager controller via Key Manager component creates a temporary private key as a secret in the namespace. Then, the Request Manager component will create a certificate request and will sign the CSR using the temporary private key. For more details on the requesting certificate process, it is recommended to review the following resource.

In the Vault UI → Secrets → app-pki/team-one you will find the two certificates recently created by the deployment.

4. The cert-manager fetches the certificate from Vault PKI, creates, and populates the certificate secret and finally copies the temporary private key into the secret:

oc describe secret client

Name:         client
Namespace:    team-one
Labels:       <none>
Annotations:  cert-manager.io/alt-names:  client.team-one.svc,client.team-one.vault.int.company.io
              cert-manager.io/certificate-name: client
              cert-manager.io/common-name: client.team-one.vault.int.company.io
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: team-one-issuer-vault
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
ca.crt:          1281 bytes
keystore.p12:    4391 bytes
tls.crt:         2550 bytes
tls.key:         1675 bytes
truststore.p12:  1210 bytes

oc describe secret server

Name:         server
Namespace:    team-one
Labels:       <none>
Annotations:  cert-manager.io/alt-names: server.team-one.svc,server.team-one.vault.int.company.io
              cert-manager.io/certificate-name: server
              cert-manager.io/common-name: server.team-one.vault.int.company.io
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: team-one-issuer-vault
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
truststore.p12:  1210 bytes
ca.crt:          1281 bytes
keystore.p12:    4391 bytes
tls.crt:         2550 bytes
tls.key:         1679 bytes

5. The client certificate is mounted as volume in the client application:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  Template:
    metadata:
      labels:
        app: client
    spec:
      volumes:
        - name: client
            secret:
            secretName: client
        - name: config
          configMap:
            name: client
      containers:
        - name: client
            ports:
            - containerPort: 8443
              protocol: TCP
            resources: {}
            volumeMounts:
            - name: client
              readOnly: true
              mountPath: /deployments/tls
            - name: config
              mountPath: /deployments/config

6. The server certificate is mounted as volume in the server application:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: server
spec:
  replicas: 1
  Selector:
  matchLabels:
    app: server
  template:
    metadata:
      labels:
        app: server
    spec:
      volumes:
        - name: server
          secret:
            secretName: server
         - name: config
             configMap:
            name: server
      Containers:
        - name: server
          ports:
            - containerPort: 8443 
              protocol: TCP
             resources: {}
             volumeMounts:
            - name: server
              readOnly: true
              mountPath: /deployments/tls
            - name: config
              mountPath: /deployments/config

7. Client-Server connection via mTLS

The sample code is based on Quarkus, we define ssl configuration in the client application.properties file as follow:

org.acme.client.mtls.GreetingService/mp-rest/url=https://server:8443
org.acme.client.mtls.GreetingService/mp-rest/trustStore=/deployments/tls/truststore.p12
org.acme.client.mtls.GreetingService/mp-rest/trustStorePassword=123423556
org.acme.client.mtls.GreetingService/mp-rest/keyStore=/deployments/tls/keystore.p12
org.acme.client.mtls.GreetingService/mp-rest/keyStorePassword=123423556

quarkus.http.ssl.certificate.key-store-file=/deployments/tls/keystore.p12
quarkus.http.ssl.certificate.key-store-password=123423556
quarkus.http.ssl.certificate.trust-store-file=/deployments/tls/truststore.p12
quarkus.http.ssl.certificate.trust-store-password=123423556
quarkus.ssl.native=true

As well, in server application.properties file:

quarkus.ssl.native=true
quarkus.http.ssl.certificate.key-store-file=/deployments/tls/keystore.p12
quarkus.http.ssl.certificate.key-store-password=123423556
quarkus.http.ssl.certificate.trust-store-file=/deployments/tls/truststore.p12
quarkus.http.ssl.certificate.trust-store-password=123423556

quarkus.http.ssl.client-auth=required

In both files the ssl.native is set to true, keystore and truststore is defined and client-auth field is set to required.

For more information see the blog, Quarkus Mutual TLS, to explore in detail how SSL and Mutual TLS works in Quarkus.

8. Access client exposed application:

oc extract secret/client --keys=ca.crt
curl --cacert ca.crt https://client-team-one.apps.${BASE_DOMAIN}/hello-client
hello from server

Conclusion

In this article, we walked through an effective and automated approach to manage the lifecycle of application certificates in a Kubernetes environment. This was accomplished by building a certificate authority (CA) in Vault, creating the CA chain hierarchy with an offline root CA and online intermediate CAs.

We also explored cert-manager, the de facto cloud-native solution for certificate issuance and renewal. Cert-manager interacts with HashiCorp Vault, an identity management system. We then introduced how Vault can be installed in a HA manner using integrated storage and leverage SSL certificates issued by cert-manager. We also described the integration of HashiCorp Vault with Kubernetes Authentication method and service account to manage access to resources within Vault.

The configuration of Vault is defined as a declarative process which is then leveraged by the Vault Config Operator. The X.509 certificates are provided by Vault’s PKI engine, using roles and policies. The cert-manager issuer will connect to the specific PKI engine and request an application certificate.

Last but not least, to understand the user and developer experience, we introduced a demo application that makes use of the full integration involving all the players in the game. Ultimately, the end user can request a certificate through cert-manager and HashiCorp Vault. Certificate secrets are available at Openshift namespace and can be mounted by any application to enable end to end security via trusted chain.