Introduction

HashiCorp Vault is a popular secret management service with integrations for cloud native applications running in Kubernetes.

Over the years, we have been getting better and better at accessing secrets in Vault in a secure fashion and consuming them in containers running on Kubernetes. There is a good amount of literature on the subject and perhaps the best place to start is Vault’s own tutorial. If you are working in an OpenShift environment, this blog is also worth reading.

But, fetching secrets out of Vault is just the tail end of a more complex workflow. This process begins with configuring Vault to be able to serve those secrets. This typically requires configuring one or more secret engines as well as possibly one or more authentication engines. If you add these components along with the orthogonal complexity of configuring multi-tenancy for both Vault and Kubernetes, Vault configurations can get complicated very quickly.

Notice that many tutorials on Vault use the Key/Value (KV) store secret engine as it is easy to set up. But, when leveraging the KV engine, someone or some type of automation has to seed the secrets in Vault. This is not the best practice because with that approach the secret needs to exist and be known outside of Vault posing the problem of where to store it. A better approach is to have Vault auto-generate the secrets by configuring one of the other secret engines, in such a way that secrets never had to exist outside of Vault, thus removing the issue of how or where they should be stored. Each secret engine type has the capacity to generate secrets for a specific subsystem that needs to be protected and the configurations vary per each subsystem.

So, it would be ideal to have a method for automating the configurations of Vault secret engines and other Vault APIs in order to have a fully operational Vault with secrets ready to be consumed with as little barrier to entry as possible.

In this blog, we will explore an operator based approach to achieve these goals.

Why an Operator Based Approach?

The folks at HashiCorp gave a speech several years ago on why they were not going to build an operator for Vault. So, this once again brings up the question: Do we need an operator to solve the problem of configuring Vault?

The core message of the speech was: if all your operator is doing is to create a bunch of manifests, then there are other possibly better approaches, such as, for example, creating an Helm chart and letting a GitOpsoperator manage their lifecycle.

This was the primary reason brought forth by HashiCorp for deciding not to build an operator to install Vault. While I believe their reasoning is sound, they were only considering the deployment of Vault, not the actual configuration. When you configure Vault, you don't just create some manifests, you actually change the state of the Vault system and these are tasks that a full blown operator is well equipped to do.

The other advantage of a Vault operator is that we can now configure Vault by creating a set of Custom Resources (CRs), which in turn can be packaged as Helm charts or kustomize manifests and managed in a GitOps way. So, configuring Vault is no longer an imperative action, but is now declarative.

Another possible objection to this approach is that there are already Vault operators and kubernetes-related automations available, such as the Vault sidecar and the Vault CSI secret storage provider from HashiCorp along with the external secrets operator in the community. All of these operators help with fetching secrets out of Vault, but I have not found much in the way of configuring Vault.

Vault Configuration Operator Design

In order to support a variety of Vault configuration workflows and conventions, the operator was designed to offer APIs in Kubernetes that reflected Vault’s APIs with a high-level of fidelity. This means that each resource field in a Vault API has a corresponding field in the Kubernetes CRD. There are, however, a few exceptions to this rule, such as for credentials that must not be visible along with some Kubernetes-dependent fields.

Another key principle that went into the design of this operator was to create isolation between different tenants. This was accomplished by not providing any credentials for accessing Vault directly to the operator and to, instead, use the Kubernetes authentication mechanism with a service account from the namespace where the CR being processed is defined. This does require that tenants using this operator must have previously given some permissions to Vault. This approach should minimize the risk of permission leaking.

With these two principles in place, we tried to make it as simple as possible to add new API types as necessary. Vault has several different secret and authentication engines that could be supported by this operator, but at the time of this initial release, the coverage is limited. Contributions in this space are welcome.

You can find more information on the vault-config-operator and instructions on how it can be installed here.

An End-to-End Example

To illustrate the capabilities of this operator, let’s take a look at an example scenario.

In this situation, we want to configure Kubernetes and Vault with the following:

1. Each development team will have a dedicated Kubernetes authentication endpoint in Vault. Each team may leverage multiple Kubernetes namespaces for their application lifecycle  environments. Each namespace will be labeled with the name of the team name and the application lifecycle environment (dev, qa, prod...). So for example team=team-a and environment=dev
2. The path for this auth endpoint should be auth/cluster1/{team}-kubernetes. We assume that we are working in one (cluster1) of many clusters that Vault serves.
3. At this authentication endpoint, teams receive one of the following roles: a secret-engine-configuration role and a secret-reader role.
4. Teams are allowed to create secret engines of type database and kv.
5. The path for these secret engines should be {team}/{engine_name}.

In order to create the Vault authentication endpoint, we need to create the same set of configuration for each dev namespace. To do that, we can use the namespace-configuration-operator . This operator allows us to create and enforce a set of manifests in multiple namespaces selected by a label. The following is the namespace configuration manifest that we need:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: NamespaceConfig
metadata:
 name: vault-config
spec:
 labelSelector:
   matchLabels:
     environment: dev
 templates:
 - objectTemplate: |
     apiVersion: redhatcop.redhat.io/v1alpha1
     kind: AuthEngineMount
     metadata:
       name: {{ index .Labels "team" }}-kubernetes
       namespace: vault-admin
     spec:
       authentication:
         path: cluster1-admin
         role: vault-admin
       type: kubernetes
       path: cluster1
 - objectTemplate: |
     apiVersion: redhatcop.redhat.io/v1alpha1
     kind: KubernetesAuthEngineConfig
     metadata:
       name: {{ index .Labels "team" }}-kubernetes
       namespace: vault-admin
     spec:
       authentication:
         path: cluster1-admin
         role: vault-admin
       path: cluster1      
 - objectTemplate: |
     apiVersion: redhatcop.redhat.io/v1alpha1
     kind: KubernetesAuthEngineRole
     metadata:
       name: {{ index .Labels "team" }}-secret-engine-admin
       namespace: vault-admin
     spec:
       authentication:
         path: cluster1-admin
         role: vault-admin
       path: cluster1/{{ index .Labels "team" }}-kubernetes
       policies:
         - {{ index .Labels "team" }}-secret-engine-admin
       targetServiceAccounts:
       - default
       targetNamespaces:
         targetNamespaceSelector:
           matchLabels:
             team: {{ index .Labels "team" }}

A NamespaceConfig allows for a templated resource to be applied to multiple namespaces matching a set of labels. In this case, this NamespaceConfig resource will create a set of objects within namespaces matching the label environment=dev.

The first object is an AuthEngineMount, which defines a Vault authentication engine configuration. It is mounted at auth/cluster1/{{ index .Labels "team" }}-kubernetes.

The second object is a KubernetesAuthEngineConfig, which configures the auth engine mount to point to a specific Kubernetes master API endpoint.

The third object is a KubernetesAuthEngineRole, which configures all of the default service accounts in namespaces with label “team: {{ index .Labels "team" }}” to receive the secret-engine-admin role.

These three objects complete the setup of the per team Vault authentication endpoint. We have redacted the object which defines the secret-reader role and the policies associated with the secret-engine-admin and the secret-reader roles to keep the code fragment to a manageable size. You can see the full code here.

The above configuration will be applied for all of the namespaces that qualify, giving each team a way to start defining their secret engines. Assuming that team-a has been onboarded as described previously and that it has created a postgresql database in their namespace, the following describes how they can manage the credentials using the Vaut PostgreSQL Database Secret Engine. Refer to this GitHub repository for a full step by step tutorial for which we will provide an overview below.  

First, we need to create a secret engine mount of type database with a SecretEngineMount CR:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: SecretEngineMount
metadata:
 name: postgresql
spec:
 authentication:
   path: cluster1/team-a-kubernetes
   role: team-a-secret-engine-admin
 type: database
 path: cluster1/team-a
Notice how every time we create a Vault configuration object, we need to specify how we want to authenticate to Vault via the authentication field.
Then, we need to create a database secret engine configuration with a DatabaseSecretEngineConfig CR, also known as connection:
apiVersion: redhatcop.redhat.io/v1alpha1
kind: DatabaseSecretEngineConfig
metadata:
 name: postgresql-dev
spec:
 authentication:
   path: cluster1/team-a-kubernetes
   role: team-a-secret-engine-admin
 pluginName: postgresql-database-plugin
 allowedRoles:
   - read-write
   - read-only
 connectionURL: postgresql://{{username}}:{{password}}@my-postgresql-database.team-a-dev.svc:5432
 rootCredentials:
   secret:
     name: postgresql-admin-password
   passwordKey: postgresql-password
 username: postgres
 path: cluster1/team-a/postgresql

Notice how we pass a secret reference to provide the administrative credentials of the PostgreSQL database. This is a basic way to initialize the secret engine which makes use of a Kubernetes secret, but we can do better:

1. If the database container supports having credentials injected from a Vault secret, we can use the RadomSecret CRD, defined by the vault-config-operator, to generate a random password for the administrative account. This random password can then be consumed by the database container as a normal Vault secret and at the same time by the DatabaseSecretEngineConfig via the rootCredentials.randomSecret field.
2. If the secret engine supports rotating the administrative credentials, then we can pass the credentials as a Kubernetes secret, as we are doing now. In this case though, the credential gets immediately rotated rendering the Kubernetes secret non usable for subsequent authentications (the Kubernetes secret becomes a use-once credential).

Finally we need to create one or more database roles with the DatabaseSecretEngineRole CR. Here is an example:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: DatabaseSecretEngineRole
metadata:
 name: read-only
spec:
 authentication:
   path: cluster1/team-a-kubernetes
   role: team-a-secret-engine-admin
 path: cluster1/team-a/postgresql
 defaultTTL: 1h
 maxTTL: 24h
 dBName: postgresql-dev
 creationStatements:
   - CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";

At this point, Vault users with the permission to read at the path cluster1/team-a/postgresql/creds/read-only will be able to retrieve credentials to access the PostgreSQL database.

Conclusions

With this operator, we hope to help streamline all of the preparation steps that need to occur in Vault before secrets can be consumed by applications. The fact that both configuration and secret consumption can now happen from within Kubernetes, and in a self-serviced and declarative fashion, should expedite the adoption of Vault while at the same time, decrease the cognitive burden of needing to understand the full set of configuration options. As an additional note, after some more research, we found KubeVault, an operator which provides similar capabilities as the vault-configuration-operator. However, since this project is not open source, we could not provide an accurate comparison of the features and capabilities.