Have you ever found yourself forgetting to set some key attributes on your Kubernetes cluster? Or perhaps others have failed to set them? Maybe you have encountered annoying missing labels, incorrect security policies, or network settings.

Gatekeeper is a great tool for alleviating all this. Until recently, it allowed us to validate incoming resources and reject nonconformant ones based on user-defined policies. This is great but still left the burden of updating the faulty resources manually. With the new mutation feature, updating nonconformant resources can be automated.

How Gatekeeper Mutation Works

The gatekeeper mutation system introduces two new mutation policy CRD’s: Assign and AssignMetadata, which allow the user to define the desired mutations. When a mutation policy is created, the mutation system parses it and adds to a set of mutators. Whenever a Kubernetes resource is created, it is intercepted by the gatekeeper mutating admission webhook, the mutators are applied, and the resource is mutated

Defining Mutations

The mutation policies are defined by means of mutation specific CRDs:

  • AssignMetadata - defines changes to the metadata section of a resource. The metadata section is sensitive to change, and hence has more limitations than other rules. Only labels and annotations can be modified
  • Assign - any change outside the metadata section

Each mutation CRD consists of three major sections:

  • Mutation scope - describes which resources are to be mutated
  • Mutation intent - the location of the mutation and the value to be assigned
  • Conditional - conditionals for performing the mutation

Mutation Scope

The mutation scope defines what resources will be mutated. It consists of two sections:

- applyTo: This defines the resources to be mutated by gvk. This allows gatekeeper to validate that the schema used in other parameters is correct (for example, the path to be modified). This is currently only valid for the Assign CRD.

- match: This allows us to filter the mutated resource by kinds, labelSelector, namespaceSelector, namespaces, and excluded namespaces.

Mutation Intent

This section contains the location to be mutated and the value of the mutation. Note that the value can be a composite value. This will be shown in an example later.

Conditionals: Conditionals will allow basing mutations on pre-existing values. As of gatekeeper version 3.4.0, this is still not implemented.

Installing Gatekeeper.

Gatekeeper mutation is in the alpha stage and as such, needs a few tweaks in the deployment.

Please use the following manifest to install gatekeeper with mutation enabled:

kubectl apply  -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/gatekeeper.yaml

A “gatekeeper-system” namespace should be created, containing the gatekeeper-controller-manager and gatekeeper-audit pods. A “gatekeeper-mutating-webhook-configuration” mutating webhook should also be present.

Installing gatekeeper using the gatekeeper operator

An alternative way to install gatekeeper is using the gatekeeper operator. The operator allows you to install and easily modify the gatekeeper configuration.

To install the operator, run:

kubectl apply -f https://raw.githubusercontent.com/gatekeeper/gatekeeper-operator/v0.1.2/deploy/gatekeeper-operator.yaml

The operator then needs a config to deploy gatekeeper. The config allows control of gatekeeper parameters such as gatekeeper pod node selectors, webhook failure policies, and more. An example of a config manifest with all values can be found here.

The simplest config enabling mutation can added by running:

kubectl apply -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/gatekeeper-operator-config.yaml

Examples

Let's  look at some use cases for using gatekeeper mutations.

Owner Annotation for All Pods

First, let’s add an owner annotation set to admin to each Pod created. Each created/updated pod will be mutated to have a field “metadata.annotations.owner" with the value “admin”. Below is the definition of an AssignMetadata for this mutation

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: AssignMetadata
metadata:
name: demo-annotation-owner
namespace: default
spec:
  match:
    scope: Namespaced
    kinds:
      - apiGroups: ["*"]
        kinds: ["Pod"]
  location: "metadata.annotations.owner"
  parameters:
    assign:
      value:  "admin"

To apply this mutation, run the following command:

kubectl apply -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/examples/1_demo_annotation_owner.yaml

Create a pod by running:

kubectl create -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/examples/pod.yaml

Examine the pod and note that an owner annotation has been added:

kubectl get pod demo -o yaml

Set securityContext.privileged=false for all pods

This example will add set securityContext.privileged to false for containers named “foo” in created pods.

Below is the definition of an Assign for this mutation. Note that the path specifies which containers the mutation will apply to: “spec.containers[name:foo]”

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
name: demo-privileged
namespace: default
spec:
  match:
    scope: Namespaced
    kinds:
      - apiGroups: ["*"]
        kinds: ["Pod"]
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.containers[name:foo].securityContext.privileged"
  parameters:
    assign:
      value: false

To apply this mutations, run the following:

kubectl apply -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/examples/2_demo_privileged.yaml

Create a pod by running:

kubectl create -f https://raw.githubusercontent.com/mmirecki/blog/main/gatekeeper/manifests/examples/pod.yaml

Examine the pod and note that the “foo” container now has securityContext.privileged set to false.

Adding a Sidecar

In this example, a sidecar container will be added to a created pod. Let’s assume we want to add a “networking” container as a sidecar to our pod.

Below is the definition of an Assign for this mutation. Note that the value in this case is a composite value, containing the definition of the container being added.

apiVersion: mutations.gatekeeper.sh/v1alpha1
kind: Assign
metadata:
name: demo-sidecar
namespace: default
spec:
  match:
    scope: Namespaced
    kinds:
      - apiGroups: ["*"]
        kinds: ["Pod"]
  applyTo:
  - groups: [""]
    kinds: ["Pod"]
    versions: ["v1"]
  location: "spec.containers[name:networking]"
  parameters:
    assign:
      value:
        name: "networking"
        imagePullPolicy: Always
        image: docker.io/centos
        command: ["/bin/bash", "-c", "sleep INF"]

Examine the created pod, and note that a sidecar container was added.