Introduction
Wouldn’t it be nice if your developers could be kept within the boundaries of a secure, signed image management process? That’s what image registries are for, naturally, and working with them is an important part of building security and quality into your developers’ everyday CI/CD processes.
Running a cloud container platform may imply dependency upon external registries. Some images can be used as-is, and others as source images for build configurations.
To keep your users happy and efficient, developers will need to pull images from external sources. But for security reasons, administrators should do a few things first:
- Whitelist a valid set of registries defined as secure sources.
- Ensure images are not tampered.
- Authorize only a set of images.
To achieve these requirements, a signature validating mechanism exists on RHEL hosts and within the OpenShift platform. In this article, we will explain the mechanism and how to implement it using a RHEL host on OpenShift. We will cover the following items:
- How to sign an image.
- How to deploy a web server signature store (SigStore).
- How to implement signatures on the OpenShift platform.
Note: Red Hat provides SigStores for quay.io, registry.redhat.io and registry.access.redhat.com with signatures of all Red Hat official images. There is no need to configure a custom sigstore for these external image hosting services, Red Hat provides already public SigStores that can be used. The following procedure explains how to configure them with OpenShift:
https://access.redhat.com/verify-images-ocp4
In this article, we will be using docker.io registry to demonstrate our use case with an external registry where we don’t have control of all the images. Administrators can replace this with a local enterprise registry in the rest of the procedure.
Signing Images
Step 1 - Generate the gpg key
To sign images, we need to use the gpg2 key mechanisms. The following procedure shows how to achieve this in a RHEL host.
First we need to generate the gpg key and trust it:
$ gpg2 --gen-key
...
Real name: saberkan
Email address: saberkan@redhat.com
You selected this USER-ID:
"saberkan <saberkan@redhat.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
...
pub rsa2048 2020-12-05 [SC] [expires: 2022-12-05]
ID_XXXXXXXXX
uid saberkan <saberkan@redhat.com>
sub rsa2048 2020-12-05 [E] [expires: 2022-12-05]
# find ID from last output
$ gpg2 --edit-key ID_XXXXXXXXX trust
1 = I don't know or won't say
2 = I do NOT trust
3 = I trust marginally
4 = I trust fully
5 = I trust ultimately
m = back to the main menu
Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
...
$ gpg> quit
Then we need to export the gpg public key to be used by consumers (in our case OpenShift):
$ mkdir -p /etc/pki/developers/
$ gpg2 --armor --export saberkan@redhat.com > /etc/pki/developers/signer-key.pub
Step 2 - Pull the image locally, or build the image locally
Let’s pull an image locally and sign it before the system-wide registry configuration:
$ podman pull docker.io/saberkan/ubi-minimal:latest
Trying to pull docker.io/saberkan/ubi-minimal:latest...
Getting image source signatures
Copying blob aebb8c556853 skipped: already exists
Copying blob 0fd3b5213a9b skipped: already exists
Copying config 28095021e5 done
Writing manifest to image destination
Storing signatures
28095021e526ad1dd5a65e11dc0fe4b34999ec398dbc60743f4b121d6bc9fc81
Step 3 - Configure a local SigStore for a registry in my host
Modify policy to check signatures with a local SigStore:
$ cat <<EOF > /etc/containers/policy.json
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports":
{
"docker": {
"docker.io": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/developers/signer-key.pub"
}
]
},
"docker-daemon":
{
"": [{"type":"insecureAcceptAnything"}]
}
}
}
EOF
Add the following in /etc/containers/registries.d/default.yaml:
$ cat <<EOF > /etc/containers/registries.d/default.yaml
docker:
docker.io:
sigstore: file:///var/lib/containers/sigstore
sigstore-staging: file:///var/lib/containers/sigstore
EOF
Step 4 - Push the image and sign it, then check the local signature
$ podman push docker.io/saberkan/ubi-minimal:latest --sign-by saberkan@redhat.com
Getting image source signatures
Copying blob 3485805ce47c done
Copying blob b0e2911c99f3 done
Copying config 28095021e5 done
Writing manifest to image destination
Signing manifest
Storing signatures
$ tree /var/lib/containers/sigstore/
/var/lib/containers/sigstore/
└── saberkan
└── ubi-minimal@sha256=f19c5b5d417cad1452ced0d174bca363ac41554190406c9147488b58394e2c56
└── signature-1
Now only the signed image can be pulled; other images from the external registry cannot be pulled.
Pull the signed image:
$ podman pull docker.io/saberkan/ubi-minimal:latest
Trying to pull docker.io/saberkan/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 438b5a18fd81 skipped: already exists
Copying blob 9d49c31ade4f skipped: already exists
Copying config 28095021e5 done
Writing manifest to image destination
Storing signatures
28095021e526ad1dd5a65e11dc0fe4b34999ec398dbc60743f4b121d6bc9fc81
Pull unsigned image:
$ podman pull docker.io/saberkan/s2i-tomcat7-jdk8-builder:latest
Trying to pull docker.io/saberkan/s2i-tomcat7-jdk8-builder:latest...
A signature was required, but no signature exists
Error: error pulling image "docker.io/saberkan/s2i-tomcat7-jdk8-builder:latest": unable to pull docker.io/saberkan/s2i-tomcat7-jdk8-builder:latest: unable to pull image: Source image rejected: A signature was required, but no signature exists
Deploy a web server SigStore
Administrators can deploy a local http web server in the same host that pushes and signs the image. The SigStore can then be exposed with an URL, and each time a signature occurs the SigStore is updated immediately.
It’s up to the administrator to choose the way of exposing the SigStore; in our case, it’s a simple httpd server where the sigstore was copied, and exposed under an URL: https://mywebserver/sigstore/
$ cp -r /var/lib/containers/sigstore /var/www/html/
Implementation on OpenShift 4
Step 1 - Configuration
There are two different ways to configure /etc/containers/policy.json under openshift:
- Using the machine config operator (MCO)
- Using image.config.openshift.io/cluster CR
The second option is limited as it only allows you to limit the resources registries, but doesn’t allow you to configure SigStores. Both options edit the same file:/etc/containers/policy.json on OpenShift nodes. So, be aware to not use both MCO and image.config.openshift.io/cluster CR to whitelist registries with registrySources.allowedRegistries parameter, because the second configuration will erase the first one.
To achieve the configuration on OpenShift, we just need to configure MCO to use our SigStore for our registry, and limit the registries that can be used to pull images by users with image.config.openshift.io/cluster with the allowedRegistriesForImport parameter, which doesn’t alter our policy.json file.
Note: The following link describes the difference between registrySources.allowedRegistries and allowedRegistriesForImport parameters: :https://access.redhat.com/solutions/4931451
In our case, we will be using the following:
$ oc edit image.config.openshift.io/cluster
kind: Image
metadata:
annotations:
name: cluster
spec:
allowedRegistriesForImport:
- domainName: quay.io
- domainName: registry.redhat.io
- domainName: docker.io
We create configuration for the external registry, and convert everything including the gpg public key to base64 to be used with the machine configuration.
$ cat > policy.json <<EOF
{
"default": [
{
"type": "insecureAcceptAnything"
}
],
"transports": {
"docker": {
"docker.io": [
{
"type": "signedBy",
"keyType": "GPGKeys",
"keyPath": "/etc/pki/developers/signer-key.pub"
}
]
},
"docker-daemon": {
"": [
{
"type": "insecureAcceptAnything"
}
]
}
}
}
EOF
$ cat <<EOF > docker.io.yaml
docker:
docker.io:
sigstore: https://mywebserver/sigstore/
EOF
$ export DOCKER_REG=$( cat docker.io.yaml | base64 -w0 )
$ export SIGNER_KEY=$(cat /etc/pki/developers/signer-key.pub | base64 -w0 )
$ export POLICY_CONFIG=$( cat policy.json | base64 -w0 )
Then we create and apply the machine configuration:
$ cat > worker-custom-registry-trust.yaml <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: worker-custom-registry-trust
spec:
config:
ignition:
config: {}
security:
tls: {}
timeouts: {}
version: 2.2.0
networkd: {}
passwd: {}
storage:
files:
- contents:
source: data:text/plain;charset=utf-8;base64,${DOCKER_REG}
verification: {}
filesystem: root
mode: 420
path: /etc/containers/registries.d/docker.io.yaml
- contents:
source: data:text/plain;charset=utf-8;base64,${POLICY_CONFIG}
verification: {}
filesystem: root
mode: 420
path: /etc/containers/policy.json
- contents:
source: data:text/plain;charset=utf-8;base64,${SIGNER_KEY}
verification: {}
filesystem: root
mode: 420
path: /etc/pki/developers/signer-key.pub
osImageURL: ""
EOF
$ oc apply -f worker-custom-registry-trust.yaml
Note: Please note that the previous deployment is only applied for worker nodes. It is recommended to apply on master nodes too, and this can be done by creating an additional machine config. But since we do not run a custom workload on master, we just apply it to workers, and this is done with “machineconfiguration.openshift.io/role: worker” label.
Step 2 - Validation
Once the machine config is applied, workers will restart one by one to apply the ignition file with the signature configurations:
$ oc get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-152-59.eu-west-1.compute.internal Ready master 3h43m v1.17.1+20ba474
ip-10-0-154-182.eu-west-1.compute.internal Ready,SchedulingDisabled worker 3h33m v1.17.1+20ba474
ip-10-0-163-170.eu-west-1.compute.internal Ready worker 3h33m v1.17.1+20ba474
ip-10-0-187-245.eu-west-1.compute.internal Ready master 3h44m v1.17.1+20ba474
ip-10-0-217-159.eu-west-1.compute.internal Ready master 3h45m v1.17.1+20ba474
Now, let’s check whether the mechanism works as expected. First, we will deploy a pod using the signed image with signature in the sigstore:
$ cat <<EOF |oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: example
namespace: saberkan-signature
spec:
containers:
- name: example
image: docker.io/saberkan/ubi-minimal:latest
EOF
Note: the pod used here is just an example ubi-minimal; it doesn't have a running application, so don’t worry if the pod fails to start (CrashLoopBackOff). We focus here on the signature aspect.
You can see in the pod events that the image is successfully pulled:
Now, let’s deploy another image from the external registry that doesn't have a valid signature in sigstore:
$ cat <<EOF |oc create -f -
apiVersion: v1
kind: Pod
metadata:
name: example-no-signature
namespace: saberkan-signature
spec:
containers:
- name: example-no-signature
image: docker.io/saberkan/s2i-tomcat7-jdk8-builder:latest
EOF
You can see in the pod events that the image can not be pulled for missing signature, and the pod will remain in ImagePullBackOff status:
Conclusion
In this article, we demonstrated how to create a sigstore, and how to add signatures to it for images built internally, or to limit images that can be deployed from an external registry. There are many different methods and tools for signing images, but with this approach, administrators can integrate the mechanism into their CI/CD pipeline. And that can incorporate security and quality assurance right into your users’ everyday development work
Categories
