How to integrate Cosign image signature with RHACS to enhance the security of the software supply chain

Introduction

Currently, among the paramount concerns for enterprise IT organizations, supply chain security holds utmost significance. This is particularly critical given the increasing integration and utilization of open source artifacts within business-critical systems and applications. As open source code constitutes 75% of application code bases, these components are now subject to heightened scrutiny, driven by a staggering 742% increase in software supply chain attacks since 2020. In response, customers are actively looking to incorporate guardrails into their software supply chain and development life cycles, aiming to foster innovation while upholding robust security measures.

An established software supply chain offers organizations the advantage of implementing DevSecOps practices and tools, ensuring early security measures for software components during the SDLC and automating security procedures throughout every phase of the software development life cycle.

One aspect of securing the software supply chain is to ensure that all the container images deployed in Openshift clusters are signed and more secured. In this Blog we will show how to leverage Cosign and RHACS tools  to enforce the signature of the container images within the CI/CD Pipelines.

Prerequisite:

• Openshift cluster v4.11 or later
• Access to Quay.io and account created  to be able to pull and push created  images
• Red hat Openshift  Pipeline operator v1.10 or later
• RHACS v.3.74 or later installed and configured  
• Have Cosign,  roxctl and tekton cli (optional) installed in the local  environment
• Have access to the  public  repo which host  the voting application used for the demo
• Have access to the  public repo which contains our pipeline manifests.

Configure Quay.io repository

Access your Quay.io account and create a new public repository. Next create  a robot account user and grant  it an admin permission. Robot Account Token will be automatically created.

Add registry credentials to Openshift

We need to use the pull secret manifest that has been generated and apply it to the Openshift ‘demo’ project within our cluster. This will allow the service account to access the registry.

1. Create a project for the sample application:  
2. Apply the pull secret containing the credentials for Quay registry to push/pull the images and signatures:
3. Link the imagePullSecret to the ServiceAccount ‘pipeline’ :

 oc project demo

oc create -f szemmour-acs-demo-secret.yml -n demo

oc secrets link pipeline <pull_secret_name> -n demo

Integrate Roxctl CLI with RHACS

The command-line interface (CLI) roxctl is used for running commands against RHACS.

Since our  goal is to perform image checks for policy violations and reporting, an integration between roxctl cli and RHACS is needed. For that, the roxctl cli needs to be able to authenticate against the RHACS central cluster. Thus, the next step is to generate the API token to be used  for the authentication.

Via the RHACS console, Go to Platform Configuration -> Integrations -> Authentication Tokens -> API Token and generate new API token as  shown in below images:

Copy the generated token and export it into the ROX_API_TOKEN variable or store it in a file in the local environment. You will not be able to view it again.

export ROX_API_TOKEN=”<api_token>”

export ACS_ROUTE=$(oc get route -n stackrox central -o jsonpath='{.spec.host}')

To validate the authentication we can install Install the roxctl cli and use the roxctl with ‘check image’ option in order verify if the API Token is working properly:

roxctl --insecure-skip-tls-verify image check --endpoint $ACS_ROUTE:443 --image registry.access.redhat.com/ubi8/ubi-minimal:8.7

The command should output a couple of violated policies with their remediations  similar to image below.

Since the roxctl will be executed as part of a task in the pipeline, we will need to provide both rox_api_token and the rox_central_endpoint as an openshift secret. Create the secret in the ‘demo’ namespace using the command below.

oc create secret generic roxauthsecret --from-literal rox_api_token="$ROX_API_TOKEN" --from-literal rox_central_endpoint="$ACS_ROUTE:443" -n demo

Generate Cosign keys

Before we can perform image signing as part of our pipeline and to enable RHACS to verify those images we need to create a key-pair signature using the Cosign tool. The key-pair needs to be stored as a secret in the application namespace, and the public key needs to be placed in RHACS for later verification. Note that for a production environment an integration with a KMS provider will procure more secure key management. This documentation contains detailed instructions on how to configure Cosign to work with KMS providers eg: Hashicorp Vault, AWS KMS …

Cosign uses keys stored in Openshift Secrets to sign and verify signatures. In order to generate a secret you have to pass the Cosign generate-key-pair command a k8s://[NAMESPACE]/[NAME] URI specifying the namespace and secret name as described here.

cosign generate-key-pair k8s://demo/cosign

The Cosign command prompts the user to enter the password for the private key. Type any password that makes sense to you and hit Enter.

After generating the key pair, Cosign will store it in Openshift Secret using your current context. The secret will contain the private and public keys, as well as the password to decrypt the private key.

To validate the creation of Cosign secret. List the secrets under the ‘demo’ namespace and  make sure Cosign secret  exist.

$ oc get secret -n demo
NAME      TYPE    DATA   AGE
cosign    Opaque  3      2m11s
…

In the same folder where the Cosign generate-key-pair command was executed you will notice the creation of the  cosign.pub file. This file represents the public key of the Cosign private key. The same base64 content of the cosign.pub file is stored in the Cosign secret under the ‘demo’ namespace.

Later, when verifying an image signature in the Tekton pipeline using Cosign verify command the key will be automatically decrypted using the password stored in the openshift secret.

Configuring signature integration

Let’s add the Cosign public key generated in the previous steps into the ACS console in order for RHACS to execute image signature verification. More details could be  found here

Under Integrations select  Signature and then click on  New Integration to create a new signature integration. Next add integration name and copy/paste the public key from the generated cosign.pub file into the public key value section as shown in the image below.

Once we've created the signature integration we need to create a policy that will use it.

Create ACS Policy for Image Signature Verification

ACS comes with a lot of default policies to help organizations better meet their security requirements. These policies apply across the build, deploy and runtime phases of the container application lifecycle. The different stages can be selected under the Lifecycle stages section of the Policy behavior configuration as shown below.

Let's create a policy that applies to the Build and Deploy stages of the container application lifecycle, and we also want to enable enforcement at deployment time.

The Build option means that when a container image check is initiated with roxctl the policy will be reported in the output. The Deploy option means that the policy applies when a deployment is created on a secured cluster. This invokes the RHACS admission controller to validate the workload against the enforced policies

Let's create a policy that applies to the Build and Deploy stages of the container application lifecycle, and we also want to enable enforcement at deployment time.

Under Platform Configuration -> Policy Management  select  Create Policy and choose all the options shown in below images.

Now we need to configure the policy criteria. To achieve this,  add the Image signature policy into the Policy Criteria as shown in the image below.

Next, Click the Select button and specify the Cosign-integration signer.

Finally, review and save the policy.

Integration validation

To evaluate the integration, a pipeline will be created to build and deploy the vote api microservice. We will present two scenarios to demonstrate the capabilities of signature validation and enforcement. In the initial scenario, the pipeline will build and attempt to deploy the application utilizing an unsigned container image. The subsequent scenario follows a similar path, but in this case, the image will be signed before its deployment.

Deploy Tekton tasks and pipelines

Create the tasks which will be used by the pipelines later using the following command:

oc create -k "github.com/szemmour-rh/acs-cosgin-integration/pipelines/tasks/?ref=main" -n demo

You can take a look at the tasks you have created:

oc get task

Operator installs a few ClusterTask which you can see by executing:

Oc get clusterTask

We will be using buildah clusterTasks, which gets installed along with the Openshift Pipeline Operator. Buildah task builds source into a container image and then pushes it to a container registry. Create the pipelines and the PVC  using the following command:

oc create -k "github.com/szemmour-rh/acs-cosgin-integration/pipelines/pipeline/?ref=main" -n demo

Build and  deploy unsigned Images

Now let’s run the pipeline which builds an unsigned image, push it to Quay and deploy the application using the same unsigned image. For the purpose of the demo we will execute the apply-manifest task regardless of the result of the image-check task in order to be able to illustrate the state of the deployment.

oc create -f  https://raw.githubusercontent.com/szemmour-rh/acs-cosgin-integration/main/pipelines/pipelinesrun/unsigned/pipelinerun-unsigned.yaml  -n demo

We can observe that the image-check task failed and upon inspecting the logs of the task, we can see that the signature check policy is being violated.

When we look at our vote-api-unsigned application deployment status and event in the Openshift console we notice that the RHACS image verification policy via the admission controller is blocking the pipeline to use the unsigned image for deployment as  shown in below  images.

If you navigate to Violations in the RHACS console  you will notice that a policy violation has been created. The policy violation lists the container image that was not signed. Under the Enforcement tab you can also verify the action taken which is that the deployment vote-api-unsigned has been scaled to 0 replicas in response to the policy violation.

Build and  deploy signed image

This time we will include the image signing task after the build and we will use the newly generated image for the vote-api-signed deployment.

oc create -f https://raw.githubusercontent.com/szemmour-rh/acs-cosgin-integration/main/pipelines/pipelinesrun/signed/pipelinerun-signed.yaml -n demo

Compared to the previous PipelineRun, there’s no signature check policy listed under the  Policy check results of the image-check task logs as shown in the image below.

To make sure the image has been signed we can look at the image Tag in Quay.io which will contain a validation sign with the message “This Tag has been signed via cosign”.

Finally, we notice that the deployment vote-api-signed was successful.

Summary 

Delving into the realms of RHACS and Cosign Integration, this article offers a concise yet insightful glimpse into their synergy as open-source security endeavors, synergistically augmenting the safety of the software supply chain. Hence, by integrating these two initiatives, developers can foster a culture of heightened safety while constructing applications, fostering a robust and more secure development ecosystem.