Some Background

The state of the art for application packaging and supply chain security for containers continues to evolve at a rapid pace. As a registry, Quay.io needs to strike a careful balance between feature stability and adoption of new technologies as they become mainstream. To that end, we are announcing that on June 15, 2022 we will be removing the support for Applications and legacy Repository Signing from our service.

Application packaging is a feature that is currently sitting behind the “Applications” tab on the Quay.io user interface::


Applications are based on
an older project called “appr” dating back to Quay’s time as a CoreOS product. As a new application packaging standard for containerized deployments, the Applications model of Quay never really caught on.  Because of this, the feature was deprecated for the on-premise Quay customers over a year ago. The largest adopters of Applications were the first iteration of Operator catalogs for OpenShift 4.5 and earlier which have moved to a container-based format in the meantime and since then usage has dwindled considerably. Today the industry has rallied behind other projects like Helm to serve a similar purpose which also opened up support for storing Helm charts in the form of OCI images Check out this blog article for more background on using Helm charts with Quay and read on for hands-on instructions below for Quay.io.

Similarly, for container image signing, Quay.io has supported a flavor of Docker Content Trust based on the project Apostille.

While this feature is very simple to use on Quay.io, its adoption has been incredibly small over the past several years (less than half of one percent of all repositories). In addition, the state of the art in this space continues to evolve and currently Red Hat is deeply engaged in the Sigstore project to serve a similar purpose. You can use the cosign utility to sign images in a portable way, which is natively supported by Quay.io and Red Hat Quay since version 3.6. Read on for example usage instructions below.

What Should You Do?

If you are currently using the Applications feature on Quay.io please migrate your packaging over to another solution like Helm V3. With Helm V3 it is very straightforward to package your charts into OCI compliant images that can be pushed to and pulled from Quay.io. Currently the Helm support on Quay.io is fairly bare-bones, but we will continue to enhance this user experience over time (in particular how we visualize Helm charts differently from regular OCI container images).

Here’s an example of how this is done using Helm version 3.8.1. First go ahead and install Helm and get an example chart that you want to push to Quay.io.

$ helm version
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}

$ helm repo add redhat-cop https://redhat-cop.github.io/helm-charts
"redhat-cop" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "redhat-cop" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm pull redhat-cop/etherpad --version=0.0.4

$ ls -l etherpad-0.0.4.tgz
-rw-r--r--. 1 bill bill 3568 Apr  6 16:18 etherpad-0.0.4.tgz

Now log into Quay.io and push the chart tarball as an OCI image.

$ helm registry login -u bdettelb quay.io
Password:
Login Succeeded

$ helm push etherpad-0.0.4.tgz oci://quay.io/bdettelb
Pushed: quay.io/bdettelb/etherpad:0.0.4
Digest: sha256:74420a94f26201498cc0cd7356952a208391a30ac9770dd36092de97df594d3b

We can verify the chart exists in Quay.io by looking at the web UI or via the command line.

$ crane manifest quay.io/bdettelb/etherpad:0.0.4 | jq .
{
"schemaVersion": 2,
"config": {
  "mediaType": "application/vnd.cncf.helm.config.v1+json",
  "digest": "sha256:0195f05c06e229b8b0e9fdcfb4663c13d3f255c913cd7318a6d65f0e9df49bab",
  "size": 318
},
"layers": [
  {
    "mediaType": "application/vnd.cncf.helm.chart.content.v1.tar+gzip",
    "digest": "sha256:c37c7efe00a1753619d30694095b6a122f63fbfe3aebf9921e9410ee65716478",
    "size": 3568
  }
]
}

As a final test, let’s remove the local tarball and re-pull it back from Quay.io.

$ rm etherpad-0.0.4.tgz

$ helm pull oci://quay.io/bdettelb/etherpad --version 0.0.4
Pulled: quay.io/bdettelb/etherpad:0.0.4
Digest: sha256:74420a94f26201498cc0cd7356952a208391a30ac9770dd36092de97df594d3b

$ ls -l etherpad-0.0.4.tgz
-rw-r--r--. 1 bill bill 3568 Apr  6 16:27 etherpad-0.0.4.tgz

You can also directly install the chart from a registry:

$ helm install oci://quay.io/bdettelb/etherpad --version 0.0.4

If you are currently signing your registries on Quay.io, there is nothing special you need to do here, however note that the ability to sign new repositories after June 1, 2022 will be removed. We strongly encourage using cosign as the method of signing your images going forward. Cosign also uses an OCI compliant image format to store signatures into your repository alongside your container images. This makes it very simple to use and understand. Much like our Helm support, the Cosign support is fairly simple and we will be continuing to expand how we make cosign signatures first-class citizens on the Quay.io web interface.

Here’s an example of how to do this with Cosign 1.7.1. First generate a keypair that you will use to sign your images.

$ cosign generate-key-pair
Enter password for private key:
Enter password for private key again:
Private key written to cosign.key
Public key written to cosign.pub

$ ls -l
total 12
-rw-------. 1 bill bill  649 Apr  6 13:58 cosign.key
-rw-r--r--. 1 bill bill  178 Apr  6 13:58 cosign.pub

Then let’s sign an image on Quay.io with our private key- this pushes a signature image alongside our image into the repository.

$ cosign login quay.io -u bdettelb
auth.go:191: logged in via /home/bill/.docker/config.json
$ cosign sign –key cosign.key quay.io/bdettelb/ubi-minimal:8.5-test
Enter password for private key:
Pushing signature to: quay.io/bdettelb/ubi-minimal

Now we can verify that our image is signed correctly against the public key.

$ cosign verify --key cosign.pub quay.io/bdettelb/ubi-minimal:8.5-test
Verification for quay.io/bdettelb/ubi-minimal:8.5-test --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- The signatures were verified against the specified public key
[{"critical":{"identity":{"docker-reference":"quay.io/bdettelb/ubi-minimal"},"image":{"docker-manifest-digest":"sha256:8fcb76d411d36b8dfac1852e8d73dd291976bc91976f43611210751b5d478d82"},"type":"cosign container image signature"},"optional":null}]

$ echo $? # Exit status 0 means the verification was successful
0

It’s also possible to get information about the signature image that was generated. Use the ‘triangulate’ command to work backwards from our signed image to discover which image contains the signature.

$ cosign triangulate quay.io/bdettelb/ubi-minimal:8.5-test
quay.io/bdettelb/ubi-minimal:sha256-8fcb76d411d36b8dfac1852e8d73dd291976bc91976f43611210751b5d478d82.sig

$ crane manifest $(cosign triangulate quay.io/bdettelb/ubi-minimal:8.5) | jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
  "mediaType": "application/vnd.oci.image.config.v1+json",
  "size": 233,
  "digest": "sha256:02755357447b49a7e872bab62a3f4f93c5a9c163deef0cf0bf2fa1f8472ae335"
},
"layers": [
  {
    "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
    "size": 244,
    "digest": "sha256:a89d24c6e6e404d37e4571779bfe91cdb607a34d38dfff584a7996189a5febc1",
    "annotations": {
      "dev.cosignproject.cosign/signature": "MEYCIQD6xMj+Fb8kheB6DKqsOsHPB2G+dYucyTEgclVYvwxG9wIhAIK79YR7JCcYrCwL8LYw3EqHnynIiKYRqrqrxZO+DMbl"
    }
  }
]
}

With the advent of Helm and Cosign now using standard OCI formatted container images, it’s relatively simple to incorporate this packaging and signing capability into your Quay.io workflows and automation. We’re excited to see projects like these continuing to leverage the extensibility of OCI’s artifact support. We look forward to continuing to extend Quay’s ability around these and yet-to-be discovered ways of securing and delivering images for your applications.

Got Questions?

Feel free to reach out to the Quay team via the Quay SIG.  We’d love to hear your questions and understand how you are using Helm or Cosign in your workflows with Quay.io! 


About the author

I joined Red Hat in 2019 to look after Quay, quay.io and the OpenShift Image Registry. I have been in software all of my career, doing development and product management both as a customer and a vendor.

Read full bio