The cert-manager Operator is now generally available in OpenShift

As the number of cloud-native workloads and applications increases, managing Transport Layer Security (TLS) certificates for each application can become daunting. Given that security is rightfully such a high priority for organizations, a tool that makes certificate management easier is critical to operationalizing TLS implementation in cloud-native applications.  

Red Hat is proud to announce the general availability of the cert-manager Operator, a cluster-wide application certificate lifecycle management service for Red Hat OpenShift. The cert-manager is an open source project that aims to automate the management and issuance of TLS certificates from various certificate authorities. The project added new resource types for certificate authorities and certificates in the Kubernetes API, which allowed Kubernetes to provide certificates to developers on-demand within a given cluster.

The cert-manager operator streamlines integration of cert-manager into OpenShift clusters, making it easier to:

• Integrate with external certificate authorities
• Manage certificates with built-in tools
• Self-serve certificates
• Renew certificates automatically

The cert-manager Operator is a “Day 2” operator, installed after the OpenShift cluster installation by the Operator Lifecycle Manager (OLM). 

Why is certificate management important?

Enterprise customers deploy hundreds or thousands of applications on OpenShift, all requiring configuration and access provisioning for external users. To secure this access, customers need SSL/TLS certificates that can be automatically generated and managed by tools on the cluster. cert-manager is the upstream project that provides certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing and using these certificates. 

With the cert-manager Operator for Red Hat OpenShift we bring the capabilities of upstream cert-manager for application owners deploying their workloads on OpenShift. The Operator is provided as a Day 2 Operator that is installed and managed using the Operator Lifecycle Manager in OpenShift. 

Overview of OpenShift cert-manager

One of the security best practices followed in the industry is the separation of trust between Control Plane and Data Plane components, thus making sure that a compromise of one does not impact the other. Keeping these practices in mind, we have provided the Operator to manage application workload certificate needs. RedHat OpenShift automates the certificate management of control plane components which are not configurable by application users (refer to documentation here to learn more.) 

However there is one exception to this. Certificates such as the Default Ingress’s certificate can be replaced with one that is issued by a public certification authority (CA) that is already included in the CA bundle, allowing external clients to connect securely to applications running under the .apps subdomain. The current GA version of the Operator does not support managing this certificate but we are evaluating this capability for a future release. 

The cert-manager operator for RedHat OpenShift includes: 

• A Day 2 Operator managed by OLM: Ability to install the Operator using CLI or Web Interface. 
• Documented and tested issuers: ACME issuers, specifically tested with LetsEncrypt, Self Signed issuers as well as CA issuer. 
• Configurations for ease of deployment across public and private networks.
• A published product lifecycle that aligns with upstream cert-manager releases (more details can be found here.)  

Key benefits of OpenShift cert-manager

There are many benefits to using OpenShift cert-manager Operator. Below are some highlights.

• The cert-manager operator helps in managing certificates in bulk, thereby helping users securely and efficiently expose hundreds or thousands workloads.
• The Operator can be configured post installation by updating the CertManager CR.
• The Operator uses the upstream deployment manifests.
• The operator automatically deploys a cluster-scoped CertManager object with default values.

As an automation tool, cert-manager reduces the engineering time for configuration and setup, thereby freeing the time for engineers to focus on delivering valuable application features. Cert-manager simplifies issuance, renewal, and management of certificates providing secure application traffic. Automating the certification process also reduces risk of human error, thereby improving the security posture of the application. A good example of the strengths of employing cert-manager certificate automation would be a web-based application whose user-facing interface, API, and admin API are all hosted under separate domain names. This means each service requires its own individual certificate. cert-manager monitors the timing of these certificates and renews them if and when necessary. In the case of larger web-based applications, automating certification using cert-manager could potentially save a very significant amount of manual engineer work. 

Cert-manager has helped in bringing Certificates and Issuers as first class citizens in Kubernetes with the use of CRDs, which has led to significantly more flexibility around configuration. You can configure multiple issuers which helps bring in different trust separation models in the cluster and helps users deliver multiple applications securely. 

Key features

Rich configuration options for detecting and honoring Outbound Proxy settings

For OpenShift clusters that have cluster-wide egress proxy configured, cert-manager needs to be configured with proxy settings so that it can communicate with the issuer servers on the internet. OLM on OpenShift clusters helps inject the environmental variables for HTTP_PROXY, HTTPS_PROXY, and NO_PROXY into the cert-manager operator subscription. The operator will then pass these Proxy settings to the cert-manager instance. Along with this, user may need to inject custom CA certificates for proxying the HTTPS connections which can be injected and verified as follows:

• Inject the CA bundle into the cert-manager instance:

oc label cm trusted-ca config.openshift.io/inject-trusted-cabundle=true -n cert-manager 

• Verify CA bundle is mounted as a volume:

oc get deployment cert-manager -n cert-manager -o=jsonpath={.spec.template.spec.'containers[0].volumeMounts'}

• Example output:

[{"mountPath":"/etc/pki/tls/certs/cert-manager-tls-ca-bundle.crt","name":"trusted-ca","subPath":"ca-bundle.crt"}]

Cert-manager supports both DNS and HTTP solvers which helps prove ownership of DNS or domain respectively based on challenge methods used – DNS or HTTP. 

Issuers

While there are several issuers supported upstream, for the GA version, we have tested ACME (with Let’s Encrypt), Self signed, as well as CA issuers. Additionally, we plan to support Vault and Venafi in our upcoming releases. 

ACME configurations support HTTP and DNS Solvers - ACME CA servers rely on a challenge to verify that a client owns the domain names that the certificate is being requested for. If the challenge is successful, the cert-manager Operator for Red Hat OpenShift can issue the certificate. The Operator supports both DNS and HTTP solvers which help prove ownership of the domain. 

In HTTP-01 challenge type, a computed key is at an HTTP URL endpoint for the domain. If the ACME CA server can get the key from the URL, it can validate the domain. Cert-manager will automatically configure your cluster ingress to route traffic for this URL to a web server that presents this key. In our testing, we have validated the HTTP-01 solver using the LetsEncrypt CA server. 

In DNS-01 challenge type, a computed key is at a DNS TXT record. If the ACME CA server can get the key by DNS lookup, it can validate the domain. The advantage of using DNS challenge versus HTTP challenge is that you can use wildcard DNS records which cover multiple domains, whereas HTTP will cover a single domain. We have tested the DNS-01 Solver on AWS with Route53, Google Cloud DNS, as well as Azure DNS. We provide a way to override the nameserver settings for the DNS-01 self check in spec.controllerConfig which is required when the target public-hosted zone overlaps with the cluster’s default private-hosted zone. Here is the configuration you can add:

apiVersion: operator.openshift.io/v1alpha1
kind: CertManager
metadata:
  name: cluster
  ...
spec:
  ...
  controllerConfig:                                
    overrideArgs:
      - '--dns01-recursive-nameservers=1.1.1.1:53' 
      - '--dns01-recursive-nameservers-only'       

Metrics and monitoring

Cert-Manager exposes controller metrics in the format expected by Prometheus Operator. A ServiceMonitor resource needs to be created to scrape metrics from cert-manager operand. 

oc -n openshift-monitoring get prometheus k8s --template='{{.spec.serviceMonitorNamespaceSelector}}{{"\n"}}{{.spec.serviceMonitorSelector}}{{"\n"}}'
map[matchLabels:map[openshift.io/cluster-monitoring:true]]
map[]

Cluster administrators can enable monitoring for user-defined projects by setting the enableUserWorkload: true field in the cluster monitoring ConfigMap object. The 'Service Monitor' will collect the metrics through the cert-manager service and will use the port name of the service as its endpoint. 

Querying Metrics 

As a cluster administrator or as a user with view permissions for all projects, you can access metrics for all default OpenShift- and user-defined projects in the Metrics UI by using the endpoints of the cert-manager service.

$ oc describe service cert-manager -n cert-manager

To query cert-manager controller metrics, select Observe → Metrics and filter the metrics of the cert-manager controller with {instance="<Endpoints>"} or {endpoint="tcp-prometheus-servicemonitor"}. 

Key use case

Securing application Ingress 

cert-manager is commonly used to provide TLS signed certificates to secure ingress resources. Annotations added to ingress resources facilitate creating Certificate resources automatically. This is possible due to the ingress-shim, which is a subcomponent of cert-manager that watches ingress resources across the cluster and when it finds annotations described in the Supported Annotations section, it will ensure a certificate resource with the name provided in the tls.secretName field is created in the same namespace as the Ingress object and configured as described in the Ingress spec. 

Here’s an example yaml showing an annotated Ingress resource:

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: ingress-test

  annotations:

    cert-manager.io/cluster-issuer: selfsigned-issuer

spec:

  rules:

  - host: <hostname>

    http:

      paths:

      - path: "/"

        backend:

          service:

            name: hello-openshift

            port:

              number: 8080

        pathType: "Prefix"

  tls:

  - hosts:

    - <hostname>

    secretName: ingress-secret

To check if the certificate is created, you can use the following:

$ oc get cert -o wide
NAME             READY   SECRET       ISSUER         STATUS     AGE
ingress-secret   True ingress-secret  selfsigned-issuer   Certificate is up to date 

Certificates can be used for providing both external and internal communication. You can secure the traffic between applications running on the cluster with TLS certificates provided by cert-manager. OpenShift helps you make smarter communication decisions while deploying applications, even if the application defaults are insecure. For example, you can use cert-manager to provide a TLS certificate to a custom ingress controller. If you have an application, such as an nginx application using HTTP instead of HTTPS, you can still configure its route to be exposed securely in OpenShift using the TLS certificates you added in the ingress controller.  

How to get started with OpenShift cert-manager

The fastest way to get started is by using the web console to install the cert-manager Operator for Red Hat OpenShift. cert-manager can be installed from the OpenShift OperatorHub.

Prerequisites

• You have access to the cluster with cluster-admin privileges.
• You have access to the OpenShift Container Platform web console.

You can learn more by following the step-by-step installation guide here.

OpenShift aims to make certificate management easy and seamless for your application workloads so that your developers, admins, and engineers can use their time developing applications that are core to your business needs. We take away the burden of provisioning and management of TLS serving certificates and help provide hardened security configurations out of the box. 

We look forward to bringing upstream cert-manager features to you more frequently with our rapid release cadence as described here. In our upcoming releases, we look to provide support for some of the core issuers supported upstream, external issuers, enhancements needed for route integration support, and integration with OpenShift components that are configurable for user workloads. 

You can learn more about the cert-manager Operator by checking out the documentation page and the upstream project website.