Cloud Experts Documentation

Dynamic Certificates for ROSA Custom Domain

This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.

There may be situations when you prefer not to use wild-card certificates. This ROSA guide talks about certificate management with cert-manager and letsencrypt, to dynamically issue certificates to routes created on a custom domain that’s hosted on AWS Route53.


  • A Red Hat OpenShift on AWS (ROSA) cluster
  • The oc CLI #logged in.
  • The aws CLI #logged in.
  • The rosa CLI #logged in.
  • jq
  • gettext
  • A Public Route53 Hosted Zone, and the related Domain to use.

Set up environment

  1. Export few environment variables

    export CLUSTER_NAME="sts-pvtlnk-cluster"
    export OIDC_PROVIDER=$(oc get cluster -o json \
    | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
    export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    export SCRATCH_DIR=/tmp/scratch
    export AWS_PAGER=""
    export HOSTED_ZONE_REGION=us-east-2
    export   #Custom Hosted Zone Domain for apps
    mkdir -p $SCRATCH_DIR
  2. Install jq & gettext

    Installing or ensuring the gettext & jq package is installed, will allow us to use envsubst to simplify some of our configuration so we can use output of CLI commands as input into YAMLs to reduce the complexity of manual editing.

    brew install gettext jq
    # or, for Linux / Windows WSL
    #sudo dnf install gettext jq

Prepare AWS Account

In order to make changes to the AWS Route53 Hosted Zone to add/remove DNS TXT challenge records by the cert-manager pod, we first need to create an IAM role with specific policy permissions & a trust relationship to allow access to the pod.

My Custom Domain Hosted Zone is in the same accout as the ROSA cluster. If these are in different accounts, few additional steps for Cross Account Accessexternal link (opens in new tab) will be required.

  1. Prepare an IAM Policy file

    cat <<EOF > $SCRATCH_DIR/cert-manager-r53-policy.json
      "Version": "2012-10-17",
      "Statement": [
          "Effect": "Allow",
          "Action": "route53:GetChange",
          "Resource": "arn:aws:route53:::change/*"
          "Effect": "Allow",
          "Action": [
          "Resource": "arn:aws:route53:::hostedzone/*"
          "Effect": "Allow",
          "Action": "route53:ListHostedZonesByName",
          "Resource": "*"
  2. Create the IAM Policy using the above created file.

    This creates a named policy for the cluster, you could use a generic policy for multiple clusters to keep things simpler.

    POLICY=$(aws iam create-policy --policy-name "${CLUSTER_NAME}-cert-manager-r53-policy" \
       --policy-document file://$SCRATCH_DIR/cert-manager-r53-policy.json \
       --query 'Policy.Arn' --output text) || \
    echo $POLICY
  3. Create a Trust Policy

    cat <<EOF > $SCRATCH_DIR/TrustPolicy.json
      "Version": "2012-10-17",
      "Statement": [
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "${OIDC_PROVIDER}:sub": [
  4. Create an IAM Role for the cert-manager Operator, with the above trust policy.

    ROLE=$(aws iam create-role \
      --role-name "${CLUSTER_NAME}-cert-manager-operator" \
      --assume-role-policy-document file://$SCRATCH_DIR/TrustPolicy.json \
      --query "Role.Arn" --output text)
    echo $ROLE
  5. Attach the permissions policy to the Role

    aws iam attach-role-policy \
       --role-name "${CLUSTER_NAME}-cert-manager-operator" \
       --policy-arn $POLICY

Set up cert-manager

  1. Create a project (namespace) in the ROSA cluster.

    oc new-project cert-manager --display-name="Certificate Manager" --description="Project  contains Certificates and Custom Domain related components."
  2. Install the cert-manager Operator

    cat <<EOF | oc create -f -
    kind: OperatorGroup
      generateName: cert-manager-
      namespace: cert-manager
    kind: Subscription
      name: cert-manager
      namespace: cert-manager
      channel: stable
      installPlanApproval: Automatic
      name: cert-manager
      source: community-operators
      sourceNamespace: openshift-marketplace

    It will take a few minutes for this operator to install and complete its set up.

  3. Wait for cert-manager operator to finish installing.

    Our next steps depends on the successful installation of the operator. I recommend that you login to your cluster console to confirm the succeess status of cert-manager operator, in the Installed Operators section.

    View cert-manager Operator
  4. Annotate the ServiceAccount.

    This is to enable the AWS SDK client code running within the cert-manager pod to interact with AWS STS service for temporary tokens, by assuming the IAM Role that was created in an earlier step. This is referred to as IRSAexternal link (opens in new tab) .

    oc annotate serviceaccount cert-manager -n cert-manager$ROLE

    Normally, after ServiceAccount annotations, a restart of the pod is required. However, the next step will automatically cause a restart of the pod.

  5. Update the CA truststore of the cert-manager pod.

    This step is usually not required. However, it was noticed that the cert-manager pod had difficulties in trusting the STSexternal link (opens in new tab) & LetsEncryptexternal link (opens in new tab) endpoints. So the below commands essentially downloads the CA chain of these endpoints, puts them into a ConfigMap, which then gets attached to the pod as a Volume. Along with this step, I’ll also be setting the NameServers that the cert-manager will use for DNS01 self-checkexternal link (opens in new tab) The Volume attachment to the pod and the setting of NameServers will be done together by patching the cert-manager CSV resource to persist these changes to the cert-manager deployment. This will cause the rollout of a new deployment and restart of the cert-manager pod.

    openssl s_client -showcerts -verify 5 -connect < /dev/null 2> /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; print}'  > $SCRATCH_DIR/extra-ca.pem
    openssl s_client -showcerts -verify 5 -connect < /dev/null 2> /dev/null | awk '/BEGIN/,/END/{ if(/BEGIN/){a++}; print}'  >> $SCRATCH_DIR/extra-ca.pem
    oc create configmap extra-ca --from-file=$SCRATCH_DIR/extra-ca.pem -n cert-manager
    CERT_MANAGER_CSV_NAME=$(oc get csv | grep 'cert-manager' | awk '{print $1}')
    CLUSTER_DNS_SERVICE_IP=$(oc get svc -n openshift-dns | grep 'dns-default' | awk '{print $3}')
    oc patch csv $CERT_MANAGER_CSV_NAME --type='json' -p '[{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/args/-", "value":'--dns01-recursive-nameservers-only'}, {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/args/-", "value":'--dns01-recursive-nameservers=$CLUSTER_DNS_SERVICE_IP:53'}]'
    oc patch csv $CERT_MANAGER_CSV_NAME --type='json' -p '[{"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/volumes", "value": [{"name": "extra-ca"}]}, {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/volumes/0/configMap", "value": {"name": "extra-ca", "defaultMode": 420}}, {"op": "add", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/volumeMounts", "value": [{"name": "extra-ca", "mountPath": "/etc/ssl/certs/extra-ca-bundle.pem", "readOnly": true, "subPath": "extra-ca-bundle.pem"}]}]'

    During an Operator upgrade, the above changes might be lost. There seems to be improvement plans to facilitate these changes directly through the Operator config, but until then, it’d be a good idea to maintain some automation around this to persist these changes if it ever gets overridden to defaults.

Create the Issuer and the Certficiate

Configure Certificate Requestor

  1. Create Cluster Issuer to use Let’s Encrypt

    envsubst  <<EOF | oc apply -f -
    kind: ClusterIssuer
      name: letsencryptissuer
        email: $LETSENCRYPT_EMAIL
        # This key doesn't exist, cert-manager creates it
          name: prod-letsencrypt-issuer-account-key
        - dns01:
             hostedZoneID: $HOSTED_ZONE_ID
             region: $HOSTED_ZONE_REGION
               name: ''
  2. Describe the ClusterIssuer to confirm it’s ready.

    oc describe clusterissuer letsencryptissuer

    You should see an output that mentions that the issuer is Registered/Ready. Note this can take a few minutes.

     Last Transition Time:  2022-11-17T10:29:37Z
     Message:               The ACME account was registered with the ACME server
     Observed Generation:   1
     Reason:                ACMEAccountRegistered
     Status:                True
     Type:                  Ready
    Events:                 <none>

    Once the above command is complete, the status of the ClusterIssues in the cluster console will look similar to the below screenshot.

    cluster issuer

Create the Certificate, which will later be used by the Custom Domain.

*I’ve used a SAN certificate here to show how SAN certificates could be created, which will be useful for clusters intended to run only a fixed set of applications. However, this is optional; a single subject/domain certificate works too *

  1. Configure the Certificate

    envsubst  <<EOF | oc apply -f -
    kind: Certificate
      name: customdomain-cert
      namespace: cert-manager
      secretName: custom-domain-certificate-tls
         name: letsencryptissuer
         kind: ClusterIssuer
      commonName: "x.apps.$DOMAIN"
      - "x.apps.$DOMAIN"
      - "y.apps.$DOMAIN"
      - "z.apps.$DOMAIN"
  2. View the Certificate status

    It’ll take up to 5 minutes for the Certificate to show as Ready status. If it takes too long, the oc describe command will mention issues if any.

    oc get certificate customdomain-cert -n cert-manager
    oc describe certificate customdomain-cert -n cert-manager

Create the Custom Domain, which will be used to access your applications.

  1. Create the Custom Domain

    envsubst  <<EOF | oc apply -f -
    kind: CustomDomain
      name: appdomain
      domain: x.apps.$DOMAIN
        name: custom-domain-certificate-tls
        namespace: cert-manager
      scope: Internal
  2. View the status of the Custom Domain

    oc get customdomain appdomain -n cert-manager

    It will take 2-3 minutes for the custom domain to change from NotReady to Ready status. When ready, an endpoint also will be visible in the output of the above command, as shown below

Custom Domains
  1. Next, we need to add a DNS record in my Custom Domain Route53 Hosted Zone to CNAME the the wildcard applications domain to the above obtained endpoint, as shown below.
CNAME the custom domain to the endpoint
CUSTOM_DOMAIN_ENDPOINT=$(oc get customdomain appdomain -n cert-manager | grep 'appdomain' | awk '{print $2}')
cat <<EOF > $SCRATCH_DIR/add_cname_record.json
    "Comment":"Add apps CNAME to Custom Domain Endpoint",
 	       "Name": "*.apps.$DOMAIN",
aws route53 change-resource-record-sets --hosted-zone-id $HOSTED_ZONE_ID --change-batch file://$SCRATCH_DIR/add_cname_record.json

The wild card CNAME’ing avoids the need to create a new record for every new application. The certificate that each of these applications use will NOT be a wildcard certificate

At this stage, you will be able to expose cluster applications on any of the listed DNS names that were specified in the previously created Certificate. But what if you have many more applications that will need to be securely exposed too. Well, one approach is to keep updating the Certificate resource with additional SAN names as more applications prepare to get onboarded, and this Certificate update which will trigger an update to the Custom Domain to honor the newly added SAN names. Another approach is to dynamically issue a Certificate to every new Route; Read on to find the details about this latter approach.

Dynamic Certificates for Custom Domain Routes.

  1. Create OpenShift resources required for issuing Dynamic Certificates to Routes.

This step will create a new deployment (and hence a pod) that’ll watch out for specifically annotated routes in the cluster, and if the issuer-kind and issuer-name annotations are found in a new route, it will request the Issuer (ClusterIssuer in my case) for a new Certificate that’s unique to this route and which will honor the hostname that was specified while creating the route.

oc apply -f -n cert-manager

Additonal OpenShift resources such as a ClusterRole (with permissions to watch and update the routes across the cluster), a ServiceAccount (with these permissions, that will be used to run this newly created pod) and a ClusterRoleBinding to bind these two resources, will be created in this step too. If the cluster does not have access to github, you may as well save the raw contents locally, and run oc apply -f localfilename.yaml -n cert-manager

  1. View the status of the new pod.

Check if all the pods are running successfully and that the events do not mention any errors.

oc get po -n cert-manager

Test an application.

  1. Create a test applciation in a new namespace.

    oc new-project testapp
    oc new-app -n testapp
  2. Expose the test application Service.

    Let’s create a Route to expose the application from outside the cluster, and annotate the Route to give it a new Certificate.

    oc create route edge --service=hello-openshift testroute --hostname hello.apps.$DOMAIN -n testapp
    oc annotate route testroute -n testapp

    It will take a 2-3 minutes for the Certificate to be created. The renewal of the certitificate will automatically be managed by the cert-manager compoenents as it approaches expiry.

  3. Access the application Route.

    Do a curl test (or any http client of your preference) to confirm there are no certificate related errors.

    Output should print “Hello OpenShfit!”, and you should also notice a line that says “subjectAltName: host hello.apps.$DOMIAN” matched cert’s “hello.apps.$DOMIAN”

    curl -vv https://hello.apps.$DOMAIN


Please note that while creating certificates, the validation process usually take upto 2-3 minutes to complete.

If you face issues during certificate create step, run ‘oc describe’ against each of - ‘certificate,certificaterequest,order & challenge’ resources to view the events/reasons that’ll mostly help with identifying the cause of the issue.

oc get certificate,certificaterequest,order,challenge

This is a very helpful guide in debugging certificatesexternal link (opens in new tab) as well.

You may also use the cmctlexternal link (opens in new tab) CLI tool for various certificate management activities such as to check the status of certificates, testing renewals etc.

Interested in contributing to these docs?

Collaboration drives progress. Help improve our documentation The Red Hat Way.

Red Hat logo LinkedIn YouTube Facebook Twitter



Try, buy & sell


About Red Hat

We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Subscribe to our newsletter, Red Hat Shares

Sign up now
© 2023 Red Hat, Inc.