ARO Custom domain with cert-manager and LetsEncrypt

ARO guide to deploying an ARO cluster with custom domain and automating certificate management with cert-manager and letsencrypt certificates to manage the *.apps and api endpoints.

Prerequisites

• az cli (already installed in Azure Cloud Shell)
• oc cli
• jq (already installed in Azure Cloud Shell)
• OpenShift 4.10+
• domain name to use (we will create zones for this domain name during this guide)

We’ll go through this setup using the bash terminal on the Azure Cloud Shell. Be sure to always use the same terminal/session for all commands since we’ll reference environment variables set or created through the steps.

Azure Cloud Shell - Bash

See Azure Docs for alternative install options.

1. Install oc CLI

   Follow the instructions in Installing the OpenShift CLI on Linux to install oc CLI.

Prepare Azure Account for Azure OpenShift

1. Register resource providers

Get Red Hat pull secret

1. Log into cloud.redhat.com

2. Browse to https://cloud.redhat.com/openshift/install/azure/aro-provisioned

3. click Copy pull secret

4. Back in the Azure Cloud Shell, enter the following. Instead of <paste_your_secret>, paste your actual secret on that line. To do so, right-click and choose paste.

Deploy Azure OpenShift

Variables and Resource Group

Set some environment variables to use later, and create an Azure Resource Group.

1. Set the following environment variables

   Change the values to suit your environment

2. Create an Azure resource group

Networking

Create a virtual network with two empty subnets

1. Create virtual network

2. Create control plane subnet

3. Create machine subnet

4. Disable network policies on the control plane subnet

   This is required for the service to be able to connect to and manage the cluster.

5. Create the cluster

   This will take between 30 and 45 minutes.

6. Wait until the ARO cluster is fully provisioned.

Create DNS Zones & Service Principal

In order for cert-manager to work with AzureDNS, we need to create the zone and add a CAA record as well as create a Service Principal that we can use to manage records in this zone so CertManager can use DNS01 authentication for validating requests.

This zone should be a public zone since letsencrypt will need to be able to read records created here.

If you use a subdomain, please be sure to create the NS records in your primary domain to the subdomain.

For ease of management, we’re using the same resource group for domain as we have the cluster in.

1. Create Zone

   You will need to configure your nameservers to point to Azure. The output of running this zone create will show you the nameservers for this record that you will need to set up within your domain registrar.

2. Create API DNS record

3. Create Wildcard DNS record

4. Add CAA Record

   CAA is a type of DNS record that allows owners to specify which Certificate Authorities are allowed to issue certificates containing their domain names.

   You should be able to view the records in your console

   

   Note - You may have to create NS records in your root zone for a subdomain if you use a subdomain zone to point to the subdomains name servers.

5. Set environment variables to build new service principal and credentials to allow cert-manager to create records in this zone.

   AZURE_CERT_MANAGER_NEW_SP_NAME = the name of the service principal to create that will manage the DNS zone automation for cert-manager.

6. Restrict service principal - remove contributor role if it exists.

   Note: If you get the error message No matched assignments were found to delete, that’s fine, and it is safe to proceed. We’re going to grant the DNS Management Role to it next.

7. Grant DNS Zone Contributor to our Service Principal

   We’ll grant DNS Zone Contributor to our DNS Service principal.

8. Assign service principal to DNS zone

9. Get OpenShift console URL

10. Get OpenShift API URL

11. Get OpenShift credentials

    You can use these to log in to the web console (will get a cert warning that you can bypass with typing “thisisunsafe” in a chrome browser or login with oc)

Log In to Cluster

1. Log in to your cluster through oc cli

   You may need to flush your local dns resolver/cache before you can see the DNS/Hostnames. On Windows you can open up a command prompt as administrator and type “ipconfig /flushdns”

   You may get a warning that the certificate isn’t trusted. We can ignore that now since we’re in the process of adding a trusted certificate.

Set up Cert-Manager

We’ll install cert-manager from operatorhub. If you experience any issues installing here, it probably means that you didn’t provide a pull-secret when you installed your ARO cluster.

1. Create namespace

2. Switch openshift-cert-manager-operator project (namespace)

3. Create OperatorGroup

4. Create subscription for cert-manager operator

   It will take a few minutes for this operator to install and complete its set up. May be a good time to take a coffee break :)

5. Wait for cert-manager operator to finish installing.

   Our next steps can’t complete until the operator has finished installing. We recommend that you log in to your cluster console with the URL and credentials you captured after you ran the az aro create and view the installed operators to see that everything is complete and successful.

   

Configure cert-manager LetsEncrypt

We’re going to set up cert-manager to use DNS verification for letsencrypt certificates. We’ll need to generate a service principal that can update the DNS zone and create short term records needed to validate certificate requests and associate this service principal with the cluster issuer.

Configure Certificate Requestor

1. Switch openshift-cert-manager project (namespace)

2. Create azuredns-config secret for storing service principal credentials to manage zone.

3. Create Cluster Issuer

4. Describe issuer

   You should see some output that the issuer is Registered/Ready

   Once the above command is complete, you should be able to log in to the OpenShift console, navigate to Installed Operators, click on the cert-manager Operator, make sure you’re in the “openshift-cert-manager-operator” project, and click the ClusterIssuer tab. You should see a screen like this. Again, if you have an ssl error and use a chrome browser - type “thisisunsafe” to get in if you get an error its an invalid cert.

   

Create & Install API Certificate

1. Switch openshift-config project /

2. Configure API certificate

3. View certificate status

4. Apply the API Certificate

   If you’re running these steps manually and you wait until the certificate is issued, you can tell the API server to use it with this command:

   Alternatively, if you’re wanting this to run as a part of automation, you can use the OCP Job resource:

Create & Install the *.apps Wildcard Certificate

1. Switch openshift-ingress project (namespace)

2. Configure Wildcard Certificate

   This will generate our API and wildcard certificate requests. We’ll now create two jobs that will install these certificates.

3. View certificate status

4. Install Wildcard Certificate

   As with the API server certificate, an oc patch ... command can be used directly:

   And again, if it’s intended to be run as part of automation, use a Job:

Debugging

The cert-manager operator keeps track of requested but not yet issued Certificate resources using another resource called a Challenge which can be seen via the oc command:

DNS-based challenges will typically resolve (and the certificate will be issued) in less than two minutes. If they don’t, looking at the messages from the specific challenge associated with the certificate will typically provide a detailed explanation of what’s causing the issue.

This is a very helpful guide in debugging certificates as well.

Validate Certificates

It will take a few minutes for the jobs to successfully complete.

Once the certificate requests are complete, you should no longer see a browser security warning and you should have a valid SSL lock in your browser and no more warnings about SSL on cli.

You may want to open an InPrivate/Private browser tab to visit the console/api via the URLs you previously listed so you can see the new SSL cert without having to expire your cache.

Delete Cluster

Once you’re done its a good idea to delete the cluster to ensure that you don’t get a surprise bill.

1. Delete the cluster

2. Delete the Azure resource group

   Only do this if there’s nothing else in the resource group.