Private ARO Cluster with access via JumpHost

A Quickstart guide to deploying a Private Azure Red Hat OpenShift cluster.

Authors: Paul Czarkowski , Ricardo Macedo Martins

Prerequisites

Azure CLI

Obviously you’ll need to have an Azure account to configure the CLI against.

MacOS

1. Install Azure CLI using homebrew

2. Install sshuttle using homebrew

Linux

1. Import the Microsoft Keys

2. Add the Microsoft Yum Repository

3. Install Azure CLI

Prepare Azure Account for Azure OpenShift

1. Log into the Azure CLI by running the following and then authorizing through your Web Browser

2. Make sure you have enough Quota (change the location if you’re not using East US)

   see Addendum - Adding Quota to ARO account if you have less than 36 Quota left for Total Regional vCPUs.

3. Register resource providers

Get Red Hat pull secret

1. Log into cloud.redhat.com

2. Browse to https://cloud.redhat.com/openshift/install/azure/aro-provisioned

3. click the Download pull secret button and remember where you saved it, you’ll reference it later.

Deploy Azure OpenShift

Variables and Resource Group

Set some environment variables to use later, and create an Azure Resource Group.

1. Set the following environment variables

   Change the values to suit your environment, but these defaults should work.

2. Create an Azure resource group

3. Create an Azure Service Principal

Networking

Create a virtual network with two empty subnets

1. Create virtual network

2. Create control plane subnet

3. Create machine subnet

4. Disable network policies for Private Link Service on the control plane subnet

   Optional. The ARO RP will disable this for you if you skip this step.

Egress

Public and Private clusters will have –outbound-type defined to LoadBalancer by default. It means all clusters by default have open egress to the internet through the public load balancer.

To change the default behavior and restrict the Internet Egress you have to set --outbound-type to UserDefinedRouting during cluster creation, and set up a traffic to run through a Firewall solution. This can either be a 3rd party solution or a native Azure solution such as Azure Firewall/Azure NAT Gateway.

If you want to proceed with the UserDefinedRouting the two examples below show how this can be achieved through either Azure NAT gateway or Azure Firewall + Internet Egress.

1a. NAT Gateway

This option replaces the routes for the cluster to go through the Azure NAT GW service for egress vs the LoadBalancer. This will incur extra Azure costs.

1. Create a Public IP

2. Create the NAT Gateway

3. Get the Public IP of the NAT Gateway

4. Reconfigure Subnets to use Nat GW

1b. Firewall + Internet Egress

This option replaces the routes for the cluster to go through Azure Firewall for egress traffic, rather than the LoadBalancer. This will incur extra Azure costs.

1. Make sure you have the AZ CLI firewall extensions

2. Create a firewall network, IP, and firewall

3. Configure the firewall and configure IP Config (this may take 15 minutes)

4. Create and configure a route table

5. Create firewall rules for ARO resources

   Note: ARO clusters do not need access to the internet, however your own workloads running on them may. You can skip this step if you don’t need any egress at all.

   • Create a Network Rule to allow all http/https egress traffic (not recommended)

   • Create Application Rules to allow to a restricted set of destinations

     replace the target-fqdns with your desired destinations

6. Update the subnets to use the Firewall

   Once the cluster is deployed successfully you can update the subnets to use the firewall instead of the default outbound loadbalancer rule.

Create the cluster

This will take between 30 and 45 minutes.

Jump Host

With the cluster in a private network, we can create a Jump host in order to connect to it. You can do this while the cluster is being created.

1. Create jump subnet

2. Create a jump host

3. Save the jump host public IP address

4. Use sshuttle to create a ssh vpn via the jump host (use a separate terminal session)

   replace the IP with the IP of the jump box from the previous step.

5. Get OpenShift console URL

   set these variables to match the ones you set at the start.

6. Get OpenShift credentials

7. log into OpenShift

Delete Cluster

Once you’re done its a good idea to delete the cluster to ensure that you don’t get a surprise bill.

1. Delete the cluster

2. Delete the Azure resource group

   Only do this if there’s nothing else in the resource group.

Addendum

Adding Quota to ARO account

1. Visit My Quotas in the Azure Console

2. Choose the appropriate filters:

   1. Set Provider to “Compute”

   2. Set Subscription to the subscription you are creating the cluster in

   3. Set Region to “East US” and uncheck the other region boxes

3. Search for the quota name that you want to increase. This may be “Total Regional vCPUs” if you checked that prior to creating the cluster, or it may be a specific resource quota named in a ResourceQuotaExceeded error message. Note that in the latter case, the Azure console uses a localized display name (for example Standard DSv3 Family vCPUs rather than an identifier name standardDSv3Family mentioned in the error message.

4. Next to the quota name you wish to increase, click the pencil in the Adjustable column to request adjustment

5. Enter the new desired quota in the New limit text box. By default, a cluster will need 36 additional Regional vCPUs beyond current usage, or the ResourceQuotaExceeded error message will tell you how much more of an additional resource is needed.

6. Click Submit. You may need to go through additional authentication.

7. Azure will review your request to adjust your quota. This may take several minutes.