Deploy ACM Submariner for connect overlay networks ARO - ROSA clusters

Submariner is an open source tool that can be used with Red Hat Advanced Cluster Management for Kubernetes to provide direct networking between pods and compatible multicluster service discovery across two or more Kubernetes clusters in your environment, either on-premises or in the cloud.

This article describes how to deploy ACM Submariner for connecting overlay networks of ARO and ROSA clusters.

NOTE: Submariner for connecting ARO and ROSA clusters only works from ACM 2.7 onwards!

Prerequisites

• OpenShift Cluster version 4 (ROSA/ARO or non-ROSA/ARO)
• az cli
• rosa cli
• aws cli (optional)

Manage Multiple Logins

• In order to manage several clusters, we will add a new Kubeconfig file to manage the logins and change quickly from one context to another:

Deploy ACM Cluster HUB

We will use the first OpenShift cluster to deploy ACM Hub.

• Login into the HUB OpenShift cluster and set the proper context

• Create the namespace for ACM

• Create the OperatorGroup for ACM

• Install Operator ACM 2.7

NOTE: you can select from ACM 2.7 onwards for install ACM Submariner for ROSA/ARO.

• Check that the Operator has installed successfully

NOTE: ACM Submariner for ROSA clusters only works with ACM 2.7 or newer!

• Install MultiClusterHub instance in the ACM namespace

• Check that the MultiClusterHub is installed and running properly

NOTE: if it’s not in Running state, wait a couple of minutes and check again.

Deploy ROSA Cluster

• Define the prerequisites for install the ROSA cluster

NOTE: it’s critical that the Machine CIDR of the ROSA and ARO clusters not overlap, for that reason we’re setting different CIDRs than the out of the box ROSA / ARO cluster install.

• Create the IAM Account Roles

• Generate a STS ROSA cluster

• Create the Operator and OIDC Roles

• Check the status of the Rosa cluster (40 mins wait until is in ready status)

• Set the admin user for the ROSA cluster

• Login into the rosa cluster and set the proper context

Generate ROSA New nodes for submariner

• Create new node/s that will be used to run Submariner gateway using the following command (check https://github.com/submariner-io/submariner/issues/1896 for more details)

NOTE: setting replicas=2 means that we allocate two nodes for SM GW , to support GW Active/Passive HA (check Gateway Failover section ), if GW HA is not needed you can set replicas=1.

• Check the machinepools requested, including the submariner machinepool requested

• After a couple of minutes, check the new nodes generated

Deploy ARO Cluster

IMPORTANT: To enable Submariner in ROSA - ARO clusters, the POD_CIDR and SERVICE_CIDR can’t overlap between them. To avoid IP address conflicts, the ARO cluster needs to modify the default IP CIDRs. Check the Submariner docs for more information.

• Define the prerequisites for install the ARO cluster

• Create an Azure resource group

• Create virtual network

• Create control plane subnet

• Create machine subnet

• Disable network policies on the control plane subnet

• Create the ARO cluster

• Get ARO OpenShift API Url

• Login into the ARO cluster and set context

• Login into the ARO cluster and set context

NOTE: ARO doesn’t need to generate extra nodes to have the ACM submariner components deployed.

Create ManagedClusterSets

• Create a ManagedClusterSet for ROSA and ARO clusters

Import ROSA cluster in ACM (CLI)

We will import the cluster using the auto-import secret and using the Klusterlet Addon Config.

If you want to import your cluster using the RHACM UI, refer to the official Importing a managed cluster by using console documentation.

• Retrieve ROSA TOKEN the ROSA API from the ROSA cluster

• Config the Hub as the current context

• Create (in ACM Hub cluster) ManagedCluster object defining the

• Create (in ACM Hub cluster) auto-import-secret.yaml secret defining the the token and server from ROSA cluster:

• Create and apply the klusterlet add-on configuration file for the ROSA cluster

• Check the imported cluster in ACM

Import ARO cluster into ACM (CLI)

• Retrieve the ARO token and the ARO API url from the ARO cluster

• Config the Hub as the current context

• Create (in the Hub) ManagedCluster object defining the ARO cluster:

• Create (in the Hub) auto-import-secret.yaml secret defining the the token and server from ARO cluster:

Review the clusters imported in ACM

• Check the managed clusters in ACM

Now it’s time to deploy submariner in our Managed Clusters (ROSA and ARO). Either deploy using the RHACM UI or with CLI (choose one).

Deploy Submariner Addon in Managed ROSA and ARO clusters from the RHACM UI

• Inside of the ClusterSets tab, go to the rosa-aro-clusters generated.

• Go to Submariner add-ons and Click in “Install Submariner Add-Ons”

• Configure the Submariner addons adding both ROSA and ARO clusters generated:

The Submariner Add-on installation will start, and will take up to 10 minutes to finish.

Deploy Submariner Addon in Managed ROSA and ARO clusters with CLI

NOTE: All of this commands are executed in the ACM Hub cluster, not in the ACM Managed Clusters (ROSA / ARO created).

• After the ManagedClusterSet is created, the submariner-addon creates a namespace called managed-cluster-set-name-broker and deploys the Submariner broker to it.

• Create the Broker configuration on the hub cluster in the rosa-clusters-broker namespace:

NOTE: Set the the value of globalnetEnabled to true if you want to enable Submariner Globalnet in the ManagedClusterSet.

• Check the Submariner Broker in the rosa-clusters-broker namespace:

• Deploy the SubmarinerConfig for the ROSA cluster imported:

• Deploy the SubmarinerConfig for the ARO cluster imported:

• Deploy Submariner on the ROSA cluster:

• Deploy Submariner on the ARO cluster:

The Submariner Add-on installation will start, and will take up to 10 minutes to finish.

Check the Status of the Submariner Networking Add-On

• Few minutes (up to 10 minutes) after we can check that the app Connection Status and the Agent Status are Healthy: