Deploy ACM Submariner for connect overlay networks of ROSA clusters

Submariner is an open source tool that can be used with Red Hat Advanced Cluster Management for Kubernetes to provide direct networking between pods and compatible multicluster service discovery across two or more Kubernetes clusters in your environment, either on-premises or in the cloud.

This article describes how to deploy ACM Submariner for connecting ROSA clusters overlay networks.

NOTE: ACM Submariner for ROSA clusters only works with ACM 2.7 or newer!

Prerequisites

• OpenShift Cluster version 4 (ROSA or non-ROSA)
• rosa cli
• aws cli (optional)
• ACM 2.7 or newer

Manage Multiple Logins

• In order to manage several clusters, we will add a new Kubeconfig file to manage the logins and change quickly from one context to another:

Deploy ACM Cluster HUB

We will use the first OpenShift cluster to deploy ACM Hub.

• Login into the HUB OpenShift cluster and set the proper context:

• Create the namespace for ACM

• Create the OperatorGroup for ACM

• Install Operator ACM 2.7

• Check that the Operator has installed successfully

NOTE: ACM Submariner will only work from 2.7 onwards! Ensure that you have a >= 2.7 ACM version.

• Install MultiClusterHub instance in the ACM namespace

• Check that the MultiClusterHub is properly installed

NOTE: if it’s not in Running state, wait a couple of minutes and check again.

Deploy First ROSA Cluster

• Define the prerequisites for install the ROSA cluster

NOTE: it’s critical that the Machine CIDR of the ROSA clusters not overlap, for that reason we’re setting different CIDRs than the out of the box ROSA cluster install.

• Create the IAM Account Roles

• Generate a STS ROSA cluster

• Create the Operator and OIDC Roles

• Check the status of the Rosa cluster (40 mins wait until is in ready status)

• Set the admin user for the ROSA cluster

• Login into the rosa cluster and set the proper context

Generate ROSA New nodes for submariner

• Create new node/s that will be used to run Submariner gateway using the following command (check the related GitHub issue for more details)

NOTE: setting replicas=2 means that we allocate two nodes for SM GW , to support GW Active/Passive HA (check Gateway Failover section ), if GW HA is not needed you can set replicas=1.

• Check the machinepools requested, including the submariner machinepool requested

• After a couple of minutes, check the new nodes generated

Deploy Second ROSA Cluster

IMPORTANT: To enable Submariner in both ROSA clusters, the POD_CIDR and SERVICE_CIDR can’t overlap between them. To avoid IP address conflicts, the second ROSA cluster needs to modify the default IP CIDRs. Check the Submariner docs for more information.

• Define the prerequisites for install the second ROSA cluster

• Create the IAM Account Roles

• Generate the second STS ROSA cluster (with the POD_CIDR and SERVICE_CIDR modified)

• Create the Operator and OIDC Roles

• Check the status of the Rosa cluster (40 mins wait until is in ready status)

• Set the admin user for the ROSA cluster

• Login into the rosa cluster and set the proper context

Generate ROSA New nodes for submariner

• Create new node/s that will be used to run Submariner gateway using the following command

• Check the machinepools requested, including the submariner machinepool requested:

• After a couple of minutes, check the new nodes generated

Create a ManagedClusterSet

• In the Hub (where ACM is installed), create the ManagedClusterSet for the rosa-clusters:

Import ROSA Sub1

We will import the cluster using the auto-import secret and using the Klusterlet Addon Config.

• Retrieve ROSA TOKEN the ROSA API from the first ROSA cluster

• Config the Hub as the current context

• Create (in the Hub) ManagedCluster object defining the rosa-subm1 cluster

• Create (in the Hub) auto-import-secret.yaml secret defining the the token and server from first ROSA cluster

• Create and apply the klusterlet add-on configuration file for the first rosa cluster

Import ROSA sub2 (CLI)

• Retrieve ROSA TOKEN the ROSA API from the second ROSA cluster

• Config the Hub as the current context

• Create (in the Hub) ManagedCluster object defining the second ROSA cluster

• Create (in the Hub) auto-import-secret.yaml secret defining the the token and server from second ROSA cluster

• Create and apply the klusterlet add-on configuration file for the second rosa cluster

• Check the managed clusters and the managed cluster set

Now it’s time to deploy submariner in our Managed ROSA Clusters. Either deploy using the RHACM UI or with CLI (choose one).

Deploy Submariner Addon in Managed ROSA clusters from the RHACM UI

• Inside of the ClusterSets tab, go to the rosa-aro-clusters generated.

• Go to Submariner add-ons and Click in “Install Submariner Add-Ons”

• Configure the Submariner addons adding both ROSA clusters generated:

Deploy Submariner Addon in ROSA clusters

• After the ManagedClusterSet is created, the submariner-addon creates a namespace called managed-cluster-set-name-broker and deploys the Submariner broker to it.

• Create the Broker configuration on the hub cluster in the managed-cluster-set-name-broker namespace

NOTE: Set the the value of globalnetEnabled: true if you want to enable Submariner Globalnet in the ManagedClusterSet.

• Check the Submariner Broker in the rosa-clusters-broker namespace:

• We don’t need to label the ManagedCluster because it was imported the proper labels within the proper ManagedClusterSet.

• Deploy SubmarinerConfig for the first rosa cluster imported:

• Deploy SubmarinerConfig for the second rosa cluster imported:

• Deploy Submariner on the first ROSA cluster cluster:

• Deploy Submariner on the second ROSA cluster cluster:

• Check the submariner status of managedclusteraddons in order to check if submariner is deployed correctly

The Submariner Add-on installation will start, and will take up to 10 minutes to finish.

Check the Status of the Submariner Networking Add-On

A few minutes (up to 10 minutes) after we can check that the app Connection Status and the Agent Status are Healthy:

Testing Submariner Networking connectivity with an example app (Optional)

This final step (totally optional), is an extra step to check if the Submariner networking tunnels are built and connected properly.

This example app deploy one FE (guestbook) in the first ROSA cluster, and two redis with active-backup replication.

One Redis will be in the first ROSA cluster and will sync and replicate the data inserted by the FE, to the second redis (in backup/passive mode) using the submariner tunnel (connecting both ROSA clusters).

The connection will be using the ServiceExport feature (DNS Discovery) from Submariner, that allows to call the Redis Service (Active or Passive) from within the Service CIDR.

• Clone the example repo app

• Deploy the GuestBook App in ROSA Cluster 1

• Deploy the Redis Master App in ROSA Cluster 1

• Apply relaxed scc only for this PoC

• Deploy the Redis Slave App in ROSA Cluster 2

• Apply relaxed SCC only for this PoC

Testing the Synchronization of the Redis Master-Slave between clusters and interacting with our FrontEnd using Submariner tunnels

To test the sync between the data from the Redis Master<->Slave, let’s write some data into our frontend. Access to the route of the guestbook App y write some data:

• Now let’s see the logs in the Redis Slave:

The sync is automatic and almost instantaneous between Master-Slave.

• We can check the data write in the redis-slave with the redis-cli and the following command:

• Let’s do this in the redis-slave pod:

And that’s how the Redis-Master in the ROSA cluster 1 sync properly the data to the redis-slave in the ROSA Cluster 2, using Submariner tunnels, all encrypted with IPSec.