Openshift on Openstack With Macvlan CNI Plug-in

This document gives steps to enable the Macvlan CNI plug-in in OCP4.7 to run on OSP 13. We assume that your OSP 13 is up and running and all the prerequisites for installing OCP are met.

What Is Macvlan?

With VLAN, you can create multiple interfaces on top of a single one and filter packages based on a VLAN tag. With macvlan you can create multiple interfaces with different Layer 2 (Ethernet MAC) addresses on top of a single one.

Without macvlan, if you wanted to connect to a physical network from a VM or namespace, you would have needed to create TAP/VETH devices and attach one side to a bridge and attach a physical interface to the bridge on the host at the same time, as shown below:

However, with MACVLAN, you can bind a physical interface that is associated with a MACVLAN directly to namespaces, without the need for a bridge:

There are five Macvlan types.

1. Private: does not allow communication between MACVLAN instances on the same physical interface, even if the external switch supports hairpin mode.
2. VEPA: data from one MACVLAN instance to the other on the same physical interface is transmitted over the physical interface. Either the attached switch needs to support hairpin mode, or there must be a TCP/IP router forwarding the packets to allow communication.
3. Bridge: all endpoints are directly connected to each other with a simple bridge via the physical interface.
4. Passthru: allows a single VM to be connected directly to the physical interface.
5. Source: the source mode is used to filter traffic based on a list of allowed source MAC addresses to create MAC-based VLAN associations.

What Is Multus CNI?

CNI is the container network interface that provides a pluggable application programming interface to configure network interfaces in Linux containers. Multus CNI is such a plug-in and is referred to as a meta-plug-in: a CNI plug-in that can run other CNI plug-ins. It works like a wrapper that calls other CNI plug-ins for attaching multiple network interfaces to pods in OpenShift/Kubernetes.

Multus allows pods to have multiple network interface connections that can address various use cases; for example:

• Splitting the control/data plane traffic, mostly applicable to VNF/CNF applications
• Legacy/storage intensive applications
• Multitenant networks

OpenShift provides and supports the following CNI plug-ins for creating additional networks in your cluster:

1. Bridge: allows Pods on the same host to communicate with each other and the host
2. Host-device: allows Pods access to a physical Ethernet network device on the host system. This simple plugin will move the requested device from the host’s network namespace to the pod network namespace. The limitation of using this plugin is that the physical device is used solely by a single pod and will not be available to multiple pods.
3. Macvlan: allows pods on a host to communicate with other hosts and pods on those hosts by using a physical network interface. Each pod that is attached to a macvlan-based additional network is provided a unique MAC address. Macvlan needs to be used in cases where a common DHCP server is used since the DHCP server would need a unique mac address which ipvlan does not have.
4. Ipvlan: allows pods on a host to communicate with other hosts and pods on those hosts, similar to a macvlan-based additional network. Unlike a macvlan-based additional network, each pod shares the same MAC address as the parent physical network interface. Ipvlan should be used in cases where some switches restrict the maximum number of mac addresses per physical port due to port security configuration.
5. SR-IOV: allows pods to attach to a virtual function (VF) interface on SR-IOV capable hardware on the host system

For more information on the official support list refer to :

https://access.redhat.com/articles/4763741

Preparing OpenStack environment

For installing Openshift on OpenStack to enable Macvlan at a later stage, we have to prepare the environment as follows:

Login as admin and create an external network:

#source overcloudrc
(overcloud)#openstack network create --external —provider-network-type vlan \ 
--provider-segment 80 --provider-physical-network datacentre vlan80
(overcloud)#openstack subnet create --network vlan80 --dhcp  \ 
--allocation-pool start=192.168.80.200,end=192.168.80.250 \
--dns-nameserver 192.168.80.1 --gateway 192.168.80.1 \
--subnet-range 192.168.80.0/24 vlan80-subnet

Create a provider network:

(Overcloud)#openstack network create --share --provider-physical-network datacentre 
  --provider-network-type vlan —provider-segment 10 vlan10
(overcloud)#openstack subnet create --network vlan10 --dhcp  \ 
--allocation-pool start=192.168.10.201,end=192.168.10.240 \
--dns-nameserver 192.168.10.1 --gateway 192.168.10.1 \
--subnet-range 192.168.10.0/24 vlan10-subnet

Installing OCP on OpenStack With Additional Network

Now let us start installing Openshift on OpenStack. Make sure that all the prerequisites are met.

1. Verify that your network configuration does not rely on a provider network. Provider networks are not supported.
2. Have a storage service installed in RHOSP, like block storage (Cinder) or object storage (Swift). Object storage is the recommended storage technology for OpenShift Container Platform registry cluster deployment. For more information, see Optimizing storage.
3. Have metadata service enabled in RHOSP

We will follow the RedHat OpenShift installation document. We will customize the installation by adding an additional network to the worker nodes. Here is a snippet of the install-config.yaml file showing where to specify the additional network:

compute:
- architecture: amd64
  hyperthreading: Enabled
  name: worker
  platform:
    openstack:
      additionalNetworkIDs: [“85fe165b-82d8-40f9-99c0-3d7b7b6f26dc"]
 replicas: 3

We are adding an additional network at installation time because this will create a trunk port on the additional network and add it to the VM instances it creates. We will later add a sub-port to it.

Start the installation of OpenShift (IPI):

./openshift-install create cluster --dir pluto —log-level=debug
…………..
DEBUG Still waiting for the cluster to initialize: Working towards 4.7.12: 663 of 669 done (99% complete) 
DEBUG Still waiting for the cluster to initialize: Cluster operator authentication is not available 
DEBUG Cluster is initialized                       
INFO Waiting up to 10m0s for the openshift-console route to be created... 
DEBUG Route found in openshift-console namespace: console 
DEBUG OpenShift console route is admitted          
INFO Install complete!                            
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/Users/aaggarwa/Work/Lab/ocp-clusters/osp/pluto/auth/kubeconfig' 
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.pluto.home.lab 
INFO Login to the console with user: "kubeadmin", and password: "UHr9S-kXWIJ-mS82f-grEV4" 
DEBUG Time elapsed per stage:
DEBUG     Infrastructure: 5m2s                     
DEBUG Bootstrap Complete: 16m52s                   
DEBUG                API: 1m29s                    
DEBUG  Bootstrap Destroy: 47s                      
DEBUG  Cluster Operators: 27m5s                    
INFO Time elapsed: 53m38s

Once the Openshift Installation is successful, our network topology will look like this:

Now let us examine the trunks created during the installation:

As we can see, each worker has two trunk ports attached to it. One is for OpenShift SDN, and another one is from an additional network we mentioned in install-config.yaml.

(Overcloud)#openstack network create --share --provider-physical-network datacentre 
  --provider-network-type vlan —provider-segment 30 vlan30
(overcloud)#openstack subnet create --network vlan30 --dhcp  \ 
--allocation-pool start=192.168.30.101,end=192.168.30.200 \
--dns-nameserver 192.168.30.1 --gateway 192.168.30.1 \
--subnet-range 192.168.30.0/24 vlan30-subnet

Now, let us add sub-port to additional network trunks. First, we have to create another provider network. If you have existing DHCP use --no-dhcp as DHCP in network might interfere with existing DHCP:

Now we will create ports in the network created above. For easy management, keep the name of the port as per the node name:

(Overcloud)#openstack port create worker-0-ftmxg --network vlan30

(Overcloud)#openstack port create worker-0-c5gd5 --network vlan30

(Overcloud)#openstack port create worker-0-47njf --network vlan30

Now add these ports as subports in the trunk. Make sure that the trunk is the one that has the parent port as the additional network:

(Overcloud)#openstack network trunk set --subport port=worker-0-c5gd5,segmentation-type=inherit 0bca67f0-613f-42e0-bb9e-dbc57fdbfa89
(Overcloud)#openstack network trunk set --subport port=worker-0-47njf,segmentation-type=inherit 541d1a9c-0c9d-4303-8842-e295f692c848
(Overcloud)#openstack network trunk set --subport port=worker-0-ftmxg,segmentation-type=inherit da42a06e-a588-4c61-99e3-346024bbc150

Here is the diagram for a better understanding:

Enabling Macvlan CNI in OpenShift

Once the installation is done and we have added sub-ports to the trunk, we will now configure OpenShift so that Macvlan can be enabled and used by the pods.

Install NMState operator

The Kubernetes NMState Operator provides a Kubernetes API for performing state-driven network configuration across the OpenShift Container Platform cluster’s nodes with NMState. The Kubernetes NMState Operator provides users with functionality to configure various network interface types, DNS, and routing on cluster nodes. Additionally, the daemons on the cluster nodes periodically report on the state of each node’s network interfaces to the API server. To install the NMState operator, please click here.

Once complete, the Operator has deployed the NMState State Controller as a daemon set across all of the cluster nodes.

Create a VLAN interface on the nodes in the cluster by applying a NodeNetworkConfigurationPolicy manifest to the nodes. We have to create configuration for each node as the MAC address has to match the sub-port attached to it. Following is an example:

#cat nmstate-worker-0-47njf.yaml 
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkConfigurationPolicy
metadata:
  name: vlan-ens4-policy-worker-0-47njf
spec:
  nodeSelector: 
    kubernetes.io/hostname: pluto-jttmc-worker-0-47njf
  desiredState:
    interfaces:
    - name: ens4.30
      description: VLAN using ens4
      type: vlan 
      state: up
      mac-address: fa:16:3e:d9:7a:27
      vlan:
        base-iface: ens4
        id: 30

#oc apply -f nmstate-worker-0-47njf.yaml

Confirm the policy has been applied to the cluster; list the policies and their status:

oc get nncp
NAME                              STATUS
vlan-ens4-policy-worker-0-47njf   SuccessfullyConfigured
vlan-ens4-policy-worker-0-c5gd5   SuccessfullyConfigured
vlan-ens4-policy-worker-0-ftmxg   SuccessfullyConfigured

Login to worker node to see if the interface has been created:

ens4.30: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether fa:16:3e:d9:7a:27  txqueuelen 1000  (Ethernet)
        RX packets 1073  bytes 120094 (117.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 800  bytes 146198 (142.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Create a project in OpenShift:

#oc new-project networking-demos

Create a network attach definition in the new namespace by adding the following snippet in network operator:

#oc edit networks.operator.openshift.io
………
Spec:
  additionalNetworks:
  - name: macvlan-30
    namespace: networking-demos
    simpleMacvlanConfig:
      ipamConfig:
        type: dhcp
      master: ens4.30
      mode: bridge
    type: SimpleMacvlan
  clusterNetwork:
…………

Check the status:

#oc get network-attachment-definitions
NAME         AGE
macvlan-30   10s

Now we will create a pod with an additional network:

#vi test-pod1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: example-pod1
  namespace: networking-demos
  annotations:
    k8s.v1.cni.cncf.io/networks: |-
      [
        {
          "name": "macvlan-30",
          "namespace": "networking-demos",
          "default-route": ["192.168.30.1"]
        }
      ]
spec:
  containers:
  - name: example-pod
    command: ["/bin/bash", "-c", "sleep 2000000000000"]
    image: centos/tools

#oc apply -f test-pod1.yaml

If everything is configured correctly, you should see that the pod is able to get IP from DHCP IP from vlan30 subnet:

You can also check in you DHCP logs:

Login to pod to check the status: