Introduction

We’re pleased to introduce Kubernetes services on secondary networks as created by Multus CNI – available as a developer preview on OpenShift 4.10, which we’re here to help you get your hands on to try today. We’ll run you through a quick installation, and a demo where you can see it in action.

Multus CNI is used in OpenShift to add additional network interfaces in pods in order to attach to “secondary networks”. Secondary networks are networks you need connectivity on in addition to the typical pod-to-pod network that you have on eth0 in a pod. This functionality is used for performance-sensitive and advanced networking use cases. One such popular case is where you need network isolation, such as isolating control and data plane traffic.

There are several articles about Multus CNI in the blog. Please take a look into it if you want to know about Multus CNI

These networks are kind of “side cars” for the pod. They exist in addition to eth0, and they aren’t necessarily bound to the same constraints as your primary network. This is a trade off, while you have lots of freedom to use these interfaces as you see fit, they’re also not as tightly integrated with some Kubernetes-specific functionality that you might expect to have. Such as Kubernetes services, which allow you to specify that a set of pods belong to a “network service”. The idea being that front-ends shouldn’t have to track where the backend exists, and allow a form of service discovery.

A Kubernetes Service is an abstraction to expose an application running on a set of Pods – the network traffic to that set of pods then load balanced across those pods. Conveniently, you can also refer to that service by DNS name (as opposed to an IP address), which aids when developing applications which run on Kubernetes. Previously, interfaces created by Multus CNI didn’t natively have services in Kubernetes, due to their nature as “side car networks”.

Tomofumi Hayashi has been working closely with the Network Plumbing Working Group to help bring services on secondary networks to light. This upstream work for this happens in a project we call multus-service. Tomo’s work includes progress on NetworkPolicy and Kubernetes Services for secondary networks, helping to meet the group’s goal of having functional parity in Kubernetes for secondary networks. We’re pleased to introduce this functionality to OpenShift today.

What does multus-service do, exactly?

In short: It creates a Kubernetes service for you! And we’ll show you just exactly how to do it here in a moment. First let’s open the hood and see what’s powering multus-service.

Multus-service is actually an alternative controller which handles Kubernetes endpointslices. Using a label to tell Kubernetes to not process these endpointslices with the default controller, we can have the service handled by an external controller – in this case, Multus-service.

Multus-service itself is modular, the initial module for Multus-proxy uses iptables in order to assert connectivity to a set of endpointslices. This is similar to the implementation of the kube-proxy, however from an advanced networking perspective, comes with some limitations that we’re looking to expand upon in the future.

What limitations come with multus-service today?

Multus service comes with some limitations as a developer preview. The first of which as noted above is that it requires iptables to be used. This means that while we have planned to extend the functionality of multus-service, today, it must use kernel networking. In order to work with user-space networking (such as DPDK, or XDP) Multus-service will need to implement modules which can handle this specific type of traffic. It also requires the use of IPv4 at the time of writing this article.

Also, currently multus-service doesn’t yet support headless services – you must use a “headed service.” We’re currently working with the upstream community to find consensus on how to implement headless services, and hope to make progress soon.

How to deploy

Currently multus-service is in community repository, in Kubernetes Network Plumbing Working Group (NPWG). You can deploy multus-service components into your OpenShift cluster  using the following command:

$ oc create -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-service/main/deploy-nft.yml

Let’s create a demo

In this demo, we will create one service, called ‘multus-nginx-macvlan’, and two nginx pods which serve a simple “Hello, world!” type HTML document. These pods attach to a macvlan interface that’s created by Multus CNI. We also create an additional pod to act as the “user” – a client that will consume our service. Then try to get http contents by ‘curl’ command from ‘fedora-net1’ to ‘multus-nginx-macvlan’ and see how its actual traffic goes.

In this article setting up connectivity for secondary networks is out of scope. In order to reduce the complexity of the demonstration, we’ve opted to utilize a NodeSelector to run the sample workloads all on one worker node in your cluster. This is intended to simplify the possible networking scenarios for demonstrative purposes. This is not a limitation of the technology, but a limitation of scope of the demonstration. You can omit doing this if you already understand network connectivity for secondary networks in your cluster and you can tailor it to your environment, just remove the NodeSelector from the deployment spec.

Creating a service for a secondary network

To create a service on a secondary network, you need to create a NetworkAttachmentDefinition (net-attach-def) first. A net-attach-def lets you express which networks you intend to connect a pod to. Let’s create a macvlan network attachment definition.

NOTE: The value for the field “master” must match the name of a network interface available on any of the host machines where this pod may be scheduled. Configuring connectivity on secondary networks is outside of the scope of this article. Secondary networks are quite flexible and there’s a lot of options for how to configure them, if you need more help check out the OpenShift 4.10 documentation for multiple networks.

You may wish to create a new project using “oc new-project multus-service-demo” before beginning these steps. It’s also important that you stay using the same OpenShift project during the steps, as the objects you create expect to be in the same namespace.

Create a file “net-attach.yml” and insert the following contents:

---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: macvlan1
spec:
config: '{
          "cniVersion": "0.3.1",
          "type": "macvlan",
          "master": "ens4",
          "mode": "bridge",
          "ipam": {
              "type": "whereabouts",
              "range":     "10.2.128.0/24"
          }
      }'

Next execute “oc create -f net-attach.yml” and you can double check the contents of it by running “oc describe net-attach-def macvlan1”

Then you create Kubernetes service as you typically would – but! There are two exceptions, adding an annotation, “k8s.v1.cni.cncf.io/service-network” and a label, “service.kubernetes.io/service-proxy-name”. This annotation specifies which network attachment definition is used for the service. In this case, ‘macvlan1’ is specified just above. The label specifies which proxy component handles the traffic. In the case of multus-service, ‘multus-proxy’ is specified.

Create a file “service.yml” with these contents:

---
kind: Service
apiVersion: v1
metadata:
name: multus-nginx-macvlan
labels:
  service.kubernetes.io/service-proxy-name: multus-proxy
annotations:
  k8s.v1.cni.cncf.io/service-network: macvlan1
spec:
selector:
  app: multus-nginx-macvlan
ports:
- protocol: TCP
  port: 80

The whole yaml file is in https://github.com/redhat-nfvpe/multus-service-demo as well as container images including network tools package.

Label your node with:

oc label nodes <node-name-here> multusservicedemo=true

Next we’ll create a deployment that accesses that network, create a file “deployment.yml” with these contents:

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: multus-nginx-macvlan
spec:
selector:
  matchLabels:
    app: multus-nginx-macvlan
replicas: 2
template:
  metadata:
    labels:
      app: multus-nginx-macvlan
    annotations:
      k8s.v1.cni.cncf.io/networks: '[
          { "name": "macvlan1" } ]'
  spec:
    containers:
    - name: multus-nginx
      image: ghcr.io/redhat-nfvpe/multus-service-demo:fedora-nginx
      ports:
      - containerPort: 80
      imagePullPolicy: IfNotPresent
      securityContext:
        privileged: true
      volumeMounts:
      - mountPath: /usr/share/nginx/html
        name: multus-nginx-macvlan-conf
    volumes:
      - name: multus-nginx-macvlan-conf
        configMap:
          name: multus-nginx-macvlan-conf
    nodeSelector:
      multusservicedemo: "true"
---
apiVersion: v1
kind: ConfigMap
metadata:
name: multus-nginx-macvlan-conf
data:
index.html: |
  <!DOCTYPE html>
  <html>
  <body>
  This is multus-nginx-macvlan!
  </body>
  </html>

Then create the deployment with “oc create -f deployment.yaml”

So, let’s consume that service! We’ll create a pod to do so, create a file “pod.yml” with the contents:

---
apiVersion: v1
kind: Pod
metadata:
name: fedora-net1
annotations:
  k8s.v1.cni.cncf.io/networks: '[ { "name": "macvlan1" } ]'
spec:
containers:
- name: fedora-net1
  image: ghcr.io/redhat-nfvpe/multus-service-demo:fedora-tools
  imagePullPolicy: Always
  command:
  - /sbin/init
  securityContext:
    privileged: true
nodeSelector:
  multusservicedemo: "true"

And again, create with “oc create -f pod.yml”

See it in action!

Before we actually make a call to create the service, let’s start a tcpdump (a packet capture tool) on our nginx pods so we can see the traffic interactively. Open them up in separate terminal windows so you can see them all happening at once.

First, start it on the first instance. This command gets the first instance and conveniently starts tcpdump on it, you can view the pods with “oc get pods” and replace the contents of “$(...)” manually if you so choose.

oc exec -it $(oc get pods | grep "multus-nginx" | awk '{print $1}' | head -n1) -- tcpdump -i net1 'tcp port http'

And in another window, do this for the second pod…

oc exec -it $(oc get pods | grep "multus-nginx" | awk '{print $1}' | tail -n1) -- tcpdump -i net1 'tcp port http'

You can now exec into the pod, and you can curl for the HTTP document

oc exec -it fedora-net1 -- bash
# curl multus-nginx-macvlan.default.svc.cluster.local

And then curl the service, repeat this process a number of times and check the output on the other terminal windows to see that the traffic is being balanced across each instance of nginx!

Or just watch the movie!

In this demo, we have the same parts from the instruction here, however we use OpenShift deployed in a bare metal environment.

Join us Upstream!

If you’ve been looking to add services on your secondary networks – we’d strongly encourage you to give multus-service a spin, and join us in the upstream community to provide early feedback so that we, as a community, can provide functionality that’s suited to a variety of cases and reliably gets traffic to your services.

As multus-service is a Developer Preview, we’d suggest that you file any issues you encounter upstream – the multus-service GitHub issues page is just the right spot to file the issues.

If you'd like to dive even deeper – we’d really like to see you at the Kubernetes Network Plumbing Working Group. Our group has a call every other Thursday, and we welcome any questions or contributions regarding multus-service as well as any of the projects from the group. You can find connection information on the Kubernetes Network Plumbing Working Group community page. Hope to see you soon!


About the authors

Doug Smith works on the network team for Red Hat OpenShift. Smith came to OpenShift engineering after focusing on network function virtualization and container technologies in Red Hat's Office of the CTO.

Read full bio