You can deploy a tang-operator to automate the deployment of a Tang server in an OpenShift cluster that requires Network Bound Disk Encryption (NBDE) internally, leveraging the tools that OpenShift provides to achieve this automation. You can also configure the Clevis tool to use the deployed TangServer. This can help you avoid the need to enter a password to access encrypted volumes while maintaining high security standards.

NBDE

Network Bound Disk Encryption (NBDE) is a subcategory of Policy Based Decryption (PBD) that allows binding encrypted volumes to a special network server. PBD allows combining different disk unlocking methods into a set of rules, called a policy, which enable the unlocking of one volume by means of different ways.

Red Hat Enterprise Linux (RHEL) provides an automated decryption policy framework (Clevis) that allows to define a policy at encryption time that must be satisfied for the data to decrypt. Clevis contains a series of plugins, called pins, that enable different unlocking capabilities. The current available pins are:

  • tpm2 -  Clevis provides support to encrypt a key in a Trusted Platform Module 2.0 () chip. The cryptographically-strong, random key used for encryption is encrypted using the TPM2 chip, and then at decryption time is decrypted using the TPM2 to allow clevis to decrypt the secret stored in the JWE.
     
  • sss - Shamir’s Secret Sharing (SSS) is an algorithm that allows Clevis to mix pins together to create sophisticated unlocking and high-availability policies.
     
  • tang - Allows volumes to be unlocked using a network server. In particular, Clevis provides support for the Tang network binding server. Tang provides a stateless, lightweight server, where encryption/decryption of the data works via HTTP.

Tang operator

This article will focus on the Tang server, and, in particular, on how you can deploy this kind of server in K8S architecture in general, and OpenShift in particular. To provide the deployment of a Tang server, a new Kubernetes operator, named tang-operator, has been implemented. With this specific operator, you can deploy automation of Tang Servers in OpenShift infrastructure.
To this end, tang-operator defines a new Custom Resource Definition (CRD), named TangServer, which allows deployment of one to many tang servers, with one entry point for each of them. Each Custom Resource object of the type TangServer will be deployable with different parameters, such as the number of replicas per deployment. You can see the different configurable parameters in following figure:

Figure 1: TangServer form

As you can see, the only mandatory parameters to specify are the name of the TangServer and the number of replicas to launch (1 by default). For each TangServer deployed, a new Kubernetes deployment will be launched, with as many Tang Server pods as the number of replicas specified in the previous form. Note that, for TangServer deploy to work appropriately, a Persistent Volume Claim (PVC) must have been created. This will be described with more detail in the section Step-by-step guide to create an NBDE scenario in OpenShift. Otherwise, the Pod will not be able to launch, because it requires a PVC to hold the keys that will be used for encryption/decryption.

The other parameters should not be modified, as they are normally fixed to previous values, which are the default ones. They have been kept as configurable for any special cases where they could be required to be modified.

TangServer deployment

The creation of a TangServer Custom Resource object implies, basically, the execution of the next steps by the logic of the controller:

  1. Launch of a Kubernetes deployment, with the number of Pods of the Tang server container equal to the number of replicas specified in the creation form. If, for example, a TangServer named “tangserver-multi-replica” with three replicas is created, the deployment created will be as follows:
  2. Startup of a network service, where the HTTP traffic should be configured to be sent by the Clevis client. The IP and port of the TangServer service can be consulted in the Networking => Service tab of the OpenShift web console:

Figure 2: TangServer deployment

If all the requirements for the PODs to be deployed are met, then each of them will be in the “Ready” state, which supposes that they are able to process HTTP traffic corresponding to the encryption/decryption requests performed by a Clevis client in a separate physical or virtual machine that can access OpenShift’s network cluster where Tang Servers are running:

image4-Nov-30-2021-04-51-27-84-PM

Figure 3: TangServer Pods

Figure 4: TangServer Service

To get the details of the Service, you can click the “service-tangserver-multi-replica”  service. Details of the service, such as Public IP (External Load Balancer IP)  and port will be shown:

Figure 5: TangServer Service details

In the “Service routing” section, you can check Load Balancer IP and service port find out the HTTP endpoint to send Tang Server traffic:

Figure 6: TangServer Service routing details

In this particular case, to configure the deployed TangServer, HTTP URL http://34.134.164.138:8081/adv must be specified.

Note that the previous URL allows retrieving the keys available in this deployment:

$ curl http://34.134.164.138:8081/adv 2>/dev/null | jq 
{
"payload": “eyJrZXlzIjpbeyJhbGciOiJFQ01SIiwiY3J2IjoiUC01MjEiLCJrZXlfb3…”,
"protected": "eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9",
"signature": "AVCUXIBvcXt9H4Dx7X8gA9Tbcd-VZJ6jaoJLyBa7NAwBrrMLh6ljNlJpH1BjYxI…"
}

Step-by-step guide to create an NBDE scenario in OpenShift

Taking the previously described information into account, this guide describes, step by step, how to get an NBDE setup where a Virtual Machine is configured with Clevis, and, in particular, with the tang pin. You will need:

  • An OpenShift cluster, where TangServer will be deployed via tang-operator.
  • A virtual machine with Fedora or Red Hat Enterprise Linux operating system, to configure the Clevis client.
  • The “operator-sdk” tool, which is required to install the tang-operator. Installation steps are defined in the Installation section.
  • OpenShift CLI tool, a.k.a. “oc”, which is required to execute some commands from the command line to the cluster.

Once previous requirements are ready, the steps to follow will be:

  1. Assignment for “anyuid” permissions to cluster. To allow deployment of a Tang Server container as root, it is required that permissions are granted in the cluster. To do so, next command must be issued:
  2. Creation of a specific project for this scenario. In order to do this, just select, in the OpenShift Web console,  “ Home => Projects => Create Project “. Once this is created, a form similar to the one below will appear. Fill the information appropriately and click on the “Create” button:
  3. Creation of a Persistent Volume Claim. For Tang Server to be deployed appropriately, a Persistent Volume Claim must be created. To do so, create:
    1. Assign permissions
    2. Persistent Volume: Select “Storage => PersistentVolume => Create PersistentVolume”
    3. Fill the name of the PersistentVolume to Create, then click on “Create” button:
    4. Persistent Volume Claim: Select “Storage => PersistentVolumeClaim => Create PersistentVolumeClaim”
    5. Fill the PersistentVolumeClaim name and the size claimed (1Gb should be enough). Take into consideration the amount of keys to handle in the Tang Server and adjust the Size appropriately. After filling required information, click on Create button:
  4. Installation of the tang-operator. Once all previous requirements are ready, it is time to install the tang-operator. This operator is not included in the operator hub, so it must be installed manually with “operator-sdk” tool:
  5. After installation, it can be observed that the tang-operator is available in the operators section. To check it has been correctly installed, select, in the OpenShift Web Console, “Operators => Installed Operators”:
  6. When tang-operator is up and running, new TangServer deployments can be created. In this example, a pair of them will be launched, to achieve an architecture similar to next one:
  7. To achieve the previous deployment, a TangServer with  two replicas is required. Follow steps similar to the ones described in the Tang operator section, but specifying two replicas. From “Installed Operators” form, click “Tang” => “Tang Server” => “Create TangServer” button:
  8. Once the “Create TangServer” button is selected, TangServer form will launch. Fill the amount of replicas, and be sure PVC is the one that was created previously. Please, not that if it was named as “tangserver-pvc”, it is not required to make any change on PVC form entry, as it is the default name to be used:
  9. After checking the Pods are up and running appropriately for TangServer deployment, find the endpoint information corresponding to this TangServer. To do so, click on “Networking” => “Services” in OpenShift Web Console. After that, select “service-tangserver”:
  10. After the previous configuration, TangServer is ready to start handling key requests and responses via HTTP. In this step, “clevis” will be configured so that it uses the TangServer deployment for encryption and decryption operations. It is assumed that you have configured a RHEL/Fedora machine with at least one encrypted partition, which can access the previous IP address and retrieve keys from the previous URL:
  11. The next steps will install “clevis”, and configure it appropriately. First, install clevis:
  12. After “clevis” is installed, check which partition is encrypted, so that it can be provided to “clevis” when configuring the Tang pin. To do so, type “lsblk”:
  13. After identifying partitions to use, configure clevis to use the tang server for each of the partitions:
  14. You can check existing bindings for each partition with the “clevis luks list -d <partition>’ command:
  15. Finally, configure dracut to allow clevis to use networking at boot time. To do so, edit the dracut clevis file with your favourite editor, include kernel_cmdline=”rd.neednet=1” in that file and enter “dracut -v -f”:
  16. Now, restart the clevis machine, and note that it is not necessary to type the password for unlocking the encrypted volumes (it can take several seconds, normally 5 to 10, for each partition to be decrypted). You can also check that, if the deployment is deleted (by removing the operator, or just destroying the TangServer deployment), rebooting the clevis machine will result in it being stuck on the password prompt, which means that the automatic unlocking is not working anymore.
$ oc adm policy add-scc-to-group anyuid system:authenticated

Figure 7: Creation of a Project in OpenShift

Figure 8: Creation of a Persistent Volume

image10-Nov-30-2021-04-56-44-39-PM

Figure 9: Creation of a Persistent Volume Claim

$ operator-sdk run bundle quay.io/sarroutb/tang-operator-bundle:latest
INFO[0008] Successfully created registry pod: quay-io-sarroutb-tang-operator-bundle-v0-0-16
INFO[0009] Created CatalogSource: tang-operator-catalog
INFO[0009] OperatorGroup "operator-sdk-og" created
INFO[0009] Created Subscription: tang-operator-v0-0-16-sub
INFO[0011] Approved InstallPlan install-lqf9f for the Subscription: tang-operator-v0-0-16-sub
INFO[0011] Waiting for ClusterServiceVersion to reach 'Succeeded' phase
INFO[0012]   Waiting for ClusterServiceVersion "default/tang-operator.v0.0.16"
INFO[0018]   Found ClusterServiceVersion "default/tang-operator.v0.0.16" phase: Pending
INFO[0020]   Found ClusterServiceVersion "default/tang-operator.v0.0.16" phase: InstallReady
INFO[0021]   Found ClusterServiceVersion "default/tang-operator.v0.0.16" phase: Installing
INFO[0031]   Found ClusterServiceVersion "default/tang-operator.v0.0.16" phase: Succeeded
INFO[0031] OLM has successfully installed "tang-operator.v0.0.16"

Figure 10: Correct Installation of Tang operator

Note that Status: “Succeeded” appears for the Tang Operator, which means that Tang Operator has been properly installed.

Figure 11: Multi Replica Deployment

Figure 12: Tang Server Creation

Figure 13: Tang Server Form

After you have filled in the required information, click the “Create” button, and the deployment will be launched. Verify that the deployment has “2 of 2” Pods running:

Figure 14: Tang Server Deployment

For the correct scenario to be achieved, both pods should be in the “Running” status, and its container should be Ready (1/1). You can verify this by clicking “Workload => Pods” in OpenShift Web Console:

Figure 15: Tang Server Pods (Running Status)


 

Figure 16: Tang Server Service

After the previous service is selected, the information about the “Service Routing” must be annotated. For this particular case, this information will be IP=34.136.106.78, port=8081:


 

Figure 17: Tang Server Service Routing details

Now, the URL to configure Clevis with Tang ping is known: http://34.136.106.78:8081/adv
Let’s configure Clevis appropriately so that it can be unlocked from the recently created Tang Server.

[clevis@fedora ~]$ ping 34.136.106.78 -c 1;
PING 34.136.106.78 (34.136.106.78) 56(84) bytes of data.
64 bytes from 34.136.106.78: icmp_seq=1 ttl=54 time=113 ms
--- 34.136.106.78 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 112.658/112.768/112.879/0.110 ms
[clevis@fedora ~]$ curl http://34.136.106.78:8081/adv 2> /dev/null | jq
{
"payload": "eyJrZXlzIjpasjdflkñajXAfasqpiu12baskdfjñsafWEja2jñajsdf…...XdEpZOGtWRWEifV19",
"protected": "eyJhbGciOiJFUzUxMiIsImN0eSI6Imp3ay1zZXQranNvbiJ9",
"signature": "SU2vCcPV5kfx6CIKIJm……..oJBrSnO6Z7XL7ItgJrRppuX3ARkPa8zoYo36o0rGmnXKYE"
}


[clevis@fedora ~]$ sudo dnf install -y clevis clevis-dracut
[sudo] password for clevis:
Last metadata expiration check: 1:17:10 ago on Tue 05 Oct 2021 11:12:57 AM CEST.
Dependencies resolved.
… (omitted output)
Installed:
clevis-18-1.fc34.x86_64       clevis-dracut-18-1.fc34.x86_64  clevis-luks-18-1.fc34.x86_64  clevis-pin-tpm2-0.3.0-1.fc34.x86_64  clevis-systemd-18-1.fc34.x86_64  luksmeta-9-10.fc34.x86_64
tpm2-tools-5.0-2.fc34.x86_64
Complete!
[clevis@fedora ~]$



[clevis@fedora ~]$ lsblk
NAME                                          MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda                                           252:0    0   12G  0 disk
├─vda1                                        252:1    0 1023M  0 part  /boot
└─vda2                                        252:2    0   11G  0 part
└─luks-c526e27d-f87c-4e18-9f15-cbc08c9359c8 253:0    0   11G  0 crypt /
vdb                                           252:16   0   12G  0 disk
└─vdb1                                        252:17   0   12G  0 part
└─luks-2bfa665e-b9ca-474b-8ca8-245a83ea5bd1 253:1    0   12G  0 crypt /home

In this case, both “vda2” and “vdb1” will be configured to use the tang server.

[clevis@fedora ~]$ sudo clevis luks bind -d /dev/vda2 tang '{"url":"http://34.136.106.78:8081"}'
Enter existing LUKS password:
Warning: Value 512 is outside of the allowed entropy range, adjusting it.
The advertisement contains the following signing keys:
vMHbTU1Kzv-8okN-HVOBEWe7ReKj0CyJy28lSm_eAB8
Do you wish to trust these keys? [ynYN] y
[clevis@fedora ~]$ sudo clevis luks bind -d /dev/vdb1 tang '{"url":"http://34.136.106.78:8081"}'
Enter existing LUKS password:
Warning: Value 512 is outside of the allowed entropy range, adjusting it.
The advertisement contains the following signing keys:
vMHbTU1Kzv-8okN-HVOBEWe7ReKj0CyJy28lSm_eAB8
Do you wish to trust these keys? [ynYN] y


[clevis@fedora ~]$ sudo clevis luks list -d /dev/vda2
1: tang '{"url":"http://34.136.106.78:8081"}'
[clevis@fedora ~]$ sudo clevis luks list -d /dev/vdb1
2: tang '{"url":"http://34.136.106.78:8081"}'


[clevis@fedora ~]$ cat /etc/dracut.conf.d/clevis.conf
kernel_cmdline="rd.neednet=1"
[clevis@fedora ~]$ sudo dracut -v -f
dracut: Executing: /usr/bin/dracut -v -f


dracut: *** Including module: clevis ***
dracut: *** Including module: clevis-pin-sss ***
dracut: *** Including module: clevis-pin-tang ***
dracut: *** Including module: clevis-pin-tpm2 ***


dracut: *** Creating initramfs image file '/boot/initramfs-5.11.12-300.fc34.x86_64.img' done ***
 

Summary

In this blog entry, tang-operator has been described, so that an OpenShift cluster administrator can be able to deploy a TangServer deployment inside OpenShift, and configure a clevis machine to use this server as decryption server, avoiding the requirement to introduce a password for encrypted volumes.

A step by step guide has also been provided to clarify the different steps that must be accomplished for the operator installation, for TangServer deployment and for Clevis installation and configuration.

References


About the author

Sergio Arroutbi is an experienced Software Designer skilled in Linux, C++/Golang/Python programming, Shell Scripting, Software Design and Development. He has a Master's Degree focused in FLOSS (Free/Libre/Open Source Software) from Universidad Rey Juan Carlos and a Software Craftsmanship Master's Degree from Universidad Politécnica de Madrid.

Read full bio