Secure boot is a compromise between the UEFI bootloader and the operating system it hands off to. Explaining the complete details of secure boot in RHEL/RHCOS is out of scope for this document. The following article does an excellent job of explaining it properly.

When using secure boot in a single node deployment of OpenShift, two methods can be used: manually setting secure boot in the system BIOS or using the bare-metal host CR to set the secure boot setting during provisioning.

More information regarding signing drivers and verifying them on RHEL8 is available in the official documentation online.

Verifying Secure Boot Status

A redfish curl command can be used to determine the current state of secure boot against a BMC endpoint. There are two examples below, one for HP and one for Dell:

# HP
curl -sS -k -u Administrator:password -H "Content-Type: application/json" -H "OData-Version: 4.0" https://[fd00:4888:x:y::z]/redfish/v1/Systems/1/SecureBoot | jq '.'
{
"@odata.context": "/redfish/v1/$metadata#SecureBoot.SecureBoot",
"@odata.etag": "W/\"4A4CB737\"",
"@odata.id": "/redfish/v1/Systems/1/SecureBoot",
"@odata.type": "#SecureBoot.v1_0_0.SecureBoot",
"Id": "SecureBoot",
"Actions": {
  "#SecureBoot.ResetKeys": {
    "target": "/redfish/v1/Systems/1/SecureBoot/Actions/SecureBoot.ResetKeys"
  }
},
"Name": "SecureBoot",
"SecureBootCurrentBoot": "Disabled",
"SecureBootEnable": false,
"SecureBootMode": "UserMode"
}
# Dell
curl -sS -k -u administrator:password -H "Content-Type: application/json" -H "OData-Version: 4.0" https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/SecureBoot | jq '.'
{
"@odata.context": "/redfish/v1/$metadata#SecureBoot.SecureBoot",
"@odata.id": "/redfish/v1/Systems/System.Embedded.1/SecureBoot",
"@odata.type": "#SecureBoot.v1_0_6.SecureBoot",
"Actions": {
  "#SecureBoot.ResetKeys": {
    "ResetKeysType@Redfish.AllowableValues": [
      "ResetAllKeysToDefault",
      "DeleteAllKeys",
      "DeletePK",
      "ResetPK",
      "ResetKEK",
      "ResetDB",
      "ResetDBX"
    ],
    "target": "/redfish/v1/Systems/System.Embedded.1/SecureBoot/Actions/SecureBoot.ResetKeys"
  },
  "Oem": {}
},
"Description": "UEFI Secure Boot",
"Id": "SecureBoot",
"Name": "UEFI Secure Boot",
"Oem": {
  "Dell": {
    "@odata.context": "/redfish/v1/$metadata#DellSecureBoot.DellSecureBoot",
    "@odata.type": "#DellSecureBoot.v1_0_0.DellSecureBoot",
    "Certificates": {
      "@odata.id": "/redfish/v1/Systems/System.Embedded.1/SecureBoot/Oem/Dell/Certificates"
    }
  }
},
"SecureBootCurrentBoot": "Enabled",
"SecureBootEnable": true,
"SecureBootMode": "DeployedMode"
}

Enabling Secure Boot

There are two methods for enabling secure boot. The first would be to interact with the hardware in question. Here is a redfish command to enable it on Dell hardware:

curl -i -k -X PATCH -u administrator:password -H "Content-Type: application/json" -d '{"SecureBootEnable":true}' https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/SecureBoot

{"@Message.ExtendedInfo":[{"Message":"Successfully Completed Request","MessageArgs":[],"MessageArgs@odata.count":0,"MessageId":"Base.1.7.Success","RelatedProperties":[],"RelatedProperties@odata.count":0,"Resolution":"None","Severity":"OK"},{"Message":"The operation is successfully completed.","MessageArgs":[],"MessageArgs@odata.count":0,"MessageId":"IDRAC.2.4.SYS430","RelatedProperties":[],"RelatedProperties@odata.count":0,"Resolution":"No response action is required.However, to make them immediately effective, restart the host server.","Severity":"Informational"}]}

curl -i -k -X POST -u administrator:password -H "Content-Type: application/json" -d '{"ResetType":"GracefulRestart"}' https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset

The following is the equivalent command for HPE hardware:

curl -i -k -X PATCH -u administrator:password -H "Content-Type: application/json" -d '{"SecureBootEnable":true}' https://10.19.x.y/redfish/v1/Systems/1/SecureBoot/
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-length: 261
Content-type: application/json; charset=utf-8
Date: Thu, 18 Nov 2021 14:08:35 GMT
ETag: W/"78E0DBAE"
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X_HP-CHRP-Service-Version: 1.0.3
{"Messages":[{"MessageID":"iLO.0.10.SystemResetRequired"}],"Type":"ExtendedError.1.0.0","error":{"@Message.ExtendedInfo":[{"MessageID":"iLO.0.10.SystemResetRequired"}],"code":"iLO.0.10.ExtendedInfo","message":"See @Message.ExtendedInfo for more information."}}
# Validating that the change has been made:
curl -i -k -X GET -u administrator:password https://10.19.x.y/redfish/v1/Systems/1/SecureBoot/
HTTP/1.1 200 OK
Allow: GET, HEAD, PATCH
Cache-Control: no-cache
Content-length: 407
Content-type: application/json; charset=utf-8
Date: Thu, 18 Nov 2021 14:10:19 GMT
ETag: W/"AEA4F742"
Link: </redfish/v1/SchemaStore/en/HpSecureBoot.json/>; rel=describedby
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X_HP-CHRP-Service-Version: 1.0.3
{"@odata.context":"/redfish/v1/$metadata#Systems/Members/1/SecureBoot$entity","@odata.id":"/redfish/v1/Systems/1/SecureBoot/","@odata.type":"#HpSecureBoot.1.0.0.HpSecureBoot","Id":"SecureBoot","Name":"SecureBoot","ResetAllKeys":false,"ResetToDefaultKeys":false,"SecureBootCurrentState":false,"SecureBootEnable":true,"Type":"HpSecureBoot.1.0.0","links":{"self":{"href":"/redfish/v1/Systems/1/SecureBoot/"}}}
curl -i -k -X POST -u administrator:password -H "Content-Type: application/json" -d '{"ResetType":"ForceRestart"}' https://10.19.x.y/redfish/v1/Systems/1/Actions/ComputerSystem.Reset/
{"Messages":[{"MessageID":"Base.0.10.Success"}],"Type":"ExtendedError.1.0.0","error":{"@Message.ExtendedInfo

The custom resource for the bare-metal host can be used to configure secure boot as well. Here is an example of the CR to use for doing an assisted service SNO deployment. For more information regarding assisted service ACM ZTP install, refer to the following unofficial repository.

spec:
online: true
automatedCleaningMode: disabled
bootMACAddress: e4:43:4b:x:y:z
bootMode: UEFISecureBoot
bmc:
  address: idrac-virtualmedia+https://10.19.x.y/redfish/v1/Systems/System.Embedded.1
  credentialsName: sno-worker-2-worker
  disableCertificateVerification: true

After the single node is installed, the secure boot state can be verified a couple of different ways. The mokutil command can be used to verify and to add custom certificates for secure boot:

mokutil --sb-state
SecureBoot enabled
mokutil --list-enrolled
[key 1]
SHA1 Fingerprint: e6:f5:06:46:20:69:aa:17:d2:e8:61:05:03:63:5c:20:f3:a9:95:c3
Certificate:
  Data:
      Version: 3 (0x2)
      Serial Number:
          83:73:0d:2b:72:80:d1:5a
      Signature Algorithm: sha256WithRSAEncryption
      Issuer: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
      Validity
          Not Before: Jun  9 08:15:36 2020 GMT
          Not After : Jan 18 08:15:36 2038 GMT
      Subject: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
      Subject Public Key Info:
          Public Key Algorithm: rsaEncryption
              RSA Public-Key: (2048 bit)
..omitted..
dmesg | egrep 'efi|secure'
[    0.000000] efi: EFI v2.70 by Dell Inc.
[    0.000000] efi:  ACPI=0x6fffe000  ACPI 2.0=0x6fffe014  SMBIOS=0x68d36000  SMBIOS 3.0=0x68d34000  MEMATTR=0x6525f020  MOKvar=0x683a9000
[    0.000000] secureboot: Secure boot enabled
..omitted..
efibootmgr -v
BootCurrent: 0008
BootOrder: 0001,0006,0008
Boot0000* Hard drive C:        VenHw(d6c0639f-c705-4eb9-aa4f-5802d8823de6)............................f.........................................................A.....................P.E.R.C. .H.7.3.0.P. .M.X. .(.b.u.s. .1.8. .d.e.v. .0.0.)...
Boot0001  NIC in Mezzanine 1A Port 1 Partition 1        VenHw(3a191845-5f86-4e78-8fce-c4cff59f9daa)
Boot0002* IBA 40G Slot 3B00 v1118        BBS(128,IBA 40G Slot 3B00 v1118,0x0)..;...................................................................................A.....................I.B.A. .4.0.G. .S.l.o.t. .3.B.0.0. .v.1.1.1.8...
Boot0003* IBA 40G Slot 8600 v1118        BBS(128,IBA 40G Slot 8600 v1118,0x0)..................`.......`...`.......................................................A.....................I.B.A. .4.0.G. .S.l.o.t. .8.6.0.0. .v.1.1.1.8...
Boot0006* Red Hat Enterprise Linux        HD(2,GPT,6fe00ee0-1948-4e3c-8254-c5d7702b677f,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)
Boot0008* Red Hat Enterprise Linux        HD(2,GPT,1e8869d4-1225-4915-866c-9e18550a9a72,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)
MirroredPercentageAbove4G: 0.00
MirrorMemoryBelow4GB: false

Real Time Kernel Interactions

After enabling or installing the RT kernel, some of the native interactions with secure boot change. The first thing to note is that the efivars are missing, so the usage of mokutil and efibootmgr does not operate normally. See the following BZ for more details.

uname -r
4.18.0-305.19.1.rt7.91.el8_4.x86_64
mokutil --sb-state
EFI variables are not supported on this system
mokutil -l
EFI variables are not supported on this system
efibootmgr -v
EFI variables are not supported on this system
dmesg | grep -i secure
[    0.000000] secureboot: Secure boot enabled
[    0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7

Adding the following kernel boot arguments will allow for interaction again:

cat <<EOF | oc create -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
  machineconfiguration.openshift.io/role: master
name: 99-efi-runtime-path-kargs
spec:
kernelArguments:
- "efi=runtime"
EOF

Out-of-tree Driver Installation and caveats

When compiling a driver out-of-tree, there are a few options for compiling that include signing and not signing drivers. When adding an unsigned driver from out-of-tree, the following error will occur when loading it:

Nov 16 22:19:15 worker-2 bash[203728]: depmod: WARNING: could not open /lib/modules/4.18.0-305.25.1.rt7.97.el8_4.x86_64/modules.order: No such file or directory
Nov 16 22:19:18 worker-2 bash[203728]: depmod: WARNING: could not open /lib/modules/4.18.0-305.25.1.rt7.97.el8_4.x86_64/modules.builtin: No such file or directory
Nov 16 22:19:18 worker-2 bash[203728]: rmmod: ERROR: Module ice is not currently loaded
Nov 16 22:19:19 worker-2 bash[203728]: modprobe: ERROR: could not insert 'ice': Required key not available

This SRO documentation includes instructions for building an OOT kernel module. The CNF features repo includes some additional assistance to sign them upon creation. It requires the following openssl commands to generate the appropriate certificates:

openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3650 -subj "/CN=Telco 5G PK Certificate/" -keyout PK.key  -outform PEM -out PK.pem
openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3650 -subj "/CN=Telco 5G KEK Certificate/" -keyout KEK.key -out KEK.pem
openssl req -new -newkey rsa:2048 -sha256 -nodes -subj "/CN=Telco 5G KEK Certificate/" -keyout KEK.key -out KEK.csr
openssl x509 -req -in KEK.csr -CA PK.pem -CAkey PK.key -set_serial 1 -days 3650 -outform DER -out KEK.der
openssl x509 -inform DER -outform PEM -in KEK.der -out KEK.crt

After the kernel module is completed and added to the registry in question, import the DER file created using the mokutil command and provide a password. Once added, reboot the server and the mokmanager.efi should start and prompt you to import the public key.

Now, the OOT driver systemd unit files should load, and the OOT drivers should be accessible.

mokutil --import /home/core/KEK.der

# After reboot and importing via mokmanager the certificate will be viewable 

keyctl list %:.platform
7 keys in keyring:
620882315: ---lswrv     0     0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
359314383: ---lswrv     0     0 asymmetric: Telco 5G KEK Certificate: 46f6d18fe9187744442352bea6f9062b7ab2e2f4

mokutil -l
..omitted..
[key 3]
SHA1 Fingerprint: c2:2b:73:21:5c:aa:dd:de:fe:dd:83:ce:0f:0c:21:cd:d8:45:07:c2
Certificate:
   Data:
       Version: 1 (0x0)
       Serial Number: 1 (0x1)
       Signature Algorithm: sha256WithRSAEncryption
       Issuer: CN=Telco 5G KEK Certificate
       Validity
           Not Before: Nov 16 20:37:56 2021 GMT
           Not After : Nov 14 20:37:56 2031 GMT
       Subject: CN=Telco 5G KEK Certificate
       Subject Public Key Info:
           Public Key Algorithm: rsaEncryption
               RSA Public-Key: (2048 bit)
               Modulus:
..omitted..

modinfo ice.ko  | egrep '^version|^signer'

version:        1.6.4
signer:         Telco 5G PK Certificate

 

Troubleshooting

Intermittently, while provisioning via Assisted Service/ACM CRs, the life cycle controller can get into a state where it can not set secure boot. The logs resemble the following:

  Normal  ProvisioningStarted     2m27s  metal3-baremetal-controller  Image provisioning started for https://assisted-image-service-open-cluster-management.apps.clus3a.t5g.lab.eng.bos.redhat.com/images/e38d94fd-1e57-4da8-bd6b-0903e856d8a3?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbmZyYV9lbnZfaWQiOiJlMzhkOTRmZC0xZTU3LTRkYTgtYmQ2Yi0wOTAzZTg1NmQ4YTMifQ.Y43XSDaUahJdRTmImzSwjKSyJIHy-ryswIcPVcSkGx1sx8qkT5SbjZrq_Zl12DgsGtcDnhPGn-40WWwfD6Rf-g&arch=x86_64&type=minimal-iso&version=4.9
 Normal  ProvisioningError       2m6s   metal3-baremetal-controller  Image provisioning failed: Failed to deploy. Exception: HTTP POST https://10.19.28.55/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia returned code 400. Base.1.7.GeneralError: A general error has occurred. See ExtendedInfo for more information Extended information: [{'Message': 'Unable to mount remote share http://10.19.32.197:6180/redfish/boot-dfa94054-d7a0-4b92-858e-2b76e77ffeff.iso?filename=tmp_uxe1rp4.iso.', 'MessageArgs': ['http://10.19.32.197:6180/redfish/boot-dfa94054-d7a0-4b92-858e-2b76e77ffeff.iso?filename=tmp_uxe1rp4.iso'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.4.RAC0720', 'RelatedProperties': ['#/Image'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Retry the operation.', 'Severity': 'Informational'}]
 Normal  DeprovisioningStarted   47s    metal3-baremetal-controller  Image deprovisioning started
 Normal  DeprovisioningComplete  37s    metal3-baremetal-controller  Image deprovisioning completed
 Normal  ProvisioningStarted     36s    metal3-baremetal-controller  Image provisioning started for https://assisted-image-service-open-cluster-management.apps.clus3a.t5g.lab.eng.bos.redhat.com/images/e38d94fd-1e57-4da8-bd6b-0903e856d8a3?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbmZyYV9lbnZfaWQiOiJlMzhkOTRmZC0xZTU3LTRkYTgtYmQ2Yi0wOTAzZTg1NmQ4YTMifQ.Y43XSDaUahJdRTmImzSwjKSyJIHy-ryswIcPVcSkGx1sx8qkT5SbjZrq_Zl12DgsGtcDnhPGn-40WWwfD6Rf-g&arch=x86_64&type=minimal-iso&version=4.9
 Normal  ProvisioningError       26s    metal3-baremetal-controller  Image provisioning failed: Deploy step deploy.deploy failed: Redfish exception occurred. Error: Failed to set secure boot state on node dfa94054-d7a0-4b92-858e-2b76e77ffeff to True: HTTP PATCH https://10.19.28.55/redfish/v1/Systems/System.Embedded.1/SecureBoot returned code 400. Base.1.7.GeneralError: Pending configuration values are already committed, unable to perform another set operation. Extended information: [{'Message': 'Pending configuration values are already committed, unable to perform another set operation.', 'MessageArgs': ['SecureBootEnable'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.4.SYS011', 'RelatedProperties': ['#/SecureBootEnable'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Wait for the scheduled job to complete or delete the configuration jobs before attempting more set attribute operations.', 'Severity':

Note:

This was resolved in 4.10 via an ironic change to clear the life cycle controller. See the following BZ.

Workaround:

  1. Remove the bare-metalhost CR.
  2. Clear the life cycle controller job queue.
  3. Re-add the bare-metal host CR.

Summary

The idea of creating a custom certificate and signing a driver can be a daunting task. Using the driver toolkit, SRO, and NFD operators help to compile an out-of-tree driver for the version of OpenShift of your choosing. Then efimanager and mokutil make importing the certificate a painless administrative process.