Red Hat blog
Secure boot is a compromise between the UEFI bootloader and the operating system it hands off to. Explaining the complete details of secure boot in RHEL/RHCOS is out of scope for this document. The following article does an excellent job of explaining it properly.
When using secure boot in a single node deployment of OpenShift, two methods can be used: manually setting secure boot in the system BIOS or using the bare-metal host CR to set the secure boot setting during provisioning.
More information regarding signing drivers and verifying them on RHEL8 is available in the official documentation online.
Verifying Secure Boot Status
A redfish curl command can be used to determine the current state of secure boot against a BMC endpoint. There are two examples below, one for HP and one for Dell:
# HP
curl -sS -k -u Administrator:password -H "Content-Type: application/json" -H "OData-Version: 4.0" https://[fd00:4888:x:y::z]/redfish/v1/Systems/1/SecureBoot | jq '.'
{
"@odata.context": "/redfish/v1/$metadata#SecureBoot.SecureBoot",
"@odata.etag": "W/\"4A4CB737\"",
"@odata.id": "/redfish/v1/Systems/1/SecureBoot",
"@odata.type": "#SecureBoot.v1_0_0.SecureBoot",
"Id": "SecureBoot",
"Actions": {
"#SecureBoot.ResetKeys": {
"target": "/redfish/v1/Systems/1/SecureBoot/Actions/SecureBoot.ResetKeys"
}
},
"Name": "SecureBoot",
"SecureBootCurrentBoot": "Disabled",
"SecureBootEnable": false,
"SecureBootMode": "UserMode"
}
# Dell
curl -sS -k -u administrator:password -H "Content-Type: application/json" -H "OData-Version: 4.0" https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/SecureBoot | jq '.'
{
"@odata.context": "/redfish/v1/$metadata#SecureBoot.SecureBoot",
"@odata.id": "/redfish/v1/Systems/System.Embedded.1/SecureBoot",
"@odata.type": "#SecureBoot.v1_0_6.SecureBoot",
"Actions": {
"#SecureBoot.ResetKeys": {
"ResetKeysType@Redfish.AllowableValues": [
"ResetAllKeysToDefault",
"DeleteAllKeys",
"DeletePK",
"ResetPK",
"ResetKEK",
"ResetDB",
"ResetDBX"
],
"target": "/redfish/v1/Systems/System.Embedded.1/SecureBoot/Actions/SecureBoot.ResetKeys"
},
"Oem": {}
},
"Description": "UEFI Secure Boot",
"Id": "SecureBoot",
"Name": "UEFI Secure Boot",
"Oem": {
"Dell": {
"@odata.context": "/redfish/v1/$metadata#DellSecureBoot.DellSecureBoot",
"@odata.type": "#DellSecureBoot.v1_0_0.DellSecureBoot",
"Certificates": {
"@odata.id": "/redfish/v1/Systems/System.Embedded.1/SecureBoot/Oem/Dell/Certificates"
}
}
},
"SecureBootCurrentBoot": "Enabled",
"SecureBootEnable": true,
"SecureBootMode": "DeployedMode"
}
Enabling Secure Boot
There are two methods for enabling secure boot. The first would be to interact with the hardware in question. Here is a redfish command to enable it on Dell hardware:
curl -i -k -X PATCH -u administrator:password -H "Content-Type: application/json" -d '{"SecureBootEnable":true}' https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/SecureBoot
{"@Message.ExtendedInfo":[{"Message":"Successfully Completed Request","MessageArgs":[],"MessageArgs@odata.count":0,"MessageId":"Base.1.7.Success","RelatedProperties":[],"RelatedProperties@odata.count":0,"Resolution":"None","Severity":"OK"},{"Message":"The operation is successfully completed.","MessageArgs":[],"MessageArgs@odata.count":0,"MessageId":"IDRAC.2.4.SYS430","RelatedProperties":[],"RelatedProperties@odata.count":0,"Resolution":"No response action is required.However, to make them immediately effective, restart the host server.","Severity":"Informational"}]}
curl -i -k -X POST -u administrator:password -H "Content-Type: application/json" -d '{"ResetType":"GracefulRestart"}' https://10.19.x.y/redfish/v1/Systems/System.Embedded.1/Actions/ComputerSystem.Reset
The following is the equivalent command for HPE hardware:
curl -i -k -X PATCH -u administrator:password -H "Content-Type: application/json" -d '{"SecureBootEnable":true}' https://10.19.x.y/redfish/v1/Systems/1/SecureBoot/
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-length: 261
Content-type: application/json; charset=utf-8
Date: Thu, 18 Nov 2021 14:08:35 GMT
ETag: W/"78E0DBAE"
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X_HP-CHRP-Service-Version: 1.0.3
{"Messages":[{"MessageID":"iLO.0.10.SystemResetRequired"}],"Type":"ExtendedError.1.0.0","error":{"@Message.ExtendedInfo":[{"MessageID":"iLO.0.10.SystemResetRequired"}],"code":"iLO.0.10.ExtendedInfo","message":"See @Message.ExtendedInfo for more information."}}
# Validating that the change has been made:
curl -i -k -X GET -u administrator:password https://10.19.x.y/redfish/v1/Systems/1/SecureBoot/
HTTP/1.1 200 OK
Allow: GET, HEAD, PATCH
Cache-Control: no-cache
Content-length: 407
Content-type: application/json; charset=utf-8
Date: Thu, 18 Nov 2021 14:10:19 GMT
ETag: W/"AEA4F742"
Link: </redfish/v1/SchemaStore/en/HpSecureBoot.json/>; rel=describedby
X-Content-Type-Options: nosniff
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
X_HP-CHRP-Service-Version: 1.0.3
{"@odata.context":"/redfish/v1/$metadata#Systems/Members/1/SecureBoot$entity","@odata.id":"/redfish/v1/Systems/1/SecureBoot/","@odata.type":"#HpSecureBoot.1.0.0.HpSecureBoot","Id":"SecureBoot","Name":"SecureBoot","ResetAllKeys":false,"ResetToDefaultKeys":false,"SecureBootCurrentState":false,"SecureBootEnable":true,"Type":"HpSecureBoot.1.0.0","links":{"self":{"href":"/redfish/v1/Systems/1/SecureBoot/"}}}
curl -i -k -X POST -u administrator:password -H "Content-Type: application/json" -d '{"ResetType":"ForceRestart"}' https://10.19.x.y/redfish/v1/Systems/1/Actions/ComputerSystem.Reset/
{"Messages":[{"MessageID":"Base.0.10.Success"}],"Type":"ExtendedError.1.0.0","error":{"@Message.ExtendedInfo
The custom resource for the bare-metal host can be used to configure secure boot as well. Here is an example of the CR to use for doing an assisted service SNO deployment. For more information regarding assisted service ACM ZTP install, refer to the following unofficial repository.
spec:
online: true
automatedCleaningMode: disabled
bootMACAddress: e4:43:4b:x:y:z
bootMode: UEFISecureBoot
bmc:
address: idrac-virtualmedia+https://10.19.x.y/redfish/v1/Systems/System.Embedded.1
credentialsName: sno-worker-2-worker
disableCertificateVerification: true
After the single node is installed, the secure boot state can be verified a couple of different ways. The mokutil command can be used to verify and to add custom certificates for secure boot:
mokutil --sb-state
SecureBoot enabled
mokutil --list-enrolled
[key 1]
SHA1 Fingerprint: e6:f5:06:46:20:69:aa:17:d2:e8:61:05:03:63:5c:20:f3:a9:95:c3
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
83:73:0d:2b:72:80:d1:5a
Signature Algorithm: sha256WithRSAEncryption
Issuer: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
Validity
Not Before: Jun 9 08:15:36 2020 GMT
Not After : Jan 18 08:15:36 2038 GMT
Subject: O=Red Hat, Inc., CN=Red Hat Secure Boot CA 5/emailAddress=secalert@redhat.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
..omitted..
dmesg | egrep 'efi|secure'
[ 0.000000] efi: EFI v2.70 by Dell Inc.
[ 0.000000] efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x68d36000 SMBIOS 3.0=0x68d34000 MEMATTR=0x6525f020 MOKvar=0x683a9000
[ 0.000000] secureboot: Secure boot enabled
..omitted..
efibootmgr -v
BootCurrent: 0008
BootOrder: 0001,0006,0008
Boot0000* Hard drive C: VenHw(d6c0639f-c705-4eb9-aa4f-5802d8823de6)............................f.........................................................A.....................P.E.R.C. .H.7.3.0.P. .M.X. .(.b.u.s. .1.8. .d.e.v. .0.0.)...
Boot0001 NIC in Mezzanine 1A Port 1 Partition 1 VenHw(3a191845-5f86-4e78-8fce-c4cff59f9daa)
Boot0002* IBA 40G Slot 3B00 v1118 BBS(128,IBA 40G Slot 3B00 v1118,0x0)..;...................................................................................A.....................I.B.A. .4.0.G. .S.l.o.t. .3.B.0.0. .v.1.1.1.8...
Boot0003* IBA 40G Slot 8600 v1118 BBS(128,IBA 40G Slot 8600 v1118,0x0)..................`.......`...`.......................................................A.....................I.B.A. .4.0.G. .S.l.o.t. .8.6.0.0. .v.1.1.1.8...
Boot0006* Red Hat Enterprise Linux HD(2,GPT,6fe00ee0-1948-4e3c-8254-c5d7702b677f,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)
Boot0008* Red Hat Enterprise Linux HD(2,GPT,1e8869d4-1225-4915-866c-9e18550a9a72,0x1000,0x3f800)/File(\EFI\redhat\shimx64.efi)
MirroredPercentageAbove4G: 0.00
MirrorMemoryBelow4GB: false
Real Time Kernel Interactions
After enabling or installing the RT kernel, some of the native interactions with secure boot change. The first thing to note is that the efivars are missing, so the usage of mokutil and efibootmgr does not operate normally. See the following BZ for more details.
uname -r
4.18.0-305.19.1.rt7.91.el8_4.x86_64
mokutil --sb-state
EFI variables are not supported on this system
mokutil -l
EFI variables are not supported on this system
efibootmgr -v
EFI variables are not supported on this system
dmesg | grep -i secure
[ 0.000000] secureboot: Secure boot enabled
[ 0.000000] Kernel is locked down from EFI secure boot; see man kernel_lockdown.7
Adding the following kernel boot arguments will allow for interaction again:
cat <<EOF | oc create -f -
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 99-efi-runtime-path-kargs
spec:
kernelArguments:
- "efi=runtime"
EOF
Out-of-tree Driver Installation and caveats
When compiling a driver out-of-tree, there are a few options for compiling that include signing and not signing drivers. When adding an unsigned driver from out-of-tree, the following error will occur when loading it:
Nov 16 22:19:15 worker-2 bash[203728]: depmod: WARNING: could not open /lib/modules/4.18.0-305.25.1.rt7.97.el8_4.x86_64/modules.order: No such file or directory
Nov 16 22:19:18 worker-2 bash[203728]: depmod: WARNING: could not open /lib/modules/4.18.0-305.25.1.rt7.97.el8_4.x86_64/modules.builtin: No such file or directory
Nov 16 22:19:18 worker-2 bash[203728]: rmmod: ERROR: Module ice is not currently loaded
Nov 16 22:19:19 worker-2 bash[203728]: modprobe: ERROR: could not insert 'ice': Required key not available
This SRO documentation includes instructions for building an OOT kernel module. The CNF features repo includes some additional assistance to sign them upon creation. It requires the following openssl commands to generate the appropriate certificates:
openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3650 -subj "/CN=Telco 5G PK Certificate/" -keyout PK.key -outform PEM -out PK.pem
openssl req -new -x509 -newkey rsa:2048 -sha256 -nodes -days 3650 -subj "/CN=Telco 5G KEK Certificate/" -keyout KEK.key -out KEK.pem
openssl req -new -newkey rsa:2048 -sha256 -nodes -subj "/CN=Telco 5G KEK Certificate/" -keyout KEK.key -out KEK.csr
openssl x509 -req -in KEK.csr -CA PK.pem -CAkey PK.key -set_serial 1 -days 3650 -outform DER -out KEK.der
openssl x509 -inform DER -outform PEM -in KEK.der -out KEK.crt
After the kernel module is completed and added to the registry in question, import the DER file created using the mokutil command and provide a password. Once added, reboot the server and the mokmanager.efi should start and prompt you to import the public key.
Now, the OOT driver systemd unit files should load, and the OOT drivers should be accessible.
mokutil --import /home/core/KEK.der
# After reboot and importing via mokmanager the certificate will be viewable
keyctl list %:.platform
7 keys in keyring:
620882315: ---lswrv 0 0 asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed522988a1bd4
359314383: ---lswrv 0 0 asymmetric: Telco 5G KEK Certificate: 46f6d18fe9187744442352bea6f9062b7ab2e2f4
mokutil -l
..omitted..
[key 3]
SHA1 Fingerprint: c2:2b:73:21:5c:aa:dd:de:fe:dd:83:ce:0f:0c:21:cd:d8:45:07:c2
Certificate:
Data:
Version: 1 (0x0)
Serial Number: 1 (0x1)
Signature Algorithm: sha256WithRSAEncryption
Issuer: CN=Telco 5G KEK Certificate
Validity
Not Before: Nov 16 20:37:56 2021 GMT
Not After : Nov 14 20:37:56 2031 GMT
Subject: CN=Telco 5G KEK Certificate
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
..omitted..
modinfo ice.ko | egrep '^version|^signer'
version: 1.6.4
signer: Telco 5G PK Certificate
Troubleshooting
Intermittently, while provisioning via Assisted Service/ACM CRs, the life cycle controller can get into a state where it can not set secure boot. The logs resemble the following:
Normal ProvisioningStarted 2m27s metal3-baremetal-controller Image provisioning started for https://assisted-image-service-open-cluster-management.apps.clus3a.t5g.lab.eng.bos.redhat.com/images/e38d94fd-1e57-4da8-bd6b-0903e856d8a3?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbmZyYV9lbnZfaWQiOiJlMzhkOTRmZC0xZTU3LTRkYTgtYmQ2Yi0wOTAzZTg1NmQ4YTMifQ.Y43XSDaUahJdRTmImzSwjKSyJIHy-ryswIcPVcSkGx1sx8qkT5SbjZrq_Zl12DgsGtcDnhPGn-40WWwfD6Rf-g&arch=x86_64&type=minimal-iso&version=4.9
Normal ProvisioningError 2m6s metal3-baremetal-controller Image provisioning failed: Failed to deploy. Exception: HTTP POST https://10.19.28.55/redfish/v1/Managers/iDRAC.Embedded.1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia returned code 400. Base.1.7.GeneralError: A general error has occurred. See ExtendedInfo for more information Extended information: [{'Message': 'Unable to mount remote share http://10.19.32.197:6180/redfish/boot-dfa94054-d7a0-4b92-858e-2b76e77ffeff.iso?filename=tmp_uxe1rp4.iso.', 'MessageArgs': ['http://10.19.32.197:6180/redfish/boot-dfa94054-d7a0-4b92-858e-2b76e77ffeff.iso?filename=tmp_uxe1rp4.iso'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.4.RAC0720', 'RelatedProperties': ['#/Image'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Retry the operation.', 'Severity': 'Informational'}]
Normal DeprovisioningStarted 47s metal3-baremetal-controller Image deprovisioning started
Normal DeprovisioningComplete 37s metal3-baremetal-controller Image deprovisioning completed
Normal ProvisioningStarted 36s metal3-baremetal-controller Image provisioning started for https://assisted-image-service-open-cluster-management.apps.clus3a.t5g.lab.eng.bos.redhat.com/images/e38d94fd-1e57-4da8-bd6b-0903e856d8a3?api_key=eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJpbmZyYV9lbnZfaWQiOiJlMzhkOTRmZC0xZTU3LTRkYTgtYmQ2Yi0wOTAzZTg1NmQ4YTMifQ.Y43XSDaUahJdRTmImzSwjKSyJIHy-ryswIcPVcSkGx1sx8qkT5SbjZrq_Zl12DgsGtcDnhPGn-40WWwfD6Rf-g&arch=x86_64&type=minimal-iso&version=4.9
Normal ProvisioningError 26s metal3-baremetal-controller Image provisioning failed: Deploy step deploy.deploy failed: Redfish exception occurred. Error: Failed to set secure boot state on node dfa94054-d7a0-4b92-858e-2b76e77ffeff to True: HTTP PATCH https://10.19.28.55/redfish/v1/Systems/System.Embedded.1/SecureBoot returned code 400. Base.1.7.GeneralError: Pending configuration values are already committed, unable to perform another set operation. Extended information: [{'Message': 'Pending configuration values are already committed, unable to perform another set operation.', 'MessageArgs': ['SecureBootEnable'], 'MessageArgs@odata.count': 1, 'MessageId': 'IDRAC.2.4.SYS011', 'RelatedProperties': ['#/SecureBootEnable'], 'RelatedProperties@odata.count': 1, 'Resolution': 'Wait for the scheduled job to complete or delete the configuration jobs before attempting more set attribute operations.', 'Severity':
Note:
This was resolved in 4.10 via an ironic change to clear the life cycle controller. See the following BZ.
Workaround:
- Remove the bare-metalhost CR.
- Clear the life cycle controller job queue.
- Re-add the bare-metal host CR.
Summary
The idea of creating a custom certificate and signing a driver can be a daunting task. Using the driver toolkit, SRO, and NFD operators help to compile an out-of-tree driver for the version of OpenShift of your choosing. Then efimanager and mokutil make importing the certificate a painless administrative process.