Managing NIST 800-53 Controls in a Multicluster OpenShift Environment - Part 1

In this three-part blog series, I discuss the challenges that organizations face when managing multicluster OpenShift deployments and examine how they affect security and compliance in large organizations. Afterwards, I explain specific security practices and how to implement them across multiple clusters with the capabilities in Red Hat OpenShift Platform Plus, to improve the compliance of your OpenShift fleet. I use NIST special publication 800-53 security controls as a standard reference. Some existing familiarity with Kubernetes and NIST 800-53 controls is assumed.

Note that software alone cannot fully address NIST 800-53 controls; many of the controls must be met through the implementation of organizational processes. However, software can be used to automate and monitor compliance with certain technical controls.

A modern way of thinking

In the past, enterprises focused on deploying and managing a few on-premise application platforms for running all their workloads. Digital transformation accelerated the need for organizations to act quickly to meet business goals using new technologies, and organizations continually face pressure to do more with less. COVID-19 changed the role of digital transformation in many organizations from a strategic effort to a Darwinian imperative. The growth in public cloud, along with the need to maintain non-cloud based investments, have resulted in an unprecedented opportunity for hybrid cloud solutions.

Kubernetes has emerged as the dominant solution for orchestration of containerized applications built for cloud-native environments. Red Hat OpenShift is an enterprise open source container orchestration platform that includes Kubernetes, but adds productivity and security features that are important to large-scale companies. Organizations may now manage multiple Kubernetes clusters that run workloads across multiple public or private clouds, as well as on-premises data centers. Each of these clusters can be designed to meet different business needs, such as:

• Running production workloads
• Performing software development testing and quality assurance functions
• Running the workloads for a specific business unit
• Performing assigned roles such as cluster management or AI/ML workloads
• Performing edge computing functions

Modern solutions create modern challenges

With the shift towards hybrid and multicloud, many new challenges were introduced within organizations. Organizations frequently ask the following questions on their journey to digital transformation:

1. How will I continue to meet my contractual and regulatory obligations, such as NIST 800-53?
2. How do I manage my Kubernetes deployments at scale?
3. How do I maintain consistent configurations across multiple clusters to meet the needs of my organization and enterprise standards for security and software engineering?
4. How do I secure my workloads in a new application architecture?

Red Hat OpenShift Platform Plus is designed to help organizations answer these questions. In April 2021, Red Hat introduced OpenShift Platform Plus, which extends OpenShift with advanced multicluster management, governance, and security for DevSecOps by including the following core solutions:

1. Red Hat Advanced Cluster Management for Kubernetes (RHACM) - Red Hat Advanced Cluster Management for Kubernetes provides end-to-end cluster and application lifecycle management, governance, and observability to manage your multicluster Kubernetes environment.
2. Red Hat Quay (Quay) - Red Hat Quay is a private container registry that stores, builds, and deploys container images. It includes Clair to analyze your container images for security vulnerabilities, identifying potential issues that can help you mitigate security risks.
3. Red Hat Advanced Cluster Security for Kubernetes (RHACS) - Red Hat Advanced Cluster Security for Kubernetes provides a Kubernetes-native architecture for cluster and application security, enabling DevOps and InfoSec teams to operationalize security.

What is NIST 800-53?

NIST 800-53 is a risk management framework for federal information systems of the U.S. government. Organizations typically need to implement a NIST 800-53 risk management program because they are contractors or vendors of the United States federal government, or have customers that must align with NIST 800-53 to maintain their contractual obligations to the U.S. government. Although originally targeted for federal information systems, these controls are generally useful and more and more enterprises have been leveraging NIST 800-53 to inform their security policies and requirements.

NIST 800-53 controls are broken down into families as shown in the following table:

Source: NIST 800-53 Rev5

These control families define technical and process management controls required to secure federal information systems. While software alone cannot fully address NIST 800-53 controls, software can be used to automate and monitor compliance with certain technical controls. This guide walks you through some of the ways that OpenShift Platform Plus can help you to address certain technical controls in your Kubernetes clusters on your journey to NIST 800-53 compliance.

Let’s start by looking at how OpenShift can help you manage compliance for a single OpenShift cluster and then explore how OpenShift Platform Plus helps you scale compliance management across your fleet of clusters.

OpenShift and the OpenShift Compliance Operator

The Compliance Operator lets OpenShift Container Platform administrators identify the set of technical controls that a cluster should comply with, and provides them with an overview of gaps and ways to remediate those gaps. The Compliance Operator assesses compliance of both the Kubernetes resources of OpenShift, as well as the operating system on the nodes running the cluster. The Compliance Operator uses OpenSCAP, a NIST-certified tool, to scan and enforce security policies provided by the profiles delivered with the Compliance Operator. The Compliance Operator includes profiles for assessing OpenShift clusters against the relevant technical controls from NIST 800-53. This solution is integrated as a core building block within OpenShift Platform Plus.

Let’s take a deeper look at how the Compliance Operator works.

After installing the OpenShift compliance operator, you can view the available compliance profiles by running the following command:

$ oc get profiles.compliance -n openshift-compliance

NAMESPACE          	  NAME          	            AGE
openshift-compliance   ocp4-cis      	                    35m
openshift-compliance   ocp4-cis-node 	                    35m
openshift-compliance   ocp4-e8       	                    35m
openshift-compliance   ocp4-moderate 	                    35m
openshift-compliance   rhcos4-e8     	                    35m
openshift-compliance   rhcos4-moderate                      35m

The ocp4-moderate profile focuses on NIST SP 800-53 moderate impact baseline configurations for OpenShift 4 deployments. To view the rules associated with this compliance profile, run the following command and observe the output in the rules section:

$ oc get profiles.compliance ocp4-moderate -n openshift-compliance -o yaml

apiVersion: compliance.openshift.io/v1alpha1
…..
rules:
- ocp4-api-server-client-ca
- ocp4-api-server-encryption-provider-cipher
- ocp4-api-server-encryption-provider-config
- ocp4-api-server-etcd-ca
- ocp4-api-server-tls-cert
- ocp4-api-server-tls-private-key
- ocp4-audit-log-forwarding-enabled
- ocp4-audit-profile-set
- ocp4-classification-banner
- ocp4-etcd-peer-cert-file
- ocp4-etcd-peer-key-file
- ocp4-kubelet-configure-tls-cert
- ocp4-kubelet-configure-tls-key
- ocp4-ocp-allowed-registries
- ocp4-ocp-allowed-registries-for-import
- ocp4-ocp-idp-no-htpasswd
title: NIST 800-53 Moderate-Impact Baseline for Red Hat OpenShift

Each rule reflects an OpenShift security practice that is associated with a NIST SP 800-53 security control. For example, the ocp4-api-server-encryption-provider-cipher rule makes sure that the etcd database is encrypted with the AES-CBC encryption provider. To get more information about the rule, run the following command:

$ oc get rule.compliance ocp4-api-server-encryption-provider-cipher -n openshift-compliance -o yaml

apiVersion: compliance.openshift.io/v1alpha1
availableFixes:
- fixObject:
	apiVersion: config.openshift.io/v1
	kind: APIServer
	metadata:
  	name: cluster
	spec:
  	encryption:
    	type: aescbc
description: To ensure the correct cipher, set the encryption type aescbc in the apiserver object which configures the API server itself …
kind: Rule
metadata:
  annotations:
	compliance.openshift.io/image-digest: pb-ocp4dvvb5
	compliance.openshift.io/rule: api-server-encryption-provider-cipher
	control.compliance.openshift.io/CIS-OCP: 1.2.34
	control.compliance.openshift.io/NIST-800-53: SC-28(1)
	policies.open-cluster-management.io/controls: 1.2.34,SC-28(1)
	policies.open-cluster-management.io/standards: CIS-OCP,NIST-800-53
  labels:
	compliance.openshift.io/profile-bundle: ocp4
    name: ocp4-api-server-encryption-provider-cipher
  namespace: openshift-compliance
rationale: aescbc is currently the strongest encryption provider, it should be preferred over other providers.

Notice that the rule is associated with the SC-28 (1) Protection of information at rest security control in the NIST SP 800-53 standard.

After running a compliance scan as described in the Compliance Operator documentation, you are able to validate if your environment is compliant with the profile or profiles you selected. In the following example, the environment is not compliant with the ocp4-api-server-encryption-provider-cipher rule:

$ oc get ComplianceCheckResult -n openshift-compliance | grep cipher

NAME                                                                    STATUS          SEVERITY
ocp4-moderate-api-server-encryption-provider-cipher                     FAIL         	medium

The compliance rules of the compliance operator frequently come with automated remediation actions. The compliance rule remediation is represented by the ComplianceRemediation custom resource. The ComplianceRemediation resource name is the same as the name of the compliance rule for which it provides remediation. For example, remediation for the ocp4-api-server-encryption-provider-cipher rule is provided by the ocp4-api-server-encryption-provider-cipher ComplianceRemediation resource:

$ oc get ComplianceRemediation -n openshift-compliance | grep cipher

NAME                                         	      STATE
ocp4-moderate-api-server-encryption-provider-cipher   NotApplied

To remediate the ocp4-api-server-encryption-provider-cipher ComplianceCheckResult, run the following command:

$ oc patch ComplianceRemediation/ocp4-moderate-api-server-encryption-provider-cipher --patch '{"spec":{"apply":true}}' --type=merge -n openshift-compliance

After you apply the compliance remediation, perform another compliance scan. The new scan shows that the environment is now compliant with the rule and the NIST 800-53 SC-28 (1) security control:

$ oc get ComplianceCheckResult -n openshift-compliance | grep cipher

NAME                                                                    STATUS          SEVERITY
ocp4-moderate-api-server-encryption-provider-cipher                     PASS         	medium

The Compliance Operator also provides profiles that review the underlying operating system, Red Hat Enterprise Linux CoreOS (RHEL CoreOS). Using the Compliance Operator, you are able to evaluate RHEL CoreOS node compliance with NIST SP 800-53 technical security controls using the rhcos4-moderate profile.

Configuring an OpenShift cluster to be compliant with relevant NIST 800-53 technical controls is straightforward. Most of the technical controls are implemented by default within OpenShift. Those that are not implemented by default can be configured using built-in OpenShift mechanisms, such as NetworkPolicy, a solution to implement software-defined network boundaries within Kubernetes. Another mechanism, ResourceQuotas, is a solution to limit the resources available to applications deployed on Kubernetes.

The more OpenShift clusters you have, the more resources, time, and attention required to keep them secured. OpenShift Platform Plus helps organizations to scale by enabling declarative governance and security policy, and automating management of those policies across a fleet of clusters with RHACM and RHACS.

Let's go through the following scenario to understand how an OpenShift multicluster environment can easily become non-compliant to applicable NIST 800-53 technical security controls. After that, let's look at how RHACM and RHACS can be used to bring the cluster back into compliance with technical controls.

Multicluster configuration and compliance drift

Let's say Company A decided to deploy multiple OpenShift clusters in different sites to mitigate the latency for its customers. All of the clusters run the same application, and are managed by the administration team in the organization, as shown in the following diagram:

Figure 1: An administrator that manages multiple, individual OpenShift clusters deployed in different sites.

How can we address the compliance needs of this setup?

Continue reading to view some of the factors that can cause security compliance issues in this setup, associated with the relevant NIST 800-53 security controls:

• AC-3 - The AC-3 control states, "The information system enforces approved authorizations for logical access to information and system resources in accordance with applicable access control policies."

  In this scenario, each cluster maintains a set of permissions that are associated with users and groups. It is difficult to maintain an approved set of permissions in a large-scale multicluster environment, especially when changes in security policies occur.

  Role-based access control (RBAC) drifts can be frequent in this scenario, making it challenging to comply with the AC-3 security control.

• SI-4(a.1) - The SI-4(a.1) control states, "The organization monitors the information system to detect attacks and indicators of potential attacks in accordance with organization-defined monitoring objectives."

  One element of meeting the control is to monitor applications for newly discovered vulnerabilities and update those applications with fixes to the vulnerabilities. If the organization does not scan vulnerabilities in application instances, some instances can become weak in the environment, making the environment non-compliant with the SI-4(a.1) security control.

  In an environment in which the application has instances on multiple clusters, it is easy to produce application versioning drifts. If not provisioned correctly, an application can end up running version 1 on cluster A and version 2 on cluster B. In such cases, cluster A is prone to security vulnerabilities that were patched in version 2 of the application.

• CM-2 - The CM-2 control states, "The organization develops, documents, and maintains under configuration control, a current baseline configuration of the information system."

  When you manage a multicluster environment, it is possible to have drifts between baseline configurations of clusters. Changes made to individual clusters can lead to unaligned baseline configurations in different clusters. The lack of configuration management in the organization can result in an unaligned environment that is non-compliant with the CM-2 security control.

• SC-8 - The SC-8 control states, "The information system protects the confidentiality; integrity of transmitted information."

  To meet this requirement, TLS needs to be enabled for data-in-transit protection and the certificates used for TLS need to be managed properly. Certificates are used both for service authentication and safe data transmission. Therefore, it is crucial to make sure that TLS is configured, and certificates used for TLS are appropriately managed, rotated, and do not expire. In a large-scale environment with manual certificate management, an administrator can lose track of all the certificates in use. A lack of automation and visibility can result in expired or unrotated certificates, thereby making the environment non-compliant with the SC-8 security control.

It does not end there. Whenever one cluster is in a different state than the rest of the clusters, a risk to its security posture may be introduced. If the permissions are not aligned, the environment may be non-compliant with the AC security controls. If there are no configuration control tools in the environment, the environment may be non-compliant with the CM security control, and so on.

All of the previous points focus on one idea — alignment to best practices or enterprise standards. An unaligned environment is an insecure environment. If the environment is unaligned, it means that not all clusters follow the security requirements and or regulatory compliance obligations of the organization. Unaligned clusters increase the attack surface in the environment, and form a weak link in your OpenShift cluster fleet.

To tackle these issues, Red Hat provides two solutions that help with managing alignment at scale: Red Hat Advanced Cluster Security and Red Hat Advanced Cluster Management for Kubernetes. For security and risk management, use Red Hat Advanced Cluster Security. For policy-based governance, use Red Hat Advanced Cluster Management. Together, these two capabilities help you achieve your enterprise security and governance goals at scale across a fleet of OpenShift clusters.

Using Red Hat Advanced Cluster Security for Kubernetes to scale workload security

Red Hat Advanced Cluster Security for Kubernetes provides advanced security for Kubernetes, enabling organizations to build, deploy, run, and manage intelligent applications securely at scale. With multiple layers of security analysis, the platform enables DevSecOps adoption and provides a holistic, ranked risk assessment across managed clusters. These can help you to meet NIST 800-53 objectives and achieve a broad spectrum of security use cases.

RHACS acts as a fundamental part of an organization's Information security architecture and risk management strategy for Kubernetes and OpenShift. RHACS enables teams to gain visibility into their workload risk using vulnerability management, configuration management, and compliance data, and couples it with runtime event alerting.

For example, you can use RHACS to enforce a required labeling strategy team to assign security attributes to Kubernetes deployments. This can be used to ensure that any deployment has organizationally defined labels. Organizations frequently require labels ranging from cost centers to security context. This security context can then be used to assign appropriate controls as part of a broader baseline configuration and configuration of access restrictions for change.

Some examples labels include:

• If data classification is supported
• If a deployment is part of a crown jewel application
• If the application deployment is test or production

This can be used to enforce advanced security controls such as:

• Allowing direct access to pods through the Kubernetes API in test environments only
• Performing vulnerability scanning across deployments that are running in your cluster, and applying different enforcement actions based on the data classification of the deployment
• Disallowing a secure shell to be part of a container image that is part of a crown jewel deployment to limit attack surface
• Disallowing privileged containers to have an external load balancer

RHACS also enables incident response teams to perform automated and manual incident handling. RHACS provides automated process discovery and baselining capabilities by observing normal application behavior to build a list of known normal processes. Security teams can use this baseline to identify, alert on, and prevent abnormal processes from running in their environment. Some common attack indicators detected out-of-the-box include execution of a package manager inside of a pod, running a crypominer, or attempting to use privileged capabilities in the pod to modify the host operating system, where existing controls are insufficient.

RHACS takes a risk management approach to manage abnormal behavior. It is relatively simple to characterize desirable and undesirable behavior. Most traditional detection techniques focus on pinpointing abnormal behavior by looking solely at runtime events, in isolation, to characterize that behavior as malicious or benign. RHACS leverages the benefits of cloud-native infrastructure to put abnormal events in context and reduce the level of fidelity required to apply enforcement, based on those events.

RHACS users can view process execution events that fall outside the baseline for a given application, within the context of how that application was built, configured, and what it is supposed to do during runtime. Users can use this information to make a judgment about whether to classify that activity as desirable behavior by adding it to the application baseline. You can determine an undesirable behavior, by creating a RHACS policy to control this behavior and initiate an incident response workflow by deleting a pod, or sending an alert to a notifier (like a team slack channel or a SIEM).

Using Red Hat Quay to securely distribute container images and enforce least privilege for your private container images

Red Hat Quay container image registry provides storage and enables you to build, distribute, and deploy containers. Organizations can enforce their access controls to gain more security over image repositories with the automation, authentication, and authorization systems available in Quay.

Container image registries play a key role in securing your containerized applications. Your teams are building containers that layer content on top of the public container images you download. You must manage access to, and promotion of, the downloaded container images and the internally built images the same way you manage other types of binaries.

Red Hat Quay includes Clair for image scanning. The Clair project is an open source engine that powers the Red Hat Quay security scanner to detect vulnerabilities in all images within Red Hat Quay. Red Hat OpenShift Container Security Operator integrates with Red Hat Quay to provide a cluster-wide view of known vulnerabilities for your deployed images in the OpenShift console.

Using Red Hat Advanced Cluster Management for Kubernetes to scale cluster configuration and governance

Red Hat Advanced Cluster Management for Kubernetes is a platform that is designed to help developers and administrators to manage cloud-native applications that run on multiple OpenShift clusters. RHACM supports the creation of OpenShift clusters across multiple cloud providers. It provides a monitoring system that visualizes resources and metrics from all clusters in the environment. It implements deployment tools that provision applications across multiple clusters. Most importantly, it provides policy-based governance that enables clusters to operate at a desired configuration state based on best practices. It also easily detects clusters that may have drifted from such a desired configuration state.

Governance, Risk and Compliance

To regulate large multicluster environments, RHACM brings an important tool to the table — its policy-based governance framework, which provides the following key capabilities:

• Uses an extensible policy framework for configuration management across multiple clusters
• Uses placement rules to bind policies to managed clusters
• Supports the integration with the OpenShift Compliance Operator
• Supports multiple policy engines, including Gatekeeper/OPA, Kyverno
• Supports GitOps deployed policies
• Applicable to entire hardware/software stack
• Provides views across clusters and applications using the multicluster observability feature
• Supports an active upstream community for collaborative development of policies in the Policy Collection Repo
• Supports third-party Policy Enforcement Points (PEPs) such as SysDig/Falco, Synopsys Black Duck
• Provides out-of-the-box policies for security, resiliency, and software engineering controls that are not provided by the Compliance Operator
• Supports Red Hat, IBM, and third-party community contributions

RHACM uses its policy framework to ensure your Kubernetes clusters are configured in the desired state to meet internal enterprise and external technical regulatory compliance standards. If a managed cluster violates a defined policy, an alert is sent to the RHACM console. An example aspect of configuration that can be managed is ensuring that various operators that deliver capabilities are deployed and running on a fleet of clusters such as the OpenShift Compliance Operator, the Red Hat OpenShift Container Security Operator, and OPA Gatekeeper operator.

To further understand the governance mechanism and concepts, refer to the following sources:

• Go through the Red Hat Advanced Cluster Management for Kubernetes Governance documentation.
• Read the Comply to standards using policy based governance of Red Hat Advanced Cluster Management for Kubernetes and Implement Policy-based Governance Using Configuration Management of Red Hat Advanced Cluster Management for Kubernetes blogs.

In this part of the blog series, I went through the main ideas and tools that support compliance in multicluster OpenShift deployments. I covered the security challenges in multicluster OpenShift implementations, and the tools that help you achieve compliance with certain NIST SP 800-53 technical controls. In the next chapter of this multi-blog series, I plan to provide you with a practical demonstration of the tools that come with OpenShift Platform Plus, to achieve compliance with certain NIST SP 800-53 security controls in a multicluster OpenShift environment.