Managing NIST 800-53 Controls in a Multicluster OpenShift Environment - Part 2

In part 1 of this three-part blog series, I covered the main ideas and tools in OpenShift Plus that support compliance in multicluster OpenShift deployments. I went through security challenges in multicluster OpenShift implementations, and went through the tools that help achieve compliance with certain NIST SP 800-53 technical controls in such environments.

In this part, I cover a practical implementation of OpenShift Plus tools and policies that help with achieving compliance with certain NIST SP 800-53 security controls in multicluster environments.

What is NIST 800-53

As a reminder, NIST 800-53 is a risk management framework for federal information systems of the U.S. government. While software alone cannot fully address NIST 800-53 controls, software can be used to automate and monitor compliance with certain technical controls. NIST 800-53 controls are broken down into families as shown in the following table:

Source: NIST 800-53 Rev5

Demo environment description

In this section, learn how an organization can implement the policy-based governance framework to help make its environment compliant with NIST 800-53 standards, using Red Hat Advanced Cluster Management for Kubernetes (RHACM).

In order to demonstrate the governance policy framework in practice, the following components are needed for this demo environment:

• Hub cluster - An OpenShift cluster where RHACM is deployed. The hub cluster is used to manage OpenShift clusters. It is used as a management portal for cluster provisioning, application deployment, and governance policy propagation. Version 2.3.1 of RHACM is deployed on the hub cluster for this demo environment.

• Managed clusters - Two OpenShift clusters are connected to the RHACM hub cluster, and RHACM manages these clusters. The clusters are affected by policies that are defined in the RHACM administration portal. For this demo, the cluster names are cluster-a and cluster-b.

• Application - A simple application is provisioned on the managed clusters. The application is used to demonstrate RHACM application security capabilities. Furthermore, use the application to understand how RHACM can improve software engineering practices in the organization. The name for the application is mariadb-app, and it is deployed in the mariadb namespace on both of the managed clusters.

  The resource in the application can be found in my forked rhacm-demo repository in GitLab, and it can be deployed by running the following command on the hub cluster:

  <hub cluster> $ oc apply -f https://gitlab.com/michael.kot/rhacm-demo/-/raw/master/rhacm-resources/application.yml
  

  View the following image of the application topology:

Governance basics

Before you go through NIST 800-53 security controls, it is important to understand the fundamentals of the governance mechanism in RHACM. In the following sections, learn about the resources that need to be created to deploy a functioning governance mechanism.

Namespace

In order to deploy the desired compliance policies, a namespace must be created on the hub cluster. The namespace for the compliance policies populates OpenShift custom resources (CRs), which are related to policy configuration. In this scenario, the name for the namespace is rhacm-policies. Run the following command:

<hub cluster> $ oc apply -f https://raw.githubusercontent.com/michaelkotelnikov/rhacm-blog/master/namespace.yml

Policies

To enforce governance and consistency between clusters with RHACM, you need to configure policies. The policies conduct a certain rule set for Kubernetes resources. The policy declares a resource that is to be monitored on a managed cluster. For example, a policy can monitor the contents of a Kubernetes ConfigMap resource. If the contents of the ConfigMap resource change, the policy controller remediates the violation in one of two ways:

• If the policy controller is set to inform, the controller registers the violation and passes it to the RHACM hub cluster. The hub cluster informs the security administrator in the organization about the violation, from the Governance dashboard.

• If the policy controller is set to enforce, the controller attempts to fix the violation (according to what is defined in the policy) by applying the desired state to the violating Kubernetes resource.

Each security control that is addressed in this article is associated with its own policy resource.

Placement rules

The placement rule aggregates a group of clusters by their label. In this example, I aggregate all clusters that contain the environment=dev label. In this case, both cluster-a and cluster-b have the environment=dev label. Run the following command to create the defined PlacementRule resource in the rhacm-policies namespace on the hub cluster:

<hub cluster> $ oc apply -f https://raw.githubusercontent.com/michaelkotelnikov/rhacm-blog/master/placementrule.yml

Placement bindings

For each policy that is created in this article, a PlacementBinding resource is created as well. The PlacementBinding resource binds between a PlacementRule resource and a Policy resource. Meaning that a policy is applied to all clusters that match the rule in the PlacementRule resource. In our scenario, all created policies are applied to all managed clusters:

Comply to NIST 800-53 standards with the help of the governance framework

Before you begin: All files and resources used in the next demonstrations are present in my forked GitHub repository. To follow the demonstrations in the article, it is recommended to fork the mentioned repository.

Comply to the AC-3 (Access Enforcement) Security Control

The AC-3 security control states, "The information system enforces approved authorizations for logical access to information and system resources in accordance with applicable access control policies". In a large-scale environment, a user that has a specific role, should be able to access only a limited set of resources across all OpenShift environments. Define specific RBAC rules across all clusters in the environment to help the organization reach compliance with the AC-3 security control.

For example, a user named user1 is created. The user is a member of the group, mariadb-operators. The purpose of the users in the group is to rollout deployments if the application pods fail. Therefore, user1 has limited access to resources in the application namespace on the managed clusters. Thereby, making the environment compliant with the AC-3 security control.

The goal of the scenario is to show how you can enforce RBAC rules on user1 (as a part of the mariadb-operators group) across multiple OpenShift clusters. Thereby, making the environment compliant with the AC-3 security control.

For example, three Policy resources are created in the rhacm-policies namespace on the hub cluster. The Policy resource defines the Kubernetes resources that are to be monitored by the policy controller. Read the following resource descriptions that are created and view the resource diagrams:

• A group-policy is created in order to define the mariadb-operators group across all managed clusters. The deployed policy is configured with the enforce remediation action, which means that the policy controller creates the defined resource if it is not available on the managed cluster. The policy defines a Group resource with the mariadb-operators name. The Group resource has user1 assigned to it:

  

• A role-policy is created in order to define the Role resource that is assigned to the mariadb-operators group. The deployed policy is configured with the enforce remediation action. The policy is only effective on the mariadb namespace, which is the namespace where our application is deployed. The policy defines a Role resource with the dc-rollout-role name. The role policy allows access to the following different resources: deploymentconfigs, replicationcontrollers and pods:

  

• A role-binding-policy is created to define a RoleBinding resource on the managed clusters. The RoleBinding resource binds between the created dc-rollout-role role and the mariadb-operators group. The policy remediation action is set to enforce in the mariadb namespace on the managed clusters:

  

Let's use the GitOps approach to deploy the policies, as described in the Contributing and deploying community policies with Red Hat Advanced Cluster Management and GitOps blog post. Using the GitOps approach allows users to treat policies like source code. When you update policies in your Git repository, the policies are updated on your RHACM hub cluster as well:

1. To use the GitOps functionality, clone the policy-collection repository. Afterwards, use the deploy/deploy.sh script from the repository to deploy all Policy and PlacementBinding resources that were mentioned previously. Deploy the resources by providing the repository you forked with your username as an argument to the deploy.sh script. Run the following commands:

   <hub-cluster> $ git clone https://github.com/open-cluster-management/policy-collection.git
   
   <hub-cluster> $ cd policy-collection/deploy/
   
   <hub-cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/AC-3 --branch master --name ac-3 --namespace rhacm-policies
   

   After you create the resources, view the RHACM Governance dashboard. All of the clusters should be in a compliant state, with no violations since the remediation action is set to enforce in the policies:

   

2. Verify that the mariadb-operators group has been created on the managed cluster, log in to cluster-b as the admin user, and list the groups that are available. Run the following commands:

   <cluster-b> $ oc whoami
   admin
   
   <cluster-b> $ oc get group
   NAME            	USERS
   mariadb-operators   user1
   

   Notice that the policy controller has successfully created the mariadb-operators Group resource on the managed cluster.

3. Verify that the Role and the RoleBinding resources were created, log in to cluster-b as user1, and try to list the pods in the mariadb namespace. Afterwards, delete a pod and view the results:

   <cluster-b> $ oc whoami
   user1
   
   <cluster-b> $ oc get pods -n mariadb
   NAME           	READY   STATUS  	RESTARTS   AGE
   mariadb-1-deploy   0/1 	Completed   0      	18m
   mariadb-1-s9zrl	1/1 	Running 	0      	18m
   
   <cluster-b> $ oc delete pod mariadb-1-s9zrl -n mariadb
   Error from server (Forbidden): pods "mariadb-1-s9zrl" is forbidden: User "user1" cannot delete resource "pods" in API group "" in the namespace "mariadb"
   

   User1 is not allowed to delete pods, as declared in the dc-rollout-role role.

   When a policy is set to enforce, even a local admin user on the managed cluster can not change the state of the resource that is defined by the policy. Enforcing policies means that alignment is applied between clusters, making them compliant to security controls at all times.

4. As a local admin user on cluster-b, attempt to edit the dc-rollout-role namespace. Add the delete verb to the pod resource, so user1 has access to delete pods:

   <cluster-b> $ oc whoami
   admin
   <cluster-b> $ oc edit role dc-rollout-role -n mariadb
   …
   - apiGroups:
     - ""
     resources:
     - pods
     verbs:
     - get
     - list
     - watch
     - delete
   …
   role.rbac.authorization.k8s.io/dc-rollout-role edited
   

   Even though the console states that the role has been edited, the policy controller enforces the original state of the resource (as stated in the policy). In order to change the role on the managed cluster, the policy that defines the Role resource needs to be modified on the hub cluster. You can list the role to verify that the delete verb has not been saved. Run the following command:

   <cluster-b> $ oc get role dc-rollout-role -o yaml -n mariadb
   …
   - apiGroups:
     - ""
     resources:
     - pods
     verbs:
     - get
     - list
     - watch
   

The cluster remains compliant to the security control defined by the organization.

If the security administrator of the organization changes the remediation action of the policy-role-mariadb-rollout policy from enforce to inform, the policy controller no longer enforces specific values to the Role resource on the managed clusters. Let's change the remediation action to inform in order to validate the previous statement.

To change the remediation action, navigate to the GitHub repository that you forked in the beginning, and override the policies/AC-3/role-policy.yml contents with this policy. Make sure to commit the changes to GitHub.

Now, if a local admin changes the value of the defined dc-rollout-role resource in the mariadb namespace, the following changes take effect:

<cluster-b> $ oc whoami
admin
<cluster-b> $ oc edit role dc-rollout-role -n mariadb
…
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
  - delete
…
role.rbac.authorization.k8s.io/dc-rollout-role edited

<cluster-b> $ oc get role dc-rollout-role -o yaml -n mariadb

…
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch
  - delete

For now, cluster-b is not compliant with the RBAC policy that is defined. The policy controller on cluster-b registers a violation and sends it to the hub cluster. The violation appears in the Governance dashboard, stating that there is one cluster (cluster-b) that is not compliant with the AC-3 NIST 800-53 security control. View the following image of the console:

Comply to the SI-4(a.1) (Information System Monitoring) Security Control

The SI-4(a.1) security control states, "The organization monitors the information system to detect attacks and indicators of potential attacks in accordance with organization-defined monitoring objectives". In order to comply with the SI-4(a.1) security control, the organization needs to monitor the workload that runs on all clusters equally.

In this section, learn how to monitor application vulnerabilities on all OpenShift clusters with RHACM.

One way to enable distributed application monitoring is by enabling the container-security-operator across all managed clusters. The operator provides visibility into images managed in Quay and scanned with Clair. If vulnerabilities are present in the image, the operator aggregates them and creates an ImageManifestVuln object. With the use of the governance policy framework, you can monitor the ImageManifestVuln objects, and initiate an alert in the Governance dashboard if they are present.

Therefore, if a cluster runs an application with an image that contains vulnerabilities, the cluster is considered non-compliant according to the security standards of the organization. In such a case, the security administrator must act according to the regulations defined by the organization. Complete the following steps:

1. In order to create the image scanning policy, add the following resources to the current setup:

   • A policy-imagemanifestvuln policy in the rhacm-policies namespace on your hub cluster. The policy monitors two object definitions -

     • The policy defines that clusters musthave the container-security-operator installed. The remediation action in this case is set to enforce; so if the operator is not present on the clusters, the policy controller attempts to install it.

     • The policy defines that clusters in the environment mustnothave resources with the ImageManifestVuln kind. The policy is used to monitor all projects besides the default openshift-\* projects. The remediation action is set to inform, so RHACM console is informed by the policy controller, if there are any ImageManifestVuln objects in the managed clusters application namespaces. Run the following command to deploy the policy:

       <hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SI-4 --branch master --name si-4 --namespace rhacm-policies
       

2. After you create the resources, view the Governance dashboard. Note that both clusters are non-compliant to the defined policy:

   

3. Navigate to the policy-imagemanifestvuln status, and view details in the Governance dashboard to see that the violating namespace is mariadb. This means that there is a vulnerability in the demo application that is deployed for the scenario. Note that the namespace contains an ImageManifestVuln object, even though a mustnothave compliance type is defined in the policy:

   

   You can obtain more information about the image vulnerabilities by clicking the View yaml button next to the violating ImageManifestVuln name:

   

4. To further analyze the vulnerability, log into the local OpenShift console of cluster-a. In the main dashboard, notice that the Quay image scanning tool has found vulnerabilities in images used by containers in the cluster:

   In the following image, notice that Clair found vulnerabilities in the mariadb namespace:

   

5. See the reference to the image manifest vulnerability monitored by the policy controller by selecting the manifest name. When the manifest name is selected, you are redirected to Quay.io, and to a full scan report from Clair. Here, you can see the CVEs behind the vulnerabilities and whether there is a fix for them:

   

For now, both cluster-a and cluster-b are compliant to the SI-4(a.1) security control since both clusters are implementing a monitoring tool, according to the organization-defined monitoring objectives. But, it's important to note that both clusters are at risk since they are running an image that has high-level vulnerability associated with it.

Now, it's up to the security administrators and application development teams of the organization to remediate the violations in the Governance dashboard.

If you do not store your application images in Quay, you can use Red Hat Advanced Cluster Security to monitor images deployed to your Kubernetes cluster for known vulnerabilities. You can find more information from the What’s New in Red Hat Advanced Cluster Security blog.

Comply to the SC-6 (Resource Availability) Security Control

The SC-6 security control states, "The information system protects the availability of resources by allocating organization-defined resources by priority". In order to comply with the SC-6 security control, the organization needs to set equal quotas across all OpenShift clusters.

Setting quotas is necessary in order to prevent low priority projects in the OpenShift clusters from interfering with the work of high priority projects. Having an environment with uncontrolled resource allocation policies can result in resource exhaustion attacks, and denial of service.

In this section, I show you how to use RHACM to distribute equal project quotas across multiple OpenShift clusters. Thereby, making the environment compliant with the SC-6 security control.

A policy-limitrange policy is created to monitor and maintain a Kubernetes LimitRange resource on managed clusters. The policy is deployed in the rhacm-policies namespace on the hub cluster. Let's name the LimitRange resource mariadb-limit-range, where the remediation action is set to enforce and a basic quota is created in the mariadb namespace on the managed clusters. Run the following command to deploy the policy:

<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-6 --branch master --name sc-6 --namespace rhacm-policies

Now, the LimitRange policy applies to both cluster-a and cluster-b. Notice in the following image that both clusters are compliant to the policy in the Governance dashboard:

Log in to cluster-a to reveal that the LimitRange quota resource is in effect in the mariadb namespace:

<cluster-a> $ oc get limitrange -n mariadb
NAME              	CREATED AT
mariadb-limit-range   2021-01-31T13:57:03Z

<cluster-a> $ oc get limitrange mariadb-limit-range -o yaml -n mariadb
apiVersion: v1
kind: LimitRange
metadata:
  creationTimestamp: "2021-01-31T13:57:03Z"
  name: mariadb-limit-range
  namespace: mariadb
  resourceVersion: "82916"
  selfLink: /api/v1/namespaces/mariadb/limitranges/mariadb-limit-range
  uid: d9440c57-7ca3-4c6d-8c83-dfd49a0e652a
spec:
  limits:
  - default:
	    cpu: 500m
	    memory: 512Mi
      defaultRequest:
	    cpu: 50m
	    memory: 256Mi
      type: Container

Based on the previous output, the policy controller on cluster-a created the defined limitrange resource. The quota is now distributed and aligned in both clusters, thereby making the environment compliant with the SC-6 security control.

Comply to the SC-28 (1) (Cryptographic Protection) Security Control

The SC-28 (1) security control states, "The information system implements cryptographic mechanisms to prevent unauthorized disclosure and modification of organization-defined information on organization-defined information system components". This security control can be applied to different aspects of an organization's OpenShift fleet. Some of these aspects are built-in the OpenShift platform, like data transport (OpenShift uses cryptographic algorithms and ciphers to protect confidentiality and integrity of data in transport). Some are configurable by the user itself, like etcd encryption throughout all OpenShift clusters in the organization.

This section discusses how to encrypt the etcd database throughout all OpenShift clusters in the organization. Encrypting the etcd database throughout the clusters pushes the environment closer towards compliance with the SC-28 (1) security control.

1. In order to encrypt the etcd database on all OpenShift clusters, an encryption policy needs to be created. Let's create an etcdencryption-policy resource named policy-etcdencryption, and deploy it in the rhacm-policies namespace. The policy defines and monitors two objects:

   • In the APIServer resource, the aescbc encryption type is defined. The remediation action can be set to enforce to force etcd encryption.

   • In the KubeAPIServer resource, be sure to verify that etcd is encrypted by monitoring that the All resources encrypted: secrets, configmaps message is present in the resource status. When you run the following command for this case, the remediation action is set to inform:

     <hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-28 --branch master --name sc-28 --namespace rhacm-policies
     

2. After applying the resources, it might take a couple of minutes for the encryption to complete. Log in to cluster-a, and note that the encryption is in progress:

   <cluster-a> $ oc get  KubeAPIServer -o yaml
   ...	
         - lastTransitionTime: "2021-01-31T11:57:51Z"
           message: Resource secrets is being encrypted
           reason: EncryptionInProgress
           status: "False"
           type: Encrypted
   ...
   

3. After the encryption is done, the message changes and notifies that all resources are encrypted, similar to the following output:

   <cluster-a> $ oc get  KubeAPIServer -o yaml
   ...	
         - lastTransitionTime: "2021-01-31T16:40:40Z"
           message: 'All resources encrypted: secrets, configmaps'
           reason: EncryptionCompleted
           status: "True"
           type: Encrypted
   ...
   

In the following image, the Governance dashboard now states that both cluster-a and cluster-b are compliant with the etcd encryption policy. Encrypting the etcd database is a major step into making your environment compliant with the SC-28 (1) security control:

Comply to the SC-8 (Transmission Confidentiality and Integrity) Security Control

The SC-8 control states, "The information system protects the confidentiality; integrity of transmitted information". As I already mentioned in this article, OpenShift communication is based on certificates. Certificates are used as an authentication mechanism and they protect the confidentiality of the data transmitted. Therefore, if the certificates expire, malfunctions can form in your OpenShift data transmission mechanism.

To keep the OpenShift clusters compliant with the SC-8 security control, certificates across OpenShift clusters must be monitored. RHACM is used to monitor the certificates and initiate an alert as soon as a certificate in the system is close to expiration.

To comply with the SC-8 security control, a certificate policy needs to be created. Let's create the cert-policy resource named policy-cert, and deploy it in the rhacm-policies namespace. The policy is defined as a CertificatePolicy in order to monitor certificates on the managed clusters. The remediation action is set to inform to notify RHACM cluster when a certificate is expired.

Let's keep the demo simple and monitor one certificate, the certificate in the openshift-ingress namespace. An alert is initiated if the certificate expires in less than 3000 hours. A policy that monitors all system certificates can be found in the community repository, policy-collection, named policy-ocp4-certs.yaml. Run the following command:

<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-8 --branch master --name sc-8 --namespace rhacm-policies

If you take a look at the Governance dashboard, it shows that one cluster is non-compliant with the defined policy:

By drilling down into the policy, you can see that cluster-a is non-compliant with the policy, and its ingress certificate is going to expire in less than 3000 hours:

Now, it is up to the security administrator in the environment to take action, in order to make cluster-a compliant to the security standard in the organization. If the certificate ends up expiring, the cluster is no longer compliant with the SC-8 security control.

Conclusion

In this part of the blog series, I went through the basic components of the RHACM governance framework. I covered the resources that need to be deployed in order to get you started with the governance framework, and I showed how policies can be created to help achieve compliance with AC-3, SI-4(a.1), SC-6, SC-28(1) and SC-8 security controls. In the next chapter of this multi-blog series, I plan to provide more practical demonstrations of the tools that come with OpenShift Platform Plus, to achieve compliance with certain NIST SP 800-53 security controls in a multicluster OpenShift environment.