In part 1 of this three-part blog series, I covered the main ideas and tools in OpenShift Plus that support compliance in multicluster OpenShift deployments. I went through security challenges in multicluster OpenShift implementations, and went through the tools that help achieve compliance with certain NIST SP 800-53 technical controls in such environments.
In this part, I cover a practical implementation of OpenShift Plus tools and policies that help with achieving compliance with certain NIST SP 800-53 security controls in multicluster environments.
What is NIST 800-53
As a reminder, NIST 800-53 is a risk management framework for federal information systems of the U.S. government. While software alone cannot fully address NIST 800-53 controls, software can be used to automate and monitor compliance with certain technical controls. NIST 800-53 controls are broken down into families as shown in the following table:
Source: NIST 800-53 Rev5
Demo environment description
In this section, learn how an organization can implement the policy-based governance framework to help make its environment compliant with NIST 800-53 standards, using Red Hat Advanced Cluster Management for Kubernetes (RHACM).
In order to demonstrate the governance policy framework in practice, the following components are needed for this demo environment:
-
Hub cluster - An OpenShift cluster where RHACM is deployed. The hub cluster is used to manage OpenShift clusters. It is used as a management portal for cluster provisioning, application deployment, and governance policy propagation. Version 2.3.1 of RHACM is deployed on the hub cluster for this demo environment.
-
Managed clusters - Two OpenShift clusters are connected to the RHACM hub cluster, and RHACM manages these clusters. The clusters are affected by policies that are defined in the RHACM administration portal. For this demo, the cluster names are
cluster-a
andcluster-b
. -
Application - A simple application is provisioned on the managed clusters. The application is used to demonstrate RHACM application security capabilities. Furthermore, use the application to understand how RHACM can improve software engineering practices in the organization. The name for the application is
mariadb-app
, and it is deployed in themariadb
namespace on both of the managed clusters.The resource in the application can be found in my forked rhacm-demo repository in GitLab, and it can be deployed by running the following command on the hub cluster:
<hub cluster> $ oc apply -f https://gitlab.com/michael.kot/rhacm-demo/-/raw/master/rhacm-resources/application.yml
View the following image of the application topology:
Governance basics
Before you go through NIST 800-53 security controls, it is important to understand the fundamentals of the governance mechanism in RHACM. In the following sections, learn about the resources that need to be created to deploy a functioning governance mechanism.
Namespace
In order to deploy the desired compliance policies, a namespace must be created on the hub cluster. The namespace for the compliance policies populates OpenShift custom resources (CRs), which are related to policy configuration. In this scenario, the name for the namespace is rhacm-policies
. Run the following command:
<hub cluster> $ oc apply -f https://raw.githubusercontent.com/michaelkotelnikov/rhacm-blog/master/namespace.yml
Policies
To enforce governance and consistency between clusters with RHACM, you need to configure policies. The policies conduct a certain rule set for Kubernetes resources. The policy declares a resource that is to be monitored on a managed cluster. For example, a policy can monitor the contents of a Kubernetes ConfigMap
resource. If the contents of the ConfigMap
resource change, the policy controller remediates the violation in one of two ways:
-
If the policy controller is set to
inform
, the controller registers the violation and passes it to the RHACM hub cluster. The hub cluster informs the security administrator in the organization about the violation, from the Governance dashboard. -
If the policy controller is set to
enforce
, the controller attempts to fix the violation (according to what is defined in the policy) by applying the desired state to the violating Kubernetes resource.
Each security control that is addressed in this article is associated with its own policy resource.
Placement rules
The placement rule aggregates a group of clusters by their label. In this example, I aggregate all clusters that contain the environment=dev
label. In this case, both cluster-a
and cluster-b
have the environment=dev
label. Run the following command to create the defined PlacementRule
resource in the rhacm-policies
namespace on the hub cluster:
<hub cluster> $ oc apply -f https://raw.githubusercontent.com/michaelkotelnikov/rhacm-blog/master/placementrule.yml
Placement bindings
For each policy that is created in this article, a PlacementBinding
resource is created as well. The PlacementBinding
resource binds between a PlacementRule
resource and a Policy
resource. Meaning that a policy is applied to all clusters that match the rule in the PlacementRule
resource. In our scenario, all created policies are applied to all managed clusters:
Comply to NIST 800-53 standards with the help of the governance framework
Before you begin: All files and resources used in the next demonstrations are present in my forked GitHub repository. To follow the demonstrations in the article, it is recommended to fork the mentioned repository.
Comply to the AC-3 (Access Enforcement) Security Control
The AC-3 security control states, "The information system enforces approved authorizations for logical access to information and system resources in accordance with applicable access control policies". In a large-scale environment, a user that has a specific role, should be able to access only a limited set of resources across all OpenShift environments. Define specific RBAC rules across all clusters in the environment to help the organization reach compliance with the AC-3 security control.
For example, a user named user1 is created. The user is a member of the group, mariadb-operators
. The purpose of the users in the group is to rollout deployments if the application pods fail. Therefore, user1 has limited access to resources in the application namespace on the managed clusters. Thereby, making the environment compliant with the AC-3 security control.
The goal of the scenario is to show how you can enforce RBAC rules on user1 (as a part of the mariadb-operators
group) across multiple OpenShift clusters. Thereby, making the environment compliant with the AC-3 security control.
For example, three Policy
resources are created in the rhacm-policies
namespace on the hub cluster. The Policy
resource defines the Kubernetes resources that are to be monitored by the policy controller. Read the following resource descriptions that are created and view the resource diagrams:
-
A
group-policy
is created in order to define themariadb-operators
group across all managed clusters. The deployed policy is configured with theenforce
remediation action, which means that the policy controller creates the defined resource if it is not available on the managed cluster. The policy defines aGroup
resource with themariadb-operators
name. TheGroup
resource has user1 assigned to it: -
A
role-policy
is created in order to define theRole
resource that is assigned to themariadb-operators
group. The deployed policy is configured with theenforce
remediation action. The policy is only effective on themariadb
namespace, which is the namespace where our application is deployed. The policy defines aRole
resource with thedc-rollout-role
name. The role policy allows access to the following different resources:deploymentconfigs
,replicationcontrollers
andpods
: -
A
role-binding-policy
is created to define aRoleBinding
resource on the managed clusters. TheRoleBinding
resource binds between the createddc-rollout-role
role and themariadb-operators
group. The policy remediation action is set toenforce
in themariadb
namespace on the managed clusters:
Let's use the GitOps approach to deploy the policies, as described in the Contributing and deploying community policies with Red Hat Advanced Cluster Management and GitOps blog post. Using the GitOps approach allows users to treat policies like source code. When you update policies in your Git repository, the policies are updated on your RHACM hub cluster as well:
-
To use the GitOps functionality, clone the policy-collection repository. Afterwards, use the
deploy/deploy.sh
script from the repository to deploy allPolicy
andPlacementBinding
resources that were mentioned previously. Deploy the resources by providing the repository you forked with your username as an argument to thedeploy.sh
script. Run the following commands:<hub-cluster> $ git clone https://github.com/open-cluster-management/policy-collection.git
<hub-cluster> $ cd policy-collection/deploy/
<hub-cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/AC-3 --branch master --name ac-3 --namespace rhacm-policiesAfter you create the resources, view the RHACM Governance dashboard. All of the clusters should be in a compliant state, with no violations since the remediation action is set to
enforce
in the policies: -
Verify that the
mariadb-operators
group has been created on the managed cluster, log in tocluster-b
as theadmin
user, and list the groups that are available. Run the following commands:<cluster-b> $ oc whoami
admin
<cluster-b> $ oc get group
NAME USERS
mariadb-operators user1Notice that the policy controller has successfully created the
mariadb-operators
Group resource on the managed cluster. -
Verify that the
Role
and theRoleBinding
resources were created, log in tocluster-b
as user1, and try to list the pods in themariadb
namespace. Afterwards, delete a pod and view the results:<cluster-b> $ oc whoami
user1
<cluster-b> $ oc get pods -n mariadb
NAME READY STATUS RESTARTS AGE
mariadb-1-deploy 0/1 Completed 0 18m
mariadb-1-s9zrl 1/1 Running 0 18m
<cluster-b> $ oc delete pod mariadb-1-s9zrl -n mariadb
Error from server (Forbidden): pods "mariadb-1-s9zrl" is forbidden: User "user1" cannot delete resource "pods" in API group "" in the namespace "mariadb"User1 is not allowed to delete pods, as declared in the
dc-rollout-role role
.When a policy is set to
enforce
, even a localadmin
user on the managed cluster can not change the state of the resource that is defined by the policy. Enforcing policies means that alignment is applied between clusters, making them compliant to security controls at all times. -
As a local
admin
user oncluster-b
, attempt to edit thedc-rollout-role
namespace. Add thedelete
verb to the pod resource, so user1 has access to delete pods:<cluster-b> $ oc whoami
admin
<cluster-b> $ oc edit role dc-rollout-role -n mariadb
…
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- delete
…
role.rbac.authorization.k8s.io/dc-rollout-role editedEven though the console states that the role has been edited, the policy controller enforces the original state of the resource (as stated in the policy). In order to change the role on the managed cluster, the policy that defines the
Role
resource needs to be modified on the hub cluster. You can list the role to verify that thedelete
verb has not been saved. Run the following command:<cluster-b> $ oc get role dc-rollout-role -o yaml -n mariadb
…
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
The cluster remains compliant to the security control defined by the organization.
If the security administrator of the organization changes the remediation action of the policy-role-mariadb-rollout
policy from enforce
to inform
, the policy controller no longer enforces specific values to the Role
resource on the managed clusters. Let's change the remediation action to inform
in order to validate the previous statement.
To change the remediation action, navigate to the GitHub repository that you forked in the beginning, and override the policies/AC-3/role-policy.yml
contents with this policy. Make sure to commit the changes to GitHub.
Now, if a local admin
changes the value of the defined dc-rollout-role
resource in the mariadb
namespace, the following changes take effect:
<cluster-b> $ oc whoami
admin
<cluster-b> $ oc edit role dc-rollout-role -n mariadb
…
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- delete
…
role.rbac.authorization.k8s.io/dc-rollout-role edited
<cluster-b> $ oc get role dc-rollout-role -o yaml -n mariadb
…
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- delete
For now, cluster-b
is not compliant with the RBAC policy that is defined. The policy controller on cluster-b
registers a violation and sends it to the hub cluster. The violation appears in the Governance dashboard, stating that there is one cluster (cluster-b
) that is not compliant with the AC-3 NIST 800-53 security control. View the following image of the console:
Comply to the SI-4(a.1) (Information System Monitoring) Security Control
The SI-4(a.1) security control states, "The organization monitors the information system to detect attacks and indicators of potential attacks in accordance with organization-defined monitoring objectives". In order to comply with the SI-4(a.1) security control, the organization needs to monitor the workload that runs on all clusters equally.
In this section, learn how to monitor application vulnerabilities on all OpenShift clusters with RHACM.
One way to enable distributed application monitoring is by enabling the container-security-operator
across all managed clusters. The operator provides visibility into images managed in Quay and scanned with Clair. If vulnerabilities are present in the image, the operator aggregates them and creates an ImageManifestVuln
object. With the use of the governance policy framework, you can monitor the ImageManifestVuln
objects, and initiate an alert in the Governance dashboard if they are present.
Therefore, if a cluster runs an application with an image that contains vulnerabilities, the cluster is considered non-compliant according to the security standards of the organization. In such a case, the security administrator must act according to the regulations defined by the organization. Complete the following steps:
-
In order to create the image scanning policy, add the following resources to the current setup:
-
A
policy-imagemanifestvuln
policy in therhacm-policies
namespace on your hub cluster. The policy monitors two object definitions --
The policy defines that clusters
musthave
thecontainer-security-operator
installed. The remediation action in this case is set toenforce
; so if the operator is not present on the clusters, the policy controller attempts to install it. -
The policy defines that clusters in the environment
mustnothave
resources with theImageManifestVuln
kind. The policy is used to monitor all projects besides the defaultopenshift-\*
projects. The remediation action is set toinform
, so RHACM console is informed by the policy controller, if there are anyImageManifestVuln
objects in the managed clusters application namespaces. Run the following command to deploy the policy:<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SI-4 --branch master --name si-4 --namespace rhacm-policies
-
-
-
After you create the resources, view the Governance dashboard. Note that both clusters are non-compliant to the defined policy:
-
Navigate to the
policy-imagemanifestvuln
status, and view details in the Governance dashboard to see that the violating namespace ismariadb
. This means that there is a vulnerability in the demo application that is deployed for the scenario. Note that the namespace contains anImageManifestVuln
object, even though amustnothave
compliance type is defined in the policy:You can obtain more information about the image vulnerabilities by clicking the View yaml button next to the violating
ImageManifestVuln
name: -
To further analyze the vulnerability, log into the local OpenShift console of
cluster-a
. In the main dashboard, notice that the Quay image scanning tool has found vulnerabilities in images used by containers in the cluster:In the following image, notice that Clair found vulnerabilities in the mariadb namespace:
-
See the reference to the image manifest vulnerability monitored by the policy controller by selecting the manifest name. When the manifest name is selected, you are redirected to Quay.io, and to a full scan report from Clair. Here, you can see the CVEs behind the vulnerabilities and whether there is a fix for them:
For now, both cluster-a
and cluster-b
are compliant to the SI-4(a.1) security control since both clusters are implementing a monitoring tool, according to the organization-defined monitoring objectives. But, it's important to note that both clusters are at risk since they are running an image that has high-level vulnerability associated with it.
Now, it's up to the security administrators and application development teams of the organization to remediate the violations in the Governance dashboard.
If you do not store your application images in Quay, you can use Red Hat Advanced Cluster Security to monitor images deployed to your Kubernetes cluster for known vulnerabilities. You can find more information from the What’s New in Red Hat Advanced Cluster Security blog.
Comply to the SC-6 (Resource Availability) Security Control
The SC-6 security control states, "The information system protects the availability of resources by allocating organization-defined resources by priority". In order to comply with the SC-6 security control, the organization needs to set equal quotas across all OpenShift clusters.
Setting quotas is necessary in order to prevent low priority projects in the OpenShift clusters from interfering with the work of high priority projects. Having an environment with uncontrolled resource allocation policies can result in resource exhaustion attacks, and denial of service.
In this section, I show you how to use RHACM to distribute equal project quotas across multiple OpenShift clusters. Thereby, making the environment compliant with the SC-6 security control.
A policy-limitrange policy is created to monitor and maintain a Kubernetes LimitRange
resource on managed clusters. The policy is deployed in the rhacm-policies
namespace on the hub cluster. Let's name the LimitRange
resource mariadb-limit-range
, where the remediation action is set to enforce
and a basic quota is created in the mariadb
namespace on the managed clusters. Run the following command to deploy the policy:
<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-6 --branch master --name sc-6 --namespace rhacm-policies
Now, the LimitRange
policy applies to both cluster-a
and cluster-b
. Notice in the following image that both clusters are compliant to the policy in the Governance dashboard:
Log in to cluster-a
to reveal that the LimitRange quota resource is in effect in the mariadb namespace:
<cluster-a> $ oc get limitrange -n mariadb
NAME CREATED AT
mariadb-limit-range 2021-01-31T13:57:03Z
<cluster-a> $ oc get limitrange mariadb-limit-range -o yaml -n mariadb
apiVersion: v1
kind: LimitRange
metadata:
creationTimestamp: "2021-01-31T13:57:03Z"
name: mariadb-limit-range
namespace: mariadb
resourceVersion: "82916"
selfLink: /api/v1/namespaces/mariadb/limitranges/mariadb-limit-range
uid: d9440c57-7ca3-4c6d-8c83-dfd49a0e652a
spec:
limits:
- default:
cpu: 500m
memory: 512Mi
defaultRequest:
cpu: 50m
memory: 256Mi
type: Container
Based on the previous output, the policy controller on cluster-a
created the defined limitrange
resource. The quota is now distributed and aligned in both clusters, thereby making the environment compliant with the SC-6 security control.
Comply to the SC-28 (1) (Cryptographic Protection) Security Control
The SC-28 (1) security control states, "The information system implements cryptographic mechanisms to prevent unauthorized disclosure and modification of organization-defined information on organization-defined information system components". This security control can be applied to different aspects of an organization's OpenShift fleet. Some of these aspects are built-in the OpenShift platform, like data transport (OpenShift uses cryptographic algorithms and ciphers to protect confidentiality and integrity of data in transport). Some are configurable by the user itself, like etcd encryption throughout all OpenShift clusters in the organization.
This section discusses how to encrypt the etcd database throughout all OpenShift clusters in the organization. Encrypting the etcd database throughout the clusters pushes the environment closer towards compliance with the SC-28 (1) security control.
-
In order to encrypt the etcd database on all OpenShift clusters, an encryption policy needs to be created. Let's create an
etcdencryption-policy
resource namedpolicy-etcdencryption
, and deploy it in therhacm-policies
namespace. The policy defines and monitors two objects:-
In the
APIServer
resource, theaescbc
encryption type is defined. The remediation action can be set toenforce
to force etcd encryption. -
In the
KubeAPIServer
resource, be sure to verify that etcd is encrypted by monitoring that theAll resources encrypted: secrets, configmaps
message is present in the resource status. When you run the following command for this case, the remediation action is set to inform:<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-28 --branch master --name sc-28 --namespace rhacm-policies
-
-
After applying the resources, it might take a couple of minutes for the encryption to complete. Log in to
cluster-a
, and note that the encryption is in progress:<cluster-a> $ oc get KubeAPIServer -o yaml
...
- lastTransitionTime: "2021-01-31T11:57:51Z"
message: Resource secrets is being encrypted
reason: EncryptionInProgress
status: "False"
type: Encrypted
... -
After the encryption is done, the message changes and notifies that all resources are encrypted, similar to the following output:
<cluster-a> $ oc get KubeAPIServer -o yaml
...
- lastTransitionTime: "2021-01-31T16:40:40Z"
message: 'All resources encrypted: secrets, configmaps'
reason: EncryptionCompleted
status: "True"
type: Encrypted
...
In the following image, the Governance dashboard now states that both cluster-a
and cluster-b
are compliant with the etcd encryption policy. Encrypting the etcd database is a major step into making your environment compliant with the SC-28 (1) security control:
Comply to the SC-8 (Transmission Confidentiality and Integrity) Security Control
The SC-8 control states, "The information system protects the confidentiality; integrity of transmitted information". As I already mentioned in this article, OpenShift communication is based on certificates. Certificates are used as an authentication mechanism and they protect the confidentiality of the data transmitted. Therefore, if the certificates expire, malfunctions can form in your OpenShift data transmission mechanism.
To keep the OpenShift clusters compliant with the SC-8 security control, certificates across OpenShift clusters must be monitored. RHACM is used to monitor the certificates and initiate an alert as soon as a certificate in the system is close to expiration.
To comply with the SC-8 security control, a certificate policy needs to be created. Let's create the cert-policy
resource named policy-cert
, and deploy it in the rhacm-policies
namespace. The policy is defined as a CertificatePolicy
in order to monitor certificates on the managed clusters. The remediation action is set to inform
to notify RHACM cluster when a certificate is expired.
Let's keep the demo simple and monitor one certificate, the certificate in the openshift-ingress
namespace. An alert is initiated if the certificate expires in less than 3000 hours. A policy that monitors all system certificates can be found in the community repository, policy-collection, named policy-ocp4-certs.yaml. Run the following command:
<hub cluster> $ ./deploy.sh --url https://github.com/<your-username>/rhacm-blog.git --path policies/SC-8 --branch master --name sc-8 --namespace rhacm-policies
If you take a look at the Governance dashboard, it shows that one cluster is non-compliant with the defined policy:
By drilling down into the policy, you can see that cluster-a
is non-compliant with the policy, and its ingress certificate is going to expire in less than 3000 hours:
Now, it is up to the security administrator in the environment to take action, in order to make cluster-a
compliant to the security standard in the organization. If the certificate ends up expiring, the cluster is no longer compliant with the SC-8 security control.
Conclusion
In this part of the blog series, I went through the basic components of the RHACM governance framework. I covered the resources that need to be deployed in order to get you started with the governance framework, and I showed how policies can be created to help achieve compliance with AC-3, SI-4(a.1), SC-6, SC-28(1) and SC-8 security controls. In the next chapter of this multi-blog series, I plan to provide more practical demonstrations of the tools that come with OpenShift Platform Plus, to achieve compliance with certain NIST SP 800-53 security controls in a multicluster OpenShift environment.
About the author
More like this
Browse by channel
Automation
The latest on IT automation that spans tech, teams, and environments
Artificial intelligence
Explore the platforms and partners building a faster path for AI
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
Explore how we reduce risks across environments and technologies
Edge computing
Updates on the solutions that simplify infrastructure at the edge
Infrastructure
Stay up to date on the world’s leading enterprise Linux platform
Applications
The latest on our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Developer resources
- Customer support
- Red Hat value calculator
- Red Hat Ecosystem Catalog
- Find a partner
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit