As the use of cloud-native applications expands, enterprises look to trusted partners and standards bodies for guidance on best practices for securing containers and Kubernetes in production. One such standards body is the National Institute of Standards and Technology (NIST). In the fall of 2017, NIST published Special Publication 800-190, Application Container Security Guide. NIST SP 800-190 is a great source of guidance on the elements of container security and security for container orchestration solutions, such as Red Hat OpenShift Container Platform.
OpenShift takes a layered approach to securing containers, integrating security throughout the container lifecycle, from building, to deploying to running containers in mission critical environments. One of the questions we’re often asked, is how OpenShift helps customers meet NIST 800-190 guidance for securing containerized applications.
To be truly effective, container and Kubernetes security needs to be automated and leverage the declarative nature of Kubernetes wherever possible. Automation requires the ability to codify guidance into technical controls. Controls related to the guidance in NIST SP 800-190 are included in NIST 800-53, and a reference list is provided in Appendix B of NIST SP800-190.
As part of Red Hat’s commitment to helping customers build, deploy and run their hybrid-cloud applications with their desired security posture, Red Hat offers a Compliance Operator that can be deployed on Red Hat OpenShift Container Platform clusters to audit for compliance with technical controls and to automatically remediate if needed. The Compliance Operator will support a number of frameworks, including the technical controls in NIST 800-53.
Currently, the Compliance Operator can audit for a subset of the NIST 800-53 controls applied at the Red Hat Enterprise Linux CoreOS layer of OpenShift. The next set of controls to be supported are inspired by the Center for Internet Security (CIS) Kubernetes benchmark v1.6.
Of course, the guidance in NIST SP 800-190 goes beyond technical controls. Red Hat solutions help customers meet the recommended best practices for securing the following elements of the container pipeline as well:
- External container registries
- Internal container registries
- Container orchestrator
- Container host
External Container Registries
The Red Hat Ecosystem Catalog is a source for trusted container images that customers need as building blocks for their custom-containerized applications. Red Hat provides signed images for the base operating system libraries needed to enable applications to run on Linux (RHEL and Universal base images) and a variety of images with runtime libraries such as Java and Python. Red Hat maintains these images, providing updated versions as new code is released, including fixes to newly discovered vulnerabilities. A Health grade is provided for each container image in the catalog providing information on the age of the individual images, as well as any known CVEs associated with the image. More information about Health index grades are available here: https://access.redhat.com/articles/2803031
Red Hat OpenShift Container Platform gives you the ability to allowlist / blocklist the registries from which container images can be pulled for deployment on the platform. Admission controllers can also be used to make sure that only images with valid signatures are deployed.
OpenShift also includes a feature called ImageStreams which can be used to monitor for changes to images pulled from external registries, making it easier to automate updating custom images that use those external images with the latest versions and fixes.
Internal Container Registries
Red Hat OpenShift Container Platform includes an integrated container registry for use with the platform.
Additionally, Red Hat Quay is an enterprise container registry available for use as a SaaS or on-premises private registry. Red Hat Quay capabilities include:
- Clair for vulnerability scanning
- Geographic replication
- Image rollback time machine
- Build image triggers
When used in combination with Red Hat OpenShift, the results of Clair vulnerability scans can be viewed in the OpenShift console for deployed container images.
Building Container Images
Red Hat OpenShift supports two different CI/CD pipelines: Jenkins and Tekton (tech preview). Both can be integrated with container security tools such vulnerability scanners and configuration analysis tools.
Red Hat CodeReady Workspaces is a collaborative Kubernetes-native development solution that delivers OpenShift workspaces and in-browser IDE for rapid cloud application development. Built on the open Eclipse Che project, Red Hat CodeReady Workspaces uses Kubernetes and containers to provide any member of the development or IT team with a consistent, more secure, and zero-configuration development environment. The user experience is as fast and familiar as an integrated development environment (IDE) on a laptop. CodeReady Workspaces also supports a developer’s own desktop IDE of choice. Container native security tools that integrate with the IDE can be used with Code Ready Workspaces.
Red Hat OpenShift Container Platform provides defense in depth for container orchestration.
OpenShift runs on Red Hat Enterprise Linux with SELinux in enforcing mode. OpenShift Security Context Constraints (implemented as a Kuberenetes admission controller) make sure that by default, no privileged containers can run on OpenShift worker nodes.
As mentioned above, OpenShift can be configured to use kubernetes admission controllers to blacklist / whitelist the registries from which container images can be pulled for deployment on the platform and checked to make sure that only images with valid signatures are deployed.
OpenShift includes a built-in OAuth server for token-based authentication of users and services. OpenShift supports integration with nine external identity providers. OpenShift platform components authenticate to each other via X.509 certificates. The CAs and certificates for platform components are managed by OpenShift.
OpenShift includes Role Based Access Control with out-of-the-box roles and the ability to create custom roles. Roles can have cluster-wide or project (see below) scope.
OpenShift projects (kube namespaces) make it easy to group users and applications with the same purpose, sensitivity, and threat posture. If desired, users can isolate applications on individual hosts by using Kubernetes taints and tolerations.
OpenShift supports Kubernetes network policies which can be used to control east-west network traffic to and from applications. OpenShift ingress and egress controls can be used to control north-south network traffic. OpenShift Service Mesh is available for additional security and management of network traffic between services.
OpenShift includes cluster monitoring and alerting, provided via Prometheus and Grafana. The ability to use the same monitoring and alerting stack for applications is currently in tech preview.
OpenShift provides event auditing and includes an optional logging stack which creates both cluster-level and application-level log files. Logging pipelines can be used to collect audit logs as well as cluster and application logs and forward these to a SIEM.
Red Hat Advanced Cluster Management, currently in tech preview, can be used to manage application deployment by policy, supporting use cases such as the requirement to deploy certain types of applications or data in specific regions.
RHEL CoreOS is a container-optimized operating system (OS). RHEL CoreOS is largely immutable and contains only those packages required to run OpenShift, thereby providing a minimized attack surface. Installing RPMs on RHCOS is not supported. The OS is built to run all processes outside the OS as a container. This enables successful upgrades and automation beyond what a traditional operating system can deliver.
- rpm-ostree is the technology used to assemble the operating system. RHEL RPMs are used to create the OS images, and versions can easily be queried using the rpm command.
- /usr is where the operating system binaries and libraries are stored and is read-only.
- /etc, /boot, /var are writable on the system but only intended to be altered by the Machine Config Operator.
- /var/lib/containers is the graph storage location for storing container images.
In OpenShift, RHEL CoreOS is managed with the Machine Config Operator, helping to make sure a uniform configuration of all hosts in the cluster. The Machine Config Operator manages the configuration for the following components:
- CRI-O (container runtime)
- Authorized registries
Updates to the host OS are managed in the same way as updates to OpenShift. The OpenShift Admin chooses when to apply updates, and the cluster applies them to the hosts in a rolling fashion. The hosts can be updated with zero application downtime for well-behaving applications.
There are many debugging tools provided to the cluster administration that can be used without the need to ssh into the node. Two, for example, are the must-gather tool and the oc debug command. For more information, see:
OpenShift and RHEL CoreOS take advantage of the following Linux features to protect container processes through a combination of the host OS configuration and the OpenShift Security Context Constraint admission controller.
- Linux namespaces provide the fundamentals of container isolation. A namespace makes it appear to the processes within the namespace that they have their own instance of global resources. Namespaces provide the abstraction that gives the impression you are running on your own operating system when you are inside a container.
- Mount namespaces isolate the set of file system mount points seen by a group of processes so that processes in different mount namespaces can have different views of the file system hierarchy. With mount namespaces, the mount() and umount() system calls cease to operate on a global set of mount points (visible to all processes) and instead perform operations that affect just the mount namespace associated with the container process. For example, each container can have its own /tmp or /var directory or even have an entirely different userspace.
- SELinux provides an additional layer of security to keep containers isolated from each other and from the host. SELinux allows administrators to enforce mandatory access controls (MAC) for every user, application, process, and file. SELinux is like a brick wall that will stop you if you manage to break out of (accidentally or on purpose) the namespace abstraction.
- Cgroups (control groups) limit, account for, and isolate the resource usage (for example, CPU, memory, disk I/O, network) of a collection of processes. Use Cgroups to make sure your container will not be stomped on by another container on the same host. Cgroups can also be used to control pseudo-devices—a popular attack vector.
- Linux capabilities can be used to lock down root in a container. Capabilities are distinct units of privilege that can be independently enabled or disabled. Capabilities allow you to do things such as send raw IP packets, or bind to ports below 1024. When running containers, you can drop multiple capabilities without impacting the vast majority of containerized applications.
- Finally, a secure computing mode (seccomp) profile can be associated with a container to restrict available system calls.
Auditing of the host can be done by enabling Auditd, and host logs can be collected and forwarded with the OpenShift logging pipeline feature.
RHEL CoreOS volumes can be encrypted using Network Bound Disk Encryption with TPM/vTPM (v2) and Tang endpoints for automatic decryption.
TLS 1.3 is the default in RHEL CoreOS.
RHEL CoreOS supports FIPS encryption. When RHEL CoreOS nodes are booted in FIPS mode, OpenShift platform components will call RHEL crypto libraries. The FIPS status for RHEL CoreOS cryptographic libraries is Modules in Process.
Container Runtime and Other Tools
OpenShift uses CRI-O for its container runtime. CRI-O allows containers to be run directly from Kubernetes without any unnecessary code or tooling. As long as the container is OCI-compliant, CRI-O can run it, cutting out extraneous tooling. CRI-O is optimized for and versioned with Kubernetes. When OpenShift needs to run a container, it calls CRI-O and the CRI-O daemon works with runc to start the container.
OpenShift also includes podman. Podman is a daemonless container engine for developing, managing, and running OCI Containers on Linux.
For more information, see https://www.redhat.com/en/blog/introducing-cri-o-10 and https://www.redhat.com/en/blog/why-red-hat-investing-cri-o-and-podman
Red Hat OpenShift Container Platform delivers a full-stack solution for those looking for enterprise-ready hybrid cloud solutions. Red Hat’s hybrid cloud portfolio provides a layered approach to security, helping customers to meet the guidance provided in NIST SP 800-90 and to maintain their desired security posture as they expand their adoption and deployment of cloud native solutions. For additional information see https://www.redhat.com/en/resources/layered-approach-security-detail.