This is the last installment in our four-part OpenShift security blog series. Don’t forget to check out our previous blog posts in the series:
Part 1 - OpenShift security best practices for designing clusters
Part 2 - OpenShift networking and cluster access best practices
Part 3 - OpenShift runtime security best practices
Read this blog post to learn about security best practices when building container images in Red Hat OpenShift environments. We will also cover field-tested steps you should take to monitor and maintain OpenShift cluster health and security.
Build Secure Images
Following a few best practices for building secure container images will minimize running containers’ exploitability and simplify security updates. Images containing only the files required for the application’s runtime make it much more difficult for malicious attacks to compromise or exploit the containers in your cluster. Avoid using container images with frequently exploited or vulnerable tools like Linux package managers, web or other network clients, or Unix shells.
What to do:
- Use a minimal, secure base image.
- Only install tools needed for the container’s application. Debugging tools should be omitted from production containers.
- Remove exploitable tools as part of the initialization process with init-containers.
- Remove the package manager from the image as a Dockerfile build step.
- Keep images up-to-date. This practice includes watching for new versions of both the base image and any third-party tools you install.
- Maintain compatibility with proper tags. Avoid the use of the “latest” tag to help downstream users avoid compatibility issues.
Manage Images with a Registry
OpenShift offers an integrated image registry located on the infrastructure nodes of the cluster. This setup allows an organization to avoid third-party hosting and public image storing services, such as Docker Hub. By keeping all necessary images within the cluster, organizations can avoid dependency on a third-party service and any outages associated with them.
Organizations can enact more tuned policies for image use and vetting by taking advantage of a local image repository. Only after an image has passed the proper controls should it be allowed to be used to deploy containers into your cluster.
Use Verified Images from the Container Catalog
To help select secure base images, Red Hat offers a container image catalog. Images in the catalog are maintained by Red Hat, meaning they are frequently scanned and updated. The container catalog also provides an image’s history; every time it is updated, a new tag is created. According to a simple scale A to F, the image is graded based on creation date and known security flaws.
Use an Image Scanner
Using containers free of known software security vulnerabilities requires ongoing vigilance. All the images deployed to a cluster should be scanned regularly by an image scanner that keeps an up-to-date database of CVEs (Common Vulnerabilities and Exposure). Several open-source and proprietary options exist, but be sure to choose a solution that scans for vulnerabilities in operating system packages and in third-party runtime libraries for the programming languages your applications use.
To address CVEs when they are found in your internally maintained images, your organization should have policies for updating and replacing images known to have serious, fixable vulnerabilities for images that are already deployed. Image scanning should be part of your CI/CD pipeline process, and images with high-severity, fixable CVEs should generate an alert and fail a build.
OpenShift Container Security Operator
OpenShift allows for a Container Security Operator (CSO) to be installed in the cluster. This Operator will view results from the OpenShift Container Platform web console for container images used to launch running pods in the cluster. The CSO can be directed to look at all pods or specific namespaces. It will then query the container registry where the image came from for any vulnerabilities. The Operator will also access vulnerabilities based on how long the container has been used without an upgrade or patch.
The CSO’s findings can be viewed through the OCP dashboard that outlines the cluster’s overall health and specific images that require attention.
Monitoring & Maintenance
Keeping up-to-date with patches for your cluster’s OpenShift version is an important requirement for your cluster and workload security. OpenShift provides full support of the newest released versions until a month after the newest version is released. OpenShift will support the maintenance of the three most recent minor releases.
OpenShift Container Platform 4.5 offers the following upgrade channels:
The candidate-4.5 channel contains candidate builds for a z-stream (4.5.z) release. Release candidates contain all the features of the product but are not supported. Use release candidate versions to test feature acceptance and assist in qualifying the next OPC version. The fast-4.5 channel is updated with new 4.5 versions as soon as Red Hat declares the given version as a general availability release. These releases are fully supported, production quality, and have performed well while available as a release candidate in the candidate-4.5 channel from where they were promoted.
While the fast-4.5 channel contains releases as soon as their errata are published, releases are added to the stable-4.5 channel after a delay. Data is collected from Red Hat SRE teams, Red Hat support services, and pre-production and production environments that participate in a connected customer program about the release’s stability during this delay.
You can use the stable-4.5 channel to upgrade from a previous minor version of the OpenShift Container Platform.
OpenShift allows for multiple ways of upgrading a cluster. A minor upgrade can be made through the web console or through the CLI. Updating a restricted cluster will need to be done via the command line and will require administrative control and Internet access to pull the most recent OpenShift images.
Regardless of the method chosen, make sure to have a recent etcd backup in case of a failed upgrade. Also, ensure all Operators are upgraded to the latest version on their channel. This will ensure that the Operators will be valid for the upgraded cluster.
Logging events and changes at both the OpenShift level and the node level creates a critical audit trail to use for evaluating your cluster’s security, especially in case of a breach or attack. Audit logs are available by default and can be easily accessed through the command line. This is vital for restricted clusters that are not using the web interface to view cluster logs.
OpenShift allows administrators to view audit logs from the node and API directly from the command line. OpenShift uses Elasticsearch and Fluentd for logging and storing your cluster events. Make sure to take advantage of these managed open source tools to debug in your cluster effectively.