What is Log4Shell?

Last week, one of the most critical 0-day vulnerabilities in several years was made public. This issue was found in the commonly used Java logging utility, Apache Log4j, version 2, which could allow remote code execution on a vulnerable system. The vulnerability is in Log4j’s use of the Java Naming and Directory Interface™ (JNDI) Lightweight Directory Access Protocol (LDAP) server lookup functionality. If exploited, a sophisticated, unauthenticated remote attacker can execute arbitrary commands that could lead to a system compromise. Log4j versions between 2.0 and 2.14.1 are impacted by this issue.

In order for the issue to be exploitable, the impacted service must meet the following criteria:

  • A remotely accessible endpoint with any protocol (e.g., HTTP, TCP) that allows an attacker to send arbitrary data
  • A log statement in the endpoint that logs the untrusted data controlled by an attacker

Red Hat has provided guidance about the impact on Red Hat products in Security Bulletin RHSB-2021-009 and in the Red Hat CVE database CVE-2021-44228. Red Hat strongly recommends applying updates as soon as errata becomes available and deploying the mitigation until updates have been applied for customers running affected versions.

If you're using an upstream version of Log4j, an initial patch is available in Log4j version 2.15.0. This patch does not fully address the vulnerability. A more recent patch is available in version 2.16.0 and disables the vulnerable application function by default.

How to identify if you're impacted

Red Hat Advanced Cluster Security for Kubernetes, powered by StackRox technology, protects your vital applications across build, deploy, and runtime as part of our layered approach to securing containers and Kubernetes. Using Advanced Cluster Security for Kubernetes, organizations can identify container images with the vulnerable Log4j software package to facilitate remediation efforts in their Kubernetes environments. Advanced Cluster Security for Kubernetes may be used to identify components in Java manifest files, as well as software packages from other package managers. Advanced Cluster Security for Kubernetes uses feeds from the National Vulnerability Database (NVD) and from Red Hat’s Product Security team to identify vulnerable software packages. These feeds are updated hourly.

When Log4j is not installed through a package manager, Advanced Cluster Security for Kubernetes identifies Java language level vulnerabilities using a JAR’s manifest file. The scanner will evaluate JAR files in container images scanned and in doing so also scan dependent libraries to evaluate the set of libraries used by the application inside the container for languages such as java, javaScript, python, and ruby, as explained here.

Using Red Hat Advanced Cluster Security for Kubernetes to address urgent vulnerabilities

Organizations can use Red Hat Advanced Cluster Security for Kubernetes to technically enforce container security best practices through policy across supported Kubernetes distributions such as Red Hat OpenShift, Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS).

Customers already using Red Hat Advanced Cluster Security for Kubernetes can use the following paths to reduce and manage the deployment of vulnerable services in their environments.

  1. Identify all existing impacted container images and deployments in their OpenShift, EKS, GKE, and AKS clusters. 
  2. Set a policy to alert on or prevent vulnerable deployments from being deployed within their environments and set notifiers to alert the security team when a specified policy is violated.
  3. Prioritize remediation efforts for mission-critical applications, externally exposed services, and containers with additional host capabilities. 

Identify all existing impacted container images and deploy your Kubernetes environment

To search for all impacted images:

  1. In vulnerability management, on the top right select the images in your environment.
  2. In the search bar, search for the CVE CVE-2021-44228.
  3. Advanced Cluster Security for Kubernetes will show users the images running in your environment that contain the impacted Log4j component.
  4. Vulnerability management teams may use this information to contact the image owners or Independent software vendors to ensure that the image is updated with the latest minimum version of Log4j. Software delivery teams may use Advanced Cluster Security for Kubernetes to remediate.

To search for all impacted deployments:

  1. In vulnerability management, on the top right select the Applications & Infrastructure, and then select Deployments.
  2. In the search bar, search for the CVE CVE-2021-44228.
  3. Advanced Cluster Security for Kubernetes will show users the deployments running in your environment that contain a container image using the impacted Log4j component.
  4. Vulnerability management teams may use this information to contact the deployment owners or Independent software vendors to remediate this issue. By requiring deployment labels with the owning software delivery team on each deployment, organizations can accelerate remediation efforts by contacting deployment owners listed in the labels of the deployment ​​details and metadata.

Policy alerting and prevention with scoped notifiers 

Organizations using Red Hat Advanced Cluster Security for Kubernetes may set organizational policies to identify all impacted deployments and any newly scheduled vulnerable deployments from entering their Kubernetes environments.

To set a system policy to alert on impacted deployments entering the environment, use the following instructions or import the linked policy:

  1. Navigate to Platform Configuration > System Policies.
  2. Create a custom policy by selecting the + sign in the top-right corner of the screen to create a new policy.
  3. Give the policy a human readable name, such as “Log4Shell.”
  4. Assign the policy a critical severity.
  5. Apply the policy to the build and deploy lifecycle stages.
  6. Give the policy a description, such as “Identify and alert on deployments with the Log4Shell vulnerability.”
  7. Provide the policy rationale, such as “Reduce the risk of a well-known critical vulnerability being exploited in the wild, Log4Shell.”
  8. Provide remediation, such as “Upgrade affected components." For updates to Red Hat components, check here. If you're using an upstream version of Log4j, a patch is available in version 2.16.0 and disables the vulnerable application function by default.
  9. Assign the system category “Vulnerability Management.”
  10. Assign a notifier integration to the policy. This allows teams to configure notifications to the appropriate team through team and system notifiers, such as Microsoft teams, Slack, e-mail, or a SEIM to implement tight feedback loops to triage issues when a policy violation occurs. For more information on setting up notifiers, visit the Red Hat Advanced Cluster Security for Kubernetes documentation.
  11. Apply an exception for deployment labels or predefined namespaces for deployments where the risk of this vulnerability may be accepted and tracked.
  12. Click Next at the top-right side of the screen.
  13. Select the CVE policy criteria under Image Contents, and drag it into the policy criteria and enter CVE-2021-44228.

 

  1. Review existing deployments that may be impacted, and consider putting in an exception for existing deployments that are vulnerable and managing those separately until a patch is available.

 

 

  1. Select if the policy will be enforced or alert. If the policy is set to enforcement mode, development team builds using Red Hat Advanced Cluster Security for Kubernetes in continuous integration workflows will be broken and admission controllers will block the deployment from being scheduled. Otherwise, users will be alerted. 
  2. Finalize the policy by clicking next and reviewing the finalized policy.

Note: To prevent images with vulnerabilities from being deployed, the admission controller must be configured to scan images inline as they are deployed. To do this, users must set the parameter admissionControl.contactImageScanners in their operator configuration or admissionControl.dynamic.scanInline in their helm chart

Testing your policy

If you'd like to test your policy, use a container image known to have vulnerable software dependencies, such as elastic/logstash:7.13.3.

Run the following command to deploy a vulnerable image without exposure in a test or sandbox environment. This should be run in a standalone namespace with an implicit deny network policy.

Using the OpenShift oc client:

Create a namespace for your log4shell test pod:

oc create namespace log4shell

Apply an implicit ingress and egress deny network policy to the namespace by copying the following network policy to a file called logshellnetpol.yml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: logshellnetpol
 namespace: log4shell
spec:
 podSelector: {}
 policyTypes:
   - Ingress
   - Egress
 ingress: []
 egress: []

Apply the network policy with the following command:

oc apply -f ./logshellnetpol.yml

Run a known vulnerable pod to test your policy and check for the alert for enforcement in the Red Hat Advanced Cluster Security for Kubernetes violations.

oc run log4shell -n log4shell --image=elastic/logstash:7.13.0

Using Kubectl:

Create a namespace for your log4shell test pod:

kubectl create namespace log4shell

Apply an implicit ingress and egress deny network policy to the namespace by copying the following network policy to a file called logshellnetpol.yml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: logshellnetpol
 namespace: log4shell
spec:
 podSelector: {}
 policyTypes:
   - Ingress
   - Egress
 ingress: []
 egress: []

Apply the network policy with the following command:

kubectl apply -f ./logshellnetpol.yml

Run a known vulnerable pod to test your policy and check for the alert for enforcement in the RHACS violations..

kubectl run log4shell -n log4shell --image=elastic/logstash:7.13.0

Prioritize externally facing deployments and mission-critical services first

Containers that run as root, privileged containers or containers that have additional host-level capabilities pose an increased risk of lateral movement and privilege escalation. When they're vulnerable to a remote code execution vulnerability, check all known deployments that contain vulnerable versions of Log4j for privileges and prioritize those alongside any externally exposed deployments as a top priority. Organizations can also add policy criteria to identify deployments running as privileged or with external exposures.

Example policy to identify services vulnerable to Log4Shell with external exposures to the cluster.

Example policy to identify privileged containers vulnerable to Log4shell.

Compensating controls and best practices

Organizations can reduce the impact of exploitation by using a defense in-depth strategy.

Linux controls

OpenShift runs on Red Hat Enterprise Linux with SELinux on by default. OpenShift provides running applications the protection of SELinux mandatory access controls without those applications needing to understand SELinux. If a compromise occurs, the user on the operating system or container that runs the logging service will be the user executing commands on behalf of the attacker. With clearly defined and hardened access controls and SELinux policies, organizations can limit the commands and files that an attacker can execute to reduce the impact of a compromise.

Container controls and best practices

Because a successfully executed exploit of the Log4Shell vulnerability may run remote commands on behalf of the user running the logging service, it's important to embed and adopt container security best practices in your organization. By disabling the capability for containers to be run as root, minimizing privileged pods and minimizing host capabilities given to containers, organizations can reduce the impact of a remote code execution vulnerability by using the inherent isolation capabilities built into container services. These best practices apply to any Kubernetes-based environment.

The OpenShift Security Context Constraint (SCC) admission controller makes it easier to implement these best practices. OpenShift SCCs are configured to help enforce these practices by default. The restricted SCC is applied to all worker nodes, preventing privileged pods from being admitted to worker nodes by regular users. The restricted SCC also prevents containers from running as root and drops the Linux capabilities KILL,MKNOD,SYS_CHROOT,SETUID,SETGID. Kubernetes Pod Security Policies (soon to be replaced by Pod Security Standards) may also be used to implement similar controls across other Kubernetes distributions, including EKS, GKE, and AKS.

Identify known bad processes and alert on their usage

As a compensating control, organizations can use Red Hat Advanced Cluster Security for Kubernetes policies at runtime to detect known bad processes running in their containers. Common attack patterns and known risky processes such as Nmap, runtime package manager execution and cryptominers are default policies in Advanced Cluster Security for Kubernetes that can help organizations detect potential attacks as a result of Log4Shell exploitation.

Identify deviations from known good process activity

Red Hat Advanced Cluster Security for Kubernetes allows teams to define a set of known good processes for each deployment in your enterprise Kubernetes environment. Setting a process baseline for each deployment in your environment can allow teams to identify deviations from known good process activity and alert on these deviations for investigation. Pods can be automatically killed to technically enforce this baseline and reduce the risk of exploitation.

Shift security left

Patching all of your container applications requires rebuilding and redeploying your container images. Red Hat CodeReady Dependency Analytics plugins and Red Hat Advanced Cluster Security for Kubernetes vulnerability scanner can be used in your pipeline to help you shift security left to help prevent packages with known vulnerabilities from being used in the future.

Security teams can enable policies that alert or block pipeline builds based on vulnerable packages and application configuration. This automatic feedback encourages developers to make the necessary changes efficiently, without teams becoming delayed with communication and large investigation efforts.

Also, since security teams can more accurately assess what workloads are vulnerable, they can prioritize their efforts to triage the specific impacted cases. This keeps security teams focused and unimpacted developer teams working on their applications.

Contact Red Hat for help

Red Hat is a partner in our customers' Kubernetes security programs and container-adoption journeys. If you have any questions or concerns around Log4Shell or your Kubernetes security program, please reach out to our support team.