Red Hat SummitRed Hat blog
In 2013 The MITRE Corporation created the first MITRE ATT&CK model, primarily focusing on the Windows enterprise environment based on contributions from the cybersecurity community. MITRE refined the matrix, and an updated version was released in 2015 with 96 techniques organized under 9 tactics. MITRE recognized a significant need to organize attack vectors and techniques into a consumable checklist for security professionals. Today, the ATT&CK Matrix is widely used in organizations to ensure they are doing all they can to secure their environments.
In parallel with the adoption of the MITRE ATT&CK Matrix, there was an exponential growth in the adoption of container technologies. The adoption of Kubernetes and its use for business-critical operations drives new security challenges that must be appropriately understood and addressed. Kubernetes represents a unique set of challenges for enterprises when addressing its security use cases, and the MITRE ATT&CK framework is a great way to simplify, categorize, and provide security recommendations for the various ways that Kubernetes and its components can be exploited. Last year, Microsoft announced its version of the threat matrix for Kubernetes and containerized applications. It was a significant step towards helping enterprises understand the threat vectors and techniques used to compromise their environments. It was popular enough that Microsoft released an updated version of its threat matrix this year.
From the updated blog, Microsoft outlines a few of the changes from the previous version:
The updated matrix adds new techniques that were found by Microsoft researchers, as well as techniques that were suggested by the community. We also deprecate several techniques, which do not apply anymore to newer versions of Kubernetes. In this version, we also add a new tactic taken from MITRE ATT&CK®: collection.
We previously broke down the threat matrix into a series of nine blogs separated by each tactic (found below). Each blog takes a deep dive into each of the nine Kubernetes threat vectors across 40 attack techniques and includes guidance on security best practices to mitigate these threats. We also highlighted whether the security controls that should be primarily used and configured to adequately protect against these threats reside within (1) Kubernetes itself, (2) cloud provider platform services, or (3) other cloud-native tooling such as container image registries, package management systems, or others. The key takeaway of the blog series is that to comprehensively secure Kubernetes, organizations must protect both the underlying infrastructure systems and the components that directly interface with them.
- Part one - Initial Access
- Part two - Execution
- Part three - Persistence
- Part four - Privilege Escalation
- Part five - Defense Evasion
- Part six - Credential Access
- Part seven - Discovery
- Part eight - Lateral Movement
- Part nine - Impact
This article will review the updated matrix, discuss the deprecated and new techniques, and where applicable, outline how Red Hat Advanced Cluster Security (RHACS) protects organizations from the threats that make up the Kubernetes attack matrix.
First, we will start with the deprecated techniques.
Deprecated Techniques
1. Initial Access - Exposed Kubernetes Dashboard
Issue
In 2018, a Kubernetes dashboard vulnerability allowed attackers to bypass authentication and use Dashboard's Service Account for reading secrets within the cluster. Large and small organizations alike experienced attacks due to access management control failures associated with their Kubernetes dashboards. Organizations were generally pushed to create controls around their exposed dashboards with RBAC controls, network policies, and policy enforcement with tools such as RHACS.
Various dashboards can be unnecessarily exposed in Kubernetes, so calling out the Kubernetes dashboard is too specific. This technique has been deprecated in favor of calling out all sensitive interfaces, as seen below.
Best Practice for Mitigation
The primary area to configure security controls: Kubernetes
In the first blog in the series, we recommended removing all dashboards that are unnecessary within your cluster. If a dashboard is necessary, it is best practice to limit the RBAC permission using a principle of least privilege and use network policies and ingress rules to ensure that the dashboards are not publicly accessible.
How RHACS Helps
Although deprecated, exposed dashboards will always be a persistent threat vector. RHACS provides a built-in policy to alert when the Kubernetes dashboard is deployed. It also automatically monitors RBAC privileges on service accounts and can identify whether elevated privileges have been granted to the dashboard. It can also ensure incoming traffic to the dashboard is blocked by configuring network policies.
2. Lateral Movement - Access Tiller Endpoint
Issue
Helm is a widely adopted package manager for Kubernetes. Version 2 of Helm had a server-side component named Tiller that operated with admin credentials within Kubernetes clusters. This component creates a significant risk as actors could leverage Tiller's permissions to gain admin privileges in the cluster.
As of version 3, Helm does not use its server-side component, Tiller, but will use the user's credentials to submit Kubernetes objects to the API server.
Best Practice for Mitigation
The primary area to configure security controls: Other - Helm
Organizations can mitigate this threat by ensuring that they upgrade to the latest version of Helm for deploying their Kubernetes applications.
How RHACS Helps
RHACS uses three deployment methods.
All three methods rely on user credentials, and RHACS itself will identify and escalate applications using administrative cluster access.
New Techniques for the Threat Matrix
1. Initial Access - Exposed Sensitive Interfaces
Issue
As part of the deprecation of the exposed Kubernetes dashboard technique, the greater security threat has been consolidated to encompass all sensitive interfaces. As mentioned above, sensitive interfaces will always be a significant threat as any unnecessary exposure can yield an abundance of information and access to bad actors.
Best Practice for Mitigation
The primary area to configure security controls: Kubernetes
The mitigation techniques remain the same. Make sure you remove any interfaces that are not necessary. If a dashboard is required, ensure RBAC permissions using a principle of least privilege and use ingress network policies to ensure that the dashboards are not publicly accessible.
How RHACS Helps
RHACS provides a built-in policy to alert when the Kubernetes dashboard is deployed. It also automatically monitors RBAC privileges on service accounts and can identify whether elevated privileges have been granted to the dashboard. It can also ensure incoming traffic to the dashboard is blocked by configuring network policies. The Network Policy dashboard will inform you of allowed external connections, so you can immediately focus on any unnecessarily exposed applications.
2. Execution - Sidecar Injection
Issue
A Kubernetes pod is a group of one or more containers. These containers will have shared storage, network resources, and specifications on permissions and capabilities. All containers in a single pod will run on the same node, and the containers can be considered tightly coupled.
Sidecar containers are secondary containers within a pod that support the main container in its workload duties. For example, service meshes use sidecar architecture and act as a proxy for the main container. Since sidecars can leverage the existing capabilities of the main container, bad actors will use this design and hide malicious activity within the sidecar.
Best Practice for Mitigation
Primary areas to configure security controls: Kubernetes and Other - Tooling
All containers in a pod will, by default, use the same permissions as each other. Users will need to set the security context at the pod and container level to differentiate between each container’s capabilities and set appropriate limits on all of the containers in a pod. It is vital to limit the capabilities of containers to only what is required, to limit the attack surface of any container breakouts.
A significant portion of securing from software injection with sidecars can also happen upstream as part of container scanning and ensuring the correct container images are used at runtime.
How RHACS Helps
Red Hat Advanced Cluster Security starts by hardening your containers upstream in your authorized registries. The RHACS container image scanner alerts on container vulnerabilities to stop injection during the build phase. RHACS will block privileged capabilities from being used as a rule in your clusters and only allow them if an exception is granted. Malicious behavior is monitored and can alert on privilege escalation or the use of package managers.
RHACS organizes your deployments and assesses their risk holistically. The same threats that an attacker could use in one container will be assessed against all containers. Both containers in a pod will be monitored when grouping threats and RHACS alerts on policy violations and abnormal process activity.
3. Persistence - Malicious Admission Controller
Issue
Kubernetes has many types of admission controllers; however, one controller can modify Kubernetes objects after the user's submission. Mutating admission controllers, as the name implies, can mutate a Kubernetes object. Using a MutatingAdmissionWebhook, attackers can alter your previously defined Kubernetes objects to gain persistence in the cluster or adjust container permissions. These changes can happen without the user's knowledge if they are not actively monitoring their container workloads and settings.
Best Practice for Mitigation
The primary area to configure security controls: Kubernetes
The MutatingAdmissionWebhook can be used to manipulate deployments to configure resources in unintended ways. If you do not need the MutatingAdmissionWebhook functionality in your clusters, it is best to disable it altogether. Using RBAC controls with a principle of least privilege will limit an attackers' ability to use a MutatingAdmissionWebhook and generally lower your cluster’s attack surface.
How RHACS Helps
RHACS contains built-in policies such as the mounting of sensitive directories, secrets, and any alterations to your workloads that are exploitable. You can customize and create policies to suit your workloads and clusters, establishing known workload behavior. If a MutatingAdmissionWebhook maliciously alters your workloads to violate one of these policies, a policy violation will be triggered that blocks the alteration using the RHACS admission controller (if enforcement is enabled).
RHACS gives you visibility into the cluster's RBAC controls, allowing you to minimize any attempts to create malicious objects like the MutatingAdmissionWebhook.
4. Credential Access - Access Managed Identity Credential
Issue
Different cloud providers have managed identities used to allocate resources, such as Kubernetes Clusters or Virtual Machines. The Instance Metadata Service (IMDS) will provide information about currently running virtual machine instances. This information is available inside a virtual machine, and if an attacker gains access inside a container, the attacker can leverage access to get the managed identity's token. An attacker could then use these tokens to gain greater access to an organization's cloud resources.
Best Practice for Mitigation
The primary areas to configure security controls: Kubernetes + Cloud Provider
IMDS is a REST API that is available at a well-known, non-routable IP address. Since the information is available only from inside the virtual machine, your containers must not have access to this information. First, prevent access to the container using principles of least privilege, security contexts, and containers without root permissions. Next, set up network policies to block the internal address of the REST API and ensure that your containers cannot mount any persistence from the host location.
How RHACS Helps
There are a few ways to mitigate a bad actor’s ability to access host information. RHACS highlights containers with elevated privileges that can access the IMDS information. RHACS blocks container execs, and a mature vulnerability management program can reduce the risk of remote code execution exploits. With RHACS, you can set up monitoring controls like runtime alerts for processes with the metadata service IP as an argument alerting you in the case of any requests to the IMDS service.
5. Lateral Movement - CoreDNS Poisoning
Issue
Kubernetes uses a Domain Name System (DNS) server called CoreDNS. CoreDNS is the primary DNS service that ships with Kubernetes and can be configured by a file called core file. CoreDNS uses a configmap to set up and propagate the configuration to any other CoreDNS pods. This architecture provides a vector of attack where malicious actors seek to change the core file and alter the behavior of the cluster's DNS.
Best Practice for Mitigation
Primary areas to configure security controls: Kubernetes + Cloud Provider
Any type of access to the cluster's DNS service is a severe threat. Typically, CoreDNS will reside in the kube-system namespace and the config map that can be used to configure it. This means that an attacker will need kube-admin access to your cluster or privilege access through a vulnerable container. In this case, users should employ the principle of least privilege. RBAC monitoring and security context functionality should be applied.
Config maps also have an immutable feature that makes it impossible to change the configmap without deleting it. While this presents a challenge for updates, it means that attackers cannot make any changes to the configmap without your becoming immediately aware.
How RHACS Helps
RHACS can monitor configmaps and actions against the Kubernetes API. Changes to the CoreDNS configmap can trigger a policy violation and alert. Monitoring configmap changes are beneficial in more cases than the CoreDNS poison tactic. Configmaps are widely used to set up container environments and can be a method to modify applications and potentially gather sensitive information.
As a secondary measure, RHACS will monitor an alert on any anomalous behavior you set, including kube-system resources. Not only are there excellent controls to stop attackers from gaining access to your cluster, but any changes or alteration of network capabilities would also be immediately detected and alerted on.
6. Lateral Movement - ARP Poisoning and IP Spoofing
Issue
Kubernetes’ modular approach to functionality leads to the rise of multiple Container Network Interfaces (CNIs) vying to help provide network functionality to your containers. While Kubenet is often the default CNI used, most cloud providers have their own recommended CNI, and you can install your preferred CNI in your clusters as well.
To network pods, a bridge is created on each node (cbr0) to which the various pods connect using veth pairs. Since pods are connected through a bridge, performing ARP poisoning in the cluster is possible. An attacker will still require elevated permissions to perform such actions; with these privileges, several network-level attacks are possible such as DNS spoofing.
Best Practice for Mitigation
Primary areas to configure security controls: Kubernetes + Cloud Provider
Ensuring that you are using your cloud provider's preferred CNI is an easy way to limit vulnerabilities that may occur from the incorrect configuration. ARP poisoning and IP spoofing occur with privileged access on the host at a container level. Ensure that your security contexts are appropriately configured, specific Linux capabilities are dropped, and containers do not contain excess network tooling.
How RHACS Helps
RHACS will monitor and alert any anomalous behavior that falls outside of your acceptable traffic. You can enforce approved network traffic using the RHACS network graph to set up and enforce your network policies.
7. Collection - Images From Private Registry
Microsoft added the collection technique in its new update using MITRE’s initial collection technique. The collection technique consists of tactics used by attackers to collect data from the cluster or through the cluster.
Issue
The container images running in a cluster can be stored in a public or private registry. Most organizations opt for a private registry to protect their software code and supply chain. To pull private images into a cluster, the container runtime engine (such as Docker or containerd) needs to have valid credentials to those registries. Private registries are used to safe-keep the underlying software running in your clusters. If a bad actor gains access to these credentials, they can pull your images and use them how they see fit.
The use case is similar to the access managed identity credential technique previously discussed in which attackers can leverage cloud credentials located inside a cloud VM. Any credentials situated in the host system need to be protected to limit the impact of any external access.
Best Practice for Mitigation
Primary areas to configure security controls: Kubernetes and Other - Tooling
Organizations must have a private registry to secure and verify that the container images running in your cluster are approved and vetted. Gaining access to these credentials is no easy feat as it would require administrator access. In this case, the typical IAM and RBAC controls will need to be monitored and applied appropriately.
How RHACS Helps
Restricting access to and from your private image repositories is out of scope for RHACS. However, gaining the repository credentials from inside the cluster can be mitigated through RHACS policies that have previously been mentioned. Limiting container privileges and RBAC controls with a principle of least privilege, identifying malicious activity in compromised containers, and removing the workloads accordingly.
Integrating the RHACS scanner into your CI pipeline provides a correlative workflow between each container’s build deploy and runtime stages. This integration limits the chance that a bad actor will be able to inject code into your containers without warning.
Get in Touch
RHACS takes a multi and hybrid cluster approach to security. That means understanding all threats present to users and their platforms. Our best-in-class features focus on the Kubernetes application life cycle, ensuring that you understand and mitigate the risks in your environments.
To learn more about Red Hat Advanced Cluster Security for Kubernetes, request a personalized demo, and start securing your Kubernetes environment.
