How are Kubernetes Secrets managed in Red Hat OpenShift?
Managing Kubernetes Secrets in Red Hat® OpenShift® can be done in several ways, and there are tools designed to make this process more secure and manageable. Organizations can use cert-manager to handle x.509 certificates and make network communications more secure. Additionally, the External Secrets Operator (ESO) and Secrets Store CSI Driver (SSCSI) can be used to pull credentials from external systems. ESO and SSCSI both make credentials available to applications in Kubernetes in different manners, which will be discussed below. Each of these tools has its own purpose, but they all aim to keep sensitive data out of the application code.
Beyond the tools themselves, the following practices help organizations manage Kubernetes Secrets more efficiently:
- Rotation: regularly changing credentials, to prevent them from becoming stale and compromised.
- Revocation: the immediate disabling of a credential when you suspect it has been compromised.
- Role-based access control (RBAC): creates rules managing and identifying the people and applications that can access specific Kubernetes Secrets.
- Configuration: maintenance and auditing of tools to make sure credentials are protected throughout their lifecycle.
- Monitoring: actively tracking events or alerts that could indicate a breach.
Learn more at: A Holistic approach to encrypting secrets, both on and off your OpenShift clusters
What will you learn?
- Introduction to the Secrets Store CSI Driver (SSCSI Driver)
- Introduction to the External Secrets Operator (ESO)
- How to choose between SSCSI Driver and ESO for managing secrets in Red Hat OpenShift
- Introduction to cert-manager
Secrets Store CSI (SSCSI) Driver
The Secrets Store CSI Driver is a tool that allows Kubernetes to access and use multiple credentials, keys, and certificates from external secret management systems. It works by mounting these credentials directly into the application's pods as a volume (a temporary storage area). Once attached, the sensitive data becomes available within the container's file system, ready for the application to use.
The SSCSI Driver is particularly useful because it communicates with a provider to retrieve the Kubernetes Secret content from an external resource. The SSCSI driver can retrieve credentials from multiple providers—preventing lock-in.
Organizations often choose to use the SSCSI Driver because they have strict compliance rules that require credentials to be stored and managed exclusively in a central store (not directly on the Kubernetes cluster). This is also a good option for organizations using environments and control planes that are externally hosted or managed, and who do not have full control over the platform and want to keep their workload credentials private from the platform administrator.
It's important to note that the SSCSI Driver doesn't directly handle all types of credentials, such as those needed as environment variables or for pulling container images.
SSCSI Driver: Key features and workflow
A major benefit of using the SSCSI Driver is that secrets are never stored directly on the cluster. Instead, they're mounted as temporary, short-lived volumes, which helps with security and compliance. The SSCSI Driver also allows for automatic rotation of credentials, so if a credential is updated in the external store, the change is automatically passed down to the applications using it. Customers are also able to reference all the credentials an application needs from a single location.
The overall workflow is straightforward: when a Pod is created, the kubelet (the agent on the node) tells the CSI Driver to mount as tmpfs. The SSCSI driver then talks to its provider (via the provider’s authentication method) to retrieve the secrets and writes them to the volume's file system. Once the Pod is deleted, the mounted volume is automatically removed, ensuring no sensitive data is left behind.
To install the SSCSI Driver, and for further working instructions, see:
External Secrets Operator (ESO)
The External Secrets Operator (ESO) is a tool that synchronizes credentials from external management systems into Kubernetes Secrets and manages them in Red Hat OpenShift. External secrets managers provide increased security via short-lived secrets generators, improved audit capabilities, and periodic credential refresh. The External Secrets Operator enhances the standard Kubernetes Secret resource on a cluster by managing all aspects of their lifecycle.
Organizations often choose ESO when they:
- are familiar with Kubernetes and want a solution that doesn’t require many changes to application code;
- want a solution that integrates with their platform;
- have a high degree of trust in the cluster's control plane;
- have multi-cluster setups that need credentials to be shared across them;
- need credentials to be modified on the cluster with templating for specific applications
ESO: Key features and workflow
ESO is particularly good for managing credentials required for Ingress certificates, image pulls, or automation. It also offers advanced Kubernetes Secrets templating—which means it can take multiple pieces of information from an external secret repository and construct complex resources, like database connection strings or JSON Web Tokens (JWT).
ESO is a "Day 2 Operator," meaning it's designed to be added to a cluster that's already up and running without disrupting its lifecycle. The workflow is as follows:
- Customers use a SecretStore (namespace scoped) or ClusterSecretStore (cluster scoped) to define what credentials should be requested from the external secrets repository.
- The SecretStore (or ClusterSecretStore) passes the request for credentials to the Provider which will authenticate with the external secret repository to retrieve requested credentials for ExternalSecrets to use.
- Defining an ExternalSecret will create a managed Kubernetes Secret with the credentials specified in the ExternalSecret populated in it.
- Pod is created and scheduled referencing the Kubernetes Secret and consumes the credentials as expected.
With ESO, Kubernetes Secrets can persist on the cluster even if the connection to the external store is lost.
To install ESO, and for further working instructions, see:
How to choose between SSCSI Driver and ESO
SSCSI Driver works best for organizations that:
- don’t want credentials stored in etcd as Kubernetes Secrets;
- have stringent compliance requirements that require them to centrally store and manage secrets;
- work in environments in which the control plane is not managed by them, and desire full control and privacy of their workload secrets
ESO works best for organizations that:
- can store credentials in etcd as Kubernetes Secrets;
- need platform integration and cross-cluster secrets integrations;
- need secrets to be modified on clusters with templating for specific applications;
- need secrets to persist on clusters;
- have a high degree of trust in the control plane of the cluster
Cert-manager
Using strong certificates within Red Hat OpenShift is crucial for modern application security, as certificates are used to encrypt all traffic between the control plane and the worker nodes, as well as connections to the Ingress controller, container registry, and etcd.
Cert-manager adds Certificates and Issuers as resource types within a Kubernetes cluster. The Certificate resource contains the necessary parameters to generate a certificate signing request (CSR), and an Issuer is a representation of a certificate authority (CA) that is able to generate signed certificates by honoring certificate signing requests generated by the Certificate resource. It simplifies the process of getting, renewing, and using certificates for various purposes—including security related to Ingress traffic, the API server, and application workloads.
Cert-manager: Key features and workflow
Cert-manager helps with large-scale certificate management. It can be used with an external Certificate Authority (CA), which provides a single, trusted source for all certificates. This is particularly valuable for large organizations that want to centralize their TLS certificate management and avoid relying on multiple, independent CAs.
Cert-manager can be configured to issue certificates with short lifespans, significantly improving a cluster's security posture so that if a breach occurs, the compromised certificates will only remain valid for a short time. While managing a large number of short-lived certificates would be a huge manual burden, cert-manager automates the entire process, managing hundreds or thousands of certificates in a production environment, and handling both issuance and rotation. This improves security by reducing the risk of stale or expired credentials, and frees administrators from the tedious and error-prone process of manual rotation. Cert-manager also monitors certificates to ensure compliance and help track expiration dates.
Cert-manager is a "Day 2 Operator," meaning it's designed to be added to a cluster that's already up and running without disrupting its lifecycle. The workflow is as follows:
- Customers use an Issuer (namespace scoped) or ClusterIssuer (cluster scoped) to define how certificate signing requests should be handled. This includes which external certificate authority will be signing the request.
- A Certificate resource is created defining the information needed to create a certificate signing request.
- Cert-manager observes the Certificate and generates a certificate signing request.
- The Issuer (or ClusterIssuer) passes the certificate signing request to the external certificate authority.
- If the certificate signing request is approved, the external CA will return a signed certificate to cert-manager.
- Cert-manager will place the signed certificate into a Kubernetes Secret.
- Pod is created and scheduled referencing the Kubernetes Secret and consumes the certificate as expected.
To install cert-manager, and for further working instructions, see:
Cert-manager and Red Hat OpenShift Service Mesh: istio-csr
While cert-manager has traditionally been a single-cluster operator, its integration with Istio and Red Hat OpenShift Service Mesh allows it to manage certificates across multiple clusters, which is valuable for organizations with complex, multi-cluster setups.
When used with a service mesh like Red Hat OpenShift Service Mesh, Cert-management offers significant additional benefits. By default, some service meshes might store their private root key as a Kubernetes Secret, which, while scoped to a specific namespace, still presents a security risk if an attacker gains access to it. Cert-manager allows organizations to move the root CA to an external location and have the istio-agent forward all signing requests to it. This allows the Issuer to authenticate with the external secrets manager, adding an extra layer of protection and insulating the root CA from attacks.
To install cert-manager, and for further working instructions, see:
