Forwarding and Monitoring ROSA HCP Control Plane Audit Logs with Filebeat

In ROSA Hosted Control Planes (HCP), the Kubernetes API server, authentication server, and OAuth server run on Red Hat-managed infrastructure. Because there are no customer-accessible control plane nodes, the standard approach of reading audit logs directly from /var/log/kube-apiserver/audit.log does not apply. Instead, Red Hat’s built-in log forwarder continuously ships control plane audit logs to an S3 bucket that you own and control.

This guide walks through the complete pipeline: configuring ROSA HCP to forward control plane audit logs to S3, setting up an Amazon SQS notification queue, deploying a Filebeat pod inside the cluster using IAM Roles for Service Accounts (IRSA) to receive and parse those logs, and validating that structured audit events (including who, what, when, and which resource) are correctly extracted. The same pipeline feeds Logstash, Elasticsearch, Splunk, or any other destination that Filebeat supports.

Use Case

Control plane audit logs record every API call made to the Kubernetes API server. This includes:

• Who made each request (user.username, e.g., developer, system:admin, or a service account)
• What they did (verb: create, delete, update, patch, get, list, watch)
• Which resource was targeted (objectRef.resource, objectRef.name, objectRef.namespace)
• When it happened (requestReceivedTimestamp, stageTimestamp)
• What was the outcome (responseStatus.code: 200, 201, 403, 409)

This data is the primary source of truth for compliance auditing (PCI-DSS, HIPAA, SOC 2), incident investigation, and anomaly detection. For example, after running through this guide you will be able to answer questions like:

• Who deployed or deleted workloads in namespace demo-kumudu?
• Which service account patched a deployment’s replica count?
• Were there any 403 Forbidden responses that might indicate privilege escalation attempts?

S3 Log Format

Before configuring ingestion, it is important to understand the format produced by ROSA’s log forwarder:

The two-layer structure means parsing requires two decode_json_fields passes: one to unwrap the array element, and a second to decode the nested audit event inside message. This is covered in the Filebeat configuration section below.

Prerequisites

• AWS CLI configured with permissions to create S3 buckets, SQS queues, and IAM roles

• ROSA CLI v1.2.64 or later, logged in (rosa login)

• OpenShift CLI (oc) , logged in to the target cluster as a cluster administrator

• A ROSA HCP cluster in Ready state. Verify with rosa describe cluster -c <cluster-name>

  Create a ROSA HCP cluster control plane logs forwarding to an S3 bucket .

• Filebeat 8.9.0 or later. This guide uses 8.15.0 in a container image; the expand_event_list_from_field: ".[]" option for top-level JSON arrays was added in 8.9.0

Set Environment Variables

Set these once and reuse them throughout the guide.

Forward Control Plane Logs to S3

Follow the Red Hat documentation to configure control plane log forwarding to your S3 bucket. Update the environment variables LOG_PREFIX and BUCKET_NAME above to match your configuration.

Verify Logs Are Flowing to S3

Wait 60 to 90 seconds, then list objects in the bucket:

Expected output:

Files above approximately 10 KB contain audit events. Very small files (under 3 KB) are application log lines from the kube-apiserver container itself, not audit events. Both types are handled correctly by the Filebeat configuration in this guide.

Inspect the format of a large file to confirm the two-level JSON structure:

Expected output:

Create the SQS Queue

Filebeat uses SQS event notifications to know when new objects arrive in S3. Create the queue in the same region as the bucket.

Get the queue ARN and apply a policy that permits S3 to send messages to it:

Configure S3 event notifications to publish to the queue when new objects appear under the log prefix:

Verify the notification configuration:

Expected output:

Create the Filebeat IAM Role

Filebeat runs as a pod in the cluster and uses IRSA (IAM Roles for Service Accounts) to authenticate to AWS without static credentials. The IAM role trust policy uses the cluster’s OIDC provider so that only the filebeat service account in the rosa-logging namespace can assume it.

Get the cluster’s OIDC issuer:

Expected output:

Create the IAM role with a trust policy scoped to the filebeat service account, and attach an inline policy granting read access to the S3 bucket and SQS queue:

Expected output:

Deploy Filebeat to the Cluster

Create the rosa-logging namespace, then apply all resources from a single manifest.

Create the manifest. The deployment mounts the service account’s projected token at the path expected by the AWS SDK for web identity authentication.

Apply the manifest:

Expected output:

Verify the pod is running:

Expected output:

Validate Structured Audit Events

Check the Filebeat pod logs. After a few seconds, you should see it connect to the SQS queue and begin processing S3 objects.

Expected output:

After 30 seconds, check the pipeline metrics. Look for added and acked values that match in large batches (hundreds per file):

Expected output showing hundreds of events per batch (one large kube-apiserver file typically contains 500 to 1500 audit events):

The core pipeline is now complete. ROSA HCP control plane audit logs are flowing from S3 through SQS into Filebeat, where they are parsed into structured events. The sections below cover reference material for querying and operating the pipeline in production.

Querying Audit Events

The queries in this section assume you have an existing Elasticsearch and Kibana deployment that Filebeat is already forwarding events to. This guide does not deploy Elasticsearch or Kibana. Once events reach Elasticsearch, the following fields are available for filtering and alerting.

Key Fields Reference

Example Queries

Find all write operations by a specific user in a namespace:

Detect secret access events:

Find privilege escalation attempts (403 responses):

Track deployment scaling events:

Understanding the Audit Trail

To show what a complete audit trail looks like, here is an example from a live ROSA HCP cluster. A developer created a new project, deployed a Java Spring Boot application from source, deleted a pod, and scaled the deployment. The audit log captured the full sequence:

Each user action is immediately followed by a cascade of system controller actions (scheduler, replicaset-controller, endpoint-controller) that are also fully recorded. This gives security teams a complete causal chain from the initial human action to every side effect.

Common Configuration Pitfalls

These are the three most frequent configuration errors when ingesting ROSA HCP audit logs with Filebeat.

Pitfall 1: content_type and expand_event_list_from_field at the Wrong Level

When file_selectors is present, Filebeat applies only the selector’s own settings to matching files. Input-level content_type and expand_event_list_from_field are silently ignored for selector-matched files.

The symptom of this mistake is a message field containing the entire raw JSON array as a string, and only one event emitted per S3 file instead of hundreds.

Pitfall 2: Only One Level of JSON Decode

The ROSA log forwarder wraps each audit event in an outer metadata object. A single decode_json_fields pass decodes the outer wrapper but leaves message (the inner audit event) as an unparsed string under audit.message. audit.auditID is never populated, so drop_event removes every event.

Pitfall 3: input and decompress are Invalid file_selectors Fields

Only regex, content_type, and expand_event_list_from_field are valid inside file_selectors. Using input or decompress causes Filebeat to silently ignore those keys, leaving content_type unset and processing every file as binary/octet-stream.

Production Considerations

The manifest in this guide uses output.console and emptyDir storage. Both are suitable for validation but require changes before running in production.

Output destination

Replace output.console with output.logstash or output.elasticsearch in the ConfigMap.

Logstash (recommended when you need pipeline routing, enrichment, or fan-out to multiple destinations):

Elasticsearch directly (simpler when you don’t need Logstash transformation):

Keep replicas at 1

Run exactly one Filebeat replica. The SQS visibility timeout (VisibilityTimeout) prevents two consumers from processing the same message simultaneously, but if a second replica picks up a message before the first acknowledges it, both will download and parse the same S3 object. There is no deduplication downstream that removes identical audit events, so multiple replicas produce duplicate events in Elasticsearch.

If you need higher throughput, increase the number_of_workers setting inside the aws-s3 input instead:

Persistent volume for Filebeat state

The data volume in the manifest uses emptyDir, which is wiped on every pod restart. Filebeat stores its SQS position and per-file cursor in /usr/share/filebeat/data. Without persistence, a pod restart causes Filebeat to re-download and re-process every unacknowledged SQS message.

Replace the emptyDir volume with a PersistentVolumeClaim:

Then update the Deployment volume entry:

SQS dead-letter queue

Configure a dead-letter queue (DLQ) on the SQS queue so that S3 objects Filebeat cannot parse do not cycle indefinitely. Without a DLQ, a corrupt or unexpected file format causes the message to become visible again after VisibilityTimeout expires and Filebeat retries it forever.

Cleanup

Remove the Filebeat deployment and namespace:

Remove the HCP log forwarder:

Remove AWS resources:

Summary

Additional Resources

• Red Hat documentation: Forwarding control plane logs on ROSA HCP
• Elastic documentation: Filebeat aws-s3 input
• GitHub PR #35475: Add support for top-level JSON arrays in aws-s3 input , which introduces expand_event_list_from_field: ".[]" (Filebeat 8.9.0)
• AWS documentation: S3 Event Notifications with SQS