OpenShift sandboxed containers on ROSA HCP

This guide shows how to install OpenShift sandboxed containers (OSC) 1.13.1 on an existing Red Hat OpenShift Service on AWS (ROSA) cluster with hosted control planes (HCP) and STS. Use the rosa, oc, aws, and jq CLIs. If you need to create a cluster, see Deploying a ROSA HCP cluster with Terraform .

The flow in this guide covers:

• creating an IAM Roles for Service Accounts (IRSA) role for OSC
• installing the Operator with ROLEARN and a Manual InstallPlan
• opening worker security group ports for peer pods
• creating peer-pods-cm and a KataConfig
• running a kata-remote sample workload (validates peer pods when metal is not available)
• adding a metal machine pool for local kata (preferred for sandboxed workloads)
• cleaning up Operator, AWS, and IAM resources

This guide was validated on ROSA HCP 4.20.32 in ap-southeast-2 with default m5.xlarge workers (peer pods) and an optional c5n.metal machine pool (local Kata).

About OSC on ROSA HCP

OSC adds optional RuntimeClass objects. On ROSA HCP, the two RuntimeClasses that matter are:

For sandboxed workloads on ROSA HCP, prefer local kata on a metal machine pool. Pods stay on ROSA workers, schedule through normal Kubernetes controls, and you can overcommit CPU and memory on the metal node like any other worker. kata-remote peer pods launch a separate EC2 instance per pod outside that model: one pod VM per workload, no cluster overcommit, and extra AWS IAM, networking, and cost outside the ROSA data plane. Use peer pods when you cannot add metal workers, or to validate the peer-pod path in a lab.

Setting enablePeerPods: true on KataConfig installs both classes (and kata-nvidia-gpu in 1.13). Local kata requires a metal machine pool. Peer pods work on standard ROSA workers but are not the preferred runtime for ongoing workloads.

ROSA HCP has no Cloud Credential Operator path for OSC. Create an IRSA role and pass ROLEARN on the Operator Subscription. Do not reuse one OSC IAM role across clusters. The OIDC issuer is per cluster.

Prerequisites

• An existing ROSA HCP cluster with STS, at least one Ready worker, and cluster-admin access. This procedure does not apply to ROSA Classic. If you create that cluster with Terraform, run Terraform in a clean shell. Inherited TF_VAR_* values (cluster name, region) override module defaults and can target a cluster that is still deleting.
• OpenShift 4.18.38 or later (OSC 1.13 install prerequisite). Confirm the current OSC compatibility matrix for HCP AWS peer pods. Some matrices list a later 4.20.z than that operator prerequisite.
• Operator catalog redhat-operators (default on ROSA)
• rosa CLI, logged in (rosa whoami)
• oc CLI
• AWS CLI v2 with permission to create IAM roles and policies, plus ec2:AuthorizeSecurityGroupIngress, ec2:DescribeInstances, and ec2:CreateTags
• jq

Decide the test scope before you start:

Do not leave a metal machine pool running after a lab test. For production sandboxed workloads, plan a dedicated metal pool and size it for expected overcommit like any other worker pool.

1. Verify identity and the cluster

1. Confirm CLI identity.

2. Set the ROSA cluster name and confirm it is HCP with Ready workers.

3. Confirm the cluster OIDC issuer (STS).

   Expect a URL such as https://oidc.op1.openshiftapps.com/<oidc-id>.

2. Create the STS IAM role

Create the role before you install the Operator. See Configure an IAM role for STS authentication .

1. Collect the account ID and OIDC provider.

2. Write the trust policy.

   Official 1.13 trust is scoped to:

   • Service account: system:serviceaccount:openshift-sandboxed-containers-operator:default
   • Audience: openshift

   The Operator projects a token with audience openshift. Use that audience in the trust policy. If AssumeRoleWithWebIdentity fails, see Troubleshooting .

3. Create the role and attach two policies:

   • Runtime (keep): AmazonEC2FullAccess so CAA can create and terminate peer-pod EC2 instances. You can replace this with a tighter EC2 policy later.
   • Image build (temporary): a customer-managed policy for the vmimport role and podvm-* S3 buckets. Detach it after the AMI exists.

   Expected attached policies: AmazonEC2FullAccess and ${POLICY_NAME}.

3. Install the Operator

1. Create the namespace and an OwnNamespace OperatorGroup.

2. Create a Subscription that pins CSV 1.13.1, sets ROLEARN, and uses Manual installPlanApproval so you can confirm permissions before CSV upgrades.

3. Approve the InstallPlan for sandboxed-containers-operator.v1.13.1.

   Wait until PHASE is Succeeded for sandboxed-containers-operator.v1.13.1.

4. Confirm STS wiring on the controller Deployment. ROLEARN should match the role you created. The Operator creates peer-pods-secret later, when you create the KataConfig. Do not share secret contents.

4. Open worker security group ports 15150 and 9000

The peer-pod shim talks to the kata-agent on the EC2 VM (TCP 15150) and the tunnel (TCP 9000). Rules are self-referential on the worker security group.

ROSA worker security groups already include TCP/UDP 9000-9999 for internal cluster communication. Still add the exact TCP 9000 and TCP 15150 self-references from the OSC documentation. When you clean up, revoke only those extra OSC rules. Do not delete the 9000-9999 range.

1. Resolve a worker instance, region, and security groups.

2. Inspect existing rules, then add the OSC self-references.

5. Create peer-pods-cm

1. Pull VPC, subnet, and security group IDs from the same worker instance CAA uses for pod VMs.

2. Create the ConfigMap. Leave PODVM_AMI_ID empty. The Operator fills it after the image job. Do not set a placeholder AMI ID.

   DISABLECVM: "true" is required on AWS unless you use confidential VMs, which this guide does not cover. OSC 1.13 uses PROXY_TIMEOUT: "8m".

   The OSC 1.13 pod VM AMI is imported with TPM 2.0. Do not include t2.* types in PODVM_INSTANCE_TYPES. CAA selects a type from that list based on the pod’s CPU and memory requests, and t2.small fails with UnsupportedOperation (AMI TPM v2.0). Use Nitro types such as t3.medium and t3.large.

6. Create KataConfig

1. Create the KataConfig. An empty selector applies Kata to all workers.

   Optional: set spec.kataConfigPoolSelector.matchLabels to limit which workers get Kata.

2. Confirm peer-pods-secret now exists and AWS_ROLE_ARN matches ${ROLEARN}.

1. Watch install progress.

   Worker RPM install can finish (COMPLETED equals TOTAL) while INPROGRESS stays True because the operator is still creating the pod VM AMI. waitingForMcoToStart stays false on ROSA HCP. Do not wait for a MachineConfig reboot.

   Expect osc-rpm-install first. After the AMI exists, also expect osc-caa-ds (one CAA pod per kata-oc node) and openshift-sandboxed-containers-monitor.

   RuntimeClasses kata and kata-nvidia-gpu appear when the RPMs are installed. kata-remote appears after the AMI job completes.

7. Wait for the pod VM AMI

This job is the first end-to-end check of STS and IRSA. The job osc-podvm-image-creation uses IRSA to create vmimport, an S3 podvm-* bucket, and an AMI. Uploading the raw disk (about 10 GiB) and AWS snapshot conversion often takes 10 minutes or more.

1. Follow the job until peer-pods-cm has a real AMI ID.

   Wait until the value is an ami-... ID. If the job fails with STS or audience errors, fix the IAM trust and re-run. Do not use static AWS access keys.

2. Optional: detach the image-creation policy after the AMI exists. Reattach it if you need to rebuild the AMI. Keep AmazonEC2FullAccess (or a tighter EC2 policy) for CAA.

8. Test peer pods (kata-remote)

Use this section to validate peer pods when you cannot add metal workers, or as a lab check after OSC install. For ongoing sandboxed workloads, prefer local Kata on Intel metal so pods stay on ROSA workers and you can overcommit CPU and memory on the metal pool.

Do not put test workloads in the Operator namespace.

1. Create a sample pod.

2. Verify the following:

   • Pod Running 1/1 (often about two to three minutes while the peer EC2 instance boots).
   • CAA launched an EC2 instance named like podvm-osc-peer-ec2-* (t3.medium unless overridden), in the same VPC, subnet, and security group as the workers.
   • The pod IP is on the cluster overlay. The peer VM has a different private IP in the worker subnet.

Optional: override the peer VM instance type on the pod:

```yaml
metadata:
  annotations:
    io.katacontainers.config.hypervisor.machine_type: t3.medium
```

9. Local Kata on Intel metal (preferred for workloads)

Prefer this path for sandboxed workloads on ROSA HCP. Pods use runtimeClassName: kata on ROSA metal workers: they schedule and scale like any other pod, CPU and memory can be overcommitted on the pool, and workloads stay inside the ROSA platform. Peer pods (kata-remote) remain useful when metal is not available, but each peer pod provisions a separate EC2 instance with no overcommit.

runtimeClassName: kata needs /dev/kvm on the worker. Nested virtualization on m5.xlarge is not available. A kata pod on m5.xlarge fails with FailedCreatePodSandBox / DeadlineExceeded after about four minutes.

Metal instances are expensive. For a lab, use one replica, pick the cheapest Intel metal type ROSA HCP offers in the cluster Availability Zone, and delete the pool in the same session. For production, size a dedicated metal pool for your workload density and overcommit targets.

1. Confirm a metal instance type is offered. rosa list instance-types may prompt for an installer role if several exist. Pick the cluster’s HCP installer role.

   If rosa list instance-types prompts for an installer role, pass the cluster HCP installer role:

   Example on-demand Linux prices from validation in ap-southeast-2. Confirm current pricing and availability in your Region:

   Type	Approx. USD/hr	vCPU / memory
   c5n.metal	5.08	72 / 192 GiB
   c5.metal	5.33	96 / 192 GiB
   z1d.metal	5.42	48 / 384 GiB

2. Create a one-node pool on the same private subnet as existing workers (HCP workers are private).

   Wait until current replicas is 1 and the node is Ready. Metal first boot is slow (ostree reboot, large disk, long POST). The EC2 instance can reach running well before the node is Ready (about 20 minutes in validation).

3. After the node is Ready, confirm /dev/kvm and that OSC DaemonSets landed on it. An empty KataConfig selector should install on the new node automatically (kata-oc role).

   Confirm osc-rpm-install / osc-caa-ds pods are on the metal node before you create the test pod.

4. Delete any failed kata pod that scheduled onto m5.xlarge, then create a local Kata pod pinned to metal.

   Verify that the pod is Running (about 16 seconds on c5n.metal in validation) and that oc exec succeeds.

10. Cleanup

Delete resources in this order to control cost: test pods (this terminates peer EC2 instances), then the metal pool if you created one, then KataConfig, then the Operator, then IAM.

Do not delete ROSA cluster roles (*-HCP-ROSA-*, ingress, CSI, EFS, cert-manager, and similar).

1. Delete workloads and confirm peer VMs are gone.

   If a podvm-* instance is still running after the pod is gone:

2. Delete the metal pool if you created one. Confirm the metal EC2 instance reaches terminated before you destroy the VPC. c5n.metal can sit in shutting-down for 10 minutes or more.

3. Delete KataConfig, then the Operator. Deleting KataConfig uninstalls RPMs (DaemonSet osc-rpm-uninstall) and triggers osc-podvm-image-deletion (AMI, S3, vmimport). Wait for the image-deletion job to complete. It removes AWS artifacts the IAM role created.

   Wait until it is gone (finalizer kataconfiguration.openshift.io/finalizer).

4. Revoke only the extra OSC security group self-references (TCP 15150 and TCP 9000). Leave ROSA’s 9000-9999 cluster communication rules.

5. Delete the OSC IAM role and customer-managed policy.

6. If image-deletion ran, vmimport and podvm-* S3/AMI should already be gone. Verify:

   get-role should return NoSuchEntity for vmimport.

7. If you created the cluster only for this test, destroy it after OSC cleanup. Wait until the metal instance is terminated first. If terraform destroy hangs on the VPC, ROSA sometimes leaves a *-vpce-private-router security group that Terraform does not manage. Delete that group, then let destroy continue.

Troubleshooting

STS assume-role or AMI job failures

The image job osc-podvm-image-creation is the first IRSA check. If logs show AssumeRoleWithWebIdentity or audience errors:

1. Confirm the Subscription ROLEARN matches the role you created.
2. Confirm trust sub is system:serviceaccount:openshift-sandboxed-containers-operator:default.
3. Confirm trust aud is openshift (the Operator-projected token audience). ROSA IRSA for other operators often also allows sts.amazonaws.com. If assume-role still fails after the product-doc trust policy, add a second statement (or a second StringEquals audience) for sts.amazonaws.com.
4. Do not use static AWS access keys.

InvalidPermission.Duplicate on security groups

The OSC authorize-security-group-ingress calls are not idempotent. Inspect rules before adding them, or ignore the duplicate error (|| true). During cleanup, revoke only TCP 15150 and the extra TCP 9000 self-referential rule. Do not remove ROSA’s 9000-9999 range.

t2 instance types fail with TPM v2.0 AMI

CAA may select t2.small from a product-doc PODVM_INSTANCE_TYPES list when the sample pod requests little CPU and memory. The OSC 1.13 pod VM AMI requires TPM 2.0, which t2 instance types do not support. Set PODVM_INSTANCE_TYPES to Nitro types (t3.medium,t3.large), restart the osc-caa-ds DaemonSet, and recreate the test pod.

Look for The t2.small instance type does not support an AMI with TPM version v2.0.

Local Kata DeadlineExceeded on m5.xlarge

runtimeClassName: kata needs /dev/kvm. Standard (non-metal) ROSA workers do not provide nested KVM. Prefer a metal machine pool and runtimeClassName: kata as in Local Kata on Intel metal . Use kata-remote only when metal is not an option.

oc exec hangs on kata-remote

This can happen even when the pod is Running and the peer EC2 instance exists. Use pod Ready and EC2 existence as the health check. For production, disable ExecProcessRequest in the Kata agent policy.

Next steps

• Deploying a ROSA HCP cluster with Terraform
• Installing OpenShift sandboxed containers on AWS
• Configuring OpenShift sandboxed containers on AWS