Use Azure Blob storage Container Storage Interface (CSI) driver on an ARO cluster

The Azure Blob Storage Container Storage Interface (CSI) is a CSI-compliant driver that can be installed on an Azure Red Hat OpenShift (ARO) cluster to provision and mount Azure Blob storage for Kubernetes workloads.

The driver mounts Azure Blob storage for use by pods.

You can also refer to the driver’s documentation here .

The Azure Blob CSI driver supports two common dynamic provisioning models:

• Driver-managed path: the driver can select or create a suitable storage account when one is not explicitly specified in the StorageClass.
• Bring your own (BYO) storage account path: the StorageClass is tied to an existing storage account that you create and manage.

Scope of validation

• We tested the steps below on ARO using dynamic provisioning with BlobFuse2 and a manually specified storage account.
• The driver-managed storage account creation path was outside the scope of this update.
• The ARO-specific settings included:
  • a dedicated cloud configuration secret in kube-system
  • the chart’s RHCOS path handling with read-ahead configuration disabled
  • an explicit BlobFuse2 working directory in the dynamic StorageClass mount options
• We also tested a statically provisioned volume with service principal authentication and an Azure RBAC data role. See Mounting without a storage account key .

Prerequisites

• ARO cluster up and running.
• Helm - command line utility
• oc - command line utility.
• jq - command line utility.
• Azure CLI logged into the correct subscription.
• Permissions to create or access an Azure Storage Account for the test.
• Permissions to create a service principal and assign Azure RBAC roles.

1. Set the environment variables related to your cluster environment:

   Update the LOCATION, CLUSTER_NAME, RG_NAME, and VNET_NAME variables in the snippet below to match your cluster details:

2. Set environment variables related to the project and secret names used to install the driver, and the testing project where a pod will use the configured storage:

3. Set environment variables for the test storage account and blob container:

The storage account name must be globally unique in Azure. Update STORAGE_ACCOUNT_NAME if the sample name is already in use.

Create an identity for the CSI Driver to access the Blob Storage

The Azure Blob CSI driver needs Azure credentials so it can access the blob storage resources used by this walkthrough.

This guide uses:

• a service principal with the required Azure permissions
• a Kubernetes secret for the driver
• additional ARO-specific controller cloud configuration in kube-system

1. Create a service principal for the Blob CSI driver:

   Note that the service principal name based on http://$CSI_BLOB_SECRET must be unique enough to avoid colliding with an existing Azure AD app.

2. Export the values from the protected credentials file:

   Delete blob-csi-driver-credentials.json after creating the cloud configuration secret.

3. Assign the required roles to the service principal.

   The lab used these permissions:

   • Contributor on the relevant resource group
   • Storage Account Contributor on the target storage account after that account is created

   These permissions supported the tested lab. They are not presented as the minimum permissions for every BYO storage account deployment. Required permissions depend on which resources the driver must discover, create, or manage.

   Assign Contributor on the resource group:

4. Create a dedicated cloud configuration secret in kube-system for the Blob CSI driver:

   If `oc create` returns `AlreadyExists`, stop and resolve the name or ownership conflict before continuing. Do not overwrite or delete the existing resource. This guide uses `blob-csi-cloud-provider` instead of the chart default, `azure-cloud-provider`. Never modify or delete `azure-cloud-provider` as part of this guide.

   After oc create succeeds, label the new secret and remove the temporary files:

   Keep the data key as cloud-config. The driver reads the secret through the Kubernetes API at runtime, so it does not appear as a mounted volume in the pod specification.

Driver installation

After creating the identity and required Azure configuration, install the Azure Blob CSI driver on the cluster.

On ARO, use the chart’s RHCOS path handling and disable its read-ahead initialization. In chart 1.27.11, linux.distro: rhcos changes the init container’s /usr/local host mount to /var/usrlocal. Setting node.blobfuseProxy.setReadAheadSize: false prevents the init script from writing a host udev rule and running udevadm control --reload, which fails on RHCOS. The init container remains in place and completes normally, so no manual DaemonSet patch is required.

1. Add the Blob CSI driver Helm repository:

2. Install the Blob CSI driver chart with the ARO-specific settings:

   The cloudConfigSecretName overrides point the controller and node components at the dedicated secret created in the previous section, instead of the chart default azure-cloud-provider.

3. Confirm the controller and node workloads reference the dedicated secret:

   Repeat the check for the node DaemonSet:

   Both commands should name blob-csi-cloud-provider and kube-system.

4. Confirm the RHCOS init settings rendered as expected:

   The output should show SET_READ_AHEAD_SIZE set to false, the init container mount path set to /host/var/usrlocal, and the host path set to /var/usrlocal.

5. Check that the Blob CSI driver pods are running:

   Expected output should include two Blob CSI controller pods and one node pod per cluster node in a Running state. Do not continue while any node pod is in an init error state.

6. Confirm that the controller and node processes actually read the dedicated secret:

   In both logs, look for reading cloud config from secret kube-system/blob-csi-cloud-provider. Checking only the container arguments proves that the configuration rendered; the startup log proves that the process read the named secret.

Test the CSI driver is working

To test the Blob CSI driver, create the required storage resources, then create a StorageClass, PersistentVolumeClaim, and a pod that mounts the provisioned storage.

1. Create the storage account:

   Assign Storage Account Contributor on the storage account:

2. Create the blob container:

3. Create the test project:

4. Create a secret in the test project containing the storage account credentials:

   This uses a storage account key. If your organization does not permit shared key access, see [Mounting without a storage account key](#mounting-without-a-storage-account-key) below for a service principal and Azure RBAC alternative.

5. Create the StorageClass:

   When using BlobFuse2 on ARO, adding `--default-working-dir=/tmp/blobfuse2` avoids mount failures caused by the default `/.blobfuse2` path being read-only.

6. Create the PersistentVolumeClaim:

7. Create a test pod that mounts the claim:

8. Check that the pod is running:

9. Check the volume mount:

   Expected result:

   • the pod is in Running state
   • the mount is present at /mnt/blob
   • the filesystem shows blobfuse2

10. Test create, read, modify, and delete operations through the mount:

    If the Blob CSI node DaemonSet is restarted later, existing BlobFuse2 mounts can report Transport endpoint is not connected even though their workload pods still show Running. Recreate only the affected workload pods after the node rollout so Kubernetes establishes fresh mounts.

Mounting without a storage account key

The test above authenticates with a storage account key. Where shared key access is not permitted, a service principal with an Azure RBAC data role can be used instead, through a statically provisioned volume.

This removes storage account keys from the mount. It is not secretless: the service principal still has a client secret. If client secrets are also prohibited, this approach does not meet that requirement either.

1. If you are starting with this section, create a dedicated test project or select one that you created for this walkthrough:

   If it does not exist, create it:

   If it already exists and is dedicated to this walkthrough, select it:

   Do not use a shared project because the cleanup section removes the test project.

2. Set unique names for a second, dedicated storage account and application:

3. Create the storage account with Shared Key access disabled, then create the container through the Azure Resource Manager API:

   Using az storage container-rm avoids relying on a storage account key to create the container. The final command must return false.

4. Create a dedicated application identity and save its generated credentials in a protected temporary file:

   The credentials file contains a client secret. Keep it out of source control and delete it after creating the Kubernetes Secret. az ad sp show returns the service principal object ID, which is different from the application client ID. Role assignments need the object ID.

5. Grant the application a data plane role scoped to the container:

   Role assignments can take several minutes to take effect.

6. If you need independent proof of the authenticating identity, enable Blob service diagnostics before mounting the volume:

   Log ingestion is not immediate. The mount and I/O tests do not need this diagnostic setting, but identity verification does.

7. Create a secret holding only the client secret. Read it from a protected file rather than passing it on the command line:

   This secret contains no azurestorageaccountkey, and is separate from both azure-secret and the driver’s blob-csi-cloud-provider secret.

8. Create a statically provisioned volume that selects service principal authentication:

   An empty storageClassName keeps this volume out of the dynamic, key based StorageClass created earlier. The volumeHandle must be unique within the cluster, and the capacity is a binding value rather than a quota on the container.

9. Create a test pod that mounts the static claim:

10. Confirm the claim binds and the pod becomes ready:

11. Test the data path from inside the pod:

    The output should be first and second on separate lines.

12. If you enabled diagnostics, query the ingested storage requests:

    Allow several minutes for ingestion. Successful PutBlob, ListBlobs, and DeleteBlob rows should have AuthenticationType set to OAuth and RequesterAppId set to $BLOB_CLIENT_ID.

    A bound claim and a successful mount prove that the data path works while Shared Key is disabled, but do not by themselves identify the caller. The diagnostic log fields provide that separate identity proof.

Troubleshooting

Blob CSI controller is not healthy after installation

If the controller pods are not running, check their status and logs:

Testing on ARO showed that dynamic provisioning required additional controller-side Azure configuration in kube-system:

• the dedicated cloud configuration secret

If the secret is missing or incomplete, the controller may fail to initialize correctly.

You can recreate the secret and then restart the controller:

PVC stays in Pending state

If the PersistentVolumeClaim does not bind, describe the PVC and review the related events:

Also check:

• the StorageClass name matches the PVC
• the storage account name and key are correct in the secret
• the blob container exists
• the controller has the required Azure permissions

Pod is stuck in ContainerCreating

If the pod does not start, describe the pod and review the events:

On ARO, BlobFuse2 returned a read-only filesystem error for its default working directory.

To avoid this, include the following mount option in the StorageClass:

Check the mount inside the pod

To confirm the mount is working:

Expected result:

• the pod is in Running state
• the mount is present at /mnt/blob
• the filesystem shows blobfuse2

Clean up

After testing is complete, remove the Blob CSI test resources.

Delete only resources that you created while following this guide. If a name existed before you started, do not overwrite or delete that resource. Never modify or delete the existing azure-cloud-provider secret.

1. Delete the test pod, PersistentVolumeClaim, and StorageClass:

2. Delete the secret from the test project:

3. If you followed Mounting without a storage account key , remove those resources as well:

   Skip the Log Analytics command if you did not enable diagnostics. Deleting the dedicated storage account also removes its diagnostic setting.

4. Delete the test project:

   Run this command only if you created the project for this walkthrough and it contains no unrelated resources.

5. Uninstall the Blob CSI driver Helm chart:

6. Remove the dedicated driver secret only if it still carries the label added by this walkthrough:

   The selector is an additional safeguard, not proof of ownership. Run this command only for the secret that you successfully created and labeled during this walkthrough. Do not delete azure-cloud-provider.

7. If no longer needed, delete the service principal created for the Blob CSI driver:

8. If you created a temporary storage account and blob container specifically for this test, delete them: