Data science pipelines with Azure Blob storage on ARO
This content is authored by Red Hat experts, but has not yet been tested on every supported configuration.
OpenShift AI pipelines store their artifacts in S3-compatible object storage. Azure Blob storage doesn’t speak the S3 API. This guide puts the OpenShift Data Foundation Multicloud Object Gateway (MCG) in front of a Blob container, so pipelines get an S3 endpoint while the data stays in your Azure storage account.
MCG runs in standalone mode, so you don’t need the extra storage nodes that a full ODF deployment uses.
Prerequisites
-
OpenShift AI installed on ARO. See Installing Red Hat OpenShift AI 3.x on ARO .
-
The
azCLI, logged in to the subscription that contains the cluster. -
oc4.18 or later (foroc wait --for=create). -
ODF must not already be installed. If it is, skip Deploy the Multicloud Object Gateway and create only the backing store and bucket class in that section.
-
Set these variables:
Create the Blob container
Note: For simplicity, this guide authenticates with a storage account key. For production, restrict network access to the storage account (for example with a private endpoint in the cluster VNet), and plan for key rotation. Rotating the key means updating the azure-blob-creds secret.
Deploy the Multicloud Object Gateway
-
Install the ODF Operator. The channel matches the cluster’s OpenShift minor version:
-
Create a standalone MCG StorageCluster:
-
Create a backing store for the Blob container and a bucket class that uses it:
Create a bucket for a data science project
-
Create the project and make it visible in the dashboard:
-
Request a bucket. MCG creates a ConfigMap and a Secret with the same name, holding the bucket details and the S3 credentials:
Configure the pipeline server
Point the pipeline server at the MCG service. The cluster service CA signs the service certificate. OpenShift publishes that CA to every namespace in the openshift-service-ca.crt ConfigMap:
Create the pipeline server:
Wait for the pipeline server to become ready:
Validate
-
In the dashboard, open the project, go to Pipelines, and import a sample pipeline. Then create a run.
-
When the run finishes, open one of its output artifacts. The dashboard reads it back through the S3 endpoint, which confirms both the write and the read.
-
Check that MCG has written objects to the Azure container:
MCG stores data in its own chunked format, so the blob names won’t match the pipeline artifact names.
Clean up
Warning: These steps permanently delete pipeline data. Run only the steps for resources you created with this guide.
Remove the project and its bucket:
Remove the shared storage only if no other project uses it. To check, run oc get obc -A.