Enabling cross account EFS mounting

The Amazon Web Services Elastic File System (AWS EFS) is a Network File System (NFS) that can be provisioned on Red Hat OpenShift Service on AWS clusters. With the release of OpenShift 4.10 the EFS CSI Driver is now GA and available.

This is a guide to enable cross-account EFS mounting on ROSA.

Important: Cross Account EFS is considered an advanced topic, and this article makes various assumptions as to knowledge of AWS terms and techniques across VPCs, Networking, IAM permissions and more.

Prerequisites

• One AWS Account containing a Red Hat OpenShift on AWS (ROSA) 4.16 or later cluster, in a VPC
• One AWS Account containing (or which will contain) the EFS filesystem, containing a VPC
• The OC CLI
• The AWS CLI
• jq command
• watch command

Set up environment

1. export some environment variables

2. As we will be swapping back and forth between two AWS accounts, set up your AWS CLI profiles to avoid confusion now:

Prepare AWS Account A IAM Roles and Policies

IMPORTANT: Run these commands in AWS ACCOUNT A

1. Swap to your Account A profile

2. Create an IAM Policy for the EFS CSI Driver (Note, this has additional permissions compared to a single account EFS CSI policy)

3. Create the Policy

4. Create a Trust Policy

5. Create Role for the EFS CSI Driver Operator

6. Attach the Policies to the Role

7. At this stage, the Role that the EFS CSI Controller uses can now assume a role inside Account B, now we need to go to Account B and set up the correct permissions.

Prepare AWS Account B IAM Roles and Policies

IMPORTANT: Run these commands in AWS ACCOUNT B

In this account, we need to allow certain permissions to allow the EFS operator in AWS Account A to reach AWS Account B.

1. Swap to your Account B profile

2. Create an IAM Policy

3. Create the Policy

4. Create a Trust Policy

5. Create Role for the EFS CSI Driver Operator to assume

6. Attach the Policies to the Role

Set up VPC Peering

Set up Account A

1. Swap to your Account A profile

2. Start a peering request to Account B from Account A

3. Accept the peering request from Account B

4. Get the route table IDs for Account A and add route to Account B VPC

5. Get the route table IDS for Account B and add route to Account A VPC

6. Enable DNS resolution for Account A to read from Account B’s VPC

Deploy and test the AWS EFS Operator

1. Create a Secret to tell the AWS EFS Operator which IAM role to request.

2. Install the EFS Operator

3. Wait until the Operator is running

4. Install the AWS EFS CSI Driver

5. Wait until the CSI driver is running

6. Create a new secret that will tell the CSI Driver the role name in Account B to assume

7. Allow the EFS CSI Controller to read this secret

Prepare the security groups on Account A to allow NFS traffic to EFS

IMPORTANT: Run these commands on Account A

1. Swap to your Account A profile

2. Run this set of commands to update the VPC to allow EFS access

3. Update the Security Groups in Account A to allow NFS traffic to your nodes from EFS

At this point you can create either a single Zone EFS filesystem, or a Region wide EFS filesystem. To simplify this document, we’re going to give only an example of a Region wide EFS filesystem.

Creating a region-wide EFS filesystem in Account B

1. Swap to your Account B profile

2. Create a region-wide EFS File System

3. Configure a region-wide Mount Target for EFS (this will create a mount point in each subnet of your VPC by default)

Create a Storage Class for the EFS volume and verify a pod can access it.

1. Create a Storage Class for the EFS volume

2. Create a namespace

3. Create a PVC

4. Create a Pod to write to the EFS Volume

   It may take a few minutes for the pod to be ready. If you see errors such as Output: Failed to resolve "fs-XXXX.efs.us-east-2.amazonaws.com" it likely means its still setting up the EFS volume, just wait longer.

5. Wait for the Pod to be ready

6. Create a Pod to read from the EFS Volume

7. Verify the second POD can read the EFS Volume

   You should see a stream of “hello efs”

Cleanup

1. Delete the Pods

2. Delete the Volume

3. Delete the Namespace

4. Delete the storage class

5. Delete the EFS Shared Volume via AWS

   Note: if you receive the error An error occurred (FileSystemInUse) wait a few minutes and try again.

   Note: if you created additional mount points for a regional EFS filesystem, remember to delete all of them before removing the file system

6. Detach the Policies to the Role

7. Delete the Role

8. Delete the Policy

9. Detach the policies from the cross-account role

10. Delete the Role

11. Delete the Policy

12. Remove peering connection from account B

13. Remove peering connection from account A