One common characteristic of many OpenShift deployments, especially in a multitenant situation, is the need to onboard development teams on the cluster. As the number of teams increase, it becomes imperative that a process be identified in order to onboard a team. And, as usual, the more automated that process is the better.
When designing such a solution, what are the primary goals that should be addressed? In theory, we want a process that automatically provisions all the needed resources and authorizations for teams to start working with OpenShift. This usually involves provisioning on at least three systems: OpenShift itself, the enterprise container registry, and the CI/CD pipeline tool. In this article, we are going to focus on the former and address what needs to be accomplished in OpenShift.
Three general problems need to be addressed in this solution:
- Map the company organization to OpenShift
- Sync the organizational structure from a directory management system to OpenShift
- Create and configure namespaces
In this article, we are going to examine those problems and propose solutions for them. We will also walk through a scenario that demonstrates an end-to-end fully automated solution for each of those problems.
Mapping the Company Organization to OpenShift
First, we need to map the company organizational structure to OpenShift so that the permissions associated with users are consistent with the actual roles they currently possess within the organization. Typically, we do not need to map the entire organization, but only those portions of it that will have to use OpenShift.
Normally, a company’s organization is stored in a directory service, such as LDAP, or in a HR product that then feeds the corporate directory service. Also, often the organization structure that we find in these directory services is hierarchical and coarse grained relative to what is desired for OpenShift. For example, we might have the following hierarchical levels: Line of Business (LOB) -> Business Unit (BU).
On the OpenShift side, permissions are given with a three-way binding between a role (which represents a group of permissions), a group, and a namespace (which often represents a specific environment of the SDLC).
Namespaces in Kubernetes are a flat structure with no concept of hierarchy. In addition, they are also relatively fine grained since they represent an SDLC environment.
So, there is a gap normally between the representation of a company organization that we can find in a company’s directory service, and the need in OpenShift. The diagram below attempts to capture the concept:
The key here is the green layer on the right side of the diagram which represents an augmentation of the company LDAP organization hierarchy with the Teams and Apps layers.
In order for these goals to be realized, a tool or set of tools must be employed that allow us to extend the company hierarchy with the layers necessary to reach the granularity that OpenShift requires. It should also provide a level of self servicing so that team managers can self configure their portion of the organization (manually or with an automation that they devise). This orgs management tool could be as simple as a file in a git repository that undergoes a review process through the form of a Pull Request (PR) and that, once updated, triggers some kind of downstream automation.
For our demo, we have decided to use RH SSO as our orgs management tool. Some of the advantages of this choice, besides that it meets our basic requirements, are that RH SSO is fully supported by RedHat and comes with no additional costs if you already have OpenShift.
You can follow the steps of the demo at this repository.
We assume we have a corporate LDAP with two hierarchical layers: Line of Business and Business Unit. Here is a representation of our LDAP structure after following the steps described within these paragraphs within the demo repository:
If everything goes well, you'll be able to see that we have pre-populated some groups and some users.
Following the next steps of the demo, we can deploy RH SSO and integrate it with the LDAP server.
The integration between RH SSO and LDAP synchronizes every five minutes. After that time elapses, you should see the imported hierarchy in RH SSO:
At this point, the idea is that each BU manager will extend the hierarchy by adding additional group layers (team, application) and then assigning employees to each of those groups.
In RH SSO with fine grained permissions, it’s possible to create a configuration by which a user is given the role to manage a portion of the group hierarchy. Once this configuration is done, this user will be able to add subgroups to the hierarchy and manage group memberships. This way, the responsibility of augmenting the original hierarchy does not have to be centrally owned by a team, but can be farmed out to the BU owners (this fine-grained permission is not configured in the demo).
In our demo, we simulate the augmentation process using a script. After the script is executed, the following should be displayed:
In this scenario, only the Online Retail Banking BU has been augmented.
As you can see, we have standardized group names on a naming convention (this is optional, but can be useful). Also we have added metadata to the created groups (this will be described in further detail later) as shown here:
So, this particular group has two pieces of metadata: type and size. We will use type to determine what actions should be applied to the group. In this case, the type is application, and as a result, we will create the SDLC namespaces for it. We use the size to determine which T-shirt size quota to assign to the created namespaces.
In Kubernetes, claims associated with a user, such as the groups they belong to, are passed by the identity manager integrated with the platform. So, groups do not need to be stored anywhere; only group bindings need to be created.
In OpenShift, the design is different: Groups and users need to exist in OpenShift for authentication and authorization to function as designed. User objects are created the first time a user logs in, so their management is relatively straightforward. Groups, instead, need to be created by some external process or automation. This, in most cases, means that groups need to be imported from a group system of record using a sync mechanism.
OpenShift has historically supported syncing groups only with LDAP via the oc command. This covered most of the use cases found in most organizations. However, an increasing number of customers have recently started integrating OpenShift with different IDP providers, mostly OIDC implementations.
The Container Community of Practice (CoP) has published the group-sync-operator to cover those use cases.
The group-sync-operator can sync groups and group membership with several IDPs and has a plugin architecture, so support for more IDPs can be added in the future. Also, as long as the IDP allows for it, it manages groups metadata such as hierarchical information and generic group labels.
In our demo, we are going to set-up OpenShift to integrate with RH-SSO as the Authentication provider and use the group-sync-operator to sync the groups from RH-SSO:
After you complete these steps (OCP-RHSSO integration, group-sync deployment and configuration), you should be able to see that groups have been imported into OpenShift. If you inspect the groups, you should be able to see the metadata associated with them, and finally you should be able to login to OCP using RH-SSO. When you login, at this point, you won’t have any permissions.
At this point, we have the desired groups and users in OpenShift. Next, we need to create namespaces with the correct permissions and configurations.
To be more specific, besides creating the namespaces, some common types of configuration that are usually requested by customers are:
- Configure role bindings
- Configure namespace quotas and multi namespace quotas
- Configure network policy (at the very least provide a sensible default)
- Configure egress network policy
Other requirements may be needed on a case-by-case basis.
We obviously want an automated process to do that and possibly one choreographed by an operator.
The namespace-configuration-operator allows you to define policies to create and enforce arbitrary Kubernetes resources based on users, groups, and namespaces.
The namespace-configuration-operator uses label and annotation selectors to choose on which objects to apply the defined configurations.
In the demo, we are going to use the namespace-configuration-operator to create namespaces based on the defined groups and their annotations (the group metadata), and furthermore, we are going to configure the created namespace with some of the resources described previously.
You can read the details of the configuration requirements here.
After you run the steps (namespace-configuration-operator deployment and policy definitions), you should see that several namespaces have been added to your cluster. You should also be able to see that now if you login, you can see several namespaces, but can operate only on some of them:
Once this setup is in place, if modifications occur in either LDAP or RH SSO, they will be quickly reflected on OCP. Users do not have to request any additional provisioning besides asking to be put in the right groups and should be able to work with OpenShift right away.
The intent of this blog was to demonstrate how to combine some building blocks to create a fairly sophisticated and fully automated team onboarding process. Hopefully, this will be a starting point for real implementations (where potentially multiple clusters are involved).
It’s also worth noting that as we mentioned in the introduction, here we covered only the OpenShift use case. Nut a full onboarding process should probably also cover the container registry and the CI/CD pipeline systems. Operators can once again play a role here: For example, it’s not difficult to imagine an operator that based on namespaces existing in OCP, creates repositories on Artifactory or Nexus and configures their pull secrets in those namespaces.