Satisfying Developer Requests

We have recently been in a situation where our customer wanted to just give access to the developers to manage the right projects in OpenShift. Developers at the same time, needed to create applications on OpenShift without needing to consult administrators for access to ArgoCD/OpenShift GitOps. This has increased in importance for non-production environments. If a developer team has gone through the effort of requesting budget, time, and resources to develop a new application on OpenShift, should they really wait for an administrator to change settings for them to get started? 

In this post we will examine the ability for administrators to create an environment in OpenShift GitOps that promotes the ability for developers to manage their own applications and content. This will speed up the development process, allowing developers the flexibility to deploy as frequently as they need to resources they have access to.

Recipe for success

Deploy GitOps. A basic deployment of GitOps in an OpenShift environment is necessary to do this successfully. More on how to do this can be found here Installing OpenShift GitOps - GitOps | CI/CD. The following custom resource definitions get installed 2 of which we are using below:

-bash $ oc get crd -o custom-columns=NAME:.metadata.name | grep argo
applications.argoproj.io
applicationsets.argoproj.io
appprojects.argoproj.io
argocds.argoproj.io

Once that has been completed, the next step is to ensure that administrator access has been given to your GitOps administrator team. In our example we have 2 teams, one being a platform admin team that has full access for managing Argo and the other being a read only team. The below illustrates an example of this:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
 finalizers:
 - argoproj.io/finalizer
 name: openshift-gitops
 namespace: openshift-gitops
 ownerReferences:
 - apiVersion: pipelines.openshift.io/v1alpha1
   blockOwnerDeletion: true
   controller: true
   kind: GitopsService
   name: cluster
spec:
***********
 rbac:
   defaultPolicy: ""
   policy: |
     p, role:platform-team-admin, applications, *, */*, allow
     p, role:platform-team-admin, *, *, *, allow
     p, role:platform-readonly, applications, get, */*, allow
     p, role:platform-readonly, repositories, get, *, allow
     p, role:platform-readonly, logs, get, *, allow
     p, role:platform-readonly, projects, get, *, allow
     p, role:platform-readonly, clusters, get, *, allow
     p, role:platform-readonly, accounts, get, *, allow
     p, role:platform-readonly, certificates, get, *, allow
     p, role:platform-readonly, gpgkeys, get, *, allow
     g, PlatformAdmins, role:platform-team-admin
     g, PlatformReaders, role:platform-readonly
   scopes: '[accounts,groups]'
***********

After the deployment of OpenShift GitOps, a list of custom resources has been imported in the OpenShift Cluster. These are the resources that could be created should a new team require access to OpenShift GitOps in order to create applications on OpenShift. The main resource that needs to be created is the AppProject. This is a realization of the ability of GitOps to create multi-tenanted environments. 

This would be a starting point for a ‘real life’ example on creating a project in GitOps to interact with a variety of namespaces in an OpenShift cluster:

kind: AppProject
metadata:
 name: teamproj
 namespace: openshift-gitops
spec:
 clusterResourceWhitelist:
 - group: '*'
   kind: '*'
 description: teamproj project
 destinations:
 - namespace: teamproj-dev
   server: https://kubernetes.default.svc
 - namespace: teamproj-uat
   server: https://kubernetes.default.svc
 - namespace: teamproj-pp
   server: https://kubernetes.default.svc
 namespaceResourceBlacklist:
 - group: '*'
   kind: NetworkPolicy
 - group: '*'
   kind: ResourceQuota
 - group: '*'
   kind: LimitRange
 roles:
 - description: Admin group for teamproj
   groups:
   - teamadmins
   name: team-admins
   policies:
   - p, proj:teamproj:team-admins, applications, *, teamproj/*, allow
   - p, proj:teamproj:team-admins, repositories, *, teamproj/*, allow
 - description: Readonly group for teamproj
   groups:
   - teamreaders
   name: teama-readers
   policies:
   - p, proj:teamproj:team-readers, applications, get, teamproj/*, allow
   - p, proj:teamproj:team-readers, repositories, get, teamproj

To allow for the namespaces mentioned above to be managed by OpenShift GitOps, we have labeled them with the following:

oc label namespace teamproj-dev argocd.argoproj.io/managed-by=openshift-gitops

What we see above are scoped resources for the project created in GitOps. Normally, an ArgoCD admin creates a project and decides in advance which clusters and Git repositories it defines. This is not possible for ever changing teams and clusters. It creates a problem in scenarios where a developer wants to add a repository or cluster after the initial creation of the project. The above project definition shows a reliable way of offering a self service definition for allowing developers to add their own repositories and applications in the GitOps area.

The above definition disallows changes to be made under the NetworkPolicy, ResourceQuota, and LimitRange resources of the OpenShift projects that the GitOps project is able to deploy to. This is a good way of ensuring that only cluster administrators are able to make those changes if the OpenShift RBAC matches the restrictions. This is useful in environments where there’s a very strict segregation of duty.

Moreover, it defines 2 separate teams of people that could be given access for both administrative and observability purposes in order to ensure the aforementioned segregation of duties.

The policy items reference Openshift-argo items as follows: openshiftGroup:argoGroup. Argo CD gets the info about the OpenShift user (like the user's group membership) from Dex, an identity service that uses OpenID Connect to drive authentication for other apps.

To test the above setup, we used https://github.com/argoproj/argocd-example-apps ‘guestbook’ application. This was successfully deployed on OpenShift and then we were able to observe said application with a user from the read-only group. 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: guestbook
 namespace: openshift-gitops
spec:
 destination:
   namespace: teamproj-dev
   server: https://kubernetes.default.svc
 project: manual
 source:
   path: guestbook
   repoURL: https://github.com/argoproj/argocd-example-apps
   targetRevision: HEAD
 syncPolicy:
   syncOptions:
   - Validate=false

So currently, only applications, repositories and clusters can be project scoped, however, it seems apparent that there is demand for scoping other types of resources such as logs, as seen in https://github.com/argoproj/argo-cd/issues/7211

Food for thought

While individual manual project creation is where most people would start with, it would be recommended to proceed with an automation mechanism onboarding teams onto GitOps in such a way that enables a team to deploy to multiple environments in a cluster. This will allow the re-use of resources and would give team autonomy to enable the developers in creating promotion processes fitting to their development practices. 

We have automated the above using jinja templates and Ansible with a group of variables that are being parsed in per-team. This allows the automation to scale as when a new team is being introduced to the platform, a process runs to automate the creation of the items above. 

Related reading: https://cloud.redhat.com/blog/managing-applications-via-a-gitops-control-plane


About the authors

James Force is a Senior Consultant at Red Hat. After working in a handful of infrastructure engineering and consulting jobs, James joined Red Hat at the start of 2020. He takes a forward-looking and straight-talking attitude in his work and promotes modern working methodologies to build easily reproducible, reliable, secure and scalable systems. He is an advocate of FOSS, DevOps and Ken Thompson's UNIX philosophy and sees himself as a problem solver rather than an expert in a specific technology. When he's not leading customer engagements James can be found cafe-hopping, playing music or travelling.

Read full bio