Please note that the capabilities covered in this post are not officially supported by Red Hat. Container technologies have had a very strong influence in application development. In addition to the well-documented value of containers related to deployment and management of applications, one of the biggest impacts has been the ability to create sets of full integration tests that can run anywhere.

The traditional way to test applications was heavily rooted in mocking due to the inability of the developers to run external services in their local development environment. The reason for mocking is to be able to programmatically replicate expected results from an external service, such as a database or messaging broker. Developers would create mock implementations using framework like Mockito and substitute these mock implementations when running in an environment where the services are not available. The problems pop up due to the limitations associated with mocking such as the inability to properly test database schema changes or transactions.

With containers, these service availability constraints can be eliminated. Developers can now spin up all kinds of infrastructure and services quickly and easily. Developers can use tools like docker or podman to start the external services in whatever environment they like and use those services for testing. For example, instead of writing a bunch of mock implementations for data access, which require their own care and maintenance, developers can now run suites of tests that can cover the full stack, not just the application code.

Enter Testcontainers

The next step in the evolution was to bring container orchestration into the testing life cycle. The first iteration was based on the build tool life cycles. For example, in Maven, a developer could use a Maven plugin to create and destroy resources as needed in the Maven life cycle. However, these quickly evolved and were replaced by more agile Java libraries like Testcontainers. The Testcontainers library provided a lightweight, easily customizable set of APIs  to plug-in containers of all types directly into JUnit. At first glance, Testcontainers seems to be the ultimate enabler for building a very powerful and thorough set of testing capabilities for any project. Developers can install Docker on their local machine and run full integration tests using a wide array of industry standard services.

Testcontainers finds my local daemon managed by Docker Desktop

The Issue: CICD

The problem comes from Testcontainers native integration with Docker. Testcontainers expects a docker daemon to be generally available on the host in which it is running, typically located at /var/run/docker.sock. If you are running some legacy CICD products that run on a single host, this is generally not a problem. However, when you want to use something that is container based, such as Tekton, then the issue rears its ugly head. The containers used to perform builds do not expose a Docker daemon and therefore the Testcontainers library will not be able to do its job.

Where is my unix domain socket for Docker? It is not there.

Docker-in-Docker

For the solution, we need to be able to run Docker. However, because we are running a Tekton pipeline, we are going to be running in containers, so the solution is to be able to run Docker in a container. This is where Docker-in-Docker comes in handy. According to the Docker blog:

What’s special in my dind? Almost nothing! It is built with a regular Dockerfile. Let’s see what is in that Dockerfile.

 

First, it installs a few packages: lxc and iptables (because Docker needs them), and ca-certificates (because when communicating with the Docker index and registry, Docker needs to validate their SSL certificates).

The Dockerfile also indicates that /var/lib/docker should be a volume. This is important, because the filesystem of a container is an AUFS mountpoint, composed of multiple branches; and those branches have to be “normal” filesystems (i.e. not AUFS mountpoints). In other words, /var/lib/docker, the place where Docker stores its containers, cannot be an AUFS filesystem. Therefore, we instruct Docker that this path should be a volume. Volumes have many purposes, but in this scenario, we use them as a pass-through to the “normal” filesystem of the host machine. The /var/lib/docker directory of the nested Docker will live somewhere in /var/lib/docker/volumes on the host system.

Now that we have a solution for being able to run Docker inside of a container, we need to figure out how to plug this into the task.

Tekton Sidecar

Luckily, Tekton has an answer for this. Sidecars allow additional containers to be configured and spun up on a task pod. For our example, we would be looking for a Maven container to execute the appropriate goals, but we need to have a running Docker daemon located at /var/run/docker.sock and available to the JUnit test life cycle.

The Task

Here is the full yaml for an example Maven build task:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 name: maven-build-task
spec:
 workspaces:
   - name: source
   - name: maven-settings
 params:
   - name: MAVEN_IMAGE
     type: string
     description: Maven base image
     default: gcr.io/cloud-builders/mvn@sha256:57523fc43394d6d9d2414ee8d1c85ed7a13460cbb268c3cd16d28cfb3859e641 #tag: latest
   - name: GOALS
     description: maven goals to run
     type: array
     default:
       - "package"
   - name: MAVEN_MIRROR_URL
     description: The Maven repository mirror url
     type: string
     default: ""
   - name: SERVER_USER
     description: The username for the server
     type: string
     default: ""
   - name: SERVER_PASSWORD
     description: The password for the server
     type: string
     default: ""
   - name: PROXY_USER
     description: The username for the proxy server
     type: string
     default: ""
   - name: PROXY_PASSWORD
     description: The password for the proxy server
     type: string
     default: ""
   - name: PROXY_PORT
     description: Port number for the proxy server
     type: string
     default: ""
   - name: PROXY_HOST
     description: Proxy server Host
     type: string
     default: ""
   - name: PROXY_NON_PROXY_HOSTS
     description: Non proxy server host
     type: string
     default: ""
   - name: PROXY_PROTOCOL
     description: Protocol for the proxy ie http or https
     type: string
     default: "http"
   - name: CONTEXT_DIR
     type: string
     description: >-
       The context directory within the repository for sources on
       which we want to execute maven goals.
     default: "."
 steps:
   - name: mvn-settings
     image: registry.access.redhat.com/ubi8/ubi-minimal:8.2
     script: |
       #!/usr/bin/env bash
       [[ -f $(workspaces.maven-settings.path)/settings.xml ]] && \
       echo 'using existing $(workspaces.maven-settings.path)/settings.xml' && exit 0
       cat > $(workspaces.maven-settings.path)/settings.xml <<EOF
       <settings>
         <servers>
           <!-- The servers added here are generated from environment variables. Don't change. -->
           <!-- ### SERVER's USER INFO from ENV ### -->
         </servers>
         <mirrors>
           <!-- The mirrors added here are generated from environment variables. Don't change. -->
           <!-- ### mirrors from ENV ### -->
         </mirrors>
         <proxies>
           <!-- The proxies added here are generated from environment variables. Don't change. -->
           <!-- ### HTTP proxy from ENV ### -->
         </proxies>
       </settings>
       EOF
       xml=""
       if [ -n "$(params.PROXY_HOST)" -a -n "$(params.PROXY_PORT)" ]; then
         xml="<proxy>\
           <id>genproxy</id>\
           <active>true</active>\
           <protocol>$(params.PROXY_PROTOCOL)</protocol>\
           <host>$(params.PROXY_HOST)</host>\
           <port>$(params.PROXY_PORT)</port>"
         if [ -n "$(params.PROXY_USER)" -a -n "$(params.PROXY_PASSWORD)" ]; then
           xml="$xml\
               <username>$(params.PROXY_USER)</username>\
               <password>$(params.PROXY_PASSWORD)</password>"
         fi
         if [ -n "$(params.PROXY_NON_PROXY_HOSTS)" ]; then
           xml="$xml\
               <nonProxyHosts>$(params.PROXY_NON_PROXY_HOSTS)</nonProxyHosts>"
         fi
         xml="$xml\
             </proxy>"
         sed -i "s|<!-- ### HTTP proxy from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
       fi
       if [ -n "$(params.SERVER_USER)" -a -n "$(params.SERVER_PASSWORD)" ]; then
         xml="<server>\
           <id>serverid</id>"
         xml="$xml\
               <username>$(params.SERVER_USER)</username>\
               <password>$(params.SERVER_PASSWORD)</password>"
         xml="$xml\
             </server>"
         sed -i "s|<!-- ### SERVER's USER INFO from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
       fi
       if [ -n "$(params.MAVEN_MIRROR_URL)" ]; then
         xml="    <mirror>\
           <id>mirror.default</id>\
           <url>$(params.MAVEN_MIRROR_URL)</url>\
           <mirrorOf>central</mirrorOf>\
         </mirror>"
         sed -i "s|<!-- ### mirrors from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
       fi
   - name: mvn-goals
     image: $(params.MAVEN_IMAGE)
     workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
     command: ["/usr/bin/mvn"]
     args:
       - -s
       - $(workspaces.maven-settings.path)/settings.xml
       - "$(params.GOALS)"
       - -ntp
     volumeMounts:
       - mountPath: /var/run/
         name: dind-socket
 sidecars:
   - image: docker:20.10-dind
     name: docker
     securityContext:
       privileged: true
     volumeMounts:
       - mountPath: /var/lib/docker
         name: dind-storage
       - mountPath: /var/run/
         name: dind-socket
 volumes:
   - name: dind-storage
     emptyDir: {}
   - name: dind-socket
     emptyDir: {}

In this task, the Docker sidecar is started, and the Docker daemon is mounted on a shared volume, dind-socket. That same volume is mounted in the Maven container on the mount path expected by Testcontainers, namely /var/run. This gives us access to the running Docker daemon and makes it available for use by Testcontainers.

Adding in the Proper Permissions

If you try to run this task in OpenShift without any additional configuration, it will not work. OpenShift, by default, does not allow containers to be run in privileged mode which enhances the overall platform security. But in this case, we need to run in privileged mode because of Docker. Therefore, we need to update the security for the service account running our tasks.

We want to isolate the permissions to a single privileged service account to the CICD  namespace. Of course, this is assuming all of your pipelines run in a single namespace and are generally managed as a centralized service. If this is not the case, then each pipeline administrator will need to set up their own privileged service account for their own namespace. To accomplish this, let’s create a new service account and provide the correct SCC  to it:

oc project cicd
oc create sa dind
oc adm policy add-scc-to-user privileged -z dind

This serviceAccountName can be specified in the TriggerTemplate for the pipeline and can be narrowed to only apply to the Maven build step that needs the access:

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: maven-build-trigger-template
spec:
 params:
   - name: git-repo-url
     description: The git repository url
   - name: git-repo-name
     description: The name of the rep
   - name: git-revision
     description: The git revision
   - name: git-ref
     description: The name of the ref

resourcetemplates:
 - apiVersion: tekton.dev/v1beta1
   kind: PipelineRun
   metadata:
     generateName: maven-build-pipeline-$(tt.params.git-repo-name)-
   spec:
     serviceAccountName: pipeline
     serviceAccountNames:
       - taskName: maven-build-task
          serviceAccountName: dind
     pipelineRef:
       name: maven-build-pipeline
     params:
       - name: git-repo-url
         value: $(tt.params.git-repo-url)
       - name: git-repo-name
         value: $(tt.params.git-repo-name)
       - name: git-revision
         value: $(tt.params.git-revision)
       - name: git-ref
         value: $(tt.params.git-ref)
     workspaces:
     - name: workspace
       volumeClaimTemplate:
         spec:
           accessModes:
             - ReadWriteOnce
           resources:
             requests:
               storage: 500Mi

Happy building!