Blog My Projects Contact Me
Adam Comer

How to set up Gitlab's CI/CD Runner on Kubernetes

Adam Comer | Feb 1, 2019

Introduction:

CI/CD Systems are quickly becoming the norm in industry and Open Source. These teams have CI/CD Systems that execute code builds and unit tests on every commit to validate any new code. Additionally, software has become containerized and tools like Kubernetes have taken over datacenters. Getting started with a CI/CD system is easy to do with Gitlab’s CI/CD Runner. In this tutorial, I will demonstrate how to setup Gitlab’s CI/CD System and deploy it to Kubernetes.

Registering a runner:

The first step to setting up a Gitlab Runner on Kubernetes is to obtain an authentication token. This token is essential because it authenticates your runner to Gitlab. To register a Runner, we need to get configuration details from Gitlab and walk through a registration process for our Runner.

In the first step, we need to get a few details from the Gitlab repository that we want our Runner to connect to. Head to your repository online and go to Settings > CI/CD > Runners. Then go to the section titled: Set up a specific Runner manually. This page has the configuration details we need to register a new Runner with Gitlab. Put this page aside. We will use the information on this page to complete the registration in the next step.

Next, we need to go through the registration process to connect a new Runner. The easiest way to register a Runner is to start a Docker container locally with the Runner, but you can use any of the methods from Gitlab if you so choose. Using the docker method we can start the Runner with this command.

$ docker run -it --entrypoint /bin/bash gitlab/gitlab-runner:latest

This command starts a Runner on our local machine to register a new Runner. Specifically, this command starts the bash shell in the Docker container. This Docker container is a perfect environment to register our new Runner.

The next is to register a new Runner.

$ gitlab-runner register

Here is where we take the information we got from our Gitlab repository and use it to register a new runner. Plug the information into the terminal for each section of the form as follows.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
https://gitlab.com/

Please enter the gitlab-ci token for this runner:
[TOKEN]

Please enter the gitlab-ci description for this runner:
[RUNNER-ID]: My first Gitlab runner!       

Please enter the gitlab-ci tags for this runner (comma separated):
tutorial

Whether to lock the Runner to current project [true/false]:
[true]: false

Registering runner... succeeded                     runner=[RUNNER ID]

Please enter the executor: shell, ssh, docker+machine, docker-ssh+machine, kubernetes, docker, docker-ssh, parallels, virtualbox:
docker

Please enter the default Docker image (e.g. ruby:2.1):
busybox:latest

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! 

Don’t worry if you want to change the executor or the default image. You can change these values in the config when we deploy our Runner on the Kubernetes cluster. To use the Runner on Kubernetes, we use a different set of settings to take advantage of Kubernetes specific deployments.

To check if the registration was successful go to your browser and refresh the settings page. Under the registration information we used to authenticate this Runner, will be the new Runner. This Runner should not be active and should have a disconnected error associated with it.

Before we deploy our Runner on Kubernetes, we need to get the authentication token. When we registered the Runner, Gitlab saved the authentication token in the docker container. All we need to do is find it. In the terminal run this command to print the configuration Gitlab Created.

$ vim /etc/gitlab-runner/config.toml

concurrent = 1
check_interval = 0

[[runners]]
  name = "temp runner"
  url = "https://gitlab.com/"
  token = [TOKEN]
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "busybox:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

The output is the configuration that Gitlab created when we registered the runner. I removed my token for security purposes but is denoted by [TOKEN]. Copy this token and make sure not to lose it. It’s the only way to authenticate our Runner to Gitlab. Once the token is saved, it is safe to delete the Docker instance.

Setting up the runner on Kubernetes:

Once we have our token and store it somewhere safe, we can start work on creating our Runner on Kubernetes. I’m running the cluster for this demo on GKE, but the principles here are similar for any of the major cloud providers(AWS, Azure, or IBM Cloud).

Before we start to add our Runner to the cluster, we must create a few resources. One is to create a Namespace for our Runner, and it’s pods. The Namespace separates our Runner from our other applications to help with future maintenance. The second is a ServiceAccount, Role, and RoleBinding to give our Runner the privileges to add new Pods to the Namespace.

Create a new Namespace called gitlab-runner.

$ kubectl create namespace gitlab-runner

Create the ServiceAccount, Role, and RoleBinding for our Runner.

# gitlab-runner-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: gitlab-runner
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: gitlab-runner
  name: gitlab-admin
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]

---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-admin
  namespace: gitlab-runner
subjects:
  - kind: ServiceAccount
    name: gitlab-admin
    namespace: gitlab-runner
roleRef:
  kind: Role
  name: gitlab-admin
  apiGroup: rbac.authorization.k8s.io
$ kubectl apply -f gitlab-runner-service-account.yaml

This ServiceAccount gives the runner access to create, modify, and view other pods in the cluster. Use this with caution. It’s common for organizations who are concerned about giving these privileges to the Runner to create separate clusters to run their CI/CD pipelines.

Once finished with the administrative work, we can start work on creating a ConfigMap to hold our Runner’s configuration on the Kubernetes Cluster. The Runner’s ConfigMap is very similar to the Runner configuration that Gitlab created for the authentication token but with some Kubernetes metadata to properly store and reference it. Take the authentication token that was created in the last section and add it the ConfigMap in the section with [TOKEN].

# gitlab-runner-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-config
  namespace: gitlab-runner
data:
  config.toml: |-
    concurrent = 4
    [[runners]]
      name = "Kubernetes Demo Runner"
      url = "https://gitlab.com/ci"
      token = "[TOKEN]"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab-runner"
        poll_timeout = 600
        cpu_request = "1"
        service_cpu_request = "200m"

Add this ConfigMap to the Kubernetes cluster.

$ kubectl apply -f gitlab-runner-config.yaml

This configuration is slightly different from the one that was generated by Gitlab when we registered our Runner. This configuration runs each Pipeline on a new Kubernetes Pod. By running directly on Kubernetes with a new Pod, the testing and build environments are clean for each pipeline.

The final thing to create is a Deployment for our Runner. This Deployment is straightforward because the Kubernetes executor on the Runner handles running each pipeline on a new Pod. The Deployment is a single Pod with no autoscaling for the Deployment(the Runner handles autoscaling). The only thing that needs to be integrated is our ConfigMap we created above. Add the Deployment to the Kubernetes cluster.

# gitlab-runner-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
spec:
  replicas: 1
  selector:
    matchLabels:
      name: gitlab-runner
  template:
    metadata:
      labels:
        name: gitlab-runner
    spec:
      serviceAccountName: gitlab-admin
      containers:
        - args:
          - run
          image: gitlab/gitlab-runner:latest
          imagePullPolicy: Always
          name: gitlab-runner
          resources:
            requests:
              cpu: "100m"
            limits:
              cpu: "100m"
          volumeMounts:
            - name: config
              mountPath: /etc/gitlab-runner/config.toml
              readOnly: true
              subPath: config.toml
      volumes:
        - name: config
          configMap:
            name: gitlab-runner-config
      restartPolicy: Always
$ kubectl apply -f gitlab-runner-deployment.yaml

To check if the runner connected to Gitlab head back to your browser and go to your project’s settings. The Runner will have a green dot next to its name to identify it has connected to your Runner. There might be a delay for the runner to connect. This was the case for me.

Now that our Runner is connected you can run CI/CD pipelines on your Gitlab repository. Although the Runner we created can execute testing stages it is unable to build Docker containers. This is the next stage in our journey.

Enabling Docker builds:

To enable our Runner to build Docker containers we must give the Runner access to the local Docker daemon. Do note that this change gives the Runner access to make changes to the Kubernetes cluster’s Docker daemon and has several security considerations to keep in mind. Running containers in a privileged mode with other production containers is not recommended.

To enable building Docker containers directly inside our CI/CD pipeline, we need to update our ConfigMap. We need to give the executor access to the local Docker daemon and run Pods in privileged mode. Change the ConfigMap the reflect these changes as such.

# gitlab-runner-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: gitlab-runner-config
  namespace: gitlab-runner
data:
  config.toml: |-
    concurrent = 4
    [[runners]]
      name = "Kubernetes Runner"
      url = "https://gitlab.com/ci"
      token = "[TOKEN]"
      executor = "kubernetes"
      [runners.kubernetes]
        namespace = "gitlab-runner"
        privileged = true
        poll_timeout = 600
        cpu_request = "1"
        service_cpu_request = "200m"
        [[runners.kubernetes.volumes.host_path]]
            name = "docker"
            mount_path = "/var/run/docker.sock"
            host_path = "/var/run/docker.sock"
$ kubectl apply -f gitlab-runner-config.yaml

Next is to restart our Runner Pod and the Runner will update with the new ConfigMap. Once completed, we can enable the docker:dind service in our CI/CD pipeline and build docker containers directly in our CI/CD pipelines.

Conclusion:

We can see that setting up a Gitlab CI/CD Runner on Kubernetes isn’t very difficult to do. Once we get an Authentication token from Gitlab, we can connect our runner to Gitlab. Then a simple ConfigMap and Deployment are needed to launch our runner on Kubernetes. To allow our Runner to build Docker containers, we can give the Runner access to the Docker daemon on the local machine. Once our Runner is up an running, we can execute our CI/CD pipelines on our Kubernetes cluster.