Patching GitLab jobs with Admission Controller

GitLab CI Runner with Kubernetes executor is a service that can run CI jobs by creating kubernetes Pods. The Runner itself has limited configuration and most of the config is abstracted by Runner Helm Chart.

Many Kubernetes features cannot be used with GitLab CI jobs without patching the runner. While sending an upstream fix may often be a viable solution, it will take months at best to be merged given GitLab’s current track record. And that is assuming they deem the feature/fix acceptable.

We can let GitLab Runner create the pods as well as it can and patch the objects before it is executed. Kubernetes has a concept of Admission Controllers, which we may leverage to update the Pods. Clusterise has published a very simple scaffolding for patching GitLab runner jobs at https://github.com/clusterise/gitlab-admission-controller, which leverages a kubewebhook library.

Once the service is deployed and admission webhook is configured, it is invoked whenever GitLab Runner creates a new Pods. Before the Pod object is scheduled and even before it is visible in the API our component has the opportunity to update it.

There are many use cases for mutating CI jobs with admission controller:

  • Add node name hosting the job by injecting spec.nodeName as an env via the downwards API. This is implemented in the example repository.
  • Limit the number of containers (helpers of a job) and set container limits with better granularity
  • Inject labels and annotations, which can be useful eg for scheduling and accounting
  • In general anything that the Runner does not support out of the box!