NERUDO
Sign in

Kubernetes runner

The kubernetes driver runs each job in a fresh pod. The runner itself runs in a pod in the same namespace. It needs no privileges and no host access.

How it works

The runner pod uses kubectl with its service account. For each job, it creates a job pod with these properties:

  • Requests equal to limits for CPU, memory, and ephemeral storage.
  • A read-only root filesystem, no capabilities, and a non-root user (UID 1000).
  • No service account token and no host paths.
  • An owner reference to the runner pod. If you delete the runner pod, Kubernetes deletes its job pods.
  • An activeDeadlineSeconds value. The kubelet stops the pod even if the runner stops.
  • A network policy that lets the pod resolve names and reach the internet only. It blocks the cloud metadata service, the cluster's pods and services, the nodes, and private address ranges.

Caution: A network policy has an effect only if the cluster's network plugin enforces it, for example Calico or Cilium. Flannel alone does not. Without enforcement, a job can reach the cluster's internal services.

The runner sends step scripts and step variables on standard input, not in command lines. The API server's audit log records command lines.

Before you start

  • Kubernetes 1.30 or later, with x86_64 or arm64 nodes.
  • Permission to make a namespace, a role, and a deployment.
  • The nerudo-runner-VERSION-kubernetes.yaml file. See Download the runner.
  • A machine with nerudo-runner for the login. It can be your laptop.

Register the runner

  1. On any machine, enter nerudo-runner login --repo OWNER/REPO --name k8s-ci --labels LABELS --state ./runner.json.
  2. Get an owner or admin approval. See Register the runner.
  3. Enter kubectl create namespace nerudo-runner.
  4. Enter kubectl -n nerudo-runner create secret generic nerudo-runner-state --from-file=runner.json.
  5. Delete the local runner.json.

The Secret now holds the runner credential. Do not give other workloads access to the namespace's Secrets.

Deploy the runner

  1. Open nerudo-runner-VERSION-kubernetes.yaml.
  2. In the nerudo-runner ConfigMap, set image, cpus, memory, disk, and timeout.
  3. Enter kubectl apply -f nerudo-runner-VERSION-kubernetes.yaml.
  4. Enter kubectl -n nerudo-runner rollout status deploy/nerudo-runner.
  5. Enter kubectl -n nerudo-runner logs deploy/nerudo-runner -f.

The log shows runner declared with the kubernetes driver. The runner shows as online on the repository's Pipelines tab.

The init container downloads the runner and kubectl. It checks each file against the SHA-256 digest in the manifest. The runner pod then uses a stock Alpine image. You do not need an image registry.

Check the runner in the cluster

  1. Enter kubectl -n nerudo-runner exec deploy/nerudo-runner -- /opt/nerudo/bin/nerudo-runner doctor --driver kubernetes --runtime /opt/nerudo/bin/kubectl --state /var/lib/nerudo-runner/state/runner.json --image IMAGE --skip-machine.
  2. Make sure that permissions shows ok.

Remove --skip-machine to also start and remove a test job pod.

Change the job machine

  • Image, CPU, memory, disk, timeout — Change the ConfigMap. Then enter kubectl -n nerudo-runner rollout restart deploy/nerudo-runner.
  • Nodes — Add --node-selector kubernetes.io/arch=arm64 to the runner's arguments. Use any node labels.
  • Stronger isolation — Add --runtime-class NAME for a sandbox runtime such as gVisor or Kata Containers.
  • Private images — Add --pull-secret NAME. The Secret must be in the runner's namespace.
  • Image user — Add --run-as UID:GID, or --run-as image for the image's own user. The namespace enforces the restricted Pod Security Standard, which refuses root.

Each node pulls the image the first time that it runs a job. The first job on a node waits for the pull. A DaemonSet that pulls the image in advance removes this wait.

Run more than one job at a time

One runner runs one job at a time. For more parallel jobs, register more runners.

  1. Log in once for each runner, with a different --name.
  2. Make a Secret for each runner.
  3. Apply a copy of the Deployment for each runner, with its own name and Secret.

Do not raise replicas. Two pods with one credential take each other's jobs. The Deployment uses the Recreate strategy, so an update never runs two pods at once.

Remove the runner

  1. Enter kubectl delete -f nerudo-runner-VERSION-kubernetes.yaml.
  2. Remove the runner from the repository. See Remove a runner.

Kubernetes deletes the job pods with the runner pod.

See also Troubleshoot a runner and Runner limits.