Troubleshoot a runner

Start with the tools that the runner gives you. Then find your symptom below.

Get information

On the Pipelines tab — The Runners table shows each runner's status, last call, version, and platform. A job page shows the failed step's log and the Runner machine panel.

Runner machine panel — The job machine's name, node, image, and runtime. Peak memory, CPU, and workspace size against their limits. A warning if the kernel killed processes for memory, or if the machine was evicted, timed out, or did not start.

nerudo-runner doctor — Checks each part that a runner needs, in sequence. Use the same options as the daemon. See Check the runner.

nerudo-runner status — Shows what the daemon with a state file does now: its phase, job, step, and job machine. It also shows the last job's result and resource use. Add --json for a machine-readable result.

The runner's own log — Each job writes lines like task 42: kubernetes nerudo-… on node-a, peak memory 1.2 GiB of 4.0 GiB, CPU 31.0 s, workspace 2.1 GiB.

Find a job machine

The machine name is on the job page, for example nerudo-0123456789abcdef0123456789abcdef. Use it with the runtime's own tools while the job runs:

To list all job machines on Kubernetes, enter kubectl -n nerudo-runner get pods -l app.kubernetes.io/name=nerudo-job.

The runner removes each machine when its job ends. To keep the step files of a failed job, add a step that copies them to an artifact store of your own.

A job waits: "No runner with these labels is online"

No runner with all of the job's labels called Nerudo in the last 60 seconds.

1. Compare the job's labels with the runner's labels on the Pipelines tab.
2. Make sure that the runner service runs.
3. Enter nerudo-runner doctor on the runner host.

A runner that runs a job is busy. One runner runs one job at a time. See Run more than one job at a time.

doctor: "credential" fails with HTTP 401

Nerudo does not know this runner. Somebody removed it, or it was ephemeral and ran its job.

1. Stop the service.
2. Log in again with a new --state path.
3. Change the service to use the new state file.

A job is cancelled: "could not create job machine"

The runtime could not start the machine. The runner reports cancelled, not failure, so the change's tests are not blamed. The runner then stops until you repair the runtime. Nerudo sends the job again when a runner is available.

The log line shows the runtime's reason. Frequent reasons:

• ImagePullBackOff, pull access denied, manifest unknown — The image name is wrong, or the node cannot read the registry. Add --pull-secret for a private registry.
• not scheduled: Insufficient memory, Insufficient cpu — No node has room for the job's requests. Lower cpus or memory, or add nodes.
• pod did not start within N s — The image pull is slow. Pull the image on each node in advance, or raise --start-timeout.
• Apple container CLI failed — Enter container system start.
• Cannot connect to the Docker daemon — Start docker. Make sure that the runner's user can use its socket.

A step fails: "the job reached its memory limit"

The kernel stopped processes because the job used all of its memory. On Kubernetes and docker, the kernel can stop the whole job machine.

1. Find the peak memory in the Runner machine panel.
2. Raise the runner's --memory, or the ConfigMap's memory on Kubernetes.
3. Or lower the build's parallelism, for example with CARGO_BUILD_JOBS=2.

Build tools often count the host's processors, not the job's CPU limit. nproc in a pod shows the node's processors. Set the parallelism of the build explicitly.

A step fails: "the job pod was evicted"

The pod used more than its ephemeral storage limit. The limit includes the workspace, /tmp, and the step logs.

1. Find the workspace size in the Runner machine panel.
2. Raise disk in the ConfigMap.
3. Or remove large build files in the step that makes them.

df in the pod shows the node's disk, not the limit.

doctor: "it ran with no memory limit"

The runtime accepted --memory but did not apply it. This occurs with rootless podman when systemd does not give the user the memory controller. See Make sure that limits apply.

A job fails with "runner job timeout reached"

The job ran longer than the runner's --timeout or the workflow's timeout-minutes. The runner stops the whole machine, which includes background processes.

The repository's [ci] timeout also applies. Its default is 1800 seconds. Its maximum is 21600 seconds.

A job shows "runner stopped" or the runner stops after a job

The runner stops when it cannot confirm that its machine is gone, or when it cannot deliver the logs. It keeps a private journal of the machine in its work directory.

1. Read the last lines of the runner log.
2. Repair the runtime or the network.
3. Start the runner again with the same state and --work-dir.

At start, the runner removes only the machines in its own journals. It never removes other containers or pods.

Stuck Kubernetes job pods

A job pod has an owner reference to its runner pod and an active deadline. Kubernetes removes it when the runner pod goes, or when the deadline passes. To remove the job pods of a runner now:

1. Enter kubectl -n nerudo-runner delete pods -l app.kubernetes.io/name=nerudo-job.

Do this only when no runner in the namespace runs a job.

See also CI runners and Runner limits.