Jenkins Production Troubleshooting: Queues, Controller Latency and Build Failures

Engineering scenario: The Jenkins UI intermittently returns 502, controller CPU spikes, agents remain idle and builds are delayed. Design a diagnosis plan that differentiates reverse-proxy, controller JVM, queue, storage and agent failures.

Investigate layers, not symptoms

Distinguish three forms of slowness

1. Queue latency: a job is accepted but cannot acquire an appropriate executor. Check label, agent eligibility, capacity and scheduling constraints.
2. Controller latency: UI/API responsiveness, job graph processing, plugin hooks, Groovy CPS, thread saturation and disk I/O. Even idle agents can coexist with a busy controller.
3. Step runtime latency: tests, SCM checkout, registry, cache, dependency servers, database and workspace I/O. Increasing controller CPU does not optimize a slow compiler on an agent.

Minimal evidence table

Signal	Possible interpretation	Confirm with
NGINX/ALB 502	No healthy upstream, controller restart, connection failure	Proxy access/error logs, upstream TCP, controller restart
Slow Jenkins pages	JVM GC pauses, plugin overhead, disk I/O, reverse proxy	JVM/GC metrics, thread dumps, I/O latency
Build queue grows	No executor/label, cloud launch delay, resource lock	Queue reason + agent cloud logs
Executor busy	Long jobs or hung step	Stage timings, agent process/thread inspection
Lost agents	Network/remoting/TLS failure or OOM kill	Agent launch logs, Kubernetes events, container exit
Disk high	Archived artifacts, build records, snapshots, workspace	Directory sizes, disk/inodes, retention settings

Authoritative commands

# Controller host, read-only; use containers/pod exec when hosted there:
uptime
top -H -p "$(pgrep -f 'jenkins.war|jenkins.Main' | head -n1)"
free -h
vmstat 1 5
iostat -xz 1 5               # Requires sysstat
df -h
df -i
ss -tanp | head -n 30
journalctl -u jenkins --since '20 min ago' --no-pager

# JVM diagnostic tools need an authorized runtime and compatible JDK:
jcmd PID VM.version
jcmd PID Thread.print
jcmd PID GC.heap_info

Safety: Thread dumps, logs and heap data may expose secrets, tokens, plugin configuration or paths. Restrict access, redact before sharing and avoid automatically producing large heap dumps on a nearly-full filesystem. pgrep can match the wrong process; verify the PID manually.

Controller UI 502 investigation

Check upstream error category (connection refused, timeout, unhealthy target, DNS resolution), controller readiness, restarts/OOM, JVM GC and expensive plugin endpoints. A 502 is a proxy-visible symptom, not proof of controller CPU exhaustion. Rollback a recent plugin change only when evidence supports it; restart may temporarily hide the root cause and interrupt resumability.

CPS/serialization failure

If a Pipeline throws NotSerializableException, identify nonserializable variables kept alive across Pipeline step suspension. Convert them to plain serializable data, scope them to a pure @NonCPS helper, or perform the transformation in an agent-executed language. Increasing heap does not make Java objects serializable.

Queue starvation incident

A busy architecture-label pool handles all builds with two executors while four other nodes have different labels. Adding nodes with the wrong label doesn't solve the problem. Inspect queue reasons and requests/limits. If a build asks for a distributed lock, the lock may be the actual constraint even when executors appear free.

Capacity and observability baseline

Measure build starts, duration per stage, time in queue, agent provision/connection latency, controller CPU/heap/GC pause, queue depth, storage usage, failure classifications, SCM indexing latency and artifact-push errors. Avoid unbounded per-build IDs as Prometheus labels; correlate unique IDs through logs and traces.

Incident command and recovery

Identify blast radius (all jobs, one repo, one agent pool, UI only, release blocked), freeze high-risk release changes, assign controller/agent/network investigations, pick the smallest reversible mitigation, verify user-facing CI journey, and record follow-up owners. Restarts should be planned with knowledge of running durable steps and downstream consequences.

Spoken explanation

“I'd separate three timelines: how long a build waits for an executor, how long it takes Jenkins to orchestrate and serve requests, and how long steps actually run. A 502 could be proxy routing or a stalled JVM, while idle agents could indicate a missing label or broken cloud provisioning. I'd correlate controller JVM, disk and network with agent logs and queue reasons before resizing anything. Then I'd implement one reversible mitigation and prove both UI and build-journey recovery.”

References: Scaling Pipelines · System administration · Jenkins troubleshooting