Jenkins Architecture: Controller, Agents, Queue and Execution Flow

Engineering scenario: A company operates hundreds of builds across different languages and architectures. Explain what the Jenkins controller owns, what agents execute, and what happens from a webhook to an archived artifact. Where can the entire system fail?

Architectural model

Jenkins is an extensible automation server. Its controller serves the UI and API, handles authentication/authorization, owns job definitions and build records, coordinates Pipeline execution, tracks agent connections, schedules work and integrates plugins. Executors run work on agents. Configure zero executors on the built-in controller node in a production design so untrusted build code does not execute alongside its management plane.

This is a logical flow; some plugin actions, SCM probes, Groovy orchestration and metadata access execute on the controller. The agent directive and node step acquire an executor and normally a workspace. A Pipeline does not automatically reserve a single executor for its entire lifetime if it is designed with agent none and stage-level agents.

Terms that must not be conflated

Term	Meaning	Failure signature
Controller	Coordinates jobs, plugins, users, history and Pipeline graph	UI/API slow, queue/flows stop progressing
Node	Registered machine or cloud-provisioned compute	Offline/mismatched labels
Agent	Remoting process providing execution on a node	Lost connection or launch failure
Executor	Slot for one concurrently allocated task	All busy even while CPU is idle
Workspace	Filesystem area for checkout/build on a node	Disk full, cross-job contamination, stale files
Queue	Jobs waiting for executor/resources/provisioning	Waiting for next available executor
Build record	Console output, artifacts/metadata and status	Storage pressure, slow job listing

A label is a scheduling constraint, not a security boundary. Multiple executors on one agent can share filesystem and operating-system identity, so running unrelated untrusted tenants together remains risky even if the jobs have different workspaces.

End-to-end sequence

Durability and CPS nuance

Pipeline's Groovy execution model supports persisted program state across some controller restarts. It is not equivalent to durable arbitrary agent processes: the step, agent availability, workspace and external commands determine what resumes safely. Pipeline CPS continuation state and build records create controller I/O and CPU demand; doing large data transformations in Groovy on the controller is a common scaling mistake. Offload heavy parsing or dependency operations to agents via sh or language-specific executables.

pipeline {
  agent none
  options { timestamps(); timeout(time: 30, unit: 'MINUTES') }
  stages {
    stage('Verify') {
      agent { label 'linux-small' }
      steps {
        checkout scm
        sh 'make test'
      }
    }
  }
}

This requires a configured eligible agent and a repository with a real test target.

Production incident: agent nodes are online, builds remain queued

Check why an item is queued before adding capacity: labels, node mode, offline cause, executor busy state, throttling/resource locks, or cloud provisioning errors. Increasing controller CPU does not resolve a missing label. Likewise, adding agent Pods is ineffective if the Kubernetes API denies creation or the agent cannot connect to the controller.

# Read-only Linux diagnostics on an authorized controller host:
systemctl status jenkins
journalctl -u jenkins --since '15 minutes ago' --no-pager
df -h
df -i
# In the UI: Build Queue -> item -> 'Why is it not running?'
# In Manage Jenkins -> Nodes: labels, online status, executor utilization

The service commands apply to a systemd deployment, not every container or Helm installation.

Trade-offs and platform leadership

Jenkins is flexible but carries stateful controller, plugin and upgrade responsibilities. Separate trusted release agents from untrusted PR agents; use ephemeral agents where appropriate; define retention and disaster-recovery objectives for the controller. Treat agent capacity and controller throughput as different resource budgets.

Spoken explanation

“Jenkins has a controller responsible for scheduling, pipeline coordination, auth, plugins and records; the agents execute actual workloads through executors and workspaces. A webhook queues a versioned job, Jenkins matches it to an eligible agent, runs build/test/publish stages, and persists logs and results. I would keep executors off the controller, isolate trust levels across agents, and diagnose queue constraints separately from controller performance. The controller is stateful, so backup, I/O and restart recovery remain essential.”

References: Using Jenkins agents · Managing nodes · Scaling Pipelines