Kubernetes Agents, Provisioning, Capacity and Workspace Isolation

Engineering scenario: Jobs queue even though Kubernetes nodes have free CPU. Explain how an ephemeral Jenkins agent is provisioned, why it may never connect, and when to choose static VMs over Pods.

Architecture and critical path

The Jenkins Kubernetes plugin provisions Pods as inbound agents. The Pod typically contains a special agent container (commonly named jnlp) and optional build containers. The Pod must become schedulable, pull its images, start the remoting process and connect to the Jenkins controller before an executor can perform work. The controller may run outside Kubernetes.

Total time-to-executor includes Jenkins queue policy + cloud provisioning + Pod scheduling + image download + startup + connection handshake. Cluster CPU headroom does not help if no Pod can be created due to RBAC, quota, taints, lack of matching nodes, image pull limits or missing network path.

Practical Pipeline using a pod template

podTemplate(
    containers: [
        containerTemplate(name: 'tools',
                          image: 'python:3.12-slim',
                          command: 'sleep',
                          args: '99d')
    ]) {
    node(POD_LABEL) {
        container('tools') {
            checkout scm
            sh 'python --version'
            sh 'python -m unittest discover -v'
        }
    }
}

Requires an installed Kubernetes plugin, configured cloud, inbound agent image (provided by plugin/template), executable sleep, and repo tests. It's a generic example, not a runnable standalone service. Pin container images to immutable digests in production.

A more declarative Jenkins Kubernetes agent is also possible using a compatible plugin and agent { kubernetes { yaml ... } }. Choose one idiom for consistency, not because one automatically avoids all provisioning or security problems.

Workspaces and filesystem

Containers inside one Pod can share an emptyDir workspace. This is temporary and can disappear with the Pod. stash/external artifacts must be used for outputs needed by a different stage or node. Avoid PVC reuse between untrusted jobs unless tenant isolation and ownership are designed. A writable Docker socket grants powerful host control; do not mount it into untrusted build Pods.

Capacity model

effective_start_delay =
  cloud provisioning + scheduler wait + image pull + agent connect

sustainable_parallelism <= min(
  Jenkins policy/executors,
  available eligible agent Pods,
  node CPU/memory/ephemeral-storage capacity,
  Kubernetes quotas, API provisioning throughput,
  downstream test DB/registry/rate limits
)

Cold image pulls, DNS failures, a controller endpoint unavailable from the cluster, or stalled admission webhook can all cause agents to remain offline while node metrics look healthy.

Inspection commands

kubectl get pods -n ci -o wide
kubectl get events -n ci --sort-by=.lastTimestamp
kubectl describe pod EXAMPLE_AGENT_POD -n ci
kubectl logs -n ci EXAMPLE_AGENT_POD -c jnlp --tail=100
kubectl get resourcequota -n ci
kubectl get nodes -L topology.kubernetes.io/zone

Never dump environment variables from a real agent Pod; they can include temporary authentication material. Use restricted access to CI namespaces because build artifacts and logs may contain sensitive data.

Hypothetical outage and recovery

An unrelated infrastructure change alters the controller URL. Kubernetes agent Pods start successfully but cannot resolve or connect to the remoting endpoint. Queue length grows and Pod images appear healthy. Inspect the agent container's connection error, Jenkins cloud configuration, proxy/TLS/DNS and network policies. Restarting every agent without fixing the endpoint just burns capacity.

Platform trade-offs

Static VMs provide stable caches and specialized hardware but accumulate drift and need patching. Ephemeral Pods improve isolation and clean state but introduce provisioning delays, image-pull cost and dependency on Kubernetes API/control plane. Select separate build pools for untrusted PR code, privileged signing and special architectures, and validate actual cold-start/recovery SLO.

Spoken explanation

“An agent Pod must pass several stages before Jenkins sees an executor: API creation, scheduling, image pull, remoting connection and agent registration. I'd inspect the reason for Pending Pods or failed inbound connections before increasing replicas. Ephemeral agents help reproducibility and isolation, but they need carefully scoped RBAC, image caching, workspace policy and capacity headroom.”

References: Jenkins Kubernetes plugin · Jenkins agents · Kubernetes Pod lifecycle