Jenkins Pipeline Syntax, Groovy Execution and Reliable Stage Design

Engineering scenario: Design a build-and-test Jenkinsfile that is readable, bounded, repeatable and recoverable. Explain how Declarative syntax differs from Scripted Pipeline, and why a Groovy script is not the same as an ordinary shell script.

Pipeline concepts from first principles

A Jenkinsfile defines a Pipeline as code. Declarative Pipeline offers structured pipeline { agent; options; environment; stages; post } syntax with controlled escape to script { }. Scripted Pipeline exposes a Groovy-based DSL where node, stage and steps are composed programmatically. Both invoke Jenkins Pipeline steps, not arbitrary Groovy functions running within build containers.

A practical Declarative Jenkinsfile

pipeline {
  agent none
  options {
    timestamps()
    timeout(time: 45, unit: 'MINUTES')
    disableConcurrentBuilds()
    buildDiscarder(logRotator(numToKeepStr: '30'))
  }
  stages {
    stage('Validate') {
      agent { label 'linux-small' }
      steps {
        checkout scm
        sh 'set -eu; ./scripts/lint.sh'
        stash name: 'verified-source', includes: '**/*', useDefaultExcludes: false
      }
    }
    stage('Test') {
      parallel {
        stage('Unit') {
          agent { label 'linux-small' }
          steps {
            unstash 'verified-source'
            sh './scripts/unit-test.sh'
          }
        }
        stage('Integration') {
          agent { label 'linux-large' }
          steps {
            unstash 'verified-source'
            sh './scripts/integration-test.sh'
          }
        }
      }
    }
  }
  post {
    always {
      echo 'Record result, links and any follow-up actions'
    }
    failure {
      echo 'Notify responsible team by an approved channel'
    }
  }
}

Illustrative only: real repositories must contain the scripts; each agent needs compatible tools. stash/unstash is convenient for small file sets, not a substitute for a large artifact repository. For large sources/artifacts, use a registry/object store or re-checkout a fixed SCM revision. The example's broad stash includes can be expensive and should be restricted to what consumers require.

Stage options and ordering

Directive / step	What it does	Avoid
agent none	No one global executor is reserved	Assuming stages inherit a workspace
when	Conditional stage execution	Using a PR-supplied parameter as deploy authorization
parallel / matrix	Independent execution variants	Sharing mutable workspace between branches
timeout	Bounds stage or pipeline time	Treating timeout as proof external side effect failed
retry	Repeats failed block	Retrying non-idempotent uploads/payment jobs
post	Cleanup/reporting after stage/pipeline	Swallowing the original error in cleanup
catchError	Adjusts recorded failure behavior	Marking a required validation failure successful
stash	Captures files for later steps	Moving gigabytes through controller storage

What CPS changes

Groovy CPS transforms much Pipeline code into resumable continuations. Local variables surviving pause/resumption generally need serializable values. A Java/Groovy iterator, regex matcher or streaming object held across a step can trigger serialization errors. @NonCPS permits some pure transformations without CPS overhead, but must not call Pipeline steps such as sh or node. Keep heavyweight computation on agents and limit controller-side Groovy to orchestration.

@NonCPS
def rankCounts(Map<String, Integer> counts) {
  counts.entrySet().sort { a, b -> b.value <=> a.value }
}
// Call this pure method with plain data.
// Do not call sh(), node(), or other Pipeline steps inside it.

Production failure

A Pipeline runs an expensive JSON parser in a large Groovy loop, causing high controller CPU while agents remain mostly idle. Moving parsing to a Python/shell command executed on an agent removes controller-side interpreter overhead and produces more testable code. Avoid treating a higher heap as the first and only fix.

Spoken explanation

“I'd default to Declarative syntax for consistency, define bounded stages, use stage-scoped agents and keep validations isolated from release authority. Parallel stages should share only immutable source revisions, not a mutable workspace. CPS means Pipeline Groovy is an orchestration layer with serialization limits, so I would push real computation into agent processes. Post blocks capture evidence without suppressing failed gates.”

References: Pipeline syntax · CPS limitations · Scaling Pipelines