> ## Knowledge Base Index > Fetch the complete knowledge base index at: https://support.stacktrack.com/sitemap.xml > Use this file to discover available pages before exploring further. > Pure-Markdown content can be obtained by appending a '.md' suffix to the content URLs listed in the sitemap (without the trailing slash). # Improving Pipeline Restartability, Reliability, and Resource Efficiency # Improving Pipeline Restartability, Reliability, and Resource Efficiency ### Overview Jenkins pipelines are increasingly being used to orchestrate complex workflows including builds, deployments, regression testing, validation activities, and downstream pipeline execution. As these workflows grow in complexity and duration, it becomes important to ensure they are resilient, maintainable, and efficient to operate. Many existing pipelines rely on a global agent configuration, where all stages execute within the same workspace and execution context. While this simplifies implementation, it can reduce operational flexibility, increase recovery time after failures, and result in unnecessary resource allocation during periods where no active work is being performed. This document proposes a standardized pipeline design approach that improves restartability, maintainability, and execution efficiency while aligning with Jenkins best practices. # Objectives The proposed approach aims to: * Improve pipeline restartability and recovery. * Reduce unnecessary re-execution of completed stages. * Promote efficient use of Jenkins agents. * Improve pipeline maintainability. * Support scalable orchestration workflows. * Establish consistent engineering standards across pipelines. # Current Challenges ## Global Agent Usage A common pattern is: ```groovy pipeline { agent { label 'build-agent' } stages { ... } } ``` In this model: * All stages inherit the same execution context. * The pipeline remains tied to a single agent allocation. * Idle stages continue consuming resources. * Recovery from failures often requires re-running the entire pipeline. --- ## Limited Restart Capability Long-running pipelines often contain multiple independent stages. Examples include: * Build * Package * Deploy * Regression execution * Reporting When a later stage fails, operators may be required to restart the entire pipeline even when earlier stages completed successfully. This can result in: * Increased execution time * Repeated downstream executions * Additional operational overhead --- ## Approval and Waiting Stages Holding Agents Manual approval stages and waiting periods frequently retain build agents even though no active work is being performed. Examples: * Deployment approvals * Production change approvals * External dependency waits * Release sign-offs This can unnecessarily reduce available build capacity. --- ## Workspace Dependency Between Stages Many pipelines assume that artifacts created in one stage will automatically be available in subsequent stages. This assumption becomes unreliable when stages execute on different agents. Without explicit artifact management, stage-level execution can introduce failures and inconsistencies. --- # Recommended Architecture ## Pipeline-Level Agent Configuration Use: ```groovy pipeline { agent none } ``` as the default configuration for orchestration pipelines. This ensures that no build resources are allocated unless explicitly required. ### Benefits * Improved pipeline flexibility. * Better resource utilization. * Clear separation of execution responsibilities. * Better support for stage restart and recovery. --- # Stage-Level Agent Allocation Allocate agents only when actual work is being performed. ## Example ```groovy stage('Build') { agent { label 'build-agent' } steps { sh './gradlew build' } } ``` ```groovy stage('Deploy') { agent { label 'deploy-agent' } steps { sh './deploy.sh' } } ``` ### Benefits * Agents are allocated only when required. * Stages can use specialized execution environments. * Improved scalability and maintainability. --- # Approval and Waiting Stages Stages that do not require a workspace should execute without agents. ## Example ```groovy stage('Approve Deployment') { agent none steps { timeout(time: 30, unit: 'MINUTES') { input 'Proceed with deployment?' } } } ``` ### Benefits * No idle agent consumption. * Improved platform efficiency. * Reduced resource contention. --- # Pipeline Timeout Controls All pipelines should define reasonable timeout limits. ## Pipeline Timeout ```groovy pipeline { agent none options { timeout(time: 2, unit: 'HOURS') } } ``` ## Stage Timeout ```groovy stage('Deploy') { options { timeout(time: 20, unit: 'MINUTES') } steps { ... } } ``` ### Benefits * Prevents abandoned executions. * Improves platform stability. * Reduces operational overhead. --- # Managing Dependencies Between Stages When stages execute on different agents, workspace contents should not be assumed to persist across stages. Any files required by downstream stages should be explicitly transferred. ## Using Stash and Unstash ### Build Stage ```groovy stage('Build') { agent { label 'build-agent' } steps { sh './gradlew build' stash( name: 'build-output', includes: 'build/**/*' ) } } ``` ### Test Stage ```groovy stage('Test') { agent { label 'test-agent' } steps { unstash 'build-output' sh './gradlew test' } } ``` ### Benefits * Eliminates dependency on shared workspaces. * Enables stages to run on different agents. * Improves restartability. * Increases portability and reliability. --- # Triggering Downstream Jobs and Pipelines Many Jenkins pipelines act purely as orchestrators and are responsible for triggering other jobs or pipelines. Examples include: * Regression orchestration * Deployment orchestration * Environment validation workflows * Release pipelines These orchestration activities generally do not require a workspace and therefore do not require a Jenkins agent. ## Recommended Pattern ```groovy stage('Regression Suite A') { agent none steps { build( job: 'regression-suite-a', wait: true, propagate: true ) } } ``` The Jenkins `build` step executes on the controller and simply schedules downstream work. Since no build activity occurs within the stage itself, an agent allocation is unnecessary. --- ## Passing Parameters Parameters can be passed explicitly to downstream jobs. ```groovy stage('Deploy Application') { agent none steps { build( job: 'application-deploy', wait: true, propagate: true, parameters: [ string(name: 'ENVIRONMENT', value: 'qa'), string(name: 'VERSION', value: env.BUILD_TAG) ] ) } } ``` --- ## Executing Multiple Downstream Pipelines Each downstream pipeline should be represented as a separate stage. ```groovy stage('Regression Suite A') { agent none steps { build job: 'regression-suite-a' } } stage('Regression Suite B') { agent none steps { build job: 'regression-suite-b' } } ``` Benefits include: * Clear visibility of execution progress. * Easier troubleshooting. * Improved stage restart capabilities. * Better separation of responsibilities. --- ## Parallel Execution Independent downstream jobs may be executed in parallel. ```groovy stage('Execute Regression Suites') { parallel { stage('Suite A') { agent none steps { build job: 'regression-suite-a' } } stage('Suite B') { agent none steps { build job: 'regression-suite-b' } } } } ``` This can significantly reduce overall execution time. --- ## Handling Downstream Failures Default behavior: ```groovy build( job: 'regression-suite', wait: true, propagate: true ) ``` The parent pipeline fails if the downstream job fails. For custom handling: ```groovy def result = build( job: 'regression-suite', wait: true, propagate: false ) echo "Result: ${result.result}" ``` This allows orchestration pipelines to: * Aggregate results. * Generate consolidated reports. * Implement custom retry strategies. * Continue execution based on business requirements. --- # Pipeline Restartability One of the primary benefits of stage-oriented execution is improved recovery from failures. To maximize restartability: * Keep stages independent. * Avoid shared workspace assumptions. * Explicitly transfer artifacts. * Use stage-level execution boundaries. * Separate orchestration from execution logic. Examples of restart-friendly stages include: * Build * Package * Deploy QA * Regression Suite A * Regression Suite B * Publish Results This structure enables operators to restart from a failed stage rather than rerunning the entire workflow. --- # Migration Approach Existing pipelines can be migrated incrementally. ## Phase 1 Introduce timeout controls. ```groovy options { timeout(time: 2, unit: 'HOURS') } ``` ## Phase 2 Move approval and waiting stages to: ```groovy agent none ``` ## Phase 3 Replace global pipeline agents with: ```groovy agent none ``` and introduce stage-level agents. ## Phase 4 Implement `stash` and `unstash` where stage dependencies exist. ## Phase 5 Validate restart and recovery behavior. --- # Recommended Standards The following standards should be adopted for all new orchestration pipelines and applied to existing pipelines where practical. 1. Use `agent none` at the pipeline level. 2. Allocate agents only to stages that perform actual work. 3. Execute approval and waiting stages using `agent none`. 4. Trigger downstream jobs and pipelines using `agent none`. 5. Configure pipeline-level timeout controls. 6. Configure stage-level timeout controls where appropriate. 7. Use `stash` and `unstash` (or approved artifact storage) when files must be transferred between stages. 8. Avoid assumptions that stages will execute on the same workspace or agent. 9. Keep stages focused on a single responsibility. 10. Explicitly define stage inputs and outputs. 11. Design downstream jobs to be independently executable and reusable. 12. Use separate stages for independent downstream pipeline executions. --- # Expected Benefits | Area | Benefit | | ---- | | Reliability | Improved recovery from failures | | Restartability | Reduced need for full pipeline reruns | | Maintainability | Consistent pipeline structure | | Scalability | Better support for large orchestration workflows | | Resource Utilization | Agents allocated only when required | | Operational Efficiency | Reduced platform overhead | | Developer Productivity | Faster troubleshooting and recovery | --- # Conclusion Adopting a stage-oriented execution model with pipeline-level `agent none`, stage-level agent allocation, explicit artifact management, and standardized timeout controls provides a scalable and maintainable foundation for Jenkins pipelines. This approach improves restartability, enhances operational efficiency, reduces unnecessary agent allocation, and establishes a consistent engineering standard for future pipeline development across the platform. For orchestration pipelines, stages that perform only control-flow activities such as approvals, notifications, waiting, or triggering downstream jobs should execute using `agent none`, while agents should be reserved exclusively for stages that require a workspace or perform computational work.