Jenkins: How to terminate a zombie build (stuck in the queue)

When working with Jenkins, one of the common issues that users may face is dealing with zombie builds. These are builds that appear to be stuck in a queue and do not progress to completion nor fail, making it hard to maintain an efficient CI/CD pipeline. This article will guide you through different strategies to identify and terminate such builds, ensuring that your Jenkins server remains healthy and responsive.

Understanding Zombie Builds

Before diving into the solutions, it’s essential to understand what zombie builds are and how they occur. A build becomes a “zombie” when Jenkins marks it as running, but it’s no longer executing any tasks. These builds can consume valuable system resources and block the queue, preventing other builds from starting. Various factors, such as network issues, resource constraints, or plugin bugs, can cause builds to become zombies.

Basic Solutions

The first step in resolving zombie builds is to identify them. You can use the Jenkins UI to manually inspect the queue and the list of running builds. However, for a more programmatic approach, you can use the Jenkins Script Console. Here’s a basic script to list all active builds:

Jenkins.instance.getAllItems(AbstractProject.class).each { p ->
  p.builds.findAll { it.isBuilding() }.each {
    println "${p.name} #${it.number}"
  }
}

This code snippet will print the name and number of all currently building projects. If you spot any builds that have been running for an unusually long time, they might be zombie builds.

Terminating Zombie Builds

Once you’ve identified a potential zombie build, the next step is to try and terminate it. Jenkins provides a few ways to do this:

1. Using the Jenkins UI: Navigate to the specific build page and click on the ‘Stop Build’ button. This is the most straightforward method but may not always work for zombie builds.
2. Script Console: You can use the following script to forcefully terminate a build via the Script Console:

Jenkins.instance.getItemByFullName("YourJobNameHere")
  .getBuildByNumber(yourBuildNumberHere)
  .finish(hudson.model.Result.ABORTED, new java.io.IOException("Build terminated."))

Replace YourJobNameHere and yourBuildNumberHere with the respective job name and build number. This script will set the build’s result to ABORTED and log a custom message indicating the termination.

Advanced Solutions

For cases where the basic solutions don’t suffice, you may need to resort to more advanced techniques. One approach is to restart the Jenkins server, but this can be disruptive. A more elegant solution involves using the Jenkins CLI or REST API to programmatically manage builds.

Here’s a CLI example to abort a build:

 java -jar jenkins-cli.jar -s http://your-jenkins-url:port/ build stop YourJobNameHere yourBuildNumberHere 

Similarly, you can use the REST API:

 curl -X POST http://your-jenkins-url:port/job/YourJobNameHere/yourBuildNumberHere/stop 

These methods provide more flexibility and can be integrated into scripts for automated build management.

Cleaning Up

After terminating zombie builds, it’s a good practice to check for any underlying issues that caused the build to become unresponsive. Investigate build logs, network connectivity, server health, and plugin compatibility. Ensuring these factors are in optimal condition can help prevent future occurrences of zombie builds.

Conclusion

Zombie builds in Jenkins can disrupt your CI/CD pipeline and consume valuable system resources. By using the strategies outlined in this tutorial, you can effectively identify and terminate these builds, helping maintain the efficiency of your Jenkins server. Remember, prevention is key—regularly monitor your system and address any underlying issues to mitigate the risk of zombie builds.