Spawning Processes From Build

It is possible to spawn a process from a build and have that process live longer than the build itself. For example, perhaps the build launches a new application server with the result of the build. In older releases, the build often did not terminate. Instead, the specific step (such as the shell script, Ant, or Maven) terminates but the build itself does not terminate.

Jenkins detects this situation and, instead of blocking indefinitely, prints out a warning and terminates the build.

Why?

This happens because of how file descriptors are used between processes in a build. Jenkins and the child process are connected by three pipes (stdin, stdout, and stderr.) This allows Jenkins to capture the output from the child process. The child process may write a lot of data to the pipe and quit immediately after that, so Jenkins waits for end-of-file (EOF) to be sure that it has drained the pipes before it terminates the build.

Whenever a process terminates, the operating system closes all the file descriptors it owned. So, even if the process did not close stdout and stderr, Jenkins gets end of file (EOF).

The complication happens when those file descriptors are inherited by other processes. Let’s say the child process forks another process to the background. The background process (which is actually a daemon) inherits all the file descriptors of the parent, including the writing side of the stdout ad stderr pipes that connect the child process and Jenkins. If the daemon forgets to close them, Jenkins does not get EOF for pipes even when the child process exits, because the daemon still has those descriptors open. This is how this problem happens.

A daemon should close all file descriptors to avoid such issues but some daemons do not follow the rule. You can mitigate this problem with various workarounds.

Workarounds

On Unix, you can use a wrapper like this to make the daemon behave. For example:

daemonize -E BUILD_ID=dontKillMe /path/to/your/command

In a Jenkins Pipeline, use JENKINS_NODE_COOKIE instead of BUILD_ID.

Note that this will set the BUILD_ID environment variable for the process being spawned to something other than the current BUILD_ID. Or you can start jenkins with -Dhudson.util.ProcessTree.disable=true.

On Windows, use the 'at' command to launch a process in the background. For example:

<scriptdef name="get-next-minute" language="beanshell">
  <attribute name="property" />

  date = new java.text.SimpleDateFormat("HH:mm")
    .format(new Date(System.currentTimeMillis() + 60000));
  project.setProperty(attributes.get("property"), date);
</scriptdef>

<get-next-minute property="next-minute" />
<exec executable="at">
  <arg value="${next-minute}" />
  <arg value="/interactive" />
  <arg value="${jboss.home}\bin\run.bat" />
</exec>

Another similar workaround on Windows is to use a wrapper script and launch your program through it:

// antRunAsync.js - Wrapper script to run an executable detached in the
// background from Ant's <exec> task.  This works by running the executable
// using the Windows Scripting Host WshShell.Run method which doesn't copy
// the standard filehandles stdin, stdout and stderr. Ant finds them closed
// and doesn't wait for the program to exit.
//
// requirements:
//   Windows Scripting Host 1.0 or better.  This is included with Windows
//   98/Me/2000/XP.  Users of Windows 95 or Windows NT 4.0 need to download
//   and install WSH support from
//   http://msdn.microsoft.com/scripting/.
//
// usage:
// <exec executable="cscript.exe">
//   <env key="ANTRUN_TITLE" value="Title for Window" />  <!-- optional -->
//   <env key="ANTRUN_OUTPUT" value="output.log" />  <!-- optional -->
//   <arg value="//NoLogo" />
//   <arg value="antRunAsync.js" />  <!-- this script -->
//   <arg value="real executable" />
// </exec>


var WshShell = WScript.CreateObject("WScript.Shell");
var exeStr = "%comspec% /c";
var arg = "";
var windowStyle = 1;
var WshProcessEnv = WshShell.Environment("PROCESS");
var windowTitle = WshProcessEnv("ANTRUN_TITLE");
var outputFile = WshProcessEnv("ANTRUN_OUTPUT");
var OS = WshProcessEnv("OS");
var isWindowsNT = (OS == "Windows_NT");

// On Windows NT/2000/XP, specify a title for the window.  If the environment
// variable ANTRUN_TITLE is specified, that will be used instead of a default.
if (isWindowsNT) {
  if (windowTitle == "")
     windowTitle = "Ant - " + WScript.Arguments(i);
  exeStr += "title " + windowTitle + " &&";
}

// Loop through arguments quoting ones with spaces
for (var i = 0; i < WScript.Arguments.count(); i++) {
  arg = WScript.Arguments(i);
  if (arg.indexOf(' ') > 0)
    exeStr += " \"" + arg + "\"";
  else
    exeStr += " " + arg;
}

// If the environment variable ANTRUN_OUTPUT was specified, redirect
// output to that file.
if (outputFile != "") {
  windowStyle = 7;  // new window is minimized
  exeStr += " > \"" + outputFile + "\"";
  if (isWindowsNT)
    exeStr += " 2>&1";
}

// WScript.Echo(exeStr);
// WshShell.Run(exeStr);
WshShell.Run(exeStr, windowStyle, false);

This self-provided wrapper script can be called from Ant for example:

<exec executable="cscript.exe">
   <env key="ANTRUN_TITLE" value="Title for Window" />  <!-- optional -->
   <env key="ANTRUN_OUTPUT" value="output.log" />  <!-- optional -->
   <arg value="//NoLogo" />
   <arg value="antRunAsync.js" />  <!-- this script -->
   <arg value="real executable" />
</exec>

Another workaround for Windows is to schedule a permanent task and force running it from the Ant script. For example, run the command:

C:\>SCHTASKS /Create /RU SYSTEM /SC ONSTART /TN Tomcat /TR
"C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\startup.bat"

Note, that ONSTART can be replaced with ONCE if you do not want to keep Tomcat running. Add the following code to your Ant script:

<exec executable="SCHTASKS">
    <arg value="/Run"/>
    <arg value="/TN"/>
    <arg value="Tomcat"/>
</exec>

User Handbook

Tutorials

Resources

Spawning Processes From Build

Why?

Workarounds