Reverse Proxy Configuration

A 'reverse proxy' allows an alternate HTTP or HTTPS provider to communicate with web browsers on behalf of Jenkins. The alternate provider may offer additional capabilities, like SSL encryption. The alternate provider may offload some work from Jenkins, like delivering static images.

General Guidelines

Jenkins actively monitors reverse proxy configuration. Jenkins reports "Your reverse proxy setup is broken" when it detects a reverse proxy configuration problem. See the Troubleshooting section if Jenkins is reporting that your reverse proxy setup is broken.

Background

Reverse proxies receive inbound HTTP requests and forward those requests to Jenkins. It receives the outbound HTTP response from Jenkins and forwards those requests to the original requester. A correctly configured reverse proxy rewrites both the HTTP request and the HTTP response.

When HTTP request rewriting is misconfigured, pages won’t be displayed at all. Refer to configuration examples if your reverse proxy is not displaying any Jenkins pages.

A reverse proxy must handle the HTTP response by either rewriting the response or setting HTTP headers on the forwarded request. When HTTP response handling is misconfigured, Jenkins may fail to show updated information on a page or it may ignore changes submitted through web pages. See the Troubleshooting section if Jenkins is reporting that your reverse proxy setup is broken or pages are not behaving as expected.

Configuration Examples

Jenkins works with many different reverse proxies. This page provides examples for specific reverse proxies, though much of the information also applies to other reverse proxies.

Running Jenkins behind Apache

In situations where you have existing web sites on your server, you may find it useful to run Jenkins (or the servlet container that Jenkins runs in) behind Apache, so that you can bind Jenkins to the part of a bigger website that you may have. This section discusses some of the approaches for doing this.

Make sure that you change the Jenkins httpListenAddress from its default of 0.0.0.0 to 127.0.0.1 or any Apache-level restrictions can be easily bypassed by accessing the Jenkins port directly.

There are several different alternatives to configure Jenkins with Apache. Choose the technique that best meets your needs:

mod_proxy

mod_proxy works by making Apache perform "reverse proxy" — when a request arrives for certain URLs, Apache becomes a proxy and forwards that request to Jenkins, then forwards the response from Jenkins back to the client.

The following Apache modules must be installed :

a2enmod proxy
a2enmod proxy_http
a2enmod headers

A typical set up for mod_proxy would look like this:

ProxyPass         /jenkins  http://localhost:8081/jenkins nocanon
ProxyPassReverse  /jenkins  http://localhost:8081/jenkins
ProxyRequests     Off
AllowEncodedSlashes NoDecode

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default
# See /etc/apache2/mods-enabled/proxy.conf in Ubuntu
<Proxy http://localhost:8081/jenkins*>
  Order deny,allow
  Allow from all
</Proxy>

This assumes that you run Jenkins on port 8081.

For this set up to work, the context path of Jenkins must be the same between your Apache and Jenkins (that is, you can’t run Jenkins on http://localhost:8081/ci and have it exposed at http://localhost:80/jenkins).
Set the context path in Windows by modifying the jenkins.xml configuration file and adding --prefix=/jenkins (or similar) to the <arguments> entry.
Set the context path when using the Ubuntu package by adding --prefix=/jenkins to JENKINS_ARGS in /etc/default/jenkins ( or in /etc/sysconfig/jenkins for RHEL/CentOS package)

When running on a dedicated server and you are using / as context, make sure you add a slash at the end of all URLs in proxy params in apache. Otherwise you might run into proxy errors. So

ProxyPass / http://localhost:8080/ nocanon

instead of

ProxyPass / http://localhost:8080 nocanon     # wont work

Note that this does not apply to the ProxyPassMatch directive, which behaves differently than ProxyPass. Below is an example of ProxyPassMatch to proxy all URLs other than /.well-known (a URL required by letsencrypt):

ProxyPassMatch  ^/(?\!.well-known)  http://localhost:8080 nocanon

The ProxyRequests Off prevents Apache from functioning as a forward proxy server (except for ProxyPass), it is advised to include it unless the server should function as a proxy.

Both the nocanon option to ProxyPass, and AllowEncodedSlashes NoDecode, are required for certain Jenkins features to work.

If you are running Apache on a Security-Enhanced Linux (SE-Linux) machine it is essential to make SE-Linux do the right thing by issuing as root

setsebool -P httpd_can_network_connect true

If this is not issued Apache will not be allowed to forward proxy requests to Jenkins and only an error message will be displayed.

Because Jenkins already compress its output, you can not use the normal proxy-html filter to modify urls:

SetOutputFilter proxy-html

Instead you can use the following:

SetOutputFilter INFLATE;proxy-html;DEFLATE
ProxyHTMLURLMap http://your_server:8080/jenkins /jenkins

But since Jenkins seems to be well behaved it’s even better to just not use SetOutputFilter and ProxyHTMLURLMap.

If there are problems with Jenkins sometimes servicing random garbage pages, then the following may help:

SetEnv proxy-nokeepalive 1

Some plug-ins determine URLs from client requests from Host header, so if you experience some problems with wrong URLs, you can try to switch on ProxyPreserveHost directive, which is switched off by default:

ProxyPreserveHost On
mod_proxy with HTTPS

You can add an additional ProxyPassReverse directive to redirect non-SSL URLs generated by Jenkins to the SSL side. Assuming that your webserver is your.host.com, placing the following within the SSL virtual host definition will do the trick:

ProxyRequests     Off
ProxyPreserveHost On
AllowEncodedSlashes NoDecode

<Proxy http://localhost:8081/jenkins*>
  Order deny,allow
  Allow from all
</Proxy>

ProxyPass         /jenkins  http://localhost:8081/jenkins nocanon
ProxyPassReverse  /jenkins  http://localhost:8081/jenkins
ProxyPassReverse  /jenkins  http://your.host.com/jenkins

Yet another option is to rewrite the Location headers that contain non-ssl URL’s generated by Jenkins. If you want to access Jenkins from https://www.example.com/jenkins, placing the following within the SSL virtual host definition also works:

ProxyRequests     Off
ProxyPreserveHost On
ProxyPass /jenkins/ http://localhost:8081/jenkins/ nocanon
AllowEncodedSlashes NoDecode

<Location /jenkins/>
  ProxyPassReverse /
  Order deny,allow
  Allow from all
</Location>

Header edit Location ^http://www.example.com/jenkins/ https://www.example.com/jenkins/

But it may also work fine to just use simple forwarding as above (the first HTTPS snippet), and add

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

in the HTTPS site configuration, as the Docker demo (below) does. (X-Forwarded-Port is not interpreted by Jenkins prior to JENKINS-23294 so it may also be desirable to configure the servlet container to specify the originating port.)

NameVirtualHost *:80
NameVirtualHost *:443

<VirtualHost *:80>
    ServerAdmin  webmaster@localhost
    Redirect permanent / https://www.example.com/
</VirtualHost>

<VirtualHost *:443>
    SSLEngine on
    SSLCertificateFile /etc/ssl/certs/cert.pem
    ServerAdmin  webmaster@localhost
    ProxyRequests     Off
    ProxyPreserveHost On
    AllowEncodedSlashes NoDecode
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyPass         /  http://localhost:8080/ nocanon
    ProxyPassReverse  /  http://localhost:8080/
    ProxyPassReverse  /  http://www.example.com/
    RequestHeader set X-Forwarded-Proto "https"
    RequestHeader set X-Forwarded-Port "443"
</VirtualHost>

mod_ajp or mod_proxy_ajp

Jenkins can be configured to use mod_ajp or mod_proxy_ajp so that Jenkins runs in a different workspace than the typical Tomcat server, but both are available via the Apache web server.

Configure Jenkins to use a different web and ajp port than Tomcat:

HTTP_PORT=9080
AJP_PORT=9009
...
nohup java -jar "$WAR" \
           --httpPort=$HTTP_PORT \
           --ajp13Port=$AJP_PORT \
           --prefix=/jenkins >> "$LOG" 2>&1 &

Then setup Apache so that it knows that the prefix /jenkins is being served by AJP in the httpd.conf file:

LoadModule jk_module          libexec/httpd/mod_jk.so

AddModule     mod_jk.c

#== AJP hooks ==
JkWorkersFile /etc/httpd/workers.properties
JkLogFile     /private/var/log/httpd/mod_jk.log
JkLogLevel    info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions     +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat     "%w %V %T"
# Here are 3 sample applications - 2 that are being served by Tomcat, and Jenkins
JkMount  /friki/* worker1
JkMount  /pebble/* worker1
JkMount  /jenkins/* worker2

Then finally the workers.conf file specified above, that just tells AJP which port to use for which web application:

# Define 2 real workers using ajp13
worker.list=worker1,worker2
# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=50
worker.worker1.cachesize=10
worker.worker1.cache_timeout=600
worker.worker1.socket_keepalive=1
# Set properties for worker2 (ajp13)
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=9009
worker.worker2.lbfactor=50
worker.worker2.cachesize=10
worker.worker2.cache_timeout=600
worker.worker2.socket_keepalive=1
worker.worker2.recycle_timeout=300

mod_proxy_ajp with SSL

AJP is an arguably cleaner alternative for an SSL-enabled reverse proxy, since Jenkins will get all pertinent HTTP headers untouched. Configuration is a snap too, in three simple steps:

1. Configure an AJP port for Jenkins (as mentioned above)

HTTP_PORT=-1
AJP_PORT=9009
...
nohup java -jar "$WAR" \
           --httpPort=$HTTP_PORT \
           --ajp13Port=$AJP_PORT \
           --prefix=/jenkins >> "$LOG" 2>&1 &

2. Enable mod_proxy_ajp in Apache:

# a2enmod proxy_ajp

3. Include the following snippet in your SSL-enabled VirtualHost:

<VirtualHost *:443>
...
    SSLEngine on
...
    AllowEncodedSlashes NoDecode
    ProxyRequests Off
    ProxyPass /jenkins ajp://localhost:9009/jenkins nocanon
</VirtualHost>

Note the use of AllowEncodedSlashes and ProxyPass...nocanon to persuade Apache to leave PATH_INFO alone.

mod_rewrite

The Apache mod_rewrite module can be used to configure an Apache reverse proxy for Jenkins.

The following Apache modules must be installed :

a2enmod rewrite
a2enmod proxy
a2enmod proxy_http

A typical mod_rewrite configuration would look like this:

# Use last flag because no more rewrite can be applied after proxy pass
RewriteRule       ^/jenkins(.*)$  http://localhost:8081/jenkins$1 [P,L]
ProxyPassReverse  /jenkins        http://localhost:8081/jenkins
ProxyRequests     Off

# Local reverse proxy authorization override
# Most unix distribution deny proxy by default
# See /etc/apache2/mods-enabled/proxy.conf in Ubuntu
<Proxy http://localhost:8081/jenkins*>
  Order deny,allow
  Allow from all
</Proxy>

This assumes that you run Jenkins on port 8081. For this set up to work, the context path of Jenkins must be the same between your Apache and Jenkins (that is, you can’t run Jenkins on http://localhost:8081/ci and have it exposed at http://localhost:80/jenkins)

The ProxyRequests Off prevents Apache from functioning as a forward proxy server (except for ProxyPass), it is advised to include it unless the server should function as a proxy.

Proxying CLI commands with the HTTP(S) transport

Using the plain CLI protocol with the HTTP(S) transport to access Jenkins through an Apache reverse proxy does not work. See JENKINS-47279 - Full-duplex HTTP(S) transport with plain CLI protocol does not work with Apache reverse proxy for more details. As a workaround, you can use the CLI over SSH.

If using Apache check that nocanon is set on ProxyPass and that AllowEncodedSlashes is set.

AllowEncodedSlashes is not inherited in Apache configs, so this directive must be placed inside the VirtualHost definition.

Running Jenkins behind Nginx

In situations where you have existing web sites on your server, you may find it useful to run Jenkins (or the servlet container that Jenkins runs in) behind Nginx, so that you can bind Jenkins to the part of a bigger website that you may have. This section discusses some of the approaches for doing this.

When a request arrives for certain URLs, Nginx becomes a proxy and further forward that request to Jenkins, then it forwards the response back to the client. A typical set up for mod_proxy would look like this:

upstream jenkins {
  keepalive 32; # keepalive connections
  server 127.0.0.1:8080; # jenkins ip and port
}

# Required for Jenkins websocket agents
map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}

server {
  listen          80;       # Listen on port 80 for IPv4 requests

  server_name     jenkins.example.com;

  # this is the jenkins web root directory
  # (mentioned in the /etc/default/jenkins file)
  root            /var/run/jenkins/war/;

  access_log      /var/log/nginx/jenkins/access.log;
  error_log       /var/log/nginx/jenkins/error.log;
  # pass through headers from Jenkins that Nginx considers invalid
  ignore_invalid_headers off;

  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    #rewrite all static files into requests to the root
    #E.g /static/12345678/css/something.css will become /css/something.css
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }

  location /userContent {
    # have nginx handle all the static requests to userContent folder
    #note : This is the $JENKINS_HOME dir
    root /var/lib/jenkins/;
    if (!-f $request_filename){
      #this file does not exist, might be a directory or a /**view** url
      rewrite (.*) /$1 last;
      break;
    }
    sendfile on;
  }

  location / {
      sendfile off;
      proxy_pass         http://jenkins;
      proxy_redirect     default;
      proxy_http_version 1.1;

      # Required for Jenkins websocket agents
      proxy_set_header   Connection        $connection_upgrade;
      proxy_set_header   Upgrade           $http_upgrade;

      proxy_set_header   Host              $host;
      proxy_set_header   X-Real-IP         $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   X-Forwarded-Proto $scheme;
      proxy_max_temp_file_size 0;

      #this is the maximum upload size
      client_max_body_size       10m;
      client_body_buffer_size    128k;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffering            off;
      proxy_request_buffering    off; # Required for HTTP CLI commands
      proxy_set_header Connection ""; # Clear for keepalive
  }

}

This assumes that you run Jenkins on port 8080. Remember to create the folder /var/log/nginx/jenkins.

For this set up to work, the context path of Jenkins must be the same between your Nginx and Jenkins (that is, you can’t run Jenkins on http://localhost:8081/ci and have it exposed at http://localhost:80/jenkins).

Set the context path by modifying the jenkins.xml configuration file and adding --prefix=/jenkins to the <arguments> entry.

If you are having problems with some paths (eg folders) with Blue Ocean, you may need to add the following snippet to your proxy configuration:

if ($request_uri ~* "/blue(/.*)") {
    proxy_pass http://YOUR_SERVER_IP:YOUR_JENKINS_PORT/blue$1;
    break;
}

To give Nginx permission to read Jenkins web root folder, add the nginx user to the Jenkins group:

useradd -aG jenkins nginx

If you are experiencing timeouts when attempting to run long CLI commands through a proxy in Jenkins, you can increase the proxy_read_timeout setting as necessary. Older versions of Jenkins may not respect the proxy_read_timeout setting.

If you are experiencing the following error when attempting to run long CLI commands in Jenkins and Jenkins is running behind Nginx, it is probably due to Nginx timing out the CLI connection. You can increase the proxy_read_timeout setting as necessary so the command will complete successfully.

WARNING: null
hudson.cli.DiagnosedStreamCorruptionException
Read back: 0x00 0x00 0x00 0x1e 0x07
           'Started reverse-proxy-test #68'
           0x00 0x00 0x00 0x01 0x07 0x0a
Read ahead:
Diagnosis problem:
    java.io.IOException: Premature EOF
        at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
        ...
    at hudson.cli.FlightRecorderInputStream.analyzeCrash(FlightRecorderInputStream.java:82)
    at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:153)
Caused by: java.io.IOException: Premature EOF
    at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
    ...
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)

Running Jenkins behind iptables

The default Jenkins installation runs on ports 8080 and 8443. Typically, HTTP/HTTPS servers run on ports 80 and 443, respectively. But these ports are considered privileged on Unix/Linux systems, and the process using them must be owned by root. Running Jenkins as root is not recommended - it should be run as its own user. One solution is to front Jenkins with a web server such as Apache, and let it proxy requests to Jenkins, but this requires maintaining the Apache installation as well. In situations where you are wanting to run Jenkins on port 80 or 443 (i.e. HTTP/HTTPS), but you do not want to setup a proxy server you can use iptables on Linux to forward traffic.

Ubuntu Installations

Follow the Ubuntu installation instructions to install and configure the initial Jenkins installation on Ubuntu 18.04 or later. These instructions are known to not work on Ubuntu versions prior to 18.04.

Prerequisites

In order to forward traffic from 80/443 to 8080/8443, first you must ensure that iptables has allowed traffic on all 4 of these ports. Use the following command to list the current iptables configuration:

 iptables -L -n

You should should see in the output entries for 80, 443, 8080,and 8443. Here is an example output for comparison.

ain INPUT (policy ACCEPT)target     prot opt source               destination
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:443
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8080
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:8443
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
target     prot opt source

If you dont see entries for these ports, then you need to run commands (as root or with sudo) to add those ports. For example, if you see none of these and need to add them all, you would need to issue the following commands:

sudo iptables -I INPUT 1 -p tcp --dport 8443 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 8080 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 443 -j ACCEPT
sudo iptables -I INPUT 1 -p tcp --dport 80 -j ACCEPT
NOTE

I used -I INPUT 1. In a lot of iptables documentation/examples, you will see -A INPUT. The difference is that -A appends to the list of rules, while -I INPUT 1 inserts before the first entry. Usually when adding new accept ports to iptables configuration, you want to put them at the beginning of the ruleset, not the end. Run iptables -L -n again and you should now see entries for these 4 ports.

Forwarding

Once traffic on the required ports are allowed, you can run the command to forward port 80 traffic to 8080, and port 443 traffic to 8443. The commands look like this:

sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080
sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8443

You can verify the forwarding rules using below command.

[root@xyz~]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:http redir ports 8080
REDIRECT   tcp  --  anywhere             anywhere             tcp dpt:https redir ports 8443

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
+

Once these rules are set and confirmed with iptables -L -n, and once your Jenkins instance is up and running on port 8080, attempt to access your Jenkins instance on port 80 instead of 8080. It should work and your URL should stay on port 80 - in other words, it should not get redirected to 8080. The fact that forwarding from 80 to 8080 (or 443 to 8443) should remain hidden from the client.

Saving iptables Configuration

Using the iptables command to change port configuration and routing rules only changes the current, in-memory configuration. It does not persist between restarts of the iptables service. So, you need to make sure you save the configuration to make the changes permanent.

Saving the configuration is slightly different between Red Hat rpm based and Debian-based systems. On a Red Hat-based system (Fedora, CentOS, Red Hat Enterprise Linux, Oracle Linux, etc), issue the following command:

sudo iptables-save > /etc/sysconfig/iptables

On a Debian-based system (Debian, Ubuntu, Mint, etc), issue the following command:

sudo sh -c "iptables-save > /etc/iptables.rules"

The iptables-restore command will need to be executed manually, or your system configured to automatically run it on boot, against the /etc/iptables.rules file you have created, in order for your iptables configuration to be retained across reboots. On Ubuntu, the fastest way is to install iptables-persistent after configuring iptables. It will automatically create the required files from the current configuration and load them on boot.

sudo apt-get install iptables-persistent

See https://help.ubuntu.com/community/IptablesHowTo for other Ubuntu options. There are many other resources describing this; please consult your system’s documentation or search on the internet for information specific to your flavor of Linux.

If you are unsure at all about what kind of system you have, consult that system’s documentation on how to update iptables configuration.

Using firewalld

Some Linux distributions (CentOS 8, Red hat Enterprise Linux 8, CentOS 7, etc.) ship with firewalld which serves as a front-end for iptables. Configuration thru firewalld is done via the firewall-cmd command. Instead of using any of the iptables commands mentioned above, all you should need to do is something like:

# allow incoming connections on port 80.
# You can also use --add-service=http instead of adding a port number
sudo firewall-cmd --add-port=80/tcp --permanent
sudo firewall-cmd --permanent \
                  --add-forward-port=port=80:proto=tcp:toaddr=127.0.0.1:toport=8080

# allow incoming connections on port 443.
# You can also use --add-service=https instead of adding a port number
sudo firewall-cmd --add-port=443/tcp --permanen
t
sudo firewall-cmd --permanent \
                  --add-forward-port=port=443:proto=tcp:toaddr=127.0.0.1:toport=8443
sudo firewall-cmd --reload

With the above commands, jenkins can be configured to run on localhost:8080 and/or localhost:8443 (depending if you need or want to do SSL or not)

firewalld will then create the required iptables rules so that incoming connections on port 80 are forwarded to jenkins on 8080 (and 443 is forwarded to 8443).

Running Jenkins behind HAProxy

In situations where you want a user friendly URL, different public ports, or to terminate SSL connections before they reach Jenkins, you may find it useful to run Jenkins (or the servlet container that Jenkins runs in) behind HAProxy. This section discusses some of the approaches for doing this.

Plain HTTP

Using HAProxy 1.7.9, here is an example HAProxy.cfg to proxy over plain HTTP:

# If you already have an haproxy.cfg file, you can probably leave the
# global and defaults section as-is, but you might need to increase the
# timeouts so that long-running CLI commands will work.
global
    maxconn 4096
    log 127.0.0.1 local0 debug

defaults
   log global
   option httplog
   option dontlognull
   option forwardfor
   maxconn 20
   timeout connect 5s
   timeout client 60s
   timeout server 60s

frontend http-in
   bind *:80
   mode http
   acl prefixed-with-jenkins  path_beg /jenkins/
   acl host-is-jenkins-example   hdr(host) eq jenkins.example.com
   use_backend jenkins if host-is-jenkins-example prefixed-with-jenkins

backend jenkins
   server jenkins1 127.0.0.1:8080
   mode http
   reqrep ^([^\ :]*)\ /(.*) \1\ /\2
   acl response-is-redirect res.hdr(Location) -m found
   # Must combine following two lines into a SINGLE LINE for HAProxy
   rspirep ^Location:\ (http|https)://127.0.0.1:8080/jenkins/(.*)
           Location:\ \1://jenkins.example.com/jenkins/\2 if response-is-redirect

This assumes Jenkins is running locally on port 8080.

This assumes that you are using the /jenkins/ context path for both the site exposed from HAProxy, and Jenkins itself. If this is not the case, you will need to adjust the configuration.

If you are experiencing the following error when attempting to run long CLI commands in Jenkins, and Jenkins is running behind HAProxy, it is probably due to HAProxy timing out the CLI connection. You can increase the timeout client and timeout server settings as necessary so the command will complete successfully.

WARNING: null
hudson.cli.DiagnosedStreamCorruptionException
Read back: 0x00 0x00 0x00 0x1e 0x07
           'Started reverse-proxy-test #68'
           0x00 0x00 0x00 0x01 0x07 0x0a
Read ahead:
Diagnosis problem:
    java.io.IOException: Premature EOF
        at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
        ...
    at hudson.cli.FlightRecorderInputStream.analyzeCrash(FlightRecorderInputStream.java:82)
    at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:153)
Caused by: java.io.IOException: Premature EOF
    at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:565)
    ...
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)

With SSL

Using HAProxy 1.7.9, here is an example HAProxy.cfg to connect to the proxy using SSL, terminate the SSL connection, and then talk to Jenkins using plain HTTP:

# If you already have an haproxy.cfg file, you can probably leave the
# global and defaults section as-is, but you might need to increase the
# timeouts so that long-running CLI commands will work.
global
    maxconn 4096
    log 127.0.0.1 local0 debug

defaults
   log global
   option httplog
   option dontlognull
   option forwardfor
   maxconn 20
   timeout connect 5s
   timeout client 5min
   timeout server 5min

frontend http-in
    bind *:80
    bind *:443 ssl crt /usr/local/etc/haproxy/ssl/server.pem
    mode http
    redirect scheme https if !{ ssl_fc } # Redirect http requests to https
    use_backend jenkins if { path_beg /jenkins/ }

backend jenkins
    server jenkins1 127.0.0.1:8080
    mode http
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    reqrep ^([^\ :]*)\ /(.*)     \1\ /\2
    acl response-is-redirect res.hdr(Location) -m found
    # Must combine following two lines into a SINGLE LINE for HAProxy
    rspirep ^Location:\ (http)://127.0.0.1:8080/(.*)
            Location:\ https://jenkins.example.com:443/\2 if response-is-redirect

Running Jenkins behind IIS

In situations where you have existing web sites on your server, you may find it useful to run Jenkins (or the servlet container that Jenkins runs in) behind Nginx, so that you can bind Jenkins to the part of a bigger website that you may have. This section discusses some of the approaches for doing this.

Make sure that you change the Jenkins httpListenAddress from its default of 0.0.0.0 to 127.0.0.1 or configure the firewall to block request on the port Jenkins is bound to, otherwise any IIS-level restrictions can be easily bypassed by accessing the Jenkins port directly.

Requirements

Example use case

I have a dedicated Jenkins installation on a Windows Server 2012 R2 server with a Common Name of VRTJENKINS01 in the Active Directory domain acme.example and is reachable by the Fully Qualified Domain Name vrtjenkins01.acme.example. Additionally Jenkins runs on port 8080 and already listens to 127.0.0.1 instead of 0.0.0.0 and the server has additional DNS names: jenkins and jenkins.acme.example.

I want to have an IIS installation which acts as a TLS/SSL terminating reverse proxy. In combination with our in-house Active Directory Certificate Services (ADCS, Microsoft’s Certificate Authority software) this should make certificate management a lot easier since Windows can be configured to automatically renew certificates, and the IIS 8.5+ Certificate Rebind feature can listen to renewal events (which contain the fingerprints of both the old and new certificate) and update the relevant bind(s) to use the fresh certificate. This would ensure that after the initial manual request it would only be necessary to manually change TLS/SSL related settings when the set of Alternate Subject Names on the certificate IIS presents should change.

IIS will only have to act as 1) a reverse proxy for Jenkins 2) redirect non-canonical URLs to the canonical URL: https://jenkins.acme.example/

I have installed the IIS (8.5) role using the Add Roles and Features Wizard with the all the default and also the following non-default features:

  • HTTP Redirection (Under Common HTTP Features, to redirect \http(s)://jenkins/, etc. to https://jenkins.acme.example/)

  • WebSocket Protocol (Under Application Development, because I felt like it)

Then I installed URL Rewrite and Application Request Routing.

Configuration Time

Enabling Reverse Proxy functionality
  1. In the Internet Information Services (IIS) Manager click on the VRTJENKINS01 server.

  2. Go to Application Request Routing Cache.

  3. In the Actions panel click on Server Proxy Settings…​

  4. Enable the proxy

  5. Disable the Reverse rewrite host in response header

    1. Don’t worry, it will work, just follow the rest of the instructions

  6. Set the Response buffer threshold (KB) to 0.

    1. This helps to prevent HTTP 502 errors on Jenkin’s Replay pages.

  7. Apply (the Actions panel again)

Configuring TLS/SSL

Out of scope, there are enough tutorials on the rest of the interwebs for this part. The rest of this tutorial will assume it has been configured with a certificate trusted by your browser of choice.

Configuring rules for response rewriting
  1. Go to the Default Web Site

  2. Go to URL Rewrite

  3. In the Actions panel click View Server Variables…​

  4. Add the following is not already define on the server level:

    1. Name: HTTP_FORWARDED

  5. Click on Back to Rules

  6. Click on Add Rule(s)…​

  7. Select Reverse Proxy and click on OK

  8. Enter jenkins.acme.example and click on OK

  9. Open the rule you just created

  10. Under Conditions add:

    1. Condition input: {CACHE_URL}

    2. Pattern: ^(http|ws)s://

  11. Under Server Variables add:

    1. Name: HTTP_FORWARDED, Value: for={REMOTE_ADDR};by={LOCAL_ADDR};host="{HTTP_HOST}";proto="https", Replace: yes

      1. Jenkins runs under Jetty, Jetty supports RFC7239, so all should be well.

  12. Under Action change:

    1. Rewrite URL to \{C:1}\://jenkins.acme.example:8080{UNENCODED_URL}

      1. Note that there is no slash between the port number and the opening curly bracket

    2. Remove the check from the Append query string checkbox

  13. Apply the changes.

  14. Edit C:\Windows\System32\drivers\etc\hosts so that jenkins.acme.example points to 127.0.0.1

    1. When resolving names Windows will check if the name is its own name before consulting the hosts file. Meaning that adding vrtjenkins01 or vrtjenkins01.acme.example to the hosts file won’t have any effect.

      1. The hosts file will however be consulted before consulting the DNS infrastructure

Experiencing the dreaded "It appears that your reverse proxy set up is broken." error for yourself
  1. https://jenkins.acme.example/configure

  2. Configure the Jenkins URL to be https://jenkins.acme.example/ and save the change

  3. Go to Configure Global Security and enable Enable proxy compatibility if you have already enabled Prevent Cross Site Request Forgery exploits

  4. Go to https://jenkins.acme.example/manage

  5. You will still experience the "It appears that your reverse proxy set up is broken." as expected

    1. If you do not get that at this point, then that is very weird…​ Continue anyway.

  6. Right click the Configure System link and choose to inspect the element.

    1. Make sure you are still on the Manage page as you will want it as your referrer

  7. Change the value of the href attribute to be administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor/test

  8. Open the link you just changed in a new tab.

    1. Keep this tab open

  9. Observe the "\https://jenkins.acme.example/manage vs http:" error and bask in its glory

    1. a white page served with HTTP status code is 200 indicates all is well

      1. If you do get that at this point, then that is very weird…​ Continue anyway.

Fixing the errors
  1. In IIS Manager got to Application Pools then edit DefaultAppPool so that the .NET CLR version is No Managed Code

    1. You might find that this is not necessary (at far as you can tell) for your setup, since IIS will only act as a TLS/SSL offloading reverse proxy, we don’t need it.

  2. Then go to SitesDefault Web SiteRequest Filtering and in the Actions panel choose Edit Feature Settings…​ and turn on Allow doube escaping

    1. This is so IIS forwards URLs like https://jenkins.acme.example/%2525 to Jenkins instead of showing an IIS error page

  3. Last, but not least, go to SitesDefault Web SiteConfiguration Editor and change the Section to system.webServer/rewrite/rules

  4. Now you should see the URL Rewrite 2.1 property useOriginalURLEncoding listed, if not install URL Rewrite 2.1 using the x86 or x64 installer, not the WebPI one and resume from here after a reboot.

  5. Change useOriginalURLEncoding to False

    1. As the URL Rewrite 2.1 announcement this will change the value of {UNENCODED_URL} to make it RFC3986 and usable for reverse proxy forwarding purposes

    2. original as in pre 2.1 behaviour.

  6. Refresh that tab you were supposed to keep open, or recreate it.

    1. Again, take some time to bask in its glory

  7. It should now be white, also the Manage page should no longer complain!

Continue configuring IIS

Some of the things you might want but I won’t cover:

  • Hypertext Strict Transport Security headers

  • Redirecting from non canonical URLs to the canonical URL (ok, sort of covered this in the web.config example)

  • The X-UA-Compatibility header so that Internet Explorer 11 (or 9, or …​) won’t claim to be IE 7 for intranet sites

  • Use IIS Crypto to configure cipher suites

  • …​

A working web.config

web.config

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <rewrite>
      <rules useOriginalURLEncoding="false">
        <rule name="CanonicalHostNameRule2" stopProcessing="true">
          <match url="(.*)" />
          <conditions trackAllCaptures="true">
            <add input="{CACHE_URL}" pattern="^(http|ws)://" />
            <add input="{HTTP_HOST}"
                 pattern="^jenkins$|^jenkins\.acme\.example$|
                          ^vrtjenkins01$|^vrtjenkins01\.acme\.example$" />
          </conditions>
          <action type="Redirect"
                  url="{C:1}s://jenkins.acme.example{UNENCODED_URL}"
                  appendQueryString="false"
                  redirectType="Permanent" />
        </rule>
        <rule name="CanonicalHostNameRule1" stopProcessing="true">
          <match url="(.*)" />
          <conditions trackAllCaptures="true">
            <add input="{CACHE_URL}" pattern="^(https|wss)://" />
            <add input="{HTTP_HOST}" pattern="^jenkins$|^vrtjenkins01$|
                                              ^vrtjenkins01\.acme\.example$" />
          </conditions>
          <action type="Redirect"
                  url="{C:1}://jenkins.acme.example{UNENCODED_URL}"
                  appendQueryString="false" redirectType="Permanent" />
        </rule>
        <rule name="ReverseProxyInboundRule1" stopProcessing="true">
          <match url="(.*)" />
          <action type="Rewrite"
                  url="{C:1}://jenkins.acme.example:8080{UNENCODED_URL}"
                  appendQueryString="false" />
          <serverVariables>
            <set name="HTTP_FORWARDED"
                 value="for={REMOTE_ADDR};
                        by={LOCAL_ADDR};
                        host=&quot;{HTTP_HOST}&quot;;
                        proto=&quot;https&quot;" />
          </serverVariables>
          <conditions trackAllCaptures="true">
            <add input="{CACHE_URL}" pattern="^(http|ws)s://" />
            <add input="{HTTP_HOST}" pattern="^jenkins\.acme\.example$" />
          </conditions>
        </rule>
      </rules>
    </rewrite>
    <security>
      <requestFiltering allowDoubleEscaping="true" />
    </security>
  </system.webServer>
</configuration>

Running Jenkins behind Squid

In situations where you want a user friendly url to access Jenkins (Not port 8080), it may make sense run Jenkins behind Squid, so that you can access Jenkins on port 80 or 443. This section discusses some of the approaches for doing this.

Squid 2.6

Using Squid 2.6:

acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
acl manager proto cache_object
acl to_localhost dst 127.0.0.0/8
acl valid_dst dstdomain .YOUR_DOMAIN ci

cache_replacement_policy heap LFUDA
memory_replacement_policy heap GDSF

cache_dir ufs /var/spool/squid 512 16 256
cache_mem 512 MB
maximum_object_size 12000 KB

## http --> https redirect
## don't forget to update "Jenkins URL" on https://ci.YOUR_DOMAIN/configure
#acl httpPort myport 80
#http_access deny httpPort
#deny_info https://ci.YOUR_DOMAIN/ httpPort

cache_peer localhost parent 8080 0 originserver name=myAccel
coredump_dir /var/spool/squid
hierarchy_stoplist cgi-bin
http_access allow localhost
http_access allow manager localhost
http_access allow valid_dst
http_access deny all
http_access deny manager

## mkdir /etc/squid/ssl/ && cd /etc/squid/ssl/
## to generate your self-signed certificate
## openssl genrsa -out jenkins.key 1024
## openssl req -new -key jenkins.key -x509 -out jenkins.crt -days 999
http_port 80 vhost
#https_port 443 cert=/etc/squid/ssl/jenkins.crt key=/etc/squid/ssl/jenkins.key vhost

http_reply_access allow all
icp_access allow all

refresh_pattern -i \.jp(e?g|gif|png|ico)   300  20%  600 override-expire

# Combine following THREE LINES into a SINGLE LINE for Squid
logformat combined %>a %ui %un \[%tl\]
          "%rm %ru HTTP/%rv" %Hs %<st
          "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh
strip_query_terms off
access_log /var/log/squid/access.log combined

visible_hostname ci.YOUR_DOMAIN

This assumes that you run Jenkins on localhost port 8080. But you can have it on an other server / different port (adjust line starting with cache_peer)

Of course replace YOUR_DOMAIN with your domain.

With ssl

Remove one level of comment

 sed s/^#// /etc/squid/squid.conf

Note: If you use the swarm client plugin, the nodes may report:

Caused by: sun.security.validator.ValidatorException:
    PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
        unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:285)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:191)
        at sun.security.validator.Validator.validate(Validator.java:218)
        at c.s.n.s.i.s.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:126)
        at c.s.n.s.i.s.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:209)
        at c.s.n.s.i.s.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:249)
        at c.s.n.s.i.s.ClientHandshaker.serverCertificate(ClientHandshaker.java:1014)
        ... 13 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException:
        unable to find valid certification path to requested target

You may be able to avoid that message with the -noCertificateCheck argument to agent.jar. That will disable server certificate checking from the agent.

Troubleshooting

Jenkins says my reverse proxy setup is broken

This message can also appear if you don’t access Jenkins through a reverse proxy: Make sure the Jenkins URL configured in the System Configuration matches the URL you’re using to access Jenkins.

Symptoms

An error message is displayed in the "Manage Jenkins" page - "It appears that your reverse proxy set up is broken"

Background

For a reverse proxy to work correctly, it needs to rewrite both the request and the response. Request rewriting involves receiving an inbound HTTP call and then making a forwarding request to Jenkins (sometimes with some HTTP headers modified, sometimes not). Failing to configure the request rewriting is easy to catch, because you just won’t see any pages at all.

But correct reverse proxying also involves one of two options, EITHER

  • rewrite the response with a "Location" header in the response, which is used during redirects. Jenkins sends Location: http://actual.server:8080/jenkins/foobar and the reverse proxy must to rewrite it to Location: http://nice.name/jenkins/foobar. Unfortunately, failing to configure this correctly is harder to catch; OR

  • set the headers X-Forwarded-Host (and perhaps X-Forwarded-Port) on the forwarded request. Jenkins will parse those headers and generate all the redirects and other links on the basis of those headers. Depending on your reverse proxy it may be easier to set X-Forwarded-Host and X-Forwarded-Port to the hostname and port in the original Host header respectively or it may be easier to just pass the original Host header through as X-Forwarded-Host and delete the X-Forwarded-Port # header from the request. You will also need to set the X-Forwarded-Proto header if your reverse proxy is changing from https to http or vice-versa.

Jenkins has proactive monitoring to make sure this is configured correctly. It uses XmlHttpRequest to request a specific URL in Jenkins (via relative path, so this will always get through provided the request is properly rewritten), which will then redirect the user to another page in Jenkins (this only works correctly if you configured the response rewriting correctly), which then returns 200.

This error message indicates that this test is failing - and the most likely cause is that the response rewriting is misconfigured. See the configuration examples for additional tips about configuring a reverse proxy.

Be sure to set the X-Forwarded-Proto header if your reverse proxy is accessed via HTTPS and then Jenkins itself is accessed via HTTP i.e. proxying HTTPS to HTTP.

Changing the context path of Jenkins with a reverse proxy is fraught with danger. There are lots of URLs that you need to rewrite correctly, and even if you get the ones in HTML files you may miss some in javascript, CSS or XML resources.

The recommendation is to ensure that Jenkins is running at the context path that your reverse proxy is serving Jenkins at. You will have the least pain if you keep to this principle.

While it is technically possible to use rewrite rules to change the context path, you should be aware that it would be a lot of work to find and fix everything in your rewrite rules and the reverse proxy will spend most of its time rewriting responses from Jenkins. Much easier to change Jenkins to run at the context path your reverse proxy is expecting, e.g. if your reverse proxy is forwarding requests at https://manchu.example.org/foobar/ to Jenkins then you could just use java -jar jenkins.war --prefix /foobar to start jenkins using /foobar as the context path

Further Diagnosis

For further diagnosis, try using cURL:

BASE=administrativeMonitor/hudson.diagnosis.ReverseProxySetupMonitor
curl -iL -e http://your.reverse.proxy/jenkins/manage \
            http://your.reverse.proxy/jenkins/${BASE}/test

(assuming your Jenkins is located at http://your.reverse.proxy/jenkins/ - and is open to anonymous read access)



Was this page helpful?

Please submit your feedback about this page through this quick form.

Alternatively, if you don't wish to complete the quick form, you can simply indicate if you found this page helpful?

    


See existing feedback here.