Back to blog

Scaling Jenkins at Jenkins World 2016

R. Tyler Croy
R. Tyler Croy
August 31, 2016

This is a guest post by R. Tyler Croy, who is a long-time contributor to Jenkins and the primary contact for Jenkins project infrastructure. He is also a Jenkins Evangelist at CloudBees, Inc.

Jenkins World

I find the topic of "scaling Jenkins" to be incredibly interesting because, more often than not, scaling Jenkins isn’t just about scaling a single instance but rather scaling an organization and its continuous delivery processes. In many cases when people talk about "scaling Jenkins" they’re talking about "Jenkins as a Service" or "Continuous Delivery as a Service" which introduces a much broader scope, and also more organization-specific requirements, to the problem.

One of my favorite parts of a big conference like Jenkins World is getting to see how other people are solving similar problems at different organizations, in essence: "how the sausage is made." This year’s Jenkins World will be no different, with a number of sessions by developers and engineers from the companies leading the way, scaling continuous delivery and Jenkins.

Register for Jenkins World in September with the code JWFOSS for a 20% discount off your pass.

In the realm of "scaling Jenkins" the following sessions stand-out to me as "must-attend" for those interested in the space:

September 14th 4:15 PM - 5:00 PM, Exhibit Hall A-1

159px National Public Radio logo.svg NPR’s Digital Media team uses Jenkins to build, test and deploy code to various staging and production environments. As the complexity of the software components, environments and tests have grown - both generally and due to our quest to achieve continuous deployment - management of Jenkins has become a challenge. In this talk, we share information about our “JenkinsOps” effort which has allowed us to automate many of the administrative tasks necessary to manage feature code branches, handle deployments, run tests and configure our environments properly.

— Paul Miles and Grant Dickie of NPR

September 15th 1:30 PM - 2:15 PM, Exhibit Hall C

Riot Games logo At Riot Games, we build a lot of software. Come learn how we built an integrated Docker solution using Jenkins that accepts Docker images submitted as build environments by engineers around the company. Our containerized farm now creates over 10,000 containers per week and handles nearly 1,000 jobs at a rate of about 100 jobs per hour. All this is done with readily available, open source Jenkins plugins. We’ll explore lessons learned, best practices and how to scale and build your own system, as well as why we chose to solve the problem this way…and whether or not we succeeded!

— Maxfield F Stewart of Riot Games

September 15th 2:30 PM - 3:15 PM, Great America J

redhat In this talk, we’ll show how to use Jenkins Pipeline together with Docker and Kubernetes to implement a complete end-to-end continuous delivery and continuous improvement system for microservices and monolithic applications using open source software. We’ll demonstrate how to easily create new microservices projects or import existing projects, have them automatically built, system and integration tested, staged and then deployed. Once deployed, we will also see how to manage and update applications using continuous delivery practices along with integrated ChatOps - all completely automated!

— James Strachan of Red Hat

September 15th 2:30 PM - 3:15 PM, Exhibit Hall C

320px CloudBees official logo The Jenkins platform can be dynamically scaled by using several Docker cluster and orchestration platforms, using containers to run agents and jobs and also isolating job execution. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? How do they compare? All of them can be used to dynamically run jobs inside containers. This talk will cover these main container clusters, outlining the pros and cons of each, the current state of the art of the technologies and Jenkins support. I believe people will be very interested in learning about the multiple options available.

— Carlos Sanchez of CloudBees

September 15th 3:45 PM - 4:30 PM, Exhibit Hall C

How can we do it? We start with some real world results realized by Jenkins users who have built large clusters and review how they got there. Next, we will do experiments scaling some individual sub-components of Jenkins in isolation and see what challenges we will face when integrated. The famous large, distributed systems undoubtedly faced problems scaling - and we can learn from them, too. The result will be recipes for building Jenkins clusters with different scaling capabilities. After all of this, you can build the biggest Jenkins cluster in the world…or maybe just make your own Jenkins cluster more efficient.

— Stephen Connolly of CloudBees

September 15th 3:45 PM - 4:30 PM, Exhibit Hall A-1

splunk logo 300x100 This session will highlight how Splunk uses Jenkins to provide an end-to-end solution in the development CI system. Attendees will see how test results are delivered to a Splunk indexer, where they can be analyzed and presented in a variety of ways. This session will also include a live demonstration.

— Bill Houston of Splunk

September 15th 4:45 PM - 5:30 PM, Exhibit Hall C

272px Google 2015 logo.svg Last year, we presented our initial investigations and stress testing as we prepared to deploy a large-scale Jenkins installation at Google. Now, with a year of real-world use under our belts, we’ll discuss how our expectations held up, what new issues we encountered and how we have addressed them.

— David Hoover of Google

In addition to these, we will also be hosting a Jenkins World Contributor Summit where "scaling" relevant topics such as "Storage Pluggability" will be discussed.

The Jenkins World agenda is packed with even more sessions, so it should be a very informational event for everybody; hope to see you there!

About the author

R. Tyler Croy

R. Tyler Croy

R. Tyler Croy has been part of the Jenkins project for the past seven years. While avoiding contributing any Java code, Tyler is involved in many of the other aspects of the project which keep it running, such as this website, infrastructure, governance, etc.