Scaling Jenkins at Jenkins World 2016
I find the topic of "scaling Jenkins" to be incredibly interesting because, more often than not, scaling Jenkins isn’t just about scaling a single instance but rather scaling an organization and its continuous delivery processes. In many cases when people talk about "scaling Jenkins" they’re talking about "Jenkins as a Service" or "Continuous Delivery as a Service" which introduces a much broader scope, and also more organization-specific requirements, to the problem.
One of my favorite parts of a big conference like Jenkins World is getting to see how other people are solving similar problems at different organizations, in essence: "how the sausage is made." This year’s Jenkins World will be no different, with a number of sessions by developers and engineers from the companies leading the way, scaling continuous delivery and Jenkins.
Register for Jenkins World in
September with the code
In the realm of "scaling Jenkins" the following sessions stand-out to me as "must-attend" for those interested in the space:
September 14th 4:15 PM - 5:00 PM, Exhibit Hall A-1
NPR’s Digital Media team uses Jenkins to build, test and deploy code to various staging and production environments. As the complexity of the software components, environments and tests have grown - both generally and due to our quest to achieve continuous deployment - management of Jenkins has become a challenge. In this talk, we share information about our “JenkinsOps” effort which has allowed us to automate many of the administrative tasks necessary to manage feature code branches, handle deployments, run tests and configure our environments properly.
September 15th 1:30 PM - 2:15 PM, Exhibit Hall C
At Riot Games, we build a lot of software. Come learn how we built an integrated Docker solution using Jenkins that accepts Docker images submitted as build environments by engineers around the company. Our containerized farm now creates over 10,000 containers per week and handles nearly 1,000 jobs at a rate of about 100 jobs per hour. All this is done with readily available, open source Jenkins plugins. We’ll explore lessons learned, best practices and how to scale and build your own system, as well as why we chose to solve the problem this way…and whether or not we succeeded!
September 15th 2:30 PM - 3:15 PM, Great America J
In this talk, we’ll show how to use Jenkins Pipeline together with Docker and Kubernetes to implement a complete end-to-end continuous delivery and continuous improvement system for microservices and monolithic applications using open source software. We’ll demonstrate how to easily create new microservices projects or import existing projects, have them automatically built, system and integration tested, staged and then deployed. Once deployed, we will also see how to manage and update applications using continuous delivery practices along with integrated ChatOps - all completely automated!
September 15th 2:30 PM - 3:15 PM, Exhibit Hall C
The Jenkins platform can be dynamically scaled by using several Docker cluster and orchestration platforms, using containers to run agents and jobs and also isolating job execution. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? How do they compare? All of them can be used to dynamically run jobs inside containers. This talk will cover these main container clusters, outlining the pros and cons of each, the current state of the art of the technologies and Jenkins support. I believe people will be very interested in learning about the multiple options available.
September 15th 3:45 PM - 4:30 PM, Exhibit Hall C
How can we do it? We start with some real world results realized by Jenkins users who have built large clusters and review how they got there. Next, we will do experiments scaling some individual sub-components of Jenkins in isolation and see what challenges we will face when integrated. The famous large, distributed systems undoubtedly faced problems scaling - and we can learn from them, too. The result will be recipes for building Jenkins clusters with different scaling capabilities. After all of this, you can build the biggest Jenkins cluster in the world…or maybe just make your own Jenkins cluster more efficient.
September 15th 3:45 PM - 4:30 PM, Exhibit Hall A-1
This session will highlight how Splunk uses Jenkins to provide an end-to-end solution in the development CI system. Attendees will see how test results are delivered to a Splunk indexer, where they can be analyzed and presented in a variety of ways. This session will also include a live demonstration.
September 15th 4:45 PM - 5:30 PM, Exhibit Hall C
Last year, we presented our initial investigations and stress testing as we prepared to deploy a large-scale Jenkins installation at Google. Now, with a year of real-world use under our belts, we’ll discuss how our expectations held up, what new issues we encountered and how we have addressed them.
In addition to these, we will also be hosting a Jenkins World Contributor Summit where "scaling" relevant topics such as "Storage Pluggability" will be discussed.
The Jenkins World agenda is packed with even more sessions, so it should be a very informational event for everybody; hope to see you there!