GUI improvements on the horizon
This past Thursday, September 24th, 2015, I presented a couple of prototypes of what I hope will be the future of the Jenkins GUI. Or perhaps more correctly, close enough to the future to start generating positive feedback from you the community that improving the Jenkins GUI is important and some pieces that I am showing are going in the right direction. If you have ~45 minutes to spare, I recommend the video (the narrator’s voice is very soothing). If not, I offer the following as a reasonable summary.
Jenkins has a lot of strengths as tool. Its robust user community along with its thoughtful and extensible design are two of the most immediate. They are the two pillars that have made Jenkins the leader in the CD/CI space and the de facto choice for most of us looking to automate our build and test processes. But let’s face it, by today’s standards, the GUI doesn’t really sing. I will even go so far as to say, I believe it is a platform liability at the moment, and even among we the Jenkins faithful, few of us look forward to using it.
In an effort to turn that tide, I traveled to this year’s 3 main JUC events, in DC, London, and Santa Clara, pushing the idea that enhancement is possible and providing an evolving sketch of what that might look like. The three main areas of enhancement I have targeted for a first round of improvement are these:
Soon to follow, but not yet prototyped by me would be pieces dedicated to monitoring jobs in Jenkins as well as node and resource utilization and efficiency. Rightly or wrongly, I have started with the create and configure side of the GUI, as I see it as somewhat primary in a typical job creation scenario (you have to create a job before you can monitor it), but this second piece is no less important. Sadly, lips service is all I can offer you today, but more prototypes and video demonstrations are on the way.
Item Creation and Configuration in Jenkins
In most use cases, item creation means creating a freestyle job, so that is what I use as my base use case example. It is important to note, however, that most configuration in Jenkins happens through a shared set of GUI components. These components are a blend of Jelly files and Javascript and can be found in the …/main/resources/lib/form directory in the Jenkins source code. In operating on these pieces, I have the opportunity to effectively enhance broad areas of the Jenkins experience, including aspects of plugin use that share these components. This greatly increases the upside of the effort as well as the possible drama and side effects, which I will go into more detail on later.
As for the upside piece, the first bit of improvement I am looking to attain is breaking up the many 'toilet paper' style unbroken configuration lists sprinkled throughout Jenkins. The first example of this appears in item creation. On first installation, this issue is not immediately obvious, but if you have installed a variety of plugins or chosen to purchase CloudBee’s Jenkins Enterprise product, you will find that Jenkins can have quite a few types of items to create. While they do have descriptive text, I still find them difficult to differentiate and almost impossible to casually scan. Thus, my first suggestion is to add some form of categorization to the item types. For this to function correctly, the GUI will need to be smart enough to apply the categories only when item counts are sufficient to justify them (if you only have 4 item creation types, it doesn’t make sense to have 8 categories with which to sort them). But if you are a long time Jenkins user with many plugins you may also know it is possible to have more than a dozen item types. So if nothing else, an extension point that allowed for the categorization of item types seems helpful.
The configuration form itself, it also can become incredibly long with few landmarks or visual differentiation points. As a remedy, I propose calling out and clearly boxing each of the existing configuration sections and making sure that their names are as meaningful as possible. As an added step, I make the sections collapsible. This allows the user to jump to specific points in the form and tuck other areas out of the way. In some cases, we can make specific sections open by link context or even by user context.
Plugin Selection
Another essential piece of the Jenkins experience is plugin configuration. Today, if you are looking to add plugins to your Jenkins environment, you are almost certainly using Google to find a 3rd party review site, collecting the name of the plugin you want and then either linking to it on this website, or filtering for it in the Plugin Manager GUI.
Neither in the product nor on this website is there a particularly good resource for comparing plugins and evaluating which you might add.
Instead, I am looking to add something akin to an application store experience to both this website and the product UI. You should be able to group sort and compare plugins by a variety of criteria, including author, installation base, and user review. You should also have a set of general use categories that fits user needs and expectations, rather than the free ranging labels that plugin authors have arbitrarily applied today.
Workflow Script Builder
Finally, I have a GUI that allows for a sort of Drag-n-drop assembly of Workflows. A major tenant of the utility of Workflows as opposed to Freestyle jobs is that they can be completely separated from the Jenkins GUI and stored in a source repository. None-the-less, with absolutely no GUI, there is little to guide the user who is looking to get started without a upfront learning investment. As it turns out, a Workflow/Groovy script is pretty straight forward, but you don’t really know that until you have made one. Also, Workflow allows for the orchestration of jobs across multiple nodes of hardware resources, making it a potentially involved little bit of configuration. Thus, my goal here is two fold. Allow the user to model a workflow quickly and easily and showcase a few of the more advanced features workflow enables. The result is this script builder. My hope is to host the prototype somewhere you all might be able to use it directly, but in the meantime, my hope is that my video pretty well explains how it works. Please take a look and post whatever comments you see fit.
…and really send along feedback…
So with all things community related, please, please, send back whatever feedback makes sense. I can be reached via Twitter @gusreiber.
Other places you can find me include, IRC (freenode/#jenkins) and https://plus.google.com/GusReiberUI[Google (https://plus.google.com/+GusReiberUI)]. I would love to hear from you.
Questions and Answers from the talk:
-
How likely is it that any of these UI changes will make it into the core open source Jenkins? When would we start seeing them there?
Most will be OSS. An exact schedule has not been determined, but most of it is still about a year away. Likely we will have an experimental wars for download along the way. -
Is there anyway to determine which GUI attributes are contributed by which plugin?
I take it that is a bit of a feature request? It came up at JUC West as well. Should be something that can be surfaced in the GUI. I agree, it would be helpful. -
What is the difference between ANT and Jenkins?
Ant is a good bit more bare-bones than Jenkins. In fact, you can add an Ant plugin to your Jenkins environment. You would typically use Ant to compile java source files. Jenkins orchestrates the fetching of the source files from some particular repository, the building of those files (often Jenkins uses Ant via its plugin to do this), running and reporting some suite of tests against that build, and then archiving or deploying the artifacts to wherever. Often times this requires navigating several computers with their own security constraints, so Jenkins helps you manage that as well. -
What version of the Jenkins it is?
This isn’t available today, but I am building against 1.621-SNAPSHOT currently, but will upgrade with Jenkins to the coming December LTS. I’m interested in seeing the list of 100 plugins that you mentioned (by Daniel?) Me too. :^) He and the community (which can be you if want to join IRC freenode.net/#jenkins and attend the hangouts and governance meetings: https://jenkins.ci.org ) -
For IRC, I assume the server is freenode.net?
Yes. -
Will there be any dashboard kind of feature for the build history in the new GUI?
So far, I have been focusing on the create and configuration portion of the Jenkins UX as I see it as a barrier to entry for new users. The read/report/analyze half of the Jenkins UX I actually see as the portion with more long term value, as you tend to read more often than you write, so I am eager to jump in here as well. ….however, in its core today, Jenkins the tool seems to me to really want to see the world in the same context of flat XML files in folders as it actually persists its configuration data. To really make meaningful dashboards, it needs to be possible to query job configurations and build artifacts by a wide set of criteria that is not at all related to the folder in which the xml file happens to be stored. Also, some of the things you care about in the Jenkins universe are compute resources (controller/agents/exactures). These are also not the same as config files in folders and need to be queryable as their own first class type of entity. …so what I am saying with a lot of words is that I see the config piece as a somewhat more immediate and urgent fix. The broccoli of the meal, if you will. I will want to get that out as fast as possible to get it out of the way. The reporting piece is actually the wine. At the moment, we are giving you Bartles and Jaymes in paper cups. …so a lot of work is still needed there. -
Have you investigated Google Polymer as UI components for jenkins UI?
I have not, but will now. I am actually quite a google fan-boy in much the way a lot of kids love Apple. (I also love Apple… being from Seattle, I even love MS). But, for the super near term, we are most focused on getting JQuery cleanly into core and Prototype.JS deprecated. Walk first, is my feeling. -
Are there any tutorials on Jenkins workflow?
Jesse Glick or KK are better people to ask about that, really. They are also on IRC: freenode.net/#jenkins. Daniel Beck as well, might be a good person to ask. My little workflow demo is still really just fiction. Will there be a 'Expand All' and 'Collapse All' buttons for the accordions in new configure GUI? (I would probably inject one if not by default) Yes. Also, they should be URL controllable so that they can be set by link or user context easily. Maybe they should also remember what you had open last? …stuff to tinker with that really needs to be right. -
What impact does the UI changes have on job configuration behind the scenes? Is configuration still stored in XML format?
None. The post string stays the same and from then on, Jenkins is Jenkins. -
Can the create item screen be configurable? At the moment, no, but ideally yes. It is still a big hand wave at the moment about how those categories are created, managed, and updated. The same categories ought to bubble back up when searching for the plugins to help relate what plugins generate what UI. I am hoping for guidance from the community. How will workflow fit in with new UI?
In some respects, the new configuration page is about enhancing the more traditional freestyle job and not workflow. However, the last bit of my presentation with the script builder is exclusively about workflow. The plugin manager is about plugins, so it would apply to both. -
How is a human notified for the wait for approval step in this workflow?
So workflow approval can be done via the web GUI. But to get real notification, you would program that into your workflow Jenkins has a fairly large set of notification plugins. So you can use Jenkins to trigger email, or SMS, or HipChat, or Slack, or pretty much whatever. As these plugins are increasingly customized for workflow, you will get nice and nice workflow syntax for instantiating those actions. When my script builder is adopted, you would have a friendly button you could drag into the stage and it would notify you prior to the manual checkpoint. -
Custom plugins still supported?
Yes. Though there is supported and supported. The highest level of support for a plugin would be a custom DSL for workflow that would make for streamlined syntax in workflow for interacting with that plugin via Groovy. But existing plugins do not need that level of support to be used within a Jenkins file / Groovy script. Instead, the syntax for accessing the plugin is likely to be more complicated. ….some plugins are freestyle specific, in which case, they no longer make sense in workflow. ….Daniel Beck or Jesse Glick are probably better suited to answering this question, however… -
Will there be an improvement in performance with docker builds, sonar scanning? From my experience sonar takes 20+ mins with jenkins plugin where as it takes 3 mins with maven plugin
Is this times it is taking the GUI to render, or the actual build to run? I am not sure I am following the question exactly, but regardless, I am not well equipped to answer many questions about performance issues in Jenkins. I know of a fairly major performance issue specifically in the configuration form that I believe will be fixed in the new GUI, but that isn’t build performance, it is just form rendering performance. -
I like the graphical configuration. Thanks. The scripting of a complex workflow looked a bit daunting.
Cool. Yeah, my main and first goal is to get something out there that would allow folks to quickly sketch and deploy an actual working workflow that reasonably reflects an 80%ish use case. No GUI can ever be as fully flexible as a script, but I don’t think most people need the 95% case to get started and see the benefit of a versionable and robust config file format. -
Will there be any effort to make the UI mobile friendly for the admin on the go?
Absolutely. Especially on the TBD read/reporting end of the UI, but everything new needs to meet a reasonably high bar of device responsiveness. Today, the Jenkins GUI is just not responsive. Which is terrible. -
As a plugin developer do I need to change implementing the ui source from jelly or groovy to some other language/technique or will it be compatible?
So you will not NEED to change from whatever you are doing, except if you have built a plugin GUI that has custom script that either relies directly on behavior.js, hudson-behavior.js, or the particulars of the existing DOM structure (you do something in the client that requires your or some other input to be in a particular TABLE TR TD DOM traversal path). …I believe 2 things are going to continue to happen at a faster and faster rate. New plugin authors are not going to want to write GUIs in Jelly and Prototype.js, but instead use some more modern client side MVC approaches like Angular, where the GUI interacts with a REST api instead of being a dom directly rendered from the server. It is a bit of a different mode of working than Jelly, and maybe slightly less direct, but it is a lot easier to find doc on how to do things with JQuery, Agile, Handlebars and the like, than it is to find doc on Jelly. And the responsiveness and breadth of gestures and controls in Jelly are already terribly behind what is now the main stream of web UI development. So I think plugin builders are, if they aren’t already, going to want better tools available to them. I also think that people are going to gravitate towards workflow or something similar. Since the UI for workflow is foremost a script, making a GUI for a plugin that works with it might be a fundamentally different beast. …depending on what the plugin is trying to do… So again, new plugins or even upgrading existing plugins to work with workflow are likely want a new technology set, not just because the existing Jenkins GUI is changing, but because new plugins will want to do different and better stuff. -
Are there connectors for other source control tools like CVS and Dimensions?
I am not sure exactly which connector plugins are already supporting Workflow or how deeply that support goes. Because Jenkins has plugins that provide access to these SCMs, you can use workflow to go and fetch those source trees. A greater level of support for workflow from these plugins would mean a more elegant workflow syntax for that interaction. At the moment, my GUI script builder is still fiction. My plan would be to add GUI buttons for whatever are the most popular SCMs and I will attempt to mask the syntax regardless of its clumsiness. ….the way I am constructing my initial prototype, there is already a reasonably clear extension point for adding buttons that generate some chunk of Groovy syntax when it is dragged into a stage. So I will add the initial set based on community feedback and then the community can continue to add their own. -
What are the compatibility issues existing plugin developers needs to be aware of?
For plugins that interact with freestyle jobs, or really most job types that aren’t workflow, plugin developers should expect the page DOM structure to change. If for whatever reason, they find they are busting into some custom script to traverse the DOM to compare 1 setting to another, that will break. Also, hudson-behaviors.js itself has a number of functions in it that do DOM traversing, like “findFollowingTR”. In some cases the signatures of those functions might need to change and the DOM structure that they return might also change. If a plugin uses what were meant to have been internal functions, they are likely to break. Finally, the page geometry is going to change. This may seem so superficial and obvious that, who cares, but sometimes changing a column width translates into an important part of a GUI being hidden or otherwise inaccessible. That ends up being as critical a break as any other. …so to combat these points of possible breakage, we are going to be looking for a handful somewhere between 20 and 100 plugins that we will want to test against. We haven’t made that list yet, let alone run any tests, so that is really a critical next step. For the plugin manager changes, I don’t see much if any of a braking issue, although I would like to add additional sorting and display power to the GUI, which means the GUI will need more metadata than currently exists, if the plugins want to take advantage of that new power in the GUI. This won’t break things, but plugin authors might want to go back to their plugins and fill in whatever the new bits of metadata end up being…. most likely they would be things like, richer descriptions, better category selections, and possibly icons. -
I’ve not seen a lot of Jenkins but what I had I didn’t really get, was awkward for all the reasons Gus mentioned. This looks brilliant. When can we have it?
Tom and I, and now our junior pledge, Keith (not actually junior at all, just more fit than me), are busily typing as fast as we can as well as lobbying the community that our vision is more or less a correct one. We have a very interesting initial plugin selection GUI that might make this years final LTS (which I did not demo), which is none-the-less a nice step forward for Jenkins. In it will be a lot of the JS library bundling that will enable most of what I have shown in this demo. Our hope is that with each LTS we will be able to push out an additional piece of the GUI puzzle. Likely starting with the job create and configure GUI, which would be the mid year LTS. I am hoping that a year from now this will be how Jenkins looks and acts. ….in the meantime, we are grappling with how best to push preview releases so people can play with it and send me hate mail. -
Is there any way to test front end of Jenkins plugins? And will that improve too?
A major and almost blocking portion of this work used to be the custom and somewhat broken version of HTMLUnit that was in core, which greatly hampered including libraries other than Prototype in Jenkins and writing code using those libraries in some sort of testable way. Our new approach to rebuilding the Jelly controls which are the foundation of the Jenkins config page and in general are shared by all plugins that need to post data back to Jenkins, already have a testing strategy backed into our design. Those Jelly form controls are extensible in Jenkins today and would remain so. Our hope would be that any plugin adding custom controls would follow our same design and test pattern we are building in core. ….so that was a long answer, but the short answer YES! Today, building GUI parts into your Jenkins plugin is a bit of a mystery, where most people copy something they saw someone else did, hack it, and the only test is, well…. it worked for me. That is no good and a fundamental piece we are looking to change. ….still a long answer… Node.js and Jasmine are the specific tools we using. -
What’s the estimated rollout date for this workflow feature?
The workflow feature is the newest concept I demonstrated, but in a lot of ways may also be the easiest to ship. As a script generator, exclusively, it could be hosted anywhere, and then you just paste your generated workflow script into the whatever existing Jenkins GUI better, submit into your source code. ….but at the moment, it isn’t actually on an official roadmap yet. Assuming the response to it remain positive, I would expect that to change fairly quickly.