External Fingerprint Storage for Jenkins

Goal: Extend Jenkins to support storing artifact usage history in external databases

Status: Completed

Team

Details

Abstract

File fingerprinting is a way to track which version of a file is being used by a job/build, making dependency tracking easy. The fingerprinting engine of Jenkins can track usages of artifacts, credentials, files, etc. within the system. It does this by maintaining a local XML-based database. This led to dependence on the physical disk of the Jenkins master.

This project extended Jenkins core to support storing of fingerprints in external storages. External storage plugins can then be used to configure connection to the storage. Once configured, the fingerprints are stored and loaded by Jenkins core via these plugins in a seamless manner. The project led to the creation of two plugins, backed by Redis and PostgreSQL. The demo shows the Redis fingerprint storage plugin in action.

Deliverables

  • Creating the External Fingerprint Storage API in core with support for migration and cleanup

  • Redis Fingerprint Storage Plugin

  • PostgreSQL Fingerprint Storage Plugin

  • Documentation and testing

Getting the Code

All the links mentioned below correspond to code which has been merged. There is no unmerged code.

Demo

External Fingerprint Storage Demo

The External Fingerprint Storage API

This section explains the external fingerprint storage API that was created in Jenkins core. For further details, please refer to JEP-226 which explains the design decisions in detail.

overview

We created the FingerprintStorage class which defines the API for allowing building of custom storage plugins. We defined the following methods in the API for plugin developers, which the plugins need to implement:

  • void save()

    • Saves the given Fingerprint in the storage.

  • Fingerprint load(String id)

    • Returns the Fingerprint with the given unique ID. The unique ID for a fingerprint is defined by Fingerprint#getHashString().

  • void delete(String id)

    • Deletes the Fingerprint with the given unique ID.

  • boolean isReady()

    • Returns true if there is some data in the fingerprint database corresponding to the particular Jenkins instance.

For more details, please refer to the Javadoc:

Fingerprint Cleanup

Fingerprint cleanup thread works by periodically iterating over the fingerprints and editing the job and build information of the ones based on whether they are still present in the system. It also deletes the fingerprints which do not have any build or job associated with them.

We extend this fingerprint cleanup functionality to be supported by external storages. Fingerprint cleanup support for external storage plugins was implemented in Jenkins-2.248. FingerprintStorage API was extended with the following methods:

  • iterateAndCleanupFingerprints(TaskListener taskListener)

    • Plugins can implement this method (which is called by Jenkins core periodically) to iterate and cleanup the fingerprints. The reason to design it this way, and not to iterate all the fingerprints via core, is because external storages may be able to implement more efficient traversal strategies on their own.

  • boolean cleanFingerprint(Fingerprint fingerprint, TaskListener taskListener)

    • This provides a reference implementation of cleanup, which external storages can use to cleanup a fingerprint. They may use this, or extend it to provide custom implementations.

This allows the plugins to implement their own cleanup strategies in efficient ways. For example, the Redis plugin uses cursors to traverse and cleanup the fingerprints.

Finally, we introduced the option to turn off fingerprint cleanup. This was done because it may be the case that storing extra data may be cheaper than performing cleanups, especially with external storages.

Fingerprint Migration

We implemented a lazy migration strategy to transfer the fingerprints from local storage to the newly configured external storage. Once an external fingerprint storage is configured, the new fingerprints are stored directly in the new storage engine. However, the old fingerprints present on the disk storage are migrated as and when they are used.

This allows the fingerprints to be migrated gradually from the local storage to the external storage and prevent huge migrations in one go. One caveat is that in case the fingerprint cleanup is turned on, the fingerprints will get transferred whenever cleanup is triggered.

Migration was introduced as part of this project in Jenkins-2.251. Both, the Redis and PostgreSQL, fingerprint storage plugins support migration.

Redis Fingerprint Storage Plugin

The Redis fingerprint storage plugin uses the external fingerprint storage API to store the fingerprints as blobs inside Redis instances.

Installation

The plugin can be installed using the Jenkins Update Center.

Follow along the following steps after running Jenkins to download and install the plugin:

  1. Select Manage Jenkins

  2. Select Manage Plugins

  3. Go to Available tab.

  4. Search for Redis Fingerprint Storage Plugin and check the box beside it.

  5. Click on Install without restart

The plugin should now be installed on your system.

Configuring the plugin using Web UI

Once the plugin has been installed, you can configure the Redis server details by following the steps below:

  1. Select Manage Jenkins

  2. Select Configure System

  3. Scroll to the section Redis Fingerprint Storage Configuration and fill in the required details:

    Configure Redis

    • Host - Enter hostname where Redis is running

    • Port - Specify the port on which Redis is running

    • SSL - Click if SSL is enabled

    • Database - Redis supports integer indexed databases, which can be specified here.

    • Connection Timeout - Set the connection timeout duration in milliseconds.

    • Socket Timeout - Set the socket timeout duration in milliseconds.

    • Credentials - Configure authentication using username and password to the Redis instance.

  4. Use the Test Redis Connection to verify that the details are correct and Jenkins is able to connect to the Redis instance.

  5. Press the Save button.

Now, all the fingerprints produced by this Jenkins instance should be saved in the configured Redis server!

PostgreSQL Fingerprint Storage Plugin

The PostgreSQL fingerprint storage plugin defines a relational structure for storing the fingerprints, and allows fingerprint metadata to be easily queried. Installing and using the plugin is very similar to the Redis fingerprint storage plugin. The usage is not explained here for the sake of brevity. The project README and phase-3 post have more information about this plugin.

Further Details

The phase wise progress can be found in the following posts:

Trying it Out!

If you are a Jenkins user, consider trying out the Redis Fingerprint Storage Plugin and the PostgreSQL Fingerprint Storage Plugin. We appreciate you trying out the plugins, and welcome any suggestions, feature requests, bug reports, etc.

Future Directions

The relational structure of the plugin allows some performance improvements that can be made when implementing cleanup, as well as improving the performance of Fingerprint#add(String job, int buildNumber). These designs were discussed and are a scope of future improvement.

The current external fingerprint storage API supports configuring multiple Jenkins instances to a single storage. This opens up the possibility of developing traceability plugins which can track fingerprints across Jenkins instances.

Please consider reaching out to us if you feel any of the use cases would benefit you, or if you would like to share some new use cases.

Acknowledgements

Special thanks to Oleg Nenashev, Andrey Falko, Mike Cirioli, Tim Jacomb, Jesse Glick and the entire Jenkins community for all the contribution to this project.

Reaching Out

Feel free to reach out to us for any questions, feedback, etc. on the project’s Gitter Channel or the Jenkins Developer Mailing list. We use Jenkins Jira to track issues. Feel free to file issues under redis-fingerprint-storage-plugin or postgresql-fingerprint-storage-plugin components.

Links