External Fingerprint Storage for Jenkins

Goal: Extend Jenkins to support storing artifact usage history in external databases

Status: Active

Team

Details

Abstract

File fingerprinting is a way to track which version of a file is being used by a job/build, making dependency tracking easy. The fingerprint engine of Jenkins can track usages of artifacts, credentials, files, etc. within the system. It does this by maintaining a local XML-based database. This leads to dependence on the physical disk of the Jenkins master.

The core idea of this project is to extend Jenkins core to support storing of fingerprints in an external storage, which would also allow tracking them across the entire CI/CD flow.

overview

Overview of Goals

  • Build a pluggable storage engine, which would allow fingerprints to be stored in external storages managed by storage system specific plugins.

  • Implementation of one such plugin (for Redis). This implementation should act as a reference for the development of other such external storage plugins for fingerprints.

  • Build the Fingerprint External Storage API Plugin to track the flow of fingerprints across the entire CI/CD flow.

Trying it out!

If you are a Jenkins user, consider trying out the Redis Fingerprint Storage Plugin. We appreciate you trying out the plugin, and welcome any suggestions, feature requests, bug reports, etc.

Feel free to reach out to us for any questions, feedback, etc. on the project’s Gitter Channel or the Jenkins Developer Mailing list. We use Jenkins Jira to track issues. Feel free to file issues under redis-fingerprint-storage-plugin component.

Deliverables

Phase 1

The following deliverables were planned for phase-1:

  • Jenkins core ready with support for pluggable storage extension point (beta API), that can be used by plugins (lacking migration and fingerprint cleanup support)

  • FileFingerprintStorage designed, implemented and conforms to maintain backwards compatibility.

  • POC reference implementation for Redis Fingerprint Storage Plugin.

  • Documentation as well as tests written for the points mentioned above.

We were able to successfully achieve them. Details can be found in the phase-1 blog post.

Phase 2

The following deliverables were planned for phase-2:

  • Alpha version of Redis Fingerprint Storage Plugin released.

  • Migration handling support added.

  • Fingerprint cleanup support added.

  • Blog Post (Phase 2)

The phase-2 deliverables were achieved, and some more. Phase-2 blog post explains them in detail.

Phase 3

  • 1.0 version of Redis plugin released.

  • FingerprintStorage API removed from Beta.

  • Setup Guides, README and contributing guide explaining how to setup, use and contribute to the plugin.

  • Alpha version of PostgreSQL Fingerprint Storage Plugin released.

  • Blog Post (Phase 3)

Meetings

Meetings are held every Monday at 15:00 UTC and every Thursday at 14:30 UTC. Meeting notes are available here.

Links