There were several reasons why it made sense to build another reference implementation, especially backed by PostgreSQL.
Redis is a key-value storage, and hence stores the fingerprints as blobs. The PostgreSQL plugin defines a relational structure for fingerprints. This offers a more powerful way to query the database for fingerprint information. Fingerprint facets can store extra information inside the fingerprints, which cannot be queried in Redis directly. PostgreSQL plugin allows powerful (indexing) and efficient querying strategies which can even query the facet metadata.
Another reason for building this plugin was to provide a basis for other relational database plugins to be built. It also validates the flexibility and design of our external fingerprint storage API.
Since PostgreSQL is a traditional disk storage database, it is more suitable for systems storing a massive number of fingerprints.
Among relational databases, PostgreSQL is quite popular, has extensive support, and is open-source. We expect the new implementation to drive more adoption, and prove to be beneficial to the community.
The plugin can be installed using the experimental update center. Follow along the following steps after running Jenkins to download and install the plugin:
Configure the Update Site URL as:
Submit, and then press the
PostgreSQL Fingerprint Storage Pluginand check the box along it.
Install without restart
The plugin should now be installed on the system.
Once the plugin has been installed, you can configure the PostgreSQL server details by following the steps below:
Scroll to the section
PostgreSQL Fingerprint Storagein the dropdown for
Fingerprint Storage Engine.
Configure the following parameters to connect to your PostgreSQL instance:
Host- Enter hostname where PostgreSQL is running
Port- Specify the port on which PostgreSQL is running
SSL- Click if SSL is enabled
Database Name- Specify the database name inside the PostgreSQL instance to be used. Please note that the database will not be created by the plugin, the user has to create the database.
Connection Timeout- Set the connection timeout duration in seconds.
Socket Timeout- Set the socket timeout duration in seconds.
Credentials- Configure authentication using username and password to the PostgreSQL instance.
Test PostgreSQL Connectionbutton to verify that the details are correct and Jenkins is able to connect to the PostgreSQL instance.
[IMPORTANT] When configuring the plugin for the first time, it is highly important to press the Perform PostgreSQL Schema Initialization button. It will automatically perform schema initialization and create the necessary indexes. The button can also be used in the case the database is wiped out and schema needs to be recreated.
Now, all the fingerprints produced by this Jenkins instance should be saved in the configured PostgreSQL instance!
Due to the relational structure defined by PostgreSQL, it allows users/developers to query the fingerprint data which was not possible using the Redis fingerprint storage plugin.
The fingerprint storage can act as a consolidated storage for multiple Jenkins instances. For example, to search for a fingerprint id across Jenkins instances using the file name, the following query could be used:
SELECT fingerprint_id FROM fingerprint.fingerprint WHERE filename = 'random_file';
A sample query is provided which can be tweaked depending on the parameters to be searched:
SELECT * FROM fingerprint.fingerprint WHERE fingerprint_id = 'random_id' AND instance_id = 'random_jenkins_instance_id' AND filename = 'random_file' AND original_job_name = 'random_job' AND original_job_build_number = 'random_build_number' AND timestamp BETWEEN '2019-12-01 23:59:59'::timestamp AND now()::timestamp
The facets are stored in the database as
PostgreSQL offers support to query
This is especially useful for querying the information stored inside fingerprint facets.
As an example, the Docker Traceability Plugin stores information like the name of Docker images inside these
These can be queried across Jenkins instances like so:
SELECT * FROM fingerprint.fingerprint_facet_relation WHERE facet_entry->>'imageName' = 'random_container';
At the moment these queries require working knowledge of the database. In future, these queries can be abstracted away by plugins and the features made available to users directly inside Jenkins.
A few API changes made in the Jenkins core were released in Jenkins-2.253. It mainly includes exposing fingerprint range set serialization methods for plugins.
The relational structure of the plugin allows some performance improvements that can be made when implementing
cleanup, as well as improving the performance of
Fingerprint#add(String job, int buildNumber).
These designs were discussed and are a scope of future improvement.
The current external fingerprint storage API supports configuring multiple Jenkins instances to a single storage. This opens up the possibility of developing traceability plugins which can track fingerprints across Jenkins instances.
Please consider reaching out to us if you feel any of the use cases would benefit you, or if you would like to share some new use cases.
The PostgreSQL Fingerprint Storage Plugin and the Redis Fingerprint Storage plugin are maintained by the Google Summer of Code (GSoC) Team for External Fingerprint Storage for Jenkins. Special thanks to Oleg Nenashev, Andrey Falko, Mike Cirioli, Tim Jacomb, and the entire Jenkins community for all the contribution to this project.
As we wrap up, we would like to point out that there are plenty of future directions and use cases for the externalized fingerprint storage, as mentioned in the previous section, and we welcome everybody to contribute.
Feel free to reach out to us for any questions, feedback, etc. on the project’s
Gitter Channel or the
Jenkins Developer Mailing list.
We use Jenkins Jira to track issues.
Feel free to file issues under either the
postgresql-fingerprint-storage-plugin or the
redis-fingerprint-storage-plugin component depending on the plugin.