13 unstable releases (3 breaking)

0.4.3	Oct 18, 2024
0.4.2	Oct 17, 2024
0.3.6	Jul 30, 2024
0.3.2	Jun 28, 2024
0.1.0	Mar 23, 2024

#273 in Filesystem

391 downloads per month

GPL-3.0-or-later

4.5MB
3K SLoC

Contains (ar library, 340KB) libHSbase-compat-batteries-0.12.3-EvvecFThiaEAGWq5U5Tpi9.a.fixed, (static library, 340KB) libHSbase-compat-batteries-0.12.3-EvvecFThiaEAGWq5U5Tpi9.a, (JAR file, 1KB) tests/cases/jars/hello.jar, (Zip file, 1KB) tests/cases/jars/hello.jar.fixed, (static library, 1KB) tests/cases/libempty.a, (static library, 2KB) tests/cases/testrelro.a and 1 more.

Build postprocessor to reset metadata fields for build reproducibility

This crate provides the program add-determinism that takes one or more paths as arguments, and will recursively process those paths, attempting to run the handlers on any files with extensions that match. (Each argument can be either a single file or a directory to be processed recursively.)

For each processed file, a temporary file is opened, the contents are rewritten, the modification timestamp is copied from the original file to the temporary copy, and the copy is renamed over the original.

If processing fails, a warning is emitted, but no modifications are made and the program returns success.

The purpose of this tool is to elimiate common source of non-determinism in builds, making it easier to create reproducible (package) builds.

Usage

Standalone

$ add-determinism /path/to/file /path/to/directory

Note that the program works in-place, replacing input files with the rewritten versions (if any modifications are made).

Some useful options:

-v — enable debug output
-j [N] — use N workers (or as many as CPUs, if N is not given)
--handler list|HANDLER|-HANDLER — constrain the list of handlers. Takes a comma-separated list of names, either a list of "positive" names, in which case only listed handlers will be used, or a list of "negative" names, each prefixed by minus, in which case the listed handlers will not be used. By default, handlers that cannot be initialized are skipped with a warning. If a "positive" list is given, failure to initialize a handler will cause an error. The special value list can be used to list known handlers.
--brp — enable "build root program" mode, see below.

In an rpm build environment

When invoked with --brp, the $RPM_BUILD_ROOT environment variable must be defined and not empty. All arguments must be below $RPM_BUILD_ROOT. This option is intended to be used in rpm macros that define post-install steps. See redhat-rpm-config pull request #293 for a pull request that added a call to add-determinism in %__os_install_post.

Verification instead of modification

When invoked with --check, the tool processes all files, but does not actually save any modifications. Instead, it'll fail if any files would have been modified. It also returns an error if any files cannot be read.

Processors

`ar`

Accepts *.a.

Resets the embedded modification times to $SOURCE_DATE_EPOCH and owner:group to 0:0.

`jar`

Accepts *.jar.

This rewrites the zip file using the zip crate. The modification times of archive entries is clamped $SOURCE_DATE_EPOCH. Extra metadata, i.e. primarily timestamps in UNIX format and DOS permissions, are stripped (also because the crate does not support them).

`javadoc`

Accepts *.html.

This looks at the <head> portion of an HTML file and finds standard lines inserted by Javadoc that specify the file creation date. For example,  is replaced by a version without the version and date, and <meta name="dc.created" content="<date>"> is replaced by a version with $SOURCE_DATE_EPOCH.

`pyc`

Accepts *.pyc.

This handler implements a .pyc file parser for Python bytecode files and cleans up unused "flag references". It is a Rust reimplementation of the MarshalParser Python module.

`pyc-zero-mtime`

Accepts *.pyc.

This handler sets the internal timestamp in .pyc file header to 0, and sets the mtime on the corresponding source .py file to 0. This is intended to be used on OSTree systems where mtimes are discarded, causing a mismatch between the timestamp embedded in the .pyc file and the filesystem metadata of the .py file.

This handler is not enabled by default and must be explicitly requested via --handler pyc-zero-mtime.

Notes

This project is inspired by strip-nondeterminism, but is written from scratch in Rust. For Debian, build tools are written in Perl and more Perl is not an issue. But in Fedora/RHEL/…, tools are written in Bash, Python, or compiled, and we don't want to pull in Perl into all buildroots. In addition, the details differ in what kinds of processing we want to do. For example, Debian does not distribute Python bytecode files.

Dependencies

~11–20MB
~295K SLoC