rpms/python-rpm-macros
6
0
Fork 0
Code Issues Pull Requests Releases Activity

New opt-in possibility to fix byte-compilation reproducibility

Browse Source 
A new script brp-fix-pyc-reproducibility creates an opt-in way of how to fix
problems with the reproducibility of byte-compiled Python files. The script
uses marshalparser [0] which currently doesn't provide solutions for all issues
but can fix at least problems with reference flags. For more info see
this Bugzilla [1].

If you want to use this new feature, you need to define
`%py_reproducible_pyc_path` to specify a path you want to fix `.pyc`
files in (recursively) and build-require /usr/bin/marshalparser.

if you forget to build-require the parser. The error message is:
```
+ /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0
Bytecompiling .py files below /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64/usr/lib/python3.9 using /usr/bin/python3.9
+ /usr/lib/rpm/redhat/brp-fix-pyc-reproducibility /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64
ERROR: If %py_reproducible_pyc_path is defined, you have to also BuildRequire: /usr/bin/marshalparser !
error: Bad exit status from /var/tmp/rpm-tmp.UUJr4v (%install)
```

A build fails if the parser is not able to parse any of the `.pyc` files.

And finally, if a build is properly configured it produces fixed `.pyc` files.

Currently, `.pyc` files in the tldr package contain a lot of unused reference flags:
```
$ dnf install -y tldr
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
… long output …
190 - Flag_ref(byte=9610, type='TYPE_SHORT_ASCII_INTERNED', content=b'init', usages=0)
191 - Flag_ref(byte=9633, type='TYPE_SHORT_ASCII_INTERNED', content=b'source', usages=0)
192 - Flag_ref(byte=9651, type='TYPE_SHORT_ASCII_INTERNED', content=b'argv', usages=0)
193 - Flag_ref(byte=9657, type='TYPE_SHORT_ASCII_INTERNED', content=b'print_help', usages=0)
194 - Flag_ref(byte=9669, type='TYPE_SHORT_ASCII_INTERNED', content=b'stderr', usages=0)
195 - Flag_ref(byte=9682, type='TYPE_SHORT_ASCII_INTERNED', content=b'parse_args', usages=0)
196 - Flag_ref(byte=9737, type='TYPE_SHORT_ASCII_INTERNED', content=b'encode', usages=0)
197 - Flag_ref(byte=9782, type='TYPE_SHORT_ASCII_INTERNED', content=b'parser', usages=0)
198 - Flag_ref(byte=9790, type='TYPE_SHORT_ASCII_INTERNED', content=b'options', usages=0)
199 - Flag_ref(byte=9799, type='TYPE_SHORT_ASCII_INTERNED', content=b'rest', usages=0)
200 - Flag_ref(byte=9821, type='TYPE_SHORT_ASCII_INTERNED', content=b'result', usages=0)
202 - Flag_ref(byte=10022, type='TYPE_SHORT_ASCII_INTERNED', content=b'__main__', usages=0)
203 - Flag_ref(byte=10102, type='TYPE_SHORT_ASCII_INTERNED', content=b'argparse', usages=0)
204 - Flag_ref(byte=10433, type='TYPE_SHORT_ASCII_INTERNED', content=b'__name__', usages=0)
205 - Flag_ref(byte=10463, type='TYPE_SHORT_ASCII_INTERNED', content=b'<module>', usages=0)
```

This new feature fixes them:

```
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
<empty output>
```

[0] https://github.com/fedora-python/marshalparser
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1686078
This commit is contained in:
Lumir Balhar 2020-07-16 10:30:55 +02:00 committed by churchyard
parent 8e9c3d8bbe
commit b983c2118b
1 changed files with 18 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
brp-fix-pyc-reproducibility Normal file
View File

@ -0,0 +1,18 @@
#!/bin/bash -e
# If using normal root, avoid changing anything.
if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then
exit 0
fi
# Defined as %py_reproducible_pyc_path macro and passed here as
# the first command-line argument
path_to_fix=$1
# First, check that the parser is available:
if [ ! -x /usr/bin/marshalparser ]; then
echo "ERROR: If %py_reproducible_pyc_path is defined, you have to also BuildRequire: /usr/bin/marshalparser !"
exit 1
fi
find "$path_to_fix" -type f -name "*.pyc" | xargs /usr/bin/marshalparser --fix --overwrite