A new script brp-fix-pyc-reproducibility creates an opt-in way of how to fix
problems with the reproducibility of byte-compiled Python files. The script
uses marshalparser [0] which currently doesn't provide solutions for all issues
but can fix at least problems with reference flags. For more info see
this Bugzilla [1].
If you want to use this new feature, you need to define
`%py_reproducible_pyc_path` to specify a path you want to fix `.pyc`
files in (recursively) and build-require /usr/bin/marshalparser.
if you forget to build-require the parser. The error message is:
```
+ /usr/lib/rpm/redhat/brp-python-bytecompile '' 1 0
Bytecompiling .py files below /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64/usr/lib/python3.9 using /usr/bin/python3.9
+ /usr/lib/rpm/redhat/brp-fix-pyc-reproducibility /builddir/build/BUILDROOT/tldr-0.5-2.fc33.x86_64
ERROR: If %py_reproducible_pyc_path is defined, you have to also BuildRequire: /usr/bin/marshalparser !
error: Bad exit status from /var/tmp/rpm-tmp.UUJr4v (%install)
```
A build fails if the parser is not able to parse any of the `.pyc` files.
And finally, if a build is properly configured it produces fixed `.pyc` files.
Currently, `.pyc` files in the tldr package contain a lot of unused reference flags:
```
$ dnf install -y tldr
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
… long output …
190 - Flag_ref(byte=9610, type='TYPE_SHORT_ASCII_INTERNED', content=b'init', usages=0)
191 - Flag_ref(byte=9633, type='TYPE_SHORT_ASCII_INTERNED', content=b'source', usages=0)
192 - Flag_ref(byte=9651, type='TYPE_SHORT_ASCII_INTERNED', content=b'argv', usages=0)
193 - Flag_ref(byte=9657, type='TYPE_SHORT_ASCII_INTERNED', content=b'print_help', usages=0)
194 - Flag_ref(byte=9669, type='TYPE_SHORT_ASCII_INTERNED', content=b'stderr', usages=0)
195 - Flag_ref(byte=9682, type='TYPE_SHORT_ASCII_INTERNED', content=b'parse_args', usages=0)
196 - Flag_ref(byte=9737, type='TYPE_SHORT_ASCII_INTERNED', content=b'encode', usages=0)
197 - Flag_ref(byte=9782, type='TYPE_SHORT_ASCII_INTERNED', content=b'parser', usages=0)
198 - Flag_ref(byte=9790, type='TYPE_SHORT_ASCII_INTERNED', content=b'options', usages=0)
199 - Flag_ref(byte=9799, type='TYPE_SHORT_ASCII_INTERNED', content=b'rest', usages=0)
200 - Flag_ref(byte=9821, type='TYPE_SHORT_ASCII_INTERNED', content=b'result', usages=0)
202 - Flag_ref(byte=10022, type='TYPE_SHORT_ASCII_INTERNED', content=b'__main__', usages=0)
203 - Flag_ref(byte=10102, type='TYPE_SHORT_ASCII_INTERNED', content=b'argparse', usages=0)
204 - Flag_ref(byte=10433, type='TYPE_SHORT_ASCII_INTERNED', content=b'__name__', usages=0)
205 - Flag_ref(byte=10463, type='TYPE_SHORT_ASCII_INTERNED', content=b'<module>', usages=0)
```
This new feature fixes them:
```
$ marshalparser --unused /usr/lib/python3.9/site-packages/__pycache__/tldr.cpython-39.pyc
<empty output>
```
[0] https://github.com/fedora-python/marshalparser
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1686078
Remove %new_package dependency on %{source_name} when creating subpackages by
suffix. This way those subpackages still work even when the packager set
%{source_name} to a value and Name: to another.
Arguably, the packager is severily conflicted, and does not know what he wants
to achieve, but this case is easy enough to accomodate by tweaking the decision
tree. So let’s just make things work instead of blaming the packager.
Make SRPM handover between macros even more graceful and reliable by auto-creating
a basic SRPM header before attempting declaration of a different sub-package.
With this change things will just work as long as the default %{source_name}
%{source_summary} and %{source_description} are set by something to sensible values.
%new_package is a wrapper around Name: and %package that abstracts their quirks
from packagers and macros. Its behavior is controled by the %{source_name}
global variable:
– when %{source_name} is not set, the first call to %new_package will create a
Name: block and set %{source_name} to the %{name} of this block.
– when %{source_name} is set:
– a call to %new_package with no arguments creates:
Name: %{source_name}
– otherwise, a call to %new_package creates the corresponding:
%package…
line, unless the resulting %{name} matches %{source_name}. In that case it
creates:
Name: %{source_name}
as before.
Arguments:
– -n and %1 like %package
– -v to print the variables %new_package sets directly.
The intended use-case it to:
– simplify coordination between macros that create subpackages,
– make it easy for packagers to declare which of the macro-created packages
owns the SRPM, and
– make %{source_name} available within spec files and not just as a dnf
synthetic variable.
Unlike %{name} %{source_name} matches the SRPM name regardless of its location
within the spec file.
The check_rhel function should return the same thing on CentOS as it
does on RHEL. Currently CentOS applies this modification downstream.
Now that CentOS is part of the Red Hat family, it would be ideal to push
this modification upstream.
This macro calls `rpm` on background, which is not good idea. Luckily,
it seems to be used just by samba package, so it should not cause any
substantial issues.
More details at \[[1]\] where the guideline to ban `rpm` call during
build is discussed.
[1]: https://pagure.io/packaging-committee/pull-request/954
This cuts the kernel provide generation time from ~33s to 2.5s on my laptop.
Tighten the path matching rule a bit while at it - it doesn't matter that
much with parametric generator but there's no point looking at entries
we don't generate dependencies on.
The Python compileall2 module in /usr/lib/rpm/redhat/
can be executed by various different Python interpreters.
We don't want to write several different `*.pyc` files
to this location - in most cases, that's not possible,
but somebody might run this as root.
For kernel builds, /usr/lib/rpm/kmod.prov is fork+execed by rpmbuild
in "Processing files:" step about 8000 times, single-threaded,
with cumulative run time of ~2 minutes.
Speed up this script, by avoiding additional fork+execing.
Tested to work, observed speedup: almost exactly 2 times faster.
While verifying correctness, noticed that old script was buggy -
it was generating a bogus "Provides:" item - kmod(modules.builtin.modinfo),
because the logic in script was filtering for */*.ko files and
for */modules.builtin* files, and wasn't prepared for
the existence of */modules.builtin.modinfo file.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
This config is to let libtool recognize that our 64bit variant of
%_libdir is actually on the standard/default library path, so libtool
doesn't think it has to be hard-wired as RPATH. This is proper solution
for libtool RPATH issues described in:
https://docs.fedoraproject.org/en-US/packaging-guidelines/#_removing_rpath
The libtool script/macros (new enough, v2.4.6+) honor this variable when
it isn't possible to detect the system-wide default library path. It is
e.g. able to parse /etc/ld.so.* configuration, but there's no info about
/usr/lib64 on Fedora.
So to not force everybody to do:
%configure LT_SYS_LIBRARY_PATH=...
... rather set this system-wide. This is low-risk change since
older libtool scripts don't use this variable, and really no other
tools should.
– handle CR in addition to LF
– be smarter in presence of lists
– fix off-by-one mistake when a LF in the processed text falls exactly on the 81st column
trim() {
printf '%s' "$*"
}
...
read shebang_line < "$f" || :
orig_shebang=$(trim $(echo "$shebang_line" | grep -Po "#!\K.*" || echo))
The "trimming", i.e. replacement of multiple spaces and removal of leading
and trailing spaces, is achieved because "trim $(cmd)" construct has an
unquoted $(), which is subject to word splitting.
This works, yes. BUT.
It is also subject to glob expansion - any ?s and *s will be attempted
to be expanded as well - definitely NOT what we want!
This change replaces this trick with code which avoids the expansion issue,
and which does not spawn any subprocesses for string manipulations -
this is ~3 times faster (fork+execs are expensive).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Resolves: rhbz#1595265
The problem this change is intended to solve is with how `real_libdir`
is calculated. Let's assume we want to recursively byte-compile all
`*.py` files in
`/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8`.
Then, `real_libdir` is this path without `$RPM_BUILD_ROOT` with
the filename at the end which displays in the error message like this:
```
Bytecompiling .py files below /builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8 using /usr/bin/python3.8
*** Error compiling '/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8/site-packages/greplin/bar.py'...
File "/usr/lib/python3.8/bar.py", line 1
import sin from math
^
SyntaxError: invalid syntax
```
`/usr/lib/python3.8/bar.py` is obviously wrong.
One of the new features of the `compileall2` module (which will
be available in stdlib in Python 3.9) is that the path byte-compiled to
`*.pyc` files is calculated for each file. This means that by using
`-s` and `-p` we can strip `$RPM_BUILD_ROOT` and prepend `/` for each
file individually which will fix the problem.
```
Bytecompiling .py files below /builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8 using /usr/bin/python3.8
*** Error compiling '/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8/site-packages/greplin/bar.py'...
File "/usr/lib/python3.8/site-packages/greplin/bar.py", line 1
import sin from math
^
SyntaxError: invalid syntax
```
This change has an effect only for Python >= 3.4.
Instead of:
%{gpgverify} --keyring='%{SOURCE2}' --signature='%{SOURCE1}' --data='%{SOURCE0}'
It is now possible to do:
%gpgverify -k2 -s1 -d0
I haven't yet assumed any defaults not to break backwards compatibility.
