This adds a new `%_python_memoize` macro that caches the values of
various python macros in a lua table based on the current value of
`%__python` / `%__python3`. The Python macros are adjusted to use this
macro for memoization instead of the current approach. This way, the
macros will return values that apply to the _current_ `%__python3` value
when packagers create multi-python specfiles that toggle the value of
`%python3_pkgversion`.
Relates: https://bugzilla.redhat.com/2209055
Co-Authored-By: Miro Hrončok <miro@hroncok.cz>
%py3_install_egg is nonfunctional; setuptools removed the easy_install
entrypoint years ago.
%py3_build_egg is technically functional but has been superseded by
newer macros. Calling setup.py directly is deprecated, and
building/installing eggs to begin with is deprecated.
The macros are not used in any Fedora packages and are broken.
It's time to remove them.
Macros like %{python3_sitelib} were evaluated at every instance in the spec
(each time, Python was started, sysconfig was imported...).
When there were many instances (40+), it might have taken more than a minute to parse the spec file.
This way, the macros are defined via %global on first usage.
Every repetitive usage reuses the actual value.
When reviewing https://src.fedoraproject.org/rpms/pyproject-rpm-macros/pull-request/291
we have discovered that there is no macronized way to get this part of some paths
and that packagers need to hardcode it as cpython-%{python3_version_nodots}.
This way, we have a standardized macro packagers (and other macros) can use.
%%py{3}_check_import now respects the custom setting of %%py{3}_shebang_flags
and invokes Python with the respective values.
If %%py{3}_shebang_flags is undefined or set to no value,
there no flags are passed to Python on invoke.
Resolves: rhbz#2018615
-f: optionally read a file with module names to test
-t: bool flag - if set, filter only top-level modules
-e: optionally exclude module names matching the given glob (Unix
shell-style wildcards)
Importing all modules may cause bogus failures in some cases,
eg. when the imported code assumes there is an existing graphical window.
Such behaviour may be by design, hence for automatic processing it's
more convinient to - in some cases - check only for top-level modules
or filter out the troublemakers.
Our Pythons currently patches distutils to install packages to
/usr/lib(64)/pythonX.Y/site-packages when the $RPM_BUILD_ROOT environment
variable is set (and to /usr/local/lib(64)/pythonX.Y/site-packages otherwise).
With the deprecation of distutils [1] we want to change the patch to create
and use a different sysconfig install scheme [2].
However, we have realized that macros defined as %(%{__python3} ...) don't
"see" the environment variables set by rpmbuild because they are expanded earlier
and hence e.g. %{python3_sitelib} evaluates to
/usr/local/lib/python3.X/site-packages -- which is not desired.
To be able to reliably detect an RPM build environment by checking
the presence of the $RPM_BUILD_ROOT environment variable,
we manually set it in the macro definitions.
Since %{buildroot} in not fully populated
(e.g. it can expand literally to
/home/anna/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64),
we don't use it here.
The variable simply needs to present in the environment.
See also the analysis of the build failures when this is not done [3].
[1] https://www.python.org/dev/peps/pep-0632/
[2] https://bugs.python.org/issue43976
[3] https://src.fedoraproject.org/rpms/python3.10/pull-request/63#comment-79042
With $PATH and $PYTHONPATH set to the %buildroot,
the macro tries to import the given Python 3 module(s).
Useful as a smoke test in %check when ruining tests is not feasible.
Accepts spaces or commas as separators.
Package python-six:
%check
%py3_check_import six
Executing(%check): ...
...
+ PATH=...
+ PYTHONPATH=...
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import six'
+ RPM_EC=0
++ jobs -p
+ exit 0
%py3_check_import six seven
...
+ /usr/bin/python3 -c 'import six, seven'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'seven'
error: Bad exit status from ... (%check)
...
%py3_check_import five, six, seven
+ /usr/bin/python3 -c 'import five, six, seven'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'five'
error: Bad exit status from ... (%check)
Package python-packaging:
%py3_check_import packaging, packaging.markers packaging.requirements, packaging.tags
Executing(%check): ...
...
+ PATH=...
+ PYTHONPATH=...
+ PYTHONDONTWRITEBYTECODE=1
+ /usr/bin/python3 -c 'import packaging, packaging.markers, packaging.requirements, packaging.tags'
+ RPM_EC=0
++ jobs -p
+ exit 0
%py3_check_import packaging, packaging.markers packaging.notachance, packaging.tags
...
+ /usr/bin/python3 -c 'import packaging, packaging.markers, packaging.notachance, packaging.tags'
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'packaging.notachance'
error: Bad exit status from ... (%check)
Distutils which were used to define the macros are deprecated in Python3.10:
https://www.python.org/dev/peps/pep-0632/.
Sysconfig isn't and it works across our Pythons, making it better choice for the task.
Versioned pathfixX.Y.py is available in main as well as in
alternative Pythons so this change enables to build
an alternative Python stack without a dependency on the main
python3-devel.
--no-index saves a ~1 minute timeout of pip possibly checking if there is
a newer pip version (in mock with disabled network).
--no-warn-script-location removes a bogus warning that says:
WARNING: The scripts easy_install and easy_install-3.9 are installed in
'/builddir/build/BUILDROOT/python-setuptools-49.2.0-1.fc33.x86_64/usr/bin'
which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning,
use --no-warn-script-location.
See https://fedoraproject.org/wiki/Changes/PythonMacroError
While doing it, make %python macros more consistent with %python3 macros,
mostly wrt whitespace but also to use python -m pip over plain pip etc.
One significant change is the removal of sleeps from python macros,
this could affect packages that use python macros to build for Python 2
while also using python3 macros to build for Python 3.
In reality, I consider that unlikely. The sleep in python2 macros stays.
The --strip-file-prefix option was already removed from %pyX_install_wheel
but we forgot to remove it from %py_install_wheel.
Previous implementation allowed for only one argument to be passed to
the %pycached macro, which made it impossible to combine it with other macros.
Current implementation allows to pass other macros as arguments to
%pycached.
Example:
%pycached %exclude /path/to/foo.py
For macro expansion limitations, the opposite order is not possible.
That is to be documented in the guidelines:
https://pagure.io/packaging-committee/pull-request/986
Added some tests.
Resolves https://bugzilla.redhat.com/show_bug.cgi?id=1838992
Co-authored-by: Marcel Plch <mplch@redhat.com>
Usage:
%files
...
%pycached %{python3_sitelib}/foo.py
This will list:
/usr/lib/python3.8/site-packages/foo.py
/usr/lib/python3.8/site-packages/__pycache__/foo.cpython-38{,.opt-?}.pyc
Assuming the Python 3 version is 3.8.
The bytecode files are globbed, their presence is not checked.
This will fail:
%pycached %{python3_sitelib}/foo
error: %pycached can only be used with paths explicitly ending with .py
And so will any of this:
%pycached %{python3_sitelib}/*
%pycached %{python3_sitelib}/foo.*
%pycached %{python3_sitelib}/foo.p?
%pycached %{python3_sitelib}/foo.?y
%pycached %{python3_sitelib}/foo.??
But this will work:
%pycached %{python3_sitelib}/foo*.py
And it will generate the following globs:
/usr/lib/python3.8/site-packages/foo*.py
/usr/lib/python3.8/site-packages/__pycache__/foo*.cpython-38{,.opt-?}.pyc
When used with paths that include Python 3 version, it globs with the version:
%pycached /opt/python3.10/foo.py
Generates:
/opt/python3.10/foo.py
/opt/python3.10/__pycache__/foo.cpython-310{,.opt-?}.pyc
While paths without version have less strict globs:
%pycached /custom/foo.py
/custom/foo.py
/custom/__pycache__/foo.cpython-3*{,.opt-?}.pyc
This will generate a warning in RPM build:
warning: File listed twice: /custom/__pycache__/foo.cpython-38.opt-1.pyc
However it ensures the optimized bytecode is there.
This is mostly used when building Sphinx docs on arched build.
Previously:
PLATFORM=$(python3 -c "import sysconfig; print(sysconfig.get_platform())")
export PYTHONPATH=../build/lib.${PLATFORM}-%{python3_version}
make man
Or:
PYTHONPATH=`realpath ../build/lib.linux*` make
Or:
PYTHONPATH=$(echo $PWD/build/lib.linux-*) make html
Now:
PYTHONPATH=../build/lib.%{python3_platform}-%{python3_version}
This is to ensure the right linker flags get propagated into binaries
accompanying python packages whenever distutils (or similar, env.
variables provided build flags aware) module is delegated to build them
(e.g. https://bugzilla.redhat.com/show_bug.cgi?id=1541106). Instead of
using "%{__global_ldflags}" as the seed for the respective env. variable
one-off application, which would be a direct parallel to "%{optflags}"
seeding CFLAGS, use rather "${RPM_LD_FLAGS}" and "${RPM_OPT_FLAGS}",
as these are not directly bound to macros out of rpm proper (like
it would have been, e.g., on redhat-rpm-config package otherwise).
Signed-off-by: Jan Pokorný <jpokorny@redhat.com>
