tests | ||
.gitignore | ||
compare_mandata.py | ||
LICENSE | ||
macros.pyproject | ||
pyproject_buildrequires_testcases.yaml | ||
pyproject_buildrequires.py | ||
pyproject_construct_toxenv.py | ||
pyproject_convert.py | ||
pyproject_preprocess_record.py | ||
pyproject_requirements_txt.py | ||
pyproject_save_files_test_data.yaml | ||
pyproject_save_files.py | ||
pyproject_wheel.py | ||
pyproject-rpm-macros.rpmlintrc | ||
pyproject-rpm-macros.spec | ||
README.md | ||
rpminspect.yaml | ||
sources | ||
test_pyproject_buildrequires.py | ||
test_pyproject_requirements_txt.py | ||
test_pyproject_save_files.py | ||
test_RECORD |
pyproject RPM macros
These macros allow projects that follow the Python packaging specifications to be packaged as RPMs.
They work for:
- traditional Setuptools-based projects that use the
setup.py
file, - newer Setuptools-based projects that have a
setup.cfg
file, - general Python projects that use the PEP 517
pyproject.toml
file (which allows using any build system, such as setuptools, flit or poetry).
These macros replace %py3_build
and %py3_install
, which only work with setup.py
.
Usage
To use these macros, first BuildRequire the devel package for the Python you
are building against. In Fedora, that's python3-devel
.
BuildRequires: python3-devel
The macros will be pulled in as a dependency on Fedora and EPEL 9+.
In other distributions you need to BuildRequire the macros as well:
BuildRequires: python3-devel
BuildRequires: pyproject-rpm-macros
Next, you need to generate more build dependencies (of your projects and
the macros themselves) by running %pyproject_buildrequires
in the
%generate_buildrequires
section:
%generate_buildrequires
%pyproject_buildrequires
This will add build dependencies according to PEP 517 and PEP 518. This also adds run-time dependencies by default and can add test-time dependencies, see the section below. If you need more dependencies, such as non-Python libraries, BuildRequire them manually.
Note that %generate_buildrequires
may produce error messages (exit 11)
in
the build log. This is expected behavior of BuildRequires generators; see
the Fedora change for details.
Then, build a wheel in %build
with %pyproject_wheel
:
%build
%pyproject_wheel
And install the wheel in %install
with %pyproject_install
:
%install
%pyproject_install
%pyproject_install
installs all wheels in pyproject-wheeldir/
located in the root of the source tree.
Adding run-time and test-time dependencies
To run tests or import checks in the %check
section,
the package's runtime dependencies need to also be included as build requirements.
Hence, %pyproject_buildrequires
also generates runtime dependencies by default.
For this to work, the project's build system must support the prepare-metadata-for-build-wheel hook. The popular buildsystems (setuptools, flit, poetry) do support it.
This behavior can be disabled
(e.g. when the project's build system does not support it)
using the -R
flag:
%generate_buildrequires
%pyproject_buildrequires -R
Alternatively, the runtime dependencies can be obtained by building the wheel and reading the metadata from the built wheel.
This can be enabled by using the -w
flag.
Support for building wheels with %pyproject_buildrequires -w
is provisional and the behavior might change.
Please subscribe to Fedora's python-devel list if you use the option.
%generate_buildrequires
%pyproject_buildrequires -w
When this is used, the wheel is going to be built at least twice,
becasue the %generate_buildrequires
section runs repeatedly.
To avoid accidentally reusing a wheel leaking from a previous (different) build,
it cannot be reused between %generate_buildrequires
rounds.
Contrarily to that, rebuilding the wheel again in the %build
section is redundant
and the packager can omit the %build
section entirely
to reuse the wheel built from the last round of %generate_buildrequires
.
Be extra careful when attempting to modify the sources after %pyproject_buildrequires
,
e.g. when running extra commands in the %build
section:
%build
cython src/wrong.pyx # this is too late with %%pyproject_buildrequires -w
%pyproject_wheel
For projects that specify test requirements using an extra
provide,
these can be added using the -x
flag.
Multiple extras can be supplied by repeating the flag or as a comma separated list.
For example, if upstream suggests installing test dependencies with
pip install mypackage[testing]
, the test deps would be generated by:
%generate_buildrequires
%pyproject_buildrequires -x testing
For projects that specify test requirements in their tox configuration,
these can be added using the -t
flag (default tox environment)
or the -e
flag followed by the tox environment.
The default tox environment (such as py37
assuming the Fedora's Python version is 3.7)
is available in the %{toxenv}
macro.
For example, if upstream suggests running the tests on Python 3.7 with tox -e py37
,
the test deps would be generated by:
%generate_buildrequires
%pyproject_buildrequires -t
If upstream uses a custom derived environment, such as py37-unit
, use:
%pyproject_buildrequires -e %{toxenv}-unit
Or specify more environments if needed:
%pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration
The -e
option redefines %{toxenv}
for further reuse.
Use %{default_toxenv}
to get the default value.
The -t
/-e
option uses tox-current-env's --print-deps-to-file
behind the scenes.
If your package specifies some tox plugins in tox.requires
,
such plugins will be BuildRequired as well.
Not all plugins are guaranteed to play well with tox-current-env,
in worst case, patch/sed the requirement out from the tox configuration.
Note that neither -x
or -t
can be used with -R
,
because runtime dependencies are always required for testing.
You can only use those options if the build backend supports the prepare-metadata-for-build-wheel hook,
or together with -w
.
Additionally to generated requirements you can supply multiple file names to %pyproject_buildrequires
macro.
Dependencies will be loaded from them:
%pyproject_buildrequires requirements/tests.in requirements/docs.in requirements/dev.in
For packages not using build system you can use -N
to entirely skip automatical
generation of requirements and install requirements only from manually specified files.
-N
option implies -R
and cannot be used in combination with other options mentioned above
(-w
, -e
, -t
, -x
).
The %pyproject_buildrequires
macro also accepts the -r
flag for backward compatibility;
it means "include runtime dependencies" which has been the default since version 0-53.
Running tox based tests
In case you want to run the tests as specified in tox configuration,
you must use %pyproject_buildrequires
with -t
or -e
as explained above.
Then, use the %tox
macro in %check
:
%check
%tox
The macro:
- Sets environment variables via
%{py3_test_envvars}
, namely:- Always prepends
$PATH
with%{buildroot}%{_bindir}
- If not defined, sets
$PYTHONPATH
to%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}
- Always prepends
- If not defined, sets
$TOX_TESTENV_PASSENV
to*
- Runs
tox
with-q
(quiet),--recreate
and--current-env
(from tox-current-env) flags - Implicitly uses the tox environment name stored in
%{toxenv}
- as overridden by%pyproject_buildrequires -e
By using the -e
flag, you can use a different tox environment(s):
%check
%tox
%if %{with integration_test}
%tox -e %{default_toxenv}-integration
%endif
If you wish to provide custom tox
flags or arguments, add them after --
:
%tox -- --flag-for-tox
If you wish to pass custom posargs
to tox, use another --
:
%tox -- --flag-for-tox -- --flag-for-posargs
Or (note the two sequential --
s):
%tox -- -- --flag-for-posargs
Generating the %files section
To generate the list of files in the %files
section, you can use %pyproject_save_files
after the %pyproject_install
macro.
It takes toplevel module names (i.e. the names used with import
in Python) and stores paths for those modules and metadata for the package (dist-info directory) to a file stored at %{pyproject_files}
.
For example, if a package provides the modules requests
and _requests
, write:
%install
%pyproject_install
%pyproject_save_files requests _requests
To add listed files to the %files
section, use %files -f %{pyproject_files}
.
Note that you still need to add any documentation manually (for now).
%files -n python3-requests -f %{pyproject_files}
%doc README.rst
You can use globs in the module names if listing them explicitly would be too tedious:
%install
%pyproject_install
%pyproject_save_files '*requests'
In fully automated environments, you can use the *
glob to include all modules (put it in single quotes to prevent Shell from expanding it). In Fedora however, you should always use a more specific glob to avoid accidentally packaging unwanted files (for example, a top level module named test
).
Speaking about automated environments, some files cannot be classified with %pyproject_save_files
, but it is possible to list all unclassified files by adding a special +auto
argument.
%install
%pyproject_install
%pyproject_save_files '*' +auto
%files -n python3-requests -f %{pyproject_files}
However, in Fedora packages, always list executables explicitly to avoid unintended collisions with other packages or accidental missing executables:
%install
%pyproject_install
%pyproject_save_files requests _requests
%files -n python3-requests -f %{pyproject_files}
%doc README.rst
%{_bindir}/downloader
%pyproject_save_files
can automatically mark license files with %license
macro
and language (*.mo
) files with %lang
macro and appropriate language code.
Only license files declared via PEP 639 License-File
field are detected.
PEP 639 is still a draft and can be changed in the future.
Note that %pyproject_save_files
uses data from the RECORD file.
If you wish to rename, remove or otherwise change the installed files of a package
after %pyproject_install
, %pyproject_save_files
might break.
If possible, remove/rename such files in %prep
.
If not possible, avoid using %pyproject_save_files
or edit/replace %{pyproject_files}
.
Performing an import check on all importable modules
If the upstream test suite cannot be used during the package build
and you use %pyproject_save_files
,
you can benefit from the %pyproject_check_import
macro.
If %pyproject_save_files
is not used, calling %pyproject_check_import
will fail.
When %pyproject_save_files
is invoked,
it creates a list of all valid and public (i.e. not starting with _
)
importable module names found in the package.
Each top-level module name matches at least one of the globs provided as an argument to %pyproject_save_files
.
This list is then usable by %pyproject_check_import
which performs an import check for each listed module.
When a module fails to import, the build fails.
The modules are imported from both installed and buildroot's %{python3_sitearch}
and %{python3_sitelib}
, not from the current directory.
Use the macro in %check
:
%check
%pyproject_check_import
By using the -e
flag, you can exclude module names matching the given glob(s) from the import check
(put it in single quotes to prevent Shell from expanding it).
The flag can be used repeatedly.
For example, to exclude all submodules ending with config
and all submodules starting with test
, you can use:
%pyproject_check_import -e '*.config' -e '*.test*'
There must be at least one module left for the import check; if, as a result of greedy excluding, no modules are left to check, the check fails.
When the -t
flag is used, only top-level modules are checked,
qualified module names with a dot (.
) are excluded.
If the modules detected by %pyproject_save_files
are requests
, requests.models
, and requests.packages
, this will only perform an import of requests
:
%pyproject_check_import -t
The modifying flags should only be used when there is a valid reason for not checking all available modules. The reason should be documented in a comment.
The %pyproject_check_import
macro also accepts positional arguments with
additional qualified module names to check, useful for example if some modules are installed manually.
Note that filtering by -t
/-e
also applies to the positional arguments.
Generating Extras subpackages
The %pyproject_extras_subpkg
macro generates simple subpackage(s)
for Python extras.
The macro should be placed after the base package's %description
to avoid
issues in building the SRPM.
For example, if the requests
project's metadata defines the extras
security
and socks
, the following invocation will generate the subpackage
python3-requests+security
that provides python3dist(requests[security])
,
and a similar one for socks
.
%pyproject_extras_subpkg -n python3-requests security socks
The macro works like %python_extras_subpkg
,
except the -i
/-f
/-F
arguments are optional and discouraged.
A filelist written by %pyproject_install
is used by default.
For more information on %python_extras_subpkg
, see the Fedora change.
These arguments are still required:
- -n: name of the “base” package (e.g. python3-requests)
- Positional arguments: the extra name(s). Multiple subpackages are generated when multiple names are provided.
PROVISIONAL: Importing just-built (extension) modules in %build
Sometimes, it is desired to be able to import the just-built extension modules
in the %build
section, e.g. to build the documentation with Sphinx.
%build
%pyproject_wheel
... build the docs here ...
With pure Python packages, it might be possible to set PYTHONPATH=${PWD}
or PYTHONPATH=${PWD}/src
.
However, it is a bit more complicated with extension modules.
The location of just-built modules might differ depending on Python version, architecture, pip version, etc.
Hence, the macro %{pyproject_build_lib}
exists to be used like this:
%build
%pyproject_wheel
PYTHONPATH=%{pyproject_build_lib} ... build the docs here ...
This macro is currently provisional and the behavior might change. Please subscribe to Fedora's python-devel list if you use the macro.
The %{pyproject_build_lib}
macro expands to an Shell $(...)
expression and does not work when put into single quotes ('
).
Depending on the pip version, the expanded value will differ:
New pip 21.3+ with in-tree-build and setuptools 62.1+ (Fedora 37+)
Always use the macro from the same directory where you called %pyproject_wheel
from.
The value will expand to something like:
/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-cpython-311
for wheels with extension modules/builddir/build/BUILD/%{name}-%{version}/build/lib
for pure Python wheels
If multiple wheels were built from the same directory,
some pure Python and some with extension modules,
the expanded value will be combined with :
:
/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-cypthon-311:/builddir/build/BUILD/%{name}-%{version}/build/lib
If multiple wheels were built from different directories, the value will differ depending on the current directory.
New pip 21.3+ with in-tree-build and older setuptools (Fedora 36)
Always use the macro from the same directory where you called %pyproject_wheel
from.
The value will expand to something like:
/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-3.10
for wheels with extension modules/builddir/build/BUILD/%{name}-%{version}/build/lib
for pure Python wheels
If multiple wheels were built from the same directory,
some pure Python and some with extension modules,
the expanded value will be combined with :
:
/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/build/lib
If multiple wheels were built from different directories, the value will differ depending on the current directory.
Older pip with out-of-tree-build (Fedora 35 and EL 9)
The value will expand to something like:
/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10
for wheels with extension modules/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib
for pure Python wheels
Note that the exact value is not stable between builds
(the xxxxxxxx
part is randomly generated,
neither you should consider the .pyproject-builddir
directory to remain stable).
If multiple wheels are built,
the expanded value will always be combined with :
regardless of the current directory, e.g.:
/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-yyyyyyyy/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-zzzzzzzz/build/lib
Note: If you manage to build some wheels with in-tree-build and some with out-of-tree-build option, the expanded value will contain all relevant directories.
Limitations
%pyproject_install
changes shebang lines of every Python script in %{buildroot}%{_bindir}
to #!%{__python3} %{py3_shbang_opt}
(#!/usr/bin/python3 -s
).
Existing Python flags in shebangs are preserved.
For example #!/usr/bin/python3 -Ru
will be updated to #!/usr/bin/python3 -sRu
.
Sometimes, this can interfere with tests that run such scripts directly by name,
because in tests we usually rely on PYTHONPATH
(and -s
ignores that).
Would this behavior be undesired for any reason,
undefine %{py3_shbang_opt}
to turn it off.
Some valid Python version specifiers are not supported.
When a dependency is specified via an URL or local path, for example as:
https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6fc54099622be.zip
/some/path/foo-1.2.3.tar.gz
git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3
The %pyproject_buildrequires
macro is unable to convert it to an appropriate RPM requirement and will fail.
If the URL contains the packageName @
prefix as specified in PEP 508,
the requirement will be generated without a version constraint:
appdirs@https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6fc54099622be.zip
foo@file:///some/path/foo-1.2.3.tar.gz
Will be converted to:
python3dist(appdirs)
python3dist(foo)
Alternatively, when an URL requirement parsed from a text file
given as positional argument to %pyproject_buildrequires
contains the #egg=packageName
fragment,
as documented in pip's documentation:
git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3#egg=sphinx
The requirements will be converted to package names without versions, e.g.:
python3dist(sphinx)
However upstreams usually only use direct URLs for their requirements as workarounds, so be prepared for problems.
Testing the macros
This repository has two kinds of tests.
First, there is RPM %check
section, run when building the python-rpm-macros
package.
Then there are CI tests.
There is currently no way to run Fedora CI tests locally,
but you can do what the tests do manually using mock.
For each $PKG.spec
in tests/
:
-
clean your mock environment:
mock -r fedora-rawhide-x86_64 clean
-
install the version of
python-rpm-macros
you're testing, e.g.:mock -r fedora-rawhide-x86_64 install .../python-rpm-macros-*.noarch.rpm
-
download the sources:
spectool -g -R $PKG.spec
-
build a SRPM:
rpmbuild -bs $PKG.spec
-
build in mock, using the path from the command above as
$SRPM
:mock -r fedora-rawhide-x86_64 -n -N $SRPM