pyproject-rpm-macros/README.md

pyproject RPM macros
====================

These macros allow projects that follow the Python [packaging specifications]
to be packaged as RPMs.

They are still *provisional*: we can make non-backwards-compatible changes to
the API.
Please subscribe to Fedora's [python-devel list] if you use the macros.

They work for:

* traditional Setuptools-based projects that use the `setup.py` file,
* newer Setuptools-based projects that have a `setup.cfg` file,
* general Python projects that use the [PEP 517] `pyproject.toml` file (which allows using any build system, such as setuptools, flit or poetry).

These macros replace `%py3_build` and `%py3_install`, which only work with `setup.py`.

[packaging specifications]: https://packaging.python.org/specifications/
[python-devel list]: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/


Usage
-----

To use these macros, first BuildRequire them:

    BuildRequires: pyproject-rpm-macros

Also BuildRequire the devel package for the Python you are building against.
In Fedora, that's `python3-devel`.
(In the future, we plan to make `python3-devel` itself require
`pyproject-rpm-macros`.)

Next, you need to generate more build dependencies (of your projects and
the macros themselves) by running `%pyproject_buildrequires` in the
`%generate_buildrequires` section:

    %generate_buildrequires
    %pyproject_buildrequires

This will add build dependencies according to [PEP 517] and [PEP 518].
To also add run-time and test-time dependencies, see the section below.
If you need more dependencies, such as non-Python libraries, BuildRequire
them manually.

Note that `%generate_buildrequires` may produce error messages `(exit 11)` in
the build log. This is expected behavior of BuildRequires generators; see
[the Fedora change] for details.

[the Fedora change]: https://fedoraproject.org/wiki/Changes/DynamicBuildRequires

Then, build a wheel in `%build` with `%pyproject_wheel`:

    %build
    %pyproject_wheel

And install the wheel in `%install` with `%pyproject_install`:

    %install
    %pyproject_install

`%pyproject_install` installs all wheels in `$PWD/pyproject-wheeldir/`.


Adding run-time and test-time dependencies
------------------------------------------

To run tests in the `%check` section, the package's runtime dependencies
often need to also be included as build requirements.
This can be done using the `-r` flag:

    %generate_buildrequires
    %pyproject_buildrequires -r

For this to work, the project's build system must support the
[`prepare-metadata-for-build-wheel` hook](https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel).
The popular buildsystems (setuptools, flit, poetry) do support it.

For projects that specify test requirements using an [`extra`
provide](https://packaging.python.org/specifications/core-metadata/#provides-extra-multiple-use),
these can be added using the `-x` flag.
Multiple extras can be supplied by repeating the flag or as a comma separated list.
For example, if upstream suggests installing test dependencies with
`pip install mypackage[testing]`, the test deps would be generated by:

    %generate_buildrequires
    %pyproject_buildrequires -x testing

For projects that specify test requirements in their [tox] configuration,
these can be added using the `-t` flag (default tox environment)
or the `-e` flag followed by the tox environment.
The default tox environment (such as `py37` assuming the Fedora's Python version is 3.7)
is available in the `%{toxenv}` macro.
For example, if upstream suggests running the tests on Python 3.7 with `tox -e py37`,
the test deps would be generated by:

    %generate_buildrequires
    %pyproject_buildrequires -t

If upstream uses a custom derived environment, such as `py37-unit`, use:

    %pyproject_buildrequires -e %{toxenv}-unit

Or specify more environments if needed:

    %pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration

The `-e` option redefines `%{toxenv}` for further reuse.
Use `%{default_toxenv}` to get the default value.

The `-t`/`-e` option uses [tox-current-env]'s `--print-deps-to-file` behind the scenes.

Note that both `-x` and `-t` imply `-r`,
because runtime dependencies are always required for testing.

[tox]: https://tox.readthedocs.io/
[tox-current-env]: https://github.com/fedora-python/tox-current-env/


Running tox based tests
-----------------------

In case you want to run the tests as specified in [tox] configuration,
you must use `%pyproject_buildrequires` with `-t` or `-e` as explained above.
Then, use the `%tox` macro in `%check`:

    %check
    %tox

The macro:

 - Always prepends `$PATH` with `%{buildroot}%{_bindir}`
 - If not defined, sets `$PYTHONPATH` to `%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}`
 - If not defined, sets `$TOX_TESTENV_PASSENV` to `*`
 - Runs `tox` with `-q` (quiet), `--recreate` and `--current-env` (from [tox-current-env]) flags
 - Implicitly uses the tox environment name stored in `%{toxenv}` - as overridden by `%pyproject_buildrequires -e`

By using the `-e` flag, you can use a different tox environment(s):

    %check
    %tox
    %if %{with integration_test}
    %tox -e %{default_toxenv}-integration
    %endif

If you wish to provide custom `tox` flags or arguments, add them after `--`:

    %tox -- --flag-for-tox

If you wish to pass custom `posargs` to tox, use another `--`:

    %tox -- --flag-for-tox -- --flag-for-posargs

Or (note the two sequential `--`s):

    %tox -- -- --flag-for-posargs



Generating the %files section
-----------------------------

To generate the list of files in the `%files` section, you can use `%pyproject_save_files` after the `%pyproject_install` macro.
It takes toplevel module names (i.e. the names used with `import` in Python) and stores paths for those modules and metadata for the package (dist-info directory) to a file stored at `%{pyproject_files}`.
For example, if a package provides the modules `requests` and `_requests`, write:

    %install
    %pyproject_install
    %pyproject_save_files requests _requests

To add listed files to the `%files` section, use `%files -f %{pyproject_files}`.
Note that you still need to add any documentation and license manually (for now).

    %files -n python3-requests -f %{pyproject_files}
    %doc README.rst
    %license LICENSE

You can use globs in the module names if listing them explicitly would be too tedious:

    %install
    %pyproject_install
    %pyproject_save_files '*requests'

In fully automated environments, you can use the `*` glob to include all modules (put it in single quotes to prevent Shell from expanding it). In Fedora however, you should always use a more specific glob to avoid accidentally packaging unwanted files (for example, a top level module named `test`).

Speaking about automated environments, some files cannot be classified with `%pyproject_save_files`, but it is possible to list all unclassified files by adding a special `+auto` argument.

    %install
    %pyproject_install
    %pyproject_save_files '*' +auto
    
    %files -n python3-requests -f %{pyproject_files}

However, in Fedora packages, always list executables explicitly to avoid unintended collisions with other packages or accidental missing executables:

    %install
    %pyproject_install
    %pyproject_save_files requests _requests
    
    %files -n python3-requests -f %{pyproject_files}
    %doc README.rst
    %license LICENSE
    %{_bindir}/downloader

`%pyproject_save_files` also automatically recognizes language (`*.mo`) files and marks them with `%lang` macro and appropriate language code.
Note that RPM might warn about such files listed twice:

    warning: File listed twice: /usr/lib/python3.9/site-packages/django/conf/locale/af/LC_MESSAGES/django.mo

The warning is harmless.


Generating Extras subpackages
-----------------------------

The `%pyproject_extras_subpkg` macro generates simple subpackage(s)
for Python extras.

The macro should be placed after the base package's `%description` to avoid
issues in building the SRPM.

For example, if the `requests` project's metadata defines the extras
`security` and `socks`, the following invocation will generate the subpackage
`python3-requests+security` that provides `python3dist(requests[security])`,
and a similar one for `socks`.

    %pyproject_extras_subpkg -n python3-requests security socks

The macro works like `%python_extras_subpkg`,
except the `-i`/`-f`/`-F` arguments are optional and discouraged.
A filelist written by `%pyproject_install` is used by default.
For more information on `%python_extras_subpkg`, see the [Fedora change].

[Fedora change]: https://fedoraproject.org/wiki/Changes/PythonExtras

These arguments are still required:

* -n: name of the “base” package (e.g. python3-requests)
* Positional arguments: the extra name(s).
  Multiple subpackages are generated when multiple names are provided.

The macro does nothing on Fedora 32 and lower, as automation around
extras was only added in f33.


Limitations
-----------

`%pyproject_install` changes shebang lines of every Python script in `%{buildroot}%{_bindir}` to `#!%{__python3} %{py3_shbang_opt}` (`#!/usr/bin/python3 -s`).
Existing Python flags in shebangs are preserved.
For example `#!/usr/bin/python3 -Ru` will be updated to `#!/usr/bin/python3 -sRu`.
Sometimes, this can interfere with tests that run such scripts directly by name,
because in tests we usually rely on `PYTHONPATH` (and `-s` ignores that).
Would this behavior be undesired for any reason,
undefine `%{py3_shbang_opt}` to turn it off.

Some valid Python version specifiers are not supported.

[PEP 517]: https://www.python.org/dev/peps/pep-0517/
[PEP 518]: https://www.python.org/dev/peps/pep-0518/


Testing the macros
------------------

This repository has two kinds of tests.
First, there is RPM `%check` section, run when building the `python-rpm-macros`
package.

Then there are CI tests.
There is currently [no way to run Fedora CI tests locally][ci-rfe],
but you can do what the tests do manually using mock.
For each `$PKG.spec` in `tests/`:

  - clean your mock environment:

        mock -r fedora-rawhide-x86_64 clean

  - install the version of `python-rpm-macros` you're testing, e.g.:

        mock -r fedora-rawhide-x86_64 install .../python-rpm-macros-*.noarch.rpm

  - download the sources:

        spectool -g -R $PKG.spec

  - build a SRPM:

        rpmbuild -bs $PKG.spec

  - build in mock, using the path from the command above as `$SRPM`:

        mock -r fedora-rawhide-x86_64 -n -N $SRPM

[ci-rfe]: https://pagure.io/fedora-ci/general/issue/4