pyproject-rpm-macros/README.md

pyproject RPM macros
====================

These macros allow projects that follow the Python [packaging specifications]
to be packaged as RPMs.

They work for:

* traditional Setuptools-based projects that use the `setup.py` file,
* newer Setuptools-based projects that have a `setup.cfg` file,
* general Python projects that use the [PEP 517] `pyproject.toml` file (which allows using any build system, such as setuptools, flit or poetry).

These macros replace `%py3_build` and `%py3_install`, which only work with `setup.py`.

[packaging specifications]: https://packaging.python.org/specifications/


Usage
-----

To use these macros, first BuildRequire the devel package for the Python you
are building against. In Fedora, that's `python3-devel`.

    BuildRequires: python3-devel

The macros will be pulled in as a dependency on Fedora and EPEL 9+.  
In other distributions you need to BuildRequire the macros as well:

    BuildRequires: python3-devel
    BuildRequires: pyproject-rpm-macros

Next, you need to generate more build dependencies (of your projects and
the macros themselves) by running `%pyproject_buildrequires` in the
`%generate_buildrequires` section:

    %generate_buildrequires
    %pyproject_buildrequires

This will add build dependencies according to [PEP 517] and [PEP 518].
This also adds run-time dependencies by default and
can add test-time dependencies, see the section below.
If you need more dependencies, such as non-Python libraries, BuildRequire
them manually.

Note that `%generate_buildrequires` may produce error messages `(exit 11)` in
the build log. This is expected behavior of BuildRequires generators; see
[the Fedora change] for details.

[the Fedora change]: https://fedoraproject.org/wiki/Changes/DynamicBuildRequires

Then, build a wheel in `%build` with `%pyproject_wheel`:

    %build
    %pyproject_wheel

And install the wheel in `%install` with `%pyproject_install`:

    %install
    %pyproject_install

`%pyproject_install` installs all wheels in `pyproject-wheeldir/` located in the root of the source tree.


Adding run-time and test-time dependencies
------------------------------------------

To run tests or import checks in the `%check` section,
the package's runtime dependencies need to also be included as build requirements.

Hence, `%pyproject_buildrequires` also generates runtime dependencies by default.

For this to work, the project's build system must support the [prepare-metadata-for-build-wheel hook].
The popular buildsystems (setuptools, flit, poetry) do support it.

This behavior can be disabled
(e.g. when the project's build system does not support it)
using the `-R` flag:

    %generate_buildrequires
    %pyproject_buildrequires -R

Alternatively, the runtime dependencies can be obtained by building the wheel and reading the metadata from the built wheel.
This can be enabled by using the `-w` flag.
Support for building wheels with `%pyproject_buildrequires -w` is **provisional** and the behavior might change.
Please subscribe to Fedora's [python-devel list] if you use the option.

    %generate_buildrequires
    %pyproject_buildrequires -w

When this is used, the wheel is going to be built at least twice,
becasue the `%generate_buildrequires` section runs repeatedly.
To avoid accidentally reusing a wheel leaking from a previous (different) build,
it cannot be reused between `%generate_buildrequires` rounds.
Contrarily to that, rebuilding the wheel again in the `%build` section is redundant
and the packager can omit the `%build` section entirely
to reuse the wheel built from the last round of `%generate_buildrequires`.
Be extra careful when attempting to modify the sources after `%pyproject_buildrequires`,
e.g. when running extra commands in the `%build` section:

    %build
    cython src/wrong.pyx  # this is too late with %%pyproject_buildrequires -w
    %pyproject_wheel

For projects that specify test requirements using an [`extra`
provide](https://packaging.python.org/specifications/core-metadata/#provides-extra-multiple-use),
these can be added using the `-x` flag.
Multiple extras can be supplied by repeating the flag or as a comma separated list.
For example, if upstream suggests installing test dependencies with
`pip install mypackage[testing]`, the test deps would be generated by:

    %generate_buildrequires
    %pyproject_buildrequires -x testing

For projects that specify test requirements in their [tox] configuration,
these can be added using the `-t` flag (default tox environment)
or the `-e` flag followed by the tox environment.
The default tox environment (such as `py37` assuming the Fedora's Python version is 3.7)
is available in the `%{toxenv}` macro.
For example, if upstream suggests running the tests on Python 3.7 with `tox -e py37`,
the test deps would be generated by:

    %generate_buildrequires
    %pyproject_buildrequires -t

If upstream uses a custom derived environment, such as `py37-unit`, use:

    %pyproject_buildrequires -e %{toxenv}-unit

Or specify more environments if needed:

    %pyproject_buildrequires -e %{toxenv}-unit,%{toxenv}-integration

The `-e` option redefines `%{toxenv}` for further reuse.
Use `%{default_toxenv}` to get the default value.

The `-t`/`-e` option uses [tox-current-env]'s `--print-deps-to-file` behind the scenes.

If your package specifies some tox plugins in `tox.requires`,
such plugins will be BuildRequired as well.
Not all plugins are guaranteed to play well with [tox-current-env],
in worst case, patch/sed the requirement out from the tox configuration.

Note that neither `-x` or `-t` can be used with `-R`,
because runtime dependencies are always required for testing.
You can only use those options if the build backend  supports the [prepare-metadata-for-build-wheel hook],
or together with `-w`.

[tox]: https://tox.readthedocs.io/
[tox-current-env]: https://github.com/fedora-python/tox-current-env/
[prepare-metadata-for-build-wheel hook]: https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel

Additionally to generated requirements you can supply multiple file names to `%pyproject_buildrequires` macro.
Dependencies will be loaded from them:

    %pyproject_buildrequires requirements/tests.in requirements/docs.in requirements/dev.in

For packages not using build system you can use `-N` to entirely skip automatical
generation of requirements and install requirements only from manually specified files.
`-N` option implies `-R` and cannot be used in combination with other options mentioned above
(`-w`, `-e`, `-t`, `-x`).

The `%pyproject_buildrequires` macro also accepts the `-r` flag for backward compatibility;
it means "include runtime dependencies" which has been the default since version 0-53.


Running tox based tests
-----------------------

In case you want to run the tests as specified in [tox] configuration,
you must use `%pyproject_buildrequires` with `-t` or `-e` as explained above.
Then, use the `%tox` macro in `%check`:

    %check
    %tox

The macro:

 - Sets environment variables via `%{py3_test_envvars}`, namely:
     - Always prepends `$PATH` with `%{buildroot}%{_bindir}`
     - If not defined, sets `$PYTHONPATH` to `%{buildroot}%{python3_sitearch}:%{buildroot}%{python3_sitelib}`
 - If not defined, sets `$TOX_TESTENV_PASSENV` to `*`
 - Runs `tox` with `-q` (quiet), `--recreate` and `--current-env` (from [tox-current-env]) flags
 - Implicitly uses the tox environment name stored in `%{toxenv}` - as overridden by `%pyproject_buildrequires -e`

By using the `-e` flag, you can use a different tox environment(s):

    %check
    %tox
    %if %{with integration_test}
    %tox -e %{default_toxenv}-integration
    %endif

If you wish to provide custom `tox` flags or arguments, add them after `--`:

    %tox -- --flag-for-tox

If you wish to pass custom `posargs` to tox, use another `--`:

    %tox -- --flag-for-tox -- --flag-for-posargs

Or (note the two sequential `--`s):

    %tox -- -- --flag-for-posargs



Generating the %files section
-----------------------------

To generate the list of files in the `%files` section, you can use `%pyproject_save_files` after the `%pyproject_install` macro.
It takes toplevel module names (i.e. the names used with `import` in Python) and stores paths for those modules and metadata for the package (dist-info directory) to a file stored at `%{pyproject_files}`.
For example, if a package provides the modules `requests` and `_requests`, write:

    %install
    %pyproject_install
    %pyproject_save_files requests _requests

To add listed files to the `%files` section, use `%files -f %{pyproject_files}`.
Note that you still need to add any documentation manually (for now).

    %files -n python3-requests -f %{pyproject_files}
    %doc README.rst

You can use globs in the module names if listing them explicitly would be too tedious:

    %install
    %pyproject_install
    %pyproject_save_files '*requests'

In fully automated environments, you can use the `*` glob to include all modules (put it in single quotes to prevent Shell from expanding it). In Fedora however, you should always use a more specific glob to avoid accidentally packaging unwanted files (for example, a top level module named `test`).

Speaking about automated environments, some files cannot be classified with `%pyproject_save_files`, but it is possible to list all unclassified files by adding a special `+auto` argument.

    %install
    %pyproject_install
    %pyproject_save_files '*' +auto
    
    %files -n python3-requests -f %{pyproject_files}

However, in Fedora packages, always list executables explicitly to avoid unintended collisions with other packages or accidental missing executables:

    %install
    %pyproject_install
    %pyproject_save_files requests _requests
    
    %files -n python3-requests -f %{pyproject_files}
    %doc README.rst
    %{_bindir}/downloader

`%pyproject_save_files` can automatically mark license files with `%license` macro
and  language (`*.mo`) files with `%lang` macro and appropriate language code.
Only license files declared via [PEP 639] `License-File` field are detected.
[PEP 639] is still a draft and can be changed in the future.

Note that `%pyproject_save_files` uses data from the [RECORD file](https://www.python.org/dev/peps/pep-0627/).
If you wish to rename, remove or otherwise change the installed files of a package
*after* `%pyproject_install`, `%pyproject_save_files` might break.
If possible, remove/rename such files in `%prep`.
If not possible, avoid using `%pyproject_save_files` or edit/replace `%{pyproject_files}`.


Performing an import check on all importable modules
----------------------------------------------------

If the upstream test suite cannot be used during the package build
and you use `%pyproject_save_files`,
you can benefit from the `%pyproject_check_import` macro.
If `%pyproject_save_files` is not used, calling `%pyproject_check_import` will fail.

When `%pyproject_save_files` is invoked,
it creates a list of all valid and public (i.e. not starting with `_`)
importable module names found in the package.
Each top-level module name matches at least one of the globs provided as an argument to `%pyproject_save_files`.
This list is then usable by `%pyproject_check_import` which performs an import check for each listed module.
When a module fails to import, the build fails.

The modules are imported from both installed and buildroot's `%{python3_sitearch}`
and `%{python3_sitelib}`, not from the current directory.

Use the macro in `%check`:

    %check
    %pyproject_check_import

By using the `-e` flag, you can exclude module names matching the given glob(s) from the import check
(put it in single quotes to prevent Shell from expanding it).
The flag can be used repeatedly.
For example, to exclude all submodules ending with `config` and all submodules starting with `test`, you can use:

    %pyproject_check_import -e '*.config' -e '*.test*'

There must be at least one module left for the import check;
if, as a result of greedy excluding, no modules are left to check, the check fails.

When the `-t` flag is used, only top-level modules are checked,
qualified module names with a dot (`.`) are excluded.
If the modules detected by `%pyproject_save_files` are `requests`, `requests.models`, and `requests.packages`, this will only perform an import of `requests`:

    %pyproject_check_import -t

The modifying flags should only be used when there is a valid reason for not checking all available modules.
The reason should be documented in a comment.

The `%pyproject_check_import` macro also accepts positional arguments with
additional qualified module names to check, useful for example if some modules are installed manually.
Note that filtering by `-t`/`-e` also applies to the positional arguments.


Generating Extras subpackages
-----------------------------

The `%pyproject_extras_subpkg` macro generates simple subpackage(s)
for Python extras.

The macro should be placed after the base package's `%description` to avoid
issues in building the SRPM.

For example, if the `requests` project's metadata defines the extras
`security` and `socks`, the following invocation will generate the subpackage
`python3-requests+security` that provides `python3dist(requests[security])`,
and a similar one for `socks`.

    %pyproject_extras_subpkg -n python3-requests security socks

The macro works like `%python_extras_subpkg`,
except the `-i`/`-f`/`-F` arguments are optional and discouraged.
A filelist written by `%pyproject_install` is used by default.
For more information on `%python_extras_subpkg`, see the [Fedora change].

[Fedora change]: https://fedoraproject.org/wiki/Changes/PythonExtras

These arguments are still required:

* -n: name of the “base” package (e.g. python3-requests)
* Positional arguments: the extra name(s).
  Multiple subpackages are generated when multiple names are provided.


PROVISIONAL: Importing just-built (extension) modules in %build
---------------------------------------------------------------

Sometimes, it is desired to be able to import the *just-built* extension modules
in the `%build` section, e.g. to build the documentation with Sphinx.

    %build
    %pyproject_wheel
    ... build the docs here ...

With pure Python packages, it might be possible to set `PYTHONPATH=${PWD}` or `PYTHONPATH=${PWD}/src`.
However, it is a bit more complicated with extension modules.

The location of just-built modules might differ depending on Python version, architecture, pip version, etc.
Hence, the macro `%{pyproject_build_lib}` exists to be used like this:

    %build
    %pyproject_wheel
    PYTHONPATH=%{pyproject_build_lib} ... build the docs here ...

This macro is currently **provisional** and the behavior might change.
Please subscribe to Fedora's [python-devel list] if you use the macro.

The `%{pyproject_build_lib}` macro expands to an Shell `$(...)` expression and does not work when put into single quotes (`'`).

Depending on the pip version, the expanded value will differ:

[python-devel list]: https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/

### New pip 21.3+ with in-tree-build and setuptools 62.1+ (Fedora 37+)

Always use the macro from the same directory where you called `%pyproject_wheel` from.
The value will expand to something like:

* `/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-cpython-311` for wheels with extension modules
* `/builddir/build/BUILD/%{name}-%{version}/build/lib` for pure Python wheels

If multiple wheels were built from the same directory,
some pure Python and some with extension modules,
the expanded value will be combined with `:`:

* `/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-cypthon-311:/builddir/build/BUILD/%{name}-%{version}/build/lib`

If multiple wheels were built from different directories,
the value will differ depending on the current directory.


### New pip 21.3+ with in-tree-build and older setuptools (Fedora 36)

Always use the macro from the same directory where you called `%pyproject_wheel` from.
The value will expand to something like:

* `/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-3.10` for wheels with extension modules
* `/builddir/build/BUILD/%{name}-%{version}/build/lib` for pure Python wheels

If multiple wheels were built from the same directory,
some pure Python and some with extension modules,
the expanded value will be combined with `:`:

* `/builddir/build/BUILD/%{name}-%{version}/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/build/lib`

If multiple wheels were built from different directories,
the value will differ depending on the current directory.


### Older pip with out-of-tree-build (Fedora 35 and EL 9)

The value will expand to something like:

* `/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10` for wheels with extension modules
* `/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib` for pure Python wheels

Note that the exact value is **not stable** between builds
(the `xxxxxxxx` part is randomly generated,
neither you should consider the `.pyproject-builddir` directory to remain stable).

If multiple wheels are built,
the expanded value will always be combined with `:` regardless of the current directory, e.g.:

* `/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-xxxxxxxx/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-yyyyyyyy/build/lib.linux-x86_64-3.10:/builddir/build/BUILD/%{name}-%{version}/.pyproject-builddir/pip-req-build-zzzzzzzz/build/lib`

**Note:** If you manage to build some wheels with in-tree-build and some with out-of-tree-build option,
the expanded value will contain all relevant directories.


Limitations
-----------

`%pyproject_install` changes shebang lines of every Python script in `%{buildroot}%{_bindir}` to `#!%{__python3} %{py3_shbang_opt}` (`#!/usr/bin/python3 -s`).
Existing Python flags in shebangs are preserved.
For example `#!/usr/bin/python3 -Ru` will be updated to `#!/usr/bin/python3 -sRu`.
Sometimes, this can interfere with tests that run such scripts directly by name,
because in tests we usually rely on `PYTHONPATH` (and `-s` ignores that).
Would this behavior be undesired for any reason,
undefine `%{py3_shbang_opt}` to turn it off.

Some valid Python version specifiers are not supported.

When a dependency is specified via an URL or local path, for example as:

    https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6fc54099622be.zip
    /some/path/foo-1.2.3.tar.gz
    git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3

The `%pyproject_buildrequires` macro is unable to convert it to an appropriate RPM requirement and will fail.
If the URL contains the `packageName @` prefix as specified in [PEP 508],
the requirement will be generated without a version constraint:

    appdirs@https://github.com/ActiveState/appdirs/archive/8eacfa312d77aba28d483fbfb6f6fc54099622be.zip
    foo@file:///some/path/foo-1.2.3.tar.gz

Will be converted to:

    python3dist(appdirs)
    python3dist(foo)

Alternatively, when an URL requirement parsed from a text file
given as positional argument to `%pyproject_buildrequires`
contains the `#egg=packageName` fragment,
as documented in [pip's documentation]:

    git+https://github.com/sphinx-doc/sphinx.git@96dbe5e3#egg=sphinx

The requirements will be converted to package names without versions, e.g.:

    python3dist(sphinx)

However upstreams usually only use direct URLs for their requirements as workarounds,
so be prepared for problems.

[PEP 508]: https://www.python.org/dev/peps/pep-0508/
[PEP 517]: https://www.python.org/dev/peps/pep-0517/
[PEP 518]: https://www.python.org/dev/peps/pep-0518/
[PEP 639]: https://www.python.org/dev/peps/pep-0639/
[pip's documentation]: https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support


Testing the macros
------------------

This repository has two kinds of tests.
First, there is RPM `%check` section, run when building the `python-rpm-macros`
package.

Then there are CI tests.
There is currently [no way to run Fedora CI tests locally][ci-rfe],
but you can do what the tests do manually using mock.
For each `$PKG.spec` in `tests/`:

  - clean your mock environment:

        mock -r fedora-rawhide-x86_64 clean

  - install the version of `python-rpm-macros` you're testing, e.g.:

        mock -r fedora-rawhide-x86_64 install .../python-rpm-macros-*.noarch.rpm

  - download the sources:

        spectool -g -R $PKG.spec

  - build a SRPM:

        rpmbuild -bs $PKG.spec

  - build in mock, using the path from the command above as `$SRPM`:

        mock -r fedora-rawhide-x86_64 -n -N $SRPM

[ci-rfe]: https://pagure.io/fedora-ci/general/issue/4