Keep the information about the requirement extras by storing the
Requirement instances in the list of the ignored requirements, rather
than the strings in the form they were initially read from metadata.
The requirements strings read from pyproject.toml don't contain the
extra information, we insert the extra marker only after converting them to
Requirement instances. When stored as the text, the information about
the extra went missing in the course of the script.
This adds a new flag, -p, to %pyproject_buildrequires.
When set, the runtime dependencies are read from the pyproject.toml's
[project] table.
See: https://bugzilla.redhat.com/2261939
pyproject_buildrequires.py already had a short `-p` option for
--python3_pkgversion (hidden from the macro users).
This change removes the one-letter option and leaves the long-one.
`-p` is now reused for reading dependencies from pyproject.toml
and made visible to the macro users.
Dependencies are recorded to a text file that is catted at the end.
This should prevent subtle bugs like https://bugzilla.redhat.com/2183519 in the future.
When the build backend prints to stdout via non-Python means,
for example when a setup.py script calls a verbose program via os.system(),
the output leaked to stdout of %pyproject_buildrequires was treated as generated BuildRequires.
Fore example, if the setup.py script has:
rv = os.system('/usr/bin/patch -N -p3 -d build/lib < lib/py-lmdb/env-copy-txn.patch')
(From https://github.com/jnwatson/py-lmdb/blob/py-lmdb_1.0.0/setup.py#L117)
The stdout of /usr/bin/patch leaked to stdout of %pyproject_buildrequires:
[lmdb-1.0.0]$ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --python3_pkgversion 3 2>/dev/null
python3dist(setuptools) >= 40.8
python3dist(wheel)
patching file lmdb.h
patching file mdb.c
python3dist(wheel)
patching file lmdb.h
patching file mdb.c
This resulted in DNF errors like this:
No matching package to install: 'lmdb.h'
No matching package to install: 'mdb.c'
No matching package to install: 'patching'
Moreover, it resulted in bogus BuildRequires that may have existed (e.g. "file").
By replacing the usage of contextlib.redirect_stdout
(which only redirects Python's sys.stdout)
with a custom context manager that captures stdout on file descriptor level
(in addition to Python's sys.stdout),
we avoid this leak.
File descriptor magic heavily inspired by the capfd pytest fixture.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2166888
Before:
usage: %pyproject_buildrequires [-w] [-R] [-e TOXENVS] [-t] [-x EXTRAS] [-N] [REQUIREMENTS.TXT ...]
After:
usage: %pyproject_buildrequires [-x EXTRAS] [-t] [-e TOXENVS] [-w] [-R] [-N] [REQUIREMENTS.TXT ...]
The order was determined as:
0. suppressed options
1. extras, the easiest way to specify test deps (x)
2. tox related options (te)
3. build wheel, as it is provisional (w)
4. "disablers" (RN)
5. varargs are always listed last
Previous order was pretty much random.
%pyproject_buildrequires macro now accepts multiple file names to load
additional dependencies from them.
New option -N was added to disable automatical generation of requirements
in case package does not use build system. Option -N cannot be used in
combination with options -r, -e, -t, -x.
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
The macro already checks if pyproject.toml exists and echoes the dependency
on python3dist(toml) early. This adds an else branch to echo the default backend.
For projects without pyproject.toml, the number of installation rounds
is reduced. Previously:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
Now:
1. (python3-devel +) pip + packaging + setuptools + wheel
2. ...
This duplicates the information about the default build backend,
because the script still needs to handle projects with pyproject.toml without
an explicit build backend option.
Hence, the script was not adapted (except a comment).
The macro checks if pyproject.toml exists and echoes the dependency early.
For projects with pyproject.toml, this saves one installation round.
Previously, the installation steps by %generate_buildrequires were:
1. (python3-devel +) pip + packaging
2. toml
3. parsed dependencies from pyproject.toml
4. ...
Now they are:
1. (python3-devel +) pip + packaging + toml
2. parsed dependencies from pyproject.toml
3. ...
For projects without pyproject.toml, the number of rounds remains the same:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
This is also more consistent:
The Python script now only outputs dependencies of the probed project,
it no longer partially outputs dependencies for itself.
The PEP 517 shows an example backend-path like this:
[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]
https://www.python.org/dev/peps/pep-0517/#source-trees
See that backend-path is a list. Our code previously only supported string path.
Obviously a string path is wrong, but we keep it to support projects that have
made the mistake, such as flit-core.
Add a small integration test for both cases.
Note that the new spec files deliberately don't do much, to save CI time.
An empty line happens when there are not deps:
Handling from tox --print-deps-only: <toxname>
WARNING: Skipping invalid requirement:
Parse error at "''": Expected W:(abcd...)
If not checked, installing runtime requirements might fail.
When a requirement is specified in setuptools' setup_requires:
setup(
...
setup_requires=["pytest-runner"],
)
It is part of the get_requires_for_build_wheel hook output.
When runtime requirements are parsed with setuptools without all setup_requires
present, it tries to get them from the internet (at least on Fedora 33).
By checking the requirements after installing "requires_for_build_wheel",
we make sure all setup_requires are already installed.
When runtime requirements are not installed, this adds an unneeded check,
but the script would end at that point anyway, so there is no real difference.
This change introduces code from pyreq2rpm, a tested set of
requirement conversion functions used in pyp2rpm and rpm's
pythondistdeps.
This adds support for the '~=' operator and wildcards.
Pros:
- projects without pyproject.toml will have 1 less dependency
- toml will be buildable with pyproject-rpm-macros out of the box
- easier bootstrap sequence (in theory)
Cons:
- projects with pyproject.toml will have 1 more %generate_buildrequires round
There is a slight problem when reporting that a dependency with extra is satisfied.
In fact, we only check the "base" dependency.
This can lead to a problem when a dependency is wrongly assumed as present
and the script proceeds to the "next stage" without restarting --
if the next stage tries to use (import) the missing dependency,
the script would crash.
However, that might be a very unlikely set of events and if such case ever happens,
we'll workaround it or fix it.
Falling back to setuptools.build_meta:__legacy__ is the standard behavior,
not setuptools.build_meta. See PEP 517:
https://www.python.org/dev/peps/pep-0517/
> If the pyproject.toml file is absent, or the build-backend key is missing,
> the source tree is not using this specification, and tools should revert
> to the legacy behaviour of running setup.py (either directly, or by
> implicitly invoking the setuptools.build_meta:__legacy__ backend).
Falling back to setuptools.build_meta had very similar results so far.,
but the behavior might change in the feature.
While working on this, I have uncovered a problem in our code.
It was not able to handle backends with ":". Looking at PEP 517 again:
> build-backend is a string naming a Python object that will be used to
> perform the build. This is formatted following the same module:object syntax
> as a setuptools entry point. For instance, if the string is "flit.api:main",
> this object would be looked up by executing the equivalent of:
>
> import flit.api
> backend = flit.api.main
>
> It's also legal to leave out the :object part, e.g.
>
> build-backend = "flit.api"
>
> which acts like:
>
> import flit.api
> backend = flit.api
We now handle such cases properly. Witch the change of the default backend,
we also test a backend with colon in our tests.
