Dependencies are recorded to a text file that is catted at the end.
This should prevent subtle bugs like https://bugzilla.redhat.com/2183519 in the future.
When the build backend prints to stdout via non-Python means,
for example when a setup.py script calls a verbose program via os.system(),
the output leaked to stdout of %pyproject_buildrequires was treated as generated BuildRequires.
Fore example, if the setup.py script has:
rv = os.system('/usr/bin/patch -N -p3 -d build/lib < lib/py-lmdb/env-copy-txn.patch')
(From https://github.com/jnwatson/py-lmdb/blob/py-lmdb_1.0.0/setup.py#L117)
The stdout of /usr/bin/patch leaked to stdout of %pyproject_buildrequires:
[lmdb-1.0.0]$ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --python3_pkgversion 3 2>/dev/null
python3dist(setuptools) >= 40.8
python3dist(wheel)
patching file lmdb.h
patching file mdb.c
python3dist(wheel)
patching file lmdb.h
patching file mdb.c
This resulted in DNF errors like this:
No matching package to install: 'lmdb.h'
No matching package to install: 'mdb.c'
No matching package to install: 'patching'
Moreover, it resulted in bogus BuildRequires that may have existed (e.g. "file").
By replacing the usage of contextlib.redirect_stdout
(which only redirects Python's sys.stdout)
with a custom context manager that captures stdout on file descriptor level
(in addition to Python's sys.stdout),
we avoid this leak.
File descriptor magic heavily inspired by the capfd pytest fixture.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2166888
Before:
usage: %pyproject_buildrequires [-w] [-R] [-e TOXENVS] [-t] [-x EXTRAS] [-N] [REQUIREMENTS.TXT ...]
After:
usage: %pyproject_buildrequires [-x EXTRAS] [-t] [-e TOXENVS] [-w] [-R] [-N] [REQUIREMENTS.TXT ...]
The order was determined as:
0. suppressed options
1. extras, the easiest way to specify test deps (x)
2. tox related options (te)
3. build wheel, as it is provisional (w)
4. "disablers" (RN)
5. varargs are always listed last
Previous order was pretty much random.
%pyproject_buildrequires macro now accepts multiple file names to load
additional dependencies from them.
New option -N was added to disable automatical generation of requirements
in case package does not use build system. Option -N cannot be used in
combination with options -r, -e, -t, -x.
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
The macro already checks if pyproject.toml exists and echoes the dependency
on python3dist(toml) early. This adds an else branch to echo the default backend.
For projects without pyproject.toml, the number of installation rounds
is reduced. Previously:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
Now:
1. (python3-devel +) pip + packaging + setuptools + wheel
2. ...
This duplicates the information about the default build backend,
because the script still needs to handle projects with pyproject.toml without
an explicit build backend option.
Hence, the script was not adapted (except a comment).
The macro checks if pyproject.toml exists and echoes the dependency early.
For projects with pyproject.toml, this saves one installation round.
Previously, the installation steps by %generate_buildrequires were:
1. (python3-devel +) pip + packaging
2. toml
3. parsed dependencies from pyproject.toml
4. ...
Now they are:
1. (python3-devel +) pip + packaging + toml
2. parsed dependencies from pyproject.toml
3. ...
For projects without pyproject.toml, the number of rounds remains the same:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
This is also more consistent:
The Python script now only outputs dependencies of the probed project,
it no longer partially outputs dependencies for itself.
The PEP 517 shows an example backend-path like this:
[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]
https://www.python.org/dev/peps/pep-0517/#source-trees
See that backend-path is a list. Our code previously only supported string path.
Obviously a string path is wrong, but we keep it to support projects that have
made the mistake, such as flit-core.
Add a small integration test for both cases.
Note that the new spec files deliberately don't do much, to save CI time.
An empty line happens when there are not deps:
Handling from tox --print-deps-only: <toxname>
WARNING: Skipping invalid requirement:
Parse error at "''": Expected W:(abcd...)
If not checked, installing runtime requirements might fail.
When a requirement is specified in setuptools' setup_requires:
setup(
...
setup_requires=["pytest-runner"],
)
It is part of the get_requires_for_build_wheel hook output.
When runtime requirements are parsed with setuptools without all setup_requires
present, it tries to get them from the internet (at least on Fedora 33).
By checking the requirements after installing "requires_for_build_wheel",
we make sure all setup_requires are already installed.
When runtime requirements are not installed, this adds an unneeded check,
but the script would end at that point anyway, so there is no real difference.
This change introduces code from pyreq2rpm, a tested set of
requirement conversion functions used in pyp2rpm and rpm's
pythondistdeps.
This adds support for the '~=' operator and wildcards.
Pros:
- projects without pyproject.toml will have 1 less dependency
- toml will be buildable with pyproject-rpm-macros out of the box
- easier bootstrap sequence (in theory)
Cons:
- projects with pyproject.toml will have 1 more %generate_buildrequires round
There is a slight problem when reporting that a dependency with extra is satisfied.
In fact, we only check the "base" dependency.
This can lead to a problem when a dependency is wrongly assumed as present
and the script proceeds to the "next stage" without restarting --
if the next stage tries to use (import) the missing dependency,
the script would crash.
However, that might be a very unlikely set of events and if such case ever happens,
we'll workaround it or fix it.
Falling back to setuptools.build_meta:__legacy__ is the standard behavior,
not setuptools.build_meta. See PEP 517:
https://www.python.org/dev/peps/pep-0517/
> If the pyproject.toml file is absent, or the build-backend key is missing,
> the source tree is not using this specification, and tools should revert
> to the legacy behaviour of running setup.py (either directly, or by
> implicitly invoking the setuptools.build_meta:__legacy__ backend).
Falling back to setuptools.build_meta had very similar results so far.,
but the behavior might change in the feature.
While working on this, I have uncovered a problem in our code.
It was not able to handle backends with ":". Looking at PEP 517 again:
> build-backend is a string naming a Python object that will be used to
> perform the build. This is formatted following the same module:object syntax
> as a setuptools entry point. For instance, if the string is "flit.api:main",
> this object would be looked up by executing the equivalent of:
>
> import flit.api
> backend = flit.api.main
>
> It's also legal to leave out the :object part, e.g.
>
> build-backend = "flit.api"
>
> which acts like:
>
> import flit.api
> backend = flit.api
We now handle such cases properly. Witch the change of the default backend,
we also test a backend with colon in our tests.
Previously, it wasn't possible to see why tox failed:
...
Requirement satisfied: tox-current-env >= 0.0.2
(installed: tox-current-env 0.0.2)
Traceback (most recent call last):
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 269, in main
generate_requires(
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 221, in generate_requires
generate_tox_requirements(toxenv, requirements)
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 184, in generate_tox_requirements
r = subprocess.run(
File "/usr/lib64/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmp96smu4rv', '-qre', 'py38']' returned non-zero exit status 1.
Now it is:
...
Requirement satisfied: tox-current-env >= 0.0.2
(installed: tox-current-env 0.0.2)
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Traceback (most recent call last):
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 270, in main
generate_requires(
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 222, in generate_requires
generate_tox_requirements(toxenv, requirements)
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 193, in generate_tox_requirements
r.check_returncode()
File "/usr/lib64/python3.8/subprocess.py", line 444, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmpwp8sffv1', '-qre', 'py38']' returned non-zero exit status 1.
Inspired by https://src.fedoraproject.org/rpms/python-chaospy/pull-request/1#comment-32750
