Commit Graph

60 Commits

Author SHA1 Message Date
Miro Hrončok
bd7890110c %pyproject_buildrequires: Add support for self-referential extras requirements 2023-04-27 11:18:18 +02:00
Miro Hrončok
456903666c Redirect stdout to stderr via Shell
Dependencies are recorded to a text file that is catted at the end.

This should prevent subtle bugs like https://bugzilla.redhat.com/2183519 in the future.
2023-04-17 15:32:26 +02:00
Miro Hrončok
f8e160d767 Remove duplicate import 2023-03-31 18:59:06 +02:00
Miro Hrončok
85fc41174d %pyproject_buildrequires: Avoid leaking stdout from subprocesses
When the build backend prints to stdout via non-Python means,
for example when a setup.py script calls a verbose program via os.system(),
the output leaked to stdout of %pyproject_buildrequires was treated as generated BuildRequires.

Fore example, if the setup.py script has:

    rv = os.system('/usr/bin/patch -N -p3 -d build/lib < lib/py-lmdb/env-copy-txn.patch')

(From https://github.com/jnwatson/py-lmdb/blob/py-lmdb_1.0.0/setup.py#L117)

The stdout of /usr/bin/patch leaked to stdout of %pyproject_buildrequires:

    [lmdb-1.0.0]$ /usr/bin/python3 -Bs /usr/lib/rpm/redhat/pyproject_buildrequires.py --python3_pkgversion 3 2>/dev/null
    python3dist(setuptools) >= 40.8
    python3dist(wheel)
    patching file lmdb.h
    patching file mdb.c
    python3dist(wheel)
    patching file lmdb.h
    patching file mdb.c

This resulted in DNF errors like this:

    No matching package to install: 'lmdb.h'
    No matching package to install: 'mdb.c'
    No matching package to install: 'patching'

Moreover, it resulted in bogus BuildRequires that may have existed (e.g. "file").

By replacing the usage of contextlib.redirect_stdout
(which only redirects Python's sys.stdout)
with a custom context manager that captures stdout on file descriptor level
(in addition to Python's sys.stdout),
we avoid this leak.

File descriptor magic heavily inspired by the capfd pytest fixture.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2166888
2023-02-06 21:17:42 +01:00
Miro Hrončok
8b3af75e2f Reorder %pyproject_buildrequires arguments for a nicer usage string
Before:

    usage: %pyproject_buildrequires [-w] [-R] [-e TOXENVS] [-t] [-x EXTRAS] [-N] [REQUIREMENTS.TXT ...]

After:

    usage: %pyproject_buildrequires [-x EXTRAS] [-t] [-e TOXENVS] [-w] [-R] [-N] [REQUIREMENTS.TXT ...]

The order was determined as:

    0. suppressed options
    1. extras, the easiest way to specify test deps (x)
    2. tox related options (te)
    3. build wheel, as it is provisional (w)
    4. "disablers" (RN)
    5. varargs are always listed last

Previous order was pretty much random.
2022-10-12 14:47:59 +02:00
Miro Hrončok
222ec5f4f5 Make %pyproject_buildrequires and %pyproject_save_files argparser errors nicer to macro users
Before:

    %pyproject_buildrequires bogus_arg
    usage: pyproject_buildrequires.py [-h] [-r] [-w] [--wheeldir PATH] [-R]
                                      [-e TOXENVS] [-t] [-x EXTRAS]
                                      [--generate-extras] [-p PYTHON3_PKGVERSION]
                                      [-N]
                                      [requirement_files ...]
    pyproject_buildrequires.py: error: argument requirement_files: can't open 'bogus_arg': ...

    %pyproject_save_files
    ...
    usage: pyproject_save_files.py [-h] --output-files OUTPUT_FILES
                                   --output-modules OUTPUT_MODULES --buildroot
                                   BUILDROOT --sitelib SITELIB --sitearch SITEARCH
                                   --python-version PYTHON_VERSION
                                   --pyproject-record PYPROJECT_RECORD --prefix
                                   PREFIX
                                   varargs [varargs ...]
    pyproject_save_files.py: error: the following arguments are required: varargs

After:

    %pyproject_buildrequires bogus_arg
    usage: %pyproject_buildrequires [-w] [-R] [-e TOXENVS] [-t] [-x EXTRAS] [-N]
                                    [REQUIREMENTS.TXT ...]
    %pyproject_buildrequires: error: argument REQUIREMENTS.TXT: can't open 'bogus_arg': ...

    %pyproject_save_files
    ...
    usage: %pyproject_save_files MODULE_GLOB [MODULE_GLOB ...] [+auto]
    %pyproject_save_files: error: the following arguments are required: MODULE_GLOB
2022-10-12 14:47:54 +02:00
Otto Liljalaakso
dd0f198400 Fix typo in function name 'evaluate_all_environamnets'
The function is only used internally in these macros,
so name change should not break any users.
2022-08-30 23:44:09 +03:00
Miro Hrončok
07577de8ad Use tomllib from the standard library on Python 3.11+ 2022-06-02 11:34:14 +02:00
Miro Hrončok
40f6765e0e Allow building wheels in %pyproject_buildrequires to support other build backends
The hook is optional, see https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel

> If a build frontend needs this information and the method is not defined,
> it should call build_wheel and look at the resulting metadata directly.

This is not yet automatically detected because the feature is provisional.
Use `%pyproject_buildrequires -w` to opt-in.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2076994
2022-04-29 12:42:37 +02:00
Lumir Balhar
3deb3f4147 Updated compatibility with tox4 2022-02-22 23:50:22 +00:00
Miro Hrončok
8c8afba774 %pyproject_buildrequires: Make -r (include runtime) the default, use -R to opt-out
See the proposal:
https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/2R6NKELTHAWE6PI3CCZBVW5PMGO5VPDG/

 - -N now implies -R
 - the macro still guards against -Nr and now also against -Rr
2022-01-18 17:50:00 +01:00
Miro Hrončok
27e23c1e87 %pyproject_buildrequires: Accept installed pre-releases for all requirements
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2014639
2021-10-25 15:02:22 +00:00
Benjamin A. Beasley
f8e0c247de Change “requirement file” to match “requirements file[s]” elsewhere 2021-07-25 17:37:01 -04:00
Benjamin A. Beasley
6ee7a6baea Fix two misspellings of “requirements,” one user-visible 2021-07-25 17:35:52 -04:00
Petr Viktorin
aeb21f671f Split requirements.txt parsing to its own module; test & improve it 2021-07-23 10:00:01 +02:00
Miro Hrončok
f8a3343abc %pyproject_buildrequires now fails when it encounters an invalid requirement
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1983053
2021-07-23 10:00:01 +02:00
Tomas Hrnciar
5b1caad68e %pyproject_buildrequires: Fallback to setuptools.build_meta:__legacy__ only if setup.py exists
Fixes: rhbz#1976459
2021-07-17 00:36:57 +02:00
Miro Hrončok
ddaf2e9fb2 %pyproject_buildrequires: Support x.* versions
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1981558
2021-07-13 12:23:35 +02:00
Tomas Hrnciar
d6ad9a778a Generate BuildRequires from file
%pyproject_buildrequires macro now accepts multiple file names to load
additional dependencies from them.

New option -N was added to disable automatical generation of requirements
in case package does not use build system. Option -N cannot be used in
combination with options -r, -e, -t, -x.

Co-authored-by: Miro Hrončok <miro@hroncok.cz>
2021-07-08 13:08:04 +02:00
Miro Hrončok
2abcad96dd Don't accidentally treat "~= X.0" requirement as "~= X"
Don't canonicalize the version twice.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1977060
2021-06-29 12:43:13 +02:00
Miro Hrončok
346c9bc80a Fix a typo 2021-06-23 12:26:28 +02:00
Miro Hrončok
99ed4639da Generate BuildRequires on extras in lower case 2021-04-07 16:25:06 +02:00
Miro Hrončok
7e1a8fd079 Handle tox provision (tox.requires / tox.minversion) 2021-04-07 16:01:33 +02:00
Miro Hrončok
ff396611dd Generate python3dist(setuptools/wheel) BuildRequires directly from the macro
The macro already checks if pyproject.toml exists and echoes the dependency
on python3dist(toml) early. This adds an else branch to echo the default backend.

For projects without pyproject.toml, the number of installation rounds
is reduced. Previously:

 1. (python3-devel +) pip + packaging
 2. setuptools + wheel
 3. ...

Now:

 1. (python3-devel +) pip + packaging + setuptools + wheel
 2. ...

This duplicates the information about the default build backend,
because the script still needs to handle projects with pyproject.toml without
an explicit build backend option.
Hence, the script was not adapted (except a comment).
2021-02-03 12:00:58 +01:00
Miro Hrončok
06b21e1976 Generate python3dist(toml) BuildRequires directly from the macro
The macro checks if pyproject.toml exists and echoes the dependency early.

For projects with pyproject.toml, this saves one installation round.
Previously, the installation steps by %generate_buildrequires were:

 1. (python3-devel +) pip + packaging
 2. toml
 3. parsed dependencies from pyproject.toml
 4. ...

Now they are:

 1. (python3-devel +) pip + packaging + toml
 2. parsed dependencies from pyproject.toml
 3. ...

For projects without pyproject.toml, the number of rounds remains the same:

 1. (python3-devel +) pip + packaging
 2. setuptools + wheel
 3. ...

This is also more consistent:
The Python script now only outputs dependencies of the probed project,
it no longer partially outputs dependencies for itself.
2021-02-03 12:00:58 +01:00
Miro Hrončok
390b9713aa Remove support for Python 3.7 from %pyproject_buildrequires
Fedora 31 is EOL and was the last one with Python 3.7 packages.
EL 8 has Python 3.6 but doesn'T have %generate_buildrequires.
2021-02-03 12:00:58 +01:00
Miro Hrončok
38ef5fb85b Allow multiple -e in %pyproject_buildrequires 2020-11-04 18:35:22 +01:00
Miro Hrončok
290941c5f3 Support PEP 517 list based backend-path
The PEP 517 shows an example backend-path like this:

    [build-system]
    # Defined by PEP 518:
    requires = ["flit"]
    # Defined by this PEP:
    build-backend = "local_backend"
    backend-path = ["backend"]

https://www.python.org/dev/peps/pep-0517/#source-trees

See that backend-path is a list. Our code previously only supported string path.

Obviously a string path is wrong, but we keep it to support projects that have
made the mistake, such as flit-core.

Add a small integration test for both cases.
Note that the new spec files deliberately don't do much, to save CI time.
2020-10-07 08:35:45 +02:00
Miro Hrončok
6f6a7636d4 Skip empty lines from tox deps
An empty line happens when there are not deps:

    Handling  from tox --print-deps-only: <toxname>
    WARNING: Skipping invalid requirement:
        Parse error at "''": Expected W:(abcd...)
2020-10-01 16:35:08 +02:00
Miro Hrončok
2ecbed7441 Support multiple -x options for %pyproject_buildrequires
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1877978
2020-10-01 16:35:07 +02:00
Miro Hrončok
9f3eea2ae5 Support the extras configuration option of tox in %pyproject_buildrequires -t
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1877977
2020-10-01 16:33:58 +02:00
Miro Hrončok
3b95c7d66c Check the requirements after installing "requires_for_build_wheel"
If not checked, installing runtime requirements might fail.

When a requirement is specified in setuptools' setup_requires:

    setup(
        ...
        setup_requires=["pytest-runner"],
    )

It is part of the get_requires_for_build_wheel hook output.

When runtime requirements are parsed with setuptools without all setup_requires
present, it tries to get them from the internet (at least on Fedora 33).

By checking the requirements after installing "requires_for_build_wheel",
we make sure all setup_requires are already installed.

When runtime requirements are not installed, this adds an unneeded check,
but the script would end at that point anyway, so there is no real difference.
2020-09-23 11:16:41 +02:00
Gordon Messmer
cb38f210d2 Support more Python version specifiers in generated BuildRequires
This change introduces code from pyreq2rpm, a tested set of
requirement conversion functions used in pyp2rpm and rpm's
pythondistdeps.

This adds support for the '~=' operator and wildcards.
2020-09-12 01:09:48 +02:00
Miro Hrončok
b4fd1c2e74 Only require toml for projects with pyproject.toml
Pros:

 - projects without pyproject.toml will have 1 less dependency
 - toml will be buildable with pyproject-rpm-macros out of the box
 - easier bootstrap sequence (in theory)

Cons:

 - projects with pyproject.toml will have 1 more %generate_buildrequires round
2020-09-07 09:48:27 +02:00
Miro Hrončok
a613e176e3 Handle Python Extras in %pyproject_buildrequires on Fedora 33+
There is a slight problem when reporting that a dependency with extra is satisfied.
In fact, we only check the "base" dependency.
This can lead to a problem when a dependency is wrongly assumed as present
and the script proceeds to the "next stage" without restarting --
if the next stage tries to use (import) the missing dependency,
the script would crash.

However, that might be a very unlikely set of events and if such case ever happens,
we'll workaround it or fix it.
2020-08-20 15:30:49 +02:00
Miro Hrončok
6a8d86ed70 Allow multiple, comma-separated extras in %pyproject_buildrequires -x 2020-08-11 15:54:42 +02:00
Lumir Balhar
cb8e334272 Use python -m tox instead of just tox
This way, macros can use whatever Python has tox module available
instead of only the main one which owns /usr/bin/tox.
2020-08-10 12:04:22 +02:00
Lumir Balhar
5561755a00 Make %pyproject_buildrequires more universal
so it can work with any non-main Python version and generate
proper dependencies.
2020-08-10 12:04:22 +02:00
Miro Hrončok
262f6d3bc3 %pyproject_buildrequires -x now implies -r
The usage without -r errored anyway, this way instead of forcing the user to add it,
we do it ourselves.

Machines stealing human's labor, yet again.
2020-07-16 13:38:50 +02:00
Miro Hrončok
ed5dd772f3 Switch from upstream deprecated pytoml to toml 2020-06-23 11:09:03 +00:00
Miro Hrončok
99d952cd6c Tox dependency generator: Handle deps read in from a text file
Resolves https://bugzilla.redhat.com/show_bug.cgi?id=1808601

tox docs: https://tox.readthedocs.io/en/latest/example/basic.html#depending-on-requirements-txt-or-defining-constraints

Relevant tox-current-env issue: https://github.com/fedora-python/tox-current-env/issues/22
2020-03-05 13:44:54 +01:00
Miro Hrončok
6210f94e46 Handle backends with colon, fallback to setuptools.build_meta:__legacy__
Falling back to setuptools.build_meta:__legacy__ is the standard behavior,
not setuptools.build_meta. See PEP 517:

https://www.python.org/dev/peps/pep-0517/

> If the pyproject.toml file is absent, or the build-backend key is missing,
> the source tree is not using this specification, and tools should revert
> to the legacy behaviour of running setup.py (either directly, or by
> implicitly invoking the setuptools.build_meta:__legacy__ backend).

Falling back to setuptools.build_meta had very similar results so far.,
but the behavior might change in the feature.

While working on this, I have uncovered a problem in our code.
It was not able to handle backends with ":". Looking at PEP 517 again:

> build-backend is a string naming a Python object that will be used to
> perform the build. This is formatted following the same module:object syntax
> as a setuptools entry point. For instance, if the string is "flit.api:main",
> this object would be looked up by executing the equivalent of:
>
>    import flit.api
>    backend = flit.api.main
>
> It's also legal to leave out the :object part, e.g.
>
>    build-backend = "flit.api"
>
> which acts like:
>
>    import flit.api
>    backend = flit.api

We now handle such cases properly. Witch the change of the default backend,
we also test a backend with colon in our tests.
2020-02-05 13:31:45 +01:00
Miro Hrončok
50464a4b19 Satisfy the flake8 linter, unify quotation marks 2019-10-25 17:07:29 +02:00
Miro Hrončok
d5c3fb3c5a When tox fails, print tox output before failing
Previously, it wasn't possible to see why tox failed:

...
Requirement satisfied: tox-current-env >= 0.0.2
   (installed: tox-current-env 0.0.2)
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 269, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 221, in generate_requires
    generate_tox_requirements(toxenv, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 184, in generate_tox_requirements
    r = subprocess.run(
  File "/usr/lib64/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmp96smu4rv', '-qre', 'py38']' returned non-zero exit status 1.

Now it is:

...
Requirement satisfied: tox-current-env >= 0.0.2
   (installed: tox-current-env 0.0.2)
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Traceback (most recent call last):
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 270, in main
    generate_requires(
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 222, in generate_requires
    generate_tox_requirements(toxenv, requirements)
  File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 193, in generate_tox_requirements
    r.check_returncode()
  File "/usr/lib64/python3.8/subprocess.py", line 444, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmpwp8sffv1', '-qre', 'py38']' returned non-zero exit status 1.

Inspired by https://src.fedoraproject.org/rpms/python-chaospy/pull-request/1#comment-32750
2019-10-25 16:57:01 +02:00
Petr Viktorin
d262d909f5 Use importlib_metadata rather than pip freeze 2019-09-18 16:16:17 +02:00
Miro Hrončok
fda0a23075 -t means "use tox", -e means "use this toxenv", -e implies -t
Further fixes
2019-08-13 14:42:21 +02:00
Miro Hrončok
c975fbe6ec --toxenv implies --runtime 2019-08-13 14:05:50 +02:00
Miro Hrončok
7ba1a33967 Use tox with --print-deps-to-file instead of parsing stdout 2019-08-12 17:29:26 +02:00
Miro Hrončok
8a60635881 Allow to fetch test dependencies from tox 2019-07-26 15:07:11 +02:00
Petr Viktorin
aca2f6a0c4 Mark multiple extras as not supported yet 2019-07-18 11:00:23 +02:00