Users invoking %pyproject_save_files with glob: '*' don't care about the
files in the Python package, hence it shouldn't error when no modules
are detected.
There may be legitimate reasons to create a package without Python
modules in it, hence we shouldn't be blocking this possibility.
When extension modules are built in %pyproject_buildrequires,
we need to create the package note file.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=2097535
This is tested via python-ldap -- %pyproject_buildrequires -w fails without the fix.
Neither python-markupsafe nor python-mistune can be used as a test
because they only warn when the extension module cannot be built
because they fallback to pure Python.
Apparently, when you repeatedly run `rpmbuild -ba`, files in %_builddir are not cleaned.
This way, we at least make sure the files are unique between different NVRAs,
so 2 unrelated builds don't share the files between each other.
Keeping files contained in the build subdirectory is the more common way of doing this,
but we cannot technically do that, because we don't know what's it gonna be (before %prep).
Should be backwards compatible, as we only modify underscore-prefixed macros and %{pyproject_files},
where the exact value should not matter to the packagers.
This package is already installed -- otherwise the macro would not even exist.
However, since python3-devel has started to Require pyproject-rpm-macros,
it is no longer possible to use `repoquery --whatrequires pyproject-rpm-macros`
to get a reliable list of packages that use the macros.
This was, all packages that use %pyproject_buildrequires will BuildRequire the macros explicitly.
(In the future, we could even include a stub version of %pyproject_buildrequires
in pyproject-srpm-macros (always installed in the buildroot),
that only echoes this package,
so packagers would not need to manually BuildRequire anything at all.)
Compressed manpages have different extension than those listed in the RECORD file,
so they were not recognized when %%pyproject_save_files '+auto' flag
was provided.
To enable the path recognition, if the manpage extension matches the one
listed in brp-compres, the extension is removed, and an asterisk is now added
to the manpages filenames.
Source: https://docs.fedoraproject.org/en-US/packaging-guidelines/#_manpages
Fixes: https://bugzilla.redhat.com/2033254
Improve handling of > operator, preventing post-release from satisfying most rpm requirements.
Improve handling of < operator, preventing pre-release from satisfying rpm requirement.
Improve handling of != operator with prefix matching, preventing pre-release from satisfying rpm requirements.
%%pyproject_save_files newly saves also a list of importable modules.
The list is used by %%pyproject_check_import to invoke the import test
on each module name.
%%pyproject_check_import accepts two options:
-t: filter only top-level modules
-e: exclude module names matching the given glob from the import check
%pyproject_buildrequires macro now accepts multiple file names to load
additional dependencies from them.
New option -N was added to disable automatical generation of requirements
in case package does not use build system. Option -N cannot be used in
combination with options -r, -e, -t, -x.
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
This allows users to do:
%build
cd somewhere
%pyproject_wheel
cd -
cd somewhere_else
%pyproject_wheel
cd -
%install
%pyproject_install
Without a need to copy paste the wheels to a common location.
This is in fact a breaking change, I'll make sure to adapt the affected packages in Fedora.
The macro already checks if pyproject.toml exists and echoes the dependency
on python3dist(toml) early. This adds an else branch to echo the default backend.
For projects without pyproject.toml, the number of installation rounds
is reduced. Previously:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
Now:
1. (python3-devel +) pip + packaging + setuptools + wheel
2. ...
This duplicates the information about the default build backend,
because the script still needs to handle projects with pyproject.toml without
an explicit build backend option.
Hence, the script was not adapted (except a comment).
The macro checks if pyproject.toml exists and echoes the dependency early.
For projects with pyproject.toml, this saves one installation round.
Previously, the installation steps by %generate_buildrequires were:
1. (python3-devel +) pip + packaging
2. toml
3. parsed dependencies from pyproject.toml
4. ...
Now they are:
1. (python3-devel +) pip + packaging + toml
2. parsed dependencies from pyproject.toml
3. ...
For projects without pyproject.toml, the number of rounds remains the same:
1. (python3-devel +) pip + packaging
2. setuptools + wheel
3. ...
This is also more consistent:
The Python script now only outputs dependencies of the probed project,
it no longer partially outputs dependencies for itself.
This is done to avoid troubles with %lang files listed as duplicated.
1. It gets rid of a warning
2. It fixes a problem described in:
http://lists.rpm.org/pipermail/rpm-list/2020-November/002041.html
This is a backwards incompatible change,
packages that rename or remove the installed files after %pyproject_install
might no longer be compatible with %pyproject_save_files.
The PEP 517 shows an example backend-path like this:
[build-system]
# Defined by PEP 518:
requires = ["flit"]
# Defined by this PEP:
build-backend = "local_backend"
backend-path = ["backend"]
https://www.python.org/dev/peps/pep-0517/#source-trees
See that backend-path is a list. Our code previously only supported string path.
Obviously a string path is wrong, but we keep it to support projects that have
made the mistake, such as flit-core.
Add a small integration test for both cases.
Note that the new spec files deliberately don't do much, to save CI time.
According to PEP 627, the RECORD file is optional and
doesn't make sense to keep it for system packages. Moreover,
its absence should indicate to other tools like pip that
they should not touch such packages.
Now, we process content of all RECORD files to one
pyproject-record (JSON) which is then used in
%pyproject_save_files. That way, we can remove the original
files in %pyproject_install and keep their content for
later.
PEP 627: https://www.python.org/dev/peps/pep-0627/#optional-record-file
If not checked, installing runtime requirements might fail.
When a requirement is specified in setuptools' setup_requires:
setup(
...
setup_requires=["pytest-runner"],
)
It is part of the get_requires_for_build_wheel hook output.
When runtime requirements are parsed with setuptools without all setup_requires
present, it tries to get them from the internet (at least on Fedora 33).
By checking the requirements after installing "requires_for_build_wheel",
we make sure all setup_requires are already installed.
When runtime requirements are not installed, this adds an unneeded check,
but the script would end at that point anyway, so there is no real difference.
This change introduces code from pyreq2rpm, a tested set of
requirement conversion functions used in pyp2rpm and rpm's
pythondistdeps.
This adds support for the '~=' operator and wildcards.
Pros:
- projects without pyproject.toml will have 1 less dependency
- toml will be buildable with pyproject-rpm-macros out of the box
- easier bootstrap sequence (in theory)
Cons:
- projects with pyproject.toml will have 1 more %generate_buildrequires round
In %pyproject_buildrequires, don't run python with -I but -s.
This allows projects used by the script itself, such as packaging or toml,
to be packaged using the macros, using "self" -- i.e. the code from $PWD.
There is a slight problem when reporting that a dependency with extra is satisfied.
In fact, we only check the "base" dependency.
This can lead to a problem when a dependency is wrongly assumed as present
and the script proceeds to the "next stage" without restarting --
if the next stage tries to use (import) the missing dependency,
the script would crash.
However, that might be a very unlikely set of events and if such case ever happens,
we'll workaround it or fix it.
Tox calls socket.getfqdn() and that call does a DNS query.
In mock with disabled networking, it takes a minute until that times out.
When a spec file uses %pyproject_buildrequires -t and %tox, it is a 3 minute delay.
Since 3.17, tox does not call socket.getfqdn() when HOSTNAME variable is set to a value:
https://github.com/tox-dev/tox/pull/1616
The value is only used in result log, so setting it to "rpmbuild"
actually makes the logs more reproducible as well.
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1856356
when tox is used in %pyproject_buildrequires -t or %tox.
With changes in PEP 610 there is new file direct_url.json created, since it is not useful
for us we prevent it's creation. This commit changes %pyproject_install macro to install wheel using
name instead of path.
This commit also includes new test to check if file direct_url.json wasn't created.
https://discuss.python.org/t/pep-610-usage-guidelines-for-linux-distributions/4012
This macro save generates file section to %pyproject_files. It should
simplify %files section and allow to build by some automatic machinery
Supposed use case in Fedora:
%install
%pyproject_install
%pyproject_save_files requests _requests
%files -n python3-requests -f %{pyproject_files}
%doc README.rst
%license LICENSE
Automatic build of arbitrary packages (e.g. in Copr):
%install
%pyproject_install
%pyproject_save_files * +bindir // save all modules with executables
%files -n python3-requests -f %{pyproject_files}
Co-Authored-By: Miro Hrončok <miro@hroncok.cz>
Falling back to setuptools.build_meta:__legacy__ is the standard behavior,
not setuptools.build_meta. See PEP 517:
https://www.python.org/dev/peps/pep-0517/
> If the pyproject.toml file is absent, or the build-backend key is missing,
> the source tree is not using this specification, and tools should revert
> to the legacy behaviour of running setup.py (either directly, or by
> implicitly invoking the setuptools.build_meta:__legacy__ backend).
Falling back to setuptools.build_meta had very similar results so far.,
but the behavior might change in the feature.
While working on this, I have uncovered a problem in our code.
It was not able to handle backends with ":". Looking at PEP 517 again:
> build-backend is a string naming a Python object that will be used to
> perform the build. This is formatted following the same module:object syntax
> as a setuptools entry point. For instance, if the string is "flit.api:main",
> this object would be looked up by executing the equivalent of:
>
> import flit.api
> backend = flit.api.main
>
> It's also legal to leave out the :object part, e.g.
>
> build-backend = "flit.api"
>
> which acts like:
>
> import flit.api
> backend = flit.api
We now handle such cases properly. Witch the change of the default backend,
we also test a backend with colon in our tests.
Previously, it wasn't possible to see why tox failed:
...
Requirement satisfied: tox-current-env >= 0.0.2
(installed: tox-current-env 0.0.2)
Traceback (most recent call last):
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 269, in main
generate_requires(
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 221, in generate_requires
generate_tox_requirements(toxenv, requirements)
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 184, in generate_tox_requirements
r = subprocess.run(
File "/usr/lib64/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmp96smu4rv', '-qre', 'py38']' returned non-zero exit status 1.
Now it is:
...
Requirement satisfied: tox-current-env >= 0.0.2
(installed: tox-current-env 0.0.2)
ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found
Traceback (most recent call last):
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 270, in main
generate_requires(
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 222, in generate_requires
generate_tox_requirements(toxenv, requirements)
File "/usr/lib/rpm/redhat/pyproject_buildrequires.py", line 193, in generate_tox_requirements
r.check_returncode()
File "/usr/lib64/python3.8/subprocess.py", line 444, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['tox', '--print-deps-to-file', '/tmp/tmpwp8sffv1', '-qre', 'py38']' returned non-zero exit status 1.
Inspired by https://src.fedoraproject.org/rpms/python-chaospy/pull-request/1#comment-32750
The `rm -v` command prints the output to stderr, polluting the generated buildrequires
$ rm -rfv pytest_harvest.dist-info/
removed 'pytest_harvest.dist-info/METADATA'
removed 'pytest_harvest.dist-info/LICENSE'
removed 'pytest_harvest.dist-info/top_level.txt'
removed 'pytest_harvest.dist-info/entry_points.txt'
removed directory 'pytest_harvest.dist-info/'
This can lead to RPM errors:
error: Dependency tokens must begin with alpha-numeric, '_' or '/': 'pytest_harvest.dist-info/METADATA'
Or bogus dependencies -- the SRPM requires "removed" and "directory".
See https://github.com/rpm-software-management/rpm/pull/889
