Commit Graph

79 Commits

Author SHA1 Message Date
Tomas Orsava
438d8d3b70 scripts/pythondistdeps: Backport switch to importlib.metadata from upstream
Upstream change to importlib.metadata: https://github.com/rpm-software-management/rpm/pull/1317

Due to extras packages being hadled slightly differently by importlib,
one test case for this was added.  And due to changes in handling
requires.txt files, comments were removed from the pyreq2rpm.tests
testing package.

Also because of the switch, we removed the dependency on setuptools and
added a dependency on packaging.

Note: Some packages with egg-info files might provide a different name
due to this change if there is a conflict between the filename and the
name in the metadata. Previously, the filename was sometimes used to
parse the name, now it is always the content of that file, which is what
packaging does, and thus also pip and other Python tooling. Currently,
this is known to affect only 1 package in Fedora (ntpsec).

The resulting script is different from upstream because of not yet upstreamed changes in Fedora:
- scripts/pythondistdeps: Rework error messages
- scripts/pythondistdeps: Add parameter --package-name
- scripts/pythondistdeps: Implement provides/requires for extras packages
- pythondistdeps.py: When parsing extras name, take the rightmost +

These changes are proposed in this upstream PR: https://github.com/rpm-software-management/rpm/pull/1546
2021-02-18 16:08:27 +01:00
Miro Hrončok
2d631762c5 Remove unused 2.7 from --majorver-provides-versions
Fixup for 8c2a1c0ac9.
This makes no real difference, just a cleanup, hence not bumping.
2021-02-08 10:53:40 +01:00
Miro Hrončok
8c2a1c0ac9 Disable the dist generators for Python 2
https://fedoraproject.org/wiki/Changes/Disable_Python_2_Dist_RPM_Generators_and_Freeze_Python_2_Macros

The regex previously matched any Python version in a form of <single digit>.<at least one digit>.

Now it matches anything from 3.0 above: <single digit (3 or higher)>.<at least one digit>

It still does not match <multiple digits>.<at least one digit>, e.g. 11.0.

This is a breaking change, hence the version bump.
2021-02-03 14:09:45 +01:00
Fedora Release Engineering
b65cf8549a - Rebuilt for https://fedoraproject.org/wiki/Fedora_34_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2021-01-27 13:13:21 +00:00
Tomas Orsava
f328c9dd18 Add executable bit to pythonbundles.py
pythondistdeps.py is executable already
2020-10-19 12:56:51 +02:00
Tomas Orsava
d77d134c10 Run scripts in an isolated environment (#1889080) 2020-10-19 12:56:43 +02:00
Miro Hrončok
bfb7f70b99 Add a test for a requires with multiple underscores 2020-09-25 14:22:03 +02:00
Tomas Orsava
cb3aaf6d26 Add a test for a requires with an underscore
We already have PyQt5_sip as a test of a provides with an underscore
2020-09-23 11:39:53 +02:00
Fedora Release Engineering
df7ed92279 - Rebuilt for https://fedoraproject.org/wiki/Fedora_33_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2020-07-29 03:42:44 +00:00
Tomas Orsava
64e5d7567a Sync tests for python dependency conversion with pyreq2rpm 2020-07-22 18:00:02 +02:00
Gordon Messmer
fbe1c77166 Sync python dependency conversion with pyreq2rpm. 2020-07-22 18:00:02 +02:00
Miro Hrončok
7398b71fbc pythondistdeps.py: When parsing extras name, take the rightmost + 2020-07-22 00:29:19 +02:00
Miro Hrončok
d1a02fdda7 pythondistdeps.py: Adapt Python version marker workaround for setuptools 42+
See https://bugzilla.redhat.com/show_bug.cgi?id=1853597#c11

pkg_resources from setuptools 42+ no longer only use platform.python_version(),
but also platform.python_version_tuple() -- this was updated in packaging 19.1+.

This fix makes it work again with both new and old setuptools,
hopefully for some while.

bf069fe9dd
86a443f318
2020-07-10 16:30:20 +02:00
Tomas Orsava
32a1b47f5b scripts/pythondistdeps: Tests: small tweaks 2020-07-10 15:46:19 +02:00
Tomas Orsava
c2e0f33565 scripts/pythondistdeps: Add tests for: Rework error messages 2020-07-10 15:37:11 +02:00
Tomas Orsava
098c48d46d scripts/pythondistdeps: Rework error messages 2020-07-10 13:45:59 +02:00
Tomas Orsava
b6e0638f7c Enable --require-extras-subpackages and bump release 2020-07-10 13:43:04 +02:00
Tomas Orsava
9df3e5bcb5 scripts/pythondistdeps: Add tests for: Implement provides/requires for extras packages 2020-07-10 13:42:12 +02:00
Tomas Orsava
0c9665427c scripts/pythondistdeps: Implement provides/requires for extras packages 2020-07-10 13:42:12 +02:00
Tomas Orsava
3b1100ba1f scripts/pythondistdeps: Add parameter --package-name 2020-07-10 13:42:12 +02:00
Miro Hrončok
48c0de39d9 Add a script to generate Python bundled provides
See https://src.fedoraproject.org/rpms/python-setuptools/pull-request/40

Strictly speaking, this is not an RPM generator, but:

 - it generates provides
 - it is tighly coupled with pythondistdeps.py

Usage:

 1. Run `$ /usr/lib/rpm/pythonbundles.py .../vendored.txt`

 2. Copy the output into the spec as a macro definition:

    %global bundled %{expand:
    Provides: bundled(python3dist(appdirs)) = 1.4.3
    Provides: bundled(python3dist(packaging)) = 16.8
    Provides: bundled(python3dist(pyparsing)) = 2.2.1
    Provides: bundled(python3dist(six)) = 1.15
    }

 3. Use the macro to expand the provides
 4. Verify the macro contents in %check:

    %check
    ...
    %{_rpmconfigdir}/pythonbundles.py src/_vendor/vendored.txt --compare-with '%{bundled}'
2020-07-07 16:01:37 +02:00
Miro Hrončok
e78c420523 Fix python(abi) requires generator, it picked files from almost good directories
The %__python_magic filter suddenly got actually working with file 5.39:

Before:

    file.cpython-38.opt-1.pyc: data

After:

    file.cpython-38.opt-1.pyc: python 3.8 byte-compiled

Hence, the filter started to pick all Python files regardless of their location.
Later, in the actual generator, paths like this were considered:

    /opt/usr/lib/python3.X/...

And generated requirements on python(abi).

We don't actually need to filter the files by file magic,
so we drop it to get the previously accidentally working behavior.

We could choose if the path and magic filters are applied as OR or AND.
However, we don't want either.

We actually want to mach any files in Python directories regardless of their magic.
We *could* filter by file type (and executable bit) for provides,
but that would require us to split the attr files into two.
2020-06-18 13:32:24 +02:00
Miro Hrončok
3a396fbf96 Use PEP 503 names for requires 2020-05-21 17:43:19 +02:00
Miro Hrončok
39315a6aa4 Adapt tests for the pythonXY -> pythonX.Y renaming
See https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/message/VIUS7WMQMDX6H2WEIH7TVTMBB6SUHY7E/
2020-05-07 17:59:18 +02:00
Miro Hrončok
33358b9a65 Deduplicate automatically provided names trough Python RPM Lua macros 2020-05-05 14:02:52 +02:00
Miro Hrončok
c3f90ed2e8 Fix reversed grep exit codes in integration tests
grep -v only fails if there are no unmatched lines, but that's not what we want to test.
2020-05-04 13:45:57 +02:00
Miro Hrončok
6beec97e9e Add integartion test for the dist generator 2020-05-04 13:45:57 +02:00
Tomas Orsava
54e4aa751b Bump version, enable new features, add test suite to Fedora CI 2020-05-04 13:45:57 +02:00
Tomas Orsava
79790d12af scripts/pythondistdeps: Modify handling of dev versions 2020-04-30 22:24:44 +02:00
Tomas Orsava
972beac29a scripts/pythondistdeps: Version handling exception with better information 2020-04-30 22:24:44 +02:00
Tomas Orsava
d48f3500d8 scripts/pythondistdeps: Do anything only when called as a main script
Note that the code is completely unchanged except for the indentation
under the new if __name__ == "__main__":

Note that this change is necessary, but not sufficient to use the
RpmVersion class.
The init of the RpmVersion class will fail when called from an outside
script, because the `parse_version()` function is lazily imported from
the code outside the class.  However, adding the import of
parse_version() to RpmVersion class is not done right now, because while
we would import it from `pkg_resources`, other scripts might want to
rely instead of the lightweight `packaging` module for the import. Thus
I'm leaving this conondrum to be addressed in the future.
2020-04-30 22:24:44 +02:00
Tomas Orsava
1523def34e scripts/pythondistdeps: Implement --normalized-name-* options
--normalized-names-format FORMAT
    FORMAT of normalized names can be `pep503` [default] or `legacy-dots` (dots allowed)

--normalized-names-provide-both
    Provede both `pep503` and `legacy-dots` format of normalized names (useful for a transition period)
2020-04-30 22:24:44 +02:00
Tomas Orsava
e33d4e94c8 scripts/pythondistdeps: Add option to generate major-version provides only for specified Python versions 2020-04-30 22:24:44 +02:00
Tomas Orsava
89e1676cee scripts/pythondistdeps: Add tests
The test data download themselves using pip if not present
2020-04-30 22:24:44 +02:00
Tomas Orsava
1634914c2e scripts/pythondistdeps: Notes from an attempted rewrite to importlib.metadata
Notes from an attempted rewrite from pkg_resources to importlib.metadata in 2020:
1. While pkg_resources can open a metadata on a specified path
   (Distribution.from_location()), importlib provides access only to
   "installed package metadata", i.e. the the dist-info or egg-info directory
   must be "discoverable", i.e. on the sys.path.
   - Thankfully only the dist/egg-info directory must exist, the
     corresponding Python module does not have to be present.
   - The problems this causes:
     (a) You have to manipulate the sys.path to add the specific location of
         the site-packages directory inside the buildroot
     (b) If you have package "foo" in this newly added directory on sys.path
         and there is some problem and its dist/egg-info metadata are not found,
         importlib.metadata continues searching the sys.path and may discover a
         package with the same name (possibly same version) outside the
         buildroot.
         To get around this, you can manipulate the sys.path to remove all
         other "site-packages" directories. But you have to leave the
         standard library there, because importlib may import other modules
         (in my testing: base64, quopri, random, socket, calendar, uu)
     (c) I have not tested how well it works if you're ispecting metadata of
         different Python versions than the one you run the script with
         (especially Python 2 vs Python 3). This might also cause problems with
         dependency specifiers (i.e. python_version != "3.4")
2. Handling of dependencies (requires) is problematic in importlib.metadata
   - pkg_resources provides a way to separately list standard requires and a
     requires for each "extras" category. importlib does not provide this, it
     only spits out a list of strings, each string in the format:
     - 'packaging>=14',
     - 'towncrier>=18.5.0; extra == "docs"', or
     - 'psutil<6,>=5.6.1; (python_version != "3.4") and extra == "testing"
     you can either parse these with a regex (fragile) or use the external
     `packaging` Python module. `packaging`, however, also doesn't have a great
     support for figuring out extra dependencies, it provides the marker api:
     - <Marker(\'python_version != "3.4" and extra == "testing"\')>
     you can use Marker api to evaluate the condition, but not to parse.
     For parsing you can access the private api Marker._markers:
     - marker._markers=[[(<Variable('python_version')>, <Op('!=')>, \
           <Value('3.4')>)], 'and', (<Variable('extra')>, <Op('==')>, \
           <Value('testing')>)]
     which beyond the problem of being private is also not very useful for
     parsing due to its structure.
   - pkg_resources also provides version parsing, which importlib does not
     and `packaging` needs to be used
   - importlib is part of the standard library, but packaging and its
     2 runtime dependencies (pyparsing and six) are not, and therefore we
     would go from 1 dependency to 3
3. A few minor issues, more in the next section about equivalents.

importlib.metadata.distribution equivalents of pkg_resources.Distribution attributes:
- pkg_resources: dist.py_version
  importlib: # not implemented (but can be guessed from the /usr/lib/pythonXX.YY/ path)
- pkg_resources: dist.project_name
  importlib: dist.metadata['name']
- pkg_resources: dist.key
  importlib: # not implemented
- pkg_resources: dist.version
  importlib: dist.version
- pkg_resources: dist.requires()
  importlib: dist.requires  # but returns strings with almost no parsing done, and also lists extras
- pkg_resources: dist.requires(extras=dist.extras)
  importlib: # not implemented, has to be parsed from dist.requires
- pkg_resources: dist.get_entry_map('console_scripts')
  importlib: [ep for ep in importlib.metadata.entry_points()['console_scripts'] if ep.name == pkg][0]
             # I have not found a better way to get the console_scripts
- pkg_resources: dist.get_entry_map('gui_scripts')
  importlib: # Presumably same as console_scripts, but untested
2020-04-30 22:24:44 +02:00
Tomas Orsava
1639424a51 Sync with upstream RPM dist generator 2020-04-30 22:24:42 +02:00
Miro Hrončok
c8249102ec Don't define global Lua variables from Python generator 2020-04-28 14:48:06 +02:00
Gordon Messmer
0ec8581037 Handle all-zero versions without crashing
From https://github.com/rpm-software-management/rpm/pull/1184
2020-04-20 13:55:53 +02:00
Igor Raits
783dcc7147 Sync with upstream RPM dist generator 2020-04-10 12:31:30 +02:00
Miro Hrončok
8eef42cbaa Use dynamic %_prefix value when matching files for python(abi) provides
See https://lists.fedoraproject.org/archives/list/packaging@lists.fedoraproject.org/thread/UFKUM5UKCTNGIT3KJVYEI5VXPI23QMBN/

Flatpak builds redefine %_prefix and the dependencies should remain present.

Also get rid of one useless ^ and prep the pattern for two digit Python major versions.

Add a test that tests that we match our default %_prefix (was the case even before this commit).
2020-04-07 16:25:11 +02:00
Miro Hrončok
bbfe4930d9 Automatically call %python_provide
This allows us to drop the %python_provide macro from most spec files,
except where we want to use it for virtual provides or empty packages.
2020-04-03 16:06:46 +02:00
Miro Hrončok
eae8dd0f57 Add CI tests for python(abi) provides 2020-04-03 14:35:16 +02:00
Miro Hrončok
486ca7e540 Drop tabs from python.attr 2020-04-01 15:57:00 +02:00
Miro Hrončok
ff7b9b1ae0 Reimplement pythondeps.sh as parametric macro generators
pythondeps.sh was written in shell and unlike the Python dist generators,
it uses no Python, it plainly determines the provide / requires from the path.
As the script was run for every Python file, we were potentially doing hundreds
of shelling outs to execute a script that calls grep and sed.

In Lua, this is much more efficient.

Some timings:
    https://github.com/rpm-software-management/rpm/pull/1153#issuecomment-607146356

Parametric macro generators require RPM 4.16+:
    https://fedoraproject.org/wiki/Changes/RPM-4.16

Fixes https://github.com/rpm-software-management/rpm/issues/1152
Upstream PR: https://github.com/rpm-software-management/rpm/pull/1153

Since this is intended for Fedora 33+ only, clean some old cruft.
2020-04-01 15:53:15 +02:00
Fedora Release Engineering
caccd3e498 - Rebuilt for https://fedoraproject.org/wiki/Fedora_32_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2020-01-30 15:05:48 +00:00
Miro Hrončok
7d819e0000 Also provide pythonXdist() with PEP 503 normalized names (#1791530)
That is, we add new provides that replace dots with a dash.

Package that used to provide python3dist(zope.component) and python3.8dist(zope.component)
now also provides python3dist(zope-component) and python3.8dist(zope-component).

Package that used to provide python3dist(a.-.-.-.a) now provides python3dist(a-a) as well.

This is consistent with pip behavior, `pip install zope-component` installs zope.component.

Historically, we have always used dist.key (safe_name) from setuptools,
but that is a non-standardized convention -- whether or not it replaces dots
with dashes is not even documented.
We say we use "canonical name" or "normalized name" everywhere, yet we didn't.

We really need to follow the standard (PEP 503):

https://www.python.org/dev/peps/pep-0503/#normalized-names

The proper function here would be packaging.utils.canonicalize_name
https://packaging.pypa.io/en/latest/utils/#packaging.utils.canonicalize_name
-- we reimplement it here to avoid an external dependency.

This is the first required step needed if we want to change our requirements later.
If we decide we don't, for whatever reason, this doesn't break anything.
2020-01-17 17:27:46 +01:00
Miro Hrončok
724a52a5f2 Fix more complicated requirement expressions by adding parenthesis
Puts bounded requirements into parenthesis

Fixes: https://github.com/rpm-software-management/rpm/issues/995
Upstream: https://github.com/rpm-software-management/rpm/pull/996

For this input: pyparsing>=2.0.1,!=2.0.4,!=2.1.2,!=2.1.6

Instead of (invalid):
(python3.8dist(pyparsing) >= 2.0.1 with
 python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0 with
 python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0 with
 python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0)

Produces (valid):
(python3.8dist(pyparsing) >= 2.0.1 with
 (python3.8dist(pyparsing) < 2.1.2 or python3.8dist(pyparsing) >= 2.1.2.0) with
 (python3.8dist(pyparsing) < 2.0.4 or python3.8dist(pyparsing) >= 2.0.4.0) with
 (python3.8dist(pyparsing) < 2.1.6 or python3.8dist(pyparsing) >= 2.1.6.0))

For this input: babel>=1.3,!=2.0

Instead of (invalid):
(python3.8dist(babel) >= 1.3 with
 python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0)

Produces (valid):
(python3.8dist(babel) >= 1.3 with
 (python3.8dist(babel) < 2 or python3.8dist(babel) >= 2.0))

For this input: pbr!=2.1.0,>=2.0.0

Instead of (invalid):
(python3.8dist(pbr) >= 2 with
 python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0)

Produces (valid):
(python3.8dist(pbr) >= 2 with
 (python3.8dist(pbr) < 2.1 or python3.8dist(pbr) >= 2.1.0))
2020-01-03 11:00:19 +01:00
Miro Hrončok
ca811dbf35 Sync with upstream RPM
- Handle version ending with ".*"
 - Handle compatible-release operator "~="
 - Use rich deps for semantically versioned dependencies
 - Match Python version if minor has multiple digits (e.g. 3.10)
 - Only add setuptools requirement for egg-info packages

https://github.com/rpm-software-management/rpm/pull/951
https://github.com/rpm-software-management/rpm/pull/973
https://github.com/rpm-software-management/rpm/pull/982

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1758141
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1777382
2020-01-01 23:21:46 +01:00
Fedora Release Engineering
b9fe0e7182 - Rebuilt for https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild
Signed-off-by: Fedora Release Engineering <releng@fedoraproject.org>
2019-07-26 16:06:05 +00:00
Miro Hrončok
ff085a044d Canonicalize Python versions and properly handle != spec
Fixes https://github.com/rpm-software-management/rpm/issues/639

From upstream PR:  https://github.com/rpm-software-management/rpm/pull/757
2019-06-24 14:44:19 +02:00