rpms/dnf
7
0
Fork 0
Code Issues Pull Requests Releases Activity

Backport PR #1038: make compose fail on missing group packages

Browse Source
This commit is contained in:
Adam Williamson 2018-09-20 20:30:50 -07:00
parent 463d813b02
commit 1a4e2c490c
2 changed files with 427 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

420
0001-Restore-strict-choice-for-group-installs-1461539.patch Normal file
View File

@ -0,0 +1,420 @@
From edbd86922b2733fd36622abadca15e744d12bfde Mon Sep 17 00:00:00 2001
From: Adam Williamson <awilliam@redhat.com>
Date: Tue, 13 Mar 2018 15:33:24 -0700
Subject: [PATCH] Restore 'strict' choice for group installs (#1461539)
This makes the `strict` parameter to `group_install` mean
something again.
When it is set `True`, DNF will behave as yum previously did,
which has been a request for several years now. See:
https://bugzilla.redhat.com/show_bug.cgi?id=1292892
https://bugzilla.redhat.com/show_bug.cgi?id=1337731
https://bugzilla.redhat.com/show_bug.cgi?id=1427365
https://bugzilla.redhat.com/show_bug.cgi?id=1461539
And commits:
91f9ce98dbb630800017f44180d636af395435cd
1e1901822748b754d038dd396655c997281a7201
We have been around the houses on this multiple times. @mikem23
actually got things right way way back in #1292892:
"yum-deprecated behaves more sanely here.
In case 1 [package exists but has dependency issues], it reports
the missing dependency and exits with an error
...
In case 2 [package cannot be found at all], it exits cleanly, but
prints a warning."
Note that this applies to yum's *default* behaviour: it had an
option, --skip-broken, which caused it to skip packages with
dependency errors.
That's exactly what we've wanted all along, and it's also what
1461539 requests. Unfortunately, the initial commit intended to
fix #1292892 - that's 91f9ce9 - did not quite behave as Mike
requested. It treated any 'mandatory' package having dependency
errors *or* not existing at all as fatal errors, while not
treating non-existence *or* errors on the part of a 'default'
or 'optional' package as a fatal error. This introduced *two*
differences from yum's behaviour (making the comps 'type' matter,
which it never did to yum, and failing on non-existence), which
I think was the source of quite a lot of confusion.
We then wound up filing bugs on this - like #1337731 - and the
response to that was 1e19018, which causes dnf to treat *neither*
case as a fatal error for *any* package, which is where we stand
at present.
When resolving a transaction, currently, if it cannot find a
package corresponding to an entry in a group, dnf will log a
warning and carry on. This is good and fine and just what we want
it to do. However, when a package is available but cannot be
installed, it does the wrong thing. The various forks of this
`trans_install` internal function all wind up passing
`optional=True` to the Goal, which tells it that it's fine to
just leave the package out if its dependencies can't be resolved.
Additionally, if this happens, `resolve()` itself does not log
anything.
If you try a group install through the CLI, it will tell you
about packages that have been left out due to dependency errors,
via the `_skipped_packages` function in `dnf/output.py` which
actually sucks them out of the Goal instance itself. But that's
no use to anything besides the CLI. Other things which use
`resolve()`, like anaconda, have no means of accessing this
information, so when this happens to them, the omission of the
package is not logged in any way at all, and they have no way to
find out about it.
If `strict` is set to `False`, the current behaviour described
above will be used: non-installable packages will be skipped. We
implement this by restoring the old separation between `install`
and `install_opt` which was removed in 1e19018. When group_install
is called with `strict=True` we add the packages to the `install`
set; when it's called with `strict=False` we add the packages to
the `install_opt` set. Then `_finalize_comps_trans` requests
strict behaviour for `install` and non-strict for `install_opt`.
It continues to be the case that the comps 'type' - mandatory,
default, or optional - does not matter; the same behaviour will be
used for all three. Again, this is how yum behaved (in the
absence of --skip-broken)
The comps 'types' were *never* intended to dictate yum/dnf
behaviour in terms of whether it should fail on error or not, as I
understand it; they were in fact related to installer UI behaviour.
In the old installer UI, after you selected a group, you would see
all the packages in the group listed. 'mandatory' packages would
be pre-checked for installation and the check would be greyed out
- you couldn't unselect them. 'default' packages would be
pre-checked for installation, but you *could* de-select them if you
wanted. 'optional' packages would not be pre-checked for
installation, but you could *select* them if you wanted. This was
the intent of the comps 'type', as I understand it.
Importantly, a package not existing at all will continue to *not*
be a fatal error whatever `strict` is set to. as this is handled
*before* we reach the `trans_install` function.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
---
dnf/base.py | 21 ++++--
dnf/comps.py | 27 +++++++-
tests/repos/broken_group.repo | 4 ++
tests/repos/main_comps.xml | 4 +-
tests/test_groups.py | 122 +++++++++++++++++++++++++++-------
5 files changed, 145 insertions(+), 33 deletions(-)
create mode 100644 tests/repos/broken_group.repo
diff --git a/dnf/base.py b/dnf/base.py
index 88e9467a..dacc42d9 100644
--- a/dnf/base.py
+++ b/dnf/base.py
@@ -1450,17 +1450,17 @@ class Base(object):
self._goal.upgrade(select=sltr)
return remove_query
- def trans_install(query, remove_query, comps_pkg):
+ def trans_install(query, remove_query, comps_pkg, strict):
if self.conf.multilib_policy == "all":
if not comps_pkg.requires:
- self._install_multiarch(query, strict=False)
+ self._install_multiarch(query, strict=strict)
else:
# it installs only one arch for conditional packages
installed_query = query.installed().apply()
self._report_already_installed(installed_query)
sltr = dnf.selector.Selector(self.sack)
sltr.set(provides="({} if {})".format(comps_pkg.name, comps_pkg.requires))
- self._goal.install(select=sltr, optional=True)
+ self._goal.install(select=sltr, optional=not strict)
else:
sltr = dnf.selector.Selector(self.sack)
@@ -1468,7 +1468,7 @@ class Base(object):
sltr.set(provides="({} if {})".format(comps_pkg.name, comps_pkg.requires))
else:
sltr.set(pkg=query)
- self._goal.install(select=sltr, optional=True)
+ self._goal.install(select=sltr, optional=not strict)
return remove_query
def trans_remove(query, remove_query, comps_pkg):
@@ -1476,7 +1476,8 @@ class Base(object):
return remove_query
remove_query = self.sack.query().filterm(empty=True)
- attr_fn = ((trans.install, trans_install),
+ attr_fn = ((trans.install, functools.partial(trans_install, strict=True)),
+ (trans.install_opt, functools.partial(trans_install, strict=False)),
(trans.upgrade, trans_upgrade),
(trans.remove, trans_remove))
@@ -1547,6 +1548,10 @@ class Base(object):
"""Installs packages of selected group
:param exclude: list of package name glob patterns
that will be excluded from install set
+ :param strict: boolean indicating whether group packages that
+ exist but are non-installable due to e.g. dependency
+ issues should be skipped (False) or cause transaction to
+ fail to resolve (True)
"""
def _pattern_to_pkgname(pattern):
if dnf.util.is_glob_pattern(pattern):
@@ -1568,8 +1573,12 @@ class Base(object):
strict)
if not trans:
return 0
+ if strict:
+ instlog = trans.install
+ else:
+ instlog = trans.install_opt
logger.debug(_("Adding packages from group '%s': %s"),
- grp_id, trans.install)
+ grp_id, instlog)
return self._add_comps_trans(trans)
def env_group_install(self, patterns, types, strict=True, exclude=None, exclude_groups=None):
diff --git a/dnf/comps.py b/dnf/comps.py
index 079d21be..8743812a 100644
--- a/dnf/comps.py
+++ b/dnf/comps.py
@@ -476,18 +476,20 @@ class CompsTransPkg(object):
class TransactionBunch(object):
def __init__(self):
self._install = set()
+ self._install_opt = set()
self._remove = set()
self._upgrade = set()
def __iadd__(self, other):
self._install.update(other._install)
+ self._install_opt.update(other._install_opt)
self._upgrade.update(other._upgrade)
self._remove = (self._remove | other._remove) - \
- self._install - self._upgrade
+ self._install - self._install_opt - self._upgrade
return self
def __len__(self):
- return len(self.install) + len(self.upgrade) + len(self.remove)
+ return len(self.install) + len(self.install_opt) + len(self.upgrade) + len(self.remove)
@staticmethod
def _set_value(param, val):
@@ -499,12 +501,28 @@ class TransactionBunch(object):
@property
def install(self):
+ """
+ Packages to be installed with strict=True - transaction will
+ fail if they cannot be installed due to dependency errors etc.
+ """
return self._install
@install.setter
def install(self, value):
self._set_value(self._install, value)
+ @property
+ def install_opt(self):
+ """
+ Packages to be installed with strict=False - they will be
+ skipped if they cannot be installed
+ """
+ return self._install_opt
+
+ @install_opt.setter
+ def install_opt(self, value):
+ self._set_value(self._install_opt, value)
+
@property
def remove(self):
return self._remove
@@ -641,7 +659,10 @@ class Solver(object):
trans = TransactionBunch()
# TODO: remove exclude
- trans.install.update(self._pkgs_of_type(comps_group, pkg_types, exclude=[]))
+ if strict:
+ trans.install.update(self._pkgs_of_type(comps_group, pkg_types, exclude=[]))
+ else:
+ trans.install_opt.update(self._pkgs_of_type(comps_group, pkg_types, exclude=[]))
return trans
def _group_remove(self, group_id):
diff --git a/tests/repos/broken_group.repo b/tests/repos/broken_group.repo
new file mode 100644
index 00000000..7f151971
--- /dev/null
+++ b/tests/repos/broken_group.repo
@@ -0,0 +1,4 @@
+=Ver: 2.0
+#
+=Pkg: brokendeps 20 2 x86_64
+=Req: nosuchpackage >= 1.2-0
diff --git a/tests/repos/main_comps.xml b/tests/repos/main_comps.xml
index 3cf8faa5..9e694d13 100644
--- a/tests/repos/main_comps.xml
+++ b/tests/repos/main_comps.xml
@@ -44,7 +44,9 @@
<name>Broken Group</name>
<packagelist>
<packagereq type="mandatory">meaning-of-life</packagereq>
- <packagereq>lotus</packagereq>
+ <packagereq type="mandatory">lotus</packagereq>
+ <packagereq type="default" requires="no-such-package">librita</packagereq>
+ <packagereq type="optional">brokendeps</packagereq>
</packagelist>
</group>
<category>
diff --git a/tests/test_groups.py b/tests/test_groups.py
index ec10a619..fe388f96 100644
--- a/tests/test_groups.py
+++ b/tests/test_groups.py
@@ -183,29 +183,6 @@ class PresetPersistorTest(tests.support.ResultTestCase):
swdb_group = self.history.group.get(comps_group.id)
self.assertIsNotNone(swdb_group)
- """
- this should be reconsidered once relengs document comps
- def test_group_install_broken(self):
- grp = self.base.comps.group_by_pattern('Broken Group')
- p_grp = self.history.group.get('broken-group')
- self.assertFalse(p_grp.installed)
-
- self.assertRaises(dnf.exceptions.MarkingError,
- self.base.group_install, grp.id,
- ('mandatory', 'default'))
- p_grp = self.history.group.get('broken-group')
- self.assertFalse(p_grp.installed)
-
- self.assertEqual(self.base.group_install(grp.id,
- ('mandatory', 'default'),
- strict=False), 1)
- inst, removed = self.installed_removed(self.base)
- self.assertLength(inst, 1)
- self.assertEmpty(removed)
- p_grp = self.history.group.get('broken-group')
- self.assertTrue(p_grp.installed)
- """
-
def test_group_remove(self):
self._install_test_group()
group_id = 'somerset'
@@ -220,6 +197,105 @@ class PresetPersistorTest(tests.support.ResultTestCase):
self._swdb_end()
+class ProblemGroupTest(tests.support.ResultTestCase):
+ """Test some cases involving problems in groups: packages that
+ don't exist, and packages that exist but cannot be installed. The
+ "broken" group lists three packages. "meaning-of-life", explicitly
+ 'default', does not exist. "lotus", implicitly 'mandatory' (no
+ explicit type), exists and is installable. "brokendeps",
+ explicitly 'optional', exists but has broken dependencies. See
+ https://bugzilla.redhat.com/show_bug.cgi?id=1292892,
+ https://bugzilla.redhat.com/show_bug.cgi?id=1337731,
+ https://bugzilla.redhat.com/show_bug.cgi?id=1427365, and
+ https://bugzilla.redhat.com/show_bug.cgi?id=1461539 for some of
+ the background on this.
+ """
+
+ REPOS = ['main', 'broken_group']
+ COMPS = True
+ COMPS_SEED_PERSISTOR = True
+
+ def test_group_install_broken_mandatory(self):
+ """Here we will test installing the group with only mandatory
+ packages. We expect this to succeed, leaving out the
+ non-existent 'meaning-of-life': it should also log a warning,
+ but we don't test that.
+ """
+ comps_group = self.base.comps.group_by_pattern('Broken Group')
+ swdb_group = self.history.group.get(comps_group.id)
+ self.assertIsNone(swdb_group)
+
+ cnt = self.base.group_install(comps_group.id, ('mandatory'))
+ self._swdb_commit()
+ self.base.resolve()
+ # this counts packages *listed* in the group, so 2
+ self.assertEqual(cnt, 2)
+
+ inst, removed = self.installed_removed(self.base)
+ # the above should work, but only 'lotus' actually installed
+ self.assertLength(inst, 1)
+ self.assertEmpty(removed)
+
+ def test_group_install_broken_default(self):
+ """Here we will test installing the group with only mandatory
+ and default packages. Again we expect this to succeed: the new
+ factor is an entry pulling in librita if no-such-package is
+ also included or installed. We expect this not to actually
+ pull in librita (as no-such-package obviously *isn't* there),
+ but also not to cause a fatal error.
+ """
+ comps_group = self.base.comps.group_by_pattern('Broken Group')
+ swdb_group = self.history.group.get(comps_group.id)
+ self.assertIsNone(swdb_group)
+
+ cnt = self.base.group_install(comps_group.id, ('mandatory', 'default'))
+ self._swdb_commit()
+ self.base.resolve()
+ # this counts packages *listed* in the group, so 3
+ self.assertEqual(cnt, 3)
+
+ inst, removed = self.installed_removed(self.base)
+ # the above should work, but only 'lotus' actually installed
+ self.assertLength(inst, 1)
+ self.assertEmpty(removed)
+
+ def test_group_install_broken_optional(self):
+ """Here we test installing the group with optional packages
+ included. We expect this to fail, as a package that exists but
+ has broken dependencies is now included.
+ """
+ comps_group = self.base.comps.group_by_pattern('Broken Group')
+ swdb_group = self.history.group.get(comps_group.id)
+ self.assertIsNone(swdb_group)
+
+ cnt = self.base.group_install(comps_group.id, ('mandatory', 'default', 'optional'))
+ self.assertEqual(cnt, 4)
+
+ self._swdb_commit()
+ # this should fail, as optional 'brokendeps' is now pulled in
+ self.assertRaises(dnf.exceptions.DepsolveError, self.base.resolve)
+
+ def test_group_install_broken_optional_nonstrict(self):
+ """Here we test installing the group with optional packages
+ included, but with strict=False. We expect this to succeed,
+ skipping the package with broken dependencies.
+ """
+ comps_group = self.base.comps.group_by_pattern('Broken Group')
+ swdb_group = self.history.group.get(comps_group.id)
+ self.assertIsNone(swdb_group)
+
+ cnt = self.base.group_install(comps_group.id, ('mandatory', 'default', 'optional'),
+ strict=False)
+ self._swdb_commit()
+ self.base.resolve()
+ self.assertEqual(cnt, 4)
+
+ inst, removed = self.installed_removed(self.base)
+ # the above should work, but only 'lotus' actually installed
+ self.assertLength(inst, 1)
+ self.assertEmpty(removed)
+
+
class EnvironmentInstallTest(tests.support.ResultTestCase):
"""Set up a test where sugar is considered not installed."""
--
2.19.0

8
dnf.spec
View File

@ -72,12 +72,15 @@ It supports RPMs, modules and comps groups & environments.
Name: dnf
Version: 3.5.1
Release: 1%{?dist}
Release: 2%{?dist}
Summary: %{pkg_summary}
# For a breakdown of the licensing, see PACKAGE-LICENSING
License: GPLv2+ and GPLv2 and GPL
URL: https://github.com/rpm-software-management/dnf
Source0: %{url}/archive/%{version}/%{name}-%{version}.tar.gz
# Backport of https://github.com/rpm-software-management/dnf/pull/1038
# Should make compose fail on missing group packages again
Patch0: 0001-Restore-strict-choice-for-group-installs-1461539.patch
BuildArch: noarch
BuildRequires: cmake
BuildRequires: gettext
@ -487,6 +490,9 @@ ln -sr %{buildroot}%{confdir}/vars %{buildroot}%{_sysconfdir}/yum/vars
%endif
%changelog
* Thu Sep 20 2018 Adam Williamson <awilliam@redhat.com> - 3.5.1-2
- Backport PR #1038 to make compose fail on missing group packages again
* Mon Sep 10 2018 Jaroslav Mracek <jmracek@redhat.com> - 3.5.1-1
- [module] Fixed list and info subcommands