1182 lines
50 KiB
Diff
1182 lines
50 KiB
Diff
diff --git a/README.md b/README.md
|
||
index 4de458b..6f8969d 100644
|
||
--- a/README.md
|
||
+++ b/README.md
|
||
@@ -29,3 +29,144 @@ We’ll gladly answer your questions and lead you to through any troubles with t
|
||
actor development.
|
||
|
||
You can reach us at IRC: `#leapp` on freenode.
|
||
+
|
||
+
|
||
+## Third-party integration
|
||
+
|
||
+If you would like to add your **signed** 3rd party packages into the upgrade process, you can use the third-party integration mechanism.
|
||
+
|
||
+There are four components for adding your information to the elevation process:
|
||
+- <vendor_name>.csv: repository mapping file
|
||
+- <vendor_name>.repo: package repository information
|
||
+- <vendor_name>.sigs: list of package signatures of vendor repositories
|
||
+- <vendor_name>.json: package migration event list
|
||
+
|
||
+All these files **must** have the same <vendor_name> part.
|
||
+
|
||
+### Repository mapping file
|
||
+
|
||
+This CSV file provides information on mappings between source system repositories (repositories present on the system being upgraded) and target system repositories (package repositories to be used during the upgrade).
|
||
+
|
||
+The first line of the file, per CSV format, should contain the headers. Standard headers for vendor.csv files look like this:
|
||
+
|
||
+```CSV
|
||
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
|
||
+```
|
||
+
|
||
+Following lines should contain the repository map entries. As an example:
|
||
+
|
||
+```CSV
|
||
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
|
||
+
|
||
+source-repoid,target-custom-repoid,target-pes-repoid,all,all,x86_64,rpm,ga,ga
|
||
+```
|
||
+
|
||
+**Source system repoid** is the ID of a repository that is expected to be present on the system before the upgrade.
|
||
+
|
||
+**Target system repoid in custom repo file** is the ID of a repository listed in the associated package repository information (<vendor_name>.repo) file. It is supposed to be used during the upgrade process.
|
||
+
|
||
+**Target system repo name in PES** is the ID which is used to refer to the target system repository in the package migration event list (<vendor_name>.json).
|
||
+
|
||
+**Repository types**:
|
||
+- rpm: normal RPM packages
|
||
+- srpm: source packages
|
||
+- debuginfo: packages with debug information
|
||
+
|
||
+**Product types**:
|
||
+- GA: general availability repositories
|
||
+- Beta: beta-testing repositories
|
||
+- HTB: High Touch Beta repositories
|
||
+
|
||
+The repository mapping file also defines whether a vendor's packages will be included into the upgrade process at all. If at least one source repository listed in the file is present on the system, the vendor is considered active, and package repositories/PES events are enabled - otherwise, they will not affect the upgrade process.
|
||
+
|
||
+In the above example, vendor's data (including the .repo and .json files) will affect the upgrade process only if a repository with `source-repoid` ID is present on the system.
|
||
+
|
||
+
|
||
+### Package repository information
|
||
+
|
||
+This file defines the vendor's package repositories to be used during the upgrade.
|
||
+
|
||
+The file has the same format normal YUM/DNF package repository files do.
|
||
+
|
||
+> NOTE: The repositories listed in this file are only used *during* the upgrade. Package repositories on the post-upgrade system should be provided through updated packages or custom repository deployment.
|
||
+
|
||
+### Package signature list
|
||
+
|
||
+This file should contain the list of public signature headers that the packages are signed with, one entry per line.
|
||
+
|
||
+You can find signature headers for your packages by running the following command:
|
||
+
|
||
+`rpm -qa --queryformat "%{NAME} || %|DSAHEADER?{%{DSAHEADER:pgpsig}}:{%|RSAHEADER?{%{RSAHEADER:pgpsig}}:{(none)}|}|\n" <PACKAGE_NAME>`
|
||
+
|
||
+rpm will return an entry like the following:
|
||
+`package-name || DSA/SHA1, Mon Aug 23 08:17:13 2021, Key ID 8c55a6628608cb71`
|
||
+
|
||
+The value after "Key ID", in this case, `8c55a6628608cb71`, is what you should put into the signature list file.
|
||
+
|
||
+### Package migration event list
|
||
+
|
||
+The Leapp upgrade process uses information from the AlmaLinux PES (Package Evolution System) to keep track of how packages change between the OS versions. This data is located in `leapp-data/files/<target_system>/vendors.d/<vendor_name>.json` in the GitHub repository and in `/etc/leapp/files/vendors.d/<vendor_name>.json` on a system being upgraded.
|
||
+
|
||
+To add new rules to the list, add a new entry to the `packageinfo` array.
|
||
+**Important**: actions from PES json files will be in effect only for those packages that are signed **and** have their signatures in one of the active <vendor_name>.sigs files. Unsigned packages will be updated only if some signed package requires a new version, otherwise they will by left as they are.
|
||
+
|
||
+Required fields:
|
||
+
|
||
+- action: what action to perform on the listed package
|
||
+ - 0 - present
|
||
+ - 1 - removed
|
||
+ - 2 - deprecated
|
||
+ - 3 - replaced
|
||
+ - 4 - split
|
||
+ - 5 - merged
|
||
+ - 6 - moved to new repository
|
||
+ - 7 - renamed
|
||
+- arches: what system architectures the listed entry relates to
|
||
+- id: entry ID, must be unique
|
||
+- in_packageset: set of packages on the old system
|
||
+- out_packageset: set of packages to switch to, empty if removed or deprecated
|
||
+- initial_release: source OS release
|
||
+- release: target OS release
|
||
+
|
||
+`in_packageset` and `out_packageset` have the following format:
|
||
+
|
||
+```json
|
||
+ "in_packageset": {
|
||
+ "package": [
|
||
+ {
|
||
+ "module_stream": null,
|
||
+ "name": "PackageKit",
|
||
+ "repository": "base"
|
||
+ },
|
||
+ {
|
||
+ "module_stream": null,
|
||
+ "name": "PackageKit-yum",
|
||
+ "repository": "base"
|
||
+ }
|
||
+ ],
|
||
+ "set_id": 1592
|
||
+ },
|
||
+```
|
||
+
|
||
+For `in_packageset`, `repository` field defines the package repository the package was installed from on the source system.
|
||
+For `out_packageset`, `repository` field for packages should be the same as the "Target system repo name in PES" field in the associated vendor repository mapping file.
|
||
+Warning: leapp doesn't force packages from out_packageset to be installed from the specific repository; instead, it enables repo from out_packageset and then dnf installs the latest package version from all enabled repos.
|
||
+
|
||
+To take the above repository map example:
|
||
+
|
||
+```CSV
|
||
+Source system repoid,Target system repoid in custom repo file,Target system repo name in PES,Source system minor versions,Target system minor versions,architecture,type (rpm/srpm/debuginfo),source product type (ga/beta,htb),target product type (ga/beta/htb)
|
||
+
|
||
+source-repoid,target-custom-repoid,target-pes-repoid,all,all,x86_64,rpm,ga,ga
|
||
+```
|
||
+
|
||
+For this configuration, `in_packageset` entries would have `source-repoid` as the `repository` field, and `out_packageset` would have `target-pes-repoid` in theirs.
|
||
+
|
||
+Please refer to [PES contribution guide](https://wiki.almalinux.org/elevate/Contribution-guide.html) for additional information on entry fields.
|
||
+
|
||
+### Providing the data
|
||
+
|
||
+Once you've prepared the vendor data for migration, you can make a pull request to https://github.com/AlmaLinux/leapp-data/ to make it available publicly.
|
||
+Files should be placed in `files/<target-system>/vendors.d/`.
|
||
+
|
||
+Alternatively, you can deploy the vendor files on a system prior to starting the upgrade. In this case, place the files into the folder `/etc/leapp/files/vendors.d/`.
|
||
diff --git a/commands/command_utils.py b/commands/command_utils.py
|
||
index da62c50..a8e7d76 100644
|
||
--- a/commands/command_utils.py
|
||
+++ b/commands/command_utils.py
|
||
@@ -12,7 +12,7 @@ LEAPP_UPGRADE_FLAVOUR_DEFAULT = 'default'
|
||
LEAPP_UPGRADE_FLAVOUR_SAP_HANA = 'saphana'
|
||
LEAPP_UPGRADE_PATHS = 'upgrade_paths.json'
|
||
|
||
-VERSION_REGEX = re.compile(r"^([1-9]\d*)\.(\d+)$")
|
||
+VERSION_REGEX = re.compile(r"^([1-9]\d*)(\.(\d+))?$")
|
||
|
||
|
||
def check_version(version):
|
||
diff --git a/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py b/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
|
||
index a2cede0..5ff1c76 100644
|
||
--- a/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
|
||
+++ b/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
|
||
@@ -17,7 +17,7 @@ def add_boot_entry(configs=None):
|
||
'/usr/sbin/grubby',
|
||
'--add-kernel', '{0}'.format(kernel_dst_path),
|
||
'--initrd', '{0}'.format(initram_dst_path),
|
||
- '--title', 'RHEL-Upgrade-Initramfs',
|
||
+ '--title', 'ELevate-Upgrade-Initramfs',
|
||
'--copy-default',
|
||
'--make-default',
|
||
'--args', '{DEBUG} enforcing=0 rd.plymouth=0 plymouth.enable=0'.format(DEBUG=debug)
|
||
diff --git a/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py b/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
|
||
index bb89c9f..2b8e7c8 100644
|
||
--- a/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
|
||
+++ b/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
|
||
@@ -42,7 +42,7 @@ run_args_add = [
|
||
'/usr/sbin/grubby',
|
||
'--add-kernel', '/abc',
|
||
'--initrd', '/def',
|
||
- '--title', 'RHEL-Upgrade-Initramfs',
|
||
+ '--title', 'ELevate-Upgrade-Initramfs',
|
||
'--copy-default',
|
||
'--make-default',
|
||
'--args',
|
||
diff --git a/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py b/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py
|
||
new file mode 100644
|
||
index 0000000..51d0c25
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py
|
||
@@ -0,0 +1,55 @@
|
||
+from leapp.actors import Actor
|
||
+from leapp.libraries.stdlib import api
|
||
+from leapp.models import (
|
||
+ RepositoriesFacts,
|
||
+ VendorRepositoriesMapCollection,
|
||
+ ActiveVendorList,
|
||
+)
|
||
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
+
|
||
+
|
||
+class CheckEnabledVendorRepos(Actor):
|
||
+ """
|
||
+ Create a list of vendors whose repositories are present on the system.
|
||
+ Only those vendors' configurations (new repositories, PES actions, etc.)
|
||
+ will be included in the upgrade process.
|
||
+ """
|
||
+
|
||
+ name = "check_enabled_vendor_repos"
|
||
+ consumes = (RepositoriesFacts, VendorRepositoriesMapCollection)
|
||
+ produces = (ActiveVendorList)
|
||
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
|
||
+
|
||
+ def process(self):
|
||
+ vendor_mapping_data = {}
|
||
+ active_vendors = []
|
||
+
|
||
+ # Make a dict for easy lookup of repoid -> vendor name.
|
||
+ for map_coll in api.consume(VendorRepositoriesMapCollection):
|
||
+ for map in map_coll.maps:
|
||
+ for repo in map.repositories:
|
||
+ # Cut the .csv, keep only the vendor name.
|
||
+ vendor_mapping_data[repo.from_repoid] = map.file[:-4]
|
||
+
|
||
+ # Is the repo listed in the vendor map as from_repoid present on the system?
|
||
+ for repos in api.consume(RepositoriesFacts):
|
||
+ for repo_file in repos.repositories:
|
||
+ for repo in repo_file.data:
|
||
+ self.log.debug(
|
||
+ "Looking for repository {} in vendor maps".format(repo.repoid)
|
||
+ )
|
||
+ if repo.repoid in vendor_mapping_data:
|
||
+ # If the vendor's repository is present in the system, count the vendor as active.
|
||
+ new_vendor = vendor_mapping_data[repo.repoid]
|
||
+ self.log.debug(
|
||
+ "Repository {} found, enabling vendor {}".format(
|
||
+ repo.repoid, new_vendor
|
||
+ )
|
||
+ )
|
||
+ active_vendors.append(new_vendor)
|
||
+
|
||
+ if active_vendors:
|
||
+ self.log.debug("Active vendor list: {}".format(active_vendors))
|
||
+ api.produce(ActiveVendorList(data=active_vendors))
|
||
+ else:
|
||
+ self.log.info("No active vendors found, vendor list not generated")
|
||
diff --git a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
|
||
index acdb93b..da1e814 100755
|
||
--- a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
|
||
+++ b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
|
||
@@ -8,7 +8,7 @@ fi
|
||
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
|
||
|
||
get_rhel_major_release() {
|
||
- local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*\.' | grep -o '[0-9]*')
|
||
+ local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*' | grep -o '[0-9]*')
|
||
[ -z "$os_version" ] && {
|
||
# This should not happen as /etc/initrd-release is supposed to have API
|
||
# stability, but check is better than broken system.
|
||
@@ -326,4 +326,3 @@ getarg 'rd.break=leapp-logs' && emergency_shell -n upgrade "Break after LEAPP sa
|
||
sync
|
||
mount -o "remount,$old_opts" $NEWROOT
|
||
exit $result
|
||
-
|
||
diff --git a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
|
||
index 14bd6e3..f6adacf 100755
|
||
--- a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
|
||
+++ b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
|
||
@@ -1,7 +1,7 @@
|
||
#!/bin/sh
|
||
|
||
get_rhel_major_release() {
|
||
- local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*\.' | grep -o '[0-9]*')
|
||
+ local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*' | grep -o '[0-9]*')
|
||
[ -z "$os_version" ] && {
|
||
# This should not happen as /etc/initrd-release is supposed to have API
|
||
# stability, but check is better than broken system.
|
||
diff --git a/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py b/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
|
||
index edf978f..7fea4ec 100644
|
||
--- a/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
|
||
+++ b/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
|
||
@@ -47,6 +47,7 @@ def get_os_release(path):
|
||
:return: `OSRelease` model if the file can be parsed
|
||
:raises: `IOError`
|
||
"""
|
||
+ os_version = '.'.join(platform.dist()[1].split('.')[:2])
|
||
try:
|
||
with open(path) as f:
|
||
data = dict(l.strip().split('=', 1) for l in f.readlines() if '=' in l)
|
||
@@ -55,7 +56,7 @@ def get_os_release(path):
|
||
name=data.get('NAME', '').strip('"'),
|
||
pretty_name=data.get('PRETTY_NAME', '').strip('"'),
|
||
version=data.get('VERSION', '').strip('"'),
|
||
- version_id=data.get('VERSION_ID', '').strip('"'),
|
||
+ version_id=os_version,
|
||
variant=data.get('VARIANT', '').strip('"') or None,
|
||
variant_id=data.get('VARIANT_ID', '').strip('"') or None
|
||
)
|
||
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/actor.py b/repos/system_upgrade/common/actors/peseventsscanner/actor.py
|
||
index fadf76b..b86d364 100644
|
||
--- a/repos/system_upgrade/common/actors/peseventsscanner/actor.py
|
||
+++ b/repos/system_upgrade/common/actors/peseventsscanner/actor.py
|
||
@@ -1,3 +1,6 @@
|
||
+import os
|
||
+import os.path
|
||
+
|
||
from leapp.actors import Actor
|
||
from leapp.libraries.actor.peseventsscanner import pes_events_scanner
|
||
from leapp.models import (
|
||
@@ -9,11 +12,15 @@ from leapp.models import (
|
||
RepositoriesMapping,
|
||
RepositoriesSetupTasks,
|
||
RHUIInfo,
|
||
- RpmTransactionTasks
|
||
+ RpmTransactionTasks,
|
||
+ ActiveVendorList,
|
||
)
|
||
from leapp.reporting import Report
|
||
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
|
||
+LEAPP_FILES_DIR = "/etc/leapp/files"
|
||
+VENDORS_DIR = "/etc/leapp/files/vendors.d"
|
||
+
|
||
|
||
class PesEventsScanner(Actor):
|
||
"""
|
||
@@ -32,9 +39,21 @@ class PesEventsScanner(Actor):
|
||
RepositoriesMapping,
|
||
RHUIInfo,
|
||
RpmTransactionTasks,
|
||
+ ActiveVendorList,
|
||
)
|
||
produces = (PESRpmTransactionTasks, RepositoriesSetupTasks, Report)
|
||
tags = (IPUWorkflowTag, FactsPhaseTag)
|
||
|
||
def process(self):
|
||
- pes_events_scanner('/etc/leapp/files', 'pes-events.json')
|
||
+ pes_events_scanner(LEAPP_FILES_DIR, "pes-events.json")
|
||
+
|
||
+ active_vendors = []
|
||
+ for vendor_list in self.consume(ActiveVendorList):
|
||
+ active_vendors.extend(vendor_list.data)
|
||
+
|
||
+ if os.path.isdir(VENDORS_DIR):
|
||
+ vendor_pesfiles = list(filter(lambda vfile: ".json" in vfile, os.listdir(VENDORS_DIR)))
|
||
+
|
||
+ for pesfile in vendor_pesfiles:
|
||
+ if pesfile[:-5] in active_vendors:
|
||
+ pes_events_scanner(VENDORS_DIR, pesfile)
|
||
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner.py b/repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner.py
|
||
index 1be2caa..8e5ca07 100644
|
||
--- a/repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner.py
|
||
+++ b/repos/system_upgrade/common/actors/peseventsscanner/libraries/peseventsscanner.py
|
||
@@ -324,7 +324,7 @@ def parse_pes_events(json_data):
|
||
:return: List of Event tuples, where each event contains event type and input/output pkgs
|
||
"""
|
||
data = json.loads(json_data)
|
||
- if not isinstance(data, dict) or not data.get('packageinfo'):
|
||
+ if not isinstance(data, dict) or data.get('packageinfo') is None:
|
||
raise ValueError('Found PES data with invalid structure')
|
||
|
||
return list(chain(*[parse_entry(entry) for entry in data['packageinfo']]))
|
||
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/tests/unit_test_peseventsscanner.py b/repos/system_upgrade/common/actors/peseventsscanner/tests/unit_test_peseventsscanner.py
|
||
index f4b02e9..c22165e 100644
|
||
--- a/repos/system_upgrade/common/actors/peseventsscanner/tests/unit_test_peseventsscanner.py
|
||
+++ b/repos/system_upgrade/common/actors/peseventsscanner/tests/unit_test_peseventsscanner.py
|
||
@@ -492,6 +492,10 @@ def test_get_events(monkeypatch):
|
||
assert reporting.create_report.called == 1
|
||
assert 'inhibitor' in reporting.create_report.report_fields['flags']
|
||
|
||
+ with open(os.path.join(CUR_DIR, 'files/sample04.json')) as f:
|
||
+ events = parse_pes_events(f.read())
|
||
+ assert len(events) == 0
|
||
+
|
||
|
||
def test_pes_data_not_found(monkeypatch):
|
||
def read_or_fetch_mocked(filename, directory="/etc/leapp/files", service=None, allow_empty=False):
|
||
diff --git a/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py b/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
|
||
index 01f6df3..4ba05f0 100644
|
||
--- a/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
|
||
+++ b/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
|
||
@@ -1,27 +1,52 @@
|
||
from leapp.actors import Actor
|
||
from leapp.libraries.common import rhui
|
||
-from leapp.models import InstalledRedHatSignedRPM, InstalledRPM, InstalledUnsignedRPM
|
||
+from leapp.models import InstalledRedHatSignedRPM, InstalledRPM, InstalledUnsignedRPM, VendorSignatures
|
||
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
|
||
|
||
-class RedHatSignedRpmScanner(Actor):
|
||
+VENDOR_SIGS = {
|
||
+ 'rhel': ['199e2f91fd431d51',
|
||
+ '5326810137017186',
|
||
+ '938a80caf21541eb',
|
||
+ 'fd372689897da07a',
|
||
+ '45689c882fa658e0'],
|
||
+ 'centos': ['24c6a8a7f4a80eb5',
|
||
+ '05b555b38483c65d',
|
||
+ '4eb84e71f2ee9d55'],
|
||
+ 'cloudlinux': ['8c55a6628608cb71']
|
||
+}
|
||
+
|
||
+VENDOR_PACKAGERS = {
|
||
+ "rhel": "Red Hat, Inc.",
|
||
+ "centos": "CentOS",
|
||
+ "cloudlinux": "CloudLinux Packaging Team",
|
||
+}
|
||
+
|
||
+
|
||
+class VendorSignedRpmScanner(Actor):
|
||
"""Provide data about installed RPM Packages signed by Red Hat.
|
||
|
||
After filtering the list of installed RPM packages by signature, a message
|
||
with relevant data will be produced.
|
||
"""
|
||
|
||
- name = 'red_hat_signed_rpm_scanner'
|
||
- consumes = (InstalledRPM,)
|
||
- produces = (InstalledRedHatSignedRPM, InstalledUnsignedRPM,)
|
||
+ name = "vendor_signed_rpm_scanner"
|
||
+ consumes = (InstalledRPM, VendorSignatures)
|
||
+ produces = (
|
||
+ InstalledRedHatSignedRPM,
|
||
+ InstalledUnsignedRPM,
|
||
+ )
|
||
tags = (IPUWorkflowTag, FactsPhaseTag)
|
||
|
||
def process(self):
|
||
- RH_SIGS = ['199e2f91fd431d51',
|
||
- '5326810137017186',
|
||
- '938a80caf21541eb',
|
||
- 'fd372689897da07a',
|
||
- '45689c882fa658e0']
|
||
+ vendor = self.configuration.os_release.release_id
|
||
+ vendor_keys = sum(VENDOR_SIGS.values(), [])
|
||
+ vendor_packager = VENDOR_PACKAGERS.get(vendor, "not-available")
|
||
+
|
||
+ for siglist in self.consume(VendorSignatures):
|
||
+ vendor_keys.extend(siglist.sigs)
|
||
+
|
||
+ self.log.debug("Signature list: {}".format(vendor_keys))
|
||
|
||
signed_pkgs = InstalledRedHatSignedRPM()
|
||
unsigned_pkgs = InstalledUnsignedRPM()
|
||
@@ -32,11 +57,11 @@ class RedHatSignedRpmScanner(Actor):
|
||
all_signed = [
|
||
env
|
||
for env in env_vars
|
||
- if env.name == 'LEAPP_DEVEL_RPMS_ALL_SIGNED' and env.value == '1'
|
||
+ if env.name == "LEAPP_DEVEL_RPMS_ALL_SIGNED" and env.value == "1"
|
||
]
|
||
|
||
- def has_rhsig(pkg):
|
||
- return any(key in pkg.pgpsig for key in RH_SIGS)
|
||
+ def has_vendorsig(pkg):
|
||
+ return any(key in pkg.pgpsig for key in vendor_keys)
|
||
|
||
def is_gpg_pubkey(pkg):
|
||
"""Check if gpg-pubkey pkg exists or LEAPP_DEVEL_RPMS_ALL_SIGNED=1
|
||
@@ -44,15 +69,15 @@ class RedHatSignedRpmScanner(Actor):
|
||
gpg-pubkey is not signed as it would require another package
|
||
to verify its signature
|
||
"""
|
||
- return ( # pylint: disable-msg=consider-using-ternary
|
||
- pkg.name == 'gpg-pubkey'
|
||
- and pkg.packager.startswith('Red Hat, Inc.')
|
||
- or all_signed
|
||
+ return ( # pylint: disable-msg=consider-using-ternary
|
||
+ pkg.name == "gpg-pubkey"
|
||
+ and (pkg.packager.startswith(vendor_packager))
|
||
+ or all_signed
|
||
)
|
||
|
||
def has_katello_prefix(pkg):
|
||
"""Whitelist the katello package."""
|
||
- return pkg.name.startswith('katello-ca-consumer')
|
||
+ return pkg.name.startswith("katello-ca-consumer")
|
||
|
||
def is_azure_pkg(pkg):
|
||
"""Whitelist Azure config package."""
|
||
@@ -68,16 +93,24 @@ class RedHatSignedRpmScanner(Actor):
|
||
for pkg in rpm_pkgs.items:
|
||
if any(
|
||
[
|
||
- has_rhsig(pkg),
|
||
+ has_vendorsig(pkg),
|
||
is_gpg_pubkey(pkg),
|
||
has_katello_prefix(pkg),
|
||
is_azure_pkg(pkg),
|
||
]
|
||
):
|
||
signed_pkgs.items.append(pkg)
|
||
+ self.log.debug(
|
||
+ "Package {} is signed, packager: {}, signature: {}".format(
|
||
+ pkg.name, pkg.packager, pkg.pgpsig
|
||
+ )
|
||
+ )
|
||
continue
|
||
|
||
unsigned_pkgs.items.append(pkg)
|
||
+ self.log.debug(
|
||
+ "Package {} is unsigned, packager: {}, signature: {}".format(pkg.name, pkg.packager, pkg.pgpsig)
|
||
+ )
|
||
|
||
self.produce(signed_pkgs)
|
||
self.produce(unsigned_pkgs)
|
||
diff --git a/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py b/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
|
||
index 3c0b04b..3480432 100644
|
||
--- a/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
|
||
+++ b/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
|
||
@@ -15,7 +15,6 @@ from leapp.models import PESIDRepositoryEntry
|
||
|
||
CUR_DIR = os.path.dirname(os.path.abspath(__file__))
|
||
|
||
-
|
||
@pytest.fixture
|
||
def adjust_cwd():
|
||
previous_cwd = os.getcwd()
|
||
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py
|
||
new file mode 100644
|
||
index 0000000..6c7f3a3
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py
|
||
@@ -0,0 +1,20 @@
|
||
+from leapp.actors import Actor
|
||
+from leapp.libraries.actor import scanvendorrepofiles
|
||
+from leapp.models import CustomTargetRepository, CustomTargetRepositoryFile, ActiveVendorList
|
||
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
+from leapp.libraries.stdlib import api
|
||
+
|
||
+
|
||
+class ScanVendorRepofiles(Actor):
|
||
+ """
|
||
+ Load and produce custom repository data from vendor-provided files.
|
||
+ Only those vendors whose source system repoids were found on the system will be included.
|
||
+ """
|
||
+
|
||
+ name = "scan_vendor_repofiles"
|
||
+ consumes = (ActiveVendorList)
|
||
+ produces = (CustomTargetRepository, CustomTargetRepositoryFile)
|
||
+ tags = (FactsPhaseTag, IPUWorkflowTag)
|
||
+
|
||
+ def process(self):
|
||
+ scanvendorrepofiles.process()
|
||
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py
|
||
new file mode 100644
|
||
index 0000000..5a65f5a
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py
|
||
@@ -0,0 +1,68 @@
|
||
+import os
|
||
+
|
||
+from leapp.libraries.common import repofileutils
|
||
+from leapp.libraries.stdlib import api
|
||
+from leapp.models import CustomTargetRepository, CustomTargetRepositoryFile, ActiveVendorList
|
||
+
|
||
+
|
||
+VENDORS_DIR = "/etc/leapp/files/vendors.d/"
|
||
+REPOFILE_SUFFIX = ".repo"
|
||
+
|
||
+
|
||
+def process():
|
||
+ """
|
||
+ Produce CustomTargetRepository msgs for the vendor repo files inside the
|
||
+ <CUSTOM_REPO_DIR>.
|
||
+
|
||
+ The CustomTargetRepository messages are produced only if a "from" vendor repository
|
||
+ listed indide its map matched one of the repositories active on the system.
|
||
+ """
|
||
+ if not os.path.isdir(VENDORS_DIR):
|
||
+ api.current_logger().debug(
|
||
+ "The {} directory doesn't exist. Nothing to do.".format(VENDORS_DIR)
|
||
+ )
|
||
+ return
|
||
+
|
||
+ for reponame in os.listdir(VENDORS_DIR):
|
||
+ if not reponame.endswith(REPOFILE_SUFFIX):
|
||
+ continue
|
||
+ # Cut the .repo part to get only the name.
|
||
+ vendor_name = reponame[:-5]
|
||
+
|
||
+ active_vendors = []
|
||
+ for vendor_list in api.consume(ActiveVendorList):
|
||
+ active_vendors.extend(vendor_list.data)
|
||
+
|
||
+ api.current_logger().debug(
|
||
+ "Active vendor list: {}".format(active_vendors)
|
||
+ )
|
||
+
|
||
+ if vendor_name not in active_vendors:
|
||
+ api.current_logger().debug(
|
||
+ "Vendor {} not in active list, skipping".format(vendor_name)
|
||
+ )
|
||
+ continue
|
||
+
|
||
+ api.current_logger().debug(
|
||
+ "Vendor {} found in active list, processing file".format(vendor_name)
|
||
+ )
|
||
+ full_repo_path = os.path.join(VENDORS_DIR, reponame)
|
||
+ repofile = repofileutils.parse_repofile(full_repo_path)
|
||
+
|
||
+ api.produce(CustomTargetRepositoryFile(file=full_repo_path))
|
||
+ for repo in repofile.data:
|
||
+ api.current_logger().debug(
|
||
+ "Loaded repository {} from file {}".format(repo.repoid, reponame)
|
||
+ )
|
||
+ api.produce(
|
||
+ CustomTargetRepository(
|
||
+ repoid=repo.repoid,
|
||
+ name=repo.name,
|
||
+ baseurl=repo.baseurl,
|
||
+ enabled=repo.enabled,
|
||
+ )
|
||
+ )
|
||
+
|
||
+ api.current_logger().info(
|
||
+ "The {} directory exists, vendor repositories loaded.".format(VENDORS_DIR)
|
||
+ )
|
||
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py
|
||
new file mode 100644
|
||
index 0000000..cb5c7ab
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py
|
||
@@ -0,0 +1,131 @@
|
||
+import os
|
||
+
|
||
+from leapp.libraries.actor import scancustomrepofile
|
||
+from leapp.libraries.common import repofileutils
|
||
+from leapp.libraries.common.testutils import produce_mocked
|
||
+from leapp.libraries.stdlib import api
|
||
+
|
||
+from leapp.models import (CustomTargetRepository, CustomTargetRepositoryFile,
|
||
+ RepositoryData, RepositoryFile)
|
||
+
|
||
+
|
||
+_REPODATA = [
|
||
+ RepositoryData(repoid="repo1", name="repo1name", baseurl="repo1url", enabled=True),
|
||
+ RepositoryData(repoid="repo2", name="repo2name", baseurl="repo2url", enabled=False),
|
||
+ RepositoryData(repoid="repo3", name="repo3name", enabled=True),
|
||
+ RepositoryData(repoid="repo4", name="repo4name", mirrorlist="mirror4list", enabled=True),
|
||
+]
|
||
+
|
||
+_CUSTOM_REPOS = [
|
||
+ CustomTargetRepository(repoid="repo1", name="repo1name", baseurl="repo1url", enabled=True),
|
||
+ CustomTargetRepository(repoid="repo2", name="repo2name", baseurl="repo2url", enabled=False),
|
||
+ CustomTargetRepository(repoid="repo3", name="repo3name", baseurl=None, enabled=True),
|
||
+ CustomTargetRepository(repoid="repo4", name="repo4name", baseurl=None, enabled=True),
|
||
+]
|
||
+
|
||
+_CUSTOM_REPO_FILE_MSG = CustomTargetRepositoryFile(file=scancustomrepofile.CUSTOM_REPO_PATH)
|
||
+
|
||
+
|
||
+_TESTING_REPODATA = [
|
||
+ RepositoryData(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
|
||
+ RepositoryData(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=False),
|
||
+ RepositoryData(repoid="repo3-stable", name="repo3name", enabled=False),
|
||
+ RepositoryData(repoid="repo4-testing", name="repo4name", mirrorlist="mirror4list", enabled=True),
|
||
+]
|
||
+
|
||
+_TESTING_CUSTOM_REPOS_STABLE_TARGET = [
|
||
+ CustomTargetRepository(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
|
||
+ CustomTargetRepository(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=False),
|
||
+ CustomTargetRepository(repoid="repo3-stable", name="repo3name", baseurl=None, enabled=False),
|
||
+ CustomTargetRepository(repoid="repo4-testing", name="repo4name", baseurl=None, enabled=True),
|
||
+]
|
||
+
|
||
+_TESTING_CUSTOM_REPOS_BETA_TARGET = [
|
||
+ CustomTargetRepository(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
|
||
+ CustomTargetRepository(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=True),
|
||
+ CustomTargetRepository(repoid="repo3-stable", name="repo3name", baseurl=None, enabled=False),
|
||
+ CustomTargetRepository(repoid="repo4-testing", name="repo4name", baseurl=None, enabled=True),
|
||
+]
|
||
+
|
||
+_PROCESS_STABLE_TARGET = "stable"
|
||
+_PROCESS_BETA_TARGET = "beta"
|
||
+
|
||
+
|
||
+class LoggerMocked(object):
|
||
+ def __init__(self):
|
||
+ self.infomsg = None
|
||
+ self.debugmsg = None
|
||
+
|
||
+ def info(self, msg):
|
||
+ self.infomsg = msg
|
||
+
|
||
+ def debug(self, msg):
|
||
+ self.debugmsg = msg
|
||
+
|
||
+ def __call__(self):
|
||
+ return self
|
||
+
|
||
+
|
||
+def test_no_repofile(monkeypatch):
|
||
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: False)
|
||
+ monkeypatch.setattr(api, 'produce', produce_mocked())
|
||
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
|
||
+ scancustomrepofile.process()
|
||
+ msg = "The {} file doesn't exist. Nothing to do.".format(scancustomrepofile.CUSTOM_REPO_PATH)
|
||
+ assert api.current_logger.debugmsg == msg
|
||
+ assert not api.produce.called
|
||
+
|
||
+
|
||
+def test_valid_repofile_exists(monkeypatch):
|
||
+ def _mocked_parse_repofile(fpath):
|
||
+ return RepositoryFile(file=fpath, data=_REPODATA)
|
||
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
|
||
+ monkeypatch.setattr(api, 'produce', produce_mocked())
|
||
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
|
||
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
|
||
+ scancustomrepofile.process()
|
||
+ msg = "The {} file exists, custom repositories loaded.".format(scancustomrepofile.CUSTOM_REPO_PATH)
|
||
+ assert api.current_logger.infomsg == msg
|
||
+ assert api.produce.called == len(_CUSTOM_REPOS) + 1
|
||
+ assert _CUSTOM_REPO_FILE_MSG in api.produce.model_instances
|
||
+ for crepo in _CUSTOM_REPOS:
|
||
+ assert crepo in api.produce.model_instances
|
||
+
|
||
+
|
||
+def test_target_stable_repos(monkeypatch):
|
||
+ def _mocked_parse_repofile(fpath):
|
||
+ return RepositoryFile(file=fpath, data=_TESTING_REPODATA)
|
||
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
|
||
+ monkeypatch.setattr(api, 'produce', produce_mocked())
|
||
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
|
||
+
|
||
+ scancustomrepofile.process(_PROCESS_STABLE_TARGET)
|
||
+ assert api.produce.called == len(_TESTING_CUSTOM_REPOS_STABLE_TARGET) + 1
|
||
+ for crepo in _TESTING_CUSTOM_REPOS_STABLE_TARGET:
|
||
+ assert crepo in api.produce.model_instances
|
||
+
|
||
+
|
||
+def test_target_beta_repos(monkeypatch):
|
||
+ def _mocked_parse_repofile(fpath):
|
||
+ return RepositoryFile(file=fpath, data=_TESTING_REPODATA)
|
||
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
|
||
+ monkeypatch.setattr(api, 'produce', produce_mocked())
|
||
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
|
||
+
|
||
+ scancustomrepofile.process(_PROCESS_BETA_TARGET)
|
||
+ assert api.produce.called == len(_TESTING_CUSTOM_REPOS_BETA_TARGET) + 1
|
||
+ for crepo in _TESTING_CUSTOM_REPOS_BETA_TARGET:
|
||
+ assert crepo in api.produce.model_instances
|
||
+
|
||
+
|
||
+def test_empty_repofile_exists(monkeypatch):
|
||
+ def _mocked_parse_repofile(fpath):
|
||
+ return RepositoryFile(file=fpath, data=[])
|
||
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
|
||
+ monkeypatch.setattr(api, 'produce', produce_mocked())
|
||
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
|
||
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
|
||
+ scancustomrepofile.process()
|
||
+ msg = "The {} file exists, but is empty. Nothing to do.".format(scancustomrepofile.CUSTOM_REPO_PATH)
|
||
+ assert api.current_logger.infomsg == msg
|
||
+ assert not api.produce.called
|
||
diff --git a/repos/system_upgrade/common/actors/setuptargetrepos/actor.py b/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
|
||
index 00de073..fb86639 100644
|
||
--- a/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
|
||
+++ b/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
|
||
@@ -12,6 +12,7 @@ from leapp.models import (
|
||
UsedRepositories
|
||
)
|
||
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
+from leapp.libraries.stdlib import api
|
||
|
||
|
||
class SetupTargetRepos(Actor):
|
||
diff --git a/repos/system_upgrade/common/actors/systemfacts/actor.py b/repos/system_upgrade/common/actors/systemfacts/actor.py
|
||
index 59b12c8..85d4a09 100644
|
||
--- a/repos/system_upgrade/common/actors/systemfacts/actor.py
|
||
+++ b/repos/system_upgrade/common/actors/systemfacts/actor.py
|
||
@@ -47,7 +47,7 @@ class SystemFactsActor(Actor):
|
||
GrubCfgBios,
|
||
Report
|
||
)
|
||
- tags = (IPUWorkflowTag, FactsPhaseTag,)
|
||
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before,)
|
||
|
||
def process(self):
|
||
self.produce(systemfacts.get_sysctls_status())
|
||
diff --git a/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py b/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
|
||
index 7a8bd99..f59c909 100644
|
||
--- a/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
|
||
+++ b/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
|
||
@@ -592,6 +592,7 @@ def _install_custom_repofiles(context, custom_repofiles):
|
||
"""
|
||
for rfile in custom_repofiles:
|
||
_dst_path = os.path.join('/etc/yum.repos.d', os.path.basename(rfile.file))
|
||
+ api.current_logger().debug("Copying {} to {}".format(rfile.file, _dst_path))
|
||
context.copy_to(rfile.file, _dst_path)
|
||
|
||
|
||
diff --git a/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py b/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py
|
||
new file mode 100644
|
||
index 0000000..f74de27
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py
|
||
@@ -0,0 +1,70 @@
|
||
+import os
|
||
+
|
||
+from leapp.actors import Actor
|
||
+from leapp.models import VendorSignatures, ActiveVendorList
|
||
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
+
|
||
+
|
||
+VENDORS_DIR = "/etc/leapp/files/vendors.d/"
|
||
+SIGFILE_SUFFIX = ".sigs"
|
||
+
|
||
+
|
||
+class VendorRepoSignatureScanner(Actor):
|
||
+ """
|
||
+ Produce VendorSignatures msgs for the vendor signature files inside the
|
||
+ <VENDORS_DIR>.
|
||
+
|
||
+ The messages are produced only if a "from" vendor repository
|
||
+ listed indide its map matched one of the repositories active on the system.
|
||
+ """
|
||
+
|
||
+ name = 'vendor_repo_signature_scanner'
|
||
+ consumes = (ActiveVendorList)
|
||
+ produces = (VendorSignatures)
|
||
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
|
||
+
|
||
+ def process(self):
|
||
+ if not os.path.isdir(VENDORS_DIR):
|
||
+ self.log.debug(
|
||
+ "The {} directory doesn't exist. Nothing to do.".format(VENDORS_DIR)
|
||
+ )
|
||
+ return
|
||
+
|
||
+ for sigfile_name in os.listdir(VENDORS_DIR):
|
||
+ if not sigfile_name.endswith(SIGFILE_SUFFIX):
|
||
+ continue
|
||
+ # Cut the suffix part to get only the name.
|
||
+ vendor_name = sigfile_name[:-5]
|
||
+
|
||
+ active_vendors = []
|
||
+ for vendor_list in self.consume(ActiveVendorList):
|
||
+ active_vendors.extend(vendor_list.data)
|
||
+
|
||
+ self.log.debug(
|
||
+ "Active vendor list: {}".format(active_vendors)
|
||
+ )
|
||
+
|
||
+ if vendor_name not in active_vendors:
|
||
+ self.log.debug(
|
||
+ "Vendor {} not in active list, skipping".format(vendor_name)
|
||
+ )
|
||
+ continue
|
||
+
|
||
+ self.log.debug(
|
||
+ "Vendor {} found in active list, processing file".format(vendor_name)
|
||
+ )
|
||
+
|
||
+ full_sigfile_path = os.path.join(VENDORS_DIR, sigfile_name)
|
||
+ with open(full_sigfile_path) as f:
|
||
+ signatures = [line for line in f.read().splitlines() if line]
|
||
+
|
||
+ self.produce(
|
||
+ VendorSignatures(
|
||
+ vendor=vendor_name,
|
||
+ sigs=signatures,
|
||
+ )
|
||
+ )
|
||
+
|
||
+ self.log.info(
|
||
+ "The {} directory exists, vendor signatures loaded.".format(VENDORS_DIR)
|
||
+ )
|
||
diff --git a/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py
|
||
new file mode 100644
|
||
index 0000000..156d78c
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py
|
||
@@ -0,0 +1,22 @@
|
||
+from leapp.actors import Actor
|
||
+from leapp.libraries.common.repomaputils import scan_vendor_repomaps, VENDOR_REPOMAP_DIR
|
||
+from leapp.models import VendorRepositoriesMapCollection, RepositoriesMap
|
||
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
|
||
+
|
||
+
|
||
+class VendorRepositoriesMapping(Actor):
|
||
+ """
|
||
+ Scan the vendor repository mapping files and provide the data to other actors.
|
||
+ """
|
||
+
|
||
+ name = "vendor_repositories_mapping"
|
||
+ consumes = ()
|
||
+ produces = (RepositoriesMap, VendorRepositoriesMapCollection,)
|
||
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
|
||
+
|
||
+ def process(self):
|
||
+ vendor_repomap_collection = scan_vendor_repomaps(VENDOR_REPOMAP_DIR)
|
||
+ if vendor_repomap_collection:
|
||
+ self.produce(vendor_repomap_collection)
|
||
+ for repomap in vendor_repomap_collection.maps:
|
||
+ self.produce(repomap)
|
||
diff --git a/repos/system_upgrade/common/libraries/config/version.py b/repos/system_upgrade/common/libraries/config/version.py
|
||
index 03f3cd4..783075d 100644
|
||
--- a/repos/system_upgrade/common/libraries/config/version.py
|
||
+++ b/repos/system_upgrade/common/libraries/config/version.py
|
||
@@ -13,7 +13,7 @@ OP_MAP = {
|
||
|
||
_SUPPORTED_VERSIONS = {
|
||
# Note: 'rhel-alt' is detected when on 'rhel' with kernel 4.x
|
||
- '7': {'rhel': ['7.9'], 'rhel-alt': ['7.6'], 'rhel-saphana': ['7.9']},
|
||
+ '7': {'rhel': ['7.9'], 'rhel-alt': ['7.6'], 'rhel-saphana': ['7.9'], 'centos': ['7.9']},
|
||
'8': {'rhel': ['8.5', '8.6']},
|
||
}
|
||
|
||
diff --git a/repos/system_upgrade/common/libraries/dnfplugin.py b/repos/system_upgrade/common/libraries/dnfplugin.py
|
||
index 4010e9f..00323a7 100644
|
||
--- a/repos/system_upgrade/common/libraries/dnfplugin.py
|
||
+++ b/repos/system_upgrade/common/libraries/dnfplugin.py
|
||
@@ -4,6 +4,8 @@ import json
|
||
import os
|
||
import shutil
|
||
|
||
+import six
|
||
+
|
||
from leapp.exceptions import StopActorExecutionError
|
||
from leapp.libraries.common import dnfconfig, guards, mounting, overlaygen, rhsm, utils
|
||
from leapp.libraries.common.config.version import get_target_major_version, get_target_version
|
||
@@ -213,10 +215,15 @@ def _transaction(context, stage, target_repoids, tasks, plugin_info, test=False,
|
||
message='Failed to execute dnf. Reason: {}'.format(str(e))
|
||
)
|
||
except CalledProcessError as e:
|
||
+ if six.PY2:
|
||
+ e.stdout = e.stdout.encode('utf-8', 'xmlcharrefreplace')
|
||
+ e.stderr = e.stdout.encode('utf-8', 'xmlcharrefreplace')
|
||
+
|
||
api.current_logger().error('DNF execution failed: ')
|
||
raise StopActorExecutionError(
|
||
message='DNF execution failed with non zero exit code.\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}'.format(
|
||
- stdout=e.stdout, stderr=e.stderr)
|
||
+ stdout=e.stdout, stderr=e.stderr
|
||
+ )
|
||
)
|
||
finally:
|
||
if stage == 'check':
|
||
diff --git a/repos/system_upgrade/common/libraries/fetch.py b/repos/system_upgrade/common/libraries/fetch.py
|
||
index 1c58148..37313b6 100644
|
||
--- a/repos/system_upgrade/common/libraries/fetch.py
|
||
+++ b/repos/system_upgrade/common/libraries/fetch.py
|
||
@@ -73,7 +73,7 @@ def read_or_fetch(filename, directory="/etc/leapp/files", service=None, allow_em
|
||
data = f.read()
|
||
if not allow_empty and not data:
|
||
_raise_error(local_path, "File {lp} exists but is empty".format(lp=local_path))
|
||
- logger.warning("File {lp} successfully read ({l} bytes)".format(lp=local_path, l=len(data)))
|
||
+ logger.debug("File {lp} successfully read ({l} bytes)".format(lp=local_path, l=len(data)))
|
||
return data
|
||
except EnvironmentError:
|
||
_raise_error(local_path, "File {lp} exists but couldn't be read".format(lp=local_path))
|
||
diff --git a/repos/system_upgrade/common/libraries/repomaputils.py b/repos/system_upgrade/common/libraries/repomaputils.py
|
||
new file mode 100644
|
||
index 0000000..7ca63d2
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/libraries/repomaputils.py
|
||
@@ -0,0 +1,154 @@
|
||
+import os
|
||
+import io # Python2/Python3 compatible IO (open etc.)
|
||
+
|
||
+from leapp.exceptions import StopActorExecutionError
|
||
+from leapp.libraries.common import config
|
||
+from leapp.libraries.common.fetch import read_or_fetch
|
||
+from leapp.libraries.stdlib import api
|
||
+from leapp.models import RepositoriesMap, RepositoryMap, VendorRepositoriesMapCollection
|
||
+from leapp.models.fields import ModelViolationError
|
||
+
|
||
+REPOMAP_FILE = "repomap.csv"
|
||
+"""Path to the repository mapping file."""
|
||
+BASE_REPOMAP_DIR = "/etc/leapp/files"
|
||
+VENDOR_REPOMAP_DIR = "/etc/leapp/files/vendors.d"
|
||
+
|
||
+
|
||
+def _raise_error(msg, details):
|
||
+ raise StopActorExecutionError(
|
||
+ msg,
|
||
+ details={
|
||
+ "details": details,
|
||
+ "hint": (
|
||
+ "Read documentation at the following link for more"
|
||
+ " information about how to retrieve the valid file:"
|
||
+ " https://access.redhat.com/articles/3664871"
|
||
+ ),
|
||
+ },
|
||
+ )
|
||
+
|
||
+
|
||
+def read_local(
|
||
+ filename,
|
||
+ directory=BASE_REPOMAP_DIR,
|
||
+ allow_empty=False,
|
||
+ encoding="utf-8",
|
||
+):
|
||
+ logger = api.current_logger()
|
||
+ local_path = os.path.join(directory, filename)
|
||
+ try:
|
||
+ with io.open(local_path, encoding=encoding) as f:
|
||
+ data = f.read()
|
||
+ if not allow_empty and not data:
|
||
+ _raise_error(
|
||
+ local_path, "File {} exists but is empty".format(local_path)
|
||
+ )
|
||
+ logger.warning(
|
||
+ "File {lp} successfully read ({l} bytes)".format(
|
||
+ lp=local_path, l=len(data)
|
||
+ )
|
||
+ )
|
||
+ return [line.strip() for line in data.splitlines()]
|
||
+ except EnvironmentError:
|
||
+ _raise_error(
|
||
+ local_path, "File {} exists but couldn't be read".format(local_path)
|
||
+ )
|
||
+ except Exception as e:
|
||
+ raise e
|
||
+
|
||
+
|
||
+def read_or_fetch_repofile(repofile, directory):
|
||
+ contents = read_or_fetch(repofile, directory)
|
||
+ return [line.strip() for line in contents.splitlines()]
|
||
+
|
||
+
|
||
+def scan_repomaps(repomap_file, repomap_dir, read_repofile_func=read_or_fetch_repofile):
|
||
+ """
|
||
+ Scan the repository mapping file and produce RepositoriesMap msg.
|
||
+
|
||
+ See the description of the actor for more details.
|
||
+ """
|
||
+ _exp_src_prod_type = config.get_product_type("source")
|
||
+ _exp_dst_prod_type = config.get_product_type("target")
|
||
+
|
||
+ repositories = []
|
||
+ line_num = 0
|
||
+ for line in read_repofile_func(repomap_file, repomap_dir)[1:]:
|
||
+ line_num += 1
|
||
+
|
||
+ api.current_logger().debug("Grabbing line {} of file {}: \"{}\"".format(line_num, repomap_file, line))
|
||
+
|
||
+ # skip empty lines and comments
|
||
+ if not line or line.startswith("#"):
|
||
+ api.current_logger().debug("Line skipped")
|
||
+ continue
|
||
+
|
||
+ try:
|
||
+ (
|
||
+ from_repoid,
|
||
+ to_repoid,
|
||
+ to_pes_repo,
|
||
+ from_minor_version,
|
||
+ to_minor_version,
|
||
+ arch,
|
||
+ repo_type,
|
||
+ src_prod_type,
|
||
+ dst_prod_type,
|
||
+ ) = line.split(",")
|
||
+
|
||
+ # filter out records irrelevant for this run
|
||
+ if (
|
||
+ arch != api.current_actor().configuration.architecture
|
||
+ or _exp_src_prod_type != src_prod_type
|
||
+ or _exp_dst_prod_type != dst_prod_type
|
||
+ ):
|
||
+ api.current_logger().debug("Line filtered out")
|
||
+ continue
|
||
+
|
||
+ new_repo_map = RepositoryMap(
|
||
+ from_repoid=from_repoid,
|
||
+ to_repoid=to_repoid,
|
||
+ to_pes_repo=to_pes_repo,
|
||
+ from_minor_version=from_minor_version,
|
||
+ to_minor_version=to_minor_version,
|
||
+ arch=arch,
|
||
+ repo_type=repo_type,
|
||
+ )
|
||
+
|
||
+ api.current_logger().debug("Map added: {}".format(new_repo_map.dump()))
|
||
+ repositories.append(new_repo_map)
|
||
+
|
||
+ except (ModelViolationError, ValueError) as err:
|
||
+ _raise_error(
|
||
+ "The repository mapping file is invalid. It is possible the file is out of date.",
|
||
+ "Offending line number: {} ({}).".format(line_num, err),
|
||
+ )
|
||
+
|
||
+ if not repositories:
|
||
+ _raise_error(
|
||
+ "The repository mapping file is invalid. Could not find any repository mapping record.",
|
||
+ "",
|
||
+ )
|
||
+
|
||
+ return RepositoriesMap(file=repomap_file, repositories=repositories)
|
||
+
|
||
+
|
||
+def scan_vendor_repomaps(repomap_dir):
|
||
+ if not os.path.isdir(repomap_dir):
|
||
+ api.current_logger().debug(
|
||
+ "The {} directory doesn't exist. Nothing to do.".format(repomap_dir)
|
||
+ )
|
||
+ return None
|
||
+
|
||
+ vendor_maps = []
|
||
+
|
||
+ for repomap_name in os.listdir(repomap_dir):
|
||
+ # Only scan the .csv files, those are the maps.
|
||
+ if not repomap_name.endswith(".csv"):
|
||
+ continue
|
||
+ scanned_map = scan_repomaps(
|
||
+ repomap_name, repomap_dir, read_repofile_func=read_local
|
||
+ )
|
||
+ vendor_maps.append(scanned_map)
|
||
+
|
||
+ return VendorRepositoriesMapCollection(maps=vendor_maps)
|
||
diff --git a/repos/system_upgrade/common/libraries/rhsm.py b/repos/system_upgrade/common/libraries/rhsm.py
|
||
index b7e4b21..dc038bf 100644
|
||
--- a/repos/system_upgrade/common/libraries/rhsm.py
|
||
+++ b/repos/system_upgrade/common/libraries/rhsm.py
|
||
@@ -92,7 +92,7 @@ def _handle_rhsm_exceptions(hint=None):
|
||
|
||
def skip_rhsm():
|
||
"""Check whether we should skip RHSM related code."""
|
||
- return get_env('LEAPP_NO_RHSM', '0') == '1'
|
||
+ return True
|
||
|
||
|
||
def with_rhsm(f):
|
||
diff --git a/repos/system_upgrade/common/models/activevendorlist.py b/repos/system_upgrade/common/models/activevendorlist.py
|
||
new file mode 100644
|
||
index 0000000..de4056f
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/models/activevendorlist.py
|
||
@@ -0,0 +1,7 @@
|
||
+from leapp.models import Model, fields
|
||
+from leapp.topics import VendorTopic
|
||
+
|
||
+
|
||
+class ActiveVendorList(Model):
|
||
+ topic = VendorTopic
|
||
+ data = fields.List(fields.String())
|
||
diff --git a/repos/system_upgrade/common/models/repositoriesmap.py b/repos/system_upgrade/common/models/repositoriesmap.py
|
||
index c187333..f5f23f4 100644
|
||
--- a/repos/system_upgrade/common/models/repositoriesmap.py
|
||
+++ b/repos/system_upgrade/common/models/repositoriesmap.py
|
||
@@ -92,3 +92,10 @@ class RepositoriesMapping(Model):
|
||
|
||
mapping = fields.List(fields.Model(RepoMapEntry), default=[])
|
||
repositories = fields.List(fields.Model(PESIDRepositoryEntry), default=[])
|
||
+ file = fields.String(default="repomap.csv")
|
||
+
|
||
+
|
||
+class VendorRepositoriesMapCollection(Model):
|
||
+ topic = TransactionTopic
|
||
+
|
||
+ maps = fields.List(fields.Model(RepositoriesMapping))
|
||
diff --git a/repos/system_upgrade/common/models/vendorsignatures.py b/repos/system_upgrade/common/models/vendorsignatures.py
|
||
new file mode 100644
|
||
index 0000000..f456aec
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/models/vendorsignatures.py
|
||
@@ -0,0 +1,8 @@
|
||
+from leapp.models import Model, fields
|
||
+from leapp.topics import VendorTopic
|
||
+
|
||
+
|
||
+class VendorSignatures(Model):
|
||
+ topic = VendorTopic
|
||
+ vendor = fields.String()
|
||
+ sigs = fields.List(fields.String())
|
||
diff --git a/repos/system_upgrade/common/topics/vendortopic.py b/repos/system_upgrade/common/topics/vendortopic.py
|
||
new file mode 100644
|
||
index 0000000..014b7af
|
||
--- /dev/null
|
||
+++ b/repos/system_upgrade/common/topics/vendortopic.py
|
||
@@ -0,0 +1,5 @@
|
||
+from leapp.topics import Topic
|
||
+
|
||
+
|
||
+class VendorTopic(Topic):
|
||
+ name = 'vendor_topic'
|