rpms
/
leapp-repository
7
0
Fork 1
Code Issues Pull Requests Releases Activity
leapp-repository/SOURCES/leapp-repository-0.17.0-ele...

3903 lines
170 KiB
Diff
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

diff --git a/README.md b/README.md
index 7d50964..c3bf594 100644
--- a/README.md
+++ b/README.md
@@ -1,7 +1,61 @@
-**Before doing anything, please read
-[Leapp framework documentation](https://leapp.readthedocs.io/).**
+# Leapp ELevate Repository
----
+**Before doing anything, please read [Leapp framework documentation](https://leapp.readthedocs.io/).**
+
+## Running
+Make sure your system is fully updated before starting the upgrade process.
+
+```bash
+sudo yum update -y
+```
+
+Install `elevate-release` package with the project repo and GPG key.
+
+`sudo yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el7.noarch.rpm`
+
+Install leapp packages and migration data for the OS you want to upgrade. Possible options are:
+ - leapp-data-almalinux
+ - leapp-data-centos
+ - leapp-data-eurolinux
+ - leapp-data-oraclelinux
+ - leapp-data-rocky
+
+`sudo yum install -y leapp-upgrade leapp-data-almalinux`
+
+Start a preupgrade check. In the meantime, the Leapp utility creates a special /var/log/leapp/leapp-report.txt file that contains possible problems and recommended solutions. No rpm packages will be installed at this phase.
+
+`sudo leapp preupgrade`
+
+The preupgrade process may stall with the following message:
+> Inhibitor: Newest installed kernel not in use
+
+Make sure your system is running the latest kernel before proceeding with the upgrade. If you updated the system recently, a reboot may be sufficient to do so. Otherwise, edit your Grub configuration accordingly.
+
+> NOTE: In certain configurations, Leapp generates `/var/log/leapp/answerfile` with true/false questions. Leapp utility requires answers to all these questions in order to proceed with the upgrade.
+
+Once the preupgrade process completes, the results will be contained in `/var/log/leapp/leapp-report.txt` file.
+It's advised to review the report and consider how the changes will affect your system.
+
+Start an upgrade. You’ll be offered to reboot the system after this process is completed.
+
+```bash
+sudo leapp upgrade
+sudo reboot
+```
+
+> NOTE: The upgrade process after the reboot may take a long time, up to 40-50 minutes, depending on the machine resources. If the machine remains unresponsive for more than 2 hours, assume the upgrade process failed during the post-reboot phase.
+> If it's still possible to access the machine in some way, for example, through remote VNC access, the logs containing the information on what went wrong are located in this folder: `/var/log/leapp`
+
+A new entry in GRUB called ELevate-Upgrade-Initramfs will appear. The system will be automatically booted into it. Observe the update process in the console.
+
+After the reboot, login into the system and check the migration report. Verify that the current OS is the one you need.
+
+```bash
+cat /etc/redhat-release
+cat /etc/os-release
+```
+
+Check the leapp logs for .rpmnew configuration files that may have been created during the upgrade process. In some cases os-release or yum package files may not be replaced automatically, requiring the user to rename the .rpmnew files manually.
## Troubleshooting
@@ -11,6 +65,15 @@
- Leapp framework: [https://github.com/oamg/leapp/issues/new/choose](https://github.com/oamg/leapp/issues/new/choose)
- Leapp actors: [https://github.com/oamg/leapp-repository/issues/new/choose](https://github.com/oamg/leapp-repository/issues/new/choose)
+### Where can I report an issue or RFE related to the AlmaLinux actor or data modifications?
+- GitHub issues are preferred:
+ - Leapp actors: [https://github.com/AlmaLinux/leapp-repository/issues/new/choose](https://github.com/AlmaLinux/leapp-repository/issues/new/choose)
+ - Leapp data: [https://github.com/AlmaLinux/leapp-data/issues/new/choose](https://github.com/AlmaLinux/leapp-data/issues/new/choose)
+
+### What data should be provided when making a report?
+
+Before gathering data, if possible, run the *leapp* command that encountered an issue with the `--debug` flag, e.g.: `leapp upgrade --debug`.
+
- When filing an issue, include:
- Steps to reproduce the issue
- *All files in /var/log/leapp*
@@ -25,7 +88,638 @@
Then you may attach only the `leapp-logs.tgz` file.
### Where can I seek help?
-We’ll gladly answer your questions and lead you to through any troubles with the
-actor development.
+We’ll gladly answer your questions and lead you to through any troubles with the actor development.
+
+You can reach the primary Leapp development team at IRC: `#leapp` on Libera.Chat.
+
+## Third-party integration
+
+If you would like to add your **signed** 3rd party packages into the upgrade process, you can use the third-party integration mechanism.
+
+There are four components for adding your information to the elevation process:
+- <vendor_name>_map.json: repository mapping file
+- <vendor_name>.repo: package repository information
+- <vendor_name>.sigs: list of package signatures of vendor repositories
+- <vendor_name>_pes.json: package migration event list
+
+All these files **must** have the same <vendor_name> part.
+
+### Repository mapping file
+
+This JSON file provides information on mappings between source system repositories (repositories present on the system being upgraded) and target system repositories (package repositories to be used during the upgrade).
+
+The file contains two sections, `mapping` and `repositories`.
+
+`repositories` descripes the source and target repositories themselves. Each entry should have a unique string ID specific to mapping/PES files - `pesid`, and a list of attributes:
+- major_version: major system version that this repository targets
+- repo_type: repository type, see below
+- repoid: repository ID, same as in *.repo files. Doesn't have to exactly match `pesid`
+- arch: system architecture for which this repository is relevant
+- channel: repository channel, see below
+
+
+**Repository types**:
+- rpm: normal RPM packages
+- srpm: source packages
+- debuginfo: packages with debug information
+
+**Repository channels**:
+- ga: general availability repositories
+ - AKA stable repositories.
+- beta: beta-testing repositories
+- eus, e4s, aus, tus: Extended Update Support, Update Services for SAP Solutions, Advanced Update Support, Telco Extended Update Support
+ - Red Hat update channel classification. Most of the time you won't need to use these.
+
+`mapping` establishes connections between described repositories.
+Each entry in the list defines a mapping between major system versions, and contains the following elements:
+- source_major_version: major system version from which the system would be upgraded
+- target_major_version: major system version to which the system would be elevated
+- entries: the list of repository mappings
+ - source: source repository, one that would be found on a pre-upgrade system
+ - target: a list of target upgrade repositores that would contain new package versions. Each source repository can map to one or multiple target repositories
+
+
+> **Important**: The repository mapping file also defines whether a vendor's packages will be included into the upgrade process at all.
+> If at least one source repository listed in the file is present on the system, the vendor is considered active, and package repositories/PES events are enabled - otherwise, they **will not** affect the upgrade process.
+
+### Package repository information
+
+This file defines the vendor's package repositories to be used during the upgrade.
+
+The file has the same format normal YUM/DNF package repository files do.
+
+> NOTE: The repositories listed in this file are only used *during* the upgrade. Package repositories on the post-upgrade system should be provided through updated packages or custom repository deployment.
+
+### Package signature list
+
+This file should contain the list of public signature headers that the packages are signed with, one entry per line.
+
+You can find signature headers for your packages by running the following command:
+
+`rpm -qa --queryformat "%{NAME} || %|DSAHEADER?{%{DSAHEADER:pgpsig}}:{%|RSAHEADER?{%{RSAHEADER:pgpsig}}:{(none)}|}|\n" <PACKAGE_NAME>`
+
+rpm will return an entry like the following:
+`package-name || DSA/SHA1, Mon Aug 23 08:17:13 2021, Key ID 8c55a6628608cb71`
+
+The value after "Key ID", in this case, `8c55a6628608cb71`, is what you should put into the signature list file.
+
+### Package migration event list
+
+The Leapp upgrade process uses information from the AlmaLinux PES (Package Evolution System) to keep track of how packages change between the OS versions. This data is located in `leapp-data/vendors.d/<vendor_name>_pes.json` in the GitHub repository and in `/etc/leapp/files/vendors.d/<vendor_name>_pes.json` on a system being upgraded.
+
+> **Warning**: leapp doesn't force packages from out_packageset to be installed from the specific repository; instead, it enables repo from out_packageset and then DNF installs the latest package version from all enabled repos.
+
+#### Creating event lists through PES
+
+The recommended way to create new event lists is to use the PES mechanism.
+
+The web interface can create, manage and export groups of events to JSON files.
+
+This video demonstration walks through the steps of adding an action event group and exporting it as a JSON file to make use of it in the elevation process.
+
+> https://drive.google.com/file/d/1VqnQkUsxzLijIqySMBGu5lDrA72BVd5A/view?usp=sharing
+
+Please refer to the [PES contribution guide](https://wiki.almalinux.org/elevate/Contribution-guide.html) for additional information on entry fields.
+
+#### Manual editing
+
+To add new rules to the list, add a new entry to the `packageinfo` array.
+
+**Important**: actions from PES JSON files will be in effect only for those packages that are signed **and** have their signatures in one of the active <vendor_name>.sigs files. Unsigned packages will be updated only if some signed package requires a new version, otherwise they will by left as they are.
+
+Required fields:
+
+- action: what action to perform on the listed package
+ - 0 - present
+ - keep the packages in `in_packageset` to make sure the repo they're in on the target system gets enabled
+ - additional behaviour present, see below
+ - 1 - removed
+ - remove all packages in `in_packageset`
+ - 2 - deprecated
+ - keep the packages in `in_packageset` to make sure the repo they're in on the target system gets enabled
+ - 3 - replaced
+ - remove all packages in `in_packageset`
+ - install parts of the `out_packageset` that are not present on the system
+ - keep the packages from `out_packageset` that are already installed
+ - 4 - split
+ - install parts of the `out_packageset` that are not present on the system
+ - keep the present `out_packageset`
+ - remove packages from `in_packageset` that are not present in `out_packageset`
+ - in case of package X being split to Y and Z, package X will be removed
+ - in case of package X being split to X and Y, package X will **not** be removed
+ - 5 - merged
+ - same as `split`
+ - additional behaviour present, see below
+ - 6 - moved to new repository
+ - keep the package to make sure the repo it's in on the target system gets enabled
+ - nothing is done to `in_packageset` as it always contains one package - the same as the "out" package
+ - 7 - renamed
+ - remove the `in_packageset` and install the `out_packageset` if not installed
+ - if already installed, keep the `out_packageset` as-is
+ - 8 - reinstalled
+ - reinstall the `in_packageset` package during the upgrade transaction
+ - mostly useful for packages that have the same version string between major versions, and thus won't be upgraded automatically
+ - Additional notes:
+ - any event except `present` is ignored if any of packages in `in_packageset` are marked for removal
+ - any event except `merged` is ignored if any of packages in `in_packageset` are neither installed nor marked for installation
+ - for `merged` events it is sufficient to have at least one package from `in_packageset` are either installed or marked for installation
+- arches: what system architectures the listed entry relates to
+- id: entry ID, must be unique
+- in_packageset: set of packages on the old system
+- out_packageset: set of packages to switch to, empty if removed or deprecated
+- initial_release: source OS release
+- release: target OS release
+
+`in_packageset` and `out_packageset` have the following format:
+
+```json
+ "in_packageset": {
+ "package": [
+ {
+ "module_stream": null,
+ "name": "PackageKit",
+ "repository": "base"
+ },
+ {
+ "module_stream": null,
+ "name": "PackageKit-yum",
+ "repository": "base"
+ }
+ ],
+ "set_id": 1592
+ },
+```
+
+For `in_packageset`, `repository` field defines the package repository the package was installed from on the source system.
+For `out_packageset`, `repository` field for packages should be the same as the "Target system repo name in PES" field in the associated vendor repository mapping file.
+
+### Providing the data
+
+Once you've prepared the vendor data for migration, you can make a pull request to https://github.com/AlmaLinux/leapp-data/ to make it available publicly.
+Files should be placed into the `vendors.d` subfolder if the data should be available for all elevation target OS variants, or into the `files/<target_OS>/vendors.d/` if intended for a specific one.
+
+Alternatively, you can deploy the vendor files on a system prior to starting the upgrade. In this case, place the files into the folder `/etc/leapp/files/vendors.d/`.
+
+## Adding complex changes (custom actors for migration)
+To perform any changes of arbitrary complexity during the migration process, add a component to the existing Leapp pipeline.
+
+To begin, clone the code repository: https://github.com/AlmaLinux/leapp-repository
+For instructions on how to deploy a development enviroment, refer to [Leapp framework documentation](https://leapp.readthedocs.io/en/latest/devenv-install.html).
+
+Create an actor inside the main system_upgrade leapp repository:
+
+```bash
+cd ./leapp-repository/repos/system_upgrade/common
+snactor new-actor testactor
+```
+
+Alternatively, you can [create your own repository](https://leapp.readthedocs.io/en/latest/create-repository.html) in the system_upgrade folder, if you wish to keep your actors separate from others.
+Keep in mind that you’ll need to link all other repositories whose functions you will use.
+The created subfolder will contain the main Python file of your new actor.
+
+The actor’s main class has three fields of interest:
+- consumes
+- produces
+- tags
+
+consumes and produces defines the [data that the actor may receive or provide to other actors](https://leapp.readthedocs.io/en/latest/messaging.html).
+
+Tags define the phase of the upgrade process during which the actor runs.
+All actors also must be assigned the `IPUWorkflowTag` to mark them as a part of the in-place upgrade process.
+The file `leapp-repository/repos/system_upgrade/common/workflows/inplace_upgrade.py` lists all phases of the elevation process.
+
+### Submitting changes
+Changes you want to submit upstream should be sent through pull requests to repositories https://github.com/AlmaLinux/leapp-repository and https://github.com/AlmaLinux/leapp-data.
+The standard GitHub contribution process applies - fork the repository, make your changes inside of it, then submit the pull request to be reviewed.
+
+### Custom actor example
+
+"Actors" in Leapp terminology are Python scripts that run during the upgrade process.
+Actors are a core concept of the framework, and the entire process is built from them.
+
+Custom actors are the actors that are added by third-party developers, and are not present in the upstream Leapp repository.
+
+Actors can gather data, communicate with each other and modify the system during the upgrade.
+
+Let's examine how an upgrade problem might be resolved with a custom actor.
+
+#### Problem
+
+If you ever ran `leapp preupgrade` on unprepared systems before, you likely have seen the following message:
+
+```
+Upgrade has been inhibited due to the following problems:
+ 1. Inhibitor: Possible problems with remote login using root account
+```
+
+It's caused by the change in default behaviour for permitting root logins between RHEL 7 and 8.
+In RHEL 8 logging in as root via password authentication is no longer allowed by default, which means that some machines can become inaccessible after the upgrade.
+
+Some configurations require an administrator's intervention to resolve this issue, but SSHD configurations where no `PermitRootLogin` options were explicitly set can be modified to preserve the RHEL 7 default behaviour and not require manual modification.
+
+Let's create a custom actor to handle such cases for us.
+
+#### Creating an actor
+
+Actors are contained in ["repositories"](https://leapp.readthedocs.io/en/latest/leapp-repositories.html) - subfolders containing compartmentalized code and resources that the Leapp framework will use during the upgrade.
+
+> Do not confuse Leapp repositories with Git repositories - these are two different concepts, independent of one another.
+
+Inside the `leapp-repository` GitHub repo, Leapp repositories are contained inside the `repos` subfolder.
+
+Everything related to system upgrade proper is inside the `system_upgrade` folder.
+`el7toel8` contains resources used when upgrading from RHEL 7 to RHEL 8, `el8toel9` - RHEL 8 to 9, `common` - shared resources.
+
+Since the change in system behaviour we're looking to mitigate occurs between RHEL 7 and 8, the appopriate repository to place the actor in is `el7toel8`.
+
+You can [create new actors](https://leapp.readthedocs.io/en/latest/first-actor.html) by using the `snactor` tool provided by Leapp, or manually.
+
+`snactor new-actor ACTOR_NAME`
+
+The bare-bones actor code consists of a file named `actor.py` contained inside the `actors/<actor_name>` subfolder of a Leapp repository.
+
+In this case, then, it should be located in a directory like `leapp-repository/repos/system_upgrade/el7toel8/actors/opensshmodifypermitroot`
+
+If you used snactor to create it, you'll see contents like the following:
+
+```python
+from leapp.actors import Actor
+
+
+class OpenSSHModifyPermitRoot(Actor):
+ """
+ No documentation has been provided for the open_ssh_actor_example actor.
+ """
+
+ name = 'openssh_modify_permit_root'
+ consumes = ()
+ produces = ()
+ tags = ()
+
+ def process(self):
+ pass
+```
+
+#### Configuring the actor
+
+Actors' `consumes` and `produces` attributes define types of [*messages*](https://leapp.readthedocs.io/en/latest/messaging.html) these actors receive or send.
+
+For instance, during the initial upgrade stages several standard actors gather system information and *produce* messages with gathered data to other actors.
+
+> Messages are defined by *message models*, which are contained inside Leapp repository's `models` subfolder, just like all actors are contained in `actors`.
+
+Actors' `tags` attributes define the [phase of the upgrade](https://leapp.readthedocs.io/en/latest/working-with-workflows.html) during which that actor gets executed.
+
+> The list of all phases can be found in file `leapp-repository/repos/system_upgrade/common/workflows/inplace_upgrade.py`.
+
+##### Receiving messages
+
+Leapp already provides information about the OpenSSH configuration through the `OpenSshConfigScanner` actor. This actor provides a message with a message model `OpenSshConfig`.
+
+Instead of opening and reading the configuration file in our own actor, we can simply read the provided message to see if we can safely alter the configuration automatically.
+
+To begin with, import the message model from `leapp.models`:
+
+```python
+from leapp.models import OpenSshConfig
+```
+
+> It doesn't matter in which Leapp repository the model is located. Leapp will gather all availabile data inside its submodules.
+
+Add the message model to the list of messages to be received:
+
+```python
+consumes = (OpenSshConfig, )
+```
+
+The actor now will be able to read messages of this format provided by other actors that were executed prior to its own execution.
+
+##### Sending messages
+
+To ensure that the user knows about the automatic configuration change that will occur, we can send a *report*.
+
+> Reports are a built-in type of Leapp messages that are added to the `/var/log/leapp/leapp-report.txt` file at the end of the upgrade process.
+
+To start off with, add a Report message model to the `produces` attribute of the actor.
+
+```python
+produces = (Report, )
+```
+
+Don't forget to import the model type from `leapp.models`.
+
+All done - now we're ready to make use of the models inside the actor's code.
+
+
+##### Running phase
+
+Both workflow and phase tags are imported from leapp.tags:
+
+```python
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+```
+
+All actors to be run during the upgrade must contain the upgrade workflow tag. It looks as follows:
+
+```python
+tags = (IPUWorkflowTag, )
+```
+
+To define the upgrade phase during which an actor will run, set the appropriate tag in the `tags` attribute.
+
+Standard actor `OpenSshPermitRootLoginCheck` that blocks the upgrade if it detects potential problems in SSH configuration, runs during the *checks* phase, and has the `ChecksPhaseTag` inside its `tags`.
+
+Therefore, we want to run our new actor before it. We can select an earlier phase from the list of phases - or we can mark our actor to run *before other actors* in the phase with a modifier as follows:
+
+```python
+tags = (ChecksPhaseTag.Before, IPUWorkflowTag, )
+```
+
+All phases have built-in `.Before` and `.After` stages that can be used this way. Now our actor is guaranteed to be run before the `OpenSshPermitRootLoginCheck` actor.
+
+
+#### Actor code
+
+With configuration done, it's time to write the actual code of the actor that will be executed during the upgrade.
+
+The entry point for it is the actor's `process` function.
+
+First, let's start by reading the SSH config message we've set the actor to receive.
+
+```python
+# Importing from Leapp built-ins.
+from leapp.exceptions import StopActorExecutionError
+from leapp.libraries.stdlib import api
+
+def process(self):
+ # Retreive the OpenSshConfig message.
+
+ # Actors have `consume` and `produce` methods that work with messages.
+ # `consume` expects a message type that is listed inside the `consumes` attribute.
+ openssh_messages = self.consume(OpenSshConfig)
+
+ # The return value of self.consume is a generator of messages of the provided type.
+ config = next(openssh_messages, None)
+ # We expect to get only one message of this type. If there's more than one, something's wrong.
+ if list(openssh_messages):
+ # api.current_logger lets you pass messages into Leapp's log. By default, they will
+ # be displayed in `/var/log/leapp/leapp-preupgrade.log`
+ # or `/var/log/leapp/leapp-upgrade.log`, depending on which command you ran.
+ api.current_logger().warning('Unexpectedly received more than one OpenSshConfig message.')
+ # If the config message is not present, the standard actor failed to read it.
+ # Stop here.
+ if not config:
+ # StopActorExecutionError is a Leapp built-in exception type that halts the actor execution.
+ # By default this will also halt the upgrade phase and the upgrade process in general.
+ raise StopActorExecutionError(
+ 'Could not check openssh configuration', details={'details': 'No OpenSshConfig facts found.'}
+ )
+```
+
+Next, let's read the received message and see if we can modify the configuration.
+
+```python
+import errno
+
+CONFIG = '/etc/ssh/sshd_config'
+CONFIG_BACKUP = '/etc/ssh/sshd_config.leapp_backup'
+
+ # The OpenSshConfig model has a permit_root_login attribute that contains
+ # all instances of PermitRootLogin option present in the config.
+ # See leapp-repository/repos/system_upgrade/el7toel8/models/opensshconfig.py
+
+ # We can only safely modify the config to preserve the default behaviour if no
+ # explicit PermitRootLogin option was set anywhere in the config.
+ if not config.permit_root_login:
+ try:
+ # Read the config into memory to prepare for its modification.
+ with open(CONFIG, 'r') as fd:
+ sshd_config = fd.readlines()
+
+ # These are the lines we want to add to the configuration file.
+ permit_autoconf = [
+ "# Automatically added by Leapp to preserve RHEL7 default\n",
+ "# behaviour after migration.\n",
+ "# Placed on top of the file to avoid being included into Match blocks.\n",
+ "PermitRootLogin yes\n"
+ "\n",
+ ]
+ permit_autoconf.extend(sshd_config)
+ # Write the changed config into the file.
+ with open(CONFIG, 'w') as fd:
+ fd.writelines(permit_autoconf)
+ # Write the backup file with the old configuration.
+ with open(CONFIG_BACKUP, 'w') as fd:
+ fd.writelines(sshd_config)
+
+ # Handle errors.
+ except IOError as err:
+ if err.errno != errno.ENOENT:
+ error = 'Failed to open sshd_config: {}'.format(str(err))
+ api.current_logger().error(error)
+ return
+```
+
+The functional part of the actor itself is done. Now, let's add a report to let the user know
+the machine's SSH configuration has changed.
+
+```python
+# These Leapp imports are required to create reports.
+from leapp import reporting
+from leapp.models import Report
+from leapp.reporting import create_report
+
+# Tags signify the categories the report and the associated issue are related to.
+COMMON_REPORT_TAGS = [
+ reporting.Tags.AUTHENTICATION,
+ reporting.Tags.SECURITY,
+ reporting.Tags.NETWORK,
+ reporting.Tags.SERVICES
+]
+
+ # Related resources are listed in the report to help resolving the issue.
+ resources = [
+ reporting.RelatedResource('package', 'openssh-server'),
+ reporting.RelatedResource('file', '/etc/ssh/sshd_config')
+ reporting.RelatedResource('file', '/etc/ssh/sshd_config.leapp_backup')
+ ]
+ # This function creates and submits the actual report message.
+ # Normally you'd need to call self.produce() to send messages,
+ # but reports are a special case that gets handled automatically.
+ create_report([
+ # Report title and summary.
+ reporting.Title('SSH configuration automatically modified to permit root login'),
+ reporting.Summary(
+ 'Your OpenSSH configuration file does not explicitly state '
+ 'the option PermitRootLogin in sshd_config file. '
+ 'Its default is "yes" in RHEL7, but will change in '
+ 'RHEL8 to "prohibit-password", which may affect your ability '
+ 'to log onto this machine after the upgrade. '
+ 'To prevent this from occuring, the PermitRootLogin option '
+ 'has been explicity set to "yes" to preserve the default behaivour '
+ 'after migration.'
+ 'The original configuration file has been backed up to'
+ '/etc/ssh/sshd_config.leapp_backup'
+ ),
+ # Reports are ordered by severity in the list.
+ reporting.Severity(reporting.Severity.MEDIUM),
+ reporting.Tags(COMMON_REPORT_TAGS),
+ # Remediation section contains hints on how to resolve the reported (potential) problem.
+ reporting.Remediation(
+ hint='If you would prefer to configure the root login policy yourself, '
+ 'consider setting the PermitRootLogin option '
+ 'in sshd_config explicitly.'
+ )
+ ] + resources) # Resources are added to the list of data for the report.
+```
+
+The actor code is now complete. The final version with less verbose comments will look something like this:
+
+```python
+from leapp import reporting
+from leapp.actors import Actor
+from leapp.exceptions import StopActorExecutionError
+from leapp.libraries.stdlib import api
+from leapp.models import OpenSshConfig, Report
+from leapp.reporting import create_report
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+
+import errno
+
+CONFIG = '/etc/ssh/sshd_config'
+CONFIG_BACKUP = '/etc/ssh/sshd_config.leapp_backup'
+
+COMMON_REPORT_TAGS = [
+ reporting.Tags.AUTHENTICATION,
+ reporting.Tags.SECURITY,
+ reporting.Tags.NETWORK,
+ reporting.Tags.SERVICES
+]
+
+
+class OpenSSHModifyPermitRoot(Actor):
+ """
+ OpenSSH doesn't allow root logins with password by default on RHEL8.
+
+ Check the values of PermitRootLogin in OpenSSH server configuration file
+ and see if it was set explicitly.
+ If not, adding an explicit "PermitRootLogin yes" will preserve the current
+ default behaviour.
+ """
+
+ name = 'openssh_modify_permit_root'
+ consumes = (OpenSshConfig, )
+ produces = (Report, )
+ tags = (ChecksPhaseTag.Before, IPUWorkflowTag, )
+
+ def process(self):
+ # Retreive the OpenSshConfig message.
+ openssh_messages = self.consume(OpenSshConfig)
+ config = next(openssh_messages, None)
+ if list(openssh_messages):
+ api.current_logger().warning('Unexpectedly received more than one OpenSshConfig message.')
+ if not config:
+ raise StopActorExecutionError(
+ 'Could not check openssh configuration', details={'details': 'No OpenSshConfig facts found.'}
+ )
+
+ # Read and modify the config.
+ # Only act if there's no explicit PermitRootLogin option set anywhere in the config.
+ if not config.permit_root_login:
+ try:
+ with open(CONFIG, 'r') as fd:
+ sshd_config = fd.readlines()
+
+ permit_autoconf = [
+ "# Automatically added by Leapp to preserve RHEL7 default\n",
+ "# behaviour after migration.\n",
+ "# Placed on top of the file to avoid being included into Match blocks.\n",
+ "PermitRootLogin yes\n"
+ "\n",
+ ]
+ permit_autoconf.extend(sshd_config)
+ with open(CONFIG, 'w') as fd:
+ fd.writelines(permit_autoconf)
+ with open(CONFIG_BACKUP, 'w') as fd:
+ fd.writelines(sshd_config)
+
+ except IOError as err:
+ if err.errno != errno.ENOENT:
+ error = 'Failed to open sshd_config: {}'.format(str(err))
+ api.current_logger().error(error)
+ return
+
+ # Create a report letting the user know what happened.
+ resources = [
+ reporting.RelatedResource('package', 'openssh-server'),
+ reporting.RelatedResource('file', '/etc/ssh/sshd_config'),
+ reporting.RelatedResource('file', '/etc/ssh/sshd_config.leapp_backup')
+ ]
+ create_report([
+ reporting.Title('SSH configuration automatically modified to permit root login'),
+ reporting.Summary(
+ 'Your OpenSSH configuration file does not explicitly state '
+ 'the option PermitRootLogin in sshd_config file. '
+ 'Its default is "yes" in RHEL7, but will change in '
+ 'RHEL8 to "prohibit-password", which may affect your ability '
+ 'to log onto this machine after the upgrade. '
+ 'To prevent this from occuring, the PermitRootLogin option '
+ 'has been explicity set to "yes" to preserve the default behaivour '
+ 'after migration.'
+ 'The original configuration file has been backed up to'
+ '/etc/ssh/sshd_config.leapp_backup'
+ ),
+ reporting.Severity(reporting.Severity.MEDIUM),
+ reporting.Tags(COMMON_REPORT_TAGS),
+ reporting.Remediation(
+ hint='If you would prefer to configure the root login policy yourself, '
+ 'consider setting the PermitRootLogin option '
+ 'in sshd_config explicitly.'
+ )
+ ] + resources)
+```
+
+Due to this actor's small size, the entire code can be fit inside the `process` function.
+If it grows beyond manageable size, or you want to run unit tests on its components, it's advised to move out all of the functional parts from the `process` function into the *actor library*.
+
+#### Libraries
+
+Larger actors can import code from [common libraries](https://leapp.readthedocs.io/en/latest/best-practices.html#move-generic-functionality-to-libraries) or define their own "libraries" and run code from them inside the `process` function.
+
+In such cases, the directory layout looks like this:
+```
+actors
++ example_actor_name
+| + libraries
+| + example_actor_name.py
+| + actor.py
+...
+```
+
+and importing code from them looks like this:
+
+`from leapp.libraries.actor.example_actor_name import example_lib_function`
+
+This is also the main way of [writing unit-testable code](https://leapp.readthedocs.io/en/latest/best-practices.html#write-unit-testable-code), since the code contained inside the `process` function cannot be unit-tested normally.
+
+In this actor format, you would move all of the actual actor code into the associated library, leaving only preparation and function calls inside the `process` function.
+
+#### Debugging
+
+The Leapp utility `snactor` can also be used for unit-testing the created actors.
+
+It is capable of saving the output of actors as locally stored messages, so that they can be consumed by other actors that are being developed.
+
+For example, to test our new actor, we need the OpenSshConfig message, which is produced by the OpenSshConfigScanner standard actor. To make the data consumable, run the actor producing the data with the –save-output option:
+
+`snactor run --save-output OpenSshConfigScanner`
+
+The output of the actor is stored in the local repository data file, and it can be used by other actors. To flush all saved messages from the repository database, run `snactor messages clear`.
+
+With the input messages available and stored, the actor being developed can be tested.
+
+`snactor run --print-output OpenSshModifyPermitRoot`
+
+#### Additional information
-You can reach us at IRC: `#leapp` on Libera.Chat.
+For more information about Leapp and additional tutorials, visit the [official Leapp documentation](https://leapp.readthedocs.io/en/latest/tutorials.html).
diff --git a/commands/command_utils.py b/commands/command_utils.py
index da62c50..a8e7d76 100644
--- a/commands/command_utils.py
+++ b/commands/command_utils.py
@@ -12,7 +12,7 @@ LEAPP_UPGRADE_FLAVOUR_DEFAULT = 'default'
LEAPP_UPGRADE_FLAVOUR_SAP_HANA = 'saphana'
LEAPP_UPGRADE_PATHS = 'upgrade_paths.json'
-VERSION_REGEX = re.compile(r"^([1-9]\d*)\.(\d+)$")
+VERSION_REGEX = re.compile(r"^([1-9]\d*)(\.(\d+))?$")
def check_version(version):
diff --git a/commands/upgrade/__init__.py b/commands/upgrade/__init__.py
index 39bfd52..b620df6 100644
--- a/commands/upgrade/__init__.py
+++ b/commands/upgrade/__init__.py
@@ -87,7 +87,7 @@ def upgrade(args, breadcrumbs):
workflow = repositories.lookup_workflow('IPUWorkflow')(auto_reboot=args.reboot)
util.process_whitelist_experimental(repositories, workflow, configuration, logger)
util.warn_if_unsupported(configuration)
- with beautify_actor_exception():
+ with util.format_actor_exceptions(logger):
logger.info("Using answerfile at %s", answerfile_path)
workflow.load_answers(answerfile_path, userchoices_path)
@@ -101,13 +101,16 @@ def upgrade(args, breadcrumbs):
logger.info("Answerfile will be created at %s", answerfile_path)
workflow.save_answers(answerfile_path, userchoices_path)
report_errors(workflow.errors)
+ util.log_errors(workflow.errors, logger)
report_inhibitors(context)
+ util.log_inhibitors(context, logger)
util.generate_report_files(context, report_schema)
report_files = util.get_cfg_files('report', cfg)
log_files = util.get_cfg_files('logs', cfg)
report_info(report_files, log_files, answerfile_path, fail=workflow.failure)
if workflow.failure:
+ logger.error("Upgrade workflow failed, check log for details")
sys.exit(1)
diff --git a/commands/upgrade/util.py b/commands/upgrade/util.py
index ce0b543..0b2eba8 100644
--- a/commands/upgrade/util.py
+++ b/commands/upgrade/util.py
@@ -2,18 +2,23 @@ import functools
import itertools
import json
import os
+import sys
import shutil
import tarfile
from datetime import datetime
+from contextlib import contextmanager
from leapp.cli.commands import command_utils
from leapp.cli.commands.config import get_config
-from leapp.exceptions import CommandError
+from leapp.exceptions import CommandError, LeappRuntimeError
from leapp.repository.scan import find_and_scan_repositories
from leapp.utils import audit
from leapp.utils.audit import get_checkpoints, get_connection, get_messages
-from leapp.utils.output import report_unsupported
+from leapp.utils.output import report_unsupported, pretty_block_text
from leapp.utils.report import fetch_upgrade_report_messages, generate_report_file
+from leapp.models import ErrorModel
+
+
def disable_database_sync():
@@ -228,3 +233,68 @@ def process_whitelist_experimental(repositories, workflow, configuration, logger
if logger:
logger.error(msg)
raise CommandError(msg)
+
+
+def process_report_schema(args, configuration):
+ default_report_schema = configuration.get('report', 'schema')
+ if args.report_schema and args.report_schema > default_report_schema:
+ raise CommandError('--report-schema version can not be greater that the '
+ 'actual {} one.'.format(default_report_schema))
+ return args.report_schema or default_report_schema
+
+
+# TODO: This and the following functions should eventually be placed into the
+# leapp.utils.output module.
+def pretty_block_log(string, logger_level, width=60):
+ log_str = "\n{separator}\n{text}\n{separator}\n".format(
+ separator="=" * width,
+ text=string.center(width))
+ logger_level(log_str)
+
+
+@contextmanager
+def format_actor_exceptions(logger):
+ try:
+ try:
+ yield
+ except LeappRuntimeError as e:
+ # TODO: This only reports the actor that raised an exception
+ # and the return code.
+ # The traceback gets eaten on the framework level, and is only
+ # seen in stderr. Changing that will require modifying the framework
+ # code itself.
+ msg = '{} - Please check the above details'.format(e.message)
+ sys.stderr.write("\n")
+ sys.stderr.write(pretty_block_text(msg, color="", width=len(msg)))
+ logger.error(e.message)
+ finally:
+ pass
+
+
+def log_errors(errors, logger):
+ if errors:
+ pretty_block_log("ERRORS", logger.info)
+
+ for error in errors:
+ model = ErrorModel.create(json.loads(error['message']['data']))
+ logger.error("{time} [{severity}] Actor: {actor}\nMessage: {message}\n".format(
+ severity=model.severity.upper(),
+ message=model.message, time=model.time, actor=model.actor))
+ if model.details:
+ print('Summary:')
+ details = json.loads(model.details)
+ for detail in details:
+ print(' {k}: {v}'.format(
+ k=detail.capitalize(),
+ v=details[detail].rstrip().replace('\n', '\n' + ' ' * (6 + len(detail)))))
+
+
+def log_inhibitors(context_id, logger):
+ from leapp.reporting import Flags # pylint: disable=import-outside-toplevel
+ reports = fetch_upgrade_report_messages(context_id)
+ inhibitors = [report for report in reports if Flags.INHIBITOR in report.get('flags', [])]
+ if inhibitors:
+ logger.error('Upgrade has been inhibited due to the following problems:')
+ for position, report in enumerate(inhibitors, start=1):
+ logger.error('{idx:5}. Inhibitor: {title}'.format(idx=position, title=report['title']))
+ logger.info('Consult the pre-upgrade report for details and possible remediation.')
diff --git a/etc/leapp/transaction/to_reinstall b/etc/leapp/transaction/to_reinstall
new file mode 100644
index 0000000..c6694a8
--- /dev/null
+++ b/etc/leapp/transaction/to_reinstall
@@ -0,0 +1,3 @@
+### List of packages (each on new line) to be reinstalled to the upgrade transaction
+### Useful for packages that have identical version strings but contain binary changes between major OS versions
+### Packages that aren't installed will be skipped
diff --git a/packaging/leapp-repository.spec b/packaging/leapp-repository.spec
index 5411fbb..95cfdfc 100644
--- a/packaging/leapp-repository.spec
+++ b/packaging/leapp-repository.spec
@@ -211,6 +211,8 @@ rm -rf %{buildroot}%{leapp_python_sitelib}/leapp/cli/commands/tests
rm -rf %{buildroot}%{repositorydir}/system_upgrade/el8toel9
%else
rm -rf %{buildroot}%{repositorydir}/system_upgrade/el7toel8
+# CloudLinux migration only supports el7 to el8
+rm -rf %{buildroot}%{repositorydir}/system_upgrade/cloudlinux
%endif
# remove component/unit tests, Makefiles, ... stuff that related to testing only
diff --git a/repos/system_upgrade/cloudlinux/.leapp/info b/repos/system_upgrade/cloudlinux/.leapp/info
new file mode 100644
index 0000000..1f16b9f
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/.leapp/info
@@ -0,0 +1 @@
+{"name": "cloudlinux", "id": "427ddd90-9b5e-4400-b21e-73d77791f175", "repos": ["644900a5-c347-43a3-bfab-f448f46d9647", "c47fbc3d-ae38-416e-9176-7163d67d94f6", "efcf9016-f2d1-4609-9329-a298e6587b3c"]}
\ No newline at end of file
diff --git a/repos/system_upgrade/cloudlinux/.leapp/leapp.conf b/repos/system_upgrade/cloudlinux/.leapp/leapp.conf
new file mode 100644
index 0000000..b459134
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/.leapp/leapp.conf
@@ -0,0 +1,6 @@
+
+[repositories]
+repo_path=${repository:root_dir}
+
+[database]
+path=${repository:state_dir}/leapp.db
diff --git a/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/actor.py b/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/actor.py
new file mode 100644
index 0000000..783e347
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/actor.py
@@ -0,0 +1,21 @@
+from leapp.actors import Actor
+from leapp.tags import FirstBootPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+from leapp.libraries.actor.addcustomrepositories import add_custom
+
+
+class AddCustomRepositories(Actor):
+ """
+ Move the files inside the custom-repos folder of this leapp repository into the /etc/yum.repos.d repository.
+ """
+
+ name = 'add_custom_repositories'
+ consumes = ()
+ produces = ()
+ tags = (IPUWorkflowTag, FirstBootPhaseTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ # We only want to run this actor on CloudLinux systems.
+ # current_version returns a tuple (release_name, version_value).
+ add_custom(self.log)
diff --git a/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/libraries/addcustomrepositories.py b/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/libraries/addcustomrepositories.py
new file mode 100644
index 0000000..74ba425
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/addcustomrepositories/libraries/addcustomrepositories.py
@@ -0,0 +1,26 @@
+import os
+import os.path
+import shutil
+import logging
+
+from leapp.libraries.stdlib import api
+
+CUSTOM_REPOS_FOLDER = 'custom-repos'
+REPO_ROOT_PATH = "/etc/yum.repos.d"
+
+
+def add_custom(log):
+ # type: (logging.Logger) -> None
+ custom_repo_dir = api.get_common_folder_path(CUSTOM_REPOS_FOLDER)
+ repofiles = os.listdir(custom_repo_dir)
+
+ # If any components are missing, halt.
+ if not repofiles or not custom_repo_dir:
+ return
+
+ for repofile in repofiles:
+ full_repo_path = os.path.join(custom_repo_dir, repofile)
+
+ log.debug("Copying repo file {} to {}".format(repofile, REPO_ROOT_PATH))
+
+ shutil.copy(full_repo_path, REPO_ROOT_PATH)
diff --git a/repos/system_upgrade/cloudlinux/actors/checkcllicense/actor.py b/repos/system_upgrade/cloudlinux/actors/checkcllicense/actor.py
new file mode 100644
index 0000000..2c935d7
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/checkcllicense/actor.py
@@ -0,0 +1,41 @@
+from leapp.actors import Actor
+from leapp import reporting
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+from leapp.libraries.stdlib import CalledProcessError, run
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+import os
+
+
+class CheckClLicense(Actor):
+ """
+ Check if the server has a CL license
+ """
+
+ name = 'check_cl_license'
+ consumes = ()
+ produces = ()
+ tags = (ChecksPhaseTag, IPUWorkflowTag)
+
+ system_id_path = '/etc/sysconfig/rhn/systemid'
+ rhn_check_bin = '/usr/sbin/rhn_check'
+
+ @run_on_cloudlinux
+ def process(self):
+ res = None
+ if os.path.exists(self.system_id_path):
+ res = run([self.rhn_check_bin])
+ self.log.debug('rhn_check result: %s', res)
+ if not res or res['exit_code'] != 0 or res['stderr']:
+ title = 'Server does not have an active CloudLinux license'
+ summary = 'Server does not have an active CloudLinux license. This renders key CloudLinux packages ' \
+ 'inaccessible, inhibiting the upgrade process.'
+ remediation = 'Activate a CloudLinux license on this machine before running Leapp again.'
+ reporting.create_report([
+ reporting.Title(title),
+ reporting.Summary(summary),
+ reporting.Severity(reporting.Severity.HIGH),
+ reporting.Tags([reporting.Tags.OS_FACTS]),
+ reporting.Flags([reporting.Flags.INHIBITOR]),
+ reporting.Remediation(hint=remediation),
+ ])
diff --git a/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/actor.py b/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/actor.py
new file mode 100644
index 0000000..a1c1cee
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/actor.py
@@ -0,0 +1,58 @@
+from leapp.actors import Actor
+from leapp import reporting
+from leapp.reporting import Report
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+from leapp.libraries.actor.version import (
+ Version, VersionParsingError,
+)
+
+import subprocess
+
+
+class CheckRhnClientToolsVersion(Actor):
+ """
+ Check the rhn-client-tools package version
+ """
+
+ name = 'check_rhn_client_tools_version'
+ consumes = ()
+ produces = (Report,)
+ tags = (ChecksPhaseTag, IPUWorkflowTag)
+
+ minimal_version = Version('2.0.2')
+ minimal_release_int = 43
+ minimal_release = '%s.el7.cloudlinux' % minimal_release_int
+
+ @run_on_cloudlinux
+ def process(self):
+ title, summary, remediation = None, None, None
+ # ex:
+ # Version : 2.0.2
+ # Release : 43.el7.cloudlinux
+ # res is: b'2.0.2\n43.el7.cloudlinux\n'
+ cmd = "yum info installed rhn-client-tools | grep '^Version' -A 1 | awk '{print $3}'"
+ res = subprocess.check_output(cmd, shell=True)
+ rhn_version, rhn_release = res.decode().split()
+ self.log.info('Current rhn-client-tools version: "%s"', rhn_version)
+ try:
+ current_version = Version(rhn_version)
+ except VersionParsingError:
+ title = 'rhn-client-tools: package is not installed'
+ summary = 'rhn-client-tools package is required to perform elevation.'
+ remediation = 'Install rhn-client-tools "%s" version before running Leapp again.' % self.minimal_version
+ else:
+ if current_version < self.minimal_version or int(rhn_release.split('.')[0]) < self.minimal_release_int:
+ title = 'rhn-client-tools: package version is too low'
+ summary = 'Current version of the rhn-client-tools package has no capability to perform elevation.'
+ remediation = 'Update rhn-client-tools to "%s %s" version before running Leapp again.' % (self.minimal_version, self.minimal_release)
+ if title:
+ reporting.create_report([
+ reporting.Title(title),
+ reporting.Summary(summary),
+ reporting.Severity(reporting.Severity.HIGH),
+ reporting.Tags([reporting.Tags.OS_FACTS]),
+ reporting.Flags([reporting.Flags.INHIBITOR]),
+ reporting.Remediation(hint=remediation),
+ ])
diff --git a/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/libraries/version.py b/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/libraries/version.py
new file mode 100644
index 0000000..149bce2
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/checkrhnclienttools/libraries/version.py
@@ -0,0 +1,46 @@
+from six import reraise as raise_
+import sys
+
+
+class VersionException(Exception):
+ pass
+
+
+class VersionParsingError(VersionException):
+ pass
+
+
+class Version(object):
+ def __init__(self, version):
+ self._raw = version
+ try:
+ self.value = tuple(
+ map(lambda x: int(x), version.split('.'))
+ )
+ except Exception:
+ tb = sys.exc_info()[2]
+ raise_(VersionParsingError, 'failed to parse version: "%s"' % self._raw, tb)
+
+ def __eq__(self, other):
+ return self.value == other.value
+
+ def __gt__(self, other):
+ return any(
+ [v[0] > v[1] for v in zip(self.value, other.value)]
+ )
+
+ def __ge__(self, other):
+ return all(
+ [v[0] >= v[1] for v in zip(self.value, other.value)]
+ )
+
+ def __lt__(self, other):
+ return any(
+ [v[0] < v[1] for v in zip(self.value, other.value)]
+ )
+
+ def __le__(self, other):
+ return all(
+ [v[0] <= v[1] for v in zip(self.value, other.value)]
+ )
+
diff --git a/repos/system_upgrade/cloudlinux/actors/checkrhnversionoverride/actor.py b/repos/system_upgrade/cloudlinux/actors/checkrhnversionoverride/actor.py
new file mode 100644
index 0000000..d832d34
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/checkrhnversionoverride/actor.py
@@ -0,0 +1,38 @@
+from leapp.actors import Actor
+from leapp import reporting
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+
+class CheckRhnVersionOverride(Actor):
+ """
+ Check if the up2date versionOverride option has not been set.
+ """
+
+ name = 'check_rhn_version_override'
+ consumes = ()
+ produces = ()
+ tags = (ChecksPhaseTag, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ up2date_config = '/etc/sysconfig/rhn/up2date'
+ with open(up2date_config, 'r') as f:
+ config_data = f.readlines()
+ for line in config_data:
+ if line.startswith('versionOverride=') and line != 'versionOverride=':
+ title = 'RHN up2date: versionOverride not empty'
+ summary = ('The RHN config file up2date has a set value of the versionOverride option.'
+ ' This value will get overwritten by the upgrade process, and non-supported values'
+ ' carry a risk of causing issues during the upgrade.')
+ remediation = ('Remove the versionOverride value from the up2date config file'
+ ' before running Leapp again.')
+ reporting.create_report([
+ reporting.Title(title),
+ reporting.Summary(summary),
+ reporting.Severity(reporting.Severity.HIGH),
+ reporting.Tags([reporting.Tags.OS_FACTS]),
+ reporting.Flags([reporting.Flags.INHIBITOR]),
+ reporting.Remediation(hint=remediation),
+ reporting.RelatedResource('file', '/etc/sysconfig/rhn/up2date')
+ ])
diff --git a/repos/system_upgrade/cloudlinux/actors/checkup2dateconfig/actor.py b/repos/system_upgrade/cloudlinux/actors/checkup2dateconfig/actor.py
new file mode 100644
index 0000000..bfc0642
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/checkup2dateconfig/actor.py
@@ -0,0 +1,48 @@
+from leapp.actors import Actor
+from leapp.tags import FirstBootPhaseTag, IPUWorkflowTag
+from leapp import reporting
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+import os
+
+
+class CheckUp2dateConfig(Actor):
+ """
+ Move up2date.rpmnew config to the old one's place
+ """
+
+ name = 'check_up2date_config'
+ consumes = ()
+ produces = ()
+ tags = (FirstBootPhaseTag, IPUWorkflowTag)
+
+ original = '/etc/sysconfig/rhn/up2date'
+ new = original + '.rpmnew'
+
+ @run_on_cloudlinux
+ def process(self):
+ """
+ For some reason we get new .rpmnew file instead of the modified `original`
+ This actor tries to save the old `serverURL` parameter to new config and move new instead of old one
+ """
+ replace, old_lines, new_lines = None, None, None
+ if os.path.exists(self.new):
+ self.log.warning('"%s" config found, trying to replace the old one', self.new)
+ with open(self.original) as o, open(self.new) as n:
+ old_lines = o.readlines()
+ new_lines = n.readlines()
+ for l in old_lines:
+ if l.startswith('serverURL=') and l not in new_lines:
+ replace = l
+ break
+ if replace:
+ for i, line in enumerate(new_lines):
+ if line.startswith('serverURL='):
+ new_lines[i] = replace
+ self.log.warning('"serverURL" parameter will be saved as "%s"', line.strip())
+ break
+ with open(self.original, 'w') as f:
+ f.writelines(new_lines)
+ self.log.info('"%s" config is overwritten by the contents of "%s"', self.original, self.new)
+ os.unlink(self.new)
+ self.log.info('"%s" config deleted', self.new)
diff --git a/repos/system_upgrade/cloudlinux/actors/clearpackageconflicts/actor.py b/repos/system_upgrade/cloudlinux/actors/clearpackageconflicts/actor.py
new file mode 100644
index 0000000..cd6801b
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/clearpackageconflicts/actor.py
@@ -0,0 +1,36 @@
+import os
+import errno
+import shutil
+
+from leapp.actors import Actor
+from leapp.libraries.common.rpms import has_package
+from leapp.models import InstalledRPM
+from leapp.tags import DownloadPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+class ClearPackageConflicts(Actor):
+ """
+ Remove several python package files manually to resolve conflicts between versions of packages to be upgraded.
+ """
+
+ name = 'clear_package_conflicts'
+ consumes = (InstalledRPM,)
+ produces = ()
+ tags = (DownloadPhaseTag.Before, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ problem_packages = ["alt-python37-six", "alt-python37-pytz"]
+ problem_packages_installed = any([has_package(InstalledRPM, pkg) for pkg in problem_packages])
+
+ if problem_packages_installed:
+ problem_dirs = [
+ "/opt/alt/python37/lib/python3.7/site-packages/six-1.15.0-py3.7.egg-info",
+ "/opt/alt/python37/lib/python3.7/site-packages/pytz-2017.2-py3.7.egg-info"]
+ for p_dir in problem_dirs:
+ try:
+ if os.path.isdir(p_dir):
+ shutil.rmtree(p_dir)
+ except OSError as e:
+ if e.errno != errno.ENOENT:
+ raise
diff --git a/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/actor.py b/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/actor.py
new file mode 100644
index 0000000..7b947f9
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/actor.py
@@ -0,0 +1,44 @@
+from leapp import reporting
+from leapp.actors import Actor
+from leapp.reporting import Report
+from leapp.tags import ChecksPhaseTag, IPUWorkflowTag
+
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+from leapp.libraries.actor.detectcontrolpanel import detect_panel, UNKNOWN_NAME
+
+
+class DetectControlPanel(Actor):
+ """
+ Check for a presence of a control panel, and inhibit the upgrade if one is found.
+ """
+
+ name = 'detect_control_panel'
+ consumes = ()
+ produces = (Report,)
+ tags = (ChecksPhaseTag, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ panel = detect_panel()
+
+ if panel:
+ summary_info = "Detected panel: {}".format(panel)
+ if panel == UNKNOWN_NAME:
+ summary_info = "Legacy custom panel script detected in CloudLinux configuration"
+
+ # Block the upgrade on any systems with a panel detected.
+ reporting.create_report([
+ reporting.Title("The upgrade process should not be run on systems with a control panel present."),
+ reporting.Summary(
+ "Systems with a control panel present are not supported at the moment."
+ " No control panels are currently included in the Leapp database, which"
+ " makes loss of functionality after the upgrade extremely likely."
+ " {}.".format(summary_info)),
+ reporting.Severity(reporting.Severity.HIGH),
+ reporting.Tags([
+ reporting.Tags.OS_FACTS
+ ]),
+ reporting.Flags([
+ reporting.Flags.INHIBITOR
+ ])
+ ])
diff --git a/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/libraries/detectcontrolpanel.py b/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/libraries/detectcontrolpanel.py
new file mode 100644
index 0000000..15b797a
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/detectcontrolpanel/libraries/detectcontrolpanel.py
@@ -0,0 +1,68 @@
+import os
+import os.path
+
+from leapp.libraries.stdlib import api
+
+
+CPANEL_NAME = 'cPanel'
+DIRECTADMIN_NAME = 'DirectAdmin'
+PLESK_NAME = 'Plesk'
+ISPMANAGER_NAME = 'ISPManager'
+INTERWORX_NAME = 'InterWorx'
+UNKNOWN_NAME = 'Unknown'
+INTEGRATED_NAME = 'Integrated'
+
+CLSYSCONFIG = '/etc/sysconfig/cloudlinux'
+
+
+def lvectl_custompanel_script():
+ """
+ Retrives custom panel script for lvectl from CL config file
+ :return: Script path or None if script filename wasn't found in config
+ """
+ config_param_name = 'CUSTOM_GETPACKAGE_SCRIPT'
+ try:
+ # Try to determine the custom script name
+ if os.path.exists(CLSYSCONFIG):
+ with open(CLSYSCONFIG, 'r') as f:
+ file_lines = f.readlines()
+ for line in file_lines:
+ line = line.strip()
+ if line.startswith(config_param_name):
+ line_parts = line.split('=')
+ if len(line_parts) == 2 and line_parts[0].strip() == config_param_name:
+ script_name = line_parts[1].strip()
+ if os.path.exists(script_name):
+ return script_name
+ except (OSError, IOError, IndexError):
+ # Ignore errors - what's important is that the script wasn't found
+ pass
+ return None
+
+
+def detect_panel():
+ """
+ This function will try to detect control panels supported by CloudLinux
+ :return: Detected control panel name or None
+ """
+ panel_name = None
+ if os.path.isfile('/opt/cpvendor/etc/integration.ini'):
+ panel_name = INTEGRATED_NAME
+ elif os.path.isfile('/usr/local/cpanel/cpanel'):
+ panel_name = CPANEL_NAME
+ elif os.path.isfile('/usr/local/directadmin/directadmin') or\
+ os.path.isfile('/usr/local/directadmin/custombuild/build'):
+ panel_name = DIRECTADMIN_NAME
+ elif os.path.isfile('/usr/local/psa/version'):
+ panel_name = PLESK_NAME
+ # ispmanager must have:
+ # v5: /usr/local/mgr5/ directory,
+ # v4: /usr/local/ispmgr/bin/ispmgr file
+ elif os.path.isfile('/usr/local/ispmgr/bin/ispmgr') or os.path.isdir('/usr/local/mgr5'):
+ panel_name = ISPMANAGER_NAME
+ elif os.path.isdir('/usr/local/interworx'):
+ panel_name = INTERWORX_NAME
+ # Check if the CL config has a legacy custom script for a control panel
+ elif lvectl_custompanel_script():
+ panel_name = UNKNOWN_NAME
+ return panel_name
diff --git a/repos/system_upgrade/cloudlinux/actors/enableyumspacewalkplugin/actor.py b/repos/system_upgrade/cloudlinux/actors/enableyumspacewalkplugin/actor.py
new file mode 100644
index 0000000..1e353b1
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/enableyumspacewalkplugin/actor.py
@@ -0,0 +1,55 @@
+from leapp.actors import Actor
+from leapp.tags import FirstBootPhaseTag, IPUWorkflowTag
+from leapp import reporting
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+try:
+ # py2
+ import ConfigParser as configparser
+ ParserClass = configparser.SafeConfigParser
+except Exception:
+ # py3
+ import configparser
+ ParserClass = configparser.ConfigParser
+
+
+class EnableYumSpacewalkPlugin(Actor):
+ """
+ Enable yum spacewalk plugin if it's disabled
+ Required for the CLN channel functionality to work properly
+ """
+
+ name = 'enable_yum_spacewalk_plugin'
+ consumes = ()
+ produces = ()
+ tags = (FirstBootPhaseTag, IPUWorkflowTag)
+
+ config = '/etc/yum/pluginconf.d/spacewalk.conf'
+
+ @run_on_cloudlinux
+ def process(self):
+ summary = 'yum spacewalk plugin must be enabled for the CLN channels to work properly. ' \
+ 'Please make sure it is enabled. Default config path is "%s"' % self.config
+ title = None
+
+ parser = ParserClass(allow_no_value=True)
+ try:
+ red = parser.read(self.config)
+ if not red:
+ title = 'yum spacewalk plugin config not found'
+ if parser.get('main', 'enabled') != '1':
+ parser.set('main', 'enabled', '1')
+ with open(self.config, 'w') as f:
+ parser.write(f)
+ self.log.info('yum spacewalk plugin enabled')
+ return
+ except Exception as e:
+ title = 'yum spacewalk plugin config error: %s' % e
+
+ if title:
+ reporting.create_report([
+ reporting.Title(title),
+ reporting.Summary(summary),
+ reporting.Severity(reporting.Severity.MEDIUM),
+ reporting.Tags([reporting.Tags.SANITY])
+ ])
diff --git a/repos/system_upgrade/cloudlinux/actors/resetrhnversionoverride/actor.py b/repos/system_upgrade/cloudlinux/actors/resetrhnversionoverride/actor.py
new file mode 100644
index 0000000..21b2164
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/resetrhnversionoverride/actor.py
@@ -0,0 +1,25 @@
+from leapp.actors import Actor
+from leapp.tags import FinalizationPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+
+class ResetRhnVersionOverride(Actor):
+ """
+ Reset the versionOverride value in the RHN up2date config to empty.
+ """
+
+ name = 'reset_rhn_version_override'
+ consumes = ()
+ produces = ()
+ tags = (FinalizationPhaseTag, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ up2date_config = '/etc/sysconfig/rhn/up2date'
+ with open(up2date_config, 'r') as f:
+ config_data = f.readlines()
+ for line in config_data:
+ if line.startswith('versionOverride='):
+ line = 'versionOverride='
+ with open(up2date_config, 'w') as f:
+ f.writelines(config_data)
diff --git a/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/actor.py b/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/actor.py
new file mode 100644
index 0000000..202e5f7
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/actor.py
@@ -0,0 +1,30 @@
+from leapp.actors import Actor
+from leapp.libraries.actor import scanrolloutrepositories
+from leapp.models import (
+ CustomTargetRepositoryFile,
+ CustomTargetRepository,
+ UsedRepositories
+)
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+
+class ScanRolloutRepositories(Actor):
+ """
+ Scan for repository files associated with the Gradual Rollout System.
+
+ Normally these repositories aren't included into the upgrade, but if one of
+ the packages on the system was installed from them, we can potentially run
+ into problems if ignoring these.
+
+ Only those repositories that had packages installed from them are included.
+ """
+
+ name = 'scan_rollout_repositories'
+ consumes = (UsedRepositories)
+ produces = (CustomTargetRepositoryFile, CustomTargetRepository)
+ tags = (FactsPhaseTag, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ scanrolloutrepositories.process()
diff --git a/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/libraries/scanrolloutrepositories.py b/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/libraries/scanrolloutrepositories.py
new file mode 100644
index 0000000..0a059f1
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/scanrolloutrepositories/libraries/scanrolloutrepositories.py
@@ -0,0 +1,49 @@
+import os
+
+from leapp.models import (
+ CustomTargetRepositoryFile,
+ CustomTargetRepository,
+ UsedRepositories
+)
+from leapp.libraries.stdlib import api
+from leapp.libraries.common import repofileutils
+
+REPO_DIR = '/etc/yum.repos.d'
+ROLLOUT_MARKER = 'rollout'
+CL_MARKERS = ['cloudlinux', 'imunify']
+
+
+def process():
+ used_list = []
+ for used_repos in api.consume(UsedRepositories):
+ for used_repo in used_repos.repositories:
+ used_list.append(used_repo.repository)
+
+ for reponame in os.listdir(REPO_DIR):
+ if ROLLOUT_MARKER not in reponame or not any(mark in reponame for mark in CL_MARKERS):
+ continue
+
+ api.current_logger().debug("Detected a rollout repository file: {}".format(reponame))
+
+ full_repo_path = os.path.join(REPO_DIR, reponame)
+ repofile = repofileutils.parse_repofile(full_repo_path)
+
+ # Ignore the repositories that are enabled, but have no packages installed from them.
+ if not any(repo.repoid in used_list for repo in repofile.data):
+ api.current_logger().debug("No used repositories found in {}, skipping".format(reponame))
+ continue
+ else:
+ api.current_logger().debug("Rollout file {} has used repositories, adding".format(reponame))
+
+ for repo in repofile.data:
+ # Don't enable all the rollout repositories wholesale, some might
+ # be disabled and we want to keep that configuration.
+ if repo.enabled:
+ api.produce(CustomTargetRepository(
+ repoid=repo.repoid,
+ name=repo.name,
+ baseurl=repo.baseurl,
+ enabled=repo.enabled,
+ ))
+
+ api.produce(CustomTargetRepositoryFile(file=full_repo_path))
diff --git a/repos/system_upgrade/cloudlinux/actors/switchclnchannel/actor.py b/repos/system_upgrade/cloudlinux/actors/switchclnchannel/actor.py
new file mode 100644
index 0000000..ae01af1
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/switchclnchannel/actor.py
@@ -0,0 +1,33 @@
+from leapp.actors import Actor
+from leapp.libraries.stdlib import api
+from leapp.tags import DownloadPhaseTag, IPUWorkflowTag
+from leapp.libraries.stdlib import CalledProcessError, run
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+
+class SwitchClnChannel(Actor):
+ """
+ Switch CLN channel from 7 to 8 to be able to download upgrade packages
+ """
+
+ name = 'switch_cln_channel'
+ consumes = ()
+ produces = ()
+ tags = (IPUWorkflowTag, DownloadPhaseTag.Before)
+
+ switch_bin = '/usr/sbin/cln-switch-channel'
+
+ @run_on_cloudlinux
+ def process(self):
+ switch_cmd = [self.switch_bin, '-t', '8', '-o', '-f']
+ yum_clean_cmd = ['yum', 'clean', 'all']
+ update_release_cmd = ['yum', 'update', '-y', 'cloudlinux-release']
+ try:
+ res = run(switch_cmd)
+ self.log.debug('Command "%s" result: %s', switch_cmd, res)
+ res = run(yum_clean_cmd) # required to update the repolist
+ self.log.debug('Command "%s" result: %s', yum_clean_cmd, res)
+ res = run(update_release_cmd)
+ self.log.debug('Command "%s" result: %s', update_release_cmd, res)
+ except OSError as e:
+ api.current_logger().error('Could not call RHN command: Message: %s', str(e), exc_info=True)
diff --git a/repos/system_upgrade/cloudlinux/actors/updatecagefs/actor.py b/repos/system_upgrade/cloudlinux/actors/updatecagefs/actor.py
new file mode 100644
index 0000000..71e3c66
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/actors/updatecagefs/actor.py
@@ -0,0 +1,36 @@
+import os
+
+from leapp.actors import Actor
+from leapp.libraries.stdlib import run, CalledProcessError
+from leapp.reporting import Report, create_report
+from leapp import reporting
+from leapp.tags import FirstBootPhaseTag, IPUWorkflowTag
+from leapp.libraries.common.cllaunch import run_on_cloudlinux
+
+
+class UpdateCagefs(Actor):
+ """
+ Force update of cagefs.
+
+ cagefs should reflect massive changes in system made in previous phases
+ """
+
+ name = 'update_cagefs'
+ consumes = ()
+ produces = ()
+ tags = (FirstBootPhaseTag, IPUWorkflowTag)
+
+ @run_on_cloudlinux
+ def process(self):
+ if os.path.exists('/usr/sbin/cagefsctl'):
+ try:
+ run(['/usr/sbin/cagefsctl', '--force-update'], checked=True)
+ self.log.info('cagefs update was successful')
+ except CalledProcessError as e:
+ # cagefsctl prints errors in stdout
+ self.log.error(e.stdout)
+ self.log.error('Command "cagefsctl --force-update" finished with exit code {}, '
+ 'the filesystem inside cagefs may be out-of-date.\n'
+ 'Check cagefsctl output above and in /var/log/cagefs-update.log, '
+ 'rerun "cagefsctl --force-update" after fixing the issues.'.format(e.exit_code)
+ )
diff --git a/repos/system_upgrade/cloudlinux/libraries/cllaunch.py b/repos/system_upgrade/cloudlinux/libraries/cllaunch.py
new file mode 100644
index 0000000..6cbab5d
--- /dev/null
+++ b/repos/system_upgrade/cloudlinux/libraries/cllaunch.py
@@ -0,0 +1,11 @@
+import functools
+from leapp.libraries.common.config import version
+
+
+def run_on_cloudlinux(func):
+ @functools.wraps(func)
+ def wrapper(*args, **kwargs):
+ if (version.current_version()[0] != "cloudlinux"):
+ return
+ return func(*args, **kwargs)
+ return wrapper
diff --git a/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py b/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
index 3836a0d..cf741d9 100644
--- a/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
+++ b/repos/system_upgrade/common/actors/addupgradebootentry/libraries/addupgradebootentry.py
@@ -17,7 +17,7 @@ def add_boot_entry(configs=None):
'/usr/sbin/grubby',
'--add-kernel', '{0}'.format(kernel_dst_path),
'--initrd', '{0}'.format(initram_dst_path),
- '--title', 'RHEL-Upgrade-Initramfs',
+ '--title', 'ELevate-Upgrade-Initramfs',
'--copy-default',
'--make-default',
'--args', '{DEBUG} enforcing=0 rd.plymouth=0 plymouth.enable=0'.format(DEBUG=debug)
diff --git a/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py b/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
index cc442f8..db60913 100644
--- a/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
+++ b/repos/system_upgrade/common/actors/addupgradebootentry/tests/unit_test_addupgradebootentry.py
@@ -42,7 +42,7 @@ run_args_add = [
'/usr/sbin/grubby',
'--add-kernel', '/abc',
'--initrd', '/def',
- '--title', 'RHEL-Upgrade-Initramfs',
+ '--title', 'ELevate-Upgrade-Initramfs',
'--copy-default',
'--make-default',
'--args',
diff --git a/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py b/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py
new file mode 100644
index 0000000..5284aec
--- /dev/null
+++ b/repos/system_upgrade/common/actors/checkenabledvendorrepos/actor.py
@@ -0,0 +1,53 @@
+from leapp.actors import Actor
+from leapp.libraries.stdlib import api
+from leapp.models import (
+ RepositoriesFacts,
+ VendorSourceRepos,
+ ActiveVendorList,
+)
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+
+
+class CheckEnabledVendorRepos(Actor):
+ """
+ Create a list of vendors whose repositories are present on the system.
+ Only those vendors' configurations (new repositories, PES actions, etc.)
+ will be included in the upgrade process.
+ """
+
+ name = "check_enabled_vendor_repos"
+ consumes = (RepositoriesFacts, VendorSourceRepos)
+ produces = (ActiveVendorList)
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
+
+ def process(self):
+ vendor_mapping_data = {}
+ active_vendors = set()
+
+ # Make a dict for easy lookup of repoid -> vendor name.
+ for vendor_src_repodata in api.consume(VendorSourceRepos):
+ for vendor_src_repo in vendor_src_repodata.source_repoids:
+ vendor_mapping_data[vendor_src_repo] = vendor_src_repodata.vendor
+
+ # Is the repo listed in the vendor map as from_repoid present on the system?
+ for repos in api.consume(RepositoriesFacts):
+ for repo_file in repos.repositories:
+ for repo in repo_file.data:
+ self.log.debug(
+ "Looking for repository {} in vendor maps".format(repo.repoid)
+ )
+ if repo.repoid in vendor_mapping_data:
+ # If the vendor's repository is present in the system, count the vendor as active.
+ new_vendor = vendor_mapping_data[repo.repoid]
+ self.log.debug(
+ "Repository {} found, enabling vendor {}".format(
+ repo.repoid, new_vendor
+ )
+ )
+ active_vendors.add(new_vendor)
+
+ if active_vendors:
+ self.log.debug("Active vendor list: {}".format(active_vendors))
+ api.produce(ActiveVendorList(data=list(active_vendors)))
+ else:
+ self.log.info("No active vendors found, vendor list not generated")
diff --git a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
index 17d6731..ddfbc19 100755
--- a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
+++ b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/85sys-upgrade-redhat/do-upgrade.sh
@@ -8,8 +8,7 @@ fi
type getarg >/dev/null 2>&1 || . /lib/dracut-lib.sh
get_rhel_major_release() {
- local os_version
- os_version=$(grep -o '^VERSION="[0-9][0-9]*\.' /etc/initrd-release | grep -o '[0-9]*')
+ local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*' | grep -o '[0-9]*')
[ -z "$os_version" ] && {
# This should not happen as /etc/initrd-release is supposed to have API
# stability, but check is better than broken system.
@@ -343,4 +342,3 @@ getarg 'rd.break=leapp-logs' && emergency_shell -n upgrade "Break after LEAPP sa
sync
mount -o "remount,$old_opts" "$NEWROOT"
exit $result
-
diff --git a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
index 5cc6fd9..6489c0c 100755
--- a/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
+++ b/repos/system_upgrade/common/actors/commonleappdracutmodules/files/dracut/90sys-upgrade/initrd-system-upgrade-generator
@@ -1,8 +1,8 @@
#!/bin/sh
get_rhel_major_release() {
- _os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*\.' | grep -o '[0-9]*')
- [ -z "$_os_version" ] && {
+ local os_version=$(cat /etc/initrd-release | grep -o '^VERSION="[0-9][0-9]*' | grep -o '[0-9]*')
+ [ -z "$os_version" ] && {
# This should not happen as /etc/initrd-release is supposed to have API
# stability, but check is better than broken system.
warn "Cannot determine the major RHEL version."
diff --git a/repos/system_upgrade/common/actors/efibootorderfix/finalization/actor.py b/repos/system_upgrade/common/actors/efibootorderfix/finalization/actor.py
index f42909f..2728cb4 100644
--- a/repos/system_upgrade/common/actors/efibootorderfix/finalization/actor.py
+++ b/repos/system_upgrade/common/actors/efibootorderfix/finalization/actor.py
@@ -1,17 +1,102 @@
+import os
+import re
+
+from leapp.libraries.stdlib import run, api
from leapp.actors import Actor
-from leapp.libraries.common import efi_reboot_fix
+from leapp.models import InstalledTargetKernelVersion, KernelCmdlineArg, FirmwareFacts, MountEntry
from leapp.tags import FinalizationPhaseTag, IPUWorkflowTag
+from leapp.exceptions import StopActorExecutionError
class EfiFinalizationFix(Actor):
"""
- Adjust EFI boot entry for final reboot
+ Ensure that EFI boot order is updated, which is particularly necessary
+ when upgrading to a different OS distro. Also rebuilds grub config
+ if necessary.
"""
name = 'efi_finalization_fix'
- consumes = ()
+ consumes = (KernelCmdlineArg, InstalledTargetKernelVersion, FirmwareFacts, MountEntry)
produces = ()
tags = (FinalizationPhaseTag, IPUWorkflowTag)
def process(self):
- efi_reboot_fix.maybe_emit_updated_boot_entry()
+ is_system_efi = False
+ ff = next(self.consume(FirmwareFacts), None)
+
+ dirname = {
+ 'AlmaLinux': 'almalinux',
+ 'CentOS Linux': 'centos',
+ 'CentOS Stream': 'centos',
+ 'Oracle Linux Server': 'redhat',
+ 'Red Hat Enterprise Linux': 'redhat',
+ 'Rocky Linux': 'rocky',
+ 'Scientific Linux': 'redhat',
+ 'CloudLinux': 'centos',
+ }
+
+ efi_shimname_dict = {
+ 'x86_64': 'shimx64.efi',
+ 'aarch64': 'shimaa64.efi'
+ }
+
+ def devparts(dev):
+ part = next(re.finditer(r'\d+$', dev)).group(0)
+ dev = dev[:-len(part)]
+ return [dev, part];
+
+ with open('/etc/system-release', 'r') as sr:
+ release_line = next(line for line in sr if 'release' in line)
+ distro = release_line.split(' release ', 1)[0]
+
+ efi_bootentry_label = distro
+ distro_dir = dirname.get(distro, 'default')
+ shim_filename = efi_shimname_dict.get(api.current_actor().configuration.architecture, 'shimx64.efi')
+
+ shim_path = '/boot/efi/EFI/' + distro_dir + '/' + shim_filename
+ grub_cfg_path = '/boot/efi/EFI/' + distro_dir + '/grub.cfg'
+ bootmgr_path = '\\EFI\\' + distro_dir + '\\' + shim_filename
+
+ has_efibootmgr = os.path.exists('/sbin/efibootmgr')
+ has_shim = os.path.exists(shim_path)
+ has_grub_cfg = os.path.exists(grub_cfg_path)
+
+ if not ff:
+ raise StopActorExecutionError(
+ 'Could not identify system firmware',
+ details={'details': 'Actor did not receive FirmwareFacts message.'}
+ )
+
+ if not has_efibootmgr:
+ return
+
+ for fact in self.consume(FirmwareFacts):
+ if fact.firmware == 'efi':
+ is_system_efi = True
+ break
+
+ if is_system_efi and has_shim:
+ efidevlist = []
+ with open('/proc/mounts', 'r') as fp:
+ for line in fp:
+ if '/boot/efi' in line:
+ efidevpath = line.split(' ', 1)[0]
+ efidevpart = efidevpath.split('/')[-1]
+ if os.path.exists('/proc/mdstat'):
+ with open('/proc/mdstat', 'r') as mds:
+ for line in mds:
+ if line.startswith(efidevpart):
+ mddev = line.split(' ')
+ for md in mddev:
+ if '[' in md:
+ efimd = md.split('[', 1)[0]
+ efidp = efidevpath.replace(efidevpart, efimd)
+ efidevlist.append(efidp)
+ if len(efidevlist) == 0:
+ efidevlist.append(efidevpath)
+ for devpath in efidevlist:
+ efidev, efipart = devparts(devpath)
+ run(['/sbin/efibootmgr', '-c', '-d', efidev, '-p', efipart, '-l', bootmgr_path, '-L', efi_bootentry_label])
+
+ if not has_grub_cfg:
+ run(['/sbin/grub2-mkconfig', '-o', grub_cfg_path])
diff --git a/repos/system_upgrade/common/actors/filterrpmtransactionevents/actor.py b/repos/system_upgrade/common/actors/filterrpmtransactionevents/actor.py
index e0d89d9..52f93ef 100644
--- a/repos/system_upgrade/common/actors/filterrpmtransactionevents/actor.py
+++ b/repos/system_upgrade/common/actors/filterrpmtransactionevents/actor.py
@@ -32,6 +32,7 @@ class FilterRpmTransactionTasks(Actor):
to_remove = set()
to_keep = set()
to_upgrade = set()
+ to_reinstall = set()
modules_to_enable = {}
modules_to_reset = {}
for event in self.consume(RpmTransactionTasks, PESRpmTransactionTasks):
@@ -39,13 +40,14 @@ class FilterRpmTransactionTasks(Actor):
to_install.update(event.to_install)
to_remove.update(installed_pkgs.intersection(event.to_remove))
to_keep.update(installed_pkgs.intersection(event.to_keep))
+ to_reinstall.update(installed_pkgs.intersection(event.to_reinstall))
modules_to_enable.update({'{}:{}'.format(m.name, m.stream): m for m in event.modules_to_enable})
modules_to_reset.update({'{}:{}'.format(m.name, m.stream): m for m in event.modules_to_reset})
to_remove.difference_update(to_keep)
# run upgrade for the rest of RH signed pkgs which we do not have rule for
- to_upgrade = installed_pkgs - (to_install | to_remove)
+ to_upgrade = installed_pkgs - (to_install | to_remove | to_reinstall)
self.produce(FilteredRpmTransactionTasks(
local_rpms=list(local_rpms),
@@ -53,5 +55,6 @@ class FilterRpmTransactionTasks(Actor):
to_remove=list(to_remove),
to_keep=list(to_keep),
to_upgrade=list(to_upgrade),
+ to_reinstall=list(to_reinstall),
modules_to_reset=list(modules_to_reset.values()),
modules_to_enable=list(modules_to_enable.values())))
diff --git a/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py b/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
index edf978f..7fea4ec 100644
--- a/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
+++ b/repos/system_upgrade/common/actors/ipuworkflowconfig/libraries/ipuworkflowconfig.py
@@ -47,6 +47,7 @@ def get_os_release(path):
:return: `OSRelease` model if the file can be parsed
:raises: `IOError`
"""
+ os_version = '.'.join(platform.dist()[1].split('.')[:2])
try:
with open(path) as f:
data = dict(l.strip().split('=', 1) for l in f.readlines() if '=' in l)
@@ -55,7 +56,7 @@ def get_os_release(path):
name=data.get('NAME', '').strip('"'),
pretty_name=data.get('PRETTY_NAME', '').strip('"'),
version=data.get('VERSION', '').strip('"'),
- version_id=data.get('VERSION_ID', '').strip('"'),
+ version_id=os_version,
variant=data.get('VARIANT', '').strip('"') or None,
variant_id=data.get('VARIANT_ID', '').strip('"') or None
)
diff --git a/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/actor.py b/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/actor.py
index 52553aa..966f9bd 100644
--- a/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/actor.py
+++ b/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/actor.py
@@ -1,7 +1,7 @@
from leapp import reporting
from leapp.actors import Actor
from leapp.exceptions import StopActorExecutionError
-from leapp.libraries.actor.opensshpermitrootlogincheck import global_value, semantics_changes
+from leapp.libraries.actor.opensshpermitrootlogincheck import global_value, semantics_changes, add_permitrootlogin_conf
from leapp.libraries.common.config.version import get_source_major_version
from leapp.libraries.stdlib import api
from leapp.models import OpenSshConfig, Report
@@ -64,25 +64,32 @@ class OpenSshPermitRootLoginCheck(Actor):
# the configuration file was locally modified, it will not get updated by
# RPM and the user might be locked away from the server with new default
if not config.permit_root_login:
+
+ newconf_res = [reporting.RelatedResource('file', '/etc/ssh/sshd_config.leapp_backup')]
+
create_report([
reporting.Title('Possible problems with remote login using root account'),
reporting.Summary(
- 'OpenSSH configuration file does not explicitly state '
- 'the option PermitRootLogin in sshd_config file, '
- 'which will default in RHEL8 to "prohibit-password".'
+ 'Your OpenSSH configuration file does not explicitly state '
+ 'the option PermitRootLogin in sshd_config file. '
+ 'Its default is "yes" in RHEL7, but will change in '
+ 'RHEL8 to "prohibit-password", which may affect your ability '
+ 'to log onto this machine after the upgrade. '
+ 'To prevent this from occuring, the PermitRootLogin option '
+ 'has been explicity set to "yes" to preserve the default behaivour '
+ 'after migration.'
+ 'The original configuration file has been backed up to'
+ '/etc/ssh/sshd_config.leapp_backup'
),
- reporting.Severity(reporting.Severity.HIGH),
+ reporting.Severity(reporting.Severity.MEDIUM),
reporting.Groups(COMMON_REPORT_TAGS),
reporting.Remediation(
- hint='If you depend on remote root logins using passwords, consider '
- 'setting up a different user for remote administration or adding '
- '"PermitRootLogin yes" to sshd_config. '
- 'If this change is ok for you, add explicit '
- '"PermitRootLogin prohibit-password" to your sshd_config '
- 'to ignore this inhibitor'
+ hint='If you would prefer to configure the root login policy yourself, '
+ 'consider setting the PermitRootLogin option '
+ 'in sshd_config explicitly.'
),
reporting.Groups([reporting.Groups.INHIBITOR])
- ] + COMMON_RESOURCES)
+ ] + COMMON_RESOURCES + newconf_res)
return
# Check if there is at least one PermitRootLogin other than "no"
@@ -90,7 +97,7 @@ class OpenSshPermitRootLoginCheck(Actor):
# This usually means some more complicated setup depending on the
# default value being globally "yes" and being overwritten by this
# match block
- if semantics_changes(config):
+ elif semantics_changes(config):
create_report([
reporting.Title('OpenSSH configured to allow root login'),
reporting.Summary(
@@ -98,7 +105,7 @@ class OpenSshPermitRootLoginCheck(Actor):
'blocks, but not explicitly enabled in global or '
'"Match all" context. This update changes the '
'default to disable root logins using paswords '
- 'so your server migth get inaccessible.'
+ 'so your server might become inaccessible.'
),
reporting.Severity(reporting.Severity.HIGH),
reporting.Groups(COMMON_REPORT_TAGS),
diff --git a/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/libraries/opensshpermitrootlogincheck.py b/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/libraries/opensshpermitrootlogincheck.py
index c223757..31418f4 100644
--- a/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/libraries/opensshpermitrootlogincheck.py
+++ b/repos/system_upgrade/common/actors/opensshpermitrootlogincheck/libraries/opensshpermitrootlogincheck.py
@@ -1,3 +1,7 @@
+import errno
+from leapp.libraries.stdlib import api
+
+
def global_value(config, default):
"""
Find the global value for PermitRootLogin option in sshd_config.
@@ -41,3 +45,30 @@ def semantics_changes(config):
in_match_enabled = True
return config_global_value is None and not in_match_enabled
+
+
+def add_permitrootlogin_conf():
+ CONFIG = '/etc/ssh/sshd_config'
+ CONFIG_BACKUP = '/etc/ssh/sshd_config.leapp_backup'
+ try:
+ with open(CONFIG, 'r') as fd:
+ sshd_config = fd.readlines()
+
+ permit_autoconf = [
+ "# Automatically added by Leapp to preserve RHEL7 default\n",
+ "# behaviour after migration.\n",
+ "# Placed on top of the file to avoid being included into Match blocks.\n",
+ "PermitRootLogin yes\n"
+ "\n",
+ ]
+ permit_autoconf.extend(sshd_config)
+ with open(CONFIG, 'w') as fd:
+ fd.writelines(permit_autoconf)
+ with open(CONFIG_BACKUP, 'w') as fd:
+ fd.writelines(sshd_config)
+
+ except IOError as err:
+ if err.errno != errno.ENOENT:
+ error = 'Failed to open sshd_config: {}'.format(str(err))
+ api.current_logger().error(error)
+ return
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/actor.py b/repos/system_upgrade/common/actors/peseventsscanner/actor.py
index 8b5cbcb..78a4665 100644
--- a/repos/system_upgrade/common/actors/peseventsscanner/actor.py
+++ b/repos/system_upgrade/common/actors/peseventsscanner/actor.py
@@ -1,4 +1,8 @@
+import os
+import os.path
+
from leapp.actors import Actor
+from leapp.libraries.actor.pes_event_parsing import get_pes_events
from leapp.libraries.actor.pes_events_scanner import process
from leapp.models import (
EnabledModules,
@@ -9,11 +13,16 @@ from leapp.models import (
RepositoriesMapping,
RepositoriesSetupTasks,
RHUIInfo,
- RpmTransactionTasks
+ RpmTransactionTasks,
+ ActiveVendorList
)
from leapp.reporting import Report
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+LEAPP_FILES_DIR = "/etc/leapp/files"
+PES_FILE_NAME = 'pes-events.json'
+VENDORS_DIR = "/etc/leapp/files/vendors.d"
+
class PesEventsScanner(Actor):
"""
@@ -32,9 +41,24 @@ class PesEventsScanner(Actor):
RepositoriesMapping,
RHUIInfo,
RpmTransactionTasks,
+ ActiveVendorList,
)
produces = (PESRpmTransactionTasks, RepositoriesSetupTasks, Report)
tags = (IPUWorkflowTag, FactsPhaseTag)
def process(self):
- process()
+ events = get_pes_events(LEAPP_FILES_DIR, PES_FILE_NAME)
+
+ active_vendors = []
+ for vendor_list in self.consume(ActiveVendorList):
+ active_vendors.extend(vendor_list.data)
+
+ pes_json_suffix = "_pes.json"
+ if os.path.isdir(VENDORS_DIR):
+ vendor_pesfiles = list(filter(lambda vfile: pes_json_suffix in vfile, os.listdir(VENDORS_DIR)))
+
+ for pesfile in vendor_pesfiles:
+ if pesfile[:-len(pes_json_suffix)] in active_vendors:
+ events.extend(get_pes_events(VENDORS_DIR, PES_FILE_NAME))
+
+ process(events)
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_event_parsing.py b/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_event_parsing.py
index 3719389..51362dd 100644
--- a/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_event_parsing.py
+++ b/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_event_parsing.py
@@ -57,6 +57,7 @@ class Action(IntEnum):
MERGED = 5
MOVED = 6
RENAMED = 7
+ REINSTALLED = 8
def get_pes_events(pes_json_directory, pes_json_filename):
diff --git a/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_events_scanner.py b/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_events_scanner.py
index 96b6328..14faf63 100644
--- a/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_events_scanner.py
+++ b/repos/system_upgrade/common/actors/peseventsscanner/libraries/pes_events_scanner.py
@@ -4,7 +4,7 @@ from functools import partial
from leapp import reporting
from leapp.exceptions import StopActorExecutionError
from leapp.libraries.actor import peseventsscanner_repomap
-from leapp.libraries.actor.pes_event_parsing import Action, get_pes_events, Package
+from leapp.libraries.actor.pes_event_parsing import Action, Package
from leapp.libraries.common.config import version
from leapp.libraries.stdlib import api
from leapp.libraries.stdlib.config import is_verbose
@@ -128,6 +128,7 @@ def compute_pkg_changes_between_consequent_releases(source_installed_pkgs,
pkgs_to_demodularize):
# Start with the installed packages and modify the set according to release events
target_pkgs = set(source_installed_pkgs)
+ pkgs_to_reinstall = set(source_installed_pkgs)
release_events = [e for e in events if e.to_release == release]
@@ -146,6 +147,8 @@ def compute_pkg_changes_between_consequent_releases(source_installed_pkgs,
# Remove packages with old repositories add packages with the new one
target_pkgs = target_pkgs.difference(event.in_pkgs)
target_pkgs = target_pkgs.union(event.in_pkgs)
+ elif event.action == Action.REINSTALLED:
+ pkgs_to_reinstall = pkgs_to_reinstall.union(event.out_pkgs)
else:
# All other packages have the same semantics - they remove their in_pkgs from the system with given
# from_release and add out_pkgs to the system matching to_release
@@ -160,7 +163,7 @@ def compute_pkg_changes_between_consequent_releases(source_installed_pkgs,
pkgs_to_demodularize = pkgs_to_demodularize.difference(event.in_pkgs)
- return (target_pkgs, pkgs_to_demodularize)
+ return (target_pkgs, pkgs_to_demodularize, pkgs_to_reinstall)
def compute_packages_on_target_system(source_pkgs, events, releases):
@@ -177,18 +180,22 @@ def compute_packages_on_target_system(source_pkgs, events, releases):
did_processing_cross_major_version = True
pkgs_to_demodularize = {pkg for pkg in target_pkgs if pkg.modulestream}
- target_pkgs, pkgs_to_demodularize = compute_pkg_changes_between_consequent_releases(target_pkgs, events,
- release, seen_pkgs,
- pkgs_to_demodularize)
+ (
+ target_pkgs,
+ pkgs_to_demodularize,
+ pkgs_to_reinstall,
+ ) = compute_pkg_changes_between_consequent_releases(
+ target_pkgs, events, release, seen_pkgs, pkgs_to_demodularize
+ )
seen_pkgs = seen_pkgs.union(target_pkgs)
demodularized_pkgs = {Package(pkg.name, pkg.repository, None) for pkg in pkgs_to_demodularize}
demodularized_target_pkgs = target_pkgs.difference(pkgs_to_demodularize).union(demodularized_pkgs)
- return (demodularized_target_pkgs, pkgs_to_demodularize)
+ return (demodularized_target_pkgs, pkgs_to_demodularize, pkgs_to_reinstall)
-def compute_rpm_tasks_from_pkg_set_diff(source_pkgs, target_pkgs, pkgs_to_demodularize):
+def compute_rpm_tasks_from_pkg_set_diff(source_pkgs, target_pkgs, pkgs_to_demodularize, pkgs_to_reinstall):
source_state_pkg_names = {pkg.name for pkg in source_pkgs}
target_state_pkg_names = {pkg.name for pkg in target_pkgs}
@@ -210,6 +217,7 @@ def compute_rpm_tasks_from_pkg_set_diff(source_pkgs, target_pkgs, pkgs_to_demodu
return PESRpmTransactionTasks(to_install=pkgs_to_install,
to_remove=pkgs_to_remove,
+ to_reinstall=pkgs_to_reinstall,
modules_to_enable=modules_to_enable,
modules_to_reset=modules_to_reset)
return None
@@ -417,9 +425,8 @@ def apply_transaction_configuration(source_pkgs):
return source_pkgs_with_conf_applied
-def process():
- # Retrieve data - installed_pkgs, transaction configuration, pes events
- events = get_pes_events('/etc/leapp/files', 'pes-events.json')
+def process(events):
+ # Retrieve data - installed_pkgs, transaction configuration, etc. from pes events
releases = get_relevant_releases(events)
source_pkgs = get_installed_pkgs()
source_pkgs = apply_transaction_configuration(source_pkgs)
@@ -429,7 +436,7 @@ def process():
repoids_of_source_pkgs = {pkg.repository for pkg in source_pkgs}
# Apply events - compute what packages should the target system have
- target_pkgs, pkgs_to_demodularize = compute_packages_on_target_system(source_pkgs, events, releases)
+ target_pkgs, pkgs_to_demodularize, pkgs_to_reinstall = compute_packages_on_target_system(source_pkgs, events, releases)
# Packages coming out of the events have PESID as their repository, however, we need real repoid
target_pkgs = replace_pesids_with_repoids_in_packages(target_pkgs, repoids_of_source_pkgs)
@@ -443,6 +450,6 @@ def process():
api.produce(repos_to_enable)
# Compare the packages on source system and the computed packages on target system and determine what to install
- rpm_tasks = compute_rpm_tasks_from_pkg_set_diff(source_pkgs, target_pkgs, pkgs_to_demodularize)
+ rpm_tasks = compute_rpm_tasks_from_pkg_set_diff(source_pkgs, target_pkgs, pkgs_to_demodularize, pkgs_to_reinstall)
if rpm_tasks:
api.produce(rpm_tasks)
diff --git a/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py b/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
index dd6db7c..21531d8 100644
--- a/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
+++ b/repos/system_upgrade/common/actors/redhatsignedrpmscanner/actor.py
@@ -1,27 +1,67 @@
from leapp.actors import Actor
from leapp.libraries.common import rhui
-from leapp.models import InstalledRedHatSignedRPM, InstalledRPM, InstalledUnsignedRPM
+from leapp.models import InstalledRedHatSignedRPM, InstalledRPM, InstalledUnsignedRPM, VendorSignatures
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
-class RedHatSignedRpmScanner(Actor):
+VENDOR_SIGS = {
+ 'rhel': ['199e2f91fd431d51',
+ '5326810137017186',
+ '938a80caf21541eb',
+ 'fd372689897da07a',
+ '45689c882fa658e0'],
+ 'centos': ['24c6a8a7f4a80eb5',
+ '05b555b38483c65d',
+ '4eb84e71f2ee9d55'],
+ 'cloudlinux': ['8c55a6628608cb71'],
+ 'almalinux': ['51d6647ec21ad6ea',
+ 'd36cb86cb86b3716'],
+ 'rocky': ['15af5dac6d745a60',
+ '702d426d350d275d'],
+ 'ol': ['72f97b74ec551f03',
+ '82562ea9ad986da3',
+ 'bc4d06a08d8b756f'],
+ 'eurolinux': ['75c333f418cd4a9e',
+ 'b413acad6275f250',
+ 'f7ad3e5a1c9fd080'],
+ 'scientific': ['b0b4183f192a7d7d']
+}
+
+VENDOR_PACKAGERS = {
+ "rhel": "Red Hat, Inc.",
+ "centos": "CentOS",
+ "cloudlinux": "CloudLinux Packaging Team",
+ "almalinux": "AlmaLinux Packaging Team",
+ "rocky": "infrastructure@rockylinux.org",
+ "eurolinux": "EuroLinux",
+ "scientific": "Scientific Linux",
+}
+
+
+class VendorSignedRpmScanner(Actor):
"""Provide data about installed RPM Packages signed by Red Hat.
After filtering the list of installed RPM packages by signature, a message
with relevant data will be produced.
"""
- name = 'red_hat_signed_rpm_scanner'
- consumes = (InstalledRPM,)
- produces = (InstalledRedHatSignedRPM, InstalledUnsignedRPM,)
+ name = "vendor_signed_rpm_scanner"
+ consumes = (InstalledRPM, VendorSignatures)
+ produces = (
+ InstalledRedHatSignedRPM,
+ InstalledUnsignedRPM,
+ )
tags = (IPUWorkflowTag, FactsPhaseTag)
def process(self):
- RH_SIGS = ['199e2f91fd431d51',
- '5326810137017186',
- '938a80caf21541eb',
- 'fd372689897da07a',
- '45689c882fa658e0']
+ vendor = self.configuration.os_release.release_id
+ vendor_keys = sum(VENDOR_SIGS.values(), [])
+ vendor_packager = VENDOR_PACKAGERS.get(vendor, "not-available")
+
+ for siglist in self.consume(VendorSignatures):
+ vendor_keys.extend(siglist.sigs)
+
+ self.log.debug("Signature list: {}".format(vendor_keys))
signed_pkgs = InstalledRedHatSignedRPM()
unsigned_pkgs = InstalledUnsignedRPM()
@@ -32,11 +72,11 @@ class RedHatSignedRpmScanner(Actor):
all_signed = [
env
for env in env_vars
- if env.name == 'LEAPP_DEVEL_RPMS_ALL_SIGNED' and env.value == '1'
+ if env.name == "LEAPP_DEVEL_RPMS_ALL_SIGNED" and env.value == "1"
]
- def has_rhsig(pkg):
- return any(key in pkg.pgpsig for key in RH_SIGS)
+ def has_vendorsig(pkg):
+ return any(key in pkg.pgpsig for key in vendor_keys)
def is_gpg_pubkey(pkg):
"""Check if gpg-pubkey pkg exists or LEAPP_DEVEL_RPMS_ALL_SIGNED=1
@@ -44,15 +84,30 @@ class RedHatSignedRpmScanner(Actor):
gpg-pubkey is not signed as it would require another package
to verify its signature
"""
- return ( # pylint: disable-msg=consider-using-ternary
- pkg.name == 'gpg-pubkey'
- and pkg.packager.startswith('Red Hat, Inc.')
- or all_signed
+ return ( # pylint: disable-msg=consider-using-ternary
+ pkg.name == "gpg-pubkey"
+ # and (pkg.packager.startswith(vendor_packager))
+ or all_signed
)
def has_katello_prefix(pkg):
"""Whitelist the katello package."""
- return pkg.name.startswith('katello-ca-consumer')
+ return pkg.name.startswith("katello-ca-consumer")
+
+ def has_cpanel_prefix(pkg):
+ """
+ Whitelist the cPanel packages.
+ A side effect of the cPanel's deployment method is that its packages both have no
+ PGP signature and aren't associated with any package repository.
+ They do, however, have a specific naming scheme that can be used to include them into
+ the upgrade process.
+ """
+
+ # NOTE: if another case like this and the above katello occurs, consider adding a
+ # mechanism (a third-party extension) to do this in a way that allows extending it to
+ # other configurations.
+ # A separate file for the "vendors.d" folder with package name wildcards?
+ return pkg.name.startswith("cpanel-")
upg_path = rhui.get_upg_path()
# AWS RHUI packages do not have to be whitelisted because they are signed by RedHat
@@ -68,16 +123,25 @@ class RedHatSignedRpmScanner(Actor):
for pkg in rpm_pkgs.items:
if any(
[
- has_rhsig(pkg),
+ has_vendorsig(pkg),
is_gpg_pubkey(pkg),
has_katello_prefix(pkg),
+ has_cpanel_prefix(pkg),
pkg.name in whitelisted_cloud_pkgs,
]
):
signed_pkgs.items.append(pkg)
+ self.log.debug(
+ "Package {} is signed, packager: {}, signature: {}".format(
+ pkg.name, pkg.packager, pkg.pgpsig
+ )
+ )
continue
unsigned_pkgs.items.append(pkg)
+ self.log.debug(
+ "Package {} is unsigned, packager: {}, signature: {}".format(pkg.name, pkg.packager, pkg.pgpsig)
+ )
self.produce(signed_pkgs)
self.produce(unsigned_pkgs)
diff --git a/repos/system_upgrade/common/actors/repositoriesblacklist/libraries/repositoriesblacklist.py b/repos/system_upgrade/common/actors/repositoriesblacklist/libraries/repositoriesblacklist.py
index b527561..44203bc 100644
--- a/repos/system_upgrade/common/actors/repositoriesblacklist/libraries/repositoriesblacklist.py
+++ b/repos/system_upgrade/common/actors/repositoriesblacklist/libraries/repositoriesblacklist.py
@@ -137,7 +137,18 @@ def process():
E.g. CRB repository is provided by Red Hat but it is without the support.
"""
- repo_mapping = next(api.consume(RepositoriesMapping), None)
+ composite_maps = []
+ composite_repositories = []
+
+ for mapping in api.consume(RepositoriesMapping):
+ composite_maps.extend(mapping.mapping)
+ composite_repositories.extend(mapping.repositories)
+
+ repo_mapping = RepositoriesMapping(
+ mapping=composite_maps,
+ repositories=composite_repositories
+ )
+
repos_facts = next(api.consume(RepositoriesFacts), None)
# Handle required messages not received
diff --git a/repos/system_upgrade/common/actors/repositoriesmapping/libraries/repositoriesmapping.py b/repos/system_upgrade/common/actors/repositoriesmapping/libraries/repositoriesmapping.py
index 2a7c320..b81f500 100644
--- a/repos/system_upgrade/common/actors/repositoriesmapping/libraries/repositoriesmapping.py
+++ b/repos/system_upgrade/common/actors/repositoriesmapping/libraries/repositoriesmapping.py
@@ -1,12 +1,10 @@
-import json
import os
from collections import defaultdict
-from leapp.exceptions import StopActorExecutionError
-from leapp.libraries.common.config.version import get_source_major_version, get_target_major_version
-from leapp.libraries.common.fetch import read_or_fetch
+from leapp.libraries.common.config.version import get_target_major_version, get_source_major_version
+from leapp.libraries.common.repomaputils import RepoMapData, read_repofile, inhibit_upgrade
from leapp.libraries.stdlib import api
-from leapp.models import PESIDRepositoryEntry, RepoMapEntry, RepositoriesMapping
+from leapp.models import RepositoriesMapping
from leapp.models.fields import ModelViolationError
OLD_REPOMAP_FILE = 'repomap.csv'
@@ -16,144 +14,9 @@ REPOMAP_FILE = 'repomap.json'
"""The name of the new repository mapping file."""
-class RepoMapData(object):
- VERSION_FORMAT = '1.0.0'
-
- def __init__(self):
- self.repositories = []
- self.mapping = {}
-
- def add_repository(self, data, pesid):
- """
- Add new PESIDRepositoryEntry with given pesid from the provided dictionary.
-
- :param data: A dict containing the data of the added repository. The dictionary structure corresponds
- to the repositories entries in the repository mapping JSON schema.
- :type data: Dict[str, str]
- :param pesid: PES id of the repository family that the newly added repository belongs to.
- :type pesid: str
- """
- self.repositories.append(PESIDRepositoryEntry(
- repoid=data['repoid'],
- channel=data['channel'],
- rhui=data.get('rhui', ''),
- repo_type=data['repo_type'],
- arch=data['arch'],
- major_version=data['major_version'],
- pesid=pesid
- ))
-
- def get_repositories(self, valid_major_versions):
- """
- Return the list of PESIDRepositoryEntry object matching the specified major versions.
- """
- return [repo for repo in self.repositories if repo.major_version in valid_major_versions]
-
- def add_mapping(self, source_major_version, target_major_version, source_pesid, target_pesid):
- """
- Add a new mapping entry that is mapping the source pesid to the destination pesid(s),
- relevant in an IPU from the supplied source major version to the supplied target
- major version.
-
- :param str source_major_version: Specifies the major version of the source system
- for which the added mapping applies.
- :param str target_major_version: Specifies the major version of the target system
- for which the added mapping applies.
- :param str source_pesid: PESID of the source repository.
- :param Union[str|List[str]] target_pesid: A single target PESID or a list of target
- PESIDs of the added mapping.
- """
- # NOTE: it could be more simple, but I prefer to be sure the input data
- # contains just one map per source PESID.
- key = '{}:{}'.format(source_major_version, target_major_version)
- rmap = self.mapping.get(key, defaultdict(set))
- self.mapping[key] = rmap
- if isinstance(target_pesid, list):
- rmap[source_pesid].update(target_pesid)
- else:
- rmap[source_pesid].add(target_pesid)
-
- def get_mappings(self, src_major_version, dst_major_version):
- """
- Return the list of RepoMapEntry objects for the specified upgrade path.
-
- IOW, the whole mapping for specified IPU.
- """
- key = '{}:{}'.format(src_major_version, dst_major_version)
- rmap = self.mapping.get(key, None)
- if not rmap:
- return None
- map_list = []
- for src_pesid in sorted(rmap.keys()):
- map_list.append(RepoMapEntry(source=src_pesid, target=sorted(rmap[src_pesid])))
- return map_list
-
- @staticmethod
- def load_from_dict(data):
- if data['version_format'] != RepoMapData.VERSION_FORMAT:
- raise ValueError(
- 'The obtained repomap data has unsupported version of format.'
- ' Get {} required {}'
- .format(data['version_format'], RepoMapData.VERSION_FORMAT)
- )
-
- repomap = RepoMapData()
-
- # Load reposiories
- existing_pesids = set()
- for repo_family in data['repositories']:
- existing_pesids.add(repo_family['pesid'])
- for repo in repo_family['entries']:
- repomap.add_repository(repo, repo_family['pesid'])
-
- # Load mappings
- for mapping in data['mapping']:
- for entry in mapping['entries']:
- if not isinstance(entry['target'], list):
- raise ValueError(
- 'The target field of a mapping entry is not a list: {}'
- .format(entry)
- )
-
- for pesid in [entry['source']] + entry['target']:
- if pesid not in existing_pesids:
- raise ValueError(
- 'The {} pesid is not related to any repository.'
- .format(pesid)
- )
- repomap.add_mapping(
- source_major_version=mapping['source_major_version'],
- target_major_version=mapping['target_major_version'],
- source_pesid=entry['source'],
- target_pesid=entry['target'],
- )
- return repomap
-
-
-def _inhibit_upgrade(msg):
- raise StopActorExecutionError(
- msg,
- details={'hint': ('Read documentation at the following link for more'
- ' information about how to retrieve the valid file:'
- ' https://access.redhat.com/articles/3664871')})
-
-
-def _read_repofile(repofile):
- # NOTE: what about catch StopActorExecution error when the file cannot be
- # obtained -> then check whether old_repomap file exists and in such a case
- # inform user they have to provde the new repomap.json file (we have the
- # warning now only which could be potentially overlooked)
- try:
- return json.loads(read_or_fetch(repofile))
- except ValueError:
- # The data does not contain a valid json
- _inhibit_upgrade('The repository mapping file is invalid: file does not contain a valid JSON object.')
- return None # Avoids inconsistent-return-statements warning
-
-
-def scan_repositories(read_repofile_func=_read_repofile):
+def scan_repositories(read_repofile_func=read_repofile):
"""
- Scan the repository mapping file and produce RepositoriesMap msg.
+ Scan the repository mapping file and produce RepositoriesMapping msg.
See the description of the actor for more details.
"""
@@ -185,10 +48,10 @@ def scan_repositories(read_repofile_func=_read_repofile):
'the JSON does not match required schema (wrong field type/value): {}'
.format(err)
)
- _inhibit_upgrade(err_message)
+ inhibit_upgrade(err_message)
except KeyError as err:
- _inhibit_upgrade(
+ inhibit_upgrade(
'The repository mapping file is invalid: the JSON is missing a required field: {}'.format(err))
except ValueError as err:
# The error should contain enough information, so we do not need to clarify it further
- _inhibit_upgrade('The repository mapping file is invalid: {}'.format(err))
+ inhibit_upgrade('The repository mapping file is invalid: {}'.format(err))
diff --git a/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py b/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
index b41693d..e18c554 100644
--- a/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
+++ b/repos/system_upgrade/common/actors/repositoriesmapping/tests/unit_test_repositoriesmapping.py
@@ -14,7 +14,6 @@ from leapp.models import PESIDRepositoryEntry
CUR_DIR = os.path.dirname(os.path.abspath(__file__))
-
@pytest.fixture
def adjust_cwd():
previous_cwd = os.getcwd()
diff --git a/repos/system_upgrade/common/actors/rpmtransactionconfigtaskscollector/libraries/rpmtransactionconfigtaskscollector.py b/repos/system_upgrade/common/actors/rpmtransactionconfigtaskscollector/libraries/rpmtransactionconfigtaskscollector.py
index fb6ae8f..70f0738 100644
--- a/repos/system_upgrade/common/actors/rpmtransactionconfigtaskscollector/libraries/rpmtransactionconfigtaskscollector.py
+++ b/repos/system_upgrade/common/actors/rpmtransactionconfigtaskscollector/libraries/rpmtransactionconfigtaskscollector.py
@@ -18,21 +18,37 @@ def load_tasks_file(path, logger):
return []
+def filter_out(installed_rpm_names, to_filter, debug_msg):
+ # These are the packages that aren't installed on the system.
+ filtered_ok = [pkg for pkg in to_filter if pkg not in installed_rpm_names]
+
+ # And these ones are the ones that are.
+ filtered_out = list(set(to_filter) - set(filtered_ok))
+ if filtered_out:
+ api.current_logger().debug(
+ debug_msg +
+ '\n- ' + '\n- '.join(filtered_out)
+ )
+ # We may want to use either of the two sets.
+ return filtered_ok, filtered_out
+
+
def load_tasks(base_dir, logger):
# Loads configuration files to_install, to_keep, and to_remove from the given base directory
rpms = next(api.consume(InstalledRedHatSignedRPM))
rpm_names = [rpm.name for rpm in rpms.items]
+
to_install = load_tasks_file(os.path.join(base_dir, 'to_install'), logger)
+ install_debug_msg = 'The following packages from "to_install" file will be ignored as they are already installed:'
# we do not want to put into rpm transaction what is already installed (it will go to "to_upgrade" bucket)
- to_install_filtered = [pkg for pkg in to_install if pkg not in rpm_names]
+ to_install_filtered, _ = filter_out(rpm_names, to_install, install_debug_msg)
- filtered = set(to_install) - set(to_install_filtered)
- if filtered:
- api.current_logger().debug(
- 'The following packages from "to_install" file will be ignored as they are already installed:'
- '\n- ' + '\n- '.join(filtered))
+ to_reinstall = load_tasks_file(os.path.join(base_dir, 'to_reinstall'), logger)
+ reinstall_debug_msg = 'The following packages from "to_reinstall" file will be ignored as they are not installed:'
+ _, to_reinstall_filtered = filter_out(rpm_names, to_reinstall, reinstall_debug_msg)
return RpmTransactionTasks(
to_install=to_install_filtered,
+ to_reinstall=to_reinstall_filtered,
to_keep=load_tasks_file(os.path.join(base_dir, 'to_keep'), logger),
to_remove=load_tasks_file(os.path.join(base_dir, 'to_remove'), logger))
diff --git a/repos/system_upgrade/common/actors/scancustomrepofile/actor.py b/repos/system_upgrade/common/actors/scancustomrepofile/actor.py
index d46018f..bb49b4e 100644
--- a/repos/system_upgrade/common/actors/scancustomrepofile/actor.py
+++ b/repos/system_upgrade/common/actors/scancustomrepofile/actor.py
@@ -1,6 +1,9 @@
from leapp.actors import Actor
from leapp.libraries.actor import scancustomrepofile
-from leapp.models import CustomTargetRepository, CustomTargetRepositoryFile
+from leapp.models import (
+ CustomTargetRepository,
+ CustomTargetRepositoryFile,
+)
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
@@ -18,7 +21,7 @@ class ScanCustomRepofile(Actor):
If the file doesn't exist, nothing happens.
"""
- name = 'scan_custom_repofile'
+ name = "scan_custom_repofile"
consumes = ()
produces = (CustomTargetRepository, CustomTargetRepositoryFile)
tags = (FactsPhaseTag, IPUWorkflowTag)
diff --git a/repos/system_upgrade/common/actors/scancustomrepofile/libraries/scancustomrepofile.py b/repos/system_upgrade/common/actors/scancustomrepofile/libraries/scancustomrepofile.py
index 1b48689..757ec76 100644
--- a/repos/system_upgrade/common/actors/scancustomrepofile/libraries/scancustomrepofile.py
+++ b/repos/system_upgrade/common/actors/scancustomrepofile/libraries/scancustomrepofile.py
@@ -17,18 +17,27 @@ def process():
"""
if not os.path.isfile(CUSTOM_REPO_PATH):
api.current_logger().debug(
- "The {} file doesn't exist. Nothing to do."
- .format(CUSTOM_REPO_PATH))
+ "The {} file doesn't exist. Nothing to do.".format(CUSTOM_REPO_PATH)
+ )
return
- api.current_logger().info("The {} file exists.".format(CUSTOM_REPO_PATH))
+
repofile = repofileutils.parse_repofile(CUSTOM_REPO_PATH)
if not repofile.data:
+ api.current_logger().info(
+ "The {} file exists, but is empty. Nothing to do.".format(CUSTOM_REPO_PATH)
+ )
return
api.produce(CustomTargetRepositoryFile(file=CUSTOM_REPO_PATH))
+
for repo in repofile.data:
- api.produce(CustomTargetRepository(
- repoid=repo.repoid,
- name=repo.name,
- baseurl=repo.baseurl,
- enabled=repo.enabled,
- ))
+ api.produce(
+ CustomTargetRepository(
+ repoid=repo.repoid,
+ name=repo.name,
+ baseurl=repo.baseurl,
+ enabled=repo.enabled,
+ )
+ )
+ api.current_logger().info(
+ "The {} file exists, custom repositories loaded.".format(CUSTOM_REPO_PATH)
+ )
diff --git a/repos/system_upgrade/common/actors/scancustomrepofile/tests/test_scancustomrepofile.py b/repos/system_upgrade/common/actors/scancustomrepofile/tests/test_scancustomrepofile.py
index 27dec8c..aaec273 100644
--- a/repos/system_upgrade/common/actors/scancustomrepofile/tests/test_scancustomrepofile.py
+++ b/repos/system_upgrade/common/actors/scancustomrepofile/tests/test_scancustomrepofile.py
@@ -4,7 +4,13 @@ from leapp.libraries.actor import scancustomrepofile
from leapp.libraries.common import repofileutils
from leapp.libraries.common.testutils import produce_mocked
from leapp.libraries.stdlib import api
-from leapp.models import CustomTargetRepository, CustomTargetRepositoryFile, RepositoryData, RepositoryFile
+
+from leapp.models import (
+ CustomTargetRepository,
+ CustomTargetRepositoryFile,
+ RepositoryData,
+ RepositoryFile,
+)
_REPODATA = [
RepositoryData(repoid="repo1", name="repo1name", baseurl="repo1url", enabled=True),
@@ -56,7 +62,7 @@ def test_valid_repofile_exists(monkeypatch):
monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
monkeypatch.setattr(api, 'current_logger', LoggerMocked())
scancustomrepofile.process()
- msg = "The {} file exists.".format(scancustomrepofile.CUSTOM_REPO_PATH)
+ msg = "The {} file exists, custom repositories loaded.".format(scancustomrepofile.CUSTOM_REPO_PATH)
assert api.current_logger.infomsg == msg
assert api.produce.called == len(_CUSTOM_REPOS) + 1
assert _CUSTOM_REPO_FILE_MSG in api.produce.model_instances
@@ -72,6 +78,6 @@ def test_empty_repofile_exists(monkeypatch):
monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
monkeypatch.setattr(api, 'current_logger', LoggerMocked())
scancustomrepofile.process()
- msg = "The {} file exists.".format(scancustomrepofile.CUSTOM_REPO_PATH)
+ msg = "The {} file exists, but is empty. Nothing to do.".format(scancustomrepofile.CUSTOM_REPO_PATH)
assert api.current_logger.infomsg == msg
assert not api.produce.called
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py
new file mode 100644
index 0000000..dd27b28
--- /dev/null
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/actor.py
@@ -0,0 +1,27 @@
+from leapp.actors import Actor
+from leapp.libraries.actor import scanvendorrepofiles
+from leapp.models import (
+ CustomTargetRepositoryFile,
+ ActiveVendorList,
+ VendorCustomTargetRepositoryList,
+)
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+from leapp.libraries.stdlib import api
+
+
+class ScanVendorRepofiles(Actor):
+ """
+ Load and produce custom repository data from vendor-provided files.
+ Only those vendors whose source system repoids were found on the system will be included.
+ """
+
+ name = "scan_vendor_repofiles"
+ consumes = ActiveVendorList
+ produces = (
+ CustomTargetRepositoryFile,
+ VendorCustomTargetRepositoryList,
+ )
+ tags = (FactsPhaseTag, IPUWorkflowTag)
+
+ def process(self):
+ scanvendorrepofiles.process()
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py
new file mode 100644
index 0000000..ba74be1
--- /dev/null
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/libraries/scanvendorrepofiles.py
@@ -0,0 +1,72 @@
+import os
+
+from leapp.libraries.common import repofileutils
+from leapp.libraries.stdlib import api
+from leapp.models import (
+ CustomTargetRepository,
+ CustomTargetRepositoryFile,
+ ActiveVendorList,
+ VendorCustomTargetRepositoryList,
+)
+
+
+VENDORS_DIR = "/etc/leapp/files/vendors.d/"
+REPOFILE_SUFFIX = ".repo"
+
+
+def process():
+ """
+ Produce CustomTargetRepository msgs for the vendor repo files inside the
+ <CUSTOM_REPO_DIR>.
+
+ The CustomTargetRepository messages are produced only if a "from" vendor repository
+ listed indide its map matched one of the repositories active on the system.
+ """
+ if not os.path.isdir(VENDORS_DIR):
+ api.current_logger().debug(
+ "The {} directory doesn't exist. Nothing to do.".format(VENDORS_DIR)
+ )
+ return
+
+ for reponame in os.listdir(VENDORS_DIR):
+ if not reponame.endswith(REPOFILE_SUFFIX):
+ continue
+ # Cut the .repo part to get only the name.
+ vendor_name = reponame[:-5]
+
+ active_vendors = []
+ for vendor_list in api.consume(ActiveVendorList):
+ active_vendors.extend(vendor_list.data)
+
+ api.current_logger().debug("Active vendor list: {}".format(active_vendors))
+
+ if vendor_name not in active_vendors:
+ api.current_logger().debug(
+ "Vendor {} not in active list, skipping".format(vendor_name)
+ )
+ continue
+
+ api.current_logger().debug(
+ "Vendor {} found in active list, processing file".format(vendor_name)
+ )
+ full_repo_path = os.path.join(VENDORS_DIR, reponame)
+ repofile = repofileutils.parse_repofile(full_repo_path)
+
+ api.produce(CustomTargetRepositoryFile(file=full_repo_path))
+
+ custom_vendor_repos = [
+ CustomTargetRepository(
+ repoid=repo.repoid,
+ name=repo.name,
+ baseurl=repo.baseurl,
+ enabled=repo.enabled,
+ ) for repo in repofile.data
+ ]
+
+ api.produce(
+ VendorCustomTargetRepositoryList(vendor=vendor_name, repos=custom_vendor_repos)
+ )
+
+ api.current_logger().info(
+ "The {} directory exists, vendor repositories loaded.".format(VENDORS_DIR)
+ )
diff --git a/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py b/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py
new file mode 100644
index 0000000..cb5c7ab
--- /dev/null
+++ b/repos/system_upgrade/common/actors/scanvendorrepofiles/tests/test_scanvendorrepofiles.py
@@ -0,0 +1,131 @@
+import os
+
+from leapp.libraries.actor import scancustomrepofile
+from leapp.libraries.common import repofileutils
+from leapp.libraries.common.testutils import produce_mocked
+from leapp.libraries.stdlib import api
+
+from leapp.models import (CustomTargetRepository, CustomTargetRepositoryFile,
+ RepositoryData, RepositoryFile)
+
+
+_REPODATA = [
+ RepositoryData(repoid="repo1", name="repo1name", baseurl="repo1url", enabled=True),
+ RepositoryData(repoid="repo2", name="repo2name", baseurl="repo2url", enabled=False),
+ RepositoryData(repoid="repo3", name="repo3name", enabled=True),
+ RepositoryData(repoid="repo4", name="repo4name", mirrorlist="mirror4list", enabled=True),
+]
+
+_CUSTOM_REPOS = [
+ CustomTargetRepository(repoid="repo1", name="repo1name", baseurl="repo1url", enabled=True),
+ CustomTargetRepository(repoid="repo2", name="repo2name", baseurl="repo2url", enabled=False),
+ CustomTargetRepository(repoid="repo3", name="repo3name", baseurl=None, enabled=True),
+ CustomTargetRepository(repoid="repo4", name="repo4name", baseurl=None, enabled=True),
+]
+
+_CUSTOM_REPO_FILE_MSG = CustomTargetRepositoryFile(file=scancustomrepofile.CUSTOM_REPO_PATH)
+
+
+_TESTING_REPODATA = [
+ RepositoryData(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
+ RepositoryData(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=False),
+ RepositoryData(repoid="repo3-stable", name="repo3name", enabled=False),
+ RepositoryData(repoid="repo4-testing", name="repo4name", mirrorlist="mirror4list", enabled=True),
+]
+
+_TESTING_CUSTOM_REPOS_STABLE_TARGET = [
+ CustomTargetRepository(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
+ CustomTargetRepository(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=False),
+ CustomTargetRepository(repoid="repo3-stable", name="repo3name", baseurl=None, enabled=False),
+ CustomTargetRepository(repoid="repo4-testing", name="repo4name", baseurl=None, enabled=True),
+]
+
+_TESTING_CUSTOM_REPOS_BETA_TARGET = [
+ CustomTargetRepository(repoid="repo1-stable", name="repo1name", baseurl="repo1url", enabled=True),
+ CustomTargetRepository(repoid="repo2-testing", name="repo2name", baseurl="repo2url", enabled=True),
+ CustomTargetRepository(repoid="repo3-stable", name="repo3name", baseurl=None, enabled=False),
+ CustomTargetRepository(repoid="repo4-testing", name="repo4name", baseurl=None, enabled=True),
+]
+
+_PROCESS_STABLE_TARGET = "stable"
+_PROCESS_BETA_TARGET = "beta"
+
+
+class LoggerMocked(object):
+ def __init__(self):
+ self.infomsg = None
+ self.debugmsg = None
+
+ def info(self, msg):
+ self.infomsg = msg
+
+ def debug(self, msg):
+ self.debugmsg = msg
+
+ def __call__(self):
+ return self
+
+
+def test_no_repofile(monkeypatch):
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: False)
+ monkeypatch.setattr(api, 'produce', produce_mocked())
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
+ scancustomrepofile.process()
+ msg = "The {} file doesn't exist. Nothing to do.".format(scancustomrepofile.CUSTOM_REPO_PATH)
+ assert api.current_logger.debugmsg == msg
+ assert not api.produce.called
+
+
+def test_valid_repofile_exists(monkeypatch):
+ def _mocked_parse_repofile(fpath):
+ return RepositoryFile(file=fpath, data=_REPODATA)
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
+ monkeypatch.setattr(api, 'produce', produce_mocked())
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
+ scancustomrepofile.process()
+ msg = "The {} file exists, custom repositories loaded.".format(scancustomrepofile.CUSTOM_REPO_PATH)
+ assert api.current_logger.infomsg == msg
+ assert api.produce.called == len(_CUSTOM_REPOS) + 1
+ assert _CUSTOM_REPO_FILE_MSG in api.produce.model_instances
+ for crepo in _CUSTOM_REPOS:
+ assert crepo in api.produce.model_instances
+
+
+def test_target_stable_repos(monkeypatch):
+ def _mocked_parse_repofile(fpath):
+ return RepositoryFile(file=fpath, data=_TESTING_REPODATA)
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
+ monkeypatch.setattr(api, 'produce', produce_mocked())
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
+
+ scancustomrepofile.process(_PROCESS_STABLE_TARGET)
+ assert api.produce.called == len(_TESTING_CUSTOM_REPOS_STABLE_TARGET) + 1
+ for crepo in _TESTING_CUSTOM_REPOS_STABLE_TARGET:
+ assert crepo in api.produce.model_instances
+
+
+def test_target_beta_repos(monkeypatch):
+ def _mocked_parse_repofile(fpath):
+ return RepositoryFile(file=fpath, data=_TESTING_REPODATA)
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
+ monkeypatch.setattr(api, 'produce', produce_mocked())
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
+
+ scancustomrepofile.process(_PROCESS_BETA_TARGET)
+ assert api.produce.called == len(_TESTING_CUSTOM_REPOS_BETA_TARGET) + 1
+ for crepo in _TESTING_CUSTOM_REPOS_BETA_TARGET:
+ assert crepo in api.produce.model_instances
+
+
+def test_empty_repofile_exists(monkeypatch):
+ def _mocked_parse_repofile(fpath):
+ return RepositoryFile(file=fpath, data=[])
+ monkeypatch.setattr(os.path, 'isfile', lambda dummy: True)
+ monkeypatch.setattr(api, 'produce', produce_mocked())
+ monkeypatch.setattr(repofileutils, 'parse_repofile', _mocked_parse_repofile)
+ monkeypatch.setattr(api, 'current_logger', LoggerMocked())
+ scancustomrepofile.process()
+ msg = "The {} file exists, but is empty. Nothing to do.".format(scancustomrepofile.CUSTOM_REPO_PATH)
+ assert api.current_logger.infomsg == msg
+ assert not api.produce.called
diff --git a/repos/system_upgrade/common/actors/setuptargetrepos/actor.py b/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
index 00de073..95cedcd 100644
--- a/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
+++ b/repos/system_upgrade/common/actors/setuptargetrepos/actor.py
@@ -9,9 +9,11 @@ from leapp.models import (
RHUIInfo,
SkippedRepositories,
TargetRepositories,
- UsedRepositories
+ UsedRepositories,
+ VendorCustomTargetRepositoryList
)
from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+from leapp.libraries.stdlib import api
class SetupTargetRepos(Actor):
@@ -30,7 +32,8 @@ class SetupTargetRepos(Actor):
RepositoriesFacts,
RepositoriesBlacklisted,
RHUIInfo,
- UsedRepositories)
+ UsedRepositories,
+ VendorCustomTargetRepositoryList)
produces = (TargetRepositories, SkippedRepositories)
tags = (IPUWorkflowTag, FactsPhaseTag)
diff --git a/repos/system_upgrade/common/actors/setuptargetrepos/libraries/setuptargetrepos.py b/repos/system_upgrade/common/actors/setuptargetrepos/libraries/setuptargetrepos.py
index 3f34aed..9428ef6 100644
--- a/repos/system_upgrade/common/actors/setuptargetrepos/libraries/setuptargetrepos.py
+++ b/repos/system_upgrade/common/actors/setuptargetrepos/libraries/setuptargetrepos.py
@@ -12,7 +12,8 @@ from leapp.models import (
RHUIInfo,
SkippedRepositories,
TargetRepositories,
- UsedRepositories
+ UsedRepositories,
+ VendorCustomTargetRepositoryList
)
@@ -58,10 +59,21 @@ def _get_used_repo_dict():
return used
-def _setup_repomap_handler(src_repoids):
- repo_mappig_msg = next(api.consume(RepositoriesMapping), RepositoriesMapping())
+def _setup_repomap_handler(src_repoids, mapping_list):
+ combined_mapping = []
+ combined_repositories = []
+ # Depending on whether there are any vendors present, we might get more than one message.
+ for msg in mapping_list:
+ combined_mapping.extend(msg.mapping)
+ combined_repositories.extend(msg.repositories)
+
+ combined_repomapping = RepositoriesMapping(
+ mapping=combined_mapping,
+ repositories=combined_repositories
+ )
+
rhui_info = next(api.consume(RHUIInfo), RHUIInfo(provider=''))
- repomap = setuptargetrepos_repomap.RepoMapDataHandler(repo_mappig_msg, cloud_provider=rhui_info.provider)
+ repomap = setuptargetrepos_repomap.RepoMapDataHandler(combined_repomapping, cloud_provider=rhui_info.provider)
# TODO(pstodulk): what about skip this completely and keep the default 'ga'..?
default_channels = setuptargetrepos_repomap.get_default_repository_channels(repomap, src_repoids)
repomap.set_default_channels(default_channels)
@@ -77,22 +89,74 @@ def _get_mapped_repoids(repomap, src_repoids):
return mapped_repoids
+def _get_vendor_custom_repos(enabled_repos, mapping_list):
+ # Look at what source repos from the vendor mapping were enabled.
+ # If any of them are in beta, include vendor's custom repos in the list.
+ # Otherwise skip them.
+
+ result = []
+
+ # Build a dict of vendor mappings for easy lookup.
+ map_dict = {mapping.vendor: mapping for mapping in mapping_list if mapping.vendor}
+
+ for vendor_repolist in api.consume(VendorCustomTargetRepositoryList):
+ vendor_repomap = map_dict[vendor_repolist.vendor]
+
+ # Find the beta channel repositories for the vendor.
+ beta_repos = [
+ x.repoid for x in vendor_repomap.repositories if x.channel == "beta"
+ ]
+ api.current_logger().debug(
+ "Vendor {} beta repos: {}".format(vendor_repolist.vendor, beta_repos)
+ )
+
+ # Are any of the beta repos present and enabled on the system?
+ if any(rep in beta_repos for rep in enabled_repos):
+ # If so, use all repos including beta in the upgrade.
+ vendor_repos = vendor_repolist.repos
+ else:
+ # Otherwise filter beta repos out.
+ vendor_repos = [repo for repo in vendor_repolist.repos if repo.repoid not in beta_repos]
+
+ result.extend([CustomTargetRepository(
+ repoid=repo.repoid,
+ name=repo.name,
+ baseurl=repo.baseurl,
+ enabled=repo.enabled,
+ ) for repo in vendor_repos])
+
+ return result
+
+
def process():
# load all data / messages
used_repoids_dict = _get_used_repo_dict()
enabled_repoids = _get_enabled_repoids()
excluded_repoids = _get_blacklisted_repoids()
+
+ mapping_list = list(api.consume(RepositoriesMapping))
+
custom_repos = _get_custom_target_repos()
+ vendor_repos = _get_vendor_custom_repos(enabled_repoids, mapping_list)
+
+ api.current_logger().debug('Custom repos: {}'.format([f.repoid for f in custom_repos]))
+ api.current_logger().debug('Vendor repos: {}'.format([f.repoid for f in vendor_repos]))
+
+ custom_repos.extend(vendor_repos)
+
+ api.current_logger().debug('Used repos: {}'.format(used_repoids_dict.keys()))
+ api.current_logger().debug('Enabled repos: {}'.format(list(enabled_repoids)))
# TODO(pstodulk): isn't that a potential issue that we map just enabled repos
# instead of enabled + used repos??
# initialise basic data
- repomap = _setup_repomap_handler(enabled_repoids)
+ repomap = _setup_repomap_handler(enabled_repoids, mapping_list)
mapped_repoids = _get_mapped_repoids(repomap, enabled_repoids)
skipped_repoids = enabled_repoids & set(used_repoids_dict.keys()) - mapped_repoids
# Now get the info what should be the target RHEL repositories
expected_repos = repomap.get_expected_target_pesid_repos(enabled_repoids)
+ api.current_logger().debug('Expected repos: {}'.format(expected_repos))
target_rhel_repoids = set()
for target_pesid, target_pesidrepo in expected_repos.items():
if not target_pesidrepo:
diff --git a/repos/system_upgrade/common/actors/systemfacts/actor.py b/repos/system_upgrade/common/actors/systemfacts/actor.py
index 59b12c8..85d4a09 100644
--- a/repos/system_upgrade/common/actors/systemfacts/actor.py
+++ b/repos/system_upgrade/common/actors/systemfacts/actor.py
@@ -47,7 +47,7 @@ class SystemFactsActor(Actor):
GrubCfgBios,
Report
)
- tags = (IPUWorkflowTag, FactsPhaseTag,)
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before,)
def process(self):
self.produce(systemfacts.get_sysctls_status())
diff --git a/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py b/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
index ee1aa8f..e0b3bb2 100644
--- a/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
+++ b/repos/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py
@@ -225,6 +225,12 @@ def _prep_repository_access(context, target_userspace):
target_etc = os.path.join(target_userspace, 'etc')
target_yum_repos_d = os.path.join(target_etc, 'yum.repos.d')
backup_yum_repos_d = os.path.join(target_etc, 'yum.repos.d.backup')
+
+ # Copy RHN data independent from RHSM config
+ if os.path.isdir('/etc/sysconfig/rhn'):
+ run(['rm', '-rf', os.path.join(target_etc, 'sysconfig/rhn')])
+ context.copytree_from('/etc/sysconfig/rhn', os.path.join(target_etc, 'sysconfig/rhn'))
+
if not rhsm.skip_rhsm():
run(['rm', '-rf', os.path.join(target_etc, 'pki')])
run(['rm', '-rf', os.path.join(target_etc, 'rhsm')])
@@ -592,6 +598,7 @@ def _install_custom_repofiles(context, custom_repofiles):
"""
for rfile in custom_repofiles:
_dst_path = os.path.join('/etc/yum.repos.d', os.path.basename(rfile.file))
+ api.current_logger().debug("Copying {} to {}".format(rfile.file, _dst_path))
context.copy_to(rfile.file, _dst_path)
diff --git a/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py b/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py
new file mode 100644
index 0000000..e28b880
--- /dev/null
+++ b/repos/system_upgrade/common/actors/vendorreposignaturescanner/actor.py
@@ -0,0 +1,72 @@
+import os
+
+from leapp.actors import Actor
+from leapp.models import VendorSignatures, ActiveVendorList
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+
+
+VENDORS_DIR = "/etc/leapp/files/vendors.d/"
+SIGFILE_SUFFIX = ".sigs"
+
+
+class VendorRepoSignatureScanner(Actor):
+ """
+ Produce VendorSignatures messages for the vendor signature files inside the
+ <VENDORS_DIR>.
+ These messages are used to extend the list of pakcages Leapp will consider
+ signed and will attempt to upgrade.
+
+ The messages are produced only if a "from" vendor repository
+ listed indide its map matched one of the repositories active on the system.
+ """
+
+ name = 'vendor_repo_signature_scanner'
+ consumes = (ActiveVendorList)
+ produces = (VendorSignatures)
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
+
+ def process(self):
+ if not os.path.isdir(VENDORS_DIR):
+ self.log.debug(
+ "The {} directory doesn't exist. Nothing to do.".format(VENDORS_DIR)
+ )
+ return
+
+ for sigfile_name in os.listdir(VENDORS_DIR):
+ if not sigfile_name.endswith(SIGFILE_SUFFIX):
+ continue
+ # Cut the suffix part to get only the name.
+ vendor_name = sigfile_name[:-5]
+
+ active_vendors = []
+ for vendor_list in self.consume(ActiveVendorList):
+ active_vendors.extend(vendor_list.data)
+
+ self.log.debug(
+ "Active vendor list: {}".format(active_vendors)
+ )
+
+ if vendor_name not in active_vendors:
+ self.log.debug(
+ "Vendor {} not in active list, skipping".format(vendor_name)
+ )
+ continue
+
+ self.log.debug(
+ "Vendor {} found in active list, processing file".format(vendor_name)
+ )
+
+ full_sigfile_path = os.path.join(VENDORS_DIR, sigfile_name)
+ with open(full_sigfile_path) as f:
+ signatures = [line for line in f.read().splitlines() if line]
+
+ self.produce(
+ VendorSignatures(
+ vendor=vendor_name,
+ sigs=signatures,
+ )
+ )
+
+ self.log.info(
+ "The {} directory exists, vendor signatures loaded.".format(VENDORS_DIR)
+ )
diff --git a/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py
new file mode 100644
index 0000000..1325647
--- /dev/null
+++ b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/actor.py
@@ -0,0 +1,19 @@
+from leapp.actors import Actor
+# from leapp.libraries.common.repomaputils import scan_vendor_repomaps, VENDOR_REPOMAP_DIR
+from leapp.libraries.actor.vendorrepositoriesmapping import scan_vendor_repomaps
+from leapp.models import VendorSourceRepos, RepositoriesMapping
+from leapp.tags import FactsPhaseTag, IPUWorkflowTag
+
+
+class VendorRepositoriesMapping(Actor):
+ """
+ Scan the vendor repository mapping files and provide the data to other actors.
+ """
+
+ name = "vendor_repositories_mapping"
+ consumes = ()
+ produces = (RepositoriesMapping, VendorSourceRepos,)
+ tags = (IPUWorkflowTag, FactsPhaseTag.Before)
+
+ def process(self):
+ scan_vendor_repomaps()
diff --git a/repos/system_upgrade/common/actors/vendorrepositoriesmapping/libraries/vendorrepositoriesmapping.py b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/libraries/vendorrepositoriesmapping.py
new file mode 100644
index 0000000..ba27cbb
--- /dev/null
+++ b/repos/system_upgrade/common/actors/vendorrepositoriesmapping/libraries/vendorrepositoriesmapping.py
@@ -0,0 +1,76 @@
+import os
+
+from leapp.libraries.common.config.version import get_target_major_version, get_source_major_version
+from leapp.libraries.common.repomaputils import RepoMapData, read_repofile, inhibit_upgrade
+from leapp.libraries.stdlib import api
+from leapp.models import VendorSourceRepos, RepositoriesMapping
+from leapp.models.fields import ModelViolationError
+
+
+VENDORS_DIR = "/etc/leapp/files/vendors.d"
+"""The folder containing the vendor repository mapping files."""
+
+
+def read_repomap_file(repomap_file, read_repofile_func, vendor_name):
+ json_data = read_repofile_func(repomap_file, VENDORS_DIR)
+ try:
+ repomap_data = RepoMapData.load_from_dict(json_data)
+
+ # What repositories associated with the vendor are expected to be present
+ # on a system with the current major version?
+ # We need to know that to know what to look for in currently enabled
+ # system repositories.
+ api.produce(VendorSourceRepos(
+ vendor=vendor_name,
+ source_repoids=repomap_data.get_version_repoids(get_source_major_version())
+ ))
+
+ mapping = repomap_data.get_mappings(get_source_major_version(), get_target_major_version())
+ valid_major_versions = [get_source_major_version(), get_target_major_version()]
+
+ # This RepositoriesMapping message is different from the one produced by the
+ # builtin actor because of the vendor field.
+ # It can be used later to distinguish the messages provided from vendors and the one
+ # from the OS upgrade data.
+ api.produce(RepositoriesMapping(
+ mapping=mapping,
+ repositories=repomap_data.get_repositories(valid_major_versions),
+ vendor=vendor_name
+ ))
+ except ModelViolationError as err:
+ err_message = (
+ 'The repository mapping file is invalid: '
+ 'the JSON does not match required schema (wrong field type/value): {}'
+ .format(err)
+ )
+ inhibit_upgrade(err_message)
+ except KeyError as err:
+ inhibit_upgrade(
+ 'The repository mapping file is invalid: the JSON is missing a required field: {}'.format(err))
+ except ValueError as err:
+ # The error should contain enough information, so we do not need to clarify it further
+ inhibit_upgrade('The repository mapping file is invalid: {}'.format(err))
+
+
+def scan_vendor_repomaps(read_repofile_func=read_repofile):
+ """
+ Scan the repository mapping file and produce RepositoriesMapping msg.
+
+ See the description of the actor for more details.
+ """
+
+ map_json_suffix = "_map.json"
+ if os.path.isdir(VENDORS_DIR):
+ vendor_mapfiles = list(filter(lambda vfile: map_json_suffix in vfile, os.listdir(VENDORS_DIR)))
+
+ for mapfile in vendor_mapfiles:
+ read_repomap_file(mapfile, read_repofile_func, mapfile[:-len(map_json_suffix)])
+ else:
+ api.current_logger().debug(
+ "The {} directory doesn't exist. Nothing to do.".format(VENDORS_DIR)
+ )
+ # vendor_repomap_collection = scan_vendor_repomaps(VENDOR_REPOMAP_DIR)
+ # if vendor_repomap_collection:
+ # self.produce(vendor_repomap_collection)
+ # for repomap in vendor_repomap_collection.maps:
+ # self.produce(repomap)
diff --git a/repos/system_upgrade/common/files/rhel_upgrade.py b/repos/system_upgrade/common/files/rhel_upgrade.py
index 6d6ad75..1219fa3 100644
--- a/repos/system_upgrade/common/files/rhel_upgrade.py
+++ b/repos/system_upgrade/common/files/rhel_upgrade.py
@@ -184,6 +184,7 @@ class RhelUpgradeCommand(dnf.cli.Command):
to_install = self.plugin_data['pkgs_info']['to_install']
to_remove = self.plugin_data['pkgs_info']['to_remove']
to_upgrade = self.plugin_data['pkgs_info']['to_upgrade']
+ to_reinstall = self.plugin_data['pkgs_info']['to_reinstall']
# Modules to enable
self._process_entities(entities=[available_modules_to_enable],
@@ -196,6 +197,9 @@ class RhelUpgradeCommand(dnf.cli.Command):
self._process_entities(entities=to_install, op=self.base.install, entity_name='Package')
# Packages to be upgraded
self._process_entities(entities=to_upgrade, op=self.base.upgrade, entity_name='Package')
+ # Packages to be reinstalled
+ self._process_entities(entities=to_reinstall, op=self.base.reinstall, entity_name='Package')
+
self.base.distro_sync()
if self.opts.tid[0] == 'check':
diff --git a/repos/system_upgrade/common/libraries/config/version.py b/repos/system_upgrade/common/libraries/config/version.py
index e148932..3b75d5d 100644
--- a/repos/system_upgrade/common/libraries/config/version.py
+++ b/repos/system_upgrade/common/libraries/config/version.py
@@ -13,8 +13,8 @@ OP_MAP = {
_SUPPORTED_VERSIONS = {
# Note: 'rhel-alt' is detected when on 'rhel' with kernel 4.x
- '7': {'rhel': ['7.9'], 'rhel-alt': ['7.6'], 'rhel-saphana': ['7.9']},
- '8': {'rhel': ['8.6', '8.7'], 'rhel-saphana': ['8.6']},
+ '7': {'rhel': ['7.9'], 'rhel-alt': ['7.6'], 'rhel-saphana': ['7.9'], 'centos': ['7.9'], 'eurolinux': ['7.9'], 'ol': ['7.9'], 'cloudlinux': ['7.9'], 'scientific': ['7.9']},
+ '8': {'rhel': ['8.5', '8.6', '8.7'], 'rhel-saphana': ['8.6'], 'centos': ['8.5'], 'almalinux': ['8.6', '8.7'], 'eurolinux': ['8.6', '8.7'], 'ol': ['8.6', '8.7'], 'rocky': ['8.6', '8.7']},
}
diff --git a/repos/system_upgrade/common/libraries/dnfplugin.py b/repos/system_upgrade/common/libraries/dnfplugin.py
index 4010e9f..09085c0 100644
--- a/repos/system_upgrade/common/libraries/dnfplugin.py
+++ b/repos/system_upgrade/common/libraries/dnfplugin.py
@@ -4,6 +4,8 @@ import json
import os
import shutil
+import six
+
from leapp.exceptions import StopActorExecutionError
from leapp.libraries.common import dnfconfig, guards, mounting, overlaygen, rhsm, utils
from leapp.libraries.common.config.version import get_target_major_version, get_target_version
@@ -85,6 +87,7 @@ def build_plugin_data(target_repoids, debug, test, tasks, on_aws):
'to_install': tasks.to_install,
'to_remove': tasks.to_remove,
'to_upgrade': tasks.to_upgrade,
+ 'to_reinstall': tasks.to_reinstall,
'modules_to_enable': ['{}:{}'.format(m.name, m.stream) for m in tasks.modules_to_enable],
},
'dnf_conf': {
@@ -213,10 +216,17 @@ def _transaction(context, stage, target_repoids, tasks, plugin_info, test=False,
message='Failed to execute dnf. Reason: {}'.format(str(e))
)
except CalledProcessError as e:
+ err_stdout = e.stdout
+ err_stderr = e.stderr
+ if six.PY2:
+ err_stdout = e.stdout.encode('utf-8', 'xmlcharrefreplace')
+ err_stderr = e.stderr.encode('utf-8', 'xmlcharrefreplace')
+
api.current_logger().error('DNF execution failed: ')
raise StopActorExecutionError(
message='DNF execution failed with non zero exit code.\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}'.format(
- stdout=e.stdout, stderr=e.stderr)
+ stdout=err_stdout, stderr=err_stderr
+ )
)
finally:
if stage == 'check':
diff --git a/repos/system_upgrade/common/libraries/fetch.py b/repos/system_upgrade/common/libraries/fetch.py
index b9f6f2e..925126f 100644
--- a/repos/system_upgrade/common/libraries/fetch.py
+++ b/repos/system_upgrade/common/libraries/fetch.py
@@ -73,7 +73,7 @@ def read_or_fetch(filename, directory="/etc/leapp/files", service=None, allow_em
data = f.read()
if not allow_empty and not data:
_raise_error(local_path, "File {lp} exists but is empty".format(lp=local_path))
- logger.warning("File {lp} successfully read ({l} bytes)".format(lp=local_path, l=len(data)))
+ logger.debug("File {lp} successfully read ({l} bytes)".format(lp=local_path, l=len(data)))
return data
except EnvironmentError:
_raise_error(local_path, "File {lp} exists but couldn't be read".format(lp=local_path))
diff --git a/repos/system_upgrade/common/libraries/repomaputils.py b/repos/system_upgrade/common/libraries/repomaputils.py
new file mode 100644
index 0000000..5c41620
--- /dev/null
+++ b/repos/system_upgrade/common/libraries/repomaputils.py
@@ -0,0 +1,147 @@
+import json
+from collections import defaultdict
+
+from leapp.exceptions import StopActorExecutionError
+from leapp.libraries.common.fetch import read_or_fetch
+from leapp.models import PESIDRepositoryEntry, RepoMapEntry
+
+
+def inhibit_upgrade(msg):
+ raise StopActorExecutionError(
+ msg,
+ details={'hint': ('Read documentation at the following link for more'
+ ' information about how to retrieve the valid file:'
+ ' https://access.redhat.com/articles/3664871')})
+
+
+def read_repofile(repofile, directory="/etc/leapp/files"):
+ # NOTE: what about catch StopActorExecution error when the file cannot be
+ # obtained -> then check whether old_repomap file exists and in such a case
+ # inform user they have to provde the new repomap.json file (we have the
+ # warning now only which could be potentially overlooked)
+ try:
+ return json.loads(read_or_fetch(repofile, directory))
+ except ValueError:
+ # The data does not contain a valid json
+ inhibit_upgrade('The repository mapping file is invalid: file does not contain a valid JSON object.')
+ return None # Avoids inconsistent-return-statements warning
+
+
+class RepoMapData(object):
+ VERSION_FORMAT = '1.0.0'
+
+ def __init__(self):
+ self.repositories = []
+ self.mapping = {}
+
+ def add_repository(self, data, pesid):
+ """
+ Add new PESIDRepositoryEntry with given pesid from the provided dictionary.
+
+ :param data: A dict containing the data of the added repository. The dictionary structure corresponds
+ to the repositories entries in the repository mapping JSON schema.
+ :type data: Dict[str, str]
+ :param pesid: PES id of the repository family that the newly added repository belongs to.
+ :type pesid: str
+ """
+ self.repositories.append(PESIDRepositoryEntry(
+ repoid=data['repoid'],
+ channel=data['channel'],
+ rhui=data.get('rhui', ''),
+ repo_type=data['repo_type'],
+ arch=data['arch'],
+ major_version=data['major_version'],
+ pesid=pesid
+ ))
+
+ def get_repositories(self, valid_major_versions):
+ """
+ Return the list of PESIDRepositoryEntry object matching the specified major versions.
+ """
+ return [repo for repo in self.repositories if repo.major_version in valid_major_versions]
+
+ def get_version_repoids(self, major_version):
+ """
+ Return the list of repository ID strings for repositories matching the specified major version.
+ """
+ return [repo.repoid for repo in self.repositories if repo.major_version == major_version]
+
+ def add_mapping(self, source_major_version, target_major_version, source_pesid, target_pesid):
+ """
+ Add a new mapping entry that is mapping the source pesid to the destination pesid(s),
+ relevant in an IPU from the supplied source major version to the supplied target
+ major version.
+
+ :param str source_major_version: Specifies the major version of the source system
+ for which the added mapping applies.
+ :param str target_major_version: Specifies the major version of the target system
+ for which the added mapping applies.
+ :param str source_pesid: PESID of the source repository.
+ :param Union[str|List[str]] target_pesid: A single target PESID or a list of target
+ PESIDs of the added mapping.
+ """
+ # NOTE: it could be more simple, but I prefer to be sure the input data
+ # contains just one map per source PESID.
+ key = '{}:{}'.format(source_major_version, target_major_version)
+ rmap = self.mapping.get(key, defaultdict(set))
+ self.mapping[key] = rmap
+ if isinstance(target_pesid, list):
+ rmap[source_pesid].update(target_pesid)
+ else:
+ rmap[source_pesid].add(target_pesid)
+
+ def get_mappings(self, src_major_version, dst_major_version):
+ """
+ Return the list of RepoMapEntry objects for the specified upgrade path.
+
+ IOW, the whole mapping for specified IPU.
+ """
+ key = '{}:{}'.format(src_major_version, dst_major_version)
+ rmap = self.mapping.get(key, None)
+ if not rmap:
+ return None
+ map_list = []
+ for src_pesid in sorted(rmap.keys()):
+ map_list.append(RepoMapEntry(source=src_pesid, target=sorted(rmap[src_pesid])))
+ return map_list
+
+ @staticmethod
+ def load_from_dict(data):
+ if data['version_format'] != RepoMapData.VERSION_FORMAT:
+ raise ValueError(
+ 'The obtained repomap data has unsupported version of format.'
+ ' Get {} required {}'
+ .format(data['version_format'], RepoMapData.VERSION_FORMAT)
+ )
+
+ repomap = RepoMapData()
+
+ # Load reposiories
+ existing_pesids = set()
+ for repo_family in data['repositories']:
+ existing_pesids.add(repo_family['pesid'])
+ for repo in repo_family['entries']:
+ repomap.add_repository(repo, repo_family['pesid'])
+
+ # Load mappings
+ for mapping in data['mapping']:
+ for entry in mapping['entries']:
+ if not isinstance(entry['target'], list):
+ raise ValueError(
+ 'The target field of a mapping entry is not a list: {}'
+ .format(entry)
+ )
+
+ for pesid in [entry['source']] + entry['target']:
+ if pesid not in existing_pesids:
+ raise ValueError(
+ 'The {} pesid is not related to any repository.'
+ .format(pesid)
+ )
+ repomap.add_mapping(
+ source_major_version=mapping['source_major_version'],
+ target_major_version=mapping['target_major_version'],
+ source_pesid=entry['source'],
+ target_pesid=entry['target'],
+ )
+ return repomap
diff --git a/repos/system_upgrade/common/libraries/rhsm.py b/repos/system_upgrade/common/libraries/rhsm.py
index 4a5b0eb..9fdec23 100644
--- a/repos/system_upgrade/common/libraries/rhsm.py
+++ b/repos/system_upgrade/common/libraries/rhsm.py
@@ -92,7 +92,7 @@ def _handle_rhsm_exceptions(hint=None):
def skip_rhsm():
"""Check whether we should skip RHSM related code."""
- return get_env('LEAPP_NO_RHSM', '0') == '1'
+ return True
def with_rhsm(f):
diff --git a/repos/system_upgrade/common/models/activevendorlist.py b/repos/system_upgrade/common/models/activevendorlist.py
new file mode 100644
index 0000000..de4056f
--- /dev/null
+++ b/repos/system_upgrade/common/models/activevendorlist.py
@@ -0,0 +1,7 @@
+from leapp.models import Model, fields
+from leapp.topics import VendorTopic
+
+
+class ActiveVendorList(Model):
+ topic = VendorTopic
+ data = fields.List(fields.String())
diff --git a/repos/system_upgrade/common/models/repositoriesmap.py b/repos/system_upgrade/common/models/repositoriesmap.py
index 824c455..8bf2885 100644
--- a/repos/system_upgrade/common/models/repositoriesmap.py
+++ b/repos/system_upgrade/common/models/repositoriesmap.py
@@ -91,3 +91,4 @@ class RepositoriesMapping(Model):
mapping = fields.List(fields.Model(RepoMapEntry), default=[])
repositories = fields.List(fields.Model(PESIDRepositoryEntry), default=[])
+ vendor = fields.Nullable(fields.String())
diff --git a/repos/system_upgrade/common/models/rpmtransactiontasks.py b/repos/system_upgrade/common/models/rpmtransactiontasks.py
index 7e2870d..05d4e94 100644
--- a/repos/system_upgrade/common/models/rpmtransactiontasks.py
+++ b/repos/system_upgrade/common/models/rpmtransactiontasks.py
@@ -10,6 +10,7 @@ class RpmTransactionTasks(Model):
to_keep = fields.List(fields.String(), default=[])
to_remove = fields.List(fields.String(), default=[])
to_upgrade = fields.List(fields.String(), default=[])
+ to_reinstall = fields.List(fields.String(), default=[])
modules_to_enable = fields.List(fields.Model(Module), default=[])
modules_to_reset = fields.List(fields.Model(Module), default=[])
diff --git a/repos/system_upgrade/common/models/targetrepositories.py b/repos/system_upgrade/common/models/targetrepositories.py
index a5a245f..a69cb6f 100644
--- a/repos/system_upgrade/common/models/targetrepositories.py
+++ b/repos/system_upgrade/common/models/targetrepositories.py
@@ -21,6 +21,12 @@ class CustomTargetRepository(TargetRepositoryBase):
enabled = fields.Boolean(default=True)
+class VendorCustomTargetRepositoryList(Model):
+ topic = TransactionTopic
+ vendor = fields.String()
+ repos = fields.List(fields.Model(CustomTargetRepository))
+
+
class TargetRepositories(Model):
topic = TransactionTopic
rhel_repos = fields.List(fields.Model(RHELTargetRepository))
diff --git a/repos/system_upgrade/common/models/vendorsignatures.py b/repos/system_upgrade/common/models/vendorsignatures.py
new file mode 100644
index 0000000..f456aec
--- /dev/null
+++ b/repos/system_upgrade/common/models/vendorsignatures.py
@@ -0,0 +1,8 @@
+from leapp.models import Model, fields
+from leapp.topics import VendorTopic
+
+
+class VendorSignatures(Model):
+ topic = VendorTopic
+ vendor = fields.String()
+ sigs = fields.List(fields.String())
diff --git a/repos/system_upgrade/common/models/vendorsourcerepos.py b/repos/system_upgrade/common/models/vendorsourcerepos.py
new file mode 100644
index 0000000..b7a219b
--- /dev/null
+++ b/repos/system_upgrade/common/models/vendorsourcerepos.py
@@ -0,0 +1,12 @@
+from leapp.models import Model, fields
+from leapp.topics import VendorTopic
+
+
+class VendorSourceRepos(Model):
+ """
+ This model contains the data on all source repositories associated with a specific vendor.
+ Its data is used to determine whether the vendor should be included into the upgrade process.
+ """
+ topic = VendorTopic
+ vendor = fields.String()
+ source_repoids = fields.List(fields.String())
diff --git a/repos/system_upgrade/common/topics/vendortopic.py b/repos/system_upgrade/common/topics/vendortopic.py
new file mode 100644
index 0000000..014b7af
--- /dev/null
+++ b/repos/system_upgrade/common/topics/vendortopic.py
@@ -0,0 +1,5 @@
+from leapp.topics import Topic
+
+
+class VendorTopic(Topic):
+ name = 'vendor_topic'
diff --git a/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/actor.py b/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/actor.py
index 69ca0f0..c488eb0 100644
--- a/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/actor.py
+++ b/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/actor.py
@@ -2,6 +2,7 @@ from leapp.actors import Actor
from leapp.libraries.stdlib import CalledProcessError, run
from leapp.models import NetworkManagerConfig
from leapp.tags import FirstBootPhaseTag, IPUWorkflowTag
+from leapp import reporting
class NetworkManagerUpdateConnections(Actor):
@@ -26,8 +27,22 @@ class NetworkManagerUpdateConnections(Actor):
return
try:
- r = run(['/usr/bin/python3', 'tools/nm-update-client-ids.py'])['stdout']
- self.log.info('Updated client-ids: {}'.format(r))
+ r = run(['/usr/bin/python3', 'tools/nm-update-client-ids.py'])
+ if r['exit_code'] == 79:
+ title = 'NetworkManager connection update failed - PyGObject bindings for NetworkManager not found.'
+ summary = 'When using dhcp=dhclient on Red Hat Enterprise Linux 7, a non-hexadecimal ' \
+ 'client-id (a string) is sent on the wire as is. On Red Hat Enterprise Linux 8, a zero ' \
+ 'byte is prepended to string-only client-ids. If you wish to preserve the RHEL 7 behaviour, ' \
+ 'you may want to convert your client-ids to hexadecimal form manually.'
+ reporting.create_report([
+ reporting.Title(title),
+ reporting.Summary(summary),
+ reporting.Severity(reporting.Severity.MEDIUM),
+ reporting.Tags([reporting.Tags.NETWORK])
+ ])
+ continue
+
+ self.log.info('Updated client-ids: {}'.format(r['stdout']))
except (OSError, CalledProcessError) as e:
self.log.warning('Error calling nm-update-client-ids script: {}'.format(e))
diff --git a/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/tools/nm-update-client-ids.py b/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/tools/nm-update-client-ids.py
index 1c8d70c..9972204 100755
--- a/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/tools/nm-update-client-ids.py
+++ b/repos/system_upgrade/el7toel8/actors/networkmanagerupdateconnections/tools/nm-update-client-ids.py
@@ -4,12 +4,25 @@ import sys
import gi
-gi.require_version('NM', '1.0')
+try:
+ gi.require_version("NM", "1.0")
+except ValueError:
+ # If we're missing NetworkManager-libnm, the script won't function.
+ print(
+ "PyGObject bindings for NetworkManager not found - do you have NetworkManager-libnm installed?"
+ )
+ print(
+ "If you have dhcp=dhclient, you may need to convert your string-formatted client IDs to hexadecimal"
+ "to preserve the format they're sent on the wire with. Otherwise, they will now have a zero byte"
+ "prepended while being sent."
+ )
+ sys.exit(79)
+
from gi.repository import NM # noqa: E402; pylint: disable=wrong-import-position
def is_hexstring(s):
- arr = s.split(':')
+ arr = s.split(":")
for a in arr:
if len(a) != 1 and len(a) != 2:
return False
@@ -22,8 +35,8 @@ def is_hexstring(s):
client = NM.Client.new(None)
if not client:
- print('Cannot create NM client instance')
- sys.exit(0)
+ print("Cannot create NM client instance")
+ sys.exit(79)
processed = 0
changed = 0
@@ -36,15 +49,20 @@ for c in client.get_connections():
client_id = s_ip4.get_dhcp_client_id()
if client_id is not None:
if not is_hexstring(client_id):
- new_client_id = ':'.join(hex(ord(x))[2:] for x in client_id)
+ new_client_id = ":".join(hex(ord(x))[2:] for x in client_id)
s_ip4.set_property(NM.SETTING_IP4_CONFIG_DHCP_CLIENT_ID, new_client_id)
success = c.commit_changes(True, None)
if success:
changed += 1
else:
errors += 1
- print('Connection {}: \'{}\' -> \'{}\' ({})'.format(c.get_uuid(),
- client_id, new_client_id,
- 'OK' if success else 'FAIL'))
+ print(
+ "Connection {}: '{}' -> '{}' ({})".format(
+ c.get_uuid(),
+ client_id,
+ new_client_id,
+ "OK" if success else "FAIL",
+ )
+ )
print("{} processed, {} changed, {} errors".format(processed, changed, errors))
diff --git a/repos/system_upgrade/el7toel8/actors/updateyumvars/actor.py b/repos/system_upgrade/el7toel8/actors/updateyumvars/actor.py
new file mode 100644
index 0000000..6252fba
--- /dev/null
+++ b/repos/system_upgrade/el7toel8/actors/updateyumvars/actor.py
@@ -0,0 +1,18 @@
+from leapp.actors import Actor
+from leapp.libraries.actor import updateyumvars
+from leapp.tags import ThirdPartyApplicationsPhaseTag, IPUWorkflowTag
+
+
+class UpdateYumVars(Actor):
+ """
+ Update the files corresponding to the current major
+ OS version in the /etc/yum/vars folder.
+ """
+
+ name = 'update_yum_vars'
+ consumes = ()
+ produces = ()
+ tags = (ThirdPartyApplicationsPhaseTag, IPUWorkflowTag)
+
+ def process(self):
+ updateyumvars.vars_update()
diff --git a/repos/system_upgrade/el7toel8/actors/updateyumvars/libraries/updateyumvars.py b/repos/system_upgrade/el7toel8/actors/updateyumvars/libraries/updateyumvars.py
new file mode 100644
index 0000000..b77f784
--- /dev/null
+++ b/repos/system_upgrade/el7toel8/actors/updateyumvars/libraries/updateyumvars.py
@@ -0,0 +1,23 @@
+import os
+
+from leapp.libraries.stdlib import api
+
+VAR_FOLDER = "/etc/yum/vars"
+
+
+def vars_update():
+ """ Iterate through and modify the variables. """
+ if not os.path.isdir(VAR_FOLDER):
+ api.current_logger().debug(
+ "The {} directory doesn't exist. Nothing to do.".format(VAR_FOLDER)
+ )
+ return
+
+ for varfile_name in os.listdir(VAR_FOLDER):
+ # cp_centos_major_version contains the current OS' major version.
+ if varfile_name == 'cp_centos_major_version':
+ varfile_path = os.path.join(VAR_FOLDER, varfile_name)
+
+ with open(varfile_path, 'w') as varfile:
+ # Overwrite the value from outdated "7".
+ varfile.write('8')
Reference in New Issue View Git Blame Copy Permalink