rpms/cloud-init
8
1
Fork 1
Code Issues Pull Requests Releases Activity

* Thu Jul 25 2024 Miroslav Rezanina <mrezanin@redhat.com> - 23.4-17

Browse Source 
- ci-fix-Clean-cache-if-no-datasource-fallback-5499.patch [RHEL-49736]
- ci-Support-setting-mirrorlist-in-yum-repository-config-.patch [RHEL-49674]
- Resolves: RHEL-49736
  ([Cloud-init] [RHEL-9] Password reset feature broken with CloudstackDataSource)
- Resolves: RHEL-49674
  (Support setting mirrorlist in yum repository config)
This commit is contained in:
Miroslav Rezanina 2024-07-25 01:35:59 -04:00
parent 8882846305
commit 95070c535c
3 changed files with 404 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

145
ci-Support-setting-mirrorlist-in-yum-repository-config-.patch Normal file
View File

@ -0,0 +1,145 @@
From d27ac077dac3474fea3c7bb1a19afe50ecbcc64d Mon Sep 17 00:00:00 2001
From: Ani Sinha <anisinha@redhat.com>
Date: Thu, 18 Jul 2024 13:36:39 +0530
Subject: [PATCH 2/2] Support setting mirrorlist in yum repository config
(#5522)
RH-Author: xiachen <xiachen@redhat.com>
RH-MergeRequest: 104: Support setting mirrorlist in yum repository config (#5522)
RH-Jira: RHEL-49674
RH-Acked-by: Ani Sinha <anisinha@redhat.com>
RH-Acked-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
RH-Commit: [1/1] 4605a0304c6bed6614990b7194465230afa79778 (xiachen/cloud-init-centos)
'mirrorlist' config can be specified instead or along with 'baseurl' in the yum
repository config. Add support for specifying mirrorlist instead of 'baseurl'.
Fixes GH-5520
Signed-off-by: Ani Sinha <anisinha@redhat.com>
(cherry picked from commit 0b4084374440d2a5a9968129e0460a1a009d9830)
Signed-off-by: Amy Chen <xiachen@redhat.com>
---
cloudinit/config/cc_yum_add_repo.py | 2 +-
.../schemas/schema-cloud-config-v1.json | 10 +++++
doc/examples/cloud-config-yum-repo.txt | 3 +-
.../unittests/config/test_cc_yum_add_repo.py | 40 ++++++++++++++++++-
4 files changed, 52 insertions(+), 3 deletions(-)
diff --git a/cloudinit/config/cc_yum_add_repo.py b/cloudinit/config/cc_yum_add_repo.py
index 4fd66250..3870d24e 100644
--- a/cloudinit/config/cc_yum_add_repo.py
+++ b/cloudinit/config/cc_yum_add_repo.py
@@ -210,7 +210,7 @@ def handle(name: str, cfg: Config, cloud: Cloud, args: list) -> None:
n_repo_config[k] = v
repo_config = n_repo_config
missing_required = 0
- req_fields = ["baseurl", "metalink"]
+ req_fields = ["baseurl", "metalink", "mirrorlist"]
for req_field in req_fields:
if req_field not in repo_config:
missing_required += 1
diff --git a/cloudinit/config/schemas/schema-cloud-config-v1.json b/cloudinit/config/schemas/schema-cloud-config-v1.json
index 4fb7fd93..c5f46f37 100644
--- a/cloudinit/config/schemas/schema-cloud-config-v1.json
+++ b/cloudinit/config/schemas/schema-cloud-config-v1.json
@@ -3447,6 +3447,11 @@
"format": "uri",
"description": "Specifies a URL to a metalink file for the repomd.xml"
},
+ "mirrorlist": {
+ "type": "string",
+ "format": "uri",
+ "description": "Specifies a URL to a file containing a baseurls list"
+ },
"name": {
"type": "string",
"description": "Optional human-readable name of the yum repo."
@@ -3484,6 +3489,11 @@
"required": [
"metalink"
]
+ },
+ {
+ "required": [
+ "mirrorlist"
+ ]
}
]
}
diff --git a/doc/examples/cloud-config-yum-repo.txt b/doc/examples/cloud-config-yum-repo.txt
index 6a4037e2..cee26677 100644
--- a/doc/examples/cloud-config-yum-repo.txt
+++ b/doc/examples/cloud-config-yum-repo.txt
@@ -11,9 +11,10 @@ yum_repos:
# Any repository configuration options
# See: man yum.conf
#
- # At least one of 'baseurl' or 'metalink' is required!
+ # At least one of 'baseurl' or 'metalink' or 'mirrorlist' is required!
baseurl: http://download.fedoraproject.org/pub/epel/testing/5/$basearch
metalink: https://mirrors.fedoraproject.org/metalink?repo=epel-$releasever&arch=$basearch&infra=$infra&content=$contentdir
+ mirrorlist: https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&
enabled: false
failovermethod: priority
gpgcheck: true
diff --git a/tests/unittests/config/test_cc_yum_add_repo.py b/tests/unittests/config/test_cc_yum_add_repo.py
index 1f27d1fb..000792b4 100644
--- a/tests/unittests/config/test_cc_yum_add_repo.py
+++ b/tests/unittests/config/test_cc_yum_add_repo.py
@@ -31,7 +31,8 @@ class TestConfig(helpers.FilesystemMockingTestCase):
"yum_repos": {
"epel-testing": {
"name": "Extra Packages for Enterprise Linux 5 - Testing",
- # At least one of baseurl or metalink must be present.
+ # At least one of baseurl or metalink or mirrorlist
+ # must be present.
# Missing this should cause the repo not to be written
# 'baseurl': 'http://blah.org/pub/epel/testing/5/$barch',
"enabled": False,
@@ -84,6 +85,43 @@ class TestConfig(helpers.FilesystemMockingTestCase):
for k, v in expected[section].items():
self.assertEqual(parser.get(section, k), v)
+ def test_mirrorlist_config(self):
+ cfg = {
+ "yum_repos": {
+ "epel-testing": {
+ "name": "Extra Packages for Enterprise Linux 5 - Testing",
+ "mirrorlist": "http://mirrors.blah.org/metalink?repo=rhel-$releasever",
+ "enabled": False,
+ "gpgcheck": True,
+ "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL",
+ "failovermethod": "priority",
+ },
+ },
+ }
+ self.patchUtils(self.tmp)
+ self.patchOS(self.tmp)
+ cc_yum_add_repo.handle("yum_add_repo", cfg, None, [])
+ contents = util.load_file("/etc/yum.repos.d/epel-testing.repo")
+ parser = configparser.ConfigParser()
+ parser.read_string(contents)
+ expected = {
+ "epel-testing": {
+ "name": "Extra Packages for Enterprise Linux 5 - Testing",
+ "failovermethod": "priority",
+ "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL",
+ "enabled": "0",
+ "mirrorlist": "http://mirrors.blah.org/metalink?repo=rhel-$releasever",
+ "gpgcheck": "1",
+ }
+ }
+ for section in expected:
+ self.assertTrue(
+ parser.has_section(section),
+ "Contains section {0}".format(section),
+ )
+ for k, v in expected[section].items():
+ self.assertEqual(parser.get(section, k), v)
+
def test_write_config(self):
cfg = {
"yum_repos": {
--
2.39.3

246
ci-fix-Clean-cache-if-no-datasource-fallback-5499.patch Normal file
View File

@ -0,0 +1,246 @@
From 914ac26ebd889b1f5cbb13d55fc011e92fc213c6 Mon Sep 17 00:00:00 2001
From: James Falcon <james.falcon@canonical.com>
Date: Thu, 18 Jul 2024 09:04:54 -0400
Subject: [PATCH 1/2] fix: Clean cache if no datasource fallback (#5499)
RH-Author: Ani Sinha <anisinha@redhat.com>
RH-MergeRequest: 103: fix: Clean cache if no datasource fallback (#5499)
RH-Jira: RHEL-49736
RH-Acked-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
RH-Commit: [1/1] 37eacd97f5e60fae2f71d401c528d508d3db517e (anisinha/cloud-init)
9929a00 added the ability to used a cached datasource when none is
found. This was supposed to be per-datasource, but the lack of cache
cleaning got applied universally. This commit makes it so cache will be
cleaned as it was before if fallback isn't implemented in datasource.
Fixes GH-5486
(cherry picked from commit 550c685c98551f65c30832b186fe091721b48477)
Signed-off-by: Ani Sinha <anisinha@redhat.com>
---
cloudinit/stages.py | 1 +
.../assets/DataSourceNoCacheNetworkOnly.py | 23 ++++
.../assets/DataSourceNoCacheWithFallback.py | 29 +++++
.../datasources/test_caching.py | 115 ++++++++++++++++++
tests/integration_tests/instances.py | 4 +-
5 files changed, 171 insertions(+), 1 deletion(-)
create mode 100644 tests/integration_tests/assets/DataSourceNoCacheNetworkOnly.py
create mode 100644 tests/integration_tests/assets/DataSourceNoCacheWithFallback.py
create mode 100644 tests/integration_tests/datasources/test_caching.py
diff --git a/cloudinit/stages.py b/cloudinit/stages.py
index 0b795624..ace94c9a 100644
--- a/cloudinit/stages.py
+++ b/cloudinit/stages.py
@@ -378,6 +378,7 @@ class Init:
ds,
)
else:
+ util.del_file(self.paths.instance_link)
raise e
self.datasource = ds
# Ensure we adjust our path members datasource
diff --git a/tests/integration_tests/assets/DataSourceNoCacheNetworkOnly.py b/tests/integration_tests/assets/DataSourceNoCacheNetworkOnly.py
new file mode 100644
index 00000000..54a7bab3
--- /dev/null
+++ b/tests/integration_tests/assets/DataSourceNoCacheNetworkOnly.py
@@ -0,0 +1,23 @@
+import logging
+
+from cloudinit import sources
+
+LOG = logging.getLogger(__name__)
+
+
+class DataSourceNoCacheNetworkOnly(sources.DataSource):
+ def _get_data(self):
+ LOG.debug("TEST _get_data called")
+ return True
+
+
+datasources = [
+ (
+ DataSourceNoCacheNetworkOnly,
+ (sources.DEP_FILESYSTEM, sources.DEP_NETWORK),
+ ),
+]
+
+
+def get_datasource_list(depends):
+ return sources.list_from_depends(depends, datasources)
diff --git a/tests/integration_tests/assets/DataSourceNoCacheWithFallback.py b/tests/integration_tests/assets/DataSourceNoCacheWithFallback.py
new file mode 100644
index 00000000..fdfc473f
--- /dev/null
+++ b/tests/integration_tests/assets/DataSourceNoCacheWithFallback.py
@@ -0,0 +1,29 @@
+import logging
+import os
+
+from cloudinit import sources
+
+LOG = logging.getLogger(__name__)
+
+
+class DataSourceNoCacheWithFallback(sources.DataSource):
+ def _get_data(self):
+ if os.path.exists("/ci-test-firstboot"):
+ LOG.debug("TEST _get_data called")
+ return True
+ return False
+
+ def check_if_fallback_is_allowed(self):
+ return True
+
+
+datasources = [
+ (
+ DataSourceNoCacheWithFallback,
+ (sources.DEP_FILESYSTEM,),
+ ),
+]
+
+
+def get_datasource_list(depends):
+ return sources.list_from_depends(depends, datasources)
diff --git a/tests/integration_tests/datasources/test_caching.py b/tests/integration_tests/datasources/test_caching.py
new file mode 100644
index 00000000..33e4b671
--- /dev/null
+++ b/tests/integration_tests/datasources/test_caching.py
@@ -0,0 +1,115 @@
+import pytest
+
+from tests.integration_tests import releases, util
+from tests.integration_tests.instances import IntegrationInstance
+
+
+def setup_custom_datasource(client: IntegrationInstance, datasource_name: str):
+ client.write_to_file(
+ "/etc/cloud/cloud.cfg.d/99-imds.cfg",
+ f"datasource_list: [ {datasource_name}, None ]\n"
+ "datasource_pkg_list: [ cisources ]",
+ )
+ assert client.execute(
+ "mkdir -p /usr/lib/python3/dist-packages/cisources"
+ )
+ client.push_file(
+ util.ASSETS_DIR / f"DataSource{datasource_name}.py",
+ "/usr/lib/python3/dist-packages/cisources/"
+ f"DataSource{datasource_name}.py",
+ )
+
+
+def verify_no_cache_boot(client: IntegrationInstance):
+ log = client.read_from_file("/var/log/cloud-init.log")
+ util.verify_ordered_items_in_text(
+ [
+ "No local datasource found",
+ "running 'init'",
+ "no cache found",
+ "Detected platform",
+ "TEST _get_data called",
+ ],
+ text=log,
+ )
+ util.verify_clean_boot(client)
+
+
+@pytest.mark.skipif(
+ not releases.IS_UBUNTU,
+ reason="hardcoded dist-packages directory",
+)
+def test_no_cache_network_only(client: IntegrationInstance):
+ """Test cache removal per boot. GH-5486
+
+ This tests the CloudStack password reset use case. The expectation is:
+ - Metadata is fetched in network timeframe only
+ - Because `check_instance_id` is not defined, no cached datasource
+ is found in the init-local phase, but the cache is used in the
+ remaining phases due to existance of /run/cloud-init/.instance-id
+ - Because `check_if_fallback_is_allowed` is not defined, cloud-init
+ does NOT fall back to the pickled datasource, and will
+ instead delete the cache during the init-local phase
+ - Metadata is therefore fetched every boot in the network phase
+ """
+ setup_custom_datasource(client, "NoCacheNetworkOnly")
+
+ # Run cloud-init as if first boot
+ assert client.execute("cloud-init clean --logs")
+ client.restart()
+
+ verify_no_cache_boot(client)
+
+ # Clear the log without clean and run cloud-init for subsequent boot
+ assert client.execute("echo '' > /var/log/cloud-init.log")
+ client.restart()
+
+ verify_no_cache_boot(client)
+
+
+@pytest.mark.skipif(
+ not releases.IS_UBUNTU,
+ reason="hardcoded dist-packages directory",
+)
+def test_no_cache_with_fallback(client: IntegrationInstance):
+ """Test we use fallback when defined and no cache available."""
+ setup_custom_datasource(client, "NoCacheWithFallback")
+
+ # Run cloud-init as if first boot
+ assert client.execute("cloud-init clean --logs")
+ # Used by custom datasource
+ client.execute("touch /ci-test-firstboot")
+ client.restart()
+
+ log = client.read_from_file("/var/log/cloud-init.log")
+ util.verify_ordered_items_in_text(
+ [
+ "no cache found",
+ "Detected platform",
+ "TEST _get_data called",
+ "running 'init'",
+ "restored from cache with run check",
+ "running 'modules:config'",
+ ],
+ text=log,
+ )
+ util.verify_clean_boot(client)
+
+ # Clear the log without clean and run cloud-init for subsequent boot
+ assert client.execute("echo '' > /var/log/cloud-init.log")
+ client.execute("rm /ci-test-firstboot")
+ client.restart()
+
+ log = client.read_from_file("/var/log/cloud-init.log")
+ util.verify_ordered_items_in_text(
+ [
+ "cache invalid in datasource",
+ "Detected platform",
+ "Restored fallback datasource from checked cache",
+ "running 'init'",
+ "restored from cache with run check",
+ "running 'modules:config'",
+ ],
+ text=log,
+ )
+ util.verify_clean_boot(client)
diff --git a/tests/integration_tests/instances.py b/tests/integration_tests/instances.py
index 3fc6558a..23c0dc98 100644
--- a/tests/integration_tests/instances.py
+++ b/tests/integration_tests/instances.py
@@ -88,7 +88,9 @@ class IntegrationInstance:
# First push to a temporary directory because of permissions issues
tmp_path = _get_tmp_path()
self.instance.push_file(str(local_path), tmp_path)
- assert self.execute("mv {} {}".format(tmp_path, str(remote_path))).ok
+ assert self.execute(
+ "mv {} {}".format(tmp_path, str(remote_path))
+ ), f"Failed to push {tmp_path} to {remote_path}"
def read_from_file(self, remote_path) -> str:
result = self.execute("cat {}".format(remote_path))
--
2.39.3

14
cloud-init.spec
View File

@ -1,6 +1,6 @@
Name: cloud-init
Version: 23.4
Release: 16%{?dist}
Release: 17%{?dist}
Summary: Cloud instance init scripts
License: ASL 2.0 or GPLv3
URL: http://launchpad.net/cloud-init
@ -59,6 +59,10 @@ Patch28: ci-Support-metalink-in-yum-repository-config-5444.patch
Patch29: ci-fix-vmware-Set-IPv6-to-dhcp-when-there-is-no-IPv6-ad.patch
# For RHEL-46873 - Suggest to update schema to support metalink
Patch30: ci-fix-add-schema-rules-for-baseurl-and-metalink-in-yum.patch
# For RHEL-49736 - [Cloud-init] [RHEL-9] Password reset feature broken with CloudstackDataSource
Patch31: ci-fix-Clean-cache-if-no-datasource-fallback-5499.patch
# For RHEL-49674 - Support setting mirrorlist in yum repository config
Patch32: ci-Support-setting-mirrorlist-in-yum-repository-config-.patch
BuildArch: noarch
@ -273,6 +277,14 @@ fi
%config(noreplace) %{_sysconfdir}/rsyslog.d/21-cloudinit.conf
%changelog
* Thu Jul 25 2024 Miroslav Rezanina <mrezanin@redhat.com> - 23.4-17
- ci-fix-Clean-cache-if-no-datasource-fallback-5499.patch [RHEL-49736]
- ci-Support-setting-mirrorlist-in-yum-repository-config-.patch [RHEL-49674]
- Resolves: RHEL-49736
([Cloud-init] [RHEL-9] Password reset feature broken with CloudstackDataSource)
- Resolves: RHEL-49674
(Support setting mirrorlist in yum repository config)
* Fri Jul 12 2024 Miroslav Rezanina <mrezanin@redhat.com> - 23.4-16
- ci-fix-add-schema-rules-for-baseurl-and-metalink-in-yum.patch [RHEL-46873]
- Resolves: RHEL-46873