almalinux
/
pungi
15
1
Fork 0
Code Issues Pull Requests Releases Activity

Update from upstream #11

Closed
soksanichenko wants to merge 158 commits from a8_updated into a8
pull from: a8_updated
merge into: almalinux:a8
Conversation 0 Commits 158 Files Changed 128 +5722 -1204
soksanichenko commented 2022-11-08 13:04:41 +00:00
Owner
Copy Link
- Updates from upstream: - changelog is taken from https://src.fedoraproject.org/rpms/pungi/blob/main/f/pungi.spec - spec is also updated from same URL - sources are taken from https://pagure.io/pungi/tree/master - KojiMock and related modules are updated from upstream code (according existing AL patches) - Unittests are fixed according existing AL patches
soksanichenko added 147 commits 2022-11-08 13:04:42 +00:00
f470599f6c React to SIGINT signal 
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
f518c1bb7c Stop copying .git directory with module defaults 
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
9ea1098eae comps: Preserve default arg on groupid 
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
39b847094a doc: remove default createrepo_checksum value from example 
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
83458f26c2 pkgset: Drop kobo.plugins usage from GatherSources 
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
0f4b0577f7 gather: Drop kobo.plugins usage from GatherMethod 
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
c87fce30ac pkgset: Drop kobo.plugin usage from PkgsetSource 
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
49a5661521 pkgset: Remove reuse file when packages are not signed 
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.

Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
d4ee42ec23 pkgset: Check tag inheritance change before reuse 
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
daa0ca6106 pkgset: Include just one version of module 
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.

JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
44f7eff1b7 Move UnsignedPackagesError to a separate file 
This file can contain all Pungi specific exceptions.

It should also fix an issue encountered on Python 2.7:

    AttributeError: 'module' object has no attribute 'pkgsets'

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
36373479db Move container metadata into compose object 
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
61e90fd7e0 osbs: Move metadata processing to standalone function 
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
40133074b3 Add image-container phase 
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
64897d7d48 pkgset: Add ability to wait for signed packages 
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
98359654cf 4.2.8 release 
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
477dcf37d9 Store extended traceback for gather errors 
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
5b5069175d pkgset: Store module tag only if module is used 
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.

If the packages are not included in any variant, they shouldn't be
available to buildinstall either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
735bfaa0d6 pkgset: Fix meaning of retries 
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
b217470464 Format code 
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2769232b72 runroot: Adjust permissions always 
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
535034ef91 image_container: Fix incorrect arch processing 
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
edb4517e80 Add Dockerfile for building testing image 
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
035b37c566 Cancel koji tasks when pungi terminated 
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
c8091899b2 gather: Copy old logs when reusing gather result 
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
ab1b5b48ec hybrid: Optimize getting lookaside packages 
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
e866d22c04 gather: Adjust reusing with lookaside 
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
da791ed15c Fix can't link XDEV using repos as pkgset_sources 
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.

Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
76d13d0062 4.2.9 release 
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
c27bfe0c59 Clean up temporary yumroot dir 
JIRA: RHELCMP-4948
Signed-off-by: Haibo Lin <hlin@redhat.com>
7fe32ae758 util: Strip file:// from local urls 
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
bf28e8d50c pkgset: Compare future events correctly 
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.

JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
9a5e901cfe Log warning when module defined in variants.xml not found 
JIRA: RHELCMP-5573
Signed-off-by: Haibo Lin <hlin@redhat.com>
edb091b7b1 Add task URL to watch task log 
JIRA: RHELCMP-5666
Signed-off-by: Haibo Lin <hlin@redhat.com>
a435fd58da gather: Add all srpms to variant lookaside repo 
The original code could cause a source RPM to be present in two variants
that have a dependency relation.

There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.

This could result in duplicated SRPMs in the compose.

Example situation:

 * Variant B depends on variant A.
 * A contains foo.x86_64.rpm (only on x86_64)
 * B pulls in subpackage foo-bar.s390x.rpm (on s390x)

Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.

The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.

JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
56a55db966 Use cachedir when createrepo 
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
cf761633f4 4.2.10 release 
JIRA: RHELCMP-6108
Signed-off-by: Haibo Lin <hlin@redhat.com>
01a52447bc doc: explain buildContainer API 
Explain how to discover the API documentation about the buildContainer
method, so users can discover more about how "scratch" and "priority"
work.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
8a2d0162d9 Better error message than 'KeyError' in pungi 
JIRA: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2a679dcb81 doc: improve signed packages retry docs 
Reword the signed_packages_retries and signed_packages_wait
configuration option documentation to use the active voice. This makes
it easier to understand who is doing what in a signing workflow.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
6afcfef919 doc: fix typo in additional_packages description 
not -> nor

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
5a8df7b69c doc: more additional_packages documentation 
Contrast the additional_packages setting with the comps_file setting.

Explain what happens when a user lists a package in additional_packages
but Pungi cannot find it.

Give an example of composing all builds in a Koji tag.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
3349585d78 Adding multithreading support for pungi/phases/image_checksum.py 
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.

Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.

Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
5831d4ae1e Better error message than 'KeyError' in pungi 
Jira: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
a7c111643d Supersede ModuleStream loading with ModuleIndex 
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
efff2c9504 Use pytest directly incl. support for posargs, e.g.: 
tox -- -s -vvv tests/path/to/a/single/test_something.py

Signed-off-by: Filip Valder <fvalder@redhat.com>
1bb038ca72 Install missing deps in ci image 
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
795bbe31e3 Fix formatting 
Signed-off-by: Haibo Lin <hlin@redhat.com>
66dacb21e0 Add createrepo_enable_cache to configuration doc 
JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
980c7ba8fb Handle the pungi failures to ensure creation of log files 
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.

Jira: RHELCMP-6077

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
9cd42a2b5e Formatted files according to flake8 and black feedback 
Signed-off-by: Dominik Rumian <drumian@redhat.com>
7c3e8d4276 Fix tests for createrepo 
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
3d9335e90e Use xorriso instead of isoinfo when createiso_use_xorrisofs is enabled 
Update get_mkisofs_cmd in createiso.py file in order to prevent using
default value. With this change it is possible to enable xorriso format

Jira: RHELCMP-6325

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
b7666ba4a4 Enable pungi to send compose_url patches to CTS 
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.

If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
e8ddacd10e Fix type detection for osbuild images 
The image type value passed to the task doesn't match the type as it
will be recorded by Koji.

JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
904a1c3271 Add authentication for updating the compose URL in CTS. 
Put authentication steps in a function in order to prevent code duplication.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
d8d1cc520b paths: Allow customizing log file extension 
If the file contents is JSON, it would be nice to have matching
extension.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
20dc4beb6b Make getting old compose config reusable 
The file will only be loaded once, it gets cached afterwards.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
195bfbefa4 Allow specifying $COMPOSE_ID in the `repo` value for osbs phase. 
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
5c26aa9127 Require requests_kerberos only when needed 
If CTS integration is not used, let's not import a module that is not
needed.

JIRA: RHELCMP-6611
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
a1ebd234a4 Only build CTS url when configured 
JIRA: RHELCMP-6611
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
72bcee01be 4.3.0 release 
JIRA: RHELCMP-6614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ba6f7429ee buildinstall: Add easy way to check if previous result was reused 
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
9612241396 Add COMPOSE_ID into the pungi log file 
Jira: RHELCMP-6739

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
ac061b2ea8 Work around ODCS creating COMPOSE_ID later 
When ODCS starts a compose, it will provide base composeinfo file, but
it doesn't create COMPOSE_ID. This leads to a crash when updating CTS,
since the compose id can't be read from the file. We can instead use the
value we already have in memory.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
7475d2a3a9 Allow ISO-Level configuration within the config file 
In order to enable this feature set "iso_level=<Value from 1 to 4>"
in config file

Jira: RHELCMP-6880

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
e42e65783d image_build: Allow reusing old image_build results 
JIRA: RHELCMP-5970
Signed-off-by: Haibo Lin <hlin@redhat.com>
8133676270 osbs: Reuse images from old compose 
JIRA: RHELCMP-5972
Signed-off-by: Haibo Lin <hlin@redhat.com>
204d88a351 Add missing mock to osbs tests 
We don't want the test to try to a dummy URL.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ab19043773 Correct irc network name & add matrix room 
Signed-off-by: Dan Čermák <dan.cermak@cgc-instruments.com>
b03490bf18 4.3.1 release 
JIRA: RHELCMP-7116
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
eb61c97cdb Remove default runroot channel 
When the value is not specified in the configuration file, let Koji pick
the default channel.

JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
ac66c3d7f3 createiso: Allow reusing old images 
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:

 * If image is bootable, buildinstall phase must have been reused too.
 * Compose configuration must have not changed (except for a few
   whitelisted options).
 * Volume ID of the ISO much not have changed.
 * No RPM on the ISO must have changed.

The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
e8305f3978 extra_isos: Allow reusing old images 
When nothing in configuration or the image itself changed, let's just
copy the older one.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
e2b3002726 repoclosure: Use --forcearch for dnf repoclosure 
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.

Based on how the list of architectures is generated, we know that the
main one will always be first.

Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
9d02f87c99 Stop trying to validate non-existent metadata 
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
94ffa1c5c6 default "with_jigdo" to False 
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
80bd254347 Check dependencies after config validation 
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
1d654522be Remove with_jigdo argument 
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
9bae86a51e doc: make dnf "backend" settings easier to discover 
Mention the corresponding "gather" or "repoclosure" backend settings in
the documentation for each setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
33d7290d78 gather: Stop requiring all variants/arches in JSON 
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
b652119d54 gather: Load JSON mapping relative to config dir 
JIRA: RHELCMP-7195
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
cfb9882269 4.3.2 release 
JIRA: RHELCMP-7182
Signed-off-by: Haibo Lin <hlin@redhat.com>
f681956cf1 Fix tests for python 2.6 
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
5e6248e3e0 Generate images.json for extra_isos phase 
JIRA: RHELCMP-7241
Signed-off-by: Haibo Lin <hlin@redhat.com>
20c2e59218 Pass compose parameter for debugging git issue 
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
260b3fce8d compose: Make sure temporary dirs are world readable 
When the temporary directory is created with 0700, other programs
(potentially on another host) will have problems reading it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-7635
894cce6a5a Ignore osbs/osbuild config when reusing iso images 
JIRA: RHELCMP-7562
Signed-off-by: Haibo Lin <hlin@redhat.com>
fe986d68b9 Add module obsoletes feature 
JIRA: MODULAR-113
Merges: https://pagure.io/pungi/pull-request/1578
Signed-off-by: Filip Valder <fvalder@redhat.com>
32221e8f36 hybrid: Explicitly pull in debugsource packages 
This should cover case where we there's a build like this:

foo-1-1.src.rpm
  foo-sub-1-1.noarch.rpm
  foo-debugsource-1-1.x86_64.rpm

The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.

JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
52c9816755 4.3.3 release 
JIRA: RHELCMP-7691
Signed-off-by: Haibo Lin <hlin@redhat.com>
330ba9b9c4 Do not clone the same repository multiple times, re-use already cloned repository 
Clone the directory to the compose tmp directory
Update the test_scm in order to create real Compose object. Mock objects are not allowed
to create/delete files for preventing multiple clones

JIRA: RHELCMP-5250

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
38810b3f13 modules: Correct a typo in loading obsoletes 
Pungi failed:

TypeError: argument obsoletes: Expected Modulemd.Obsoletes, but got list
Frame collect_module_obsoletes in /usr/lib/python3.10/site-packages/pungi/module_util.py at line 91
<CODE>
      84     mod_index = mod_index or Modulemd.ModuleIndex()
      85
      86     for module_name, obsoletes in iter_module_defaults_or_obsoletes(
      87         obsoletes_dir, obsoletes=True
      88     ):
      89         for obsolete in obsoletes:
      90             if not modules_to_load or module_name in modules_to_load:
-->   91                 mod_index.add_obsoletes(obsoletes)
      92
      93     return mod_index
</CODE>
<LOCALS>
           mod_index = <Modulemd.ModuleIndex object at 0x7f01a40fae40 (ModulemdModuleIndex at 0x7f0484338f90)>
         module_name = 'perl'
     modules_to_load = {'perl-Date-Manip', 'subversion', 'sway', 'nginx', 'perl-YAML', 'ghc', 'perl-App-cpanminus', 'perl-XML-Parser', 'varnish', 'nodejs', 'cri-o', 'perl-DBD-Pg', 'perl-DBI', 'perl', 'swig', 'perl-FCGI', 'p
            obsolete = <Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>
           obsoletes = [<Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>]
       obsoletes_dir = '/mnt/koji/compose/rawhide/Fedora-Rawhide-20220203.n.1/work/global/module_obsoletes'
</LOCALS>

This patches fixes the typo in add_obsoletes() argument.

https://pagure.io/releng/failed-composes/issue/3058
Signed-off-by: Petr Písař <ppisar@redhat.com>
aabf8faea0 profiler: Respect provided output stream 
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
6c280f2c46 Filter out environment groups unmatch given arch 
JIRA: RHELCMP-7926
Signed-off-by: Haibo Lin <hlin@redhat.com>
ecb1646042 Fix the wrong working directory for the progress_notification script 
Jira: RHELCMP-7901

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
0e82663327 Update the default greedy_method value in doc 
JIRA: RHELCMP-6308

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
b805ce3d12 osbs: only handle archives of type "image" 
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():

  File "pungi/phases/osbs.py", line 81, in process
    self.worker(compose, variant, config)
  File "pungi/phases/osbs.py", line 141, in worker
    nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
  File "pungi/phases/osbs.py", line 447, in add_metadata
    arch = archive["extra"]["image"]["arch"]
  KeyError: 'image'

Tell Koji to only return container image archives, and ignore these
remote-source archives.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
903ab076ba doc: improve osbs_registries explanation 
Explain the use-case for this setting, and use the active voice to
explain what actions Pungi performs relative to other tools.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
d55770898c nomacboot option for livemedia koji tasks 
Merges: https://pagure.io/pungi/pull-request/1591
Signed-off-by: Christopher O'Brien <cobrien@redhat.com>
bebbefe46e Variants file in config can contain path 
rcm-metadata configs contain definition of variants file. It can
be in form of SCM or file path. Before the fix, only variants
file's basename was consireded. Now the path can be written.
Example: variants_file = "comps/variants-rcmtools-2.0-rhel-8.xml"

JIRA: RHELCMP-8705
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
f8c7ad28e4 kojiwrapper: Add retries to login call 
The gssapi_login call is not retried automatically by Koji yet (see
koji#3170). Let's try to work around that by retrying in the calling
code.

JIRA: RHELCMP-8700
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
707a2c8d10 4.3.4 release 
JIRA: RHELCMP-8627
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
e490764985 Involve bandit 
JIRA: RHELCMP-8562
Signed-off-by: Haibo Lin <hlin@redhat.com>
c5cdd498ac Revert "Do not clone the same repository multiple times, re-use already cloned repository" 
This reverts commit 330ba9b9c4.

As of RHELCMP-8874, revert this patch as a quick fix.

Signed-off-by: Haibo Lin <hlin@redhat.com>
e8d79e9269 Restrict jsonschema version 
There's a new major version released on PyPI, and it doesn't seem to
work with Pungi yet. Until code is updated to be compatible, let's
ensure tox won't try to install it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
80957f5205 kojiwrapper: Ignore warnings before task id 
When looking for task ID in output of koji runroot command, do not check
just the first line. Instead look for first line that contains just a
number.

Most of the time, this should really be the first line. But if koji
client decides to print any warnings, this patch should skip that.

JIRA: RHELCMP-8944
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
c4aa45beab Add skip_branding to ostree_installer. 
Fixes: #1594
Merges: https://pagure.io/pungi/pull-request/1609
Signed-off-by: Lingyan Zhuang <lzhuang@redhat.com>
895b3982d7 Update the cts_keytab field in order to get the hostname of the server 
- This change is required for the following issue. Authentication is required for
importing composes to the CTS and finding generic keytabs
in different servers.

JIRA: RHELCMP-8930

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
ca185aaea8 Fix module defaults and obsoletes validation 
- Remove validation for modules obsoletes
  We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
  modules defaults && obsoletes
- Add additional check for invalid yaml file
  in Defaults. Previously, empty list of default would
  be returned when invalid yaml is present in Defaults
  directory.
- Using MergeIndex for Obsoletes only (for now).

https://pagure.io/pungi/issue/1592

Signed-off-by: Marek Kulik <mkulik@redhat.com>
d7aebfc7f9 4.3.5 release 
JIRA: RHELCMP-9389
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
960c85efde extra_isos: Fix detection of changed packages 
Checking start of the line is not sufficient for extra_isos that have
the variants in separate directories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
da336f75f8 Avoid crash when loading pickle file failed 
The pickle files are used for reusing results from old compose and the
failure should not block the compose process.

JIRA: RHELCMP-9494
Signed-off-by: Haibo Lin <hlin@redhat.com>
b27301641a Log time taken of each phase 
Signed-off-by: Haibo Lin <hlin@redhat.com>
19cb013fec Print more logs for git_ls_remote 
e.output probably contains the root cause of git ls-remote failure.

JIRA: RHELCMP-9598
JIRA: RHELCMP-9599
Signed-off-by: Haibo Lin <hlin@redhat.com>
b0b494fff0 Convert _ssh_run output to str for python3 
This is for fixing "a bytes-like object is required, not 'str'" issue
in runroot task.

JIRA: RHELCMP-9224
Signed-off-by: Haibo Lin <hlin@redhat.com>
ea8020473d
doc: fix osbuild's image_types field name 
It's actually image_types, not just image_type. See
https://pagure.io/fork/obudai/pungi/blob/master/f/pungi/checks.py#_1160

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
778dcfa587 Fix black complaint 
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
0abf937b0e Fix compatibility with jsonschema >= 4.0.0 
Fedora Rawhide (to be 37) packages jsonschema 4.9.0 at the moment, so we
can no longer get by with limiting the requirements. This patch makes
the validation work with both old and new version.

Fixes: rhbz#2113607
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
13ea8e5834 Create DVDs with xorriso 
Use a different approach for building DVDs when xorriso is enabled.

The default of using genisoimage is not changed at all. When the config
option is set to use xorriso, the actual execution is different between
bootable and non-bootable images.

The non-bootable images are still created by running xorrisofs (which is
a compatibility tool with same UI as genisoimage). Since the image is
not bootable, there should be no problems with boot options.

For bootable images, Pungi will instead take the boot.iso generated by
Lorax, and use xorriso to inject all the extra files into the image.

The shell script that used to invoke all the commands to build the ISO
now runs the `xorriso` command in interactive mode and feeds another
file into it. The new file contains the xorriso commands to add the
required files to the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
11fa342507 createiso: Make ISO level more granular 
Make it possible to set the level separately for each variant and
architecture.

JIRA: RHELCMP-9341

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
603c61a033 ostree: Add unified core mode for compose in rpm-ostree 
rpm-ostree is moving to unified core composes and this is now working
for Silverblue & Kinoite.

This is untested for IoT but they should move to os-build with Fedora
37.

See: https://github.com/coreos/rpm-ostree/issues/729
Merges: https://pagure.io/pungi/pull-request/1626
Signed-off-by: Timothée Ravier <tim@siosm.fr>
779793386c osbuild: add support for building ostree artifacts 
In order to start building Fedora IoT images with osbuild, we need to be able
to pass ostree options from pungi to the koji's osbuildImage task.

This commit adds support for it via new configuration options: ostree_url,
ostree_url and ostree_parent.

A test was added to cover these new options and they are were also added
into the documentation.

JIRA: COMPOSER-1702
Merges: https://pagure.io/pungi/pull-request/1624
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
8aba2363e2 pkgset: Report better error when module is missing an arch 
Pungi expects each module to be built for all architectures by default.
Unless the module is filtered out, missing metadata for a particular
arch would cause it to crash with a incomprehensible error message. This
should make it a little better.

Relates: https://pagure.io/releng/failed-composes/issue/3889
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
146b88e1e9 4.3.6 release 
JIRA: RHELCMP-9914
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
c7121f9378 profiler: Flush stdout before printing 
Apparently redirecting stderr to the same pipe as stdout does not
guarantee that the data will not be mangled together.

Flushing stdout before the profiler data is printed should ensure that
it does not end up in the middle of some RPM path.

Fixes: https://pagure.io/pungi/issue/1627
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
57ea640916 Add Jenkinsfile for CI 
JIRA: RHELCMP-9800
Signed-off-by: Haibo Lin <hlin@redhat.com>
805a1083a2
osbuild: accept only a single image type in the configuration 
Modify the osbuild configuration schema to accept only an array with a
single value as the `image_types`, in addition to a single string. The
single string was supported by the schema also before, but this fact was
not mentioned in the documentation, nor it was supported by the
`koji-osbuild` plugin of version lower than `9`.

Update the documentation accordingly.

Add unit test for invalid configuration containing more than one image
type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
57739c238f
osbuild: support specifying upload_options 
Since version 9, the `koji-osbuild` plugin supports specifying upload
options as part of a Koji build. This enables one to upload the built
image directly to the cloud environment as part of the image build in
Koji.

Extend the configuration schema with `upload_options`.
Extend the documentation and describe valid `upload_options` values.
Add a unit test testing a scenario when `upload_options` are specified.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
fa967f79b5 Ignore existing kerberos ticket for CTS auth 
When there is an existing kerberos ticket, it gets precedence over the
environment variable with path to a keytab. That is not expected and the
user ticket can possibly lack permissions in CTS to be able to run the
compose successfully.

This patch fixes that by setting KRB5CCNAME to a fresh path. That way
there will not be any valid ticket, since the credentials cache does not
exist yet.

JIRA: RHELCMP-9742
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
8cd19605bd Retry failed cts requests 
JIRA: RHELCMP-10033
Signed-off-by: Haibo Lin <hlin@redhat.com>
479849042f init: Filter comps for modular variants with tags 
Modular variants can either be specified by a list of modules, or by a
list of Koji tags. In terms of comps preprocessing there should not be
any difference between the two.

Resolves: https://pagure.io/pungi/issue/1640
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
fce5493f09 Merge remote-tracking branch 'centos-origin/master' 
# Conflicts:
#	pungi/phases/init.py
#	pungi/wrappers/comps.py
323d31df2b Merge branch 'master' into a8_updated 
# Conflicts:
#	pungi.spec
#	pungi/wrappers/kojiwrapper.py
#	setup.py
#	tests/test_extra_isos_phase.py
#	tests/test_pkgset_pkgsets.py
soksanichenko added 3 commits 2022-11-08 15:11:31 +00:00
soksanichenko added 1 commit 2022-11-09 07:06:16 +00:00
soksanichenko added 1 commit 2022-11-09 10:38:40 +00:00
soksanichenko added 1 commit 2022-11-09 15:59:57 +00:00
soksanichenko added 1 commit 2022-11-09 16:18:17 +00:00
soksanichenko added 1 commit 2022-11-09 18:57:02 +00:00
soksanichenko added 1 commit 2022-11-09 19:01:36 +00:00
soksanichenko added 1 commit 2022-11-09 19:27:57 +00:00
soksanichenko added 1 commit 2022-11-09 19:42:19 +00:00
soksanichenko closed this pull request 2022-11-11 21:58:50 +00:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No Label
Milestone
Clear milestone
No items
No Milestone
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: almalinux/pungi#11
No description provided.
Delete Branch "a8_updated"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?