Commit Graph

2595 Commits

Author SHA1 Message Date
Haibo Lin
19cb013fec Print more logs for git_ls_remote
e.output probably contains the root cause of git ls-remote failure.

JIRA: RHELCMP-9598
JIRA: RHELCMP-9599
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-14 11:11:01 +08:00
Haibo Lin
b27301641a Log time taken of each phase
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-12 16:56:41 +08:00
Haibo Lin
da336f75f8 Avoid crash when loading pickle file failed
The pickle files are used for reusing results from old compose and the
failure should not block the compose process.

JIRA: RHELCMP-9494
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-01 10:33:52 +08:00
Lubomír Sedlář
960c85efde extra_isos: Fix detection of changed packages
Checking start of the line is not sufficient for extra_isos that have
the variants in separate directories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-06-27 09:46:35 +02:00
Lubomír Sedlář
d7aebfc7f9 4.3.5 release
JIRA: RHELCMP-9389
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-06-15 12:27:04 +02:00
Marek Kulik
ca185aaea8 Fix module defaults and obsoletes validation
- Remove validation for modules obsoletes
  We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
  modules defaults && obsoletes
- Add additional check for invalid yaml file
  in Defaults. Previously, empty list of default would
  be returned when invalid yaml is present in Defaults
  directory.
- Using MergeIndex for Obsoletes only (for now).

https://pagure.io/pungi/issue/1592

Signed-off-by: Marek Kulik <mkulik@redhat.com>
2022-06-10 11:35:26 +00:00
Ozan Unsal
895b3982d7 Update the cts_keytab field in order to get the hostname of the server
- This change is required for the following issue. Authentication is required for
importing composes to the CTS and finding generic keytabs
in different servers.

JIRA: RHELCMP-8930

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-06-01 07:49:58 +00:00
Lingyan Zhuang
c4aa45beab Add skip_branding to ostree_installer.
Fixes: #1594
Merges: https://pagure.io/pungi/pull-request/1609
Signed-off-by: Lingyan Zhuang <lzhuang@redhat.com>
2022-05-11 15:19:53 +02:00
Lubomír Sedlář
80957f5205 kojiwrapper: Ignore warnings before task id
When looking for task ID in output of koji runroot command, do not check
just the first line. Instead look for first line that contains just a
number.

Most of the time, this should really be the first line. But if koji
client decides to print any warnings, this patch should skip that.

JIRA: RHELCMP-8944
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-26 08:15:13 +02:00
Lubomír Sedlář
e8d79e9269 Restrict jsonschema version
There's a new major version released on PyPI, and it doesn't seem to
work with Pungi yet. Until code is updated to be compatible, let's
ensure tox won't try to install it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-19 12:56:13 +02:00
Haibo Lin
c5cdd498ac Revert "Do not clone the same repository multiple times, re-use already cloned repository"
This reverts commit 330ba9b9c4.

As of RHELCMP-8874, revert this patch as a quick fix.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-13 16:21:55 +08:00
Haibo Lin
e490764985 Involve bandit
JIRA: RHELCMP-8562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-07 14:03:57 +08:00
Ondrej Nosek
707a2c8d10 4.3.4 release
JIRA: RHELCMP-8627
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-04-04 11:12:21 +02:00
Lubomír Sedlář
f8c7ad28e4 kojiwrapper: Add retries to login call
The gssapi_login call is not retried automatically by Koji yet (see
koji#3170). Let's try to work around that by retrying in the calling
code.

JIRA: RHELCMP-8700
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-04 07:51:12 +00:00
Ondrej Nosek
bebbefe46e Variants file in config can contain path
rcm-metadata configs contain definition of variants file. It can
be in form of SCM or file path. Before the fix, only variants
file's basename was consireded. Now the path can be written.
Example: variants_file = "comps/variants-rcmtools-2.0-rhel-8.xml"

JIRA: RHELCMP-8705
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-03-30 22:39:18 +02:00
Christopher O'Brien
d55770898c nomacboot option for livemedia koji tasks
Merges: https://pagure.io/pungi/pull-request/1591
Signed-off-by: Christopher O'Brien <cobrien@redhat.com>
2022-03-23 09:36:51 +01:00
Ken Dreyer
903ab076ba doc: improve osbs_registries explanation
Explain the use-case for this setting, and use the active voice to
explain what actions Pungi performs relative to other tools.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-21 14:53:11 -04:00
Ken Dreyer
b805ce3d12 osbs: only handle archives of type "image"
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():

  File "pungi/phases/osbs.py", line 81, in process
    self.worker(compose, variant, config)
  File "pungi/phases/osbs.py", line 141, in worker
    nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
  File "pungi/phases/osbs.py", line 447, in add_metadata
    arch = archive["extra"]["image"]["arch"]
  KeyError: 'image'

Tell Koji to only return container image archives, and ignore these
remote-source archives.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-17 11:04:02 -04:00
Ozan Unsal
0e82663327 Update the default greedy_method value in doc
JIRA: RHELCMP-6308

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-10 15:35:13 +01:00
Ozan Unsal
ecb1646042 Fix the wrong working directory for the progress_notification script
Jira: RHELCMP-7901

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-01 11:25:06 +00:00
Haibo Lin
6c280f2c46 Filter out environment groups unmatch given arch
JIRA: RHELCMP-7926
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-02-22 11:33:25 +08:00
Lubomír Sedlář
aabf8faea0 profiler: Respect provided output stream
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-02-07 09:07:59 +00:00
Petr Písař
38810b3f13 modules: Correct a typo in loading obsoletes
Pungi failed:

TypeError: argument obsoletes: Expected Modulemd.Obsoletes, but got list
Frame collect_module_obsoletes in /usr/lib/python3.10/site-packages/pungi/module_util.py at line 91
<CODE>
      84     mod_index = mod_index or Modulemd.ModuleIndex()
      85
      86     for module_name, obsoletes in iter_module_defaults_or_obsoletes(
      87         obsoletes_dir, obsoletes=True
      88     ):
      89         for obsolete in obsoletes:
      90             if not modules_to_load or module_name in modules_to_load:
-->   91                 mod_index.add_obsoletes(obsoletes)
      92
      93     return mod_index
</CODE>
<LOCALS>
           mod_index = <Modulemd.ModuleIndex object at 0x7f01a40fae40 (ModulemdModuleIndex at 0x7f0484338f90)>
         module_name = 'perl'
     modules_to_load = {'perl-Date-Manip', 'subversion', 'sway', 'nginx', 'perl-YAML', 'ghc', 'perl-App-cpanminus', 'perl-XML-Parser', 'varnish', 'nodejs', 'cri-o', 'perl-DBD-Pg', 'perl-DBI', 'perl', 'swig', 'perl-FCGI', 'p
            obsolete = <Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>
           obsoletes = [<Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>]
       obsoletes_dir = '/mnt/koji/compose/rawhide/Fedora-Rawhide-20220203.n.1/work/global/module_obsoletes'
</LOCALS>

This patches fixes the typo in add_obsoletes() argument.

https://pagure.io/releng/failed-composes/issue/3058
Signed-off-by: Petr Písař <ppisar@redhat.com>
2022-02-04 11:15:31 +01:00
Ozan Unsal
330ba9b9c4 Do not clone the same repository multiple times, re-use already cloned repository
Clone the directory to the compose tmp directory
Update the test_scm in order to create real Compose object. Mock objects are not allowed
to create/delete files for preventing multiple clones

JIRA: RHELCMP-5250

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-26 15:57:13 +01:00
Haibo Lin
52c9816755 4.3.3 release
JIRA: RHELCMP-7691
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-01-13 15:45:30 +08:00
Lubomír Sedlář
32221e8f36 hybrid: Explicitly pull in debugsource packages
This should cover case where we there's a build like this:

foo-1-1.src.rpm
  foo-sub-1-1.noarch.rpm
  foo-debugsource-1-1.x86_64.rpm

The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.

JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-01-12 13:59:09 +00:00
Filip Valder
fe986d68b9 Add module obsoletes feature
JIRA: MODULAR-113
Merges: https://pagure.io/pungi/pull-request/1578
Signed-off-by: Filip Valder <fvalder@redhat.com>
2022-01-12 12:51:37 +01:00
Ozan Unsal
42f668d969 buildinstall: Add ability to install extra packages in runroot
Resolves: https://pagure.io/pungi/issue/1461
Merges: https://pagure.io/pungi/pull-request/1580
JIRA: RHELCMP-2911
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-05 14:35:37 +01:00
Haibo Lin
894cce6a5a Ignore osbs/osbuild config when reusing iso images
JIRA: RHELCMP-7562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-01-04 16:35:29 +08:00
Lubomír Sedlář
260b3fce8d compose: Make sure temporary dirs are world readable
When the temporary directory is created with 0700, other programs
(potentially on another host) will have problems reading it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-7635
2021-12-17 08:10:48 +01:00
Haibo Lin
20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Haibo Lin
5e6248e3e0 Generate images.json for extra_isos phase
JIRA: RHELCMP-7241
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-15 14:38:53 +08:00
Haibo Lin
f681956cf1 Fix tests for python 2.6
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-12 17:04:17 +08:00
Haibo Lin
cfb9882269 4.3.2 release
JIRA: RHELCMP-7182
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-11 16:08:54 +08:00
Lubomír Sedlář
b652119d54 gather: Load JSON mapping relative to config dir
JIRA: RHELCMP-7195
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Lubomír Sedlář
33d7290d78 gather: Stop requiring all variants/arches in JSON
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Ken Dreyer
9bae86a51e doc: make dnf "backend" settings easier to discover
Mention the corresponding "gather" or "repoclosure" backend settings in
the documentation for each setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-11-04 10:55:54 -04:00
Lubomír Sedlář
1d654522be Remove with_jigdo argument
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
80bd254347 Check dependencies after config validation
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Ken Dreyer
94ffa1c5c6 default "with_jigdo" to False
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
9d02f87c99 Stop trying to validate non-existent metadata
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
2021-11-04 09:57:20 +01:00
fdiprete
7b9e08ab28 test images for metadata deserialization error
Merges: https://pagure.io/pungi/pull-request/1559
Jira: https://issues.redhat.com/browse/RHELCMP-6685
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-11-03 08:50:07 +01:00
Lubomír Sedlář
e2b3002726 repoclosure: Use --forcearch for dnf repoclosure
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.

Based on how the list of architectures is generated, we know that the
main one will always be first.

Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 08:51:40 +01:00
Lubomír Sedlář
e8305f3978 extra_isos: Allow reusing old images
When nothing in configuration or the image itself changed, let's just
copy the older one.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
ac66c3d7f3 createiso: Allow reusing old images
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:

 * If image is bootable, buildinstall phase must have been reused too.
 * Compose configuration must have not changed (except for a few
   whitelisted options).
 * Volume ID of the ISO much not have changed.
 * No RPM on the ISO must have changed.

The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
eb61c97cdb Remove default runroot channel
When the value is not specified in the configuration file, let Koji pick
the default channel.

JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-25 15:56:24 +02:00
Ozan Unsal
b03490bf18 4.3.1 release
JIRA: RHELCMP-7116
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-10-25 15:18:26 +02:00
Dan Čermák
ab19043773 Correct irc network name & add matrix room
Signed-off-by: Dan Čermák <dan.cermak@cgc-instruments.com>
2021-10-25 07:26:18 +00:00
Lubomír Sedlář
204d88a351 Add missing mock to osbs tests
We don't want the test to try to a dummy URL.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-21 13:34:37 +02:00
Haibo Lin
8133676270 osbs: Reuse images from old compose
JIRA: RHELCMP-5972
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:22:19 +08:00