Commit Graph

2687 Commits

Author SHA1 Message Date
Stepan Oksanichenko cfe6ec3f4e Merge pull request 'ALBS-334: Make the ability of Pungi to give module_defaults from remote sources' (#4) from ALBS-334 into aln8
Reviewed-on: #4
2022-05-04 17:05:45 +00:00
soksanichenko e6c6f74176 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-03 18:18:17 +03:00
soksanichenko 8676941655 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-02 02:25:32 +03:00
soksanichenko 5f74175c33 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-01 03:41:40 +03:00
soksanichenko 1e18e8995d ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-01 03:32:01 +03:00
soksanichenko 38ea822260 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-30 00:27:31 +03:00
soksanichenko 34eb45c7ec ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:39:51 +03:00
soksanichenko 7422d1e045 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:33:28 +03:00
soksanichenko 97801e772e ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:25:59 +03:00
soksanichenko dff346eedb - Unit tests are fixed 2022-04-28 16:44:47 +03:00
soksanichenko de53dd0bbd - Unit tests are fixed 2022-04-28 16:30:03 +03:00
Lubomír Sedlář 80957f5205 kojiwrapper: Ignore warnings before task id
When looking for task ID in output of koji runroot command, do not check
just the first line. Instead look for first line that contains just a
number.

Most of the time, this should really be the first line. But if koji
client decides to print any warnings, this patch should skip that.

JIRA: RHELCMP-8944
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-26 08:15:13 +02:00
Lubomír Sedlář e8d79e9269 Restrict jsonschema version
There's a new major version released on PyPI, and it doesn't seem to
work with Pungi yet. Until code is updated to be compatible, let's
ensure tox won't try to install it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-19 12:56:13 +02:00
Haibo Lin c5cdd498ac Revert "Do not clone the same repository multiple times, re-use already cloned repository"
This reverts commit 330ba9b9c4.

As of RHELCMP-8874, revert this patch as a quick fix.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-13 16:21:55 +08:00
Haibo Lin e490764985 Involve bandit
JIRA: RHELCMP-8562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-07 14:03:57 +08:00
Ondrej Nosek 707a2c8d10 4.3.4 release
JIRA: RHELCMP-8627
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-04-04 11:12:21 +02:00
Lubomír Sedlář f8c7ad28e4 kojiwrapper: Add retries to login call
The gssapi_login call is not retried automatically by Koji yet (see
koji#3170). Let's try to work around that by retrying in the calling
code.

JIRA: RHELCMP-8700
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-04 07:51:12 +00:00
Ondrej Nosek bebbefe46e Variants file in config can contain path
rcm-metadata configs contain definition of variants file. It can
be in form of SCM or file path. Before the fix, only variants
file's basename was consireded. Now the path can be written.
Example: variants_file = "comps/variants-rcmtools-2.0-rhel-8.xml"

JIRA: RHELCMP-8705
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-03-30 22:39:18 +02:00
Christopher O'Brien d55770898c nomacboot option for livemedia koji tasks
Merges: https://pagure.io/pungi/pull-request/1591
Signed-off-by: Christopher O'Brien <cobrien@redhat.com>
2022-03-23 09:36:51 +01:00
Ken Dreyer 903ab076ba doc: improve osbs_registries explanation
Explain the use-case for this setting, and use the active voice to
explain what actions Pungi performs relative to other tools.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-21 14:53:11 -04:00
soksanichenko 88121619bc ALBS-226: Patch pungi/lorax for building AL9
- Defaults modules can be empty, but pungi detects
  empty folder while copying and raises the exception in this case
2022-03-18 22:37:57 +02:00
Ken Dreyer b805ce3d12 osbs: only handle archives of type "image"
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():

  File "pungi/phases/osbs.py", line 81, in process
    self.worker(compose, variant, config)
  File "pungi/phases/osbs.py", line 141, in worker
    nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
  File "pungi/phases/osbs.py", line 447, in add_metadata
    arch = archive["extra"]["image"]["arch"]
  KeyError: 'image'

Tell Koji to only return container image archives, and ignore these
remote-source archives.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-17 11:04:02 -04:00
Ozan Unsal 0e82663327 Update the default greedy_method value in doc
JIRA: RHELCMP-6308

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-10 15:35:13 +01:00
Ozan Unsal ecb1646042 Fix the wrong working directory for the progress_notification script
Jira: RHELCMP-7901

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-01 11:25:06 +00:00
Haibo Lin 6c280f2c46 Filter out environment groups unmatch given arch
JIRA: RHELCMP-7926
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-02-22 11:33:25 +08:00
Lubomír Sedlář aabf8faea0 profiler: Respect provided output stream
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-02-07 09:07:59 +00:00
Petr Písař 38810b3f13 modules: Correct a typo in loading obsoletes
Pungi failed:

TypeError: argument obsoletes: Expected Modulemd.Obsoletes, but got list
Frame collect_module_obsoletes in /usr/lib/python3.10/site-packages/pungi/module_util.py at line 91
<CODE>
      84     mod_index = mod_index or Modulemd.ModuleIndex()
      85
      86     for module_name, obsoletes in iter_module_defaults_or_obsoletes(
      87         obsoletes_dir, obsoletes=True
      88     ):
      89         for obsolete in obsoletes:
      90             if not modules_to_load or module_name in modules_to_load:
-->   91                 mod_index.add_obsoletes(obsoletes)
      92
      93     return mod_index
</CODE>
<LOCALS>
           mod_index = <Modulemd.ModuleIndex object at 0x7f01a40fae40 (ModulemdModuleIndex at 0x7f0484338f90)>
         module_name = 'perl'
     modules_to_load = {'perl-Date-Manip', 'subversion', 'sway', 'nginx', 'perl-YAML', 'ghc', 'perl-App-cpanminus', 'perl-XML-Parser', 'varnish', 'nodejs', 'cri-o', 'perl-DBD-Pg', 'perl-DBI', 'perl', 'swig', 'perl-FCGI', 'p
            obsolete = <Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>
           obsoletes = [<Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>]
       obsoletes_dir = '/mnt/koji/compose/rawhide/Fedora-Rawhide-20220203.n.1/work/global/module_obsoletes'
</LOCALS>

This patches fixes the typo in add_obsoletes() argument.

https://pagure.io/releng/failed-composes/issue/3058
Signed-off-by: Petr Písař <ppisar@redhat.com>
2022-02-04 11:15:31 +01:00
Ozan Unsal 330ba9b9c4 Do not clone the same repository multiple times, re-use already cloned repository
Clone the directory to the compose tmp directory
Update the test_scm in order to create real Compose object. Mock objects are not allowed
to create/delete files for preventing multiple clones

JIRA: RHELCMP-5250

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-26 15:57:13 +01:00
Haibo Lin 52c9816755 4.3.3 release
JIRA: RHELCMP-7691
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-01-13 15:45:30 +08:00
Lubomír Sedlář 32221e8f36 hybrid: Explicitly pull in debugsource packages
This should cover case where we there's a build like this:

foo-1-1.src.rpm
  foo-sub-1-1.noarch.rpm
  foo-debugsource-1-1.x86_64.rpm

The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.

JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-01-12 13:59:09 +00:00
Filip Valder fe986d68b9 Add module obsoletes feature
JIRA: MODULAR-113
Merges: https://pagure.io/pungi/pull-request/1578
Signed-off-by: Filip Valder <fvalder@redhat.com>
2022-01-12 12:51:37 +01:00
Ozan Unsal 42f668d969 buildinstall: Add ability to install extra packages in runroot
Resolves: https://pagure.io/pungi/issue/1461
Merges: https://pagure.io/pungi/pull-request/1580
JIRA: RHELCMP-2911
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-05 14:35:37 +01:00
Haibo Lin 894cce6a5a Ignore osbs/osbuild config when reusing iso images
JIRA: RHELCMP-7562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-01-04 16:35:29 +08:00
soksanichenko 0484426e0c ALBS-97: Build AlmaLinux PPC64le repos and ISOs with pungi
- Changelog
- Version is bumped

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I933925b7a27a5e1b642020e060f59212fdc6ebf4
2021-12-30 12:42:34 +02:00
soksanichenko b9d86b90e1 ALBS-97: Build AlmaLinux PPC64le repos and ISOs with pungi
- Scripts `create_packages_json` & `gather_modules` can process lzma compressed yaml files
- Script `create_package_json` can use repodata there are packages with different
  arch by compare with passed to the script

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ia9a3bacfa4344f0cf33b9f416649fd4a5f8d3c37
2021-12-28 16:08:04 +02:00
soksanichenko 58a16e5688 - The version is bumped
- The changelog is updated
- The test `create_packages_json` is fixed

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I173013da990eb296e58ca8f3555a05913ca1c852
2021-12-20 14:11:17 +02:00
Lubomír Sedlář 260b3fce8d compose: Make sure temporary dirs are world readable
When the temporary directory is created with 0700, other programs
(potentially on another host) will have problems reading it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-7635
2021-12-17 08:10:48 +01:00
Haibo Lin 20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Haibo Lin 5e6248e3e0 Generate images.json for extra_isos phase
JIRA: RHELCMP-7241
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-15 14:38:53 +08:00
Haibo Lin f681956cf1 Fix tests for python 2.6
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-12 17:04:17 +08:00
Haibo Lin cfb9882269 4.3.2 release
JIRA: RHELCMP-7182
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-11 16:08:54 +08:00
soksanichenko f2ed64d952 ALBS-66: Prepare Jenkins jobs for building distribution of AlmaLinux 8.5
- Script `create_packages_json` can duplicate the packages with
  same version in different variants

Change-Id: I3c79ad06c4c22442423c12d5fa06baf82d663a3f
2021-11-10 15:29:59 +02:00
Lubomír Sedlář b652119d54 gather: Load JSON mapping relative to config dir
JIRA: RHELCMP-7195
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Lubomír Sedlář 33d7290d78 gather: Stop requiring all variants/arches in JSON
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-10 11:01:53 +01:00
Ken Dreyer 9bae86a51e doc: make dnf "backend" settings easier to discover
Mention the corresponding "gather" or "repoclosure" backend settings in
the documentation for each setting.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-11-04 10:55:54 -04:00
Lubomír Sedlář 1d654522be Remove with_jigdo argument
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Lubomír Sedlář 80bd254347 Check dependencies after config validation
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Ken Dreyer 94ffa1c5c6 default "with_jigdo" to False
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
2021-11-04 13:37:51 +00:00
Lubomír Sedlář 9d02f87c99 Stop trying to validate non-existent metadata
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
2021-11-04 09:57:20 +01:00
fdiprete 7b9e08ab28 test images for metadata deserialization error
Merges: https://pagure.io/pungi/pull-request/1559
Jira: https://issues.redhat.com/browse/RHELCMP-6685
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-11-03 08:50:07 +01:00