Commit Graph

2909 Commits

Author SHA1 Message Date
Tomas Hozza
805a1083a2
osbuild: accept only a single image type in the configuration
Modify the osbuild configuration schema to accept only an array with a
single value as the `image_types`, in addition to a single string. The
single string was supported by the schema also before, but this fact was
not mentioned in the documentation, nor it was supported by the
`koji-osbuild` plugin of version lower than `9`.

Update the documentation accordingly.

Add unit test for invalid configuration containing more than one image
type.

Signed-off-by: Tomas Hozza <thozza@redhat.com>
2022-09-06 10:55:25 +02:00
Haibo Lin
57ea640916 Add Jenkinsfile for CI
JIRA: RHELCMP-9800
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-08-29 17:38:04 +08:00
Lubomír Sedlář
c7121f9378 profiler: Flush stdout before printing
Apparently redirecting stderr to the same pipe as stdout does not
guarantee that the data will not be mangled together.

Flushing stdout before the profiler data is printed should ensure that
it does not end up in the middle of some RPM path.

Fixes: https://pagure.io/pungi/issue/1627
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-29 05:57:14 +00:00
Lubomír Sedlář
146b88e1e9 4.3.6 release
JIRA: RHELCMP-9914
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-26 11:13:43 +02:00
Lubomír Sedlář
8aba2363e2 pkgset: Report better error when module is missing an arch
Pungi expects each module to be built for all architectures by default.
Unless the module is filtered out, missing metadata for a particular
arch would cause it to crash with a incomprehensible error message. This
should make it a little better.

Relates: https://pagure.io/releng/failed-composes/issue/3889
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-26 08:46:01 +00:00
Ondřej Budai
779793386c osbuild: add support for building ostree artifacts
In order to start building Fedora IoT images with osbuild, we need to be able
to pass ostree options from pungi to the koji's osbuildImage task.

This commit adds support for it via new configuration options: ostree_url,
ostree_url and ostree_parent.

A test was added to cover these new options and they are were also added
into the documentation.

JIRA: COMPOSER-1702
Merges: https://pagure.io/pungi/pull-request/1624
Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-08-25 15:35:17 +02:00
Timothée Ravier
603c61a033 ostree: Add unified core mode for compose in rpm-ostree
rpm-ostree is moving to unified core composes and this is now working
for Silverblue & Kinoite.

This is untested for IoT but they should move to os-build with Fedora
37.

See: https://github.com/coreos/rpm-ostree/issues/729
Merges: https://pagure.io/pungi/pull-request/1626
Signed-off-by: Timothée Ravier <tim@siosm.fr>
2022-08-23 10:52:45 +02:00
Lubomír Sedlář
11fa342507 createiso: Make ISO level more granular
Make it possible to set the level separately for each variant and
architecture.

JIRA: RHELCMP-9341

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-16 14:38:18 +02:00
Lubomír Sedlář
13ea8e5834 Create DVDs with xorriso
Use a different approach for building DVDs when xorriso is enabled.

The default of using genisoimage is not changed at all. When the config
option is set to use xorriso, the actual execution is different between
bootable and non-bootable images.

The non-bootable images are still created by running xorrisofs (which is
a compatibility tool with same UI as genisoimage). Since the image is
not bootable, there should be no problems with boot options.

For bootable images, Pungi will instead take the boot.iso generated by
Lorax, and use xorriso to inject all the extra files into the image.

The shell script that used to invoke all the commands to build the ISO
now runs the `xorriso` command in interactive mode and feeds another
file into it. The new file contains the xorriso commands to add the
required files to the image.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-16 14:30:41 +02:00
Lubomír Sedlář
0abf937b0e Fix compatibility with jsonschema >= 4.0.0
Fedora Rawhide (to be 37) packages jsonschema 4.9.0 at the moment, so we
can no longer get by with limiting the requirements. This patch makes
the validation work with both old and new version.

Fixes: rhbz#2113607
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-10 09:52:23 +02:00
Lubomír Sedlář
778dcfa587 Fix black complaint
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-08-10 09:04:41 +02:00
Ondřej Budai
ea8020473d
doc: fix osbuild's image_types field name
It's actually image_types, not just image_type. See
https://pagure.io/fork/obudai/pungi/blob/master/f/pungi/checks.py#_1160

Signed-off-by: Ondřej Budai <ondrej@budai.cz>
2022-08-09 17:39:42 +02:00
Haibo Lin
b0b494fff0 Convert _ssh_run output to str for python3
This is for fixing "a bytes-like object is required, not 'str'" issue
in runroot task.

JIRA: RHELCMP-9224
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-15 13:45:15 +08:00
Haibo Lin
19cb013fec Print more logs for git_ls_remote
e.output probably contains the root cause of git ls-remote failure.

JIRA: RHELCMP-9598
JIRA: RHELCMP-9599
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-14 11:11:01 +08:00
Haibo Lin
b27301641a Log time taken of each phase
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-12 16:56:41 +08:00
Haibo Lin
da336f75f8 Avoid crash when loading pickle file failed
The pickle files are used for reusing results from old compose and the
failure should not block the compose process.

JIRA: RHELCMP-9494
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-07-01 10:33:52 +08:00
Lubomír Sedlář
960c85efde extra_isos: Fix detection of changed packages
Checking start of the line is not sufficient for extra_isos that have
the variants in separate directories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-06-27 09:46:35 +02:00
Lubomír Sedlář
d7aebfc7f9 4.3.5 release
JIRA: RHELCMP-9389
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-06-15 12:27:04 +02:00
Marek Kulik
ca185aaea8 Fix module defaults and obsoletes validation
- Remove validation for modules obsoletes
  We can have multiple obsoletes for one module
- Add unit tests to cover basic scenarios for
  modules defaults && obsoletes
- Add additional check for invalid yaml file
  in Defaults. Previously, empty list of default would
  be returned when invalid yaml is present in Defaults
  directory.
- Using MergeIndex for Obsoletes only (for now).

https://pagure.io/pungi/issue/1592

Signed-off-by: Marek Kulik <mkulik@redhat.com>
2022-06-10 11:35:26 +00:00
Ozan Unsal
895b3982d7 Update the cts_keytab field in order to get the hostname of the server
- This change is required for the following issue. Authentication is required for
importing composes to the CTS and finding generic keytabs
in different servers.

JIRA: RHELCMP-8930

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-06-01 07:49:58 +00:00
Lingyan Zhuang
c4aa45beab Add skip_branding to ostree_installer.
Fixes: #1594
Merges: https://pagure.io/pungi/pull-request/1609
Signed-off-by: Lingyan Zhuang <lzhuang@redhat.com>
2022-05-11 15:19:53 +02:00
soksanichenko
f21ed6f607 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-04 20:16:23 +03:00
soksanichenko
cfe6ec3f4e Merge pull request 'ALBS-334: Make the ability of Pungi to give module_defaults from remote sources' (#4) from ALBS-334 into aln8
Reviewed-on: #4
2022-05-04 17:05:45 +00:00
soksanichenko
e6c6f74176 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-03 18:18:17 +03:00
soksanichenko
8676941655 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-02 02:25:32 +03:00
soksanichenko
5f74175c33 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-01 03:41:40 +03:00
soksanichenko
1e18e8995d ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-05-01 03:32:01 +03:00
soksanichenko
38ea822260 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-30 00:27:31 +03:00
soksanichenko
34eb45c7ec ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:39:51 +03:00
soksanichenko
7422d1e045 ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:33:28 +03:00
soksanichenko
97801e772e ALBS-334: Make the ability of Pungi to give module_defaults from remote sources 2022-04-29 21:25:59 +03:00
soksanichenko
dff346eedb - Unit tests are fixed 2022-04-28 16:44:47 +03:00
soksanichenko
de53dd0bbd - Unit tests are fixed 2022-04-28 16:30:03 +03:00
Lubomír Sedlář
80957f5205 kojiwrapper: Ignore warnings before task id
When looking for task ID in output of koji runroot command, do not check
just the first line. Instead look for first line that contains just a
number.

Most of the time, this should really be the first line. But if koji
client decides to print any warnings, this patch should skip that.

JIRA: RHELCMP-8944
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-26 08:15:13 +02:00
Lubomír Sedlář
e8d79e9269 Restrict jsonschema version
There's a new major version released on PyPI, and it doesn't seem to
work with Pungi yet. Until code is updated to be compatible, let's
ensure tox won't try to install it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-19 12:56:13 +02:00
Haibo Lin
c5cdd498ac Revert "Do not clone the same repository multiple times, re-use already cloned repository"
This reverts commit 330ba9b9c4.

As of RHELCMP-8874, revert this patch as a quick fix.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-13 16:21:55 +08:00
Haibo Lin
e490764985 Involve bandit
JIRA: RHELCMP-8562
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-04-07 14:03:57 +08:00
Ondrej Nosek
707a2c8d10 4.3.4 release
JIRA: RHELCMP-8627
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-04-04 11:12:21 +02:00
Lubomír Sedlář
f8c7ad28e4 kojiwrapper: Add retries to login call
The gssapi_login call is not retried automatically by Koji yet (see
koji#3170). Let's try to work around that by retrying in the calling
code.

JIRA: RHELCMP-8700
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-04-04 07:51:12 +00:00
Ondrej Nosek
bebbefe46e Variants file in config can contain path
rcm-metadata configs contain definition of variants file. It can
be in form of SCM or file path. Before the fix, only variants
file's basename was consireded. Now the path can be written.
Example: variants_file = "comps/variants-rcmtools-2.0-rhel-8.xml"

JIRA: RHELCMP-8705
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2022-03-30 22:39:18 +02:00
Christopher O'Brien
d55770898c nomacboot option for livemedia koji tasks
Merges: https://pagure.io/pungi/pull-request/1591
Signed-off-by: Christopher O'Brien <cobrien@redhat.com>
2022-03-23 09:36:51 +01:00
Ken Dreyer
903ab076ba doc: improve osbs_registries explanation
Explain the use-case for this setting, and use the active voice to
explain what actions Pungi performs relative to other tools.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-21 14:53:11 -04:00
soksanichenko
88121619bc ALBS-226: Patch pungi/lorax for building AL9
- Defaults modules can be empty, but pungi detects
  empty folder while copying and raises the exception in this case
2022-03-18 22:37:57 +02:00
Ken Dreyer
b805ce3d12 osbs: only handle archives of type "image"
Prior to this change, if a container image used Cachito with OSBS, then
OSBS would store additional "remote-sources" files in the Koji archives
for the build. Pungi cannot parse the metadata for these archive
entries, so it would crash in add_metadata():

  File "pungi/phases/osbs.py", line 81, in process
    self.worker(compose, variant, config)
  File "pungi/phases/osbs.py", line 141, in worker
    nvr, archive_ids = add_metadata(variant, task_id, compose, scratch)
  File "pungi/phases/osbs.py", line 447, in add_metadata
    arch = archive["extra"]["image"]["arch"]
  KeyError: 'image'

Tell Koji to only return container image archives, and ignore these
remote-source archives.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2022-03-17 11:04:02 -04:00
Ozan Unsal
0e82663327 Update the default greedy_method value in doc
JIRA: RHELCMP-6308

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-10 15:35:13 +01:00
Ozan Unsal
ecb1646042 Fix the wrong working directory for the progress_notification script
Jira: RHELCMP-7901

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-03-01 11:25:06 +00:00
Haibo Lin
6c280f2c46 Filter out environment groups unmatch given arch
JIRA: RHELCMP-7926
Signed-off-by: Haibo Lin <hlin@redhat.com>
2022-02-22 11:33:25 +08:00
Lubomír Sedlář
aabf8faea0 profiler: Respect provided output stream
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2022-02-07 09:07:59 +00:00
Petr Písař
38810b3f13 modules: Correct a typo in loading obsoletes
Pungi failed:

TypeError: argument obsoletes: Expected Modulemd.Obsoletes, but got list
Frame collect_module_obsoletes in /usr/lib/python3.10/site-packages/pungi/module_util.py at line 91
<CODE>
      84     mod_index = mod_index or Modulemd.ModuleIndex()
      85
      86     for module_name, obsoletes in iter_module_defaults_or_obsoletes(
      87         obsoletes_dir, obsoletes=True
      88     ):
      89         for obsolete in obsoletes:
      90             if not modules_to_load or module_name in modules_to_load:
-->   91                 mod_index.add_obsoletes(obsoletes)
      92
      93     return mod_index
</CODE>
<LOCALS>
           mod_index = <Modulemd.ModuleIndex object at 0x7f01a40fae40 (ModulemdModuleIndex at 0x7f0484338f90)>
         module_name = 'perl'
     modules_to_load = {'perl-Date-Manip', 'subversion', 'sway', 'nginx', 'perl-YAML', 'ghc', 'perl-App-cpanminus', 'perl-XML-Parser', 'varnish', 'nodejs', 'cri-o', 'perl-DBD-Pg', 'perl-DBI', 'perl', 'swig', 'perl-FCGI', 'p
            obsolete = <Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>
           obsoletes = [<Modulemd.Obsoletes object at 0x7f00c0fe4a00 (ModulemdObsoletes at 0x7f024c0268b0)>]
       obsoletes_dir = '/mnt/koji/compose/rawhide/Fedora-Rawhide-20220203.n.1/work/global/module_obsoletes'
</LOCALS>

This patches fixes the typo in add_obsoletes() argument.

https://pagure.io/releng/failed-composes/issue/3058
Signed-off-by: Petr Písař <ppisar@redhat.com>
2022-02-04 11:15:31 +01:00
Ozan Unsal
330ba9b9c4 Do not clone the same repository multiple times, re-use already cloned repository
Clone the directory to the compose tmp directory
Update the test_scm in order to create real Compose object. Mock objects are not allowed
to create/delete files for preventing multiple clones

JIRA: RHELCMP-5250

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2022-01-26 15:57:13 +01:00