Commit Graph

921 Commits

Author SHA1 Message Date
Filip Valder
fe986d68b9 Add module obsoletes feature
JIRA: MODULAR-113
Merges: https://pagure.io/pungi/pull-request/1578
Signed-off-by: Filip Valder <fvalder@redhat.com>
2022-01-12 12:51:37 +01:00
soksanichenko
58a16e5688 - The version is bumped
- The changelog is updated
- The test `create_packages_json` is fixed

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I173013da990eb296e58ca8f3555a05913ca1c852
2021-12-20 14:11:17 +02:00
Haibo Lin
20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Haibo Lin
f681956cf1 Fix tests for python 2.6
It failed to build RHEL 6 package as logging.NullHandler does not exist
in python 2.6

JIRA: RHELCMP-7188
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-12 17:04:17 +08:00
Lubomír Sedlář
1d654522be Remove with_jigdo argument
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
80bd254347 Check dependencies after config validation
This way the checks can rely on default values from the config.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-04 13:37:51 +00:00
Ken Dreyer
94ffa1c5c6 default "with_jigdo" to False
Fedora has not composed with jigdo in a long time. Disable it by
default.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Merges: https://pagure.io/pungi/pull-request/1561
Fixes: https://pagure.io/pungi/issue/1560
2021-11-04 13:37:51 +00:00
Lubomír Sedlář
9d02f87c99 Stop trying to validate non-existent metadata
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
2021-11-04 09:57:20 +01:00
Lubomír Sedlář
e2b3002726 repoclosure: Use --forcearch for dnf repoclosure
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.

Based on how the list of architectures is generated, we know that the
main one will always be first.

Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 08:51:40 +01:00
Lubomír Sedlář
e8305f3978 extra_isos: Allow reusing old images
When nothing in configuration or the image itself changed, let's just
copy the older one.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
ac66c3d7f3 createiso: Allow reusing old images
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:

 * If image is bootable, buildinstall phase must have been reused too.
 * Compose configuration must have not changed (except for a few
   whitelisted options).
 * Volume ID of the ISO much not have changed.
 * No RPM on the ISO must have changed.

The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.

JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-11-02 07:08:00 +00:00
Lubomír Sedlář
eb61c97cdb Remove default runroot channel
When the value is not specified in the configuration file, let Koji pick
the default channel.

JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-25 15:56:24 +02:00
Lubomír Sedlář
204d88a351 Add missing mock to osbs tests
We don't want the test to try to a dummy URL.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-10-21 13:34:37 +02:00
Haibo Lin
8133676270 osbs: Reuse images from old compose
JIRA: RHELCMP-5972
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:22:19 +08:00
Haibo Lin
e42e65783d image_build: Allow reusing old image_build results
JIRA: RHELCMP-5970
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-10-20 14:07:02 +08:00
Jan Kaluza
0530cf2712 When cts_url is configured, use CTS /repo API for buildContainer yum_repourls.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-24 10:29:28 +02:00
Jan Kaluza
195bfbefa4 Allow specifying $COMPOSE_ID in the repo value for osbs phase.
There should be an option for `yum_repourls` to point to static
URL, for example when CTS is used. The idea is that instead of
setting `repo` to `AppStream`, we could use link similar to this one:

`https://cts.localhost/api/1/composes/$COMPOSE_ID/repo/?variant=AppStream`

This would be translated to real static link during the OSBS phase:

`https://cts.localhost/api/1/composes/CentOS-Stream-9-20210803.0/repo/?variant=AppStream`

That way this statis link would appear in the yum_repourls.

Merges: https://pagure.io/pungi/pull-request/1543
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2021-09-09 10:46:33 +02:00
Lubomír Sedlář
20dc4beb6b Make getting old compose config reusable
The file will only be loaded once, it gets cached afterwards.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-09 10:38:34 +02:00
Dominik Rumian
7c3e8d4276 Fix tests for createrepo
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 13:48:29 +02:00
Dominik Rumian
9cd42a2b5e Formatted files according to flake8 and black feedback
Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 09:55:17 +02:00
Haibo Lin
1bb038ca72 Install missing deps in ci image
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 12:01:51 +08:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00
Dominik Rumian
5831d4ae1e Better error message than 'KeyError' in pungi
Jira: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-12 10:52:36 +00:00
Dominik Rumian
8a2d0162d9 Better error message than 'KeyError' in pungi
JIRA: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-10 07:56:47 +00:00
fdiprete
446334fb95 show and log command when using the run_blocking_cmd() method [RHELCMP-2243]
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-08-03 10:58:18 +00:00
Haibo Lin
56a55db966 Use cachedir when createrepo
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-07-27 09:23:47 +08:00
stepan_oksanichenko
084321dd97 LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
  An reference repo is used as main source of packages. A not reference repo is used as source
  of packages which don't exist in the reference repos.
- All cases are covered by the unittest

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
2021-06-15 17:42:12 +03:00
stepan_oksanichenko
941d6b064a LNX-318: Modify build scripts for building CloudLinux OS 8.4
- [Fixed] The script `create_packages_json` selects a first
          comer package from variant, but it should select the
          higher by version of package

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
2021-06-04 12:36:03 +03:00
stepan_oksanichenko
cc4d99441c LNX-108: Add multiarch support to pungi
@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ibfd540454941922d790ae4e56cc0992c0c85635d
2021-05-24 18:07:11 +03:00
Lubomír Sedlář
7fe32ae758 util: Strip file:// from local urls
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-06 12:37:55 +02:00
stepan_oksanichenko
b9f554bf39 LNX-311: Add ability to productmd set a main variant while dumping TreeInfo
@BS-NOBUILD
@BS-TARGET-CL8
@BS-LINKED-608ab56299ce8ac801a396c5  # python3-productmd

Change-Id: Id86d627ae8ae0b9a73b5ce6531c20538f3d040b1
2021-04-29 17:01:49 +03:00
stepan_oksanichenko
ebf028ca3b LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The modules from a parsend output of FUS should be have a stream
  with replaced dash by underscore

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
2021-04-27 13:39:09 +03:00
stepan_oksanichenko
305103a38e LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- Some modules can be absent in koji env but be present in variants.xml.
  And Pungi will fail in this case. So we must filter out those modules
  from expected modules list by list from pungi build config

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
2021-04-23 13:03:05 +03:00
stepan_oksanichenko
01bce26275 LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The script `gather_modules` should replace `-` by `_`
  in stream of modules as pungi does it in self

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
2021-04-22 13:40:48 +03:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
e866d22c04 gather: Adjust reusing with lookaside
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-16 10:37:47 +08:00
Lubomír Sedlář
ab1b5b48ec hybrid: Optimize getting lookaside packages
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-04-15 11:02:04 +02:00
Haibo Lin
c8091899b2 gather: Copy old logs when reusing gather result
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-01 14:42:20 +08:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
soksanichenko
02686d7bdf LU-2186 .treeinfo file in AlmaLinux public kickstart repo should contain AppStream variant
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo

Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
2021-02-23 06:48:15 -05:00
soksanichenko
2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
soksanichenko
5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Haibo Lin
d4ee42ec23 pkgset: Check tag inheritance change before reuse
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-01 09:22:42 +08:00
Stepan Oksanichenko
94ad7603b8 LNX-104: Create gather_prepopulate file generator for Pungi
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
2021-01-25 16:17:34 +02:00
oshyshatskyi
903db91c0f LNX-102: Patch pungi tool to use local koji mock
Instead of koji.mbox use local koji-like wrapper.

@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8

Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
2021-01-25 16:17:33 +02:00
oshyshatskyi
552343fffe LNX-102: Add tool that gathers directory for all rpms
Tool that finds all available rpm files in directory
and creates special tree for pungi:
 # ls /mnt/koji/
   i686/  noarch/  x86_64/

Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
2021-01-25 16:17:33 +02:00
oshyshatskyi
5806217041 LNX-102: Add tool that collects information about modules
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
 - module_defaults
 - modules

 First one is used by pungi during repocreate phase and
 the second one is used by koji mock to get list of
 available modules and their versions.

Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
2021-01-25 16:17:33 +02:00
Haibo Lin
2c1603c414 Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
(cherry picked from commit f518c1bb7c)
2021-01-25 14:06:33 +02:00
Haibo Lin
f518c1bb7c Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-14 10:36:38 +08:00
Lubomír Sedlář
98ddc74c16 osbuild: Fix not failing on failable tasks
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.

JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:30:20 +01:00
Lubomír Sedlář
4a048d4a85 kojiwrapper: Use gssapi_login
The krb_login method is deprecated and will be removed in 1.22

JIRA: RHELCMP-3383
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-02 14:21:31 +01:00
Christian Kellner
6998ffe694 osbuild: use task result to get build info
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
    result = {
        "composer": {
            "server": <COMPOSER_URL>,
            "id": <COMPOSE_ID>
        },
        "koji": {
            "build": <BUILD_ID>
        }
    }
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.

Adapt the test accordingly.

Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
2020-11-30 09:57:48 +01:00
Lubomír Sedlář
e7af6d2ac2 osbuild: Only send release when not empty
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 15:37:49 +01:00
Lubomír Sedlář
27bab19a5e Fix config validation for osbuild
Tests are now added for it as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-25 14:14:18 +01:00
Haibo Lin
4c88e7dc0e Use xorrisofs for creating ISOs when needed
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.

By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.

JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-11-20 09:47:49 +08:00
Jan Kaluza
c27e21ccf8 Add --respin-of argument.
It is used to defined the relation between original compose
and respun compose with a hotfix in CTS.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-19 12:26:38 +00:00
Jan Kaluza
bb8cd030ec Try reuse old gather_phase even if pkgset_koji_builds changed.
The pkgset_koji_builds influences the gather phase, but the
change itself is not a reason to not reuse old gather phase. if
new pkgset_koji_builds value leads to significant change in input
package set, we will find that later in this function when comparing
old and new package set.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 12:52:02 +01:00
Jan Kaluza
94bc5e286d Do not use shlex_quote in get_pungi_buildinstall_cmd and get_pungi_ostree_cmd.
These methods return command as list which is never serialized to str and
never executed using bash. It is instead passed directly to
`kobo.shortcuts.run`.

The `shlex_quote` usage here actually breaks the code, because it adds
quotes there which are needed only if this command would be serialized
to string and passed to bash. But this never happens. As a result,
the arguments passed to `kobo.shortcuts.run` contain those extra
quotes.

In this commit the shlex_quote is removed completely from this
part of code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-11-10 07:45:05 +00:00
Lubomír Sedlář
a45f4969f3 Add phase for building images with osbuild
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.

The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.

JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-11-04 13:04:43 +01:00
Lubomír Sedlář
f27b120cfc tests: Fix mock usage for Python 2
The `args` attribute doesn't work, let's instead access the arguments
via index.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-09-25 12:41:05 +02:00
Jan Kaluza
4623536b24 Fix wrong condition when reusing old gather phase results.
When list is used in gather_lookaside_repos, the Pungi currently
fails with an exception. This is caused by inverted condition
in the code which tries to filter-out the lookaside repos
generated during the Pungin execution pointing to different
compose variants.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-09-23 07:02:11 +00:00
Jiri Konecny
fdb2449c0e
Adapt tests to new patch-iso --work-dir parameter
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-09-15 16:07:18 +02:00
Haibo Lin
b3a55fd863 gitwrapper: Re-run git init before do full clone
Sometimes full clone failed with error fatal: Not a git repository
and we found that .git/refs/ dir is missing after shallow clone failed.

JIRA: RHELCMP-724
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-09-10 10:09:46 +08:00
Lubomír Sedlář
a6a96e40db Preserve environment when running koji commands
JIRA: RHELBLD-2479
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-25 09:07:28 +02:00
Lubomír Sedlář
e628bb91ec Fix formatting
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-08-20 15:15:13 +02:00
Jiri Konecny
40bbb4325a Adapt tests to a new get_graft_points() argument
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:13 +02:00
Haibo Lin
1e1c8533ac tests: Fix tests for old version of git
GitSCMTestCaseReal failed with error 'Unknown option: -C' when building
rpm for el7.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-08-18 12:38:20 +08:00
Jan Kaluza
2657a12c96 Allow setting CTS parent_compose_ids using --parent-compose-id option.
This is needed to track dependencies between composes in the Compose
Tracking Service.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-08-13 13:22:43 +02:00
Bohdan Khomutskyi
7a6d8303dc Replace -c parameter with --config
Also, reformat the invocation of Lorax, since
Lorax does not accept -c=%s, only -c %s.

Jira: RHELCMP-713
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-13 11:16:43 +02:00
Bohdan Khomutskyi
b899126b7e Allow squashfs-only and configuration_file in lorax_options
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.

squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.

The final goal of this change is to allow for optimization of
the installation medium size.

This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.

Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
2020-08-11 09:29:50 +00:00
Haibo Lin
3c72755814 Use pytest instead of nosetests
Nose has been in maintenance mode for the past several years
and pytest is a good replacement.

Other changes:
- Replace deprecated assertRegexpMatches with assertRegex
- Replace deprecated assertRaisesRegexp with assertRaisesRegex
- Replace deprecated SafeConfigParser with ConfigParser
- Force reinstall pytest and mock in tox virtualenv. This is because
  the globally installed packages may not work as expected(occured in
  jenkins job).

JIRA: RHELCMP-1619
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-29 14:57:16 +08:00
Haibo Lin
7e6bed9713 Retry buildinstall tasks on losetup error
JIRA: RHELCMP-1394
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-21 17:15:25 +08:00
Haibo Lin
f7167fa3b6 Allow including scratch module builds
JIRA: RHELCMP-439
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-17 09:08:36 +08:00
Lubomír Sedlář
7d00942d13 Allow test calls in any order
There are threads involved and the order is undefined. We need to accept
any order.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-07-08 08:26:59 +02:00
Haibo Lin
0ab6f48de3 Create arch repo when reusing failed
Reusing old arch repo may fail for reasons such as arch not
available in old compose or unexpected error when copying data
from old compose.

JIRA: RHELCMP-994
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-07 17:59:12 +08:00
Ondrej Nosek
d9f111edae Remove buffering when running koji commands
If the compose is aborted while koji tasks are running, we can be
left with empty log files. That complicates debugging.

JIRA: RHELCMP-1218

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2020-07-02 09:45:46 +02:00
Haibo Lin
b6605827b3 buildinstall: Improve error reporting when lorax fails
Logs of DepsolveError will be printed in the main log.

Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-17 13:17:22 +08:00
Jan Kaluza
e35c250700 Move test for unsigned packages with pkgset_koji_scratch_tasks to PkgsetPhase class.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Jan Kaluza
4a15d1351a Allow building compose with scratch builds defined by pkgset_koji_scratch_tasks.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-06-11 12:05:47 +00:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Haibo Lin
f5bfd509ab notification: Add compose_path into the messages
It's needed by the notification script added in RHELCMP-401.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 06:40:26 +00:00
Haibo Lin
7c2743fb50 Include the output of getisoimage in the error message
JIRA: RHELCMP-720
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-01 13:29:55 +08:00
Lubomír Sedlář
d6caf0785b tests: Patch time in CTS related tests
The value must be fixed to match the expected JSON.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-29 10:24:17 +02:00
Jan Kaluza
b8c3ca1abe Allow using Pungi Koji plugin for ostree phases.
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.

It is similar to `buildinstall` phase running with Koji plugin.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:23:58 +00:00
Jan Kaluza
f1eea0b5a6 Allow getting the compose id from CTS (Compose Tracking Service).
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-05-26 13:18:38 +00:00
Lubomír Sedlář
59e2aa9607 Fix flake8 issues
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-05-26 11:49:50 +02:00
Haibo Lin
d4efe17328 Support --skip-branding option in lorax
JIRA: RHELCMP-572
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-05-14 11:49:40 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Jan Kaluza
fdfaae8b71 Allow gather phase reuse on product_id change.
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.

The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-06 08:23:19 +00:00
Haibo Lin
65251d983a Reuse arch pkgset repos
JIRA: COMPOSE-4217
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-31 13:32:12 +08:00