Commit Graph

2524 Commits

Author SHA1 Message Date
Haibo Lin
1bb038ca72 Install missing deps in ci image
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 12:01:51 +08:00
Filip Valder
efff2c9504 Use pytest directly incl. support for posargs, e.g.:
tox -- -s -vvv tests/path/to/a/single/test_something.py

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:35:47 +02:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00
Dominik Rumian
5831d4ae1e Better error message than 'KeyError' in pungi
Jira: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-12 10:52:36 +00:00
JamesKunstle
3349585d78 Adding multithreading support for pungi/phases/image_checksum.py
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.

Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.

Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
2021-08-12 10:13:15 +02:00
Ken Dreyer
5a8df7b69c doc: more additional_packages documentation
Contrast the additional_packages setting with the comps_file setting.

Explain what happens when a user lists a package in additional_packages
but Pungi cannot find it.

Give an example of composing all builds in a Koji tag.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
6afcfef919 doc: fix typo in additional_packages description
not -> nor

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
2a679dcb81 doc: improve signed packages retry docs
Reword the signed_packages_retries and signed_packages_wait
configuration option documentation to use the active voice. This makes
it easier to understand who is doing what in a signing workflow.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-11 14:07:12 -04:00
Dominik Rumian
8a2d0162d9 Better error message than 'KeyError' in pungi
JIRA: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-10 07:56:47 +00:00
Ken Dreyer
01a52447bc doc: explain buildContainer API
Explain how to discover the API documentation about the buildContainer
method, so users can discover more about how "scratch" and "priority"
work.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-05 12:42:32 -04:00
Haibo Lin
cf761633f4 4.2.10 release
JIRA: RHELCMP-6108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-04 17:23:19 +08:00
fdiprete
446334fb95 show and log command when using the run_blocking_cmd() method [RHELCMP-2243]
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-08-03 10:58:18 +00:00
Haibo Lin
56a55db966 Use cachedir when createrepo
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-07-27 09:23:47 +08:00
Lubomír Sedlář
a435fd58da gather: Add all srpms to variant lookaside repo
The original code could cause a source RPM to be present in two variants
that have a dependency relation.

There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.

This could result in duplicated SRPMs in the compose.

Example situation:

 * Variant B depends on variant A.
 * A contains foo.x86_64.rpm (only on x86_64)
 * B pulls in subpackage foo-bar.s390x.rpm (on s390x)

Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.

The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.

JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-07-19 14:12:44 +02:00
Haibo Lin
edb091b7b1 Add task URL to watch task log
JIRA: RHELCMP-5666
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-29 09:10:28 +08:00
Haibo Lin
9a5e901cfe Log warning when module defined in variants.xml not found
JIRA: RHELCMP-5573
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-25 14:55:23 +08:00
Lubomír Sedlář
bf28e8d50c pkgset: Compare future events correctly
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.

JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-12 16:17:28 +02:00
Lubomír Sedlář
7fe32ae758 util: Strip file:// from local urls
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-06 12:37:55 +02:00
Haibo Lin
c27bfe0c59 Clean up temporary yumroot dir
JIRA: RHELCMP-4948
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-29 15:57:56 +08:00
Ondrej Nosek
76d13d0062 4.2.9 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-04-29 06:59:45 +02:00
Romain Forlot
da791ed15c Fix can't link XDEV using repos as pkgset_sources
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.

Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
2021-04-22 14:03:12 +02:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
e866d22c04 gather: Adjust reusing with lookaside
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-16 10:37:47 +08:00
Lubomír Sedlář
ab1b5b48ec hybrid: Optimize getting lookaside packages
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-04-15 11:02:04 +02:00
Haibo Lin
c8091899b2 gather: Copy old logs when reusing gather result
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-01 14:42:20 +08:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Lubomír Sedlář
5b5069175d pkgset: Store module tag only if module is used
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.

If the packages are not included in any variant, they shouldn't be
available to buildinstall either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-22 10:21:56 +01:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Ondrej Nosek
98359654cf 4.2.8 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-02-12 15:20:40 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
Lubomír Sedlář
61e90fd7e0 osbs: Move metadata processing to standalone function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:28 +01:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Lubomír Sedlář
44f7eff1b7 Move UnsignedPackagesError to a separate file
This file can contain all Pungi specific exceptions.

It should also fix an issue encountered on Python 2.7:

    AttributeError: 'module' object has no attribute 'pkgsets'

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 10:16:32 +00:00
Haibo Lin
daa0ca6106 pkgset: Include just one version of module
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.

JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-02 18:16:52 +08:00
Haibo Lin
d4ee42ec23 pkgset: Check tag inheritance change before reuse
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-01 09:22:42 +08:00
Lubomír Sedlář
49a5661521 pkgset: Remove reuse file when packages are not signed
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.

Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-29 13:06:29 +01:00
Lubomír Sedlář
c87fce30ac pkgset: Drop kobo.plugin usage from PkgsetSource
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
0f4b0577f7 gather: Drop kobo.plugins usage from GatherMethod
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Lubomír Sedlář
83458f26c2 pkgset: Drop kobo.plugins usage from GatherSources
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00
Ken Dreyer
39b847094a doc: remove default createrepo_checksum value from example
createrepo_checksum already defaults to sha256. Remove this setting from
the documented Minimal Example configuration to make it easier to read.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-01-25 08:22:17 +00:00
Lubomír Sedlář
9ea1098eae comps: Preserve default arg on groupid
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-22 08:33:09 +01:00
Haibo Lin
f518c1bb7c Stop copying .git directory with module defaults
JIRA: RHELCMP-3016
Fixes: https://pagure.io/pungi/issue/1464

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-14 10:36:38 +08:00
Haibo Lin
f470599f6c React to SIGINT signal
ODCS sends SIGINT signal.

JIRA: RHELCMP-3687
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-01-11 10:02:10 +08:00
Lubomír Sedlář
cda67776d9 scm: Only copy debugging data if we have a compose
If we don't have a compose, this copy will fail. Let's prevent that.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-12-08 08:57:06 +01:00