Commit Graph

2534 Commits

Author SHA1 Message Date
Lubomír Sedlář
d8d1cc520b paths: Allow customizing log file extension
If the file contents is JSON, it would be nice to have matching
extension.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-09 10:38:34 +02:00
Ozan Unsal
904a1c3271 Add authentication for updating the compose URL in CTS.
Put authentication steps in a function in order to prevent code duplication.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-09-09 07:27:55 +00:00
Lubomír Sedlář
e8ddacd10e Fix type detection for osbuild images
The image type value passed to the task doesn't match the type as it
will be recorded by Koji.

JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-09-08 09:07:00 +00:00
Ozan Unsal
b7666ba4a4 Enable pungi to send compose_url patches to CTS
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.

If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.

Jira: RHELCMP-6318

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-09-07 16:41:35 +02:00
Ozan Unsal
3d9335e90e Use xorriso instead of isoinfo when createiso_use_xorrisofs is enabled
Update get_mkisofs_cmd in createiso.py file in order to prevent using
default value. With this change it is possible to enable xorriso format

Jira: RHELCMP-6325

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-08-31 08:09:09 +00:00
Dominik Rumian
7c3e8d4276 Fix tests for createrepo
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 13:48:29 +02:00
Dominik Rumian
9cd42a2b5e Formatted files according to flake8 and black feedback
Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 09:55:17 +02:00
Ozan Unsal
980c7ba8fb Handle the pungi failures to ensure creation of log files
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.

Jira: RHELCMP-6077

Signed-off-by: Ozan Unsal <ounsal@redhat.com>
2021-08-25 11:15:23 +00:00
Haibo Lin
66dacb21e0 Add createrepo_enable_cache to configuration doc
JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-18 11:14:55 +08:00
Haibo Lin
795bbe31e3 Fix formatting
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 14:23:43 +08:00
Haibo Lin
1bb038ca72 Install missing deps in ci image
tests requiring libmodulemd are skipped due to missing deps and
this patch could fix the issue.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-17 12:01:51 +08:00
Filip Valder
efff2c9504 Use pytest directly incl. support for posargs, e.g.:
tox -- -s -vvv tests/path/to/a/single/test_something.py

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:35:47 +02:00
Filip Valder
a7c111643d Supersede ModuleStream loading with ModuleIndex
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.

Signed-off-by: Filip Valder <fvalder@redhat.com>
2021-08-12 16:32:38 +02:00
Dominik Rumian
5831d4ae1e Better error message than 'KeyError' in pungi
Jira: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-12 10:52:36 +00:00
JamesKunstle
3349585d78 Adding multithreading support for pungi/phases/image_checksum.py
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.

Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.

Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
2021-08-12 10:13:15 +02:00
Ken Dreyer
5a8df7b69c doc: more additional_packages documentation
Contrast the additional_packages setting with the comps_file setting.

Explain what happens when a user lists a package in additional_packages
but Pungi cannot find it.

Give an example of composing all builds in a Koji tag.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
6afcfef919 doc: fix typo in additional_packages description
not -> nor

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-12 07:01:49 +00:00
Ken Dreyer
2a679dcb81 doc: improve signed packages retry docs
Reword the signed_packages_retries and signed_packages_wait
configuration option documentation to use the active voice. This makes
it easier to understand who is doing what in a signing workflow.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-11 14:07:12 -04:00
Dominik Rumian
8a2d0162d9 Better error message than 'KeyError' in pungi
JIRA: RHELCMP-6107

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-10 07:56:47 +00:00
Ken Dreyer
01a52447bc doc: explain buildContainer API
Explain how to discover the API documentation about the buildContainer
method, so users can discover more about how "scratch" and "priority"
work.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2021-08-05 12:42:32 -04:00
Haibo Lin
cf761633f4 4.2.10 release
JIRA: RHELCMP-6108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-08-04 17:23:19 +08:00
fdiprete
446334fb95 show and log command when using the run_blocking_cmd() method [RHELCMP-2243]
Signed-off-by: fdiprete <fdipretre@redhat.com>
2021-08-03 10:58:18 +00:00
Haibo Lin
56a55db966 Use cachedir when createrepo
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-07-27 09:23:47 +08:00
Lubomír Sedlář
a435fd58da gather: Add all srpms to variant lookaside repo
The original code could cause a source RPM to be present in two variants
that have a dependency relation.

There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.

This could result in duplicated SRPMs in the compose.

Example situation:

 * Variant B depends on variant A.
 * A contains foo.x86_64.rpm (only on x86_64)
 * B pulls in subpackage foo-bar.s390x.rpm (on s390x)

Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.

The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.

JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-07-19 14:12:44 +02:00
Haibo Lin
edb091b7b1 Add task URL to watch task log
JIRA: RHELCMP-5666
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-29 09:10:28 +08:00
Haibo Lin
9a5e901cfe Log warning when module defined in variants.xml not found
JIRA: RHELCMP-5573
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-25 14:55:23 +08:00
Lubomír Sedlář
bf28e8d50c pkgset: Compare future events correctly
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.

JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-12 16:17:28 +02:00
Lubomír Sedlář
7fe32ae758 util: Strip file:// from local urls
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-06 12:37:55 +02:00
Haibo Lin
c27bfe0c59 Clean up temporary yumroot dir
JIRA: RHELCMP-4948
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-29 15:57:56 +08:00
Ondrej Nosek
76d13d0062 4.2.9 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-04-29 06:59:45 +02:00
Romain Forlot
da791ed15c Fix can't link XDEV using repos as pkgset_sources
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.

Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
2021-04-22 14:03:12 +02:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
e866d22c04 gather: Adjust reusing with lookaside
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-16 10:37:47 +08:00
Lubomír Sedlář
ab1b5b48ec hybrid: Optimize getting lookaside packages
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-04-15 11:02:04 +02:00
Haibo Lin
c8091899b2 gather: Copy old logs when reusing gather result
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-01 14:42:20 +08:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Lubomír Sedlář
5b5069175d pkgset: Store module tag only if module is used
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.

If the packages are not included in any variant, they shouldn't be
available to buildinstall either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-22 10:21:56 +01:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Ondrej Nosek
98359654cf 4.2.8 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-02-12 15:20:40 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
Lubomír Sedlář
61e90fd7e0 osbs: Move metadata processing to standalone function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:28 +01:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Lubomír Sedlář
44f7eff1b7 Move UnsignedPackagesError to a separate file
This file can contain all Pungi specific exceptions.

It should also fix an issue encountered on Python 2.7:

    AttributeError: 'module' object has no attribute 'pkgsets'

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 10:16:32 +00:00
Haibo Lin
daa0ca6106 pkgset: Include just one version of module
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.

JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-02 18:16:52 +08:00