Commit Graph

2895 Commits

Author SHA1 Message Date
Lubomír Sedlář
a435fd58da gather: Add all srpms to variant lookaside repo
The original code could cause a source RPM to be present in two variants
that have a dependency relation.

There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.

This could result in duplicated SRPMs in the compose.

Example situation:

 * Variant B depends on variant A.
 * A contains foo.x86_64.rpm (only on x86_64)
 * B pulls in subpackage foo-bar.s390x.rpm (on s390x)

Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.

The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.

JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-07-19 14:12:44 +02:00
Haibo Lin
edb091b7b1 Add task URL to watch task log
JIRA: RHELCMP-5666
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-29 09:10:28 +08:00
Haibo Lin
9a5e901cfe Log warning when module defined in variants.xml not found
JIRA: RHELCMP-5573
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-06-25 14:55:23 +08:00
stepan_oksanichenko
b2c49dcaf6 - The version is bumped
- The changelog is updated

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Iadbf3d7223db85a58ba82f41597de27dbfffe1ca
2021-06-18 14:47:09 +03:00
stepan_oksanichenko
14dd6a195f LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The reference packages should be replaced only by the newer reference packages
- The non-reference packages should be replaced by both of types packages

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I881bd4e58527ae219ef6e1adbc6332b3b05933c1
2021-06-18 14:23:42 +03:00
stepan_oksanichenko
084321dd97 LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
  An reference repo is used as main source of packages. A not reference repo is used as source
  of packages which don't exist in the reference repos.
- All cases are covered by the unittest

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
2021-06-15 17:42:12 +03:00
stepan_oksanichenko
941d6b064a LNX-318: Modify build scripts for building CloudLinux OS 8.4
- [Fixed] The script `create_packages_json` selects a first
          comer package from variant, but it should select the
          higher by version of package

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
2021-06-04 12:36:03 +03:00
stepan_oksanichenko
aaeee7132d - It's bumped version
- It's added changelog

@BS-TARGET-CL8

Change-Id: I51eef1eb45ba54d034e6bed46d99b0470f4e9221
2021-05-25 21:28:47 +03:00
stepan_oksanichenko
cc4d99441c LNX-108: Add multiarch support to pungi
@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Ibfd540454941922d790ae4e56cc0992c0c85635d
2021-05-24 18:07:11 +03:00
Lubomír Sedlář
bf28e8d50c pkgset: Compare future events correctly
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.

JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-12 16:17:28 +02:00
Lubomír Sedlář
7fe32ae758 util: Strip file:// from local urls
Make sure that the function returns a path even for local files
specified by file:// urls.

JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-05-06 12:37:55 +02:00
stepan_oksanichenko
a435eeed06 - it's added changelog
@BS-NOBUILD

Change-Id: I3a0a0377f9c1cefabf52c33fbc0d19ab0e4fe4f1
2021-04-29 17:15:17 +03:00
stepan_oksanichenko
b9f554bf39 LNX-311: Add ability to productmd set a main variant while dumping TreeInfo
@BS-NOBUILD
@BS-TARGET-CL8
@BS-LINKED-608ab56299ce8ac801a396c5  # python3-productmd

Change-Id: Id86d627ae8ae0b9a73b5ce6531c20538f3d040b1
2021-04-29 17:01:49 +03:00
Haibo Lin
c27bfe0c59 Clean up temporary yumroot dir
JIRA: RHELCMP-4948
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-29 15:57:56 +08:00
Ondrej Nosek
76d13d0062 4.2.9 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-04-29 06:59:45 +02:00
stepan_oksanichenko
ebf028ca3b LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The modules from a parsend output of FUS should be have a stream
  with replaced dash by underscore

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
2021-04-27 13:39:09 +03:00
stepan_oksanichenko
305103a38e LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- Some modules can be absent in koji env but be present in variants.xml.
  And Pungi will fail in this case. So we must filter out those modules
  from expected modules list by list from pungi build config

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
2021-04-23 13:03:05 +03:00
Romain Forlot
da791ed15c Fix can't link XDEV using repos as pkgset_sources
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.

Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
2021-04-22 14:03:12 +02:00
stepan_oksanichenko
01bce26275 LNX-286: Prepare pungi configuration and setup Jenkins job for AlmaLinux 8.4 beta
- The script `gather_modules` should replace `-` by `_`
  in stream of modules as pungi does it in self

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
2021-04-22 13:40:48 +03:00
Lev Veyde
00a9861367 Updated the deprecated ks argument name (to the current inst.ks)
Signed-off-by: Lev Veyde <lveyde@redhat.com>
2021-04-21 21:04:01 +03:00
Haibo Lin
e866d22c04 gather: Adjust reusing with lookaside
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused

JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-16 10:37:47 +08:00
Lubomír Sedlář
ab1b5b48ec hybrid: Optimize getting lookaside packages
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.

The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.

Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:

 * dnf repoquery - this doesn't include modular packages unless the
   stream is default
 * dnf reposync - requires `--urls` option to only print the names,
   which is not available on RHEL 7

JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-04-15 11:02:04 +02:00
Haibo Lin
c8091899b2 gather: Copy old logs when reusing gather result
This would be helpful for debugging.

JIRA: RHELCMP-4594
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-04-01 14:42:20 +08:00
Haibo Lin
035b37c566 Cancel koji tasks when pungi terminated
JIRA: RHELCMP-4148
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-23 14:47:48 +08:00
Haibo Lin
edb4517e80 Add Dockerfile for building testing image
There are two images because it's hard to install both Python 2
and Python 3 packages (e.g. libcomps) in latest fedora release.

JIRA: RHELCMP-4580
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-15 18:13:25 +08:00
Lubomír Sedlář
535034ef91 image_container: Fix incorrect arch processing
OSBS will reject no scratch builds with arch_override.

When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.

With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.

JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-03-08 11:57:48 +01:00
Haibo Lin
2769232b72 runroot: Adjust permissions always
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.

JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-04 16:08:06 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
soksanichenko
4d763514c1 - Version is bumped
- Changelog is added

Change-Id: I440b44f12c4a1aa41619acd3ba5ca354dc71b419
2021-02-24 17:42:22 +02:00
Lubomír Sedlář
735bfaa0d6 pkgset: Fix meaning of retries
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-23 16:01:41 +01:00
Danylo Kuropiatnyk
41381df6a5 LU-2202: Start unittests during installation or build of pungi
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version

@BS-TARGET-CL8

Change-Id: I036db225646875eb610736cd26f473850a78447c
2021-02-23 07:55:36 -05:00
soksanichenko
02686d7bdf LU-2186 .treeinfo file in AlmaLinux public kickstart repo should contain AppStream variant
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo

Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
2021-02-23 06:48:15 -05:00
soksanichenko
2e48c9a56f LU-2195 Change path to sources and iso when generating repositories
- We should add the images to the compose if they will be used only as netinstall image.
  E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
  E.g. *-minimal.iso

Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
2021-02-22 13:23:48 +02:00
Lubomír Sedlář
5b5069175d pkgset: Store module tag only if module is used
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.

If the packages are not included in any variant, they shouldn't be
available to buildinstall either.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-22 10:21:56 +01:00
Lubomír Sedlář
477dcf37d9 Store extended traceback for gather errors
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.

This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.

JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-17 11:11:26 +01:00
Ondrej Nosek
98359654cf 4.2.8 release
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2021-02-12 15:20:40 +01:00
Lubomír Sedlář
64897d7d48 pkgset: Add ability to wait for signed packages
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.

JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-11 15:31:14 +01:00
soksanichenko
b3a8c3f28a - Version is bumped
- Changelog is added

Change-Id: Ib1366f1fe2639037db99b8e939537bb63801058e
2021-02-11 14:50:12 +02:00
Lubomír Sedlář
40133074b3 Add image-container phase
This phase runs after image-build and osbuild and can embed an image
into a container.

JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-09 14:47:57 +01:00
soksanichenko
5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
Lubomír Sedlář
61e90fd7e0 osbs: Move metadata processing to standalone function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:28 +01:00
Lubomír Sedlář
36373479db Move container metadata into compose object
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 13:23:26 +01:00
Lubomír Sedlář
44f7eff1b7 Move UnsignedPackagesError to a separate file
This file can contain all Pungi specific exceptions.

It should also fix an issue encountered on Python 2.7:

    AttributeError: 'module' object has no attribute 'pkgsets'

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-02-08 10:16:32 +00:00
Haibo Lin
daa0ca6106 pkgset: Include just one version of module
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.

JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-02 18:16:52 +08:00
Haibo Lin
d4ee42ec23 pkgset: Check tag inheritance change before reuse
JIRA: RHELCMP-2453
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-02-01 09:22:42 +08:00
Lubomír Sedlář
49a5661521 pkgset: Remove reuse file when packages are not signed
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.

Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-29 13:06:29 +01:00
soksanichenko
3b5501b4bf LNX-133: Create a server for building nightly builds of AlmaLinux
- It's added key argument '--json-output-path' to script `pungi-generate-package-json`

Change-Id: Ic18fa2708cc4913002023828b3be018d4907de25
2021-01-28 14:03:40 +02:00
soksanichenko
cea8d92906 Bump version for setup.py
Change-Id: I980e9ebb728c3a88597c987d585e1b5937499e81
2021-01-28 00:06:40 +02:00
soksanichenko
1a29de435e - It's added changelog
- Its bumped version

Change-Id: I4c7b8d9c64da3379a24d93837657cec2686a8511
2021-01-27 23:47:39 +02:00
Lubomír Sedlář
c87fce30ac pkgset: Drop kobo.plugin usage from PkgsetSource
Relates: https://pagure.io/pungi/issue/1488
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2021-01-27 15:55:21 +01:00