This should cover case where we there's a build like this:
foo-1-1.src.rpm
foo-sub-1-1.noarch.rpm
foo-debugsource-1-1.x86_64.rpm
The compose contains the noarch package, and should also have the
debugsource package. The original code only checked for
foo-sub-debugsource though.
JIRA: RHELCMP-7628
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- Scripts `create_packages_json` & `gather_modules` can process lzma compressed yaml files
- Script `create_package_json` can use repodata there are packages with different
arch by compare with passed to the script
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: Ia9a3bacfa4344f0cf33b9f416649fd4a5f8d3c37
When the temporary directory is created with 0700, other programs
(potentially on another host) will have problems reading it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: RHELCMP-7635
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.
This does not fix the issue but it would be helpful for debugging when
it occurs again.
JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
- Script `create_packages_json` can duplicate the packages with
same version in different variants
Change-Id: I3c79ad06c4c22442423c12d5fa06baf82d663a3f
The JSON source file should not require a mapping for all
variants/architectures. When something is specified, it should be
included.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It was checked in a condition together with the configuration value, and
only ever explicitly used with the same value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a compose doesn't build any images, it won't produce any metadata
file for them, and thus it makes no sense to validate it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1565
DNF repoclosure requires this option when checking a repository that is
not compatible with host architecture. It seems that when it is
compatible, it works as well.
Based on how the list of architectures is generated, we know that the
main one will always be first.
Fixes: https://pagure.io/pungi/issue/1562
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When nothing in configuration or the image itself changed, let's just
copy the older one.
JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch allows Pungi to reuse ISO image created in previous compose
if a list of assumptions proves to hold:
* If image is bootable, buildinstall phase must have been reused too.
* Compose configuration must have not changed (except for a few
whitelisted options).
* Volume ID of the ISO much not have changed.
* No RPM on the ISO must have changed.
The ISO also contains other files. Changes in extra files and product ID
certificates should be visible in configuration (the SHA will differ).
Similarly any repodata configuration would be reflected in
configuration.
JIRA: RHELCMP-5969
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the value is not specified in the configuration file, let Koji pick
the default channel.
JIRA: RHELBLD-8088
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When ODCS starts a compose, it will provide base composeinfo file, but
it doesn't create COMPOSE_ID. This leads to a crash when updating CTS,
since the compose id can't be read from the file. We can instead use the
value we already have in memory.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The image type value passed to the task doesn't match the type as it
will be recorded by Koji.
JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.
If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.
Jira: RHELCMP-6318
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
Update get_mkisofs_cmd in createiso.py file in order to prevent using
default value. With this change it is possible to enable xorriso format
Jira: RHELCMP-6325
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.
Jira: RHELCMP-6077
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.
Signed-off-by: Filip Valder <fvalder@redhat.com>
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.
Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.
Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
The original code could cause a source RPM to be present in two variants
that have a dependency relation.
There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.
This could result in duplicated SRPMs in the compose.
Example situation:
* Variant B depends on variant A.
* A contains foo.x86_64.rpm (only on x86_64)
* B pulls in subpackage foo-bar.s390x.rpm (on s390x)
Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.
The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.
JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- The reference packages should be replaced only by the newer reference packages
- The non-reference packages should be replaced by both of types packages
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I881bd4e58527ae219ef6e1adbc6332b3b05933c1
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
An reference repo is used as main source of packages. A not reference repo is used as source
of packages which don't exist in the reference repos.
- All cases are covered by the unittest
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
- [Fixed] The script `create_packages_json` selects a first
comer package from variant, but it should select the
higher by version of package
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.
JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make sure that the function returns a path even for local files
specified by file:// urls.
JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- The modules from a parsend output of FUS should be have a stream
with replaced dash by underscore
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
- Some modules can be absent in koji env but be present in variants.xml.
And Pungi will fail in this case. So we must filter out those modules
from expected modules list by list from pungi build config
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.
Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
- The script `gather_modules` should replace `-` by `_`
in stream of modules as pungi does it in self
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused
JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.
The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.
Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:
* dnf repoquery - this doesn't include modular packages unless the
stream is default
* dnf reposync - requires `--urls` option to only print the names,
which is not available on RHEL 7
JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OSBS will reject no scratch builds with arch_override.
When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.
With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.
JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.
JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version
@BS-TARGET-CL8
Change-Id: I036db225646875eb610736cd26f473850a78447c
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo
Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
- We should add the images to the compose if they will be used only as netinstall image.
E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
E.g. *-minimal.iso
Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.
If the packages are not included in any variant, they shouldn't be
available to buildinstall either.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.
This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.
JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.
JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs after image-build and osbuild and can embed an image
into a container.
JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
@BS-TARGET-CL8
@BS-NOBUILD
- It's added the script which can collect packages/modules
from the remote repos (including BS repos) and merge them
to an one local repo with the right repodata (including
modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages
Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This file can contain all Pungi specific exceptions.
It should also fix an issue encountered on Python 2.7:
AttributeError: 'module' object has no attribute 'pkgsets'
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.
JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.
Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.
@BS-LINKED-5ffda6156f44affc6c5ea239 # pungi & dependencies
@BS-TARGET-CL8
Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
Instead of koji.mbox use local koji-like wrapper.
@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8
Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
Tool that finds all available rpm files in directory
and creates special tree for pungi:
# ls /mnt/koji/
i686/ noarch/ x86_64/
Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
- module_defaults
- modules
First one is used by pungi during repocreate phase and
the second one is used by koji mock to get list of
available modules and their versions.
Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 9ea1098eae)
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Filip Valder
Ozan Unsal
Haibo Lin
soksanichenko
Ken Dreyer
fdiprete
Jan Kaluza
Dominik Rumian
JamesKunstle
Romain Forlot
Lev Veyde
Danylo Kuropiatnyk
oshyshatskyi