The image type value passed to the task doesn't match the type as it
will be recorded by Koji.
JIRA: RHELCMP-5727
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If cts_keytab is also enabled then the HTTP requests are handled with
Kerberos Authentication otherwise no authentication is used.
If cts_url is defined in the configuration, translate_paths is required.
This is needed in order to get the host and the path of the composes.
Jira: RHELCMP-6318
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
Update get_mkisofs_cmd in createiso.py file in order to prevent using
default value. With this change it is possible to enable xorriso format
Jira: RHELCMP-6325
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
If the given directory is not a valid git directory, it raises RuntimeError.
This can be catched and raised as GitUrlResolveError, so compose can continue
to log the failure.
Jira: RHELCMP-6077
Signed-off-by: Ozan Unsal <ounsal@redhat.com>
- Use ModuleIndex's update_from_file/update_from_string instead of ModuleStream's
read_file/read_string which is deprecated.
- Extend tests to work with real module streams instead of mocks.
Signed-off-by: Filip Valder <fvalder@redhat.com>
Multithreading was added to parallelize the
computation of image checksums. Resulting memory structures
are protected via synchronization primitives. Max number of
threads is uncapped- experiments were done to determine
whether a maximum number of threads would yield greater
efficiency and there were no gains from this.
Likewise, experiments were done to determine whether pools of
threads computed in separate processes could likewise decrease
compute-time. Evidence did not suggest that this was the
case. This indicate that the checksum operation is bounded
by I/O read/write times.
Merges: https://pagure.io/pungi/pull-request/1520
Jira: RHELCMP-5967
Signed-off-by: James Kunstle jkunstle@redhat.com
The original code could cause a source RPM to be present in two variants
that have a dependency relation.
There is always only one source repo for a variant in the final compose.
When gathering packages for a variant that depends on another variant,
we need to build a temporary lookaside repo that has similar content to
the parent variant. This lookaside only contained source RPMs for
packages present the the architecture.
This could result in duplicated SRPMs in the compose.
Example situation:
* Variant B depends on variant A.
* A contains foo.x86_64.rpm (only on x86_64)
* B pulls in subpackage foo-bar.s390x.rpm (on s390x)
Source repo for A will correctly contain foo.src.rpm. With original code
the srpm would also end up in B.src. By adding all sources to the
temporary lookaside Pungi will know that source repo for B doesn't need
to duplicate the package.
The refactoring to use a set to store the packages is meant to avoid
listing the same SRPM multiple times in the repo in the most common
situation when SRPM is listed in multiple architectures.
JIRA: RHELCMP-6002
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- The reference packages should be replaced only by the newer reference packages
- The non-reference packages should be replaced by both of types packages
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I881bd4e58527ae219ef6e1adbc6332b3b05933c1
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
An reference repo is used as main source of packages. A not reference repo is used as source
of packages which don't exist in the reference repos.
- All cases are covered by the unittest
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
- [Fixed] The script `create_packages_json` selects a first
comer package from variant, but it should select the
higher by version of package
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I36268f2a493897fc11e787c040066d2d501a1c81
It is possible to try to re-run a compose with old event. When trying to
reuse pkgset data, we must use set the bounds not based on
current/reused event, but actually check which was first.
JIRA: CWFHEALTH-495
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Make sure that the function returns a path even for local files
specified by file:// urls.
JIRA: RHELCMP-5340
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- The modules from a parsend output of FUS should be have a stream
with replaced dash by underscore
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: If36d3d0a1ef8010bf85a4a0218b9838e0888453c
- Some modules can be absent in koji env but be present in variants.xml.
And Pungi will fail in this case. So we must filter out those modules
from expected modules list by list from pungi build config
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: I22c15c42868412e34fd554030130bd7c3e25b8ef
Trying to compose from external classic repositories return an error trying the hardling from
a yum cache directory located in /tmp to the target directory in another filesystem.
This commit fixes this using the 'link' method form linker module which handle the link_type
configuration parameter instead of the hardcoded method 'hardlink'.
Change-Id: Ib79cfbd72f9def6462fddb2ae368730c55f257cd
Signed-off-by: Romain Forlot <romain.forlot@iot.bzh>
- The script `gather_modules` should replace `-` by `_`
in stream of modules as pungi does it in self
@BS-NOBUILD
@BS-TARGET-CL8
Change-Id: Iea05b70afbf80f3ccd20ad4943c9d86c7ed7aa90
- Do not reuse when there is any external lookaside repo
- Do not reuse when lookaside variant is not reused
JIRA: RHELCMP-4596
Signed-off-by: Haibo Lin <hlin@redhat.com>
The original code ended up downloading all repodata from the lookaside
repo. This could cause a lot of memory to be used.
The new code only downloads the repomd.xml and then primary record,
which is sufficient to obtain all needed information. A lot less memory
is used and the code is also significantly faster.
Here are some alternative ways of getting a list of packages from the
lookaside repo and reasons why they did not work:
* dnf repoquery - this doesn't include modular packages unless the
stream is default
* dnf reposync - requires `--urls` option to only print the names,
which is not available on RHEL 7
JIRA: RHELCMP-4761
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OSBS will reject no scratch builds with arch_override.
When the option is not specified in Pungi, it would do `"".split(" ")`
to get list of arches, which returns a list with empty string instead of
an empty list.
With this fixed, it might be possible to have multiple images match the
spec (unless arch is used in the filter). To fix that, we can replace
arch with $basearch variable.
JIRA: RHELCMP-3824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Previously commands to adjust permissions do not run when main
command failed and then files can't be cleaned up due to
Permission Denied problem.
JIRA: RHELCMP-4253
Signed-off-by: Haibo Lin <hlin@redhat.com>
The name brings a different expectation than how it actually worked.
This patch makes the code work similarly to the expectation.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* added section with tests and pytest module to requires
IMPORTANT - build.sh script is commented
* added pyfakefs dependency
* fixed little mock_open issue for runroot test
* bumped version
@BS-TARGET-CL8
Change-Id: I036db225646875eb610736cd26f473850a78447c
- We are modifying existing repo's .treeinfo:
-- Take info about included variants from iso's .treeinfo and put it to repo's .treeinfo
Change-Id: I29bf655d90994e8a1bda40ad04568dd7364f5dca
- We should add the images to the compose if they will be used only as netinstall image.
E.g. *-boot.iso.
- And we shouldn't add if the images will be modified in phase `extra_isos`.
E.g. *-minimal.iso
Change-Id: I9095cfd87414ecca46b1213553589731c82dd2e2
When a module is skipped from the compose, we should not add it to a
mapping of module tags. If it's there, we then spend time building a
repo for the module, and it get's passed to buildinstall, despite the
packages not being supposed to be included in the compose.
If the packages are not included in any variant, they shouldn't be
available to buildinstall either.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a gathering thread raises an exception, it gets forwarded to the
main thread and re-raised there. However, during this transition it
loses details about exact location of the problem.
This patch creates an extended traceback in the worker, which should
make it easier to track the problem down later.
JIRA: RHELCMP-4259
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If packages are appearing quickly in Koji, and signing them is triggered
by automation, there may be a delay between the package being signed and
compose running. In such case it may be preferable to wait for the
signed copy rather than fail the compose.
JIRA: RHELCMP-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs after image-build and osbuild and can embed an image
into a container.
JIRA: RHELCMP-3820
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
@BS-TARGET-CL8
@BS-NOBUILD
- It's added the script which can collect packages/modules
from the remote repos (including BS repos) and merge them
to an one local repo with the right repodata (including
modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages
Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
Rather than tracking this directly in OSBS phase, move this into Compose
object, which will allow access to this from multiple phases.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This file can contain all Pungi specific exceptions.
It should also fix an issue encountered on Python 2.7:
AttributeError: 'module' object has no attribute 'pkgsets'
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When adding extra modules via option *pkgset_koji_module_builds*, all
other versions of the same stream potentially available in a Brew tag
should be skipped.
JIRA: RHELCMP-3689
Signed-off-by: Haibo Lin <hlin@redhat.com>
In such case we never want to reuse the pkgset, as it risks leaking
unsigned packages. Safest option is to remove the file completely.
Fixes: https://pagure.io/pungi/issue/1480
JIRA: RHELCMP-3720
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.
@BS-LINKED-5ffda6156f44affc6c5ea239 # pungi & dependencies
@BS-TARGET-CL8
Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
Instead of koji.mbox use local koji-like wrapper.
@BS-LINKED-5ff8b8cb6f44affc6c5e9a7a
@BS-TARGET-CL8
Change-Id: I82a2bc8bc71ae06240656898f3df71bb28bcb9e9
Tool that finds all available rpm files in directory
and creates special tree for pungi:
# ls /mnt/koji/
i686/ noarch/ x86_64/
Change-Id: Ibcf2d23c46411ad89477058f4d56e07ca117f0d1
Add special tool that gathers given modules.tar.gz files
and collects information about modules into two dirs:
- module_defaults
- modules
First one is used by pungi during repocreate phase and
the second one is used by koji mock to get list of
available modules and their versions.
Change-Id: I50a095a5f3bafa7e7a1effc2c0d4a2fc52ba603b
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
(cherry picked from commit 9ea1098eae)
When the wrapper processes comps file, it wasn't emitting "default"
argument for groupid element. The default is false and most entries are
actually using the default, so let's only emit it if set to true.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1882358
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The task can only fail as a whole, thus it only makes sense to set all
architectures (`*`) as failable. The correct value needs to be checked
though.
JIRA: RHELCMP-3412
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of parsing the log file to get the NVR and then in turn
use that to get to the build info use the structured return value
from the koji task. The return value of the osbuild plugin is:
result = {
"composer": {
"server": <COMPOSER_URL>,
"id": <COMPOSE_ID>
},
"koji": {
"build": <BUILD_ID>
}
}
This means we have direct access to the koji build id, which was
returned by composer to the plugin via its status API. Using that
removes the need to parse the log file.
Adapt the test accordingly.
Merges: https://pagure.io/pungi/pull-request/1475
Signed-off-by: Christian Kellner <christian@kellner.me>
A new configuration *createiso_use_xorrisofs* is added to determine
which tool to use for creating ISOs.
By default, createiso_use_xorrisofs = False and genisoimage will be used.
When set to True, xorrisofs will be used.
JIRA: RHELCMP-2875
Fixes: https://pagure.io/pungi/issue/1130
Signed-off-by: Haibo Lin <hlin@redhat.com>
Previously, the pkg_object has been found by iterating over pkg_sets
for each package. This was quite slow given the number of RPMs in the
compose.
In this commit, the temporary dict is created which stores mapping
between path and pkg_obj and is used instead.
In this commit, the get_package_path is called just once instead of
twice in the loop which might also save some time.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The pkgset_koji_builds influences the gather phase, but the
change itself is not a reason to not reuse old gather phase. if
new pkgset_koji_builds value leads to significant change in input
package set, we will find that later in this function when comparing
old and new package set.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
These methods return command as list which is never serialized to str and
never executed using bash. It is instead passed directly to
`kobo.shortcuts.run`.
The `shlex_quote` usage here actually breaks the code, because it adds
quotes there which are needed only if this command would be serialized
to string and passed to bash. But this never happens. As a result,
the arguments passed to `kobo.shortcuts.run` contain those extra
quotes.
In this commit the shlex_quote is removed completely from this
part of code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When testing if a variant has some modules,
it's no longer enough to look at `variant.modules`. That attribute
contains only names of modules configured in variants.xml.
With modules being specified in `pkgset_koji_module_builds` or
`pkgset_scratch_modules` options it is possible the code wouldn't
trigger even if there was a module.
JIRA: RHELCMP-3054
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When Pungi receives a signal to terminate, it can sometimes get stuck
if there are threads running. It has to wait for all worker threads to
finish. They generally do finish, unless they get stuck waiting on a
subprocess.
This patch should reduce the likelihood of this happening by stopping
all subprocesses.
JIRA: RHELCMP-3056
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is similar to image-build in terms of what it does, and somewhat
similar to OSBS phase in how it's implemented.
The phase reads configuration, submits the build via XMLRPC call and
waits for the task to finish. Then it downloads the built image and
includes it in the compose metadata.
JIRA: RHELCMP-315
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Without that, the resulting string is `--foo="'bar'"`
which results in `'bar'` being passed to Koji task.
With this commit, the resulting string is `"'--foo=bar'"`
which results in `bar` being passed to Koji which is expected.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When list is used in gather_lookaside_repos, the Pungi currently
fails with an exception. This is caused by inverted condition
in the code which tries to filter-out the lookaside repos
generated during the Pungin execution pointing to different
compose variants.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This is useful when running pungi-patch-iso on VM with low amount of
memory but higher disk space. Without this option the operation will
fail because /tmp is tmpfs filesystem.
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
There are two thread pools for making package sets. If Pungi is being
terminated by external event and the exception is handled in the first
thread, the second one never gets to the `stop` method and the process
keeps hanging.
This patch should make sure that `stop()` is called on both pools.
JIRA: RHELCMP-2459
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Sometimes full clone failed with error fatal: Not a git repository
and we found that .git/refs/ dir is missing after shallow clone failed.
JIRA: RHELCMP-724
Signed-off-by: Haibo Lin <hlin@redhat.com>
When running a Bodhi update, somehow the raise condition was triggered
before any exception was raised, which caused the compose to fail.
This shouldn't really happen often, but it's possible if the machine
adjusts time for any reason (e.g. DST).
This commit moves the check for timeout after the function is called.
This can cause an extra invocation of the function, which shouldn't be a
huge deal really.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Also, reformat the invocation of Lorax, since
Lorax does not accept -c=%s, only -c %s.
Jira: RHELCMP-713
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
The change allows for setting the parameters as described below to Lorax.
Lorax, a program called during the buildInstall phase, creates the SquashFS
during the buildInstall phase.
The Squash filesystem is present both on the DVD and the BOOT.ISO.
squashfs_only --- (str) passes --squashfs-only option.
configuration_file --- (str or scm_dict) passes -c option to Lorax.
The final goal of this change is to allow for optimization of
the installation medium size.
This pull request is related to the Fedora change proposal, which is available
at this location:
https://fedoraproject.org/wiki/Category:Changes/OptimizeSquashFS
See the change proposal for more information about the benefits of higher
compression ratio.
Jira: RHELCMP-693
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
This uses a previous change that added ability to retry and simply
extends the conditions when it fires.
JIRA: RHELCMP-1863
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Reusing old arch repo may fail for reasons such as arch not
available in old compose or unexpected error when copying data
from old compose.
JIRA: RHELCMP-994
Signed-off-by: Haibo Lin <hlin@redhat.com>
If the compose is aborted while koji tasks are running, we can be
left with empty log files. That complicates debugging.
JIRA: RHELCMP-1218
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
Previously, we checked only for `compose_dir`, but it makes
more sense to really check for `composeinfo-base.json` instead.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This commit also adds a --config argument allowing to
override/specify a specific fedora-messaging configuration
file to use.
Signed-off-by: Pierre-Yves Chibon <pingou@pingoured.fr>
Logs of DepsolveError will be printed in the main log.
Fixes: https://pagure.io/pungi/issue/1399
JIRA: RHELCMP-955
Signed-off-by: Haibo Lin <hlin@redhat.com>
This patch updates the documentation to match the actual behavior, and
adds a configuration option to remove the leading prefix.
The extra wildcard is causing problems when there are two variants in
the compose and one UID is a suffix of the other (e.g. DevTools and
Tools), since multiple files will match the shorter name and an error
will be reported.
JIRA: RHELCMP-1086
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.
JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This commits changes `ostree` and `ostree_installer` phases
so they can run with Koji Pungi plugin instead of the plain runroot.
It is similar to `buildinstall` phase running with Koji plugin.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This will eliminate 2 calls to os.stat per one invocation of the _link_file function.
Assuming during the compose build 50000 files are linked, this change will eliminate 100000 redundant calls to os.stat.
Jira: RHELCMP-797
Signed-off-by: Bohdan Khomutskyi <bkhomuts@redhat.com>
We have seen the directory created with wrong permissions. Since we
haven't been able to find out why it happens this is a workaround.
JIRA: RHELCMP-142
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If phase is skipped in a config option (and not on command line), we
should just print a warning or do nothing in quiet mode.
JIRA: RHELCMP-399
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the input packages contain e.g. `ansible`, and there's
`ansible-runner-service` in the package set, we don't want to pull it
in.
JIRA: RHELCMP-446
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After a file is copied, a check was done if size and mtime of the files
match.
This has very few benefits in what problems it can detect, and can cause
problems on NFS where it can take a short time before the updated
attributes are visible on the new location.
JIRA: RHELCMP-378
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:
- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.
This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:
- lorax commandline arguments are the same (except of expected
differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
the ones in package sets in the current compose.
By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add tests for buildinstall reuse and buildinstall_allow_reuse option.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.
JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.
The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
GitWrapper._clone crashes occasionally but unfortunately
the root cause is not found yet. We need more info for debugging.
JIRA: COMPOSE-4219
Signed-off-by: Haibo Lin <hlin@redhat.com>
The `re.match` function already anchors the pattern at the start of the
string, but allows for other characters to continue after match.
This is causing problems with packages like `elfutils-debuginfod-client`
which are not debuginfo.
Let's be safe and explicitly anchor both start and end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...
The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.
This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.
This saves around 40 seconds from pkgset phase for RHEL compose.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
This change adds an option which allows users to specify
the number of version components (i.e. values between dots)
The default behavior is preserved, and is equivalent to using
'-1' as the value.
- Negative values remove items from the end.
- Positive values specify the count of of components
- Zero will remove the version field entirely from the symlink
- When nonzero, at least one version component will appear
Merges: https://pagure.io/pungi/pull-request/1361
Signed-off-by: Lon Hohberger <lhh@redhat.com>
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
did not change, the "names" of RPMs in global package set
did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.
In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:
- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`
The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.
This commit removes this check and therefore symlinks are allowed
when searching for old compose.
I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.
I also executed test compose and it worked as expected.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If the signing machinery crashes, we don't want to wait forever. This
way at least we can shout for help periodically.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1350
JIRA: COMPOSE-4166
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When running with `--quiet`, the warning should not be printed. It would
confuse tools that expect only compose path to be printed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It is useful to actually check how the schema changed after applying
--schema-override. This commit changes --dump-schema in a way that
--schema-override is taken into account and dumped schema contains
the changes done using the --schema-override.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.
There is currently no way how to check that Pungi configuration fulfills
these extra requirements.
This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.
For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:
```
{
"properties": {
"pkgset_source": {
"enum": ["koji"]
}
}
}
```
It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.
Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Newer versions of Python report an error when a file is not closed.
Let's avoid it by using with statement.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.
The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939
This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.
The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).
JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.
In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.
The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.
However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.
The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.
The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.
This commit allows running `createiso` phase locally while keeping the other
phases in Koji.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
It is possible we only ever see a package from non-lookaside repo, but
it actually is in lookaside. This patch should check for that before
adding the package to result set.
JIRA: RCM-71946
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The final set of packages can contain multiple copies of the same
package if it's present in multiple repositories. Since they have the
same NVRA and were built in Koji, we know they are identical and should
merge their flags.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Jan Kaluza
Ozan Unsal
Lubomír Sedlář
Dominik Rumian
Haibo Lin
Filip Valder
JamesKunstle
fdiprete
stepan_oksanichenko
Romain Forlot
Lev Veyde
Danylo Kuropiatnyk
oshyshatskyi
Christian Kellner
Jiri Konecny
Bohdan Khomutskyi
Ondrej Nosek
Pierre-Yves Chibon
Frédéric Pierret (fepitre)
Dusty Mabe
Buvanesh Kumar
Lon Hohberger