When running a Bodhi update, somehow the raise condition was triggered
before any exception was raised, which caused the compose to fail.
This shouldn't really happen often, but it's possible if the machine
adjusts time for any reason (e.g. DST).
This commit moves the check for timeout after the function is called.
This can cause an extra invocation of the function, which shouldn't be a
huge deal really.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `re.match` function already anchors the pattern at the start of the
string, but allows for other characters to continue after match.
This is causing problems with packages like `elfutils-debuginfod-client`
which are not debuginfo.
Let's be safe and explicitly anchor both start and end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...
The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.
This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.
This saves around 40 seconds from pkgset phase for RHEL compose.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:
- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`
The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.
This commit removes this check and therefore symlinks are allowed
when searching for old compose.
I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.
I also executed test compose and it worked as expected.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
When probing lookasides for platform definition, we need to make sure it
works for repos specified as HTTP urls. Createrepo doesn't seem to
automatically download the repodata, so we have to help it.
JIRA: COMPOSE-3958
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make it possible to only import the library only when it's
really needed.
DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option. This is expected to be a name of
subdirectory in the repo with module defaults. If supplied, overrides
from that location are loaded every time defaults are loaded.
This raises the minimal required version of libmodulemd to 2.8.0
JIRA: COMPOSE-3828
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.
This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.
Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.
The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.
JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For scm dict resolving the return value should be git ref (or source
branch for offline mode).
JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the repo contains the same name under multiple directories, make the
resolving work by filtering only to refs/heads and refs/tags.
Fixes: https://pagure.io/pungi/issue/1180
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Split out a smaller function that receives a git repo URL and ref and
returns the commit id. The caching resolver class is modified to use
this second function if branch is given to it.
The new function checks if the ref received already looks like a commit
ID, and if so it does not attempt to do anything with it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a repo or install tree is specified as an absolute path on the local
filesystem, we should either translate it using the configured mappings,
or if no mapping matches, it should be return unchanged. A variant name
can not start with a slash, so attempting that translation does not make
much sense.
JIRA: COMPOSE-3290
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of multiple places handling the same thing duplicating the
logic, it's better to do it once upfront. This allows easy caching of
the results.
Additional advantage of this approach is that the config dump will
include resolved URLs. The original reference will still be available in
the copy of the original config.
JIRA: COMPOSE-3065
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently the `copy_all` function resolves all symlinks and crashes on
broken symlinks. Instead it should copy symlinks as symlinks.
Fixes: https://pagure.io/pungi/issue/1084
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not like the flags kwargs for re.sub(), which should not really
be needed, since \W is case insensitive anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This was not configurable for users, and the default was always used,
which meant no escaping. Might as well just remove the dead code.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not abort the compose, but it's still an error. It should be
marked as such in the log. This will allow easier searching for the
failures.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.
Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.
Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995
Signed-off-by: Sinny Kumari <sinny@redhat.com>
There are now two places where we need to do this, so we can simplify
the logic of finding and filtering them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).
JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not exist on Python 3. Converting the exception to string works
identically.
The validate methods on many phases are simplified by not calling the
parent (which does not do anything).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.
Related: https://pagure.io/pungi/issue/778
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
The previous attempt - caed78e - is not really correct. It sorts
the dict item tuples according to the alphabetical sort order of
the first item of each tuple (reversed). This will always work
when both substitutions *start* with the same characters, as in
the case of two strings that start with the same characters but
have a different length, the shorter one sorts alphabetically
first, and we reverse that. But it is not safe if the shorter
substitution doesn't start with the same characters, as in the
case I put in the tests: we should sort 'zzzaaaaaazzz' before
'aaaaaa' (and hence apply the 'zzzaaaaaazzz' substitution to a
volume ID that contains that string and not the 'aaaaaa' one),
but the previous commit did not.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Haibo Lin
Lubomír Sedlář
Jan Kaluza
Sinny Kumari
Patrick Uiterwijk
Patrick Uiterwijk
Ondrej Nosek
Adam Williamson