The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.
This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.
Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Nothing is calling this file as an executable, so there is no reason to
have logic for parsing arguments.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Creating an OSTree repository is a notable event; the general
expectation is that rather than having lots of repositories,
one has branches inside a single repository.
For $reasons, Fedora is not currently doing this, but we will
change it to do so.
The reason I'm making this change is we discovered that
it looked like Fedora had somehow made a repo inside a repo,
presumably due to a configuration error.
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/GBFSOLULGGZFGEFCIW6FG23NZZV5VH4K/
Signed-off-by: Colin Walters <walters@verbum.org>
There can be multiple images listed for a single variant, the config
validation should not reject it.
The syntax with a single config object is still accepted. The price for
that is less descriptive error message when there are errors.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.
Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple repos are configured in pkgset_repos, the logs should
contain information on where exactly the package was pulled from. The
log file in question should be in
`logs/<arch>/pkgset_source.<arch>.log`.
Fixes: #545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.
Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.
Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there's a temporary network issue, Pungi will fail to turn a branch
into a commit hash. This would abort the whole compose. Instead we
should just retry a few times.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.
This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.
Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.
Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This option is currently only checked in the ostree phase, and it does
not make sense as a string. When any non-empty string was given, it
enabled the check.
Relates: #590
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A `requires` attribute is taken from a wrong package (because of a wrong
variable used: `pkg` vs. `package`). On RHEL 6 this actually leads to a
crash. Let's use only one name to avoid such problems.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.
In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
As long as the assertion is not violated, this would not be a problem as
the second argument is not evaluated. However if the condition was
broken, a NameError would be raised instead of AssertionError with a
nice message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple arguments that are never used. We can just remove
them. This also fixes a bug where temporary files would be left
undeleted in tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When dependencies are pulled in, it's useful to know not only the
package that requires them, but also the specific requires.
This patch only implement this for the YUM version.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of going in random order, we should do it deterministically and
process the dependencies in order.
This should actually fix a problem where `glibc-langpack` packages are
missing on some architectures for Fedora Rawhide.
Fixes: https://pagure.io/pungi-fedora/issue/214
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.
This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).
Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config contains a git url pointing to a non-existing branch,
pungi will fail to get commit hash from that branch and die with a
confusing error message.
Fixes: #583
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When we fail to generate a volume ID that fits in 32 characters, the
error message should include the options that were considered. It could
show that there might be a substitution that could fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.
Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Instead of replacing yum.comps with an something from DNF, we can go
directly to libcomps. DNF does not have the equivalent functionality
(particularly it's impossible to load comps from file directly).
We would have depended on libcomps anyway transitively, so this is not a
big deal.
Fixes: #587
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple methods that are never used in the codebase. We don't
support using this module as a library from other programs either, so we
can just drop this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.
For example:
with schema:
schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"definitions": {
"list_of_strings": {
"type": "array",
"items": {"type": "string"},
},
"strings": {
"anyOf": [
{"type": "string"},
{"$ref": "#/definitions/list_of_strings"},
]
},
},
"properties": {
"release_name": {"type": "string"},
"repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
},
"additionalProperties": False,
}
and config:
repo = "http://url/to/repo"
repo_from = "Server"
config will be updated to:
repo = ["http://url/to/repo", "Server"]
It supports multiple append options too, like:
"repo": {
"$ref": "#/definitions/strings",
"append": ["repo_from", "source_repo_from"],
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Show warning message for any alias option find in config instance.
Example warning message:
WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This way we pay once for the expansion, and lookups are then done in
constant time instead of iterating over the list over and over again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We want to exclude all multilib packages matching the blacklist (that
are not noarch) from the package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Qixiang Wan
Colin Walters
Martin Curlej
Dennis Gilmore
Jan Kaluza
Ralph Bean
Lubomír Sedlář