When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.
Signed-off-by: Ralph Bean <rbean@redhat.com>
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.
Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.
This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.
The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.
Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.
Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.
This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.
In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.
Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.
For example, if "ceph-3" was missing from the SCM:
OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'
This occurred even if product_id_allow_missing was set to True.
Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.
We now see the following warning in the logs instead:
[WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}
and the compose succeeds.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.
Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.
Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.
Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.
Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
As long as the assertion is not violated, this would not be a problem as
the second argument is not evaluated. However if the condition was
broken, a NameError would be raised instead of AssertionError with a
nice message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Lubomír Sedlář
Ralph Bean
Jan Kaluza
Ken Dreyer
Qixiang Wan
Martin Curlej
Dennis Gilmore
Lubomír Sedlář