When allowing empty list of modules, the check for variant tags got
broken, causing Everything to no longer have an associated list of
allowed packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Relates: https://pagure.io/pungi/issue/862
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no guarantee that it will print any text. We don't even need
the output, we just print it to error log if there is a problem.
Fixes: https://pagure.io/pungi/issue/847
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no place that would possibly call it, so we can drop the check
and function for generating cli.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can use a key function instead of relying to the deprecated cmp. This
makes the code work on Python 2.6 and on recent versions it makes it
faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config for image-build command contains multiple values, they
should be joined with commas into a single value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3, configparser will reject non string values, and
theoretically we could have some in the configuration.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a custom server CA certificate, it needs to be generally
available, and not just used when logging in so that SSL verification
works.
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.
The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.
Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.
There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972
As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.
Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.
In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.
Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.
Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.
This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A `requires` attribute is taken from a wrong package (because of a wrong
variable used: `pkg` vs. `package`). On RHEL 6 this actually leads to a
crash. Let's use only one name to avoid such problems.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple arguments that are never used. We can just remove
them. This also fixes a bug where temporary files would be left
undeleted in tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.
This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).
Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of replacing yum.comps with an something from DNF, we can go
directly to libcomps. DNF does not have the equivalent functionality
(particularly it's impossible to load comps from file directly).
We would have depended on libcomps anyway transitively, so this is not a
big deal.
Fixes: #587
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple methods that are never used in the codebase. We don't
support using this module as a library from other programs either, so we
can just drop this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Remove dead imports
* Remove dead code
* Fix extra empty lines
* Fix formatting
* Break too long lines
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Split the function for getting implant MD5 into two functions, so that
we can have access to other information retrieved by checkisomd5
command. A test case is added for both functions.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.
This fixes tests leaving garbage around in /tmp.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.
Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a lot of options that are not used by any code path in Pungi.
There is not reason to provide access to them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Using getattr so we also work with versions of koji that do not
read principal and keytab from the config file.
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of spawning a separate process for each test, move the code to
run in the main process. This gives us correct coverage information and
makes the tests a lot faster.
The input is still provided in a kickstart file. The output also goes
into a log file that is later parsed.
There is some extra work needed with logging setup to make the logs go
to standard output so that nose can correctly capture them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Tell lorax to use a specific directory for log files so that we preserve
them despite koji not having any idea about them.
Fixes: #457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It really is just a group of independent functions, so we can simplify
it by removing the unused wrapper class. Instead of importing the
wrapper, instantiating it and calling its methods we can import the
module and call its functions directly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
to complement https://pagure.io/koji/pull-request/162 we need to adjust pungi
rpm-ostree uses bublewrap that does not work in mock. --new-chroot to mock
enables the use of systemd-nspawn instead of chroot resulting in working
rpm-ostree again
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
Some modules can be executed as a sort-of test. However, the files do
not have executable bit set, so there is no need for them to have
shebangs. If someone wants to call them directly, they should do so via
python.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The address is no longer correct. We can just as well simply point to
the web page describing the license.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a variant has optional, it is possible to explictly list it in the
XML file and add extra groups. However, the original DTD did not allow
multiple variants with `id="optional"` as `id` attribute has to be
globally unique. This patch changes its type to `CDATA`, which has less
restrictions. This also means we can no longer define `<ref>` as
`IDREF` and instead check the existence of referenced variant in parser.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.
Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Introduces a new metadata file to track arbitrary files added during the
extra-files phase. This file is placed in the root of each tree and is
called ``extra_files.json``. It is a JSON file containing a single
object, which contains a "header" key with an object describing the
metadata, and a "data" key, which is an array of objects, where each
object represents a file. Each object contains the "file", "checksums",
and "size" keys. "file" is the relative path from the tree root to the
extra file. "checksums" is an object containing one or more checksums,
where the key is the digest type and the value of that key is the hex
digest. Finally, the size is the size of the file in bytes.
For example:
{
"header": {"version": "1.0},
"data": [
{
"file": "GPL",
"checksums": {
"sha256": "8177f97513213526df2cf6184d8ff986c675afb514d4e68a404010521b880643"
},
"size": 18092
},
{
"file": "release-notes/notes.html",
"checksums": {
"sha256": "82b1ba8db522aadf101dca6404235fba179e559b95ea24ff39ee1e5d9a53bdcb"
},
"size": 1120
}
]
}
Signed-off-by: Jeremy Cline <jeremy@jcline.org>
Fixes: #295
This will avoid confusion when the file can not be found, but previous
export from remote location worked well. In such case the log said:
Exporting from SCM...
Boom, no files found...
This commit changes it into:
Exporting from SCM...
Exporting file from current working dir...
Boom, no files found...
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It will take RPM repo from a variant in this compose and a Dockerfile
from configured git and use it to build an image.
The build images are uploaded to some a Docker registry by OSBS and are
not directly part of compose (because there is no export function).
There is a new metadata file `osbs.json` that has some information that
can be used to find the image.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is for layered products that have a variant mapped
to multiple variants in a base product, for example:
* Foo-Tools (id: FooTools, uid: Foo-Tools, name: Tools)
* Bar-Tools (id: BarTools, uid: Bar-Tools, name: Tools)
Requires productmd >= 1.2.
Signed-off-by: Daniel Mach <dmach@redhat.com>
On Python 2.6, it requires the format placeholder to have explicit index
of argument, so using % formatting is easier.
There are a couple places where the method is still used because the
same argument is used twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
rhbz#1331317 when we refactored how we make dvds in
df400002d8 we lost the ability to boot
the dvd as a disk image.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
It should have raised NotImplementedError, but we better remove it
altogher since we are missing export_dir anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is a sure way to have it removed, without having to duplicate the
logic in all places where it is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The logic for expanding the list of patterns into a list of actual RPM
paths was duplicated for both exporting a directory and file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of spawning `cp x/* y` there is now Python code to the same
thing. This should help with debugging if something fails as the
traceback will be more informative (rather than saying a command
failed). As another benefit the tests get much simpler.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple threads call `get_profile_module` at the same time, some
of them may get an exception from koji. This should prevent the problem.
Fixes: #253
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of looking for the DTD at module load time do it only when
required. Use a logger (if supplied) to print the warning.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There already were config options tree_arches and tree_variants, but the
filtering was done at the compose level and there was not interaction
between the two options. This led to problems when a variant would have
all its arches filtered out.
This patch moves the filtering to the variant loader. It adds better
logging, so whenever a variant is filtered (for any reason), it will be
explicitly stated in the logs.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This phase runs lorax with extra templates in Koji runroot task, links
the boot.iso to proper location in compose directory and adds the
installer iso to image manifest. This phase runs concurrently with live
media etc.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Simplify the code a bit by removing conditions that don't help in any
way: executing a loop zero times is the same as not executing it at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch Pungi should be more tolerant of network failures when
running a blocking command (creating live media or live images).
If the connection drops and the output indicates network problems, Pungi
will try to watch the task with `koji watch-task`. This will be retried
until it finishes (successfully or with some other failure). There is an
increasing timeout after each retry. Currently the maximum number of
retries is not limited.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.
This only makes sense for a variant that will have some other
deliverables like live media or images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch updates tests so that they pass, and fixes a couple places in
image build that did not work correctly with the nested configuration.
The documentation is reformatted so that it horizontal scrolling is less
likely.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If Koji fails runroot task for some reason, the output will most likely
not have the required format and will crash Pungi.
Pagure: #140
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds support for live media creator in Koji. The intended
workflow is to create a command , run it and finally collect built
artifacts.
get_live_media_cmd()
run_blocking_cmd()
get_image_paths()
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The methods mentioning image build are generic and can work for other
task types.
get_image_build_paths -> get_image_paths
run_create_image_cmd -> run_blocking_cmd
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When pungi crashes, the logs will often contain useless description of
a variant. With this patch, the traceback will have details on the
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given a list of arches, koji can build multiple images in one go
(automatically starting children tasks for each one).
This causes a bunch of changes:
* The configuration no longer allows changing config based on
architecture, only variants are allowed. It is however possible to
filter which arches are used for the building in the variant.
* The configuration files for koji image-build are stored in
work/image-build/$variant (not split based on arch).
This patch also changes the option name that is passed to koji
image-build: the repos should be specified under key `repo` (without the
trailing slash).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The lorax wrapper class now understands the --installpkgs argument and
can use it to pass multiple package names to the command.
There is a simple test to make sure the commands includes all specified
options. A bug is fixed where the bug URL would not be correctly
included.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch updates the variants DTD and parsing to allow specifying
multiple build install packages for each variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of hardcoding /usr/bin/python in shebangs, use /usr/bin/env.
This allows Pungi to work with dependencies installed in virtualenv.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These phases were computing checksums for images:
* buildinstall
* createiso
* image_build
* live_images
* product_img
In each phase the checksummed thing would ultimately end-up in image
manifest.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Add add_template, add_template_var, add_arch_template, and
add_arch_template_var to the call for lorax.get_lorax_cmd for the Atomic
ISO composes.
Remove duplicate if statement for add_template_var in lorax wrapper
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
Rename //variant/product to //variant/release in variants XML.
CHANGE: pkgset_koji_url and pkgset_koji_path_prefix config options replaced with koji_profile.
ACTION: Add 'koji_profile = "<profile_name>"' (use "koji" for Fedora) to config files.
You can safely remove and pkgset_koji_url and pkgset_koji_path_prefix from config files.
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
Set 'createrepo_c = False' if you need legacy createrepo.
CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
rename the pungi binary to pungi-koji since it does is tasks in koji
rename pungi-gather to pungi as it is the standalone old pungi binary
there is scripts that expect pungi to be the old pungi, the new binary
is not yet in use, pungi-koji semes to make sense, open to better ideas
Lubomír Sedlář
Jan Kaluza
Ondrej Nosek
Ralph Bean
Owen W. Taylor
Qixiang Wan
Daniel Mach
Dennis Gilmore
Patrick Uiterwijk
Colin Walters
Jeremy Cline
Dennis Gilmore
Peter Robinson
Lubomír Sedlář
Tomas Mlcoch
Lubos Kocman
Adam Miller