If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.
To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The task can take a while to finish, and it's possible for the
authenticated session to expire in the mean time. Watching the task will
work, because that happens by spawning `koji watch-task` as subprocess.
We can work around this by creating a fresh unauthenticated session for
getting the task results.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This fixes a case where extra packages are pulled in.
The scenario is:
* there's a lookaside repo which contains group G which has package P
* we want to pull group G into the compose, but our definition of G
does not contain P
* lookaside repo does not contain package P
* current package set has P
DNF depsolver will then merge the two definitions and try to get all the
packages. For most cases this is not a problem, since the package is in
the lookaside repo and will not be pulled into the compose. But in the
example above since P is not in lookaside, Pungi will put it into
current compose.
This is also ugly in the depsolving log says it includes package P
because it was in input. But checking comps file does not show it.
The result of this change is that some packages might disappear from
current composes. This can only happen if there is a lookaside which
defines groups with similar IDs, which should be very rare. In cases
where this would be a problem the fix is to explicitly add wanted
packages either to comps or to additional packages.
Fixes: https://pagure.io/pungi/issue/978
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
During the createrepo phase for Modular variants, this will now
interrogate the repodata from the "work" repositories for the set of
licenses in use by each of the RPMs in a module and add those to the
metadata to be written out into the final repodata location.
Merges: https://pagure.io/pungi/pull-request/968
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).
JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Modulemd.copy() method has been available since libmodulemd 1.1
and is much faster than dumping to a string and parsing it again.
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Merges: https://pagure.io/pungi/pull-request/965
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.
JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The `add_noarch` option of `get_valid_arches` is broken and doesn't
really do anything (noarch is always present in the result).
This causes packages that have ExclusiveArch including noarch to
actually not be excluded. They should be.
Changing this globally could have a very big impact. Therefore we can
hide it behind a configuration option so that it's opt-in.
JIRA: COMPOSE-2457
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is used for mapping basearch to binary arch.
One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.
The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not exist on Python 3. Converting the exception to string works
identically.
The validate methods on many phases are simplified by not calling the
parent (which does not do anything).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
This way some parts of the code will be reusable. This should have no
effects on the outcome, the tests still pass without any changes needed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For variants that contain all packages (Fedora's Everything) we don't
want to lose any environments.
Fixes: https://pagure.io/pungi/issue/940
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.
Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With one compose combining traditional and modular content there might
be different requirements for tag inheritance. This patch adds a new
option that controls whether builds in modular tags should be inherited.
It defaults to False, which is the right option for current MBS
behavior.
JIRA: COMPOSE-2148
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code checked with `assert` that there is only one module matching
given NSV. In actual package that would not do anything and we would
silently pick the first value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On stdout it gets mixed with listing of gathered packages. This has no
effect in a real pungi-koji run, where both streams are merged into a
single file which is then parsed, but in manual debugging runs it's a
little obnoxious.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This speeds up the compose quite a bit and there is no need for the
database anyway.
Merges: https://pagure.io/pungi/pull-request/922
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package is in lookaside, we don't really need a debuginfo nor
source for it. Definitely we should not pull them from the package set
even if there is some suitable package. Therefore we should include them
in the output but marked as lookaside.
Fixes: https://pagure.io/pungi/issue/915
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The log file used to contain a dump of the configuration, which could
potentially be very large. This patch removes data that is directly
visible in the input kickstart file. Instead it just displays the number
of items.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.
Only defaults for modules present in the variant are included.
This requires libmodulemd 1.2.0+.
Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.
This allows us to do the filtering once in init phase than just use the
prepared file in comps source.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a variant contains both modules and comps groups, we need to include
builds from the compose tag in the package whitelist. However only
packages that are not already provided by any module should be added.
JIRA: COMPOSE-2435
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for
* keeping only items with matching arch
* not reindenting the file
These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The code should not live directly in the executable, that makes it very
hard to test.
Other than the move there is no functional change.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no need to create the directory in work/, as it will get
deleted immediately. Let's move it to /tmp and use the context manager
to clean it up.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
When ref is not modified via pungi config, we read it from the treefile
and substitute in basearch.
When pungi is configured to replace it, it modifies the treefile and
then used the value from config to avoid parsing the file. This however
did not substitute the basearch value.
We can simply use one code path for getting the value. This will work
for both cases.
Fixes: https://pagure.io/pungi/issue/866
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When allowing empty list of modules, the check for variant tags got
broken, causing Everything to no longer have an associated list of
allowed packages.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Relates: https://pagure.io/pungi/issue/862
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We now have a way to select even older version of package (since the
newer one can be left out of the whitelist), so we can include multiple
versions of the same package into global package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to check tags for the variant, not for the whole compose. This
results in merge always being done even if there is a single tag. For
f29 tag in Fedora this takes about 2 hours for each variant.
Relates: https://pagure.io/pungi/issue/860
Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1551653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.
The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This already works on YUM backend, and this patch makes it work for DNF
as well.
Both native and multilib debuginfo and debugsource packages will be
excluded. This matches the yum behavior.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.
Related: https://pagure.io/pungi/issue/778
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
There is a valid use case for modules without any RPMs in them. This
patch makes it possible to include such modules in the repodata.
Merges: https://pagure.io/pungi/pull-request/856
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
The previous attempt - caed78e - is not really correct. It sorts
the dict item tuples according to the alphabetical sort order of
the first item of each tuple (reversed). This will always work
when both substitutions *start* with the same characters, as in
the case of two strings that start with the same characters but
have a different length, the shorter one sorts alphabetically
first, and we reverse that. But it is not safe if the shorter
substitution doesn't start with the same characters, as in the
case I put in the tests: we should sort 'zzzaaaaaazzz' before
'aaaaaa' (and hence apply the 'zzzaaaaaazzz' substitution to a
volume ID that contains that string and not the 'aaaaaa' one),
but the previous commit did not.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
There is no guarantee that it will print any text. We don't even need
the output, we just print it to error log if there is a problem.
Fixes: https://pagure.io/pungi/issue/847
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In one place, there was an explicit check if pkgset_koji_tag was set,
in another it was blindly referenced and assumed to be a list (with
accidental semi-success for a scalar.)
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.
This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.
Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For sources and debuginfo package the flag should be set based on the
repo we pull that particular RPM from, not for the corresponding binary
package.
There could be a case where they come from different repos, so this
would cause problems.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the same package is contained in the main repo for depsolving as
well as in a lookaside repo, there was a chance that we would pick a
package from main repo, but corresponding source from lookaside.
This would cause a crash when trying to link the packages into the
compose.
A solution is to try to get source rpm from the same repo as the binary
package. Only when it's not available there we get some other one.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Background story: if a compose is combining modular and traditional
compose, the configuration will contain multiple Koji tags to build
package set from (one tag for each module, plus at least one tag for the
traditional content). However some packages might be present in multiple
tags, and if the package set contains both, there's no way to control
which one will end up in the compose.
The solution for this is to give preference to the modular compose. If a
package with the same name exists in multiple tags, we only take the
first one we find. This relies on ordering of collected tags: modular
ones are always first, and traditional tags are at the end of the list.
If there are multiple modules that contain the same package, only one of
them will be used, which is not correct. Allegedly this should not
happen. In any case such use case does not work without this patch
either, so we're not losing anything.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Ideally, pungi would generate repository IDs like `fedora-updates`
or so, and we'd have versioning inside the rpm-md. But for now
let's do this to avoid invalidating rpm-ostree's change detection.
Closes: https://pagure.io/pungi/issue/811
Signed-off-by: Colin Walters <walters@verbum.org>
Since we drop these files in a separate workdir each time,
there's no need to datestamp them. Doing so is part of the
cause for invalidating's rpm-ostree input change hashing.
Issue: https://pagure.io/pungi/issue/811
Signed-off-by: Colin Walters <walters@verbum.org>
When buildinstall fails, there will be no lorax generated files copied
into the compose directory. However they may still be mentioned in the
.treefile in work/ subdirectory where lorax runs.
To avoid possible issues, we should use the lorax created .treeinfo only
if the run was successful.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We already filtered a list of packages compatible with the architecture.
There's no need to do that again (with bugs). Instead of using the
global package set we should just restrict the code to an arch package
set.
This should not break anything. If a package is included such that it's
in global but not arch package set, the compose would crash as linking
packages takes paths from the arch package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Use global koji tag for populating package set even for modular
composes. To preserve backwards compatibility value "not-used" is
ignored. This should however be fixed in the configuration to simply not
specify the option at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Followup from discussion in: https://pagure.io/pungi/issue/811
It's likely now that for Fedora Atomic Host we'll use this, to work
around other issues, after we fix the FAW change detection.
Signed-off-by: Colin Walters <walters@verbum.org>
This essentially forces everyone to install cvs even though they most
likely don't need it. There is no easy way to check what is actually
going to be needed, so let's just put the responsibility of decoding a
potential error message to users.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no place that would possibly call it, so we can drop the check
and function for generating cli.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When cherry picking packages from Koji tag, we need to make sure that
for every binary package we always have a corresponding source package.
Even if it does not go into the compose, we need it to get values for
Exclusive or Exclude Arch tags.
This means we need to process the binary packages first and only then
look at source ones. Instead of sorting a potentially very long list,
let's just iterate twice.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The list of RPMs in a module can contain multilib packages, but they are
never included because we only ask for compatible arches without
multilib. For x86_64 that only adds noarch, but not i686.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Additionally ostree_ref (if parameter is given) should be placed into treefile.
Relates: https://pagure.io/pungi/issue/777
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
It allows specify what ref we want this compose to commit to.
New parameter 'ostree_ref' overrides the default value from the treefile json.
Relates: https://pagure.io/pungi/issue/777
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.
The files are created in this external directory and then copies into
the usual location.
Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
If buildinstall is supposed to run, we can't pick only some packages
from the tag. The temporary repo is used as a source for lorax. There
could very well be packages not meant to be shipped in the compose that
are still essential for building the bootable images.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to use `repository`, not `os_tree` path. For any non-addon
variant they are the same, but for addons the original path does not
work as it points to the parent really.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
Without this there is almost no information on what's happening. This
patch should provide basic debugging capabilities.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can use a key function instead of relying to the deprecated cmp. This
makes the code work on Python 2.6 and on recent versions it makes it
faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.
Pungi should not require users to specify the suffix, since it can just
as well just know the right value.
The old configuration will continue working, only the extension will be
ignored.
Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config for image-build command contains multiple values, they
should be joined with commas into a single value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The input is given as unicode value, not a bytestring. We need universal
newlines to handle the conversion as needed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3, configparser will reject non string values, and
theoretically we could have some in the configuration.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Let's not duplicate the list. Productmd exports a list of valid values,
so Pungi should just pick and use that.
Closes: https://pagure.io/pungi/pull-request/773
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.
It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It only affects gathering packages when gather_source is set to comps,
but it could still make sense to include the file in the repository even
if the not used for gathering.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently `None` has to be included to allow using unsigned packages.
ODCS has trouble with including non-string value in the list though, so
we can treat empty string the same way (it's not a valid key ID anyway).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the package name matches, we should take the package either if
architecture matches or the package is noarch.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The path prefix for packages is identical for all architectures. There's
no reason to compute it multiple times and check the directory existence
repeatedly.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When splitting is not possible, we should not print a huge number as
free space on single media. This patch also adds more information so
that the numbers can actually be connected to variant and architecture.
The list of ignored files will only be printed now if not empty.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The name can contains spaces, which would potentially cause problems.
Also this is not consistent with how it works in other phases.
Relates: https://pagure.io/pungi-fedora/pull-request/354
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.
Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to negate the value: the values are in seconds west of UTC, but
ISO 8601 wants the offset to be negative for times behind UTC (i.e. to
the west).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Displaying the offset in seconds makes very little sense. We should
adhere to ISO 8601 format of `+HH:MM` which is much easier to
understand.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the variant is not empty, it buildinstall will run lorax on it and
create some files that would otherwise be overwritten by the ostree
installer.
The validation script is updated to load variants file as it needs a
list of variants to find out if the config is wrong.
Fixes: https://pagure.io/pungi/issue/695
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
`pungi-gather` (the tool that underlies both the `pkgset` and `gather`
phases) contains profiling code that will log statistics about how long
different function calls take. However, pungi-koji did not contain a
way to pass the ``--profiler`` argument to enable this.
This change adds a new configuration option ``gather_profiler`` which,
when set to true, simply passes the argument to `pungi-koji`. Hopefully
this can help shed some light on what is happening in some of our
longer-running composes.
Merges: https://pagure.io/pungi/pull-request/727
Signed-off-by: Ralph Bean <rbean@redhat.com>
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.
Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.
Example:
image_build = {
'^Server$': [
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['x86_64'],
'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
}
},
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['ppc64le'],
}
},
],
}
In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config can change the default compose type. This can still be
overwritten by a CLI argument. A `--production` option is now added to
CLI (because that was the default before).
Fixes: https://pagure.io/pungi/issue/694
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In once case we can completely drop try-except statements: it only
logs the exception details but that will be logged again anyway.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is updated in all files for consistency, even the modules that will
never be ported to Py 3 completely due to dependency on Yum.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a custom server CA certificate, it needs to be generally
available, and not just used when logging in so that SSL verification
works.
Signed-off-by: Owen W. Taylor <otaylor@fishsoup.net>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When using repos as gather_source, we should use DNF backend even for
constructing initial package set and to download the packages from
source repos. Without this the repos source would not be usable on
Python 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Due to overwriting an existing variable the logs are getting duplicated
line about missing comps packages instead of announcement of gathering
being finished. Rename the variable to fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is needed for correct trimming of addons and optional. A package
pulled in as a dependency but that matches something on fulltree exclude
list should have this flag. It will then be moved from addon to base
variant and therefore excluded from optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't reliably tell user what system packages are missing as the name
might be different on different systems. Addiotionally there's no reason
why not rely on the packaging to be correct.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.
This preserves full backwards compatibility, the old config format is
still accepted.
Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch adds a helper function for easy creation of a structure where
either X or list of Xs is accepted.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.
The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.
Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a package mentioned in comps is not available in the package set,
print a warning about this. Additionally there is a config option that
allows to turn this warning into a fatal error.
Fixes: https://pagure.io/pungi/issue/50
Fixes: https://pagure.io/pungi/issue/683
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of just printing the error directly to stderr, capture the
output and use proper logger. This makes sure the error is included in
the log file and also fixes `--quiet` option which was not properly
honored originally.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently, this will fail two lines later when we try to access
`pdc_module['modulemd']` with an unhelpful `TypeError` since
`pdc_module` is `None`.
Signed-off-by: Ralph Bean <rbean@redhat.com>
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.
There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972
As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.
Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
These packages should behave like regular debuginfo packages (at least
for now).
Fixes: https://pagure.io/pungi/issue/684
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a noarch subpackage, all compatible debuginfo would be
pulled in, which is not desirable.
Example: Server.x86_64 needs pkg.x86_64 and pkg-data.noarch. We only
want pkg-debuginfo.x86_64, but without this patch even
pkg-debuginfo.i686 would get in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are no groups, we shouldn't try to read comps file (because
it may very well not be there).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.
This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.
The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.
Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are composes with two digit respin, the code would prefer 9
over 10 as latest. Respin needs to be treated as a number.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.
Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.
This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When gather_method is set to nodeps, we should not ignore the comps
group that the method received. Instead it should find out which
packages are in those groups and take them into the compose.
In order for this to be of any reasonable use, the comps file needs to
include all dependencies for the packages.
Fixes: #653
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.
For example, if "ceph-3" was missing from the SCM:
OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'
This occurred even if product_id_allow_missing was set to True.
Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.
We now see the following warning in the logs instead:
[WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}
and the compose succeeds.
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.
This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.
Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Nothing is calling this file as an executable, so there is no reason to
have logic for parsing arguments.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Creating an OSTree repository is a notable event; the general
expectation is that rather than having lots of repositories,
one has branches inside a single repository.
For $reasons, Fedora is not currently doing this, but we will
change it to do so.
The reason I'm making this change is we discovered that
it looked like Fedora had somehow made a repo inside a repo,
presumably due to a configuration error.
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/GBFSOLULGGZFGEFCIW6FG23NZZV5VH4K/
Signed-off-by: Colin Walters <walters@verbum.org>
There can be multiple images listed for a single variant, the config
validation should not reject it.
The syntax with a single config object is still accepted. The price for
that is less descriptive error message when there are errors.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.
Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple repos are configured in pkgset_repos, the logs should
contain information on where exactly the package was pulled from. The
log file in question should be in
`logs/<arch>/pkgset_source.<arch>.log`.
Fixes: #545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.
Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.
Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there's a temporary network issue, Pungi will fail to turn a branch
into a commit hash. This would abort the whole compose. Instead we
should just retry a few times.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.
This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.
Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.
Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This option is currently only checked in the ostree phase, and it does
not make sense as a string. When any non-empty string was given, it
enabled the check.
Relates: #590
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A `requires` attribute is taken from a wrong package (because of a wrong
variable used: `pkg` vs. `package`). On RHEL 6 this actually leads to a
crash. Let's use only one name to avoid such problems.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.
In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
As long as the assertion is not violated, this would not be a problem as
the second argument is not evaluated. However if the condition was
broken, a NameError would be raised instead of AssertionError with a
nice message.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The get_system_release_packages function can never be called without a
variant, so it makes no sense to check for that condition.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* add comments explaining what's going on
* break too long lines
* simplify the logic where possible
* use with statement to work with files
* remove commented out and unused code
* introduce helpers to reduce code duplication
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple arguments that are never used. We can just remove
them. This also fixes a bug where temporary files would be left
undeleted in tests.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When dependencies are pulled in, it's useful to know not only the
package that requires them, but also the specific requires.
This patch only implement this for the YUM version.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of going in random order, we should do it deterministically and
process the dependencies in order.
This should actually fix a problem where `glibc-langpack` packages are
missing on some architectures for Fedora Rawhide.
Fixes: https://pagure.io/pungi-fedora/issue/214
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the koji profile we are using is configured to use keytab, we should
run koji executable with a fresh credentials cache. Otherwise we risk a
race condition as multiple processes will trample over the same
directory in /tmp/krbcc_0.
This is currently only implemented for calling `koji runroot`. We might
need to do it for other commands as well (currently there is a sleep to
avoid the race condition for other commands).
Fixes: https://pagure.io/releng/issue/6715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In order to avoid conflicting tags, OSBS allows only one build for a
repo/branch pair at the same time. To avoid race conditions, we should
make sure we always pass in the branch. This commit makes it a required
option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding a new config option, we can just reuse the existing
`media_checksums` value. If the value is good for image checksums, it
should work for extra files as well.
Relates: #591
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config contains a git url pointing to a non-existing branch,
pungi will fail to get commit hash from that branch and die with a
confusing error message.
Fixes: #583
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When we fail to generate a volume ID that fits in 32 characters, the
error message should include the options that were considered. It could
show that there might be a substitution that could fix the problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We have some hooks yield ConfigOptionWarning. When it happens within
anyOf validator, anyOf validator yield ValidationError and reports the
config as incorrect. We need to overwrite it to pass not break.
Fixes: #598
Merges: #599
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Instead of replacing yum.comps with an something from DNF, we can go
directly to libcomps. DNF does not have the equivalent functionality
(particularly it's impossible to load comps from file directly).
We would have depended on libcomps anyway transitively, so this is not a
big deal.
Fixes: #587
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We never need anything more than the ID, so passing around a big
complicated object is not necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a couple methods that are never used in the codebase. We don't
support using this module as a library from other programs either, so we
can just drop this.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The config now uses similar logic what previous commit did for OSTree.
Also we should report error when an unknown generator is used.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Running depsolving with no requested inputs will only lead to a hard to
decipher error. We should instead explicitly tell the user that there is
a problem.
Unit tests are added to add to test this functionality.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When variants XML lists a group that does not match any known group in
input comps, report a warning. This is not necessarily a problem in
itself, but having this information in the log can help debug problems.
Relates: #585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Config option 'repo' and 'repo_from' are used in several phases, merge
them with one option 'repo'. 'append' in schema is used for appending
the values from deprecated options to 'repo', so it won't break on any
existing config files that have the old options of 'repo_from' and
'source_repo_from' (which is an alias of 'repo_from').
And 'repo' schema is updated to support repo dict as the value or an
item in the values, a repo dict is just a dict contains repo options,
'baseurl' is required in the dict, like:
{"baseurl": "http://example.com/url/to/repo"}
or:
{"baseurl": "Serer"}
currently this is used in ostree phase to support extra repo options
like:
{"baseurl": "Server", "exclude": "systemd-container"}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
If 'append' is defined for a property, append the values from append
options to the property. Note: The property must support to be a list
of values.
For example:
with schema:
schema = {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"definitions": {
"list_of_strings": {
"type": "array",
"items": {"type": "string"},
},
"strings": {
"anyOf": [
{"type": "string"},
{"$ref": "#/definitions/list_of_strings"},
]
},
},
"properties": {
"release_name": {"type": "string"},
"repo": {"$ref": "#/definitions/strings", "append": "repo_from"}
},
"additionalProperties": False,
}
and config:
repo = "http://url/to/repo"
repo_from = "Server"
config will be updated to:
repo = ["http://url/to/repo", "Server"]
It supports multiple append options too, like:
"repo": {
"$ref": "#/definitions/strings",
"append": ["repo_from", "source_repo_from"],
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Show warning message for any alias option find in config instance.
Example warning message:
WARNING: Config option 'product_name' is deprecated and now an alias to
'release_name', please use 'release_name' instead. In:
{'release_name': 'dummy product', 'product_name': 'dummy product'}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Do not create empty skeleton dirs for empty variants which we do for rpm
variants in some phases (some others already have the check):
1. createiso phase
2. extra_files phase
3. gather phase
In addtion to this, compose metadata (composeinfo.json) doesn't include
variant paths which don't exist or are just empty dirs now.
Fixes: #497
Merges: #572
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Calling it gather_backend is similar to repoclosure_backend we already
have. It's also more obvious what it does.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We also rename the old multilib module used by dnf code to multilib_yum
to make it clear what is imported where.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This way we pay once for the expansion, and lookups are then done in
constant time instead of iterating over the list over and over again.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is no guarantee __del__ will ever be called, and we were leaving a
ton of stuff in /tmp. With this patch we pass the temporary directories
explictly and make sure they are deleted at the end.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
These requires are accessible from a separate attribute, but we want to
handle them the same ways as regular Requires.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We want to exclude all multilib packages matching the blacklist (that
are not noarch) from the package set.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This only works in non-greedy mode. When greedy, the same provides can
be linked to multiple packages that should be pulled in.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
* Remove dead imports
* Remove dead code
* Fix extra empty lines
* Fix formatting
* Break too long lines
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
the second most demanding filter
it is called 8297 times and gets 189s out of 15min run which is 21%
spent here
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
this is most demanding filter
it is called 10826 times and gets 210.277010s out of 15min run which is
23% spent here
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Split the function for getting implant MD5 into two functions, so that
we can have access to other information retrieved by checkisomd5
command. A test case is added for both functions.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch refactors logic for creating a temporary mount point,
mounting an image, running arbitrary code on it, unmounting the image
and removing the mount point. It immediately uses it in the buildinstall
phase.
Similar mounting is present in product_img phase as well, but due to
different usage pattern it's not changed yet.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Koji expects the value as comma separated list, otherwise it will ignore
it. This makes it possible for the image building to fail on per arch
basis.
Relates: #128
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Currently if there are multiple ostree installers for the same
architecture, the logfiles all end up in the same location and overwrite
each other.
This patch moves the logs to logs/<arch>/<variant>/ostree_installer-X/
for a unique value of X so that there can be multiple runs even for the
same tree.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When even after retries the unmounting still fails, this patch runs `ls
-lA`, `fuser -vm` and `lsof +D` on the directory to give some idea of
what's blocking it.
Relates: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the image can not be unmounted because the device is busy, we should
retry. There will be increasing pauses between the attempts. At most 10
attempts will be done before giving up.
Fixes: #559
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the checkisomd5 command exits successfully but returns a wrong value,
we should catch and log that. In theory this should be impossible, but
we have seen it in production.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If task does not produce the expected number of images, it should fail
at Koji level. Also, this check does not really work when some
architectures are allowed to fail.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
live_images: additional_repos -> repo
ostree: source_repo_from -> repo_from
extra_source_repos -> repo
ostree_installer: source_repo_from -> repo_from
With the change, the phases have consolidate option names for variant
repos and external repos.
Old option names will continue to work, old names will be converted
to new names after validation automatically if new options are not
specified in config.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
When a property has 'alias' defined, and it's not present in instance,
if the alias property is present, add the property with value from alias
property before remove the alias property from instance.
Examples:
with schema:
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Pungi Configuration",
"type": "object",
"properties": {
"release_name": {"type": "string", "alias": "product_name"},
},
"required": ["release_name"],
"additionalProperties": False,
}
1. config = {"release_name": "dummy product"}:
validate pass, config not changed after validation.
2. config = {"product_name": "dummy product"}:
validate pass, config updated to the following after validation:
config: {"release_name": "dummy product"}
3. config = {"name": "dummy product"}:
validate fail, errror message is "Failed validation in : 'release_name' is a required property",
and warning message is "WARNING: Unrecognized config option: name."
4. config = {"product_name": "dummy product", "release_name": "dummy product"}
validate fail, error message is "Failed validation in : product_name is an alias of release_name, only one can be used."
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The line of log printing can result in "No handlers could be found for
logger" warning in quiet mode when custom paths module is enabled,
because file handler is setup after that, since this line is only
printed to screen (when pungi-koji is not ran in quiet mode) and doesn't
contain important info, it doesn't hurt to remove it.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
This is a follow up of 8bc65a8be5. When
resolving packages that go into optional variant, we only want to skip
filtering groups when working on a top-level variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For different cases where runroot is used it's now possible to set
custom weight. The usecase for this is to avoid one builder taking too
many tasks. Especially buildinstall is quite resource intensive, so one
builder taking multiple tasks at the same time leads to very slow
compose time.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The optional variant can defined by just adding has_optional=True into
variant xml. In such case it has no comps groups and Pungi would copy
the original file unmodified. This leads to extra packages being pulled
into the optional variant.
In this case the correct solution is to filter the comps and remove all
groups.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of creating and configuring the logger at module import time, we
can only get the logger if it's actually needed and configure it from
the main script.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
In multiple situations we need to create temporary files or directories
that should not be preserved after compose is finished. Let's add
context managers that ensure these get cleaned up.
This fixes tests leaving garbage around in /tmp.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If there is an empty images.json file exists, re-running pungi-koji
in debug-mode will fail with exception. We can just ignore the exception
in such cases, pungi-koji will create a correct images.json later.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In order to do something meaningful, the class needs to be instantiated
with arguments pointing the blacklist and whitelist.
The `file` multilib method used via `pungi-koji` or `pungi` directly has
no way to pass those in.
The only way this class can be useful would be if someone actually
imported the class directly in their own code. Pungi is not meant to be
used as a library though, so this is not really a supported use case.
Not to mention that the `select` method always returned `False`.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We call pungi to get the packages from external repos if
pkgset_source=repos. In this case we need to look at additional packages
as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: #528
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.
Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are a lot of options that are not used by any code path in Pungi.
There is not reason to provide access to them.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of joining the arches as a comma separated string and splitting
it again later. Ultimately we do need the original format to pass to
koji wrapper, but we can produce that value later.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Write file sizes of images in checksum files with comment lines,
checksum files are in BSD-style which supports comments by starting
a line with '#'.
Example:
$ cat RHEL-7.4-20170123.n.4/compose/Server/x86_64/iso/RHEL-Server-7.4-x86_64-20170123.n.4-CHECKSUM
# RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso: 3725590528 bytes
# RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso: 377487360 bytes
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-dvd1.iso) = fa3de37fe4b859a0285f16ea1123f44f15aec169aea84bf010aa3821bd58fc41
SHA256 (RHEL-7.4-20170123.n.4-Server-x86_64-boot.iso) = 74bf68c54665328adb08b09daf773c67e633b5907e3e2797338ab3c1b58fdf48
(No space at the start of line, because git commit message drops lines
start with '#', added one space to avoid that.)
When there are multiple checksum types specified and checksums are
written to individual files, file size of the image will also be
written to every checksum files.
Fixes: #493
Signed-off-by: Qixiang Wan <qwan@redhat.com>
pungi-make-ostree has to run on the target arch so that rpm
scriptlets can be ran. as a reult we can ask rpm what the
basearch is for the running environment. For notifications
we have to pass in the arch we are running for.
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
If gpgkey option is defined in config, set gpgcheck=1 and set
gpgkey=<value> in variant repo files.
Fixes: #487
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add new key 'repo' to allow specifying multiple repos as the source
repositories. And change 'source_repo_from' to allow specifying multiple
vairant names to use variant repos as source repositories.
Doc of 'source_repo_from' is updated to not mention URL is supported,
though we still support that in code. User should add url of repos in
'repo' key instead of 'source_repo_from'.
Fixes: #508
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Add compose.mkdtemp which creates and returns a unique temporary
directory under <compose_topdir>/work/{global,<arch>}/tmp[-<variant>]/
Change tempfile.mkdtemp in code to compose.mkdtemp, so temporary
dirs are created under the compose's workdir, this makes it more
easier to maintain the temporary dirs/files especially when compose
fails with amount of temporary files left there.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Productmd does validation on the parsed value. There is a long list of
rules to populate `short` name in the productmd code.
There is a shim in pungi to make sure the short is populated if no rule
matches. After renaming product to release, the shim stopped working.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.
Known issues:
* The filename does not respect settings. This is tricky because the
name could include variant name, which we don't have here (by design
of unified ISO).
* The same is true for volume id.
In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This makes sure the test configurations will be accepted in real usage.
It also enables us to remove some manual error checking that will be
performed by validator.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The same way live_media and image_build accept additional external repos
or variants list, there is now a `repo` and `repo_from` configuration
key to add these.
Fixes: #486
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Using getattr so we also work with versions of koji that do not
read principal and keytab from the config file.
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
Also remove the TODO comment from live images phase: the appliances are
already submitted one task per single arch, so this change is not
necessary.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This uses the --can-fail option in koji. Failing an optional image will
not abort whole task. If the whole task fails (or there is a problem on
the compose side), we abort unless all arches are optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This patch uses the `--can-fail` option of koji command line. If only
optional arches fail, the task will report as success. Failures on
compose box side are ignored if and only if all architectures are
optional.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Buildinstall only runs for top-level variants. Addons, optionals and
integrated layered products must reuse install tree from their parent,
because otherwise there would be no boot.iso.
Fixes: #472
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Given the way config files can include other files, it is entirely
possible to use the same config file for both Pungi and Distill. Pungi
will however complain about unknown options.
This patch reverts part of c38bb480 and moves deprecation handling back
into schema validation. The validation method then returns a list of
errors and a list of warnings.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When running with this option, if a debuginfo package is included in the
final package set, there would be a crash due to missing SRPM. To
replicate this issue, just use the argument with input kickstart listing
a single package which has debuginfo.
This is not an issue for Pungi 4, as it never uses the option in the
first place.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Source packages can be excluded while binary packages should still go
into the compose. This patch makes it so that the mapping from binary
packages to source packages contains None in such case. The code is
already capable of handling that. A warning will be emitted for each
binary package without source.
This also allows us to remove some code from `createSourceHashes` that
is now unused.
A test for excluding source package is added as well.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The new sub-command 'installer' is added to support build OSTree
installer image with pungi-make-ostree. It can take an optional argument
'--extra-config' to read some of configurations from a json file. The
content of the json file can contains the configuration which are
supported in OSTree installer phase, the difference is variant UID is
not supported as a repo url in this case. A valid json file can be like
the following:
{
"source_repo_from": "http://www.example.com/repo/workstation/os",
"installpkgs": [
"fedora-productimg-workstation"
],
"add_template": [
"/path/to/installer/template/lorax-configure-repo.tmpl"
],
"add_template_var": [
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
],
"add_arch_template": [
"/path/to/installer/template/lorax-embed-repo.tmpl"
],
"add_arch_template_var": [
"ostree_repo=https://www.example.com/compose/ostree",
"ostree_osname=fedora-workstation",
"ostree_ref=fedora/25/x86_64/workstation"
]
}
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Update pungi-make-ostree to supourt sub-command 'tree', which is just
as the original feature of pungi-make-ostree to compose OSTree tree.
With the change we can add other sub commands later to build other
OSTree artifacts, like the installer image.
Inaddtional to the change, now the the 'tree' command can accept an
optional '--extra-config' parameter to update the original tree
configuration with extra configurations specified in a json file
before composing the OSTree tree.
Example:
pungi-make-ostree tree --repo=/ostree --treefile=/path/to/treefile \
--log-dir=/path/to/log --extra-config=/path/to/extra-config.json
The extra-config file can contains the same configuration as OSTree
phase, the difference is it doesn't understand variant UID as source
repo since it's not ran in the chain of phases. A valid configuration
can be like:
{
"source_repo_from": "http://example.com/repo/x86_64/Server",
"extra_source_repos": [
{
"name": "optional",
"baseurl": "http://example.com/repo/x86_64/optional",
"exclude": "systemd-container",
"gpgcheck": False
},
{
"name": "extra",
"baseurl": "http://example.com/repo/x86_64/extra",
}
],
"keep_original_sources": True
}
The OSTree phase is updated to move out the task of updating treefile,
instead of that, it writes the extra configurations to a json file,
then 'pungi-make-ostree tree' will take it by option '--extra-config'.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The internal flag in productmd is meant to indicate that a compose is
not meant for publishing. This is potentially useful to allow filtering
in PDC or similar service.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Phases createiso, liveimages, image_build, ostree_installer and osbs are
done in parallel, logs from these phases are mixed and and it's not
obvious which log message belongs to which phase. This change adds phase
name in log message for these phases.
The new mixin 'PhaseLoggerMixin' is added to extend a Pungi phase with a
logging logger which copy handlers from compose's logger but with
formatter changed.
Fixes: #58
Signed-off-by: Qixiang Wan <qwan@redhat.com>
The check for arch compatibility should not be performed if the
debuginfo package is noarch. Such packages should be included
unconditionally.
Fixes: #450
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the media is bootable, we can not split it. The limit is not used in
that case and we may overflow it with an warning message. The warning
should correctly mention what is going on instead of printing a
non-sensical message about free space on media being some huge number.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
