And include variant in repo file name. The whole path is unique already,
but not the filename itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can specify !VERSION_FROM_VERSION in version field during
image_build to expand it to correct release number without any label
information.
Also implemented !RELEASE_FROM_DATE_RESPIN to provide correct
release number. This helps to keep Atomic Host media files name
produced by image_build during bodhi updates compose run
consistent with nightly run.
Fixes: https://pagure.io/pungi/issue/987
Merges: https://pagure.io/pungi/pull-request/995
Signed-off-by: Sinny Kumari <sinny@redhat.com>
If a module says to filter a package out, we can do it immediately when
getting the build information from Koji.
This avoids a possible problem of something pulling the module package
in as a dependency, but it should also make the package set slightly
smaller.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a package name contains leading or trailing whitespace, it will
eventually lead to issues: pungi will try to include that group, but
since it does not exist, the packages will not make it in.
The root cause is hard to find. Better report an error immediately.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If a file has multiple hard links, genisoimage will put the wrong number
on the ISO. This patch can work around it by copying hard-linked files
into a temporary staging directory.
JIRA: COMPOSE-2610
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
PDC is deprecated in upstream. The usecase for getting list of modules
by NS, NSV or NSVC can however be satisfied by querying modules imported
into Koji.
This makes it possible to deprecate PDC configuration.
Merges: https://pagure.io/pungi/pull-request/985
Signed-off-by: Martin Curlej <mcurlej@redhat.com>
If there is leading or trailing whitespace in a comps group name, it
will not be included in the compose and there will even be no error
message. Whitespace on module name results in a failure.
To avoid these errors, validating the variants file will now also check
that there is no whitespace in significant places, and abort the compose
if there a problem.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
After cloning the repository with defaults, open each file and try to
check if there is more than one definition for the same module. It's not
a problem for the compose process, but consumers of the compose would
get confused and possibly explode. Better alert people early.
Conceptually this should be part of the test phase, but that would mean
waiting for the compose to finish before reporting the error. The
earlier the better.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the compose is not using comps, we can't pass the comps to lorax,
since it doesn't exist and it would cause a crash.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When multiple composes are chained, they should reuse the same event.
However it is tricky as the value would have to be passed by hand. This
patch makes it possible to read the value from another compose (the
first one in the chain).
JIRA: COMPOSE-2571
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The commands in runroot run as root every time. If they create files
that are not readable to other users, the reset of compose could have
problems with it if it does not run as root too. Particularly updates
composes in Bodhi run under apache user.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All the components are there already separately, but having the full NVR
should simplify searching the metadata with grep.
JIRA: COMPOSE-2519
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is an optimization for Yum. DNF does not care at all.
The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.
Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It should not be needed there, since the repo is empty anyway. A test is
added for the variant specific comps repo.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This is used for mapping basearch to binary arch.
One use for this is when running depsolving to tell yum/dnf what arch to
work with. With this change it will get i686 instead of athlon when
working on i386 basearch.
The other use case is finding arches compatible with given basearch. The
change will remove athlon from the list for i386.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should give us better error reporting. The `copy_all` function
should preserve permissions on all files.
Relates: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It checks config file value 'skip_phases' for valid phases names.
Also checks command-line attribute 'skip-phase' of 'bin/pungi-koji'.
JIRA: COMPOSE-2493
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
This was added back when rpmutils module was used. Since we migrated to
Python 3 we have included the code directly in Pungi (because it was
coming from yum which is not going to Py3 any time soon). The mocks can
be dropped.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There is really no need to write out megabytes of logs that are not
really interesting. This should also help the parallelization. With the
verbose log createrepo fills the output buffer and needs to wait for the
busy python program to read it first.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The files created in koji runroot will be owned by root. If the compose
is done under different user, there could be a problem with copying the
files preserving the owner. Let's just copy them without that.
Fixes: https://pagure.io/pungi/issue/932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the ISO is meant to be bootable but lorax fails, there's no point in
creating the ISO as it will not behave as expected.
Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1574585
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the parent task is successful, there can still be failed child tasks
for failable arches. We need to log those and potentially mark the
compose as incomplete.
Fixes: https://pagure.io/pungi/issue/874
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the phase is skipped, it could mean that we are doing a debug run and
we don't want to mess up the .treefile by missing arch specific images.
The other alternative is that the phase was really skipped, in which
case there will be no files generated and we already handle that fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps could potentially be different in different variants, so
instead we can create the comps repo for every variant separately and
use two repos instead of one (packages in one repository, comps in
another one).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the compose configuration includes the module_defaults_dir (an
scm_dict), clone the directory, read the module defaults contained
therein and include relevant defaults in the combined modulemd file.
Only defaults for modules present in the variant are included.
This requires libmodulemd 1.2.0+.
Merges: https://pagure.io/pungi/pull-request/891
Signed-off-by: Petr Šabata <contyk@redhat.com>
Even for Everything we want to filter the comps file to make sure we
remove the stuff that is not compatible with current arch. All groups
are still preserved in that case.
This allows us to do the filtering once in init phase than just use the
prepared file in comps source.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
All use cases that are actually used by pungi-koji are tested. There is
missing coverage for
* keeping only items with matching arch
* not reindenting the file
These aren't currently used and should be removed in the future, but
there may be other tools depending on the comps_filter executable.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't rely on the UID to be correctly joined with colons. There may
be historical data that still uses dashes.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
A new configuration option is added that allows users to point lorax to
extra repositories. This can be handy if some tools to create the
bootable image are not part of the product itself.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
JIRA: COMPOSE-2253
Otherwise it tries to ensure it exists, and since the default is
/etc/dnf/modules.d, it's causing problems if the directory does not
exist and user does not have permissions to create it.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should indicate that it's a modular variant, but there is no
modular content yet. We don't want to treat that as Everything.
The end result will be an empty repository.
Fixes: https://pagure.io/pungi/issue/871
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to include all relevant arches, not just the base one (including
noarch and src). However the list can be shortened by only listing
NEVRs, because that should be unique.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
For the first pass we don't need to filter out exclusive architectures,
and we don't need to exclude source packages without any binary
packages. We just want to merge the two package sets as fast as
possible.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Basically everything not on the list is excluded. This has to be applied
before we filter only the latest versions (otherwise we could lose
packages that are on the whitelist).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If we have a package set for the variant (which happens if there are
modules), include a list of all NEVRAs in the pungi kickstart.
This can be used to make sure only packages from correct tag get into
the compose. If two packages with same name but different version get
into the compose, this can help get even older version into a particular
variant.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The comps source should not return all groups when there are only
modules defined. This fixes part of the problem: non-modular packages
will not go in by default.
The second part is the comps file in the created repository. It will be
filtered to not contain any groups (because packages from there will not
be in the repo).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This already works on YUM backend, and this patch makes it work for DNF
as well.
Both native and multilib debuginfo and debugsource packages will be
excluded. This matches the yum behavior.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
OstreeInstaller phase will be moved to a different timeslot
and therefore needs different repo not to depend on Gather
phase which runs at the same time.
Related: https://pagure.io/pungi/issue/778
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
The previous attempt - caed78e - is not really correct. It sorts
the dict item tuples according to the alphabetical sort order of
the first item of each tuple (reversed). This will always work
when both substitutions *start* with the same characters, as in
the case of two strings that start with the same characters but
have a different length, the shorter one sorts alphabetically
first, and we reverse that. But it is not safe if the shorter
substitution doesn't start with the same characters, as in the
case I put in the tests: we should sort 'zzzaaaaaazzz' before
'aaaaaa' (and hence apply the 'zzzaaaaaazzz' substitution to a
volume ID that contains that string and not the 'aaaaaa' one),
but the previous commit did not.
Signed-off-by: Adam Williamson <awilliam@redhat.com>
Use 'get_packages_to_gather' to fail early if these packages are not
signed with right key. This prevents us from having to wait for the
repo to be created and depsolving to finish. Unsigned dependencies will
still be reported later than previously.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
There can be packages in the tag that will not end up in the compose.
Instead of failing immediately with error, this patch delays the check
until after depsolving finishes and only checks packages that will
really be included.
This is not an issue for nodeps compose, as that already pulls in only
packages that will be composed and nothing else.
Merges: https://pagure.io/pungi/pull-request/843
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.
The method used for gathering can be configured for each variant and
gather source separately.
Additional packages are only added to the comps source.
Each gathering step is logged separately. All the logs are preserved for
later inspection.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There are valid use cases for not specifying this option: specifically a
modular compose will get the tags to use from modules listed in the
variants file.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When configured to use createrepo_c, we should also use modifyrepo_c.
That allows us to relax the check for createrepo package.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Additionally ostree_ref (if parameter is given) should be placed into treefile.
Relates: https://pagure.io/pungi/issue/777
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
A new `buildinstall_topdir` option allows using buildinstall even when
the compose is created on a different volume that Koji is using.
The files are created in this external directory and then copies into
the usual location.
Merges: https://pagure.io/pungi/pull-request/807
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Options that are currently marked as deprecated do not have any effect
anymore (other than printing warning). We should remove them and update
the message so that we can mark options as deprecated even when they
still work.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
We can use a key function instead of relying to the deprecated cmp. This
makes the code work on Python 2.6 and on recent versions it makes it
faster.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would make sure that e.g. "updates" composes don't try to use "updates-testing" as an
old_compose_path, which would create practically useless deltarpms and for no repodata
reuse at all.
Signed-off-by: Patrick Uiterwijk <patrick@puiterwijk.org>
The file extension in configuration is only used to tell Pungi which
files from the task results should be downloaded. The user has to get it
right or the phase will fail. Each format has a single valid suffix.
Pungi should not require users to specify the suffix, since it can just
as well just know the right value.
The old configuration will continue working, only the extension will be
ignored.
Fixes: https://pagure.io/pungi/issue/753
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When the config for image-build command contains multiple values, they
should be joined with commas into a single value.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The input is given as unicode value, not a bytestring. We need universal
newlines to handle the conversion as needed.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
RHEL has an older version of the library which does not backport all the
assertions that we used. In order for the tests to pass there we need to
use names that exist everywhere.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On EPEL 6 modulemd is not available, so the test fails with an import
error. We should just skip this particular tests, as the rest of the
functionality is still perfectly fine.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
On Python 3, configparser will reject non string values, and
theoretically we could have some in the configuration.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This should make all tests pass on both Python 2 and Python 3.
Unittest2 is required on Py 2.6 and Py 3.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
This would be useful for modularity. The templates can be added now and
variables set via the existing `lorax_option`.
It's not possible to use custom templates not shipped with lorax, as
passing the path to a random directory is a little bit more tricky.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It only affects gathering packages when gather_source is set to comps,
but it could still make sense to include the file in the repository even
if the not used for gathering.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of the old default value and custom handling introduce new
option `live_images_target` (for consistency with other phases) and use
the usual inheritance rules.
Fixes: https://pagure.io/pungi/issue/749
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When saving new metadata fails, the images.json file would be left
empty. Instead we should not touch it at all.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We need to negate the value: the values are in seconds west of UTC, but
ISO 8601 wants the offset to be negative for times behind UTC (i.e. to
the west).
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Displaying the offset in seconds makes very little sense. We should
adhere to ISO 8601 format of `+HH:MM` which is much easier to
understand.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
If the compose failed, it may not have repos to compute deltas against,
and even if it has them, they were never shipped so no one will have the
older version of the package. We should instead go deeper in history and
pick a successful compose.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.
Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.
Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Pungi write image config file with name of <format>-<name>.cfg, if there
are two or more image configs present for different arches under the same
variant and with same format & name, the config file can be overwritten,
and result in invalid image conf file.
Example:
image_build = {
'^Server$': [
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['x86_64'],
'repo': ["http://example.com/linux/fedora/26/Everything/x86_64/os", ]
}
},
{
'image-build': {
'format': [('qcow2', 'qcow2'),],
'name': 'fedora-guest-image',
'target': 'guest-fedora-26-image',
'version': '26',
'ksurl': "git://git.example.com/ks.git?fedora#HEAD",
'kickstart': "fedora-26-kvm.ks",
'ksversion': 'f26',
'distro': 'fedora-26',
'disk-size': '10',
'arches': ['ppc64le'],
}
},
],
}
In this case, config file "qcow2_guest-fedora-26-image.cfg" will be
created for both x86_64 and ppc64le under the same variant dir, and
there is a high chance it will be over-written while Pungi creating the
koji task. We can add arch name(s) in config filename to avoid that.
Signed-off-by: Qixiang Wan <qwan@redhat.com>
In many cases we need to open files as binary to avoid errors on Py3
about writing binary data to file opened in text mode.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Even a failed lorax task can leave behind some in-progress images. Pungi
needs to copy the files into compose/ subdirectory only if the task
finished successfully.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1485021
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It's possible a variant is excluded via tree_variants option and the
section does not match anything. It can be confusing to users why
nothing is happening. This patch lets Pungi log all unmatched patterns.
Fixes: https://pagure.io/pungi/issue/692
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
We can't reliably tell user what system packages are missing as the name
might be different on different systems. Addiotionally there's no reason
why not rely on the packaging to be correct.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
It makes no sense to repeat the same configuration for multiple
architectures. Instead we should just list the architectures as another
key in the mapping. There is an option to specify multiple config dicts.
This preserves full backwards compatibility, the old config format is
still accepted.
Fixes: https://pagure.io/pungi/issue/678
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When a file should be obtained from a git repository, allow running an
arbitrary command (like `make`) after clone but before copying the files
out. This only works for the Git backend.
The downside is that a clone is needed and we can no longer use `git
archive` to speed things up.
Fixes: https://pagure.io/pungi/issue/5
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The paths need to be absolute so that subprocesses started during the
compose with a modified cwd will still work.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
There's no reason for reading the whole log of depsolving into memory
just to split it into lines and process one line at a time.
We can just as well read it in chunks.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.
Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
With this patch it is possible to run the test compose script from the
top level directory. This makes it slightly easier to use.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Instead of adding images to metadata and then creating hardlinks in a
separate step, do it immediately while we still have accurate
information about what variants it should be linked to.
Fixes: https://pagure.io/pungi/issue/670
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The rpmUtils module is provided yum-utils package, which is only
available for Python 2. There is no replacement for the functionality in
DNF.
There is a proposal to add this functionality to rpm itself, but it's
not really moving forward very much:
https://bugzilla.redhat.com/show_bug.cgi?id=1072972
As a short term solution let's copy the needed parts of rpmUtils.arch
module directly to pungi code base.
Fixes: https://pagure.io/pungi/issue/533
Signed-off-by: Qixiang Wan <qwan@redhat.com>
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.
This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.
The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.
Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
When there are composes with two digit respin, the code would prefer 9
over 10 as latest. Respin needs to be treated as a number.
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.
Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
The message announcing new ostree commit contains hash of the commit,
the ref it's for, but there is no information about where the repo
actually is.
This patch adds `repo_path` key into the message with URL of the repo
and `local_repo_path` with path to the repo on local filesystem.
Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
