Commit Graph

2218 Commits

Author SHA1 Message Date
Haibo Lin
cf52665a8d Use createrepo_checksum option instead of hardcoded sha256
JIRA: COMPOSE-3392
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-19 17:53:59 +08:00
Lubomír Sedlář
2dfb1cd4c8 pkgset: Skip adding modulemd if there is none
When libmodulemd is available, but the compose contains no modules, we
don't want to add the modular metadata anywhere. This patch actually
avoids a crash as `ModuleIndex.dump_to_string()` will return `None` if
there is no data.

JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 09:02:49 +02:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář
eeec62756f gather: Remove unused argument
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
da78b99fc0 pkgset: Load modulemd only when needed
We can avoid parsing source modulemd information since we can get the
same information from the Koji build info.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
a31fe998d5 Remove unused import
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
a340663f00 doc: Clarify relationship to Koji
Fixes: https://pagure.io/pungi/issue/1143
JIRA: COMPOSE-3372
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-03 09:06:49 +02:00
Lubomír Sedlář
8d00f56117 gather: Correctly sort list with multiple data types
On Python 3 it is not possible to sort str and None or RpmWrapper.

First convert everything to strings and then sort it. The sorting is
really to simplify diffing the files, so exact order does not have to be
preserved.

Fixes: https://pagure.io/pungi/issue/1227
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:13:23 +02:00
Lubomír Sedlář
fc362c5347 git-changelog: Fix running on Python 3
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 15:05:25 +02:00
Lubomír Sedlář
46333b01ad 4.1.38 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 10:32:01 +02:00
Lubomír Sedlář
bf1b3e8421 Remove remaining mentions of runroot option
This also cleans up the runroot method detection code to not rely on the
now removed option.

JIRA: COMPOSE-2634
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-02 09:43:13 +02:00
Lubomír Sedlář
49d0ab797c pkgset: Include module metadata in the repos
If the package set repo contains any modular package, the module
metadata is added there as well.

This is needed to accomodate change in DNF that refuses to work with
repo with modular packages if the metadata is not there. This DNF change
can cause issues in buildinstall phase.

Related: https://bugzilla.redhat.com/show_bug.cgi?id=1623128

The hybrid solver is modified to not create a separate repo with the
module metadata anymore, since it will be available in the repo with
packages. This also allows us to drop code to look into lookaside repos.

We still need to iterate over local modules in order to find out what
platform should be used.

JIRA: COMPOSE-3621
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-01 09:09:17 +02:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Daris A Nevil
920eceaa5a Respect --nomacboot flag when calling isohybrid
Fixes: https://pagure.io/pungi/issue/1222
Merges: https://pagure.io/pungi/pull-request/1223
Signed-off-by: Daris A Nevil <dnevil@intrusion.com>
2019-06-26 13:21:22 +02:00
Lubomír Sedlář
32a6415e58 config: Keep known options defined on CLI
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 15:13:16 +02:00
Lubomír Sedlář
d063217d6f config-dump: Report better error for non-existing sources
JIRA: COMPOSE-3624
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 14:42:50 +02:00
Lubomír Sedlář
4e59c7595e config: Improve config validation for anyOf and oneOf
Instead of just saying that there is a problem, give all possible
reasons.

JIRA: COMPOSE-3636
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 12:50:07 +02:00
Lubomír Sedlář
acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Lubomír Sedlář
32624c59b1 config: Report validation warning if variants fail to load
In a real compose this would be a blocker issue and the compose would be
aborted, but for validation it may make sense to continue. Instead of
crashing, let's report a clear warning.

JIRA: COMPOSE-3606
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 10:02:25 +02:00
Lubomír Sedlář
2f05a71c61 Allow customizing nosetests command
In some environments there may not be a version of the command without a
version. In such case it's quite convenient to be able to specify what
is the actual name of the nosetests command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-20 13:26:53 +02:00
Lubomír Sedlář
e6a26571e0 scm: Close stdin of processing command
It is possible the user set a command to run in cloned Git repository,
but that command can ask for additional input. However Pungi will
capture all output, so if there is a prompt, it will never be shown.

In order to prevent confusion ("Did this hang?"), let's send empty
string to stdin of the program. That will cause any possible read to see
EOF immediately, which should cause an error that will then be reported
by Pungi to the user.

It is still possible the program will wait for input if it reads
directly from TTY. However in such case the prompt should hopefully also
be sent to TTY directly, so that possible confusion should be cleared.

JIRA: COMPOSE-3598
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-20 09:07:07 +02:00
Lubomír Sedlář
82580ed5b3 pkgset: Create arch repos in parallel
This patch reuses the existing createrepo_num_threads options to limit
maximum number of parallel createrepo processes.

Fixes: https://pagure.io/pungi/issue/955
JIRA: COMPOSE-2575
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 15:42:46 +02:00
Lubomír Sedlář
f1263eeacb util: Resolve HEAD in repos that have a remote
JIRA: COMPOSE-3597
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-19 09:04:01 +02:00
Lubomír Sedlář
d01084337e tests: Avoid using threads in tests
Starting tests just to run mock functions slows the tests down for no
good reason. Let's instead mock the runner and run the dummy tasks
serially.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 13:17:17 +02:00
Lubomír Sedlář
2c3e6a5a74 pkgset: Use highest pickle protocol
Higher protocols should be more efficient in terms of performance and
storage size. Since we don't really care about interoperability with
different python version, we can safely go to the highest version.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-18 09:33:11 +02:00
Haibo Lin
6ec206f9ae gather: fix crash issue when gather_method = "nodeps"
JIRA: COMPOSE-3089
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-14 17:50:19 +08:00
Lubomír Sedlář
e68e17d8fe pkgset: Check for unused module patterns across all tags
Since there can be multiple tags, the check must be done once for all of
them at the same time. Otherwise any module found only in some and not
all tags would raise this error.

The code builds a set of all existing patterns and then removes items
from it. If there is something left once all tags are processed, it
means such patterns were not matched by anything.

JIRA: COMPOSE-3609
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-13 10:10:30 +02:00
Lubomír Sedlář
f784bbd519 util: Fix offline resolving for scm dict
For scm dict resolving the return value should be git ref (or source
branch for offline mode).

JIRA: COMPOSE-3614
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-12 08:22:38 +02:00
Lubomír Sedlář
2a9490526a pkgset: Make serialization more resilient
If the package set is loaded from a pickle, it will not contain this
key, and thus it can not be again written down into a new pickle. While
this is not a common use case, it's still better not to crash in that
case.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-11 08:45:16 +02:00
Lubomír Sedlář
0207260b9f fus: Support HTTP repos
Recent version of fus gained ability to download repodata if repo is
provided as HTTP url.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-07 14:43:04 +02:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
4e5f74b78d pkgset: Set correct nsvc for devel modules
We need to update the source modulemd to match what we want. For most
modules the name will be correct already, but in case of devel modules
the original name will still be present there.

Fixes: COMPOSE-3596
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 16:02:19 +02:00
Lubomír Sedlář
f6162b90dd Remove unused variable
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 15:57:55 +02:00
Lubomír Sedlář
c55ed742cd Whitespace fixes
This patch fixes some issues with inconsistent use of whitespace. It
only modifies lines that do not contain any code to not break git blame
too much.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:31:01 +02:00
Lubomír Sedlář
68115f3502 Whitespace cleanup
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:14:26 +02:00
Lubomír Sedlář
fa6197246b Remove unused variables
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:14:15 +02:00
Lubomír Sedlář
0a90f18b1f Remove unused imports
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-31 09:05:50 +02:00
Lubomír Sedlář
163d69713d 4.1.37 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-27 10:40:38 +02:00
Lubomír Sedlář
217fcd6c02 config-dump: Allow dumping config for multi compose
Clone the main config, all referenced parts and included variants or
comps files.

This requires a new argument: destination directory to which to write
the copied files.

JIRA: COMPOSE-3289
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-27 08:42:42 +02:00
Lubomír Sedlář
33471c38bb pkgset: Ignore modules without metadata in Koji
This is fairly similar to a package only being built for particular
arches.

Fixes: https://pagure.io/pungi/issue/1198
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-23 13:51:15 +02:00
Lubomír Sedlář
551f52922f runroot: Remove useless argument output_path
The same logic is handled by `chown_paths`, which does the same thing
but supports multiple paths.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
1951b0a521 buildinstall: Change owner of lorax logs
It's created in runroot, the owner and permissions should be updated so
that anyone can read it and user running compose can delete the files.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
f2bbf35429 kojiwrapper: Allow changing mode of multiple files
The directory with logs should have updated owner and permission as
well as the actual output. This patch lays foundation for that by
allowing multiple paths to be specified.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
Lubomír Sedlář
8acd2c9689 buildinstall: Create toplevel directory on compose host
When using lorax, this directory will contain subdirectories for
individual variants. Those are created in runroot and their ownership
and permissions are fixed there. However the top level dir was only
created and not updated.

JIRA: COMPOSE-3545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-22 15:10:32 +02:00
fanjinke
a74470c18d arch_utils: add Hygon Dhyana CPU support
Add Hygon CPU vendor ID("HygonGenuine") detection and optimize arch to
"athlon" in x86 and "amd64" in x86_64.

Merges: https://pagure.io/pungi/pull-request/1196
Signed-off-by: fanjinke <fanjinke@hygon.cn>
2019-05-21 12:24:28 +02:00
Lubomír Sedlář
187ce8df79 gather: Introduce module source again
This reverts commit ac15f21135.

It is still needed if nodeps gather method is used. It simply returns
all packages listed in all modules.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1708661
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-21 08:23:18 +02:00
Lubomír Sedlář
fe723a2094 metadata: Include empty directories in metadata
For example when a variant does not have any debuginfo packages, the
metadata will contain path to a repository, but it will be missing the
package path despite the (empty) directory being present on the
filesystem.

We should really only skip missing directories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-05-20 13:26:27 +02:00