Commit Graph

2369 Commits

Author SHA1 Message Date
Lubomír Sedlář
4efdacd0a0 4.2.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-13 12:55:44 +01:00
Jan Kaluza
a209bda73c Create MaterializedPackageSets in threads to make pkgset faster.
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...

The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.

This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.

This saves around 40 seconds from pkgset phase for RHEL compose.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-13 12:51:02 +01:00
Lon Hohberger
9391ce3065 pungi-koji: Allow user to specify version component count in latest symlink
This change adds an option which allows users to specify
the number of version components (i.e. values between dots)

The default behavior is preserved, and is equivalent to using
'-1' as the value.

- Negative values remove items from the end.
- Positive values specify the count of of components
- Zero will remove the version field entirely from the symlink
- When nonzero, at least one version component will appear

Merges: https://pagure.io/pungi/pull-request/1361
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2020-03-13 10:50:08 +01:00
Haibo Lin
3543f8fb3e pkgset: Reuse pkgset repos
JIRA: COMPOSE-4158
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-13 17:29:53 +08:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Lubomír Sedlář
a32bf4046f Remove fnmatch from nodeps gather method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-12 09:47:00 +01:00
Lubomír Sedlář
fb7f7396be Remove fnmatch from pkg_is_debug
On Python 2.7 fnmatch is not thread-safe, and this can cause crashes in
nodeps gather source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-12 09:11:32 +01:00
Jan Kaluza
e70ad8aaa5 Run extra_files phase in parallel to other phases.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-12 07:58:57 +01:00
Jan Kaluza
afcb3e969b Add Compose.old_compose_path and use it when searching for files in old compose.
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.

In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-09 12:47:23 +01:00
Jan Kaluza
fc3b5063ca Do not skip symlinks when searching for old compose.
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:

- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`

The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.

This commit removes this check and therefore symlinks are allowed
when searching for old compose.

I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.

I also executed test compose and it worked as expected.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-06 09:03:10 +00:00
Lubomír Sedlář
887c68d05d Display black output in tests
This should make fixing it much easier since it will directly tell users
what it doesn't like.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-05 09:51:47 +01:00
Lubomír Sedlář
16d6e5b0dd Resend message while waiting for ostree ref
If the signing machinery crashes, we don't want to wait forever. This
way at least we can shout for help periodically.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1350
JIRA: COMPOSE-4166
2020-03-04 14:48:03 +01:00
Lubomír Sedlář
4734f9859a Silence all productimg related warnings
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-03 11:06:52 +01:00
Jan Kaluza
145c3adbef Move buildinstall results to final directories if using pungi-buildinstall Koji plugin.
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-03 10:35:25 +01:00
Lubomír Sedlář
3cd94a4aa5 Silence productimg warning in quiet mode
When running with `--quiet`, the warning should not be printed. It would
confuse tools that expect only compose path to be printed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-02 09:24:01 +01:00
Haibo Lin
af5ee7030d Check skip_phases in config file
JIRA: COMPOSE-4143
Fixes: https://pagure.io/pungi/issue/1344
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-19 11:46:15 +08:00
Haibo Lin
e4f878a498 Fix typos in comments
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-19 11:26:15 +08:00
Lubomír Sedlář
d23b576a60 Remove dependency on dict.sorted
This is no longer used, and we don't need to pull it in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-02-17 12:45:49 +01:00
Jan Kaluza
5808733270 The pungi-config-validate --dump-schema now respects --schema-override.
It is useful to actually check how the schema changed after applying
--schema-override. This commit changes --dump-schema in a way that
--schema-override is taken into account and dumped schema contains
the changes done using the --schema-override.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-13 07:27:09 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Haibo Lin
3a31d47c83 doc: add code convention requirements
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 17:00:39 +08:00
Haibo Lin
9b12be7300 Fix formatting issues
Code should be formatted via black.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:16:20 +08:00
Haibo Lin
56fea60595 Run flake8 and black via tox
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:45 +08:00
Haibo Lin
65aa8fde2f Fix other flake8 complaints
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:09 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
41a629969c Format code base with black
https://black.readthedocs.io/en/stable/

JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:35:47 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Jan Kaluza
ef33d00f5b Add --schema-override to pungi-config-validate script
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.

There is currently no way how to check that Pungi configuration fulfills
these extra requirements.

This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.

For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:

```
{
    "properties": {
        "pkgset_source": {
            "enum": ["koji"]
        }
    }
}
```

It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.

Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-05 10:03:30 +01:00
Lubomír Sedlář
6f23c7b8ba 4.2.0 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-02-03 08:26:02 +01:00
Lubomír Sedlář
46ea0743a9 iso: Clean up cache for guestmount
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-30 10:35:42 +01:00
Lubomír Sedlář
4c6396f491 Remove deprecated warn() call
Instead warning() should be called.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 15:34:00 +01:00
Lubomír Sedlář
116c617b86 arch_utils: Fix ResourceWarnings
Newer versions of Python report an error when a file is not closed.
Let's avoid it by using with statement.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 13:08:44 +00:00
Jan Kaluza
3cde5c3a87 Add support for new Pungi Buildinstall Koji plugin.
We would like to start generating the buildinstall phase using the safer
Koji Pungi Buildinstall plugin and stop the direct use of Runroot plugin.

The plugin so far exists only as PR for Koji:
https://pagure.io/koji/pull-request/1939

This commit adds support for this plugin when `lorax_use_koji_plugin`
is set to `True`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-23 16:09:15 +01:00
Haibo Lin
6eb6511aa6 Make --task-id mandatory in get_runroot_cmd
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-20 12:56:01 +08:00
Haibo Lin
bce57c2e66 Wait on runroot tasks with wait-task command
JIRA: COMPOSE-4027
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-19 17:56:31 +08:00
Haibo Lin
52f82ccc6e pkgset: Ignore deleted module builds
JIRA: COMPOSE-4058
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-14 16:23:55 +08:00
Lubomír Sedlář
e7a58ccd07 Add tests for new exclude options
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-13 10:54:11 +01:00
Haibo Lin
93ed40ad2d Remove invalid parameters from osbs config
JIRA: COMPOSE-4049
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-13 09:05:55 +00:00
Frédéric Pierret (fepitre)
c624aab945
pungi-gather: add options for excluding debug and source packages
Signed-off-by: Frédéric Pierret (fepitre) <frederic.pierret@qubes-os.org>
2020-01-11 22:13:51 +01:00
Haibo Lin
01ab1d2e24 Hide latest koji package via pkgset_koji_builds option
JIRA: COMPOSE-4010
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-09 07:33:46 +00:00
Lubomír Sedlář
aa6a213c8d Add a new option for customizing version in .treeinfo
There was an attempt to achieve this by customizing --version and
--release arguments for Lorax, but Pungi does not take the [general] and
[release] sections from its .treeinfo. Instead it was always using
release version.

The value from this new option will be put into .treeinfo and used for
--version and --release arguments in Lorax (unless explicitly defined in
lorax_options to another value).

JIRA: COMPOSE-4029
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:55:48 +01:00
Lubomír Sedlář
2c2462970d Kickstart on RHEL 7 does not have metalink attr
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-07 10:30:16 +01:00
Jan Kaluza
fcf1442f71 Support generating ISOs when using link_type="symlink".
When `link_type = "symlink"` is used, the packages are in fact symlinks
to /mnt/koji. When graft points file is generated, the paths in this graft
points file point to symlinks and therefore symlinks are copied into the
generated ISO file instead of real files.

In this commit, the code to generate the graft points file is changed
so it resolves the symlink to real file stored on /mnt/koji. To make
this code safer, it does such resolving only in case the symlink points
outside of `compose.paths.compose.topdir()`. Therefore you can still
generate ISO file with symlink pointing to file stored within the ISO
file itself, although this is not done currently afaik.

The main reason for this is to be able to generate ISO files even
without hardlinks (which would need read-write access on /mnt/koji)
and without copying all the packages from /mnt/koji to local storage.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 12:30:13 +00:00
Lubomír Sedlář
02ace28fe4 pkgset: Hardlink downloaded packages
When pungi-gather (or old pungi) download the packages from repos into
work/$arch/, they are linked to work/global/. This was using link_type
configuration option.

However if that is set to symlink, we get a relative symlink in
work/global/ which is later copied under the compose/ directory. Since
it's a relative symlink, it gets broken by this.

The fix is to hardlink the downloaded packages instead in the first
step. Since both the source and destination are in work/ directory, we
know it's the same volume and hardlinks should work. There is a fallback
to copy just to be sure, but it shouldn't ever be used.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-06 10:49:25 +01:00
Haibo Lin
3f111b559f Check dependency of --just-phase
JIRA: COMPOSE-4020
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-06 08:52:47 +00:00
Jan Kaluza
3750d6795f Update runroot_method documentation.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-06 07:47:38 +01:00
Haibo Lin
794d151bef Remove productimg phase
JIRA: COMPOSE-4004
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-01-03 11:42:38 +00:00
Lubomír Sedlář
38f6162b46 Fix unified isos with missing images.json
The metadata file can be missing if the compose contains no images. We
need to handle that by creating a new empty file to add unified images
to.

JIRA: COMPOSE-4048
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-03 09:20:16 +01:00
Jan Kaluza
817acdbbac Allow setting runroot_method per Pungi phase.
The `runroot_method` now accepts `dict` value with phase name as a key
and runroot method as a value. For backward compatibility, the `str`
value is still supported.

The new `global_runroot_method` option has been added which defines
the runroot method in case it is not set in `dict` in the `runroot_method`.

This commit allows running `createiso` phase locally while keeping the other
phases in Koji.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-01-02 14:10:20 +00:00