Commit Graph

2536 Commits

Author SHA1 Message Date
Lubomír Sedlář
9fced77140 scm: Workaround incorrect permissions on created directory
We have seen the directory created with wrong permissions. Since we
haven't been able to find out why it happens this is a workaround.

JIRA: RHELCMP-142
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-29 09:51:53 +02:00
Lubomír Sedlář
9a1b9dd154 Fix warning about productimg in skip_phases option
If phase is skipped in a config option (and not on command line), we
should just print a warning or do nothing in quiet mode.

JIRA: RHELCMP-399
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-28 07:51:40 +00:00
Haibo Lin
0525768519 Add pkgset_allow_reuse option
JIRA: RHELCMP-492
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-28 09:54:36 +08:00
Haibo Lin
153eb628e8 Delete outdated comments
JIRA: RHELCMP-395
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-22 17:14:51 +08:00
Lubomír Sedlář
f7944a406e gather: Fix nodeps method to not prefix match
If the input packages contain e.g. `ansible`, and there's
`ansible-runner-service` in the package set, we don't want to pull it
in.

JIRA: RHELCMP-446
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-21 09:58:00 +02:00
Lubomír Sedlář
1a5cd9e0bf linker: Remove check after copying file
After a file is copied, a check was done if size and mtime of the files
match.

This has very few benefits in what problems it can detect, and can cause
problems on NFS where it can take a short time before the updated
attributes are visible on the new location.

JIRA: RHELCMP-378
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-20 14:52:00 +02:00
Haibo Lin
30f4771db1 Stop creating iso_stage_dir before deleting
It's unreasonable to create the dir and then delete it immediately.

JIRA: RHELCMP-151
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-20 18:10:58 +08:00
Jan Kaluza
5395af416c Allow reusing old buildinstall phase results.
New `buildinstall.metadata` file is created once the buildinstall
phase is done. This file contains:

- list of lorax command line arguments.
- list of RPMs installed in the buildinstall buildroot.
- list of RPMs installed in the resulting boot.iso.

This file is checked in the next compose run to find out if
the result of buildinstall phase from the previous compose
can be reused. Following is checked:

- lorax commandline arguments are the same (except of expected
  differences).
- The NVRAs of RPMs in the runroot_tag are the same as the ones
  installed in the old buildinstall buildroot.
- The NVRAs of RPMs installed in the boot.iso are the same as
  the ones in package sets in the current compose.

By its implementation, this reuse strategy is used only if
pungi_buildinstall Koji plugin is used.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add tests for buildinstall reuse and buildinstall_allow_reuse option.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-17 11:37:13 +00:00
Dusty Mabe
3509d7a36c ostree: set umask to be more permissive for ostree operations
We need to set the umask to be more permissive so directories get group
write permissions. See https://pagure.io/releng/issue/8811#comment-629051.

Signed-off-by: Dusty Mabe <dusty@dustymabe.com>
Merges: https://pagure.io/pungi/pull-request/1373
2020-04-17 09:20:24 +02:00
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Lubomír Sedlář
e187b5ea79 Break too long line
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-04-06 14:20:47 +02:00
Buvanesh Kumar
4cf11906e8 RCM-79601: Increase time delay in race condition
Signed-off-by: Buvanesh Kumar <bsivasub@redhat.com>
2020-04-06 08:15:57 -04:00
Jan Kaluza
fdfaae8b71 Allow gather phase reuse on product_id change.
I've analysed multiple nigthly composes and often the only difference
in the configuration between two nightly composes is different
`product_id` commit hash.

The `product_id` is used in later `createrepo` phase and does not
influence the gather phase at all. I therefore think it can be
whitelisted in gather phase reuse code.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-04-06 08:23:19 +00:00
Haibo Lin
c5e59fa732 Gather more debug data for GitWrapper clone
GitWrapper._clone crashes occasionally but unfortunately
the root cause is not found yet. We need more info for debugging.

JIRA: COMPOSE-4219
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-03 17:08:51 +08:00
Haibo Lin
65251d983a Reuse arch pkgset repos
JIRA: COMPOSE-4217
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-31 13:32:12 +08:00
Haibo Lin
63ec1adc22 Get non-rpm build to pungi's extra_files with inheritance
JIRA: COMPOSE-4214
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-20 13:19:57 +08:00
Lubomír Sedlář
4a78514162 util: Fix regex for detecting debuginfo packages
The `re.match` function already anchors the pattern at the start of the
string, but allows for other characters to continue after match.

This is causing problems with packages like `elfutils-debuginfod-client`
which are not debuginfo.

Let's be safe and explicitly anchor both start and end.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-16 11:13:02 +01:00
Lubomír Sedlář
4efdacd0a0 4.2.1 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-13 12:55:44 +01:00
Jan Kaluza
a209bda73c Create MaterializedPackageSets in threads to make pkgset faster.
When modules are used, there are lot of small package sets. These
package sets have usually less than 500 packages. The createrepo
part of `MaterializedPackageSet.create` executed for such small
set of packages takes around 1 second. Most of this time
the createrepo_c runs in single thread. It does the initialization,
it writes the XML files, ...

The parts of createrepo which can be run in parallel and therefore
would use all the CPUs are quite small for very small package sets.

This commit therefore executes multiple threads with
`MaterializedPackageSet.create` for these very small package sets.

This saves around 40 seconds from pkgset phase for RHEL compose.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-13 12:51:02 +01:00
Lon Hohberger
9391ce3065 pungi-koji: Allow user to specify version component count in latest symlink
This change adds an option which allows users to specify
the number of version components (i.e. values between dots)

The default behavior is preserved, and is equivalent to using
'-1' as the value.

- Negative values remove items from the end.
- Positive values specify the count of of components
- Zero will remove the version field entirely from the symlink
- When nonzero, at least one version component will appear

Merges: https://pagure.io/pungi/pull-request/1361
Signed-off-by: Lon Hohberger <lhh@redhat.com>
2020-03-13 10:50:08 +01:00
Haibo Lin
3543f8fb3e pkgset: Reuse pkgset repos
JIRA: COMPOSE-4158
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-03-13 17:29:53 +08:00
Jan Kaluza
169fa5b453 Allow reusing gather phase results.
- Get also requires/provides of RPMs in package set.
- Store the results of gather phase as pickle file.
- Reuse old gather phase results in case Pungi configuration
  did not change, the "names" of RPMs in global package set
  did not change and their requires/provides did not change.
- Add `gather_allow_reuse` option to enable this feature.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>

Add gather_allow_reuse, add more tests and better handling of gather_lookaside_repos.
2020-03-13 10:17:59 +01:00
Lubomír Sedlář
a32bf4046f Remove fnmatch from nodeps gather method
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-12 09:47:00 +01:00
Lubomír Sedlář
fb7f7396be Remove fnmatch from pkg_is_debug
On Python 2.7 fnmatch is not thread-safe, and this can cause crashes in
nodeps gather source.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-12 09:11:32 +01:00
Jan Kaluza
e70ad8aaa5 Run extra_files phase in parallel to other phases.
It seems the other phases executed in Weaver phase does not need
extra_files output, so it can be run in parallel to them.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-12 07:58:57 +01:00
Jan Kaluza
afcb3e969b Add Compose.old_compose_path and use it when searching for files in old compose.
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.

In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-09 12:47:23 +01:00
Jan Kaluza
fc3b5063ca Do not skip symlinks when searching for old compose.
ODCS creates symlinks to real directories containing the composes.
The directory structure is similar to following one:

- `./nightly/Fedora-Rawhide-20200304.n.0` -> `../odcs-3`
- `./nightly/Fedora-Rawhide-20200305.n.0` -> `../odcs-4`
- `./nightly/latest-Fedora-Rawhide` -> `../odcs-5`

The current Pungi code to search for old composes skips symlinks
and therefore old ODCS composes are not found.

This commit removes this check and therefore symlinks are allowed
when searching for old compose.

I think this check existed to prevent using `latest-*` symlink as
source for the compose. But this is not possible, because the
code checks that the old compose directory name has certain pattern
constructed from release_short, release_version, ... The `latest-*`
symlink definitely does not match this pattern.

I also executed test compose and it worked as expected.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-06 09:03:10 +00:00
Lubomír Sedlář
887c68d05d Display black output in tests
This should make fixing it much easier since it will directly tell users
what it doesn't like.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-05 09:51:47 +01:00
Lubomír Sedlář
16d6e5b0dd Resend message while waiting for ostree ref
If the signing machinery crashes, we don't want to wait forever. This
way at least we can shout for help periodically.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
Fixes: https://pagure.io/pungi/issue/1350
JIRA: COMPOSE-4166
2020-03-04 14:48:03 +01:00
Lubomír Sedlář
4734f9859a Silence all productimg related warnings
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-03 11:06:52 +01:00
Jan Kaluza
145c3adbef Move buildinstall results to final directories if using pungi-buildinstall Koji plugin.
If Koji pungi-buildinstall is used, then the buildinstall results are
stored in the `output_dir` dir, but in "results" and "logs" subdirectories.
We need to move them to final_output_dir.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-03 10:35:25 +01:00
Lubomír Sedlář
3cd94a4aa5 Silence productimg warning in quiet mode
When running with `--quiet`, the warning should not be printed. It would
confuse tools that expect only compose path to be printed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-03-02 09:24:01 +01:00
Haibo Lin
af5ee7030d Check skip_phases in config file
JIRA: COMPOSE-4143
Fixes: https://pagure.io/pungi/issue/1344
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-19 11:46:15 +08:00
Haibo Lin
e4f878a498 Fix typos in comments
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-19 11:26:15 +08:00
Lubomír Sedlář
d23b576a60 Remove dependency on dict.sorted
This is no longer used, and we don't need to pull it in.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-02-17 12:45:49 +01:00
Jan Kaluza
5808733270 The pungi-config-validate --dump-schema now respects --schema-override.
It is useful to actually check how the schema changed after applying
--schema-override. This commit changes --dump-schema in a way that
--schema-override is taken into account and dumped schema contains
the changes done using the --schema-override.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-13 07:27:09 +01:00
Haibo Lin
ff269a8675 gather: Make additional_packages accept debuginfo packages
Now debuginfo packages defined in additional_packages option
could be handled correctly.

JIRA: COMPOSE-4085
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-12 16:38:08 +08:00
Haibo Lin
3a31d47c83 doc: add code convention requirements
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 17:00:39 +08:00
Haibo Lin
9b12be7300 Fix formatting issues
Code should be formatted via black.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:16:20 +08:00
Haibo Lin
56fea60595 Run flake8 and black via tox
JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:45 +08:00
Haibo Lin
65aa8fde2f Fix other flake8 complaints
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:09 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
41a629969c Format code base with black
https://black.readthedocs.io/en/stable/

JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:35:47 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Jan Kaluza
ef33d00f5b Add --schema-override to pungi-config-validate script
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.

There is currently no way how to check that Pungi configuration fulfills
these extra requirements.

This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.

For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:

```
{
    "properties": {
        "pkgset_source": {
            "enum": ["koji"]
        }
    }
}
```

It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.

Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-05 10:03:30 +01:00
Lubomír Sedlář
6f23c7b8ba 4.2.0 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-02-03 08:26:02 +01:00
Lubomír Sedlář
46ea0743a9 iso: Clean up cache for guestmount
When partial cleanup messes up the guestfs cache, the call to guestmount
will fail. To fix that, let's check if there is a problem first and
clean up everything if needed.

Relates: https://bugzilla.redhat.com/show_bug.cgi?id=1771976
JIRA: COMPOSE-3932
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-30 10:35:42 +01:00
Lubomír Sedlář
4c6396f491 Remove deprecated warn() call
Instead warning() should be called.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 15:34:00 +01:00
Lubomír Sedlář
116c617b86 arch_utils: Fix ResourceWarnings
Newer versions of Python report an error when a file is not closed.
Let's avoid it by using with statement.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-29 13:08:44 +00:00