Commit Graph

28 Commits

Author SHA1 Message Date
Haibo Lin
477b43d4e9 Split repoclosure into separate phase
Move repoclosure out from test phase into its own phase and
run parallel with image building phases(osbs, imagebuild, ...)
to speed things up.

JIRA: RHELCMP-8
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-04-14 09:25:57 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Lubomír Sedlář
aefe9b186d repoclosure: Parse all fus logs
Originally the list of solvables for fus was growing with each iteration
and nothing was ever removed. That later changed so that fus iterations
are only done on newly added stuff. It's great for performance, but
means that the last log is not a superset of all others.

To get all dependency problems we need to look into all log files, not
just the last one.

JIRA: COMPOSE-3964
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-12-02 11:08:55 +01:00
Haibo Lin
e0dd20dffe Prefix repoclosure cache dir with compose id
JIRA: COMPOSE-3922
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-20 13:51:13 +08:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Haibo Lin
53a93f016a Refactor code for removing repoclosure cache
For 'yum' backend, only cache dirs following repoclosure-$COMPOSE_ID-$variant.$arch
name convention are created, e.g. repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64

But for 'dnf' backend, the dir name looks like
repoclosure-$COMPOSE_ID-$variant.$arch-$suffix and there are other files
created, e.g.
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-df9fe164317e314e
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64-filenames.solvx
    repoclosure-DP-1.0-20190822.t.0-Bar-Tools.x86_64.solv

JIRA: COMPOSE-2565
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-23 11:00:17 +08:00
Haibo Lin
048698d885 Delete repoclosure cache
JIRA: COMPOSE-2565
Merges: https://pagure.io/pungi/pull-request/1253
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-15 08:42:16 +02:00
Haibo Lin
c00162413c Add compose_id to repoclosure/lookaside directory names
JIRA: COMPOSE-3616
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-08-05 15:57:27 +08:00
Lubomír Sedlář
3097019338 test: Allow turning strictness off as well
When the value is explicitly set to `False`, the check should not be
strict.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-16 08:24:18 +02:00
Lubomír Sedlář
62c6d4ddcf test: Option to make size check strict
Sometimes it's practical not just warn when ISO is larger than expected,
but to also abort the compose.

JIRA: COMPOSE-3658
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 13:57:50 +02:00
Lubomír Sedlář
6c6d4759f5 isos: Check maximum expected size
This patch allows the configuration to express maximum expected size for
ISOs created in createiso and extra_isos phases. If the image is larger
than this limit, a warning is emitted in test phase. The compose itself
is not affected in any way.

JIRA: COMPOSE-2824
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-12 15:15:30 +01:00
Lubomír Sedlář
fe39056431 repoclosure: Extract logs from hybrid solver
There is no repoclosure that correctly understands modules. The best
thing we can offer is the errors reported by the depsolver.

JIRA: COMPOSE-2321
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-26 08:42:21 +02:00
Lubomír Sedlář
d5a0316877 Use dnf backend for repoclosure on PY3
When yum is not available, it makes no sense to offer it as an option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-12-06 13:26:43 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
e2962dc547 repoclosure: Allow aborting compose when repoclosure fails
Alternatively the call to repoclosure can be turned off. This is
customizable per variant and architecture.

Fixes: https://pagure.io/pungi/issue/676
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-08-07 15:17:33 +02:00
Lubomír Sedlář
dc7d3b36ab repoclosure: Don't run build deps check
Fixes: #521
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-30 10:16:03 +01:00
Lubomír Sedlář
e3fe67be53 repoclosure: add option to use dnf backend
This adds a new option repoclosure_backend that changes what tool is
used for repoclosure.

Checking build dependencies is currently not supported, as `dnf` does
not have the corresponding option.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:39:12 +01:00
Lubomír Sedlář
95fc0fa4ab repoclosure: Add test for repoclosure in test phase
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-26 13:27:48 +01:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Lubomír Sedlář
ef99e28849 [test] Only check bootability for images on x86_64 and i386
Current check does not really work for other arches.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-08-10 13:02:56 +02:00
Lubomír Sedlář
463088d580 Refactor failables
This is a breaking change as big part of current failable_deliverables
options will be ignored.

There is no change for buildinstall and creatiso phase.

Failability for artifacts in other phases is now configured per
artifact. It already works correctly for ostree and ostree_installer
phases (even per-arch). For OSBS phase there is currently only a binary
switch as it does not handle multiple arches yet. When it gains that
support, the option should contain list of non-blocking architectures.

For live images, live media and image build phases each config block can
configure list of failable arches. If the list is not empty, it can
fail. Once we have a way to fail only some arches, the config will not
need to change.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-27 13:06:01 +02:00
Lubomír Sedlář
22e94caf3c [test] Correctly check bootable ISOs
ISO image without MBR and GPT can still be bootable if it has an El
Torito boot catalog. The test phase must accept such images.

This slightly defeats the point of the check: to verify the ISO is
hybrid. However, based on the metadata we have no way to actually tell
if the image is supposed to be hybrid.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-07-22 14:11:06 +02:00
Lubomír Sedlář
b2a266c3e4 Fix checking optional ISO images in test phase
Instead of iterating over image manifest, loop through all variants and
arches and see if there are any images. This avoids a crash for variants
nested under other variants (layered products, optionals or addons).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-06 10:41:06 +02:00
Lubomír Sedlář
d5512fd6ff Use unittest2 if available
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
4e3d87e658 [test] Add checks for created images
The performed checks:
 * If format is ISO, the file must have correct magic string
 * If it's bootable, there must be MBR or GPT

When a check fails on any failable deliverable, it will be logged and
the file removed from metadata (it will still remain on the disk). This
required a change to write the images.json file later (after test
phase).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-05 15:38:00 +02:00