Commit Graph

26 Commits

Author SHA1 Message Date
Lubomír Sedlář
31ef7736aa orchestrator: Monitor status of parts
When a phase is started or stopped, add a line to the to output. This
should help users keep track of what is happening in case the part takes
a long time to run.

JIRA: COMPOSE-3287
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 14:20:04 +01:00
Lubomír Sedlář
45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
088ea7fe37 orchestrator: Support generic pre- and post- scripts
Run arbitrary commands before and after the compose.

The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
86fb93d603 orchestrator: Support getting kerberos ticket
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
afd2d3ae66 orchestrator: Log exception to log file
This data can be useful for debugging.

JIRA: COMPOSE-3284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-27 09:56:29 +01:00
Lubomír Sedlář
59c9c150ef orchestrator: Use prefix for config substitutions
This will avoid possible conflicts. The names for parts are chosen by
users, and if we decide to add more options in the future, this can
become handy.

JIRA: COMPOSE-3285
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 09:42:14 +01:00
Lubomír Sedlář
f92c71683f orchestrator: Add missing function arguments
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-21 09:13:03 +01:00
Lubomír Sedlář
7320bf4943 orchestrator: Compatibility with Python 2.6
This is why we can't have nice things.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-14 10:57:05 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
cb33b0278d unified_isos: Add extra variants to metadata
Each unified ISO contains all toplevel variants, and they are listed in
the metadata multiple times (once for each variant). In each case the
metadata should include all other variants that are included on the
image.

JIRA: COMPOSE-2918
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář
f3806f7c77 Stop using deprecated pipes.quote
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed9d7f69a6 Use universal_newlines when running other commands
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
842e2e810e unified-isos: Stop erasing metadata on failure
When saving new metadata fails, the images.json file would be left
empty. Instead we should not touch it at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-06 08:37:56 +02:00
Lubomír Sedlář
c66f2228b5 unified-iso: Only link to non-empty variants
Instead of adding images to metadata and then creating hardlinks in a
separate step, do it immediately while we still have accurate
information about what variants it should be linked to.

Fixes: https://pagure.io/pungi/issue/670
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-31 15:15:17 +02:00
Lubomír Sedlář
81cb0952ca checksum: Refactor creating checksum files
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.

This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.

The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.

Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Ken Dreyer
0350b715dd unified-iso: handle empty arch
Prior to this change, when running pungi-create-unified-iso on a compose
with zero builds present for an arch, unified-iso crashes.

The problem is that unified-iso does not set up the arch's debuginfo
destination directory at all before trying to dump the productmd
treeinfo for that arch's debuginfo. productmd tries to write to the
destination directory that does not exist.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-10 12:14:34 -06:00
Lubomír Sedlář
d4b7293acb unified-iso: Use different type for debuginfo iso
This requires productmd>=1.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-06 09:08:08 +02:00
Lubomír Sedlář
f6121f0887 unified-iso: Handle missing paths in metadata
For empty variants the path is no longer stored, so we need to handle
the possible exception. This has no effect on the actual result, as if
the path was empty, we would bail anyway on missing .treeinfo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-06 08:24:03 +02:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
a1214525f5 unified-iso: Log better error when linking fails
When linking files fails due to target path already existing, we should
print details about the conflict.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář
08b9d275a8 unified-iso: Blacklist extra files metadata
The file is in each variant that has extra files, and would create a
conflict when linking the files.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář
2c76313382 unified-iso: Fall back to default config
When the configuration can not be read from the compose, we should use
default values instead of aborting. This allows us to work with composes
produced by Distill.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-17 08:15:13 +01:00
Lubomír Sedlář
afaa0e025e unified-iso: Create isos with debuginfo packages
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.

There is no switch to turn this off. The images can simply not be
shipped if not wanted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:36:25 +01:00
Lubomír Sedlář
dc19363a5c unified-iso: Create work/ dir if missing
In theory the directory could be deleted before generating the isos.
This patch will recreate it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-05 13:21:49 +01:00
Lubomír Sedlář
e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00