Commit Graph

39 Commits

Author SHA1 Message Date
Jiri Konecny
59727f84b1
Support change of the patch-iso temp dir
This is useful when running pungi-patch-iso on VM with low amount of
memory but higher disk space. Without this option the operation will
fail because /tmp is tmpfs filesystem.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-09-15 15:47:16 +02:00
Jiri Konecny
8d4fea7890 Fix pungi-patch-iso to work with a new compose top dir solution
get_graft_points has changed but without adapting pungi-patch-iso utility so the
exception was raised when executed.

Fixes: https://pagure.io/pungi/issue/1438
Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2020-08-20 15:15:02 +02:00
Lubomír Sedlář
54882a0fc4 Fix typos
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-24 14:16:42 +02:00
Haibo Lin
0196d7fd00 Allow only creating unified ISO for specified arch
Fixes: https://pagure.io/pungi/issue/1393
JIRA: RHELCMP-807
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-09 09:47:12 +08:00
Haibo Lin
7c2743fb50 Include the output of getisoimage in the error message
JIRA: RHELCMP-720
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-06-01 13:29:55 +08:00
Haibo Lin
65aa8fde2f Fix other flake8 complaints
E231 missing whitespace after ','
E265 block comment should start with '# '
E266 too many leading '#' for block comment
E302 expected 2 blank lines, found 1
E501 line too long (115 > 88 characters)
E713 test for membership should be 'not in'
E722 do not use bare 'except'
F812 list comprehension redefines 'g' from line 1499
F821 undefined name 'cmp'
F841 local variable 'ex' is assigned to but never used

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 16:14:09 +08:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
41a629969c Format code base with black
https://black.readthedocs.io/en/stable/

JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:35:47 +08:00
Lubomír Sedlář
38f6162b46 Fix unified isos with missing images.json
The metadata file can be missing if the compose contains no images. We
need to handle that by creating a new empty file to add unified images
to.

JIRA: COMPOSE-4048
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-01-03 09:20:16 +01:00
Haibo Lin
6afbe6d20a buildinstall: Log message for boot config change
JIRA: COMPOSE-3945
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-11-21 12:16:05 +08:00
Lubomír Sedlář
32a6415e58 config: Keep known options defined on CLI
If the validation or dumping script is given some options, they should
only be removed if they are not valid. We have to remove the invalid
ones, otherwise that would cause a warning about unknown options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-25 15:13:16 +02:00
Lubomír Sedlář
acd3c19618 config-validate: Allow defining variables
When trying to validate a template that should later be filled in with
`pungi-config-dump`, there will be errors about undefined variables.
These are meant to be set when the template is populated.

This patch adds support for `-e`, `--define` argument to the validation
script that can be used to suppress these errors.

Alternatively a JSON file is read from the directory with config file
that can contain values for the variables.

The `--define` option is changed in both validation and dumping to allow
empty string as an accepted value.

JIRA: COMPOSE-3599
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-06-21 12:47:50 +02:00
Jiri Konecny
0cd089802f
patch-iso supports multiple graft directories
Pungi patch iso now supports multiple graft directories. This should
make usage more comfortable.

Signed-off-by: Jiri Konecny <jkonecny@redhat.com>
2019-04-12 15:27:54 +02:00
Lubomír Sedlář
31ef7736aa orchestrator: Monitor status of parts
When a phase is started or stopped, add a line to the to output. This
should help users keep track of what is happening in case the part takes
a long time to run.

JIRA: COMPOSE-3287
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 14:20:04 +01:00
Lubomír Sedlář
45cdbb2faf orchestrator: Send messages about the main compose
Only start/finish messages will be sent if a handler is configured.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
088ea7fe37 orchestrator: Support generic pre- and post- scripts
Run arbitrary commands before and after the compose.

The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
86fb93d603 orchestrator: Support getting kerberos ticket
If the configuration sets keytab path and principal, run kinit with
custom cache file, and delete the file at the end of the run.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
afd2d3ae66 orchestrator: Log exception to log file
This data can be useful for debugging.

JIRA: COMPOSE-3284
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-27 09:56:29 +01:00
Lubomír Sedlář
59c9c150ef orchestrator: Use prefix for config substitutions
This will avoid possible conflicts. The names for parts are chosen by
users, and if we decide to add more options in the future, this can
become handy.

JIRA: COMPOSE-3285
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-02-21 09:42:14 +01:00
Lubomír Sedlář
f92c71683f orchestrator: Add missing function arguments
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-21 09:13:03 +01:00
Lubomír Sedlář
7320bf4943 orchestrator: Compatibility with Python 2.6
This is why we can't have nice things.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-14 10:57:05 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
cb33b0278d unified_isos: Add extra variants to metadata
Each unified ISO contains all toplevel variants, and they are listed in
the metadata multiple times (once for each variant). In each case the
metadata should include all other variants that are included on the
image.

JIRA: COMPOSE-2918
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-05 12:01:43 +02:00
Lubomír Sedlář
f3806f7c77 Stop using deprecated pipes.quote
Instead use the definition from python-six. Once we drop Py 2 support
completely, we'll just swap underscores with dots.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed9d7f69a6 Use universal_newlines when running other commands
This will automatically convert the output to unicode/str and we will
not have to worry about decoding ourselves.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
842e2e810e unified-isos: Stop erasing metadata on failure
When saving new metadata fails, the images.json file would be left
empty. Instead we should not touch it at all.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-06 08:37:56 +02:00
Lubomír Sedlář
c66f2228b5 unified-iso: Only link to non-empty variants
Instead of adding images to metadata and then creating hardlinks in a
separate step, do it immediately while we still have accurate
information about what variants it should be linked to.

Fixes: https://pagure.io/pungi/issue/670
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-31 15:15:17 +02:00
Lubomír Sedlář
81cb0952ca checksum: Refactor creating checksum files
Instead of iterating over the images metadata and appending the checksum
to relevant files immediately, we should store them and write only once.

This avoid an issue when the same image is mentioned in the metadata
multiple times. This happens for source images that are listed under
each binary arch.

The unified isos script is updated to use the exact same logic and code.
This also uncovered a problem with the metadata for debuginfo unified
isos: their paths in metadata were incorrect, which lead to missing
checksums.

Fixes: https://pagure.io/pungi/issue/667
Fixes: https://pagure.io/pungi/issue/668
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-17 08:26:51 +02:00
Ken Dreyer
0350b715dd unified-iso: handle empty arch
Prior to this change, when running pungi-create-unified-iso on a compose
with zero builds present for an arch, unified-iso crashes.

The problem is that unified-iso does not set up the arch's debuginfo
destination directory at all before trying to dump the productmd
treeinfo for that arch's debuginfo. productmd tries to write to the
destination directory that does not exist.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-10 12:14:34 -06:00
Lubomír Sedlář
d4b7293acb unified-iso: Use different type for debuginfo iso
This requires productmd>=1.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-06 09:08:08 +02:00
Lubomír Sedlář
f6121f0887 unified-iso: Handle missing paths in metadata
For empty variants the path is no longer stored, so we need to handle
the possible exception. This has no effect on the actual result, as if
the path was empty, we would bail anyway on missing .treeinfo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-04-06 08:24:03 +02:00
Lubomír Sedlář
4b90822115 Add a script for modifying ISO images
With this script it's possible to add additional files into an ISO file.
If the file happens to be ks.cfg, the boot configs are tweaked so that
the kickstart is actually used.

Resolves: #503
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-21 07:55:19 +01:00
Lubomír Sedlář
a1214525f5 unified-iso: Log better error when linking fails
When linking files fails due to target path already existing, we should
print details about the conflict.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář
08b9d275a8 unified-iso: Blacklist extra files metadata
The file is in each variant that has extra files, and would create a
conflict when linking the files.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-03-13 14:45:13 +01:00
Lubomír Sedlář
2c76313382 unified-iso: Fall back to default config
When the configuration can not be read from the compose, we should use
default values instead of aborting. This allows us to work with composes
produced by Distill.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-17 08:15:13 +01:00
Lubomír Sedlář
afaa0e025e unified-iso: Create isos with debuginfo packages
When creating unified ISOs, the script will now also create one iso per
architecture containing a repo with debuginfo packages.

There is no switch to turn this off. The images can simply not be
shipped if not wanted.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-11 14:36:25 +01:00
Lubomír Sedlář
dc19363a5c unified-iso: Create work/ dir if missing
In theory the directory could be deleted before generating the isos.
This patch will recreate it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-05 13:21:49 +01:00
Lubomír Sedlář
e260fe5581 Add script to generate unified ISOs
This a standalone script that will look into a compose and create
unified ISO for each architecture. The ISO contains RPM repositories for
all variants that have the arch.

Known issues:
 * The filename does not respect settings. This is tricky because the
   name could include variant name, which we don't have here (by design
   of unified ISO).
 * The same is true for volume id.

In order to test the feature without running actual compose, we need to
add essentially a big chunk of compose. Most of the files are empty, as
their content is never accessed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-01-04 10:22:20 +01:00