Commit Graph

1647 Commits

Author SHA1 Message Date
Ken Dreyer
0350b715dd unified-iso: handle empty arch
Prior to this change, when running pungi-create-unified-iso on a compose
with zero builds present for an arch, unified-iso crashes.

The problem is that unified-iso does not set up the arch's debuginfo
destination directory at all before trying to dump the productmd
treeinfo for that arch's debuginfo. productmd tries to write to the
destination directory that does not exist.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-10 12:14:34 -06:00
Ken Dreyer
5acfb90b23 createrepo: handle missing product ids scm dir
Prior to this change, if the entire product IDs SCM directory was missing, pungi would crash with an error.

For example, if "ceph-3" was missing from the SCM:

    OSError: [Errno 2] No such file or directory: '/tmp/tmpMb9O6r/product_ids/ceph-3'

This occurred even if product_id_allow_missing was set to True.

Make product_id_allow_missing cover this case as well, and gracefully
skip all product IDs.

We now see the following warning in the logs instead:

    [WARNING ] No product IDs in {'scm': 'git', 'repo': 'git://example.com/rcm/rcm-metadata.git', 'dir': 'product_ids/ceph-3'}

and the compose succeeds.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
2017-07-07 09:59:17 -06:00
Lubomír Sedlář
afffb27f94 comps_wrapper: Code clean up
* Remove explicit option requirement. Argparse can take care of that
  while also making this information visible in help output.
* Simplify writing resulting comps.
* Remove unused code.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 15:51:48 +02:00
Pavel Holica
9a3d04c305 comps_filter: Filter environments by arch
If the environment has arch attribute, the environment should be removed
from the file on all other arches. This mirrors similar behaviour for
groups and packages.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 14:45:59 +02:00
Lubomír Sedlář
a21c8a555d notification: Allow specifying multiple scripts
The notification hooks can be useful for doing other things than just
announcing status on message bus. For this to be truly usable, we need
the ability to use multiple scripts.

This patch allows the command line option to be specified multiple
times. Each given script will be called. Even if the script fails, it
does not block the compose.

Additionally the output of the notification scripts is logged now to
make it possible to debug possible failure.

Relates: https://pagure.io/pungi/issue/650
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-29 08:30:56 +02:00
Qixiang Wan
18bd37ff2c pkgset: Allow populating packages from multiple koji tags
One of the use cases is https://pagure.io/odcs/issue/7.

Merges: #660
Signed-off-by: Qixiang Wan <qwan@redhat.com>
2017-06-29 08:27:34 +02:00
Lubomír Sedlář
9193a6902e pungi: Port to argparse
And fix PEP8 violations while modifying the file.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:37:03 +02:00
Lubomír Sedlář
87c485f2d0 comps_filter: Port to argparse
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:37:03 +02:00
Lubomír Sedlář
ecbb43ab86 variants-wrapper: Remove main() function
Nothing is calling this code, and it's only really useful for manual
testing of parsing.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:48 +02:00
Lubomír Sedlář
69a6046bf8 multilib_yum: Remove main() function
Nothing is calling this file as an executable, so there is no reason to
have logic for parsing arguments.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:39 +02:00
Lubomír Sedlář
01026858f7 pungi-koji: Port to argparse
Optparse is deprecated since Python 2.7, long live argparse.

This also allows us to remove some of the manual error checking and make
the library check required or conflicting options.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 09:36:23 +02:00
Lubomír Sedlář
68098bec37 ostree: Update tests for no ostree init
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-27 07:46:35 +02:00
Colin Walters
af631f560e ostree: Don't automatically create a repo
Creating an OSTree repository is a notable event; the general
expectation is that rather than having lots of repositories,
one has branches inside a single repository.

For $reasons, Fedora is not currently doing this, but we will
change it to do so.

The reason I'm making this change is we discovered that
it looked like Fedora had somehow made a repo inside a repo,
presumably due to a configuration error.

https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/GBFSOLULGGZFGEFCIW6FG23NZZV5VH4K/
Signed-off-by: Colin Walters <walters@verbum.org>
2017-06-26 17:47:35 -04:00
Lubomír Sedlář
bfc1cebbc4 osbs: Config validation should accept a list
There can be multiple images listed for a single variant, the config
validation should not reject it.

The syntax with a single config object is still accepted. The price for
that is less descriptive error message when there are errors.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-23 12:39:48 +02:00
Martin Curlej
079454c502 pkgset: Use release number of a module
Signed-off-by: Martin Curlej <mcurlej@redhat.com>

updated regex, added comment, pep8 fix
2017-06-21 15:53:39 +02:00
Lubomír Sedlář
65bc6969e2 docs: Add a basic info about gathering packages
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-21 09:55:29 +02:00
Dennis Gilmore
65b75e7049 Merge #646 ostree-installer: Clean up output dir 2017-06-20 15:31:09 +00:00
Dennis Gilmore
8e89168ad3 Merge #647 gather: Log tag from which we pulled a package 2017-06-20 15:27:38 +00:00
Lubomír Sedlář
3e64238fca docs: Kobo can be installed via pip now
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-20 11:11:16 +02:00
Dennis Gilmore
72081a3a35 Merge #649 docs: Add overview of what each phase does 2017-06-19 17:22:43 +00:00
Lubomír Sedlář
9ab3dc89b2 docs: Add overview of what each phase does
Fixes: #520
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:58:01 +02:00
Lubomír Sedlář
b2554ce663 gather: Log tag from which we pulled a package
For each tag we ask Koji about (there might be more than one in
modularity case), we create a log file with list of RPMs and details
about which tag they were pulled from. This makes it easier to find out
where the package is inherited from.

Fixes: #547
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:37 +02:00
Lubomír Sedlář
2a8d7f8843 docs: Document config file format
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:31 +02:00
Lubomír Sedlář
bef573d222 docs: Move logo to _static subdir
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 10:51:23 +02:00
Lubomír Sedlář
d586368515 Add dropped livemedia phase
After the patch to ensure that all phases are stopped the livemedia
phase was apparently dropped by accident.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-19 08:58:53 +02:00
Lubomír Sedlář
4fb28979cf gather: Display source repo of packages
When multiple repos are configured in pkgset_repos, the logs should
contain information on where exactly the package was pulled from. The
log file in question should be in
`logs/<arch>/pkgset_source.<arch>.log`.

Fixes: #545
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-16 12:39:04 +02:00
Lubomír Sedlář
c337018294 pkgset: Use descriptive name for log file
When repos are used as pkgset source, the logs are stored in a file that
should have a better name than `fooo.<arch>.log`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-16 09:57:49 +02:00
Lubomír Sedlář
6e6e250cec ostree-installer: Clean up output dir
The Koji task can be restarted and lorax will fail if the output
directory already exists. Let's start the work in runroot by removing
the output directory.

Relates: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-15 08:17:07 +02:00
Lubomír Sedlář
1c3637c48f Ignore more pycodestyle warnings
These warnings are ignored by default, but setting our own ignore list
removes these defaults.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-14 13:21:31 +02:00
Jan Kaluza
f7197ddbcc Allow gather source classes to return SimpleRpmWrapper objects from pkgset phase directly.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-06-14 08:51:37 +02:00
Lubomír Sedlář
a54b68d08b tests: use unittest2 if available
This fixes a test failure on RHEL 6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-13 08:55:41 +02:00
Lubomír Sedlář
81b71b9ed3 koji-wrapper: Handle failed subtasks
If a subtask fails, we can't ask about it's results as that would raise
an exception. We can safely assume that since the parent succeeded, any
failed child is actually allowed to fail.

Fixes: #641
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-12 18:15:29 +02:00
Lubomír Sedlář
d3dd7aa7ce 4.1.16 release
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-09 13:12:19 +02:00
Lubomír Sedlář
3a59e8f266 Fix changelog generator script
* Correctly exclude merge commits (there's a git log option for it)
 * Stop mangling log lines with @ symbol. This should only be cut for
   author of a change.
 * Fix PEP8 issues
 * Remove unused code
 * Port to argparse (optparse is deprecated)

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-08 11:00:22 +02:00
Lubomír Sedlář
0cdf996e6e util: Retry resolving git branches
When there's a temporary network issue, Pungi will fail to turn a branch
into a commit hash. This would abort the whole compose. Instead we
should just retry a few times.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-06 15:31:39 +02:00
Lubomír Sedlář
700106facf arch: Move exclu(de|sive)arch check to a function
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-06 15:28:12 +02:00
Jan Kaluza
3601d6d1a8 gather-source: Check arch in module source
Skip the RPM if it is excluded on this arch or exclusive for different
arch.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-06-06 15:25:41 +02:00
Lubomír Sedlář
1db1abbb82 koji-wrapper: Stop mangling env variables
When koji is authenticated with a keytab, by setting the private
directory we erased rest of existing environment. In non-keytab path,
the environment variables got removed as well.

This patch makes sure that the environment will not be modified more
than necessary (by setting KRB5CCNAME if needed).

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-05 12:46:45 +02:00
Dennis Gilmore
3ddfd4d172 Merge #621 gather: Don't resolve dependencies in lookaside 2017-06-02 14:01:46 +00:00
Lubomír Sedlář
247a1a71ba Ensure all phases are stopped
If a phase is started successfully, it needs to be stopped as well. In
most cases when `stop` is called immediately after `start`, this is not
a problem.

Only when something else happens while a phase is runnning and this
something fails, Pungi will deadlock and never exit. This something
could be another phase or just main thread raising an exception.

Fixes: #625
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-02 09:52:18 +02:00
Lubomír Sedlář
8c237b78c2 comps-wrapper: Report unknown package types
When there is a typo in the comps file, instead of crashing with a
non-descript KeyError we should raise a nice error with details about
the problem.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-06-01 09:59:41 +02:00
Jan Kaluza
118444a311 Generate proper modular metadata when there are different versions of the same package in the variant
Merges: #629
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-30 15:18:09 +02:00
Lubomír Sedlář
58a6affd65 checks: Make gpgkey a boolean option
This option is currently only checked in the ostree phase, and it does
not make sense as a string. When any non-empty string was given, it
enabled the check.

Relates: #590
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:31:47 +02:00
Lubomír Sedlář
a2c42aee3c ostree: Refactor writing repo file
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:26:49 +02:00
Lubomír Sedlář
63c0bbc5c9 iso-wrapper: Capture debug information for mounting
Occasionally we have seen the mount command fail. The default error
message says to set some environment variables and try again. We can
just always set the environment and only print the output on failure.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:18:07 +02:00
Lubomír Sedlář
77ee882c21 comps-wrapper: Fix crash on conditional packages
A `requires` attribute is taken from a wrong package (because of a wrong
variable used: `pkg` vs. `package`). On RHEL 6 this actually leads to a
crash. Let's use only one name to avoid such problems.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 13:11:49 +02:00
Lubomír Sedlář
3130d837c0 gather: Don't resolve dependencies in lookaside
When looking at a package in a lookaside repo, it does not make much
sense to process its dependencies. We should just assume that the
lookaside can satisfy them.

In the worst case, this could result in packages being pulled into the
compose just so that they could satisfy a dep of something in lookaside.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:55:38 +02:00
Lubomír Sedlář
f27f3ce4ba koji-wrapper: Run all blocking commands with fresh ccache
If keytab is used for authentication, other commands than runroot can
possibly fail due to the credentials cache being overwritten.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-29 12:53:29 +02:00
Jan Kaluza
68351fa5a8 Add @retry decorator and use it to retry connection on PDC on IOError and in SCM's retry_run.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2017-05-29 11:33:23 +02:00
Lubomír Sedlář
dc410b58b5 Remove shebang from non-executable files
This fixes an rpmlint error on built package.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-05-10 09:07:16 +02:00