Commit Graph

48 Commits

Author SHA1 Message Date
Haibo Lin
20c2e59218 Pass compose parameter for debugging git issue
With this param, get_dir_from_scm will try to copy the tmp git dir to
compose target dir when error occurs.

This does not fix the issue but it would be helpful for debugging when
it occurs again.

JIRA: RHELCMP-7244
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-11-26 15:40:35 +08:00
Dominik Rumian
7c3e8d4276 Fix tests for createrepo
Tests for createrepo failed when pungi is installed in system.

JIRA: RHELCMP-6209

Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 13:48:29 +02:00
Dominik Rumian
9cd42a2b5e Formatted files according to flake8 and black feedback
Signed-off-by: Dominik Rumian <drumian@redhat.com>
2021-08-30 09:55:17 +02:00
Haibo Lin
56a55db966 Use cachedir when createrepo
Then createrepo can reuse checksum values from cache to make it faster.

JIRA: RHELCMP-5984
Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-07-27 09:23:47 +08:00
Haibo Lin
b217470464 Format code
Code didn't get well formatted when jenkins unusable.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2021-03-02 18:31:45 +08:00
Haibo Lin
3c72755814 Use pytest instead of nosetests
Nose has been in maintenance mode for the past several years
and pytest is a good replacement.

Other changes:
- Replace deprecated assertRegexpMatches with assertRegex
- Replace deprecated assertRaisesRegexp with assertRaisesRegex
- Replace deprecated SafeConfigParser with ConfigParser
- Force reinstall pytest and mock in tox virtualenv. This is because
  the globally installed packages may not work as expected(occured in
  jenkins job).

JIRA: RHELCMP-1619
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-07-29 14:57:16 +08:00
Lubomír Sedlář
3bb1e3df11 createrepo: Add extra modulemd files to the repo
This is a workaround for modularity design issues and DNF bugs. If there
were gaps in contexts, DNF has trouble handling the upgrades. Thus we
may need to add module metadata for older versions of previously
released module streams and add the missing contexts.

JIRA: RHELCMP-982
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2020-06-10 09:50:44 +02:00
Jan Kaluza
afcb3e969b Add Compose.old_compose_path and use it when searching for files in old compose.
The current code calls `find_old_compose` followed by multiple `os.path.*`
calls to find out if particular file exists in the old compose. This
duplicates code a lot and makes it harder to read.

In this commit, new `Compose.old_compose_path` is introduced and
used instead of direct calls of `find_old_compose`.

Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-03-09 12:47:23 +01:00
Haibo Lin
c0193c9fca Fix flake8 complaints - E501
E501 line too long (92 > 88 characters)
E501 line too long (103 > 88 characters)
...

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 14:36:46 +08:00
Haibo Lin
3eddcfccd8 Fix flake8 complaints - F401
F401 'dnf' imported but unused
F401 'imp' imported but unused
F401 'os' imported but unused
F401 'subprocess' imported but unused
F401 'sys' imported but unused
F401 'yum' imported but unused

JIRA: COMPOSE-4108
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-07 11:48:31 +08:00
Haibo Lin
38142d30ba Format tests with black
JIRA: COMPOSE-4086
Signed-off-by: Haibo Lin <hlin@redhat.com>
2020-02-05 17:29:15 +08:00
Haibo Lin
0ed70fc8b6 tests: drop sys.path modification
It's unnecessary to add the path manually.

Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-12-10 10:53:38 +08:00
Lubomír Sedlář
908a6a759d Use custom assertion for checking content of files
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:44 +02:00
Lubomír Sedlář
562b770b8d Drop custom ANY object
There is already one such thing in mock library.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
2f54745715 Remove usage of unittest2 on Python 3
It was needed to provide assertItemsEqual method. Starting with Python
3.2, there's assertCountEqual that does the same thing. Six provides a
helper that will dispatch to the existing method. With this change,
unittest 2 is only needed on Python 2.6 to backport the method.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:25:18 +02:00
Lubomír Sedlář
43fb06e00f Remove shebangs from test files
They should be executed with a runner like nose, not directly.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-04 15:24:17 +02:00
Lubomír Sedlář
8ab7d9f7ba Move import of modulemd to a separate module
This should make it possible to only import the library only when it's
really needed.

DNF does not work with libmodulemd v2. If we import libmodulemd2 and
then dnf, the program will just hang forever. We only need DNF in
pungi-gather, where libmodulemd is not needed, and also where we do need
libmodulemd, we don't have DNF.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:40:31 +02:00
Lubomír Sedlář
3824eab15b createrepo: Remove arch_repo usage
There is no longer a single repo with all packages. This means that the
metadata has to be loaded from another location.

When taking packages from Koji, we can assume that the non-modular
package tag will be processed last. The repo for this tag will be used.
This has better chance of being useful than using a random module.

For repo sources, there is only one package set anyway, so this change
makes no difference.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-09-03 09:31:18 +02:00
Lubomír Sedlář
61e3cb0ef1 Port to libmodulemd v2
Fixes: https://pagure.io/pungi/issue/1225
JIRA: COMPOSE-3662
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-15 15:27:05 +02:00
Lubomír Sedlář
04baa2a4db Stop storing modulemd without arch
Historically each variant had a list of modules. This is no longer
needed and can be dropped. We can also stop logging the modulemd since
we know it was retrieved from Koji and not modified locally.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-07-11 14:25:42 +02:00
Lubomír Sedlář
9939d09643 createrepo: Stop processing modulemd
The file we get from Koji has all important bits already filled in.
There is no need to add anything.

This patch also stops filtering the artifacts to only contain packages
that are actually in the repo. Thus debuginfo and source packages will
appear there.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-25 08:48:16 +01:00
Lubomír Sedlář
1a161982c0 gather: Make devel modules configurable
By default nothing should change. This patch adds a new config option
that enables all this new craziness:

 * it turns of applying module filters at pkgset level
 * it creates new modules and adds them to the compose

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
06b32b5d80 pkgset: Stop prefilling RPM artifacts
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-10-08 14:41:04 +02:00
Lubomír Sedlář
1350684c31 Index arch modulemd by full NSVC
There can be multiple modules with the same name and stream. They should
all have the same version, but will have different contexts. Fus takes
only N:S as input, but should pull in all matching modules. We just need
to give it correct data in the repo.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-09-05 13:10:53 +02:00
Lubomír Sedlář
7c5020e82d Reduce duplication in tests
Set the dummy compose into non-debug mode in a single place, and only
change it when needed.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-29 15:52:45 +02:00
Lubomír Sedlář
2fa1f09827 createrepo: Allow passing arbitrary arguments
This could be used to enable zchunk generation, which can require up to
4 different options. Instead of hardcoding every single one, let's just
allow more direct access to the executed command.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-08-08 16:15:51 +02:00
Lubomír Sedlář
096075848b createrepo: Include empty modules
A module without any RPMs is still valid and should be included. It
should also be in the metadata.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-22 12:54:39 +02:00
Lubomír Sedlář
71f2c61020 Add test for getting licenses from a repo
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-06-04 08:26:26 +02:00
Lubomír Sedlář
b4e746aa71 createrepo: Allow disabling SQLite database
This is an optimization for Yum. DNF does not care at all.

The behaviour is configurable, but the default depends on gather
backend, as that is what users should be using to consume the packages
from the repo.

Fixes: https://pagure.io/pungi/issue/951
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-21 08:13:29 +02:00
Lubomír Sedlář
490f079c44 Update tests for libmodulemd 1.4.0
It got a little picky about allowed inputs. We need full RPM NEVRA in
all tests.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-02 15:36:52 +02:00
Ondrej Nosek
e3aa2f769b Write module metadata
Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2018-03-29 09:47:55 +02:00
Jan Kaluza
1574f306c7 Move Modulemd import to pungi/__init__.py to remove duplicated code.
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2018-03-20 12:59:17 +01:00
Jan Kaluza
3f71cdd384 Use Modulemd.Module for 'variant.arch_mmds' instead of yaml dump
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
Merges: https://pagure.io/pungi/pull-request/872
2018-03-19 14:47:03 +01:00
Ondrej Nosek
e0308a74ff remove remaining hard coded createrepo threads
To be more precise, new createrepo parameter "workers" was customized and
new default value was set. This fixes issue #752.

Signed-off-by: Ondrej Nosek <onosek@redhat.com>
2017-10-30 10:37:10 +01:00
Lubomír Sedlář
f21e3a2d6d tests: Work with older unittest2
RHEL has an older version of the library which does not backport all the
assertions that we used. In order for the tests to pass there we need to
use names that exist everywhere.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-30 09:16:18 +01:00
Lubomír Sedlář
ed22e07ef9 Port to Python 3
This should make all tests pass on both Python 2 and Python 3.

Unittest2 is required on Py 2.6 and Py 3.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-24 15:15:31 +02:00
Lubomír Sedlář
dec00fe2f4 createrepo: Allow customizing number of threads
The default is now to use one thread per CPU.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-10-03 15:21:37 +02:00
Lubomír Sedlář
e21a27bdc9 createrepo: Allow selecting variants for delta RPMs
The configuration needs to be more granular than a single global option.
With this patch each tree can enable deltas separately.

Fixes: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
90be25c14c createrepo: Only create delta RPMs for binary repos
It does not make much sense to have deltas for source and debug repos.
No one benefits from it really and it takes a long time.

Relates: https://pagure.io/pungi/issue/715
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-09-04 09:15:20 +02:00
Lubomír Sedlář
a831d65c40 createrepo: Don't use existing metadata with deltas
When creating the final repo, we reuse metadata from arch repo used for
depsolving. This however breaks creating deltas with createrepo_c.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 14:29:16 +02:00
Lubomír Sedlář
910f816be4 createrepo: Use correct paths for old package dirs
Createrepo expects to be pointed to a directory with the actual RPM
files, not the previous repo. This means that when hashed directories
are used, we need to pass in a lot of directories.

Fixes: https://pagure.io/pungi/issue/344
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-14 10:40:44 +02:00
Lubomír Sedlář
5148adf233 createrepo: Add tests for adding product certificates
If the certificate is enabled and exists, it will currently be added to
all rpm repositories.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Lubomír Sedlář
96fc6fb11e createrepo: Add tests for retrieving product certificates
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-23 11:17:04 +01:00
Patrick Uiterwijk
59dbd9d35f Remove executable permissions on test scripts
Signed-off-by: Patrick Uiterwijk <puiterwijk@redhat.com>
2016-10-25 10:42:16 +00:00
Lubomír Sedlář
bf46048cbf createrepo: Add option to use xz compression
This patch adds a new config option createrepo_use_xz, which when set to
true will cause createrepo to compress sqlite databases with xz. The
default setting is False.

Fixes: #387
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-12 11:10:43 +02:00
Lubomír Sedlář
d5512fd6ff Use unittest2 if available
The module backports features to Python 2.6 and 2.7. If it is available,
the tests will use it. If it is not available, it will fall back to
regular unittest. On Python 2.7, the tests pass anyway. On Python 2.6,
there are failures with Python 2.6.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-05-25 13:41:49 +02:00
Lubomír Sedlář
27375788c2 [createrepo] Compute delta RPMS against old compose
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-07 14:49:17 +01:00
Lubomír Sedlář
223344e04a [createrepo] Add tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-02 16:11:25 +01:00