Commit Graph

33 Commits

Author SHA1 Message Date
stepan_oksanichenko
084321dd97 LNX-326: Add the ability to include any package by mask in packages.json to the generator
- The ability is added
- Also the generator includes only the latest by version packages to packages.json
- The generator has key `--is-reference` for an each repo. This key marks a repo as reference.
  An reference repo is used as main source of packages. A not reference repo is used as source
  of packages which don't exist in the reference repos.
- All cases are covered by the unittest

@BS-NOBUILD
@BS-TARGET-CL8

Change-Id: I2f80ba4fbfce27fb9a30500ae46c0b8a2f2aabcd
2021-06-15 17:42:12 +03:00
soksanichenko
5434d24027 LU-2133: Prepare CI for iso builds of CLOSS 8
@BS-TARGET-CL8
@BS-NOBUILD

- It's added the script which can collect packages/modules
  from the remote repos (including BS repos) and merge them
  to an one local repo with the right repodata (including
  modules.yaml.gz)
- The script `create_packages_json` can use regexps for list of excluded packages

Change-Id: I1365b712460959db6bb451d1199d640bff6ffe5e
2021-02-09 10:47:46 +02:00
Stepan Oksanichenko
94ad7603b8 LNX-104: Create gather_prepopulate file generator for Pungi
- It's added the tool which can generate json like as `centos-packages.json` using repodata from completed repos.

@BS-LINKED-5ffda6156f44affc6c5ea239  # pungi & dependencies
@BS-TARGET-CL8

Change-Id: Ib0466a1d8e06feb855e81fb7160fe170e2e82e04
2021-01-25 16:17:34 +02:00
Jan Kaluza
ef33d00f5b Add --schema-override to pungi-config-validate script
Some composes might need extra validation to ensure they are following
certain strict rules - for example containing only signed packages or
packages only from particular Koji tag.

There is currently no way how to check that Pungi configuration fulfills
these extra requirements.

This commit adds new `--schema-override` option to
`pungi-config-validate` script which allows caller to specify path to
JSON schema overriding the default JSON schema and therefore limitting
it further.

For exmaple, to limit the `pkgset_source` to `koji`, one can use
following JSON schema override:

```
{
    "properties": {
        "pkgset_source": {
            "enum": ["koji"]
        }
    }
}
```

It is possible to use `--schema-override` multiple times to apply
multiple schema overrides.

Merges: https://pagure.io/pungi/pull-request/1341
Signed-off-by: Jan Kaluza <jkaluza@redhat.com>
2020-02-05 10:03:30 +01:00
Lubomír Sedlář
ff526d1dd2 Regenerate test fixture repo
Make dummy-bash -> dummy-glibc dependency require archful. This avoids
potential race condition where order of dependency processing can result in
different packages being pulled in. The tests where this could happen are
updated.

Make dummy-glibc-debuginfo depend on dummy-glibc-debuginfo-common.

The filenames for the repo no longer include hash, and sqlite databases are not
generated.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-10-03 10:46:21 +02:00
Haibo Lin
3811c0a176 config: Deprecate bootable option
JIRA: COMPOSE-2635
Fixes: https://pagure.io/pungi/issue/976
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-07-22 16:51:38 +08:00
Haibo Lin
c4ed2bf3b2 config: Deprecate runroot option
Fixes: https://pagure.io/pungi/issue/975
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-27 18:00:07 +08:00
Haibo Lin
ce066707c1 config: Deprecate release_is_layered option
Fixes: https://pagure.io/pungi/issue/977
Merges: https://pagure.io/pungi/pull-request/1204
Signed-off-by: Haibo Lin <hlin@redhat.com>
2019-06-06 13:45:49 +02:00
Lubomír Sedlář
e71e91982b Update test data
The parts in multi compose are now named reasonably.

The package set repos are selected only once and not per variant. We
can't use repos with packages for specific arch only as that would
require more transformations during generating test data.

Lookasides for ResilientStorage part are added to point to Server part.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-04-01 09:49:53 +02:00
Lubomír Sedlář
088ea7fe37 orchestrator: Support generic pre- and post- scripts
Run arbitrary commands before and after the compose.

The example config is updated to generate latest symlink with a
post-compose script. The pre compose script runs always, post compose
runs only if the compose is not doomed.

JIRA: COMPOSE-3288
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-03-21 09:29:54 +01:00
Lubomír Sedlář
90c60f8e64 Add script to orchestrate multiple composes
It may make sense to break a big compose into smaller chunks that can be
done independently. This script allows describing the smaller parts,
runs them with correct dependencies and arranges the result to look like
a single big compose.

All parts use the same koji event, that is either obtained from Koji, or
from command line argument.

JIRA: COMPOSE-2654
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2019-01-08 09:40:36 +01:00
Lubomír Sedlář
85bf5535bc init: Keep parent groups in addon comps environments
The environment in comps for a variant can refer to groups in parent
variant (either for addons, or because of other configuration). We
should not remove the groups in this case.

This requires changes in two places:
 * teaching `comps_filter` about groups that should not be removed
 * fixing writing comps so that it does not actually change the data as
   well

JIRA: COMPOSE-2612
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-11-02 12:43:56 +01:00
Lubomír Sedlář
b2f995d516 Add a phase for creating extra ISOs
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-05-16 12:20:01 +02:00
Lubomír Sedlář
a217eea24e Allow filtering comps for different variants
We already filter for arches, so this just extends the filter to work
with variants as well.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-04-13 14:56:06 +02:00
Lubomír Sedlář
364d7f5229 Support multiple sources in one variant
With this patch the gather_source option is no longer used. Instead, all
sources are always used. If they return at least some input packages,
then a configured method is used and the returned lists of packages from
all sources are merged.

The method used for gathering can be configured for each variant and
gather source separately.

Additional packages are only added to the comps source.

Each gathering step is logged separately. All the logs are preserved for
later inspection.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2018-02-01 14:31:09 +01:00
Lubomír Sedlář
e1eacd456e gather: Add debugsource package to tests
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-07-27 08:31:39 +02:00
Daniel Mach
ad23efd323 Refactor depsolving tests
This adds a new test for Requires(pre) and (post). The general structure
of the test now makes it easy to use the same test scenarios for
different backend.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2017-02-13 12:32:07 +01:00
Lubomír Sedlář
deb8623bd1 pungi: Include noarch debuginfo
The check for arch compatibility should not be performed if the
debuginfo package is noarch. Such packages should be included
unconditionally.

Fixes: #450
Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-16 14:50:03 +01:00
Lubomír Sedlář
8e5b197e3f pungi: Add tests for depsolving
These are tests written by Daniel Mach originally for Distill-NG. They
are ported to current Pungi. The test repositories are committed in the
tests/fixtures/ directory. This is the same data that is used for test
compose, but the actual RPM files are not present. Some tests are
adapted from dmach's fork of Pungi.

Some of the packages are marked with a comment saying they are
important. These are the packages that the test is specifically trying
to get included in the package set. There are also explicit tests for
packages that should not be included.

Two tests are skipped for now as there is a bug preventing them from
passing. This is related to fulltree being done for packages that are
explicitly multilib.

The depsolver is called by invoking a separate executable, so the
coverage data is wrong.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-11-09 09:36:23 +01:00
Lubomír Sedlář
b46af7acbb createiso: Include layered product name in iso name
When creating an ISO for a layered product, the name of the integrated
product should be included in the file name.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-10-17 15:15:50 +02:00
Lubomír Sedlář
a31be0a3c4 Fix test data build script
It uses bash specific features, so it should not claim /bin/sh in
shebang. We also want to use `set -e` to catch possible errors instead
of claiming success every time.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-09-01 08:46:23 +02:00
Lubomír Sedlář
11bbbae2ed Treat variants without comps groups as having all of them
If the `<groups></groups>` section is not specified in the variants XML
file, all groups will be used in this variant. The section must be
omitted completely, not just empty. This is (and was) correct according
to the DTD, it just lead to crash before.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-06-28 09:01:30 +02:00
Daniel Mach
f0aecf6744 Add support for top-level variant IDs with dashes.
This is for layered products that have a variant mapped
to multiple variants in a base product, for example:
 * Foo-Tools (id: FooTools, uid: Foo-Tools, name: Tools)
 * Bar-Tools (id: BarTools, uid: Bar-Tools, name: Tools)

Requires productmd >= 1.2.

Signed-off-by: Daniel Mach <dmach@redhat.com>
2016-06-02 10:16:16 -04:00
Lubomír Sedlář
337d2a999c [checks] Don't always require jigdo
If the configuration specifically requests no jigdos, there is no point
in checking for the binary existence.

This is not 100% reliable. The jigdo option defaults to True, so if the
option is not specified the binary is required even if there are no
images configured.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-03-10 09:08:19 +01:00
Lubomír Sedlář
454363fba8 Allow specifying empty variants
The variants.xml file can list a variant with is_empty="true" and no
groups. If such variant is found, not package gathering will be run for
it, and no repos will be created.

This only makes sense for a variant that will have some other
deliverables like live media or images.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-02-16 15:08:15 +01:00
Lubomír Sedlář
8b4ad1ea02 Toggle multilib per variant
There is now a single option `multilib`, that maps variants and arches
to multilib methods. This replaces old `multilib_methods` option.

Multilib arches are implicitly deduced instead of using the
`multilib_arches` option.

The test compose is updated to only enable multilib on Server and its
addons.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2016-01-20 12:53:08 +01:00
Lubomír Sedlář
539736a11e Use lowercase hashed directories
On case-insensitive filesystems it is not such a good idea to have
directories that only differ in case. Packages should be always split
into lowercased directories.

The test data is modified to include some packages starting with
uppercase letters. The example in code can be verified by running
`nosetests --with-doctest`.

Signed-off-by: Lubomír Sedlář <lsedlar@redhat.com>
2015-12-03 09:08:15 +01:00
Daniel Mach
0d8ad9a111 gather: Implement hashed directories.
Set 'hashed_directories = True' config option to enable the feature.
2015-07-24 04:40:55 -04:00
Daniel Mach
1f313b39ad createiso: Add createiso_skip options to skip createiso on any variant/arch. 2015-07-23 10:58:35 -04:00
Daniel Mach
72302bd98e Rename product_* to release_*.
CHANGE: Rename product_* config options to release_* for consistency with productmd.
ACTION: Rename product_name, product_short, product_version, product_is_layered to release_* in config files.
        Rename //variant/product to //variant/release in variants XML.
2015-07-09 06:58:30 -04:00
Daniel Mach
6286e87d3b Config option create_optional_isos now defaults to False.
CHANGE: create_optional_isos config option now defaults to False.
ACTION: You can safely remove 'create_optional_isos = False' from config files.
2015-07-08 09:00:30 -04:00
Daniel Mach
90918dc34d Change createrepo config options defaults.
CHANGE: createrepo_c is config option now defaults to True.
ACTION: You can safely remove 'createrepo_c = True' from config files.
        Set 'createrepo_c = False' if you need legacy createrepo.

CHANGE: createrepo_checksum config option is now mandatory.
ACTION: Add 'createrepo_checksum = "sha256"' (or "sha") to config files.
2015-07-08 08:57:50 -04:00
Daniel Mach
81e935fe0e Fix test data, improve Makefile.
Previous test data was insufficient for proper testing.
Test compose runs and depsolving tests require precisely
set NVRs, dependencies, sub-packages, etc.
Using rpmfluff for these would be an overkill, it's better
to create RPMs directly from specs.
2015-06-25 08:02:57 -04:00