264 lines
11 KiB
Plaintext
264 lines
11 KiB
Plaintext
= filtermods.py(1)
|
|
:ext: adoc
|
|
:doctype: manpage
|
|
|
|
== NAME
|
|
filtermods.py - filter kernel modules
|
|
|
|
== DESCRIPTION
|
|
Filter/sort/allocate kmods into groups for packaging. Main use case for
|
|
the script is to run as part of kernel packaging scripts (e.g. kernel.spec)
|
|
to decide where each kmod should go (modules, modules-extra, etc.)
|
|
|
|
While the tool gives a user flexibility in how yaml config is organized,
|
|
there are some best practices for configs used in kernel builds:
|
|
|
|
* Make modules package the default
|
|
* Write as few rules for modules package as possible
|
|
* Keep the rule list alphabetical where possible
|
|
|
|
*usage:* filtermods.py [-h] [-v] [-q] [-l LOG_FILENAME] {sort,rulemap,selftest,cmp2rpm} ...
|
|
|
|
== Common command line options
|
|
These options apply to all subcommands.
|
|
|
|
* -h / --help : display help
|
|
* -v / --verbose : increases stdout verbosity
|
|
* -q / --quiet : lowers stdout verbosity
|
|
* -l / --log-filename : name of the logfile which captures detailed logs (filtermods.log by default)
|
|
|
|
== SUBCOMMANDS
|
|
=== sort
|
|
$ filtermods.py sort [-h] -c CONFIG -d DEPMOD [-o OUTPUT] [-r VARIANTS] [-g]
|
|
|
|
Using modules.dep and yaml config as input, sort kmods into groups,
|
|
satisfy all rules and assign each kmod to a package, respect kmod
|
|
dependency relations, respect module dependency relations.
|
|
|
|
This means, that if you install any combination of module packages,
|
|
all kmods dependencies have to be available as well - either directly
|
|
come from the package of one of packages it depends on.
|
|
|
|
==== yaml config (-c option)
|
|
The yaml config describes module packages, their dependencies and rules
|
|
(or constraints) for kmods.
|
|
|
|
* There are two main rule types:
|
|
** "wants" - rule is satisfied if kmod ends up in listed package
|
|
(or any package it depends on - meaning it can move
|
|
down in package hierarchy)
|
|
** "default" - specifies that all kmods which were not covered
|
|
by other rules (implicitly or explicitely) become
|
|
"wants" for listed package
|
|
* Rules are evaluated in order they appear in yaml config.
|
|
* Rules can be regexes to match multiple kmods.
|
|
* If multiple rules apply to a kmod, first one wins.
|
|
* It is not required to explicitely list all kmods or their dependencies.
|
|
* Rules can propagate transitively to children or parents of a kmod.
|
|
For example "kmod A" dependencies have to end up in a package, which is
|
|
in dependency chain of "kmod A"'s package (see example below).
|
|
* General rule format is: "kmod_regex: module_name"
|
|
* Packages and rules can be conditional (see 'if_variant_in' attribute below)
|
|
|
|
Rule examples:
|
|
|
|
----
|
|
- drivers/base/.*(kunit|test).*: modules-internal
|
|
- drivers/net/amt.ko: modules-core
|
|
- fs/.*test.*: modules-internal
|
|
- net/appletalk/appletalk.*: modules-extra
|
|
- default: modules
|
|
----
|
|
|
|
==== depmod file (-d option)
|
|
This is file that describes kmod dependencies. It follows the format
|
|
of modules.dep (created by depmod), which you can find in
|
|
/lib/modules/$(uname -r)/ directory.
|
|
|
|
==== output directory (-o option)
|
|
Directory where filtermods.py should store resulting package lists.
|
|
Filenames are derived from package names defined in yaml config.
|
|
|
|
==== variants (-r option)
|
|
This option conditionally enables parts of config which are specific
|
|
to provided variants. A good example is -rt variant, which is 99% identical
|
|
to base kernel, with the exception of kvm subpackage. It's an alternative
|
|
to creating entirely new yaml config that would be 99% identical with base one.
|
|
Sample usage can be seen in `filtermods-testdata/test1.yaml`.
|
|
|
|
==== graphviz (-g option)
|
|
For experiments and debugging it's handy to vizualize what's going on.
|
|
If you have graphviz installed, -g option will generate svg files
|
|
for initial and final states. These contain graphs representing
|
|
kmod and package relations.
|
|
|
|
==== Example 1
|
|
Let's take a look at first example with just 3 kmods.
|
|
|
|
.example1.dep
|
|
[source,txt]
|
|
----
|
|
kernel/drivers/input/joystick/iforce/iforce.ko.xz:
|
|
kernel/drivers/input/joystick/iforce/iforce-serio.ko.xz: kernel/drivers/input/joystick/iforce/iforce.ko.xz
|
|
kernel/drivers/input/joystick/iforce/iforce-usb.ko.xz: kernel/drivers/input/joystick/iforce/iforce.ko.xz
|
|
----
|
|
|
|
.example1.yaml
|
|
[source,yaml]
|
|
----
|
|
packages:
|
|
- name: modules-core
|
|
depends-on: []
|
|
- name: modules
|
|
depends-on:
|
|
- modules-core
|
|
- name: modules-extra
|
|
depends-on:
|
|
- modules-core
|
|
- modules
|
|
|
|
rules:
|
|
- drivers/input/joystick/iforce/iforce.ko: modules-extra
|
|
- default: modules
|
|
----
|
|
|
|
In yaml config above we are saying 2 things:
|
|
1. we want iforce.ko to be in modules-extra
|
|
2. and any unassigned kmods to go to modules package (default bucket)
|
|
|
|
----
|
|
$ ./redhat/scripts/filtermods.py sort -c example1.yaml -d example1.dep -o .
|
|
...
|
|
06:10:44 INFO write_modules_lists: 772 Module list ./modules-core.list created with 0 kmods
|
|
06:10:44 INFO write_modules_lists: 772 Module list ./modules.list created with 0 kmods
|
|
06:10:44 INFO write_modules_lists: 772 Module list ./modules-extra.list created with 3 kmods
|
|
----
|
|
Not surprisingly all kmods ended up in modules-extra package. That's because
|
|
iforce-serio.ko and iforce-usb.ko depend on iforce.ko. If they were in
|
|
modules-core or modules package, and modules-extra were not installed they
|
|
would be missing a dependency (iforce.ko).
|
|
|
|
Now consider, we modify rules to:
|
|
|
|
.example1.yaml
|
|
[source,yaml]
|
|
----
|
|
...
|
|
rules:
|
|
- drivers/input/joystick/iforce/iforce.ko: modules-extra
|
|
- drivers/input/joystick/iforce/iforce-usb.ko: modules
|
|
- default: modules
|
|
----
|
|
|
|
This seems impossible at first look, but remember that these are
|
|
"soft rules". That means that kmod can end up in specified package
|
|
*or* any package it depends on. In other words, if those packages
|
|
are installed and kmod is available, that is considered as satisfying
|
|
rules as well.
|
|
|
|
----
|
|
$ ./redhat/scripts/filtermods.py sort -c example1.yaml -d example1.dep -o .
|
|
...
|
|
06:14:18 INFO print_report: 709 ************************** REPORT **************************
|
|
06:14:18 INFO print_report: 745 iforce.ko: wanted by ['modules-extra'] but ended up in ['modules']
|
|
06:14:18 INFO print_report: 747 has conflicting parent: iforce-serio.ko(modules), iforce-usb.ko(modules)
|
|
06:14:18 INFO print_report: 753 No. of kmod(s) assigned to preferred package: 2
|
|
06:14:18 INFO print_report: 754 No. of kmod(s) moved to a related package: 1
|
|
06:14:18 INFO print_report: 755 No. of kmod(s) which could not be assigned: 0
|
|
06:14:18 INFO print_report: 756 ************************************************************
|
|
06:14:18 INFO write_modules_lists: 772 Module list ./modules-core.list created with 0 kmods
|
|
06:14:18 INFO write_modules_lists: 772 Module list ./modules.list created with 3 kmods
|
|
06:14:18 INFO write_modules_lists: 772 Module list ./modules-extra.list created with 0 kmods
|
|
----
|
|
What happened? We asked iforce-usb.ko to be in modules, but that would lead
|
|
to broken dependency with iforce.ko (in modules-extra). So the tool does the
|
|
next best thing, it moves iforce.ko to modules, and all kmods end up in modules
|
|
package. This move, to a "related" package is allowed for "soft rules".
|
|
|
|
==== More examples
|
|
Have a look at filtermods-testdata directory for more examples.
|
|
|
|
You can also run all self tests with -g option:
|
|
----
|
|
$ filtermods.py selftest -g
|
|
----
|
|
and then inspect generated test*_f.svg files to easily see what was input
|
|
and what the tool decided to do.
|
|
|
|
=== rulemap
|
|
$ filtermods.py rulemap [-h] -c CONFIG -d DEPMOD [-r VARIANTS]
|
|
|
|
Expand all rules and for each kmod print its desired module package name.
|
|
With complex yaml config rules it may be handy to double check that a specific
|
|
kmod is covered by correct rule. This doesn't do any sorting, it only prints
|
|
yaml config rules in expanded form. Since a kmod can be covered by multiple
|
|
rules and packages, this output is "what rule/package won for each kmod".
|
|
|
|
----
|
|
$ ./redhat/scripts/filtermods.py rulemap -c redhat/fedora_files/def_variants.yaml.fedora -d ~/tmp/modules.dep | grep 'kernel/drivers/block/'
|
|
modules-core kernel/drivers/block/aoe/aoe.ko.xz
|
|
modules-core kernel/drivers/block/brd.ko.xz
|
|
modules-core kernel/drivers/block/drbd/drbd.ko.xz
|
|
modules-extra kernel/drivers/block/floppy.ko.xz
|
|
modules-core kernel/drivers/block/loop.ko.xz
|
|
modules-core kernel/drivers/block/mtip32xx/mtip32xx.ko.xz
|
|
modules-core kernel/drivers/block/nbd.ko.xz
|
|
modules-core kernel/drivers/block/null_blk/null_blk.ko.xz
|
|
modules-core kernel/drivers/block/pktcdvd.ko.xz
|
|
modules-core kernel/drivers/block/rbd.ko.xz
|
|
modules kernel/drivers/block/rnbd/rnbd-client.ko.xz
|
|
modules kernel/drivers/block/rnbd/rnbd-server.ko.xz
|
|
modules-core kernel/drivers/block/ublk_drv.ko.xz
|
|
modules-core kernel/drivers/block/virtio_blk.ko.xz
|
|
modules-core kernel/drivers/block/xen-blkback/xen-blkback.ko.xz
|
|
modules-core kernel/drivers/block/xen-blkfront.ko.xz
|
|
modules-core kernel/drivers/block/zram/zram.ko.xz
|
|
----
|
|
|
|
|
|
=== cmp2rpm
|
|
$ filtermods.py cmp2rpm [-h] -c CONFIG -k KMOD_RPMS
|
|
|
|
Compare yaml config rules with kmod RPMs. This is a check that helps to review how
|
|
yaml config rules deviate from existing RPMs. It unpacks supplied RPMs and for
|
|
each kmod it compares where config would like this kmod to end up, with the
|
|
package name where it's present in existing RPMs. This is useful when creating
|
|
a new config from scratch and you want to see how is the config different from
|
|
existing RPMs. The alternative is to go through the build, and then compare old
|
|
and new RPMs.
|
|
|
|
----
|
|
$ mkdir ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136
|
|
$ cd ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136
|
|
$ koji download-build kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136
|
|
$ cd -
|
|
$ ./redhat/scripts/filtermods.py cmp2rpm -c redhat/rhel_files/def_variants.yaml.rhel -k "$(ls -1 ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136/*modules*.rpm)"
|
|
...
|
|
09:34:06 WARNIN do_rpm_mapping_test: 914 kmod kernel/lib/percpu_test.ko.xz wanted by config in ['modules-internal'], in tree it is: ['modules-core']
|
|
...
|
|
----
|
|
|
|
=== selftest
|
|
$ filtermods.py selftest [-h] [-g]
|
|
|
|
Run selftests using data from filtermods-testdata directory.
|
|
|
|
== How does it work?
|
|
It is inspired by "label propagation algorithm". Each kmod keeps a
|
|
track of plausible packages that won't break the rules. Initially
|
|
only kmods mentioned in yaml config have these set. Then it
|
|
iterates over all kmods and traverses kmod's children and parents
|
|
and tries to refine set of plausible packages by removing ones
|
|
that would break any of the rules.
|
|
|
|
.There are 3 phases:
|
|
1. Apply initial labels based on yaml config.
|
|
2. If some kmods satisfy rules for more than one package,
|
|
pick the preferred one specified by config "wants" rules.
|
|
3. If some kmods still satisfy rules for more than one package,
|
|
prefer one from default rule.
|
|
For all remaining (not yet assigned) kmods, try to use default rule.
|
|
|
|
== AUTHOR
|
|
Jan Stancek <jstancek@redhat.com>
|