= filtermods.py(1) :ext: adoc :doctype: manpage == NAME filtermods.py - filter kernel modules == DESCRIPTION Filter/sort/allocate kmods into groups for packaging. Main use case for the script is to run as part of kernel packaging scripts (e.g. kernel.spec) to decide where each kmod should go (modules, modules-extra, etc.) While the tool gives a user flexibility in how yaml config is organized, there are some best practices for configs used in kernel builds: * Make modules package the default * Write as few rules for modules package as possible * Keep the rule list alphabetical where possible *usage:* filtermods.py [-h] [-v] [-q] [-l LOG_FILENAME] {sort,rulemap,selftest,cmp2rpm} ... == Common command line options These options apply to all subcommands. * -h / --help : display help * -v / --verbose : increases stdout verbosity * -q / --quiet : lowers stdout verbosity * -l / --log-filename : name of the logfile which captures detailed logs (filtermods.log by default) == SUBCOMMANDS === sort $ filtermods.py sort [-h] -c CONFIG -d DEPMOD [-o OUTPUT] [-r VARIANTS] [-g] Using modules.dep and yaml config as input, sort kmods into groups, satisfy all rules and assign each kmod to a package, respect kmod dependency relations, respect module dependency relations. This means, that if you install any combination of module packages, all kmods dependencies have to be available as well - either directly come from the package of one of packages it depends on. ==== yaml config (-c option) The yaml config describes module packages, their dependencies and rules (or constraints) for kmods. * There are two main rule types: ** "wants" - rule is satisfied if kmod ends up in listed package (or any package it depends on - meaning it can move down in package hierarchy) ** "default" - specifies that all kmods which were not covered by other rules (implicitly or explicitely) become "wants" for listed package * Rules are evaluated in order they appear in yaml config. * Rules can be regexes to match multiple kmods. * If multiple rules apply to a kmod, first one wins. * It is not required to explicitely list all kmods or their dependencies. * Rules can propagate transitively to children or parents of a kmod. For example "kmod A" dependencies have to end up in a package, which is in dependency chain of "kmod A"'s package (see example below). * General rule format is: "kmod_regex: module_name" * Packages and rules can be conditional (see 'if_variant_in' attribute below) Rule examples: ---- - drivers/base/.*(kunit|test).*: modules-internal - drivers/net/amt.ko: modules-core - fs/.*test.*: modules-internal - net/appletalk/appletalk.*: modules-extra - default: modules ---- ==== depmod file (-d option) This is file that describes kmod dependencies. It follows the format of modules.dep (created by depmod), which you can find in /lib/modules/$(uname -r)/ directory. ==== output directory (-o option) Directory where filtermods.py should store resulting package lists. Filenames are derived from package names defined in yaml config. ==== variants (-r option) This option conditionally enables parts of config which are specific to provided variants. A good example is -rt variant, which is 99% identical to base kernel, with the exception of kvm subpackage. It's an alternative to creating entirely new yaml config that would be 99% identical with base one. Sample usage can be seen in `filtermods-testdata/test1.yaml`. ==== graphviz (-g option) For experiments and debugging it's handy to vizualize what's going on. If you have graphviz installed, -g option will generate svg files for initial and final states. These contain graphs representing kmod and package relations. ==== Example 1 Let's take a look at first example with just 3 kmods. .example1.dep [source,txt] ---- kernel/drivers/input/joystick/iforce/iforce.ko.xz: kernel/drivers/input/joystick/iforce/iforce-serio.ko.xz: kernel/drivers/input/joystick/iforce/iforce.ko.xz kernel/drivers/input/joystick/iforce/iforce-usb.ko.xz: kernel/drivers/input/joystick/iforce/iforce.ko.xz ---- .example1.yaml [source,yaml] ---- packages: - name: modules-core depends-on: [] - name: modules depends-on: - modules-core - name: modules-extra depends-on: - modules-core - modules rules: - drivers/input/joystick/iforce/iforce.ko: modules-extra - default: modules ---- In yaml config above we are saying 2 things: 1. we want iforce.ko to be in modules-extra 2. and any unassigned kmods to go to modules package (default bucket) ---- $ ./redhat/scripts/filtermods.py sort -c example1.yaml -d example1.dep -o . ... 06:10:44 INFO write_modules_lists: 772 Module list ./modules-core.list created with 0 kmods 06:10:44 INFO write_modules_lists: 772 Module list ./modules.list created with 0 kmods 06:10:44 INFO write_modules_lists: 772 Module list ./modules-extra.list created with 3 kmods ---- Not surprisingly all kmods ended up in modules-extra package. That's because iforce-serio.ko and iforce-usb.ko depend on iforce.ko. If they were in modules-core or modules package, and modules-extra were not installed they would be missing a dependency (iforce.ko). Now consider, we modify rules to: .example1.yaml [source,yaml] ---- ... rules: - drivers/input/joystick/iforce/iforce.ko: modules-extra - drivers/input/joystick/iforce/iforce-usb.ko: modules - default: modules ---- This seems impossible at first look, but remember that these are "soft rules". That means that kmod can end up in specified package *or* any package it depends on. In other words, if those packages are installed and kmod is available, that is considered as satisfying rules as well. ---- $ ./redhat/scripts/filtermods.py sort -c example1.yaml -d example1.dep -o . ... 06:14:18 INFO print_report: 709 ************************** REPORT ************************** 06:14:18 INFO print_report: 745 iforce.ko: wanted by ['modules-extra'] but ended up in ['modules'] 06:14:18 INFO print_report: 747 has conflicting parent: iforce-serio.ko(modules), iforce-usb.ko(modules) 06:14:18 INFO print_report: 753 No. of kmod(s) assigned to preferred package: 2 06:14:18 INFO print_report: 754 No. of kmod(s) moved to a related package: 1 06:14:18 INFO print_report: 755 No. of kmod(s) which could not be assigned: 0 06:14:18 INFO print_report: 756 ************************************************************ 06:14:18 INFO write_modules_lists: 772 Module list ./modules-core.list created with 0 kmods 06:14:18 INFO write_modules_lists: 772 Module list ./modules.list created with 3 kmods 06:14:18 INFO write_modules_lists: 772 Module list ./modules-extra.list created with 0 kmods ---- What happened? We asked iforce-usb.ko to be in modules, but that would lead to broken dependency with iforce.ko (in modules-extra). So the tool does the next best thing, it moves iforce.ko to modules, and all kmods end up in modules package. This move, to a "related" package is allowed for "soft rules". ==== More examples Have a look at filtermods-testdata directory for more examples. You can also run all self tests with -g option: ---- $ filtermods.py selftest -g ---- and then inspect generated test*_f.svg files to easily see what was input and what the tool decided to do. === rulemap $ filtermods.py rulemap [-h] -c CONFIG -d DEPMOD [-r VARIANTS] Expand all rules and for each kmod print its desired module package name. With complex yaml config rules it may be handy to double check that a specific kmod is covered by correct rule. This doesn't do any sorting, it only prints yaml config rules in expanded form. Since a kmod can be covered by multiple rules and packages, this output is "what rule/package won for each kmod". ---- $ ./redhat/scripts/filtermods.py rulemap -c redhat/fedora_files/def_variants.yaml.fedora -d ~/tmp/modules.dep | grep 'kernel/drivers/block/' modules-core kernel/drivers/block/aoe/aoe.ko.xz modules-core kernel/drivers/block/brd.ko.xz modules-core kernel/drivers/block/drbd/drbd.ko.xz modules-extra kernel/drivers/block/floppy.ko.xz modules-core kernel/drivers/block/loop.ko.xz modules-core kernel/drivers/block/mtip32xx/mtip32xx.ko.xz modules-core kernel/drivers/block/nbd.ko.xz modules-core kernel/drivers/block/null_blk/null_blk.ko.xz modules-core kernel/drivers/block/pktcdvd.ko.xz modules-core kernel/drivers/block/rbd.ko.xz modules kernel/drivers/block/rnbd/rnbd-client.ko.xz modules kernel/drivers/block/rnbd/rnbd-server.ko.xz modules-core kernel/drivers/block/ublk_drv.ko.xz modules-core kernel/drivers/block/virtio_blk.ko.xz modules-core kernel/drivers/block/xen-blkback/xen-blkback.ko.xz modules-core kernel/drivers/block/xen-blkfront.ko.xz modules-core kernel/drivers/block/zram/zram.ko.xz ---- === cmp2rpm $ filtermods.py cmp2rpm [-h] -c CONFIG -k KMOD_RPMS Compare yaml config rules with kmod RPMs. This is a check that helps to review how yaml config rules deviate from existing RPMs. It unpacks supplied RPMs and for each kmod it compares where config would like this kmod to end up, with the package name where it's present in existing RPMs. This is useful when creating a new config from scratch and you want to see how is the config different from existing RPMs. The alternative is to go through the build, and then compare old and new RPMs. ---- $ mkdir ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136 $ cd ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136 $ koji download-build kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136 $ cd - $ ./redhat/scripts/filtermods.py cmp2rpm -c redhat/rhel_files/def_variants.yaml.rhel -k "$(ls -1 ~/tmp/kernel-6.8.0-0.rc6.20240227git45ec2f5f6ed3.50.eln136/*modules*.rpm)" ... 09:34:06 WARNIN do_rpm_mapping_test: 914 kmod kernel/lib/percpu_test.ko.xz wanted by config in ['modules-internal'], in tree it is: ['modules-core'] ... ---- === selftest $ filtermods.py selftest [-h] [-g] Run selftests using data from filtermods-testdata directory. == How does it work? It is inspired by "label propagation algorithm". Each kmod keeps a track of plausible packages that won't break the rules. Initially only kmods mentioned in yaml config have these set. Then it iterates over all kmods and traverses kmod's children and parents and tries to refine set of plausible packages by removing ones that would break any of the rules. .There are 3 phases: 1. Apply initial labels based on yaml config. 2. If some kmods satisfy rules for more than one package, pick the preferred one specified by config "wants" rules. 3. If some kmods still satisfy rules for more than one package, prefer one from default rule. For all remaining (not yet assigned) kmods, try to use default rule. == AUTHOR Jan Stancek