The time to run on tests/* data decreased from 34.517s to 30.201s.
Do not allow combining of any glob meta characters, either in first
or second line.
The code is still buggy, it may attempt to combine
modalias(abc[def]123)
modalias(abc[dXf]123)
into
modalias(abc[d[eX]f]123)
(presumably there won't be any such cases in real modules, but still).
Eliminate "| head -n1" - do it with bash string ops.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
On my machine, out of 2604 modules, this reduces number of
"provides" items by ~800 (to 12689 items).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
This creates a subshell: ... && ( [ -z "$pos" ] || [ $n = $pos ] )
For example, processing ath9k.ko needed ~80 forks here.
Rather expensive.
This should do the same: ... && [ -z "$pos" -o "$n" = "$pos" ]
While at it, explain algorithm in comments.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
While (in this case) their effect is the same,
many more people know that "local" makes variables function-local,
than that "declare" in bash does the same. I didn't know.
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
basename(): return one value, not two (principle of least confusion).
strip_compress_sfx(): ditto
strip_compress_sfx(): fix it, it had several problems:
(a) incorrect escaping of . (period) ... masked by incorrect use of backslash
(b) lua has no | "or" pattern
Use simpler gsub patterns / replacements, presumably faster (did not measure).
Signed-off-by: Denys Vlasenko <dvlasenk@redhat.com>
Starting with RHEL 9.2 and Fedora 35, debuginfod is enabled by default,
which leads to significant startup times of various tools
from the elfutils package (namely, readelf and objdump).
Unset DEBUGINFOD_URLS to avoid it.
* find-provides.ksyms (DEBUGINFOD_URLS): Unset.
* find-requires.ksyms (DEBUGINFOD_URLS): Ditto.
* kernel-srpm-macros.spec (Release): Bump to 18.
(%changelog): Add a record.
Suggested-by: Panu Matilainen <pmatilai@redhat.com>
Co-Authored-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Linux commit v5.19-rc1~139^2~2 ("kbuild: link symbol CRCs at final link,
removing CONFIG_MODULE_REL_CRCS") has broken the assumption aobut
the placement of non-absolute CRC symbols in .rodata (and also affects
the architectures that used to have these symbols as absolute); rewrite
the parsing by utilising "objdump -t" output to figure out
the section(s) where the __crc_* symbols are stored and process
it instead of the hard-coded ".rodata" section. The change also speeds
up the processing a bit, around 33% on synthetic tests:
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" | find-provides.ksyms.old > /dev/null 2> /dev/null'
sh -c 10.36s user 5.58s system 137% cpu 11.613 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" | find-provides.ksyms.new > /dev/null 2> /dev/null'
sh -c 7.82s user 4.59s system 142% cpu 8.686 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.old" \; > /dev/null 2> /dev/null'
sh -c 11.85s user 6.76s system 129% cpu 14.318 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.new" \; > /dev/null 2> /dev/null'
sh -c 8.91s user 5.51s system 135% cpu 10.647 total
* find-provides.ksyms: Process "objdump -t" output to get the list
of sections where __crc_* symbol contents are placed; retrieve
each one with "readelf -R" and supply it to an awk script that cuts
the required part of it for each __crc_* symbol in that section.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
Commit binutils-2_33~1385[1] has changed (and binutils-2_35~1768[2]
has not reverted) the calculated type for symbols in read-write .rodata section
from 'R' to 'D' on architectures where CONFIG_MODULE_REL_CRCS is enabled
(namely, ppc64* and s390*), since, apparently, many kernel modules have it indeed
read-write. Work around it by matching both D and R as possible __crc_* symbol
type.
[1] https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=a288c270991d
[2] https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=49d9fd42acef
* find-provides.ksyms: Match D as possible __crc_* symbol type in
addition to R.
* find-requires.ksyms: Likewise.
* kernel-srpm-macros.spec (Release): Bump to 11%{?dist}.
(%changelog): Mention the change.
Resolves: #2055464
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Avoid running sed on the whole "nm" output and also avoid "sed | awk"
pipe in favor of a single awk call. Overall, this gives around 20%
speedup on some quick synthetic tests:
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" | find-provides.ksyms.old > /dev/null 2> /dev/null'
sh -c 14.20s user 8.93s system 144% cpu 16.014 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" | find-provides.ksyms.new > /dev/null 2> /dev/null'
sh -c 12.01s user 7.46s system 143% cpu 13.567 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.old" \; > /dev/null 2> /dev/null'
sh -c 16.31s user 10.77s system 134% cpu 20.092 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.new" \; > /dev/null 2> /dev/null'
sh -c 13.95s user 8.92s system 135% cpu 16.836 total
* find-provides.ksyms: Check presence of absolute __crc_* symbols with
"grep -q" exit code and not presence of sed output; rewrite awk script to match
the __crc_* symbols instead of preprocessing the nm output with sed.
* find-requires.ksyms: Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
This makes the sort order a bit more predictable across various
environments and also speeds up grep/sed/sort a bit as they no longer
need to deal with non-trivial collations and multibyte sequences
that may appear when some garbage is being parsed accidentally.
* find-provides.ksyms: Add "export LC_ALL=C", remove "LC_ALL=C" from the
specific calls.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
Add conflicts of kernel-srpm-macros with kernel-rpm-macros < 185-9
as macros.kmp, kmodtool, and rpmsort were moved from the latter
to the former.
* kernel-srpm-macros.spec (Conflicts): Add kernel-rpm-macros < 185-9.
(%changelog): New entry.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* rpmsort: Chnge the FSF address from "59 Temple Place, Suite 330,
Boston, MA 02111-1307 USA" to "51 Franklin Street, Fifth Floor,
Boston, MA 02110-1301 USA".
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* modalias.prov: Replace everyithing that is not "0-9a-zA-Z.:"
in a kernel module version with underscores; replace everything
that is not "][0-9a-zA-Z.:*_?/-" in a kernel module alias with
underscores.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* modalias.prov: Chnge interpreter from "/bin/sh" to "/bin/bash" because
it uses bashisms; add "-efu" flags to keep things in check; initialise
is_kernel_package variable; check for presence of /sbin/modinfo, sed,
and sort.
(combine_modaliases): Initialise the "variants" and "pos" variables.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
After split of the kmods into a separate packages (like kernel-modules
and kernel-modules-extra), kernel() provides for the inbox kmods are generated
for incorrect package, as they were handled by kabi.sh that uses symvers
as a basis for the dependency list. Stop using it for kmod dependencies
(but continue using it for the symbols provided by vmlinux itself) and
employ find-provides.sh for that purpose.
* kabi.sh: Filter only those symbols that are exported by vmlinux.
* find-provides.ksyms: Generate tags with the "kernel" prefix for kernel
modules inside /lib/modules/[1-9][^/]*/kernel.
Resolves: #1942563Resolves: #1975927Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
redhat-rpm-macros has broken kmod SRPM generation when stopped providing
in the base buildroot by moving macros.kmp to kernel-rpm-macros
and not depending on the latter due to superficial reasons. Since
it depends on kernel-srpm-macros now, we can fix that breakage.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
It leads to some regressions in requires generation precision, though,
as it is no longer possible to exclude requires that are satisfied
by other kmods in the package. And calling a script on each file (instead
of calling it on the whole file list at once) does not make things faster,
either (so much for "sanity and benefit").
* find-provides.ksyms: Check for "$@" as well.
* find-requires.ksyms: Likewise.
* kernel-srpm-macros.spec (Source104, Source105, Source106): New
attribute files.
(%install): Install provided_ksyms.attr, required_ksyms.attr,
and modalias.attr into "%{buildroot}%{_fileattrsdir}".
(%files -n kernel-rpm-macros): Add provided_ksyms.attr,
required_ksyms.attr, and modalias.attr.
kmodtool: Remove "%global _use_internal_dependency_generator 0".
macros.kmp: Remove %__find_provides and %__find_requires.
* modalias.attr: New file.
* provided_ksyms.attr: Likewise.
* required_ksyms.attr: Likewise.
Resolves: #1942072Resolves: #1942563
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
If a kernel module are compressed by a post-install hook, then its name
would differ from the one saved by brp-kmod-set-exec-bit; try to
recover from this by guessing a possible name of the compressed kmod and
using it instead.
* brp-kmod-restore-perms (while read perm path): Check for
"$RPM_BUILD_ROOT/$path.gz", "$RPM_BUILD_ROOT/$path.bz2",
and "$RPM_BUILD_ROOT/$path.xz" if "$RPM_BUILD_ROOT/$path"
does not exist, add the respective suffix to the $path.
Resolves: #1942537
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* modalias.prov: Add "(\.gz|\.bz2|\.xz)?" to the end of the module path
matching regular expression, use "$@" instead of $*.
Resolves: #1942537Resolves: #2024117
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
The enablement of debuginfo generation broke custom %install scripts
that perform module signing, and in an attempt to address that,
additional hooks are added that can be used for module compression,
signing, and other means of kmod processing.
* macros.kmp (redhat_kmp_has_post_hooks): New macro, set to 1.
(kernel_module_package_release): Add __brp_kmod_pre_sign_process,
__brp_kmod_sign, __brp_kmod_post_sign_process, __brp_kmod_compress,
and __brp_kmod_post_compress_process to the generated
__spec_install_post.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* kmodtool (myver): Change revision to "rhel9", may be useful
for external users.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
This hack temporary adds executable permission for the *.ko files
during post-install stage so they are picked up by find-debuginfo.sh.
Since the intention is to avoid changing of behaviour in non-kmod
workloads, it's done by adding two additional brp-* scripts to the
__spec_install_post macro when kernel_module_package is instantiated.
* brp-kmod-restore-perms: New file.
* brp-kmod-set-exec-bit: Likewise.
* macros.kmp (%__brp_kmod_set_exec_bit, %__brp_kmod_restore_perms,
%__kmod_brps_added): New macros.
(%kernel_module_package): Rewrite __spec_install_post macro.
* redhat-rpm-config.spec (Source701, Source702): Add
brp-kmod-restore-perms and brp-kmod-set-exec-bit.
(Requires): Add find requirement for brp-kmod-set-exec-bit.
(%files): Explicitly list brp-* scripts for redhat-rpm-config and
kernel-rpm-macros packages.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
If no kernel-devel packages are installed (this is possible, for example,
during SRPM generation in an isolated environment), no output would be
generated for latest_kernel macro, which would lead to an error (since
an RPM macro can't have an empty definition). Append %nil to the input
of the "head" command to avoid this situation.
* macros.kmp (kernel_module_package): Feed additional line with %nil
to the input of head command in the latest_kernel macro definition.
Resolves: #1971748Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Providing "kernel-modules" with "${verrel}${dotvariant}" looks strange,
as it differs from the package and version that are actualy used for
flavored kernels.
* kmodtool (get_rpmtemplate): Add "kernel${dashvariant}-modules >= ${verrel}"
Provides to the generated spec.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
A patch[1] to the kernel spec file changed the separator that is used
for kernel flavor designation from "." to "+", which broke kmodtool and
%kernel_module_package in multiple places. Try to accomodate it by
accepting both "+" and "." during parsing, and guessing which one should
be used for kernel osurce path generation.
PS. It looks like the analogous breakage was before, when it was changed
from "-" to ".". May be we should start support it back again as well.
[1] https://lists.fedoraproject.org/pipermail/kernel/2013-June/004262.html
* kmodtool (get_verrel, get_variant): Accept both "." and "+" as separators.
(get_variant_char, print_variant_char, print_kernel_source): New functions.
(get_filelist): Convert "+%1" in file list to $dotvariant, in addition
to ".%1" and "-%1".
(get_rpmtemplate): Determine a char that is used as a variant separator.
Use it in $dotvariant.
(usage): Mention "veriant_char" and "kernel_source" commands.
* macros.kmp (%kernel_module_package): Use "kernel_source" command in
"%kernel_source" definition when non-default variant is used; fall back
to ls-based heuristics when it is not available (for example, when a
version of kmodtool that doesn't support this command is used).
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
* kmodtool: Put $@ instances into quotes as otherwise arguments
will undergo word splitting.
Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
