Since the immediate symbols can be retrieved for "objdump -t" output
as well, it makes some sence to use it as the sole data source and
handle both vaiants inside the awk script. This simplification comes
at some runtime cost for the simpler case of the immediate symbols,
though, and overall it is about 5% faster in the relative symbols case
and about the same 5% slower in the absolute symbols case, with the
latter not as important moving forward, probably:
$ for i in ./lib/modules/*; do \
echo "====== $i ====="; \
diff -u <(find $i | ./find-provides.ksyms.old) <(find $i | ./find-provides.ksyms.new); \
echo -n "old: "; find $i | time ./find-provides.ksyms.old > /dev/null; \
echo -n "new: "; find $i | time ./find-provides.ksyms.new > /dev/null; \
done
====== ./lib/modules/4.18.0-372.57.1.el8_6.s390x =====
old: ./find-provides.ksyms.old > /dev/null 3.98s user 3.04s system 129% cpu 5.411 total
new: ./find-provides.ksyms.new > /dev/null 3.78s user 3.06s system 132% cpu 5.181 total
====== ./lib/modules/4.18.0-372.57.1.el8_6.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 6.18s user 4.00s system 124% cpu 8.161 total
new: ./find-provides.ksyms.new > /dev/null 6.57s user 4.84s system 132% cpu 8.644 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.s390x =====
old: ./find-provides.ksyms.old > /dev/null 4.70s user 2.94s system 126% cpu 6.061 total
new: ./find-provides.ksyms.new > /dev/null 4.37s user 3.03s system 127% cpu 5.793 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 6.66s user 4.35s system 123% cpu 8.884 total
new: ./find-provides.ksyms.new > /dev/null 7.07s user 5.00s system 130% cpu 9.218 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.s390x =====
old: ./find-provides.ksyms.old > /dev/null 3.81s user 2.62s system 128% cpu 5.018 total
new: ./find-provides.ksyms.new > /dev/null 3.55s user 2.56s system 128% cpu 4.743 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 13.79s user 8.59s system 125% cpu 17.817 total
new: ./find-provides.ksyms.new > /dev/null 13.18s user 8.78s system 127% cpu 17.247 total
* find-provides.ksyms: Do not perform an intial "nm | awk" run in an attempt
to capture absolute symbols, just parse "objdump -t" output and handle both
absolute and relative symbols in the awk script based on the section name.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Add a comment that, hopefully, sheds some light on what is going on in
the following code.
* find-provides.ksyms: Add a comment.
Co-Authored-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Since the section retrieval code is implemented inside the awk script
now, the next step is to eliminate external iteration over the section
list and just retrieve the required sections inside awk on demand. This
allows calling (and processing the output of) objdump -t just once,
which saves around 8-12% of time on kernels that store modversions
as non-absolute symbols:
$ for i in ./lib/modules/*; do \
echo "====== $i ====="; \
diff -u <(find $i | ./find-provides.ksyms.old) <(find $i | ./find-provides.ksyms.new); \
echo -n "old: "; find $i | time ./find-provides.ksyms.old > /dev/null; \
echo -n "new: "; find $i | time ./find-provides.ksyms.new > /dev/null; \
done
====== ./lib/modules/4.18.0-372.57.1.el8_6.s390x =====
old: ../find-provides.ksyms.old > /dev/null 4.41s user 3.46s system 136% cpu 5.756 total
new: ../find-provides.ksyms.new > /dev/null 3.90s user 2.94s system 129% cpu 5.279 total
====== ./lib/modules/4.18.0-372.57.1.el8_6.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 6.04s user 3.88s system 124% cpu 7.993 total
new: ../find-provides.ksyms.new > /dev/null 6.08s user 3.91s system 124% cpu 8.012 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.s390x =====
old: ../find-provides.ksyms.old > /dev/null 5.09s user 3.51s system 133% cpu 6.452 total
new: ../find-provides.ksyms.new > /dev/null 4.65s user 2.89s system 126% cpu 5.949 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 6.52s user 4.14s system 123% cpu 8.638 total
new: ../find-provides.ksyms.new > /dev/null 6.64s user 4.12s system 123% cpu 8.690 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.s390x =====
old: ../find-provides.ksyms.old > /dev/null 4.45s user 3.29s system 136% cpu 5.661 total
new: ../find-provides.ksyms.new > /dev/null 3.84s user 2.54s system 127% cpu 4.980 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 7.34s user 4.33s system 129% cpu 9.019 total
new: ../find-provides.ksyms.new > /dev/null 6.67s user 3.51s system 122% cpu 8.278 total
* find-provides.ksyms: Remove "objdump -t" section loop, do not supply
sectname to the awk script, treat sectdata as an array keyed on section
name, convert section retrieval to a function with a section name
as an argument, call it if a section name is not present in sectdata.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Instead of running the nm output through grep and then running it
through awk again, run it directly through awk, and bail of immediately
if a non-absolute symbol is found.
Also, while we're here, avoid executing the indirect parsing code
if no __crc_* symbols are present at all, which shaves around 8-18% off
of the run time:
% for i in ./lib/modules/*; do \
echo "====== $i ====="; \
diff -u <(find $i | ./find-provides.ksyms.old) <(find $i | ./find-provides.ksyms.new); \
echo old:; find $i | time ./find-provides.ksyms.old > /dev/null; \
echo new:; find $i | time ./find-provides.ksyms.new > /dev/null; \
done
====== ./lib/modules/4.18.0-372.57.1.el8_6.s390x =====
old: ../find-provides.ksyms.old > /dev/null 5.10s user 4.52s system 150% cpu 6.412 total
new: ../find-provides.ksyms.new > /dev/null 4.41s user 3.48s system 136% cpu 5.764 total
====== ./lib/modules/4.18.0-372.57.1.el8_6.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 7.80s user 6.19s system 143% cpu 9.740 total
new: ../find-provides.ksyms.new > /dev/null 6.07s user 3.97s system 124% cpu 8.066 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.s390x =====
old: ../find-provides.ksyms.old > /dev/null 5.87s user 4.35s system 143% cpu 7.113 total
new: ../find-provides.ksyms.new > /dev/null 5.12s user 3.56s system 132% cpu 6.533 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 8.44s user 6.78s system 144% cpu 10.535 total
new: ../find-provides.ksyms.new > /dev/null 6.55s user 4.17s system 123% cpu 8.666 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.s390x =====
old: ../find-provides.ksyms.old > /dev/null 5.04s user 3.95s system 145% cpu 6.166 total
new: ../find-provides.ksyms.new > /dev/null 4.64s user 3.17s system 136% cpu 5.719 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.x86_64 =====
old: ../find-provides.ksyms.old > /dev/null 8.49s user 5.54s system 139% cpu 10.043 total
new: ../find-provides.ksyms.new > /dev/null 7.41s user 4.31s system 129% cpu 9.046 total
* find-provides.ksyms: Rewrite the "if nm | grep -q; then nm | awk"
with straight "nm | awk", update the awk script to return the exit code
accordingly, execute indirect symbol parsing only if inderect __crc_*
symbols are present.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Previous approach with passing the section contents as a command line
option proven to be error-prone, as it hits the command line option size
limit on some kmods. Move the reading and processing of the section
contents inside the awk script itself; it also saves a bit (around 3—4%)
of time on the kernels with indirect __crc_* symbols:
$ for i in ./lib/modules/*; do \
echo "====== $i ====="; \
diff -u <(find $i | ./find-provides.ksyms.old) <(find $i | ./find-provides.ksyms.new); \
echo -n "old: "; find $i | time ./find-provides.ksyms.old > /dev/null; \
echo -n "new: "; find $i | time ./find-provides.ksyms.new > /dev/null; \
done
====== ./lib/modules/4.18.0-372.57.1.el8_6.s390x =====
old: ./find-provides.ksyms.old > /dev/null 5.38s user 4.66s system 150% cpu 6.693 total
new: ./find-provides.ksyms.new > /dev/null 5.20s user 4.52s system 149% cpu 6.484 total
====== ./lib/modules/4.18.0-372.57.1.el8_6.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 7.85s user 6.34s system 143% cpu 9.864 total
new: ./find-provides.ksyms.new > /dev/null 7.75s user 6.32s system 143% cpu 9.809 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.s390x =====
old: ./find-provides.ksyms.old > /dev/null 6.19s user 4.68s system 144% cpu 7.506 total
new: ./find-provides.ksyms.new > /dev/null 5.93s user 4.46s system 143% cpu 7.219 total
====== ./lib/modules/5.14.0-284.15.1.el9_2.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 8.47s user 6.71s system 144% cpu 10.523 total
new: ./find-provides.ksyms.new > /dev/null 8.44s user 6.59s system 144% cpu 10.435 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.s390x =====
old: ./find-provides.ksyms.old > /dev/null 5.21s user 4.09s system 145% cpu 6.383 total
new: ./find-provides.ksyms.new > /dev/null 5.04s user 3.85s system 145% cpu 6.120 total
====== ./lib/modules/6.4.0-0.rc1.20230511git80e62bc8487b.19.eln126.x86_64 =====
old: ./find-provides.ksyms.old > /dev/null 8.68s user 5.77s system 139% cpu 10.326 total
new: ./find-provides.ksyms.new > /dev/null 8.47s user 5.57s system 139% cpu 10.067 total
* find-provides.ksyms: Add assign to revbytes the result of endianness
chack test instead of the code snippet; do not generate SECTDATA
variable; do not pass SECTDATA to the awk script; pass module
and revbytes to the awk script; retrieve the section data
into the sectdata variable and perform the byte re-shuffling
in accordance to the revbytes value in the BEGIN section of the awk
script.
* find-requires.ksyms (all_provides): Likewise.
Fixes: cd7e9e8a2f "find-provides.ksyms, find-requires.ksyms: rewrite indirect CRC parsing"
Resolves: #2135901
References: https://bugzilla.redhat.com/show_bug.cgi?id=2178935
Co-Authored-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Currently, it just matches the 2nd to 5th fields with awk, which leads
to occasional capturing of some garbage at the end of the section data
and minor issues when supplying it to the symbol extracting awk script:
awk: cmd. line:1: (FILENAME=- FNR=19) fatal: argument index with `$' must be > 0
awk: warning: escape sequence `\.' treated as plain `.'
awk: cmd. line:1: (FILENAME=- FNR=9) fatal: not enough arguments to satisfy format string
`78b624dbc125415351be965ex.$..%ASQ..^'
^ ran out for this one
While at it, also eliminate grep and sed calls (with the pair of awk and
sed being the most offending).
* find-provides.ksyms: Generate part of the awk script that performs
the byte reversal based on the endianness in the readelf -h output,
inject it into the awk script; use substr for matching parts
of the section data instead of fields.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
To avoid all kinds of fun that might stem from interpreting the user-supplied
input.
* find-provides.ksyms: Supply -r option to the read call.
* find-requires.ksyms (all_provides): Likewise.
Fixes: cd7e9e8a2f "find-provides.ksyms, find-requires.ksyms: rewrite indirect CRC parsing"
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Instead of ignoring the section name in the objdump -t output, section
name has to be checked in the awk script to produce only the meaningful
output; hit when both __kcrctab and __kcrctab_gpl are present.
* find-requires.ksyms (all_provides): Pass $sectname to the awk script,
check that the relevant field of the input matches it.
* find-provides.ksyms: Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Apparently, these are local in the recent kernels.
* find-provides.ksyms (all_provides): Filter local __crc_* symbols
in additional to global in the objdump -t call output.
* find-requires.ksyms: Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Starting with RHEL 9.2 and Fedora 35, debuginfod is enabled by default,
which leads to significant startup times of various tools
from the elfutils package (namely, readelf and objdump).
Unset DEBUGINFOD_URLS to avoid it.
* find-provides.ksyms (DEBUGINFOD_URLS): Unset.
* find-requires.ksyms (DEBUGINFOD_URLS): Ditto.
* kernel-srpm-macros.spec (Release): Bump to 18.
(%changelog): Add a record.
Suggested-by: Panu Matilainen <pmatilai@redhat.com>
Co-Authored-by: Denys Vlasenko <dvlasenk@redhat.com>
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Linux commit v5.19-rc1~139^2~2 ("kbuild: link symbol CRCs at final link,
removing CONFIG_MODULE_REL_CRCS") has broken the assumption aobut
the placement of non-absolute CRC symbols in .rodata (and also affects
the architectures that used to have these symbols as absolute); rewrite
the parsing by utilising "objdump -t" output to figure out
the section(s) where the __crc_* symbols are stored and process
it instead of the hard-coded ".rodata" section. The change also speeds
up the processing a bit, around 33% on synthetic tests:
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" | find-provides.ksyms.old > /dev/null 2> /dev/null'
sh -c 10.36s user 5.58s system 137% cpu 11.613 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" | find-provides.ksyms.new > /dev/null 2> /dev/null'
sh -c 7.82s user 4.59s system 142% cpu 8.686 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.old" \; > /dev/null 2> /dev/null'
sh -c 11.85s user 6.76s system 129% cpu 14.318 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.ppc64le -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.new" \; > /dev/null 2> /dev/null'
sh -c 8.91s user 5.51s system 135% cpu 10.647 total
* find-provides.ksyms: Process "objdump -t" output to get the list
of sections where __crc_* symbol contents are placed; retrieve
each one with "readelf -R" and supply it to an awk script that cuts
the required part of it for each __crc_* symbol in that section.
* find-requires.ksyms (all_provides): Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
Commit binutils-2_33~1385[1] has changed (and binutils-2_35~1768[2]
has not reverted) the calculated type for symbols in read-write .rodata section
from 'R' to 'D' on architectures where CONFIG_MODULE_REL_CRCS is enabled
(namely, ppc64* and s390*), since, apparently, many kernel modules have it indeed
read-write. Work around it by matching both D and R as possible __crc_* symbol
type.
[1] https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=a288c270991d
[2] https://sourceware.org/git/?p=binutils-gdb.git;a=commitdiff;h=49d9fd42acef
* find-provides.ksyms: Match D as possible __crc_* symbol type in
addition to R.
* find-requires.ksyms: Likewise.
* kernel-srpm-macros.spec (Release): Bump to 11%{?dist}.
(%changelog): Mention the change.
Resolves: #2055464
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Avoid running sed on the whole "nm" output and also avoid "sed | awk"
pipe in favor of a single awk call. Overall, this gives around 20%
speedup on some quick synthetic tests:
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" | find-provides.ksyms.old > /dev/null 2> /dev/null'
sh -c 14.20s user 8.93s system 144% cpu 16.014 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" | find-provides.ksyms.new > /dev/null 2> /dev/null'
sh -c 12.01s user 7.46s system 143% cpu 13.567 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.old" \; > /dev/null 2> /dev/null'
sh -c 16.31s user 10.77s system 134% cpu 20.092 total
$ time sh -c 'find ./lib/modules/5.14.0-258.el9.x86_64 -name "*.ko.xz" -exec sh -c "echo {} | find-provides.ksyms.new" \; > /dev/null 2> /dev/null'
sh -c 13.95s user 8.92s system 135% cpu 16.836 total
* find-provides.ksyms: Check presence of absolute __crc_* symbols with
"grep -q" exit code and not presence of sed output; rewrite awk script to match
the __crc_* symbols instead of preprocessing the nm output with sed.
* find-requires.ksyms: Likewise.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
This makes the sort order a bit more predictable across various
environments and also speeds up grep/sed/sort a bit as they no longer
need to deal with non-trivial collations and multibyte sequences
that may appear when some garbage is being parsed accidentally.
* find-provides.ksyms: Add "export LC_ALL=C", remove "LC_ALL=C" from the
specific calls.
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
Resolves: #2135047
After split of the kmods into a separate packages (like kernel-modules
and kernel-modules-extra), kernel() provides for the inbox kmods are generated
for incorrect package, as they were handled by kabi.sh that uses symvers
as a basis for the dependency list. Stop using it for kmod dependencies
(but continue using it for the symbols provided by vmlinux itself) and
employ find-provides.sh for that purpose.
* kabi.sh: Filter only those symbols that are exported by vmlinux.
* find-provides.ksyms: Generate tags with the "kernel" prefix for kernel
modules inside /lib/modules/[1-9][^/]*/kernel.
Resolves: #1942563Resolves: #1975927Resolves: #2002887
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
It leads to some regressions in requires generation precision, though,
as it is no longer possible to exclude requires that are satisfied
by other kmods in the package. And calling a script on each file (instead
of calling it on the whole file list at once) does not make things faster,
either (so much for "sanity and benefit").
* find-provides.ksyms: Check for "$@" as well.
* find-requires.ksyms: Likewise.
* kernel-srpm-macros.spec (Source104, Source105, Source106): New
attribute files.
(%install): Install provided_ksyms.attr, required_ksyms.attr,
and modalias.attr into "%{buildroot}%{_fileattrsdir}".
(%files -n kernel-rpm-macros): Add provided_ksyms.attr,
required_ksyms.attr, and modalias.attr.
kmodtool: Remove "%global _use_internal_dependency_generator 0".
macros.kmp: Remove %__find_provides and %__find_requires.
* modalias.attr: New file.
* provided_ksyms.attr: Likewise.
* required_ksyms.attr: Likewise.
Resolves: #1942072Resolves: #1942563
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
- Add support for compressed kernel modules to find-provides.ksyms,
find-requires.ksyms, firmware.prov (#1622019)
* find-provides.ksyms: Try to process files whose names end with
something else than "*.ko" by guessing a decompressor and uncompressing
it in a temporary file and parsing it instead.
* find-requires.ksyms (all_provides): Likewise.
Add "(\.gz|\.bz2|\.xz)?" to the end of the module path matching regular
expression.
* firmware.prov: Add "(\.gz|\.bz2|\.xz)?" to the end of the module path
matching regular expression.
Resolves: #1942537
Signed-off-by: Eugene Syromiatnikov <esyr@redhat.com>
