redhat-rpm-config/brp-python-bytecompile
Lumir Balhar 6335a7ff4a Fix brp-python-bytecompile with the new features from compileall2
Resolves: rhbz#1595265

The problem this change is intended to solve is with how `real_libdir`
is calculated. Let's assume we want to recursively byte-compile all
`*.py` files in
`/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8`.
Then, `real_libdir` is this path without `$RPM_BUILD_ROOT` with
the filename at the end which displays in the error message like this:

```
Bytecompiling .py files below /builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8 using /usr/bin/python3.8
*** Error compiling '/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8/site-packages/greplin/bar.py'...
  File "/usr/lib/python3.8/bar.py", line 1
    import sin from math
               ^
SyntaxError: invalid syntax
```

`/usr/lib/python3.8/bar.py` is obviously wrong.

One of the new features of the `compileall2` module (which will
be available in stdlib in Python 3.9) is that the path byte-compiled to
`*.pyc` files is calculated for each file. This means that by using
`-s` and `-p` we can strip `$RPM_BUILD_ROOT` and prepend `/` for each
file individually which will fix the problem.

```
Bytecompiling .py files below /builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8 using /usr/bin/python3.8
*** Error compiling '/builddir/build/BUILDROOT/python-scales-1.0.9-250.fc32.x86_64/usr/lib/python3.8/site-packages/greplin/bar.py'...
  File "/usr/lib/python3.8/site-packages/greplin/bar.py", line 1
    import sin from math
               ^
SyntaxError: invalid syntax
```

This change has an effect only for Python >= 3.4.
2019-12-02 16:39:32 +01:00

145 lines
5.3 KiB
Bash
Executable File

#!/bin/bash
errors_terminate=$2
extra=$3
# If using normal root, avoid changing anything.
if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then
exit 0
fi
# Figure out how deep we need to descend. We could pick an insanely high
# number and hope it's enough, but somewhere, somebody's sure to run into it.
depth=`(find "$RPM_BUILD_ROOT" -type f -name "*.py" -print0 ; echo /) | \
xargs -0 -n 1 dirname | sed 's,[^/],,g' | sort -u | tail -n 1 | wc -c`
if [ -z "$depth" -o "$depth" -le "1" ]; then
exit 0
fi
# This function now implements Python byte-compilation in two different ways:
# Python >= 3.4 uses a new module compileall2 - https://github.com/fedora-python/compileall2
# Python < 3.4 (inc. Python 2) uses compileall module from stdlib with some hacks
# When we drop support for Python 2, we'd be able to use all compileall2 features like:
# - -s and -p options to manipulate with a path baked into pyc files instead of $real_libdir
# - -o 0 -o 1 to produce multiple files in one run - each with a different optimization level - instead of $options
# - removed useless $depth - both compileall and compileall2 are limited by sys.getrecursionlimit()
# These changes will make this script much simpler
function python_bytecompile()
{
local options=$1
local python_binary=$2
local exclude=$3
local python_libdir=$4
local depth=$5 # Not used for Python >= 3.4
local real_libdir=$6 # Not used for Python >= 3.4
python_version=$($python_binary -c "import sys; sys.stdout.write('{0.major}{0.minor}'.format(sys.version_info))")
#
# Python 3.4 and higher
#
if [ "$python_version" -ge 34 ]; then
[ ! -z $exclude ] && exclude="-x '$exclude'"
# /usr/lib/rpm/redhat/ contains compileall2 Python module
# -q disables verbose output
# -f forces the process to overwrite existing compiled files
# -x excludes paths defined by regex
# -e excludes symbolic links pointing outside the build root
# -x and -e together implements the same functionality as the Filter class below
# -s strips $RPM_BUILD_ROOT from the path
# -p prepends the leading slash to the path to make it absolute
PYTHONPATH=/usr/lib/rpm/redhat/ $python_binary $options -m compileall2 -q -f $exclude -s $RPM_BUILD_ROOT -p / -e $RPM_BUILD_ROOT $python_libdir
else
#
# Python 3.3 and lower (incl. Python 2)
#
cat << EOF | $python_binary $options
import compileall, sys, os, re
python_libdir = "$python_libdir"
depth = $depth
real_libdir = "$real_libdir"
build_root = "$RPM_BUILD_ROOT"
exclude = r"$exclude"
class Filter:
def search(self, path):
ret = not os.path.realpath(path).startswith(build_root)
if exclude:
ret = ret or re.search(exclude, path)
return ret
sys.exit(not compileall.compile_dir(python_libdir, depth, real_libdir, force=1, rx=Filter(), quiet=1))
EOF
fi
}
# .pyc/.pyo files embed a "magic" value, identifying the ABI version of Python
# bytecode that they are for.
#
# The files below RPM_BUILD_ROOT could be targeting multiple versions of
# python (e.g. a single build that emits several subpackages e.g. a
# python26-foo subpackage, a python31-foo subpackage etc)
#
# Support this by assuming that below each /usr/lib/python$VERSION/, all
# .pyc/.pyo files are to be compiled for /usr/bin/python$VERSION.
#
# For example, below /usr/lib/python2.6/, we're targeting /usr/bin/python2.6
# and below /usr/lib/python3.1/, we're targeting /usr/bin/python3.1
shopt -s nullglob
for python_libdir in `find "$RPM_BUILD_ROOT" -type d|grep -E "/usr/lib(64)?/python[0-9]\.[0-9]$"`;
do
python_binary=/usr/bin/$(basename $python_libdir)
real_libdir=${python_libdir/$RPM_BUILD_ROOT/}
echo "Bytecompiling .py files below $python_libdir using $python_binary"
# Generate normal (.pyc) byte-compiled files.
python_bytecompile "" "$python_binary" "" "$python_libdir" "$depth" "$real_libdir"
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
# Generate optimized (.pyo) byte-compiled files.
python_bytecompile "-O" "$python_binary" "" "$python_libdir" "$depth" "$real_libdir"
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
done
# Handle other locations in the filesystem using the default python implementation
# if extra is set to 0, don't do this
if [ 0$extra -eq 0 ]; then
exit 0
fi
# If we don't have a default python interpreter, we cannot proceed
default_python=${1:-/usr/bin/python}
if [ ! -x "$default_python" ]; then
exit 0
fi
# Figure out if there are files to be bytecompiled with the default_python at all
# this prevents unnecessary default_python invocation
find "$RPM_BUILD_ROOT" -type f -name "*.py" | grep -Ev "/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]|/usr/share/doc" || exit 0
# Generate normal (.pyc) byte-compiled files.
python_bytecompile "" $default_python "/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
# Generate optimized (.pyo) byte-compiled files.
python_bytecompile "-O" $default_python "/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]|/usr/share/doc" "$RPM_BUILD_ROOT" "$depth" "/"
if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then
# One or more of the files had a syntax error
exit 1
fi
exit 0