The support for delta_root allows to build a delta container
image from a given base container. Due to the refactoring of
the kiwi code base using context managers no explicit deletion
of instances happens anymore. This uncovered a weakness of
the delta root code at the level of the overlay mount. At
the time of the umount there are still active temporary
mount handlers which keeps the mountpoint busy. In order to
fix this properly also the PackageManager factory is now
a context manager and the Repository factory received a
cleanup method which is called when the PackageManager goes
out of scope. This refactoring also fixes the busy state
when building deltas
an instance of CommandCapabilities allows to check for specific
options of a command. If the parsing of options has failed a
warning message is created by default. Under certain circumstances
like the check for the --help option of setfiles, such a warning
message can be misleading information in the build log file.
Therefore the new silent flag allows to suppress the warning
message and the flag is used for the capabilities of the
setfiles utility. This Fixes#2350
Command.run() currently has a bit of a confusing behavior: if raise_on_error is
False and the executable is not found, then a weird CommandT is returned (return
code is -1 and stdout+stderr is None). This makes it possible to hanlde command
not found errors separately, but it makes that needlessly verbose. So instead,
let's just return None in *this* special case.
That in turn uncovered, that in most cases when we set `raise_on_error=True`, we
actually want an error if the command is not present but no error if the command
fails to execute (e.g. because it returns -1 if you run `$cmd --version`). Hence we
introduce the flag `raise_on_command_not_found`, which causes an exception to
be raised if the command is not found. This makes it independent of the
`raise_on_error` flag.
Additionally, we add a small optimization: if command starts with /, then we
assume it's a full path and we omit the call to which (and just check whether it
exists).
Co-authored-by: Marcus Schäfer <marcus.schaefer@gmail.com>
systemd-boot tools like kernel-install expect a certain
entry naming policy. This commit adapts kiwi to adapt to
this policy. The name for the default entry is constructed
out of the ID information from /etc/os-release followed
by the name of the kernel as it is represented by the
directory name in /lib/modules/... This Fixes#2417
Only if libc reports errno 95 Operation not supported the method
should return that extended attributes are not supported. Also
add a debug information about the result of the call to get further
information in the log file
The attribute disk_start_sector allows to specify a custom
start sector for the first partition of the disk. On GPT
tables everything works nicely, on DOS tables the used tools
fdisk/sfdisk are not able to manage the start/end values of
subsequent partitions if the first partition doesn't start
with the tooling default. This patch allows to set the
start sector after the partition table has been created
The class allows to check for data produced on stdout
and stderr. However, programs reporting data on stderr
usually fails with an exit code != 0. If the command
is not called with raise_on_error=False it will never
be possible to catch information from stderr. As we
don't know if programs returns a failed exit code
even on their e.g --usage message we should always
pass the no raise option to make this more useful
There is support in kiwi to use dm_integrity in combination
with the LUKS header and dm_crypt. However there is also the
use case to setup dm_integrity in standalone mode. This commit
allows to create the dm_integrity layer outside of LUKS using
/etc/integritytab to activate the map through a systemd
generator if systemd is used.
Regarding systemd it's required to use a version of system which
provides: system-generators/systemd-integritysetup-generator.
If this generator does not exist in the distribution it will
also be missing in the dracut generated initrd and the boot
will not be able to succeed. It's mentioned here because even
newer distributions might be missing the generator
Along with the implementation there are two new optional
attributes in the <type> section:
standalone_integrity="true|false"
embed_integrity_metadata="true|false"
standalone_integrity activates/deactivates the dm_integrity map
on top of the root filesystem. Similar to the veritysetup support
there is the opportunity to create an embedded magic metadata
block at the end of the device containing the root
filesystem via embed_integrity_metadata
The method added information about the PARTUUID as useful
information. However, PARTUUID's are not supported by all
partition tables. The Linux generated artificial values
from the disk identifier are not wanted in this scope.
As the information is not mandatory it's better to not
provide it at all and avoid confusion to users.
Do not raise of blkid is not able to read the requested ID.
It is expected that the methods of the BlockID class either
returns a value or none but do not raise and cause the
complete process to terminate
veritysetup uses tabs to align values. The way kiwi parsed
the values did not strip out the tabs and later on keeps
them in the verification metadata block. The unit test
did not catch this because the mock output used for
veritysetup did not contain tabs. This commit fixes the
test to catch this condition and also fixes the code to
handle all space characters (tabs, space, newlines) in
a safe way
Along with creating a filesystem including device mapper features
like dm_verity (see verity_blocks) or dm_crypt/dm_integrity (see luks)
there is always the question where to store the metadata information
required to setup the device map. This can include information about
blocksizes, offset addresses and more. The create_verification_metadata()
method allows to write a signed custom data block of a documented
format at the end of the given block special which stores this type
of information such that tools at boot time gets the opportunity to
read this information. In this commit only information connected
to the dm_verity feature activated via the verity_blocks attribute
will be part of the verification block. With future changes other
data might be added
So far the verity support was only available with the
overlayroot layout and the read-only squashfs root. This
commit adds a new attribute: verity_blocks="number|all"
which allows to create the verity setup also on the
standard root partition
In addition to the change it was needed to extend the
Filesystem API with an additional optional paramter to
allow setup of the filesystem UUID. Having the opportunity
to set the UUID at filesystem creation is generally useful
and with regards to this particular change it became also
required
veritysetup was called as part of the disk builder. However,
the veritysetup should be a responsibility of the classes which
implements the sync_data method. This allows to use the creation
of a verity hash format right after sync as a feature to these
classes and in a broader scope. In addition to that change the
VeritySetup::format method now returns the metadata from the
format call and stores it as debug information to the log file.
A concept to persistently store the verification metadata as part
of the partition(s) will be a next step.
Setting this attribute to a number or 'all' in an overlayroot
configuration will create a dm verity hash from the number of
given blocks (or all) placed at the end of the squashfs compressed
read-only root filesystem. For later verification of the device,
and without further image description settings, the credentials
information produced by veritysetup from the cryptsetup tools, is
created as a file in /boot/overlayroot.verity and is stored as
such into the image by default.
utils/checksum: drop factor calls
For portability we don't need to call linux coreutils' factor,
instead we use pure python prime number generation implementation.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
To avoid conflicts with python standard `dir` function
and for better matching purpose, the Temporary class
constructor parameter it is the best to rename from dir
to path.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
Using of --temp-dir argument does not make an effect,
because optional 'dir' parameter defaults to the
global TEMP_DIR value before it's changed.
This patch address this issue.
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@gmail.com>
A speciality of the rsync tool is that it behaves differently
if the given source_dir ends with a '/' or not. If it ends
with a slash the data structure below will be synced to the
target_dir. If it does not end with a slash the source_dir
and its contents are synced to the target_dir. For example:
source
└── some_data
1. $ rsync -a source target
target
└── source
└── some_data
2. $ rsync -a source/ target
target
└── some_data
The parameter force_trailing_slash in the DataSync::sync_data
method can be used to make sure rsync behaves like shown in
the second case. This Fixes#1786
Due to the change on the temporary directory an error
in the volume manager classes for LVM and btrfs was
exposed. There was code which uses a sub-part of the
volume mount point directory which is a tempdir based
directory name. The sub-part was choosen by an index
based path split which worked for /tmp but no longer for
/var/tmp. With tempdir now being also a commandline
option this code has to become more robust.
Moving use of mkdtemp, NamedTemporaryFile and TemporaryDirectory
into its own class called Temporary: By default all temporary
data is created below /var/tmp but can be changed via the
global commandline option --temp-dir. This Fixes#1870
If the Codec.decode() method cannot encode the given data to utf-8
an Encoding exception is raised which causes kiwi to raise a
KiwiDecodingError. That way to handle the error causes an image
build to fail. However, this sort of error for example happens if
the .changes file of a package contains invalid characters. From
a user perspective this cannot be fixed and you cannot build the
image because of a stupid character error in a .changes file
outside your control. Because of this reason this commit handles
the decode error case differently and applies the 'replace'
strategy on error. The characters in questions gets replaced and
the rest of the data which can be decoded is returned.
The new attribute <volume ... filesystem_check="true|false"/>
allows to change the default value for the fs_passno field in
the generated fstab file. By default kiwi sets "0" in this
field and leaves it up to the user to customize this as
appropriate via script code. Coding changes to the fstab file
via scripts are not very user friendly and with respect that
systemd takes over control and generates checkers depending on
the value of fs_passno it would be good if there is a way to
explicitly specify if checks to the filesystem are wanted or
not. Therefore the new attribute now exists. If set to: true
this results in a value of "2" for the fs_passno field. Please
note the root/boot and efi entries are excluded from this
setup. This Fixes#1728
If devicepersistency="by-label" is set in the KIWI description
it will correctly operate on the fstab values but still uses
the UUID based setting for root= in the grub config. This commit
allows to set root=LABEL=... in the grub config in case the
devicepersistency requested it. In order for this to work this
commit also had to increase the scope of the grub helper
method _fix_grub_root_device_reference which is now called in
any case. This Fixes#1757
Python open function can fail when a file is not found encoded in
utf-8, depending on the execution environment. In particular on
Debian 10 an error message regarding utf8 was encountered.
Since checksum files always use only ascii characters, force it.
Allow to read multiple markup formats. Supported are XML
and YAML. The parsing and transformation is based on the
anymarkup module. The use of anymarkup is optional and
implemented as an on demand loading module. If a user
uses a yaml config file or a request to convert into
yaml is made without an installed anymarkup module an
exception is thrown
This commit ensures that when trying to parse the version of a tool
the comparison is done with the longest match for the given regular
expression. This solves cases such in `grub2` where the tool name
already provides some digit that could be seen as a version.
With the new Fstab class from prior pull request there is an
opportunity to handle all fstab related actions to be done
by that class. This commit extends the Fstab class with an
add_entry method such that we can avoid the extra lists
holding raw fstab lines in e.g the disk builder. In the end
all fstab related data is stored in an instance of the Fstab
class. This also extends the KIWI api by an fstab management
class. Related to #1329 and #1349
Handling of fstab should be done in its own namespace and class.
The current handling of fstab entries is spread at several places.
There should be only one code that writes the fstab entries and
that code should also care for the correct canonical order of
the mountpoints
This commit makes sure to empty RPM macros files are created during
the build. Additionally it also ensures that no needless `rm` and `mkdir`
calls are done.
Fixes#1316
This commit refactors the use of Path.which in several parts of the
code. Since dd4d2ed78 the Path utility is capable to run Path.which
on certain chroot env, thus no need to adapt the PATH environment
variable.
Fixes#1281
This commit adds a stderr_to_stdout option to Command.run method to
redirect stderr to stdout.
Additionally this commit also includes to the debug log the dracut
call output, including both, stderr and stdout.
Fixes#1114
This commit extends the bootstrap macro management to also consider
the case where image was already bootstrap. Note this is a common case
for building derived container images and also the situation when
`--allow-existing-root` flag is in use.
The method checked for the presence of /usr/bin/rpm. But
that binary is also provided by another toolkit named
busybox. Thus to check if the rpm we are aiming for is
present the check has been modified to look for /usr/bin/rpmdb
which is exclusively provided by rpm only. This Fixes#1037
