Change the DiskFormat Factory to be a context manager.
All code using DiskFormat was updated to the following
with statement:
with DiskFormat(...).new as disk_format:
disk_format.some_member()
This is related to Issue #2412
Command.run() currently has a bit of a confusing behavior: if raise_on_error is
False and the executable is not found, then a weird CommandT is returned (return
code is -1 and stdout+stderr is None). This makes it possible to hanlde command
not found errors separately, but it makes that needlessly verbose. So instead,
let's just return None in *this* special case.
That in turn uncovered, that in most cases when we set `raise_on_error=True`, we
actually want an error if the command is not present but no error if the command
fails to execute (e.g. because it returns -1 if you run `$cmd --version`). Hence we
introduce the flag `raise_on_command_not_found`, which causes an exception to
be raised if the command is not found. This makes it independent of the
`raise_on_error` flag.
Additionally, we add a small optimization: if command starts with /, then we
assume it's a full path and we omit the call to which (and just check whether it
exists).
Co-authored-by: Marcus Schäfer <marcus.schaefer@gmail.com>
Change the RaidDevice, LuksDevice and IntegrityDevice classes
to context manager:
with RaidDevice(...) as raid:
raid.some_member()
with LuksDevice(...) as luks:
luks.some_member()
with IntegrityDevice(...) as integrity:
integrity.some_member()
In the context of the disk builder an ExitStack is used to
handle the new context manager based classes
This is related to Issue #2412
Change the Disk class to be a context manager.
All code using Disk was updated to the following
with statement:
with Disk(...) as disk:
disk.some_member()
This is related to Issue #2412
Change the FileSystem class to be a context manager. All code using
FileSystem was updated to the following with statement:
with FileSystem.new(...) as filesystem:
filesystem.some_member()
This is related to Issue #2412
Change the LoopDevice class to be a context manager.
All code using LoopDevice was updated to the following
with statement:
with LoopDevice(...) as loop_provider:
loop_provider.some_member()
This is related to Issue #2412
The keyfile path was not correctly set in etc/crypttab which
caused systemd not being able to read the keyfile, consequently
asking for a passphrase. This commit fixes the writing of the
crypttab and also fixes a python name clash with the "os"
namespace.
The recent change from kpartx to partx caused some appliance
builds to break depending on their configuration. I spotted
issues when building disks with veritysetup or integritysetup
root devices. There are also issues with grub-install on other
architectures e.g s390. It seems partx cannot be used as a drop
in replacement and so I suggest to make this a runtime
configuration option with partx as the new default but also
allow for the old method
partx can clean up all partition loops associated with a loop device
when deleting the main loop device. Apparently, sometimes it goes and
does this even when only deleting the partition loop, so to avoid
this problem, we will just eliminate the redundant call.
Fixes: 8f2b8fda82
The device-mapper (DM) based loop devices that kiwi has historically
generated creates issues when trying to run kiwi in confined build
environments. The DM tools prefer to have a tighter coupling between
userspace and kernel interfaces, and we cannot necessarily guarantee
that in the variety of build environments that kiwi can be run in.
In particular, Koji uses either nspawn containers or chroots through
Mock to run kiwi and that is where this fails.
However, we do not need to use DM for this purpose when util-linux
provides a perfectly serviceable alternative. This commit changes
kiwi's loop device setup to use partx(8) from util-linux instead.
As long as the appliance image being created doesn't use a disk
scheme that requires DM, it becomes entirely possible to produce
an image without needing DM at runtime at all.
Add a new attribute integrity_legacy_hmac="true|false" which
allows to use old flawed HMAC calculation (does not protect superblock).
Add a new attribute integrity_legacy_padding="true|false" which
allows to use inefficient legacy padding. Do not use these attributes
until compatibility with a specific old kernel is required!
The attribute disk_start_sector allows to specify a custom
start sector for the first partition of the disk. On GPT
tables everything works nicely, on DOS tables the used tools
fdisk/sfdisk are not able to manage the start/end values of
subsequent partitions if the first partition doesn't start
with the tooling default. This patch allows to set the
start sector after the partition table has been created
When using squashfs in a custom partitions setup like the following:
<partitions>
<partition ... filesystem="squashfs"/>
</partitions>
The build fails because the filesystem needs to be created
using the create_on_file() API and not the create_on_device()
API. In addition the size estimation is bogus when using
squashfs and cannot be pre-calculated because we only know
how much space the filesystem really needs after mksquashfs
as worked on the data and the compression. Thus this commit
also relaxes the required size check in case of squashfs.
Last but not least a squashfs filesystem does not provide
label or UUID and can only be referenced by the PARTUUID
it gets dumped on or by the native unix device node. As
the unix node is a loop during build time of the image and
meaningless this commit also forces by-partuuid mapping in
fstab when mounting the squashfs based device.
Subformats should also not be compressed when encryption is enabled
This is a follow on change to bdba953. When the filesystem is encrypted the
resulting image should not be compressed. Also explain why we ignore the
compression seeting in the user configuration for encrypted images.
Allow to protect the opening of the integrity data map and
journal through a keyfile. For setting the key file two new
optional type attributes were added:
* integrity_keyfile
* integrity_metadata_key_description
The key file format must be correct according to the selected
integrity algorithm. As of now the kiwi default hmac-sha256
algorithm is used with the selected keyfile
The optional integrity_metadata_key_description attribute
allows to specify a custom description of an integrity key
as it is expected to be present in the kernel keyring. The
information is placed in the integrity metadata block. If
not specified kiwi creates a key argument string instead
which is based on the given integrity_keyfile filename.
The format of this key argument is:
:BASENAME_OF_integrity_keyfile_WITHOUT_FILE_EXTENSION
There is support in kiwi to use dm_integrity in combination
with the LUKS header and dm_crypt. However there is also the
use case to setup dm_integrity in standalone mode. This commit
allows to create the dm_integrity layer outside of LUKS using
/etc/integritytab to activate the map through a systemd
generator if systemd is used.
Regarding systemd it's required to use a version of system which
provides: system-generators/systemd-integritysetup-generator.
If this generator does not exist in the distribution it will
also be missing in the dracut generated initrd and the boot
will not be able to succeed. It's mentioned here because even
newer distributions might be missing the generator
Along with the implementation there are two new optional
attributes in the <type> section:
standalone_integrity="true|false"
embed_integrity_metadata="true|false"
standalone_integrity activates/deactivates the dm_integrity map
on top of the root filesystem. Similar to the veritysetup support
there is the opportunity to create an embedded magic metadata
block at the end of the device containing the root
filesystem via embed_integrity_metadata
The Disk class provides methods to create partition(s)
and map names according to its scope and independent of
the actual partition tools. For example: create_root_partition().
This commit adds an additional optional clone parameter to
all methods for which we want to allow partition clones
Added CloneDevice class to the storage interface.
The class allows to create clone(s) from a given source
block device into a list of target block devices.
The target block devices are clones of the source but
prevents device naming conflicts for unique identifiers
like the UUID. This is requires to still allow to boot
from images containing device clones and needs to be
handled by tools that might work on top of the cloned
devices.
This commit adds the following new type attribute
<type ... dosparttable_extended_layout="true|false"/>
If set it specifies to make use of logical partitions inside
of an extended one. Effective only on type configurations which
uses the msdos table type, it will cause the fourth partition
to be an extended partition and all following partitions will be
placed as logical partitions inside of that extended partition.
This setting is useful if more than 4 partitions needs to be
created in an msdos table.
In addition to the support for extended/logical partitions the
the attributes 'mountpoint' and 'filesystem' in the <partitions>
section becomes optional. This also allows to place partitions
as placeholders not mounted into the system
The modifications in this commit allows the unit tests
to run on both, pytest 6.x (nose test layout) and the new
pytest 7.x (xunit test layout). This Fixes#2072 in a
much nicer way. Thanks much to @smarlowucf
This reverts commit 0dc2e803e0.
The pytest interface from version v6 to v7 has received
changes which requires the tests to be adapted to work for
either the old or the new interface. As there are still many
distributions which uses v6 as the standard we decided to
revert back the adaptions done to support v7 and create
a version requirement to v6 in .virtualenv.dev-requirements.txt
This Fixes#2072
The pytest interface setup() method call has changed
in a way that an additional parameter is passed to
the method which leads to a python error at invocation
time if the setup method does not define it.
In addition to the volume volume management settings also
allow to setup low level table entries like in the following
example:
<partitions>
<partition name="var" size="100" mountpoint="/var" filesystem="ext3"/>
</partitions>
Moving use of mkdtemp, NamedTemporaryFile and TemporaryDirectory
into its own class called Temporary: By default all temporary
data is created below /var/tmp but can be changed via the
global commandline option --temp-dir. This Fixes#1870
To support cloud platforms better we should allow the
creation of an initial(insecure) LUKS encrypted image
with an empty passphrase/keyfile. This Fixes
bsc#1187461 and bsc#1187460
When building a vmdk image with pvscsi as adapter type, kiwi
implicitly changed the adapter_type from pvscsi to lsilogic
because qemu only knows lsilogic. At the end kiwi patched
the adapter type in the descriptor of the vmdk header back
to pvscsi. That patching seems to be wrong according to
information from users and VMware support. This commit
deletes the descriptor patching and only leaves the pvscsi
setting in the guest configuration(vmx).
This Fixes bsc#1180539 and Fixes#1847
The option --target-arch allows to set the architecture
used to build the image. By default this is the host
architecture. Please note, if the specified architecture
name does not match the host architecture and is therefore
requesting a cross architecture image build, it's important
to understand that for this process to work a preparatory
step to support the image architecture and binary format
on the building host is required and is not considered a
responsibility of kiwi. There will be a followup effort
on providing a plugin for kiwi which should be used to
manage the needed binfmt settings for cross arch image
builds
The former implementation read the runtime config file every
time a new RuntimeConfig instance was created. The runtime
config is however static during the runtime of an image build
process and not supposed to change. Therefore the file should
be read in once and any new instance of RuntimConfig should
just use what has been read in at the first invocation. This
commit refactors the RuntimeConfig class to hold an application
global RUNTIME_CONFIG variable and allows to reread the
config on explicit request.
This commit refactors the Partitioner class and turns it into a
proper factory class and also includes type hints to facilitate
it's use from an API POV. Related to #1498
Selecting the xfs filesystem made kiwi to create an extra
boot partition. This is from times when grub was not able
to read from XFS. As grub doesn't have this limitation since
quite some time the bootpartition default in kiwi for XFS
should be changed. This is realted to #1611
This commit refactors DiskFormat to turn it into a proper
factory class and to also include type hints to facilitate it's
use from an API POV. Related to #1498
In addition to the custom size of the root volume it's now
also possible to setup the name of the root volume as follows:
<volume name="@root=rootlv"/>
If no name for the root volume is specified the default
name: LVRoot applies as before. This Fixes#1530
