Note the exception string checking around compose_type. I didn't really
want to introduce a new exception type just for this, but also didn't
want to duplicate strings. I'd be open to other suggestions for how to
do this.
(cherry picked from commit b3bb438254)
This adds some fairly redundant code to the beginning of all the
blueprint routes to attempt reading a commit from git for the
blueprint's recipe. If it succeeds, the blueprint exists and the route
can continue. Otherwise, return an error. Hopefully this doesn't slow
things down too much.
(cherry picked from commit a925cc7ddb)
Note that this also changes the return type of uuid_info to return None
when an unknown ID is given. The other uuid_* functions are fine
because they are checked ahead of time.
(cherry picked from commit 6497b4fb65)
Each element in the errors value is now a dict, with a msg field and an
id field. The id field contains a value out of errors.py that can be
used by the front end to key on. The msg field is the same as what's
been there.
The idea is to keep the number of IDs somewhat limited so there's not a
huge number of things for the front end to know.
(cherry picked from commit 9677b012da)
This should make it easier to return more complex error structures. It
also doesn't appear to matter - tests still pass without changes.
(cherry picked from commit 4c3f93e329)
Make sure no UTF8 characters are allowed and return an error if they
are.
Also includes tests to make sure the correct error is returned.
(cherry picked from commit 86d79cd8a6)
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.
This also adds tests for the routes to confirm that an error is
correctly returned.
(cherry picked from commit 74f5def3d4)
This handles the case where a route is requested, but without a required
parameter. So, /blueprints/info is requested instead of
/blueprints/info/http-server. It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.
Typo'd URLs (/blueprints/nfo for instance) will still return a 404. I
think this is a reasonable thing to do.
(cherry picked from commit 5daf2d416a)
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules. yum doesn't traceback when doPackageLists is given
something that doesn't exist. It just returns an empty list. If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened. If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.
Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.
(cherry picked from commit 8e948e4a4d)
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).
(cherry picked from commit 3ba9d53b8b)
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.
(cherry picked from commit 17c40ef271)
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose. This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.
(cherry picked from commit b59d59b124)
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.
(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve. However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.
(cherry picked from commit 5fe4b47072)
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
Pass --dnfplugin='*' to enable all of them.
Pass --dnfplugin='plugin-name' to enable one fo them. You can use it
multiple times to enable multiple plugins. Globs work as well.
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.
To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.
Resolves: rhbz#1589084
(cherry picked from commit c746e8b0c3)
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.
--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...
Resolves: rhbz#1452220
Make it possible to manipulate the simple and regexp
tests the LogRequestHandler class uses to check error
messages for potential error states.
This is accomplished by moving the simple and regexp test
strings to class members, where they can be easily
manipulated by users of the pylorax module.
It's also now possible to set the log request handler class
for a LogMonitor.
This functionality can then be used for example like this:
customized_log_request_handler = monitor.LogRequestHandler
customized_log_request_handler.simple_tests.remove("Call Trace:")
log_monitor = monitor.LogMonitor(install_log,
timeout=opts.timeout,
log_request_handler_class = customized_log_request_handler)
This way installation will continue even if there was a call
trace in the logs. In a similar way additional tests and regexps can be
also added.
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.
Add a test for the new function.
Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.
Update test_server.py to check for the correct error code.
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.
<kalev> I still think it's a fallout from 27a90d973f?branch=master
Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.
https://bugzilla.redhat.com/show_bug.cgi?id=1581838
First is Anaconda uses 6k blocks per file for its estimate, and it
fudges by 10% so adjust for those with an extra 10% of headroom just in
case.
Second is an Anaconda bug that won't allow it to do a kickstart install
to a disk smaller than 3000 MB. There is a PR to fix it upstream, but
for now the minimum size has to be 3000e9
This adds support for the optional blueprint section [customizations].
Use it like this:
[customizations]
hostname = yourhostnamehere
[[customiations.sshkey]]
user = root
key = root user key
Different versions of libgit2 act differently. Using TIME results in
some commits (like a revert) being listed correctly, but the rest being
listed in reverse order. Leaving it at the default works for
libgit2-0.26.3
This moves everything except the cmdline checking into run_creator in
pylorax.creator
It also rearranges some functions to prevent import loops, and adds a
utility function to imgutils (mkfsimage_from_disk for copying a
partition into a filesystem image).
This no longer uses the enabled configuration setting to select repos to
use. It uses everything in the repo_dir, and if system repos have not
been disabled it copies them into the repo_dir at startup, overwriting
the previous copy.
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
filter(provides=...) doesn't work with paths. The release packages
provide system-release so just look for that instead of a file.
Now it finds the release package and selects it along with the
corresponding logos package.
Note, this has been broken since commit 431ca6ce
Commit 8edaefd4d1 added the ability to install specific NVR's of
packages, but it did not adjust the exclude operation to account for
this.
This patch fixes that, applying the exclude only to the name part of the
package NVR, and changes some variable names to pkgnvr/pkgnvrs to make
it more clear that the content has changed to <name>-<version>-<release>
Some lorax users run it from inside mock, which isn't able to detect
whether the host is in Permissive mode. This can lead to confusing
error messages, so this points them in the right direction.
When multiple units are passed to systemctl and one fails it doesn't
finish the others. Change the template command to call systemctl for
each unit individually.
This also removes the lvm2-activation-generator in runtime-cleanup.tmpl
This will allow anaconda to fetch kickstarts using https when installing
with fips=1
Leave vmlinuz and .vmlinuz.hmac in /boot
dracut-fips module needs the vmlinuz.hmac file in order to boot.
It seems that on rare occasions losetup can return before the /dev/loopX
is ready for use, causing problems with mkfs. This tries to make sure
that the loop device really is associated with the backing file before
continuing.
NOTE that using losetup --list -O to return the backing store
associated with the loop device can fail due to losetup truncating
the output filename if sysfs isn't setup. Instead of printing the full
path it will truncate it to 64 characters with a * at the end.
See util-linux lib/loopdev.c for the code that does this.
Use the existing get_loop_name function, which uses losetup -j, to lookup
the loop device associated with the backing store which should work the
same, just in the opposite direction.
For historical reasons, lorax used the 'anaconda' package as a
touchstone to determine the architecture for the build. At some
point, this package became a metapackage that pulls in both the
GUI and headless installers.
In the modular world, it's possible that only the core and TUI bits
may be available for use. The only subpackage of anaconda that is
guaranteed to be on any viable system is anaconda-core, so let's
switch to using that for the touchstone instead of the metapackage.
Signed-off-by: Stephen Gallagher <sgallagh@redhat.com>
Also sort the expanded list of packages so that any failures will
be consistent instead of depending on the randomness of a set().
And add better logging when things fail.
The core issue is that repodata may have packages that match globs, but
they cannot actually be installed (eg. sigrok-firmware). This can cause
*some* of the globbed packages to be installed before hitting the
failure package.
With this change it will log the expanded list of packages if a glob is
used. It will skip any packages that fail to install when using
--optional with the glob, and continue to install the rest.
Related: rhbz#1440417
Previously lorax had no way to use repos with self-signed certificates.
This adds the --noverifyssl cmdline option which will ignore certificate
errors.
Resolves: rhbz#1430483
OSTree is a deduplicating hardlink store using a new file path
`/ostree`, which SELinux policy doesn't know about. However, OSTree
has SELinux support built in, and rpm-ostree (for example) uses this
to ensure the attributes on files stored there are simply always
correct. Relabeling it will corrupt it.
Hence, let's skip it.
Right now we dump all subprocess output to `program.log`. Unfortunately,
The pungi/koji stack doesn't know how to scrape out the lorax logs.
And even when running interactively, it's annoying that *some* fatal
errors show up on stderr, but if it's from a subprocess, I need to go
over and `tail program.log`.
Let's output the subprocess stderr directly, since the user is
going to want it prominently anyways.
anaconda-26.1 changed how package scriptlet failures are handled. They
are now fatal, and anaconda hangs after logging an Installation failure.
ERR packaging: Installation failed: PayloadInstallError('DNF error:
Non-fatal POSTIN scriptlet failure in rpm package mlocate',)
Catch this (the 'packaging: Installation failed' part) and terminate the
image creation.
This controls how big the root filesystem is for the squashfs used in
the boot.iso, the default is 2GiB.
Note that larger rootfs sizes will require more memory and may cause the
build to fail.
I'm working on
https://fedoraproject.org/wiki/Changes/WorkstationOstree and when
using lorax to make an installer ISO with content embedded, I run out
of disk space since the desktop+various apps is large.
Since this ends up being compressed anyways, let's just bump the
currently arbitrary `2` to `10` - the only real cost I can think of is
going to be a few more superblock entries.
If the query filter doesn't return anything it would just ignore the
install request instead of logging and raising an error when
required=True.
This checks for no packages matching, and if required is True raises an
error after all of the requested packages have been processed, instead
of after the first one to fail.
Previous versions of lorax assumed that installpkg was optional, and
would continue on if the PKGGLOB didn't match anything. But the majority
of the packages are required so this allows the boot.iso to be built
with missing packages that are hard to track down.
It makes more sense to make the PKGGLOB required and to flag the
few exceptions to this with --optional.
Fedora now has a edk2 package so use the OVMF code from there. This also
adds using a copy of OVMF_VARS for each boot instead of reusing the one
provided by the package.
There's no reason to require the initramfs when we can rebuild it using
the version from the kernel. This adds handling of missing initramfs so
that lmc kickstarts can remove it from the squashfs, saving about 40M on
the iso.
umount tries to delete a mountpoint if it has lorax.imgutils in the
path. This doesn't work right if you try to umount something mounted
deeper on the path.
This adds a delete option, which is True by default, to skip the delete.
When using the template install command copying the same file to itself
shouldn't crash. Just log the error and continue.
Also copy the s390 configuration files for use with livemedia-creator
Resolves: rhbz#1269213
Allow the template to select a different compression type or arguments
for the installimg command.
On 32bit builds running inside a mock xz sees the full amount of system
memory which can result in xz failing with a memory error. This allows
the template to limit the amount of memory it tries to use.
If there isn't enough space for DNF to download packages it will log:
"Not enough disk space to download the packages."
So add this to the messages in monitor that trigger an error.
This makes package selection a little more roundabout, but it allows for
unused packages (and their dependencies) to be removed from globs during
the install phase.
dnf.subject.Subject is the class used by dnf's Base.install to select
packages, so the behavior of installpkg without --except options is the
same as it was before.
When using no-virt the runtime filesystem size comes from the kickstart.
For virt installs lmc was creating a runtime filesystem that was just
slightly larger than the space used by the files installed by anaconda.
This can run into problems with larger filesystem. It is also
inconsistent behavior between virt and no-virt installations.
With this commit the virt runtime filesystem will also come from the
kickstart.
This allows lorax to support multiple templates.
If there is no templates.d under the sharedir (/usr/share/lorax or the
directory passed by --sharedir) then the templates in that directory
will be used as they were previously.
If there are directories under templates.d the first one will be used,
unless --sharedir points to a specific one.
Use 4k blocks for the ext4 filesystem. Run fsck on the filesystem to
make sure deleted blocks are actually zeroed, and pass -Xbcj to
mksquashfs.
4k blocks and -Xbcj decreases the size by 2-6% depending on the
filesystem size. Zeroing the blocks of the ext4 fs improves things
dramatically. The problem is that DNF downloads the rpms before
installing them. In addition to forcing us to use a larger filesystem
than we would like it leaves data that is difficult to compress on the
image. The downloaded files are removed, but need to be zeroed out so
that mksquashfs can compress it.
Sometimes debugging a boot.iso requires using gdb, and finding the
corresponding debuginfo packages can be difficult. This writes the
matching -debuginfo package names and full ENVR to a file on the iso.
This can then be fed to dnf to install the correct debug packages.
Sometimes you don't want to include the selinux xattrs in the tar (eg.
bsdtar has problems extracting them). They are still included by
default, but pass selinux=False to remove '--selinux --acls --xattrs'
from the tar cmdline.
This allows the partition to be mounted on a directory underneath the
temporary directory, eg. /rootfs/, to help support creating other image
types without needed to move the files around.
When running the transaction in a separate process it crashes if you use
a https repo source. There's really no need for threads or processes in
lorax so drop it.
Also switched to using the DNF TransactionProgress API for progress
reporting.
The system the image boots on will likely not match the host where lorax
was run, and in some cases this can cause systems to hang.
Resolves: rhbz#1258498
fedup is deprecated and abandoned. Let's save time and disk by not
building `upgrade.img` when nothing is going to use it anymore.
For the record, performing upgrades using an initramfs from the new
system turns out to be fragile and hard to support:
* dracut initramfs isn't generic enough to handle booting all systems
(e.g. missing vconsole.conf means you get keymaps wrong, so users
can't unlock encrypted disks)
* The ABI differences between the two versions of plymouth, systemd,
etc. requires nasty workarounds at best and causes nightmarish
systemd crashes at worst
This patch removes all the code that built and installed `upgrade.img`.
For backwards compatibility, the API retains the `doupgrade` keyword
argument, and the `--noupgrade` flag is still accepted.
Some callers expect CalledProcessError.output to have the output, so
pass up the stdout + stderr output.
This means failed runcmd template commands will log to program.log and
lorax.log
After the cleanup step, check that everything in /usr/bin and /usr/sbin
can still run. Currently, this just checks that ELF files have
everything they need to link, and scripts have an interpreter.
Verifying is on by default but can be skipped with --noverify
When this is too small the rootfs can run into problems when used with a
live system. Doubling it leaves enough space for the system to run
properly during the installation and since it's all compresses it
doesn't make the image noticeably bigger.
Add a 'lower' filter to the templates to replace string.lower which no
longer exists. Fix udev_escape, the strings are already unicode, and
drop --chdir from runcmd. It wasn't ever used, and passing cwd to the
new runcmd isn't supported.
Fix up 2to3 complaints. I've decided to do with wrapping list
comprehension inside list() to get the generators to run in several
places instead of list(map( or list(filter( which seem less readable to
me.
If it terminates really badly (e.g. with SIGSEGV), it doesn't report any error,
just doesn't put anything to the queue. So instead of just blindly waiting on
the queue forever, check that the process is still alive if we don't get any
message in long time interval.
I originally added --add-template to support doing something similar
to pungi, which injects content into the system to be used by default.
However, this causes the content to be part of the squashfs, which
means PXE installations have to download significantly more data that
they may not need (if they actually want to pull the tree data from
the network, which is not an unusual case).
What I actually need is to be able to modify *both* the runtime image
and the arch-specific content. For the runtime, I need to change
/usr/share/anaconda/interactive-defaults.ks to point to the new
content. (Although, potentially we could patch Anaconda itself to
auto-detect an ostree repository configured in disk image, similar to
what it does for yum repositories)
For the arch-specfic image, I want to drop my content into the ISO
root.
So this patch adds --add-arch-template and --add-arch-template-var
in order to do the latter, while preserving the --add-template
to affect the runtime image.
Further, the templates will automatically graft in a directory named
"iso-graft/" from the working directory (if it exists).
(I suggest that external templates create a subdirectory named
"content" to avoid clashes with any future lorax work)
Thus, this will be used by the Atomic Host lorax templates to inject
content/repo, but could be used by e.g. pungi to add content/rpms as
well.
I tried to avoid code deduplication by creating a new template for the
product.img bits and this, but that broke because the parent boot.iso
code needs access to the `${imggraft}` variable. I think a real fix
here would involve turning the product.img, content/, *and* boot.iso
into a new template.
removekmod GLOB [GLOB...] --allbut KEEPGLOB [KEEPGLOB...]
This can be used to remove kernel modules from under
/lib/modules/*/kernel/ while keeping specific items. This should be
easier than constructing find arguments to select the right things to
save.
It appears that reset+fill_sack will now do the right thing and load the
state of the installed packages. Drop the hack with deleting the object.
Also add a double-check to make sure there really is a list of files
for anaconda-core before we run off and make an image without removing
anything.
This is a workaround for a current dnf bug, it doesn't update the state
of the packages after they are installed so we tear down the base dnf
object and create a new one pointing to the installroot.
There is an additional issue with the list of files returned, hawkey and
dnf don't appear to make a distinction between files, dirs and ghosted
dirs like yum did, this can result in too much being removed (eg. all of
/etc/selinux/) so we only remove files not directories.
pylorax users will need to change to using dnf and pass a dnf.Base()
object as the dbo argument instead of a yum object as the yum or ybo
argument. See the lorax script for an example of how to do this.
The lorax cmdline argument --excludepkgs has been removed since dnf
doesn't appear to have any way to support it and packages should be
controlled using templates anyway.
--make-pxe-live target generate live squashfs and initrd for pxe boot.
Also generates pxe config template.
--make-ostree-live is used for installations of Atomic Host. Additionally to
--make-pxe-live it ensures using deployment root instead of physical root of
installed disk image where needed. Atomic installation needs to be virt
installation with /boot on separate partition (the only way supported by
Anaconda currently). Content of boot partition is added to live root fs so that
ostree can find deployment by boot configuration.
Previously if there was an error during a novirt installation that
didn't exit the process there was no way to detect it. This uses the new
--remotelog option for anaconda to monitor the logs for errors using the
same criteria as it does when monitoring a virt install. If there is an
error the anaconda process will be terminated and the logs will be
gathered up into ./anaconda/
Recently, Fedora has been trying to do a 3 product split. As part of
that, lorax was changed to do "installpkg lorax-product-*" via
provides.
I think that approach is awkward; a much simpler approach is to simply
specify the product package as input to lorax on the command line, via
external rel-eng scripts.
This patch therefore adds --includepkg (and we should probably add an
option to remove the implicit lorax-product-* glob).
installimg SRCDIR DESTFILE
Create a compressed cpio archive of the contents of SRCDIR and place
it in DESTFILE.
If SRCDIR doesn't exist or is empty nothing is created.
Examples:
installimg ${LORAXDIR}/product/ images/product.img
tar recurses into directories by default, but find is feeding it all the
files and directories so the tar it produces is considerably larger than
it needs to be due to duplicate files. Add --no-recursion flag so that
tar will only add the specific files and directories piped to it by find.
Related: rhbz#1144140
(cherry picked from commit a8681aca4e)
The 32MiB size limit does not apply to upgrade.img since it's installed
to /boot by redhat-upgrade-tool instead of downloaded through TFTP. The
warning in rebuild_initrds will still be triggered by an upgrade.img
over the limit, but this doesn't halt the compose and it's probably not
a bad thing to know about.
Chris Lumens
Brian C. Lane
Adam Williamson
Martin Kolman
Colin Walters
Jiri Konecny
Yuval Turgeman
Claudio Zumbo
Alexander Todorov
Stephen Gallagher
Jaroslav Mracek
Pat Riehecky
Manjunath A Kumatagi
David Shea
Will Woods
Vratislav Podzimek
Radek Vykydal