Add a 'lower' filter to the templates to replace string.lower which no
longer exists. Fix udev_escape, the strings are already unicode, and
drop --chdir from runcmd. It wasn't ever used, and passing cwd to the
new runcmd isn't supported.
Fix up 2to3 complaints. I've decided to do with wrapping list
comprehension inside list() to get the generators to run in several
places instead of list(map( or list(filter( which seem less readable to
me.
If it terminates really badly (e.g. with SIGSEGV), it doesn't report any error,
just doesn't put anything to the queue. So instead of just blindly waiting on
the queue forever, check that the process is still alive if we don't get any
message in long time interval.
I originally added --add-template to support doing something similar
to pungi, which injects content into the system to be used by default.
However, this causes the content to be part of the squashfs, which
means PXE installations have to download significantly more data that
they may not need (if they actually want to pull the tree data from
the network, which is not an unusual case).
What I actually need is to be able to modify *both* the runtime image
and the arch-specific content. For the runtime, I need to change
/usr/share/anaconda/interactive-defaults.ks to point to the new
content. (Although, potentially we could patch Anaconda itself to
auto-detect an ostree repository configured in disk image, similar to
what it does for yum repositories)
For the arch-specfic image, I want to drop my content into the ISO
root.
So this patch adds --add-arch-template and --add-arch-template-var
in order to do the latter, while preserving the --add-template
to affect the runtime image.
Further, the templates will automatically graft in a directory named
"iso-graft/" from the working directory (if it exists).
(I suggest that external templates create a subdirectory named
"content" to avoid clashes with any future lorax work)
Thus, this will be used by the Atomic Host lorax templates to inject
content/repo, but could be used by e.g. pungi to add content/rpms as
well.
I tried to avoid code deduplication by creating a new template for the
product.img bits and this, but that broke because the parent boot.iso
code needs access to the `${imggraft}` variable. I think a real fix
here would involve turning the product.img, content/, *and* boot.iso
into a new template.
pylorax unconditionally calls reset() on the dbo, so provide an empty
method to keep it happy.
The lmc dbo is minimal because it is only used for creating the iso, not
anything related to package installation.
The stage2 image can be either LiveOS/squashfs.img or it can be
images/install.img, adjust the IsoMountpoint for this and rename the
flag to .stage2 instead of .liveos
removekmod GLOB [GLOB...] --allbut KEEPGLOB [KEEPGLOB...]
This can be used to remove kernel modules from under
/lib/modules/*/kernel/ while keeping specific items. This should be
easier than constructing find arguments to select the right things to
save.
It appears that reset+fill_sack will now do the right thing and load the
state of the installed packages. Drop the hack with deleting the object.
Also add a double-check to make sure there really is a list of files
for anaconda-core before we run off and make an image without removing
anything.
--cachedir allows the user to specify where the DNF cache is located.
This doesn't actually appear to do much since dnf erases the cache when
it is done. May be useful in the future.
--workdir sets the top level directory for lorax to use for installing
packages, creating installtree and installroot. Normally a temporary
directory under /var/tmp.
Note that the workdir will *not* be removed if there is an error setting
up the DNF object.
--force skips checking if the output directory exists, allowing things
like pungi to use lorax to place the output next to the repo tree it has
already created.
This is a workaround for a current dnf bug, it doesn't update the state
of the packages after they are installed so we tear down the base dnf
object and create a new one pointing to the installroot.
There is an additional issue with the list of files returned, hawkey and
dnf don't appear to make a distinction between files, dirs and ghosted
dirs like yum did, this can result in too much being removed (eg. all of
/etc/selinux/) so we only remove files not directories.
pylorax users will need to change to using dnf and pass a dnf.Base()
object as the dbo argument instead of a yum object as the yum or ybo
argument. See the lorax script for an example of how to do this.
The lorax cmdline argument --excludepkgs has been removed since dnf
doesn't appear to have any way to support it and packages should be
controlled using templates anyway.
The directory where the --logfile is located is also used for other log
files and for the anaconda logs when using --no-virt. Create the parent
directories if they don't exist.
--make-pxe-live target generate live squashfs and initrd for pxe boot.
Also generates pxe config template.
--make-ostree-live is used for installations of Atomic Host. Additionally to
--make-pxe-live it ensures using deployment root instead of physical root of
installed disk image where needed. Atomic installation needs to be virt
installation with /boot on separate partition (the only way supported by
Anaconda currently). Content of boot partition is added to live root fs so that
ostree can find deployment by boot configuration.
Previously if there was an error during a novirt installation that
didn't exit the process there was no way to detect it. This uses the new
--remotelog option for anaconda to monitor the logs for errors using the
same criteria as it does when monitoring a virt install. If there is an
error the anaconda process will be terminated and the logs will be
gathered up into ./anaconda/
Recently, Fedora has been trying to do a 3 product split. As part of
that, lorax was changed to do "installpkg lorax-product-*" via
provides.
I think that approach is awkward; a much simpler approach is to simply
specify the product package as input to lorax on the command line, via
external rel-eng scripts.
This patch therefore adds --includepkg (and we should probably add an
option to remove the implicit lorax-product-* glob).
installimg SRCDIR DESTFILE
Create a compressed cpio archive of the contents of SRCDIR and place
it in DESTFILE.
If SRCDIR doesn't exist or is empty nothing is created.
Examples:
installimg ${LORAXDIR}/product/ images/product.img
The aarch64 change to use shim (6907567) also stopped using mixed-case
names for BOOT${efiarch}.efi (so it's always .EFI and ${efiarch} is X64
IA32 AA64 etc. now), and mkefiboot needs to match that.
Related: rhbz#1043274
Related: rhbz#1100048
tar recurses into directories by default, but find is feeding it all the
files and directories so the tar it produces is considerably larger than
it needs to be due to duplicate files. Add --no-recursion flag so that
tar will only add the specific files and directories piped to it by find.
Related: rhbz#1144140
(cherry picked from commit a8681aca4e)
The 32MiB size limit does not apply to upgrade.img since it's installed
to /boot by redhat-upgrade-tool instead of downloaded through TFTP. The
warning in rebuild_initrds will still be triggered by an upgrade.img
over the limit, but this doesn't halt the compose and it's probably not
a bad thing to know about.
The ppc config files were missing from the live config_files directory
and ppc needs the correct lib directory so lmc has been switched to use
ArchData driven from the installed kernel arch.
Resolves: rhbz#1102318
(cherry picked from commit 59f256e989)
When using GPT it reserves a few more sectors at the end of the disk for
the copy of the GPT so pad the size of the partitions in the kickstart
by 2MiB instead of 1MiB to account for this.
Depending on the environment that --no-virt is run inside the resulting
filesystem may need to have the SELinux labels updates. Run setfiles on
the new filesystem after the anaconda run has finished.
eg. when run from inside mock the labels will be incorrect.
This no longer seems to be needed, and causes images created without
passing --vnc vnc to start up with a serial console. If you need a
serial console you can pass it using --kernel-args
Add check for running traceback script and when the retry fails.
This also indicates something went wrong with the installation, exit
when they are seen in the logs.
Also drop looking for WARNING in the regex errors, they will be errors
after the syslog level name remap patch goes into anaconda.
Spaces cause various bugs like #923374 and #855849 , and it would be
better if we just didn't use them.
Note that there's a corresponding pungi change to go with this.
Signed-off-by: Peter Jones <pjones@redhat.com>
Watch the logs for WARNING packaging: .* requires .* which indicates
that there are missing packages and it is sitting at a prompt waiting
for user input.
What I need is to make something like the traditional DVD which also
includes packages. At present this is apparently handled by the
entirely separate pungi tool.
At the moment for me, it's the least bad option to modify lorax to
inject data from an external source than to create a new tool, or
attempt to also modify pungi to do this.
This would also allow pungi's DVD creation to eventually be a set of
external templates for Lorax.
If the user supplies a fsimage to use for the iso it may not be on the
same filesystem as the working directory. Try to hardlink first, but if
that fails copy the image.
Commit d2ae92b4b3 patched up the download counter and progress
display. Yum no longer provides the needed information so now we get the
total number of packages from the start of the transaction.
This also turns off colors when stdout is not a tty, and only prints the
install progress once so that piping to a logfile isn't flooded with
useless characters.
This adds the --make-tar option which will produce a xz compressed tar
of the root filesystem. This works with either virt-install or no-virt
modes. Use --image-name to set the output filename.
--compression is used to set the compression type to use, which defaults
to xz. Supported types are xz, lzma, gzip and bzip2.
--compress-arg is used to pass arguments to the compression utility.
--make-fsimage was only working with --no-virt, this re-structures
things so that virt-install partitioned disk images can be converted to
a fsimage. --make-ami was actually already doing this, so change it to
use --make-fsimage and set the default image name to "ami-root.img" with
a label of "AMI".
This also adds the ability to set the fs label on iso fsimage and
fsimages created with --make-fsimage and --make-ami by passing
--fs-label, but note that bootable iso's expect the Anaconda label.
This is used as a kickstart %post interpreter to streamline
modifications to images.
Also adds an example kickstart.
This obsoletes the old appliance-tools-minimizer and includes a Provide
so that the transition will be seamless.
When doing an image install there is no .buildstamp file to pull the
information from so use the cmdline variables (or their defaults) and
pass them to anaconda in the environment.
Resolves: rhbz#1067746
Remove more drivers and remove plymouth and drm dracut modules. Only on
PPC64 initrd, all other arches have the full set of drivers and modules.
Resolves: rhbz#1060691
Make sure the data is written before we do anything else with the disk
image. This shouldn't be needed, umount should take care of it, but it
also can't hurt.
Resolves: rhbz#1052175
Also adds a check for a bad url repo, and fix ram calculation for
appliance mode. Updates the README.livemedia-creator documentation.
Resolves: rhbz#1019728
Fix "lorax -V" and add a "version is ???" to the log file so we can easily know
what version of lorax is used to build an iso.
Changed to try/import for versioning because the version file is autogenerated.
We should probably let the user know if something goes wrong with the
transaction or if a scriptlet fails. So: log the messages so we can find
(and, one hopes, fix) them later.
Some package scripts may call utilities using dbus. Since this is just a
chroot that will fail. This unsets DESKTOP and DBUS_SESSION_BUS_ADDRESS
to keep them from crashing.
This adds support for installing to a filesystem image instead of a
partitioned disk image. It requires Anaconda's --dirinstall support.
Also re-organized the code to break it up into smaller methods.
This speeds up iso creation in no-virt mode by removing the need to copy
the filesystem from the partitioned disk image to the filesystem image
that is used to make the squashfs image.
It used to give us (x/y) packagename, but now it only sends package
name. This was changed in yum commit 7e8c76173. For now default to
showing the zeros along with the package name so that it doesn't look
like an error.
kpartx can return before the devices are created. Use -s to wait.
Also remove -p p and let kpartx handle adding pX if needed, we use
whatever it outputs so there is no need to force a name.
This makes findkernels() look for any image named something like:
$PREFIX-$KERNELVER.img
and adds a corresponding entry to its returned data like:
kernel.$PREFIX.path = [path]
As a special backwards-compatibility case we use 'initrd' for the
attribute name if $PREFIX is 'initramfs'.
This gives us any extra initramfs images that may have been built using
the 'prefix' argument to rebuild_initrds().
If 'prefix' is passed to rebuild_initrds(), it will build a *new*
initramfs with a name like $PREFIX-$KERNELVER.img, rather than
overwriting the existing initramfs.
Dracut now makes the initrd with 600 permissions
for security reasons. These reasons do not apply
to install images, and we want the other tools
that use lorax to be able to read the initrd file.
When I switched execution over to execWith* functions I failed to
account for the use of CalledProcessError in various places. This
patch restores that behavior. All places that used check_call or
check_output now pass raise_err=True to the execWith* call.
--location specifies an iso directory tree to be used by virt-install
instead of the iso. This allows you to update the initrd in the tree for
debugging.
virt-install uses the images/pxeboot/ directory for initrd.img and
vmlinux.
An iso is still required for the LiveOS/squashfs.img stage2 file.
live media isn't exactly the same as the Anaconda install media. Right
now this amounts to needing a root= cmdline argument but in the future
there may be other differences.
This also reverts 5437557846 on the new copies of the templates.
Switch to using execWith* so that the command and its output can be
logged. To capture the output setup a logger named "program"
livemedia-creator captures all of this into program.log
- Add a check to livemedia-creator for /usr/bin/virt-install (#841566)
- Suggest the correct package for livemedia-creator's libvirt (#841552)
- Add to list of packages needed to build a livemedia-creator iso (#841594)
The 'systemctl' command can be used to enable, disable, or mask systemd
units inside the runtime being modified. Modify runtime-postinstall.tmpl
to use the 'systemctl' command.
We also no longer remove quota*.service or kexec*.service, since
these aren't enabled by default. And systemd-remount-api-vfs.service
should work correctly now, so we can leave it alone as well.
The '-cmd' functionality depends on the individual lorax template
commands raising errors, so they shouldn't do sys.exit().
Also, capture stderr along with stdout, and put both in the log.
There's something strange going on where unmounting a hfsplus volume
immediately after mounting it will fail with EBUSY.
This makes the umount fail, which makes the rmdir fail, which causes a
traceback, which breaks mkefiboot --apple.
It works fine if you wait a second and retry.. so do that.
Also, add the "lazy" argument so you can do lazy unmounts if you like.
Generally it's not a good idea for python libraries to set up loggers in
the body of the library.
Set up a NullHandler by default (as the logging module suggests), and
add a function to do the current logging setup during run().
From Harald Hoyer <harald@redhat.com>:
pollcdrom is used to poll for the install medium. A lot of CDROM drives
are not polled by the kernel correctly, so we have to actively poll for
the medium.
Some packages are critical to the compose. If --required
is specified in the template's installpkg command, lorax
will exit if the package is not available.
This adds support for creating an appliance description file for the
disk image. Mako templates are used to make it easy to support other
appliance targets. The included example works with virt-image.
Mac boot images are optional. Don't require hfsplus-tools
by default, but warn the user that he needs to install them
if he wants to create mac boot images.
When running with no-virt mode libvirt and virt-install are not needed
so make the import optional and update the usage info reflect that the
virt options are disabled.
There's a small amount of additional metadata required for the Mac boot
images to appear as bootable devices in the startup preferencs, so add
support for generating that.
Signed-off-by: Brian C. Lane <bcl@redhat.com>
Install the anaconda dracut module during 'install', use it when
rebuilding initramfs, and clean it up afterward.
Also install '.buildstamp' into the initramfs (the anconda module wants
it).
AMI images are un-partitioned filesystem images with a grub.conf that
is read by the pv-grub bootloader used by EC2. Most of the actual work
making the AMI is done in the kickstart. This just creates the image
file.
Fedora 17 changes top level directories like /bin, /lib, etc. to
symlinks to the corresponding dirs in /usr/
dracut can convert old systems to the new layout using its convertfs
module.
The installer no longer has access to the initrd's root. We need to
copy any needed files over to /sysroot before switching root. This
copies *.cfg and *.ks files.
It also adds the ability to add dracut hook scripts to the initramfs
from /usr/share/lorax/dracut_hooks/
This re-adds commit af6d4e2c50 which was
lost during the switch to the treebuilder branch.
In the latest method for booting the rootfs is in the LiveOS
directory of the media, not appended to the initrd. Detect this
and mount the iso and pass the CDLABEL to virt-install.
Some repos may contain anaconda packages for more architectures,
so it's not possible to get the right buildarch.
This patch allows optional specifying of the buildarch on the
command line when running lorax.
If the buildarch is not specified manually, lorax tries to get
it from the anaconda package as before.
If there's no anaconda package in the repository, don't fallback
to the system architecture and continue, because this is pointless.
We need the anaconda package.
When considering a package's filelist (e.g. for things like removefrom
--allbut), we might need to also include the %ghost files, so make
_filelist() add them to its returned list.
This runs anaconda directly, using the --image install feature. The host
system should be the same release as the target system in order to avoid
unexpected problems.
livemedia-creator uses an anaconda install media iso to install to a
file image. virt-install is used to execute the kickstart. lorax is used
to post-process the image file and create a bootable .iso from it.
Future additions will allow creation of EC2 images and output xml
details about the install.
The anaconda execWithRedirect and execWithCapture functions are too
useful not to include. They also allow you to log all the execuatable's
output to a logfile. Added them under executils.py module which uses
the pylorax and a new program logger.
Allow passing of size to create_runtime, add PartitionMount context
to use kpartx to mount partitioned file images. Add resetting the
selinux context on the newly created rootfs.
This means that any (non-ignored) command error will cause lorax to
exit.
Do note, however, that some commands (e.g. remove, installpkg) don't
raise exceptions and therefore will not cause lorax to exit.
Makefile-style "-cmd" syntax lets us run a command and ignore any
resulting errors. This is a more general version of what copyif/moveif
were trying to accomplish, so we can drop those commands.
New images find their root device by looking at the CDLABEL. Since pungi
is building ISO images separately from lorax, if it uses a different ISO
Volume Label we'll end up with unbootable images.
This changes the volume labels to match what pungi uses, so both should
boot OK.
- Log the error message (and the line causing the error) to console
- Log the whole exception (with some excess junk removed) to debuglog
- don't immediately SystemExit if the template won't parse
- clean up some comments
This lets us easily do whitelisting instead of blacklisting during
runtime cleanup. For example:
removefrom xfsprogs --allbut /sbin/* /usr/sbin/xfs_admin
would remove everything from the xfsprogs package except files in /sbin
and /usr/sbin/xfs_admin.
A few things in runtime-cleanup have been converted to use --allbut. The
only difference in the created runtime image is that we're deleting
/usr/share/kde4 from fedora-logos.
make sure that runner.templatedir gets set to a useful default if
initialized with None, and and make sure we pass templatedir to
LoraxTemplateRunner.__init__ (so it'll get initialized properly)
emit a useful log message if we attempt to removepkg something that
isn't installed, or if a glob used in a removefrom line matches no
files.
this will help us keep the templates cleaner.
* add bcj arch dict to ArchData
* add "compression" settings back to __init__.py
* pass them to treebuilder.create_runtime
* pass them through to imgutils.mksquashfs
To build F15 images we need to remove systemd and set up loader as init
(see runtime-cleanup and runtime-postinstall).
We also need to add a hack to dracut so loader won't freak out when it
gets started by anaconda - see the file we're adding to the initramfs in
treebuilder.py.
(There's also an extra bonus hack for working around a bug in dracut if
/proc/cmdline is empty - SEE IF YOU CAN SPOT IT!!!)
Let's let yum handle the magic of figuring out what basearch is. And
since basearch will match userspace, libdir should match basearch.
This fixes stuff on pre-F16 ppc64 (where basearch is ppc and thus has a
different libdir).
module-info and the updated depmod stuff need to be inside the runtime
image, so this needs to happen before create_runtime. Accordingly,
generate_module_data gets moved to RuntimeBuilder.
dracut's dmsquash-live-root won't recognize the image as a squashfs live
image unless its name matches "*squashfs.img", so choose the filename
accordingly. Then pass the name to the TreeBuilder so it can move it
into place.
add lib/dracut/hooks/pre-pivot/99anaconda-umount.sh to unmount things
before starting anaconda (because loader explodes if /{dev,sys,proc}
are mounted when it starts)
Add setup_init() and setup_s390_init() to installtree.py to handle
init setup, and stop using systemd so we can make F15 images.
This reverts commit b58190d660.
"from glob import glob" instead of "import glob"
rename args to BaseBuilder.runtemplate
set up 'exists' and 'glob' in runtemplate rather than getdefaults
Since we want all the modules in the runtime image, there's no need
to deal with the individual kernels. And workdir was only being used
to mess with the modules, so we don't need that either.
This allows us to create these objects without needing workdir,
which means we can use them outside of __init__.py.
We can also write them directly to their final destination
instead of writing them to the workdir and then copying them in.
Any decisions about arch-specific stuff should happen in the Lorax class
or the arch-specific templates/code. Move that logic up to Lorax.run()
and remove installtree.basearch.
getdata(cmd) will return a generator that yields every token on every
line that starts with the token "cmd". getdata(cmd, mode="lines") will
yield a list for each line rather than every individual token.
this simplifies some things in __init__.py.
arch has three attributes: .buildarch, .basearch, and .libdir
product has six: .name, .version, .release, .variant, .bugurl, and
is_beta
This makes it easier to pass this data into functions/templates.
TreeBuilder uses templates full of commands (like ramdisk.ltmpl) to
create the output tree and boot images. There are 4 arch-specific
templates, plus a bonus EFI template which can handle EFI image creation
for any arch that implements EFI.
This adds the new "mkefiboot" cmd, which creates an efiboot img in the
magical way that EFI requires. There doesn't seem to be a good tool for
this (unlike the existing tools for all the other weirdo boot image
types) so it was necessary to create one.
This contains simple functions for creating disk images:
mkcpio, mksquashfs, mkdosimg, mkext4img, mkbtrfsimg
And the helper functions they use:
truncate, loop_{attach,detach}, dm_{attach,detach},
mount/umount, estimate_size, roundup, cpio_copytree
This adds the remove() function, which works a lot like rm -rf - if you
remove() a file, it uses os.unlink, and if you remove() a directory it
uses shutils.rmtree().
We're already using find and cpio subprocesses, so using
one more subprocess is not a problem. With this approach
we can pipe cpio to the xz/gzip command, which should
help with the memory issues.
We do what create_gconf is trying to achieve in get_config_files(). What's
more, the files in crete_gconf() end up in the root directory where
nothing can possibly use them because the root user's home is now /root.
The way we create the images had to be changed a little.
Some code had to be moved to other places, so we can create
images for other architectures too.
