Commit Graph

65 Commits

Author SHA1 Message Date
Brian C. Lane
1c6b083260 Try all packages when installpkg --optional is used.
Also sort the expanded list of packages so that any failures will
be consistent instead of depending on the randomness of a set().
And add better logging when things fail.

The core issue is that repodata may have packages that match globs, but
they cannot actually be installed (eg. sigrok-firmware). This can cause
*some* of the globbed packages to be installed before hitting the
failure package.

With this change it will log the expanded list of packages if a glob is
used. It will skip any packages that fail to install when using
--optional with the glob, and continue to install the rest.

Related: rhbz#1440417
2017-05-30 09:47:37 -07:00
Colin Walters
5aa6fea8c3 templates: When a subprocess fatally errors, output its stderr directly
Right now we dump all subprocess output to `program.log`.  Unfortunately,
The pungi/koji stack doesn't know how to scrape out the lorax logs.

And even when running interactively, it's annoying that *some* fatal
errors show up on stderr, but if it's from a subprocess, I need to go
over and `tail program.log`.

Let's output the subprocess stderr directly, since the user is
going to want it prominently anyways.
2016-11-14 09:04:08 -08:00
Brian C. Lane
658df4f3f9 Fix installpkg error handling
If the query filter doesn't return anything it would just ignore the
install request instead of logging and raising an error when
required=True.

This checks for no packages matching, and if required is True raises an
error after all of the requested packages have been processed, instead
of after the first one to fail.
2016-06-24 13:40:58 -07:00
Brian C. Lane
fe17f970c9 Switch installpkg default to --required
Previous versions of lorax assumed that installpkg was optional, and
would continue on if the PKGGLOB didn't match anything. But the majority
of the packages are required so this allows the boot.iso to be built
with missing packages that are hard to track down.

It makes more sense to make the PKGGLOB required and to flag the
few exceptions to this with --optional.
2016-06-24 13:40:58 -07:00
Brian C. Lane
701ab02619 Copying same file shouldn't crash (#1269213)
When using the template install command copying the same file to itself
shouldn't crash. Just log the error and continue.

Also copy the s390 configuration files for use with livemedia-creator

Resolves: rhbz#1269213
2016-04-06 16:59:43 -07:00
Brian C. Lane
422a9a5859 ltmpl: Add compressor selection and argument passing to installimg
Allow the template to select a different compression type or arguments
for the installimg command.

On 32bit builds running inside a mock xz sees the full amount of system
memory which can result in xz failing with a memory error. This allows
the template to limit the amount of memory it tries to use.
2016-03-29 09:46:07 -07:00
David Shea
9612f9e16f Add a means of excluding packages from a glob
This makes package selection a little more roundabout, but it allows for
unused packages (and their dependencies) to be removed from globs during
the install phase.

dnf.subject.Subject is the class used by dnf's Base.install to select
packages, so the behavior of installpkg without --except options is the
same as it was before.
2016-03-22 13:25:01 -04:00
Brian C. Lane
a3097b3085 Update things to make pylint 1.5.1 happy 2015-12-16 12:59:48 -08:00
Brian C. Lane
ceabb7e214 Write a list of debuginfo packages to /root/debug-pkgs.log (#1068675)
Sometimes debugging a boot.iso requires using gdb, and finding the
corresponding debuginfo packages can be difficult. This writes the
matching -debuginfo package names and full ENVR to a file on the iso.
This can then be fed to dnf to install the correct debug packages.
2015-12-16 10:23:23 -08:00
Brian C. Lane
b3bf61bfdd Drop multiprocessing for do_transaction (#1208296)
When running the transaction in a separate process it crashes if you use
a https repo source. There's really no need for threads or processes in
lorax so drop it.

Also switched to using the DNF TransactionProgress API for progress
reporting.
2015-09-09 15:58:57 -07:00
Brian C. Lane
eb22c69f29 Additional python3 changes
Add a 'lower' filter to the templates to replace string.lower which no
longer exists. Fix udev_escape, the strings are already unicode, and
drop --chdir from runcmd. It wasn't ever used, and passing cwd to the
new runcmd isn't supported.
2015-05-15 17:11:20 -07:00
Brian C. Lane
de0e662f51 Convert pylorax to python3
Fix up 2to3 complaints. I've decided to do with wrapping list
comprehension inside list() to get the generators to run in several
places instead of list(map( or list(filter( which seem less readable to
me.
2015-05-15 17:11:20 -07:00
Brian C. Lane
a6e469e0e6 Clean up some pylint warnings
Mostly logging, using .format() on the strings.
2015-05-15 17:11:20 -07:00
Brian C. Lane
80ed5006d4 Add some more details about template rendering errors 2015-04-10 11:01:59 -07:00
Vratislav Podzimek
d2c64b1b26 Check that the transaction process is still alive
If it terminates really badly (e.g. with SIGSEGV), it doesn't report any error,
just doesn't put anything to the queue. So instead of just blindly waiting on
the queue forever, check that the process is still alive if we don't get any
message in long time interval.
2015-03-25 16:59:02 +01:00
Colin Walters
67217f5642 Add ability for external templates to graft content into boot.iso
I originally added --add-template to support doing something similar
to pungi, which injects content into the system to be used by default.
However, this causes the content to be part of the squashfs, which
means PXE installations have to download significantly more data that
they may not need (if they actually want to pull the tree data from
the network, which is not an unusual case).

What I actually need is to be able to modify *both* the runtime image
and the arch-specific content.  For the runtime, I need to change
/usr/share/anaconda/interactive-defaults.ks to point to the new
content.  (Although, potentially we could patch Anaconda itself to
auto-detect an ostree repository configured in disk image, similar to
what it does for yum repositories)

For the arch-specfic image, I want to drop my content into the ISO
root.

So this patch adds --add-arch-template and --add-arch-template-var
in order to do the latter, while preserving the --add-template
to affect the runtime image.

Further, the templates will automatically graft in a directory named
"iso-graft/" from the working directory (if it exists).

(I suggest that external templates create a subdirectory named
 "content" to avoid clashes with any future lorax work)

Thus, this will be used by the Atomic Host lorax templates to inject
content/repo, but could be used by e.g. pungi to add content/rpms as
well.

I tried to avoid code deduplication by creating a new template for the
product.img bits and this, but that broke because the parent boot.iso
code needs access to the `${imggraft}` variable.  I think a real fix
here would involve turning the product.img, content/, *and* boot.iso
into a new template.
2015-03-19 16:48:16 -04:00
Brian C. Lane
11c9e0e8ee Add removekmod template command
removekmod GLOB [GLOB...] --allbut KEEPGLOB [KEEPGLOB...]

This can be used to remove kernel modules from under
/lib/modules/*/kernel/ while keeping specific items. This should be
easier than constructing find arguments to select the right things to
save.
2015-03-06 11:26:14 -08:00
Brian C. Lane
13de711548 Drop the dnf Base object deletion code and use reset
It appears that reset+fill_sack will now do the right thing and load the
state of the installed packages. Drop the hack with deleting the object.

Also add a double-check to make sure there really is a list of files
for anaconda-core before we run off and make an image without removing
anything.
2015-02-16 08:47:30 -08:00
Brian C. Lane
345cc89ee0 dnf: remove files from installed packages
This is a workaround for a current dnf bug, it doesn't update the state
of the packages after they are installed so we tear down the base dnf
object and create a new one pointing to the installroot.

There is an additional issue with the list of files returned, hawkey and
dnf don't appear to make a distinction between files, dirs and ghosted
dirs like yum did, this can result in too much being removed (eg. all of
/etc/selinux/) so we only remove files not directories.
2015-02-16 08:47:30 -08:00
Brian C. Lane
431ca6cea4 Switch lorax to use dnf instead of yum
pylorax users will need to change to using dnf and pass a dnf.Base()
object as the dbo argument instead of a yum object as the yum or ybo
argument. See the lorax script for an example of how to do this.

The lorax cmdline argument --excludepkgs has been removed since dnf
doesn't appear to have any way to support it and packages should be
controlled using templates anyway.
2015-02-16 08:47:30 -08:00
Brian C. Lane
b064ae6166 Add installimg command for use in the templates
installimg SRCDIR DESTFILE
  Create a compressed cpio archive of the contents of SRCDIR and place
  it in DESTFILE.

  If SRCDIR doesn't exist or is empty nothing is created.

Examples:
  installimg ${LORAXDIR}/product/ images/product.img
2014-11-05 09:58:51 -08:00
Brian C. Lane
d25c0b0594 Update copyright statements 2014-05-09 15:13:39 -07:00
Brian C. Lane
6d47689a33 Cleanup pylorax pylint warnings 2014-05-09 08:10:41 -07:00
Brian C. Lane
9e15705bb9 Clean up download and install output
Commit d2ae92b4b3 patched up the download counter and progress
display. Yum no longer provides the needed information so now we get the
total number of packages from the start of the transaction.

This also turns off colors when stdout is not a tty, and only prints the
install progress once so that piping to a logfile isn't flooded with
useless characters.
2014-04-25 06:55:56 -07:00
Martin Gracik
62920272c7 Fix rexists 2012-12-19 12:39:39 +01:00
Dennis Gilmore
78e3d0f115 add installupgradeinitrd function and use it to install the upgrade initrds 2012-11-30 13:28:38 -06:00
Martin Gracik
91afe5883f Set permissions on the initrd (#863018)
Dracut now makes the initrd with 600 permissions
for security reasons. These reasons do not apply
to install images, and we want the other tools
that use lorax to be able to read the initrd file.
2012-10-11 10:24:11 +02:00
Brian C. Lane
43bd549480 add convienience functions for running commands
Adds runcmd() and runcmd_output() which set raise_err=True
2012-09-17 11:26:53 -07:00
Brian C. Lane
12ad7d0d99 restore CalledProcessError handling
When I switched execution over to execWith* functions I failed to
account for the use of CalledProcessError in various places. This
patch restores that behavior. All places that used check_call or
check_output now pass raise_err=True to the execWith* call.
2012-09-17 11:26:50 -07:00
Brian C. Lane
134eec24d5 clean up command execution
Switch to using execWith* so that the command and its output can be
logged. To capture the output setup a logger named "program"

livemedia-creator captures all of this into program.log
2012-07-30 09:22:10 -07:00
Will Woods
3a75b2e07d add 'systemctl' command and use it in postinstall
The 'systemctl' command can be used to enable, disable, or mask systemd
units inside the runtime being modified. Modify runtime-postinstall.tmpl
to use the 'systemctl' command.

We also no longer remove quota*.service or kexec*.service, since
these aren't enabled by default. And systemd-remount-api-vfs.service
should work correctly now, so we can leave it alone as well.
2012-06-19 20:00:00 -04:00
Will Woods
3fda56b499 Add some documentation to LoraxTemplateRunner 2012-06-19 19:59:59 -04:00
Will Woods
daacb4465d fix '-runcmd' and improve logging
The '-cmd' functionality depends on the individual lorax template
commands raising errors, so they shouldn't do sys.exit().

Also, capture stderr along with stdout, and put both in the log.
2012-06-19 19:59:59 -04:00
Martin Gracik
cbe001638e Verify the yum transaction
Check if all packages that were supposed to be installed,
ended up installed.
2012-06-04 16:14:38 +02:00
Martin Gracik
7446d0d1e2 Add a --required switch to installpkg
Some packages are critical to the compose. If --required
is specified in the template's installpkg command, lorax
will exit if the package is not available.
2012-06-04 16:14:38 +02:00
Martin Gracik
b61927badc Log all removed files
This makes finding what glob removed some file easier.
2012-04-12 08:28:58 -07:00
Brian C. Lane
ac6512a531 Fix missing CalledProcessError import 2012-02-17 08:32:00 -08:00
Dan Horák
17179f2120 use internal implementation of the addrsize utility
Anaconda dropped the addrsize utility in F-16, so we need to implement
it directly in lorax.
2012-02-09 14:08:56 +01:00
Martin Gracik
681f67f954 Log output of failed command
If we run a command from template with runcmd,
log it's output if it exits with non-zero return code.
2012-01-19 18:13:03 +01:00
Will Woods
5e411b09c0 consider %ghost files part of the filelists in templates
When considering a package's filelist (e.g. for things like removefrom
--allbut), we might need to also include the %ghost files, so make
_filelist() add them to its returned list.
2012-01-05 16:15:54 -05:00
Will Woods
c8f06bd8ff ltmpl: be more careful about checking command names 2011-11-04 13:41:10 -04:00
Will Woods
dfa7742a6d default to fatalerrors=True in LoraxTemplateRunner
This means that any (non-ignored) command error will cause lorax to
exit.

Do note, however, that some commands (e.g. remove, installpkg) don't
raise exceptions and therefore will not cause lorax to exit.
2011-10-26 13:09:50 -04:00
Will Woods
0e64f08ff8 Add '-cmd' syntax to ignore errors, drop copyif/moveif
Makefile-style "-cmd" syntax lets us run a command and ignore any
resulting errors. This is a more general version of what copyif/moveif
were trying to accomplish, so we can drop those commands.
2011-10-26 13:08:39 -04:00
Will Woods
305476eefe add comment about symlink root-escape problem 2011-10-25 16:19:23 -04:00
Will Woods
cd1225c5fb minor comment cleanups 2011-10-24 16:27:36 -04:00
Will Woods
262a05e539 ltmpl: continue loop if installpkg can't find something
Otherwise, something like:
  installpkg missing-thingy kernel kernel-firmware
will leave you without a kernel.
2011-09-15 19:27:31 -04:00
Will Woods
318c843e37 ltmpl: improve exception handling/reporting
- Log the error message (and the line causing the error) to console
- Log the whole exception (with some excess junk removed) to debuglog
- don't immediately SystemExit if the template won't parse
- clean up some comments
2011-09-15 19:24:35 -04:00
Will Woods
473f01c696 Add docstrings for the ltmpl commands in LoraxTemplateRunner 2011-09-14 18:33:30 -04:00
Will Woods
faafd04dcb add '--allbut' flag to 'removefrom'
This lets us easily do whitelisting instead of blacklisting during
runtime cleanup. For example:
 removefrom xfsprogs --allbut /sbin/* /usr/sbin/xfs_admin
would remove everything from the xfsprogs package except files in /sbin
and /usr/sbin/xfs_admin.

A few things in runtime-cleanup have been converted to use --allbut. The
only difference in the created runtime image is that we're deleting
/usr/share/kde4 from fedora-logos.
2011-08-31 19:40:20 -04:00
Will Woods
916402b023 handle templatedir more cleanly in ltmpl/treebuilder
make sure that runner.templatedir gets set to a useful default if
initialized with None, and and make sure we pass templatedir to
LoraxTemplateRunner.__init__ (so it'll get initialized properly)
2011-08-08 19:01:38 -04:00