Commit Graph

1086 Commits

Author SHA1 Message Date
David Shea c941b82b0c Add a new output type, tar-disk.
This option will create an optionally compressed tarball containing a
disk image. This format is used by Google's Compute Engine.

This also adds a new option, tar_disk_name, to set the name of the disk
image that will be wrapped in the final tarball. opts.image_name
continues to be the final output file name.
2019-04-02 10:15:42 -04:00
David Shea c585b91422 Support compressing single files.
Modify imgutils.compress to allow the "rootdir" argument to be either a
directory or a single file to add to an archive.
2019-04-02 10:15:42 -04:00
David Shea 8ef4f431d8 Add an option to align the image size to a multiplier.
If provided, round the disk image size up to a multiple of the value.
This allows for image formats with specific size-alignment requirements
(e.g., disk size must be in GiB).
2019-04-02 10:15:42 -04:00
Brian C. Lane 049f68cb60 lorax-composer: Fix installing files from [[repos.git]] to /
rpmfluff was including / in the rpm, which conflicts with
filesystem.rpm

The rpm globs are pretty limited, and we don't actually know the file
paths until later, so we have to use a glob or a directory.

So when the destination is / it now uses /* to select all the files and
sub-directories in the archive. The limitation of this is that it cannot
support dotfiles directly under /, they will cause a rpmbuild error.

For destinations other than / it uses the name of the directory, so
dotfiles are fine in that situation.
2019-03-29 08:45:39 -07:00
Brian C. Lane d178189ef3 qemu wasn't restoring the terminal if it was terminated early
You would need to run reset to regain control of your terminal after
this happened, so this turns off the monitor and serial port mux to
stdout.
2019-03-26 13:33:47 -07:00
Brian C. Lane 6c6e4fced1 Switch the --virt-uefi method to use SecureBoot
This updates the qemu arguments so that it will actually work, and
switches to using SecureBoot OVMF firmware.
2019-03-26 13:32:12 -07:00
Brian C. Lane 5dea308080 lorax-composer: pass customization.kernel append to extra_boot_args
This allows iso builds to include the extra kernel boot parameters by
passing them to the arch-specific live/*tmpl template.

Also adds tests to make sure it is written to config.toml in the build
metadata.
2019-03-26 11:06:57 -07:00
Brian C. Lane f9665940bb Improve logging for template syntax errors
The shlex splitting can fail, resulting in error messages like:

ERROR livemedia-creator: No closing quotation

without any context in the log files. This logs the line that failed to
be split and expanded.
2019-03-26 11:06:57 -07:00
Brian C. Lane 235813212f Add extra boot args to the livemedia-creator iso templates
This adds a new livemedia-creator argument, --extra-boot-args, which can
be used to add arguments to the kernel command-line in the templates.
2019-03-26 11:06:57 -07:00
Brian C. Lane 59464286f9 lorax-composer: Add the ability to append to the kernel command-line
Sometimes it is necessary to modify the kernel command-line of the
image, this adds support for a [customizations.kernel] section to the
blueprint:

[customizations.kernel]
append = "nosmt=force"

This will be appended to the kickstart's bootloader --append argument.

Includes tests for modifying the bootloader line, the kickstart
template, and examining the final-kickstart.ks created for a compose.
2019-03-26 11:06:57 -07:00
Brian C. Lane 61efa91a03 rpmgit: catch potential errors while running git
Log them and report them as RuntimeError. Also add a couple tests for
them.
2019-03-15 11:18:12 -07:00
Brian C. Lane cd8c884adb Add repos.git support to lorax-composer builds
This hooks up creation of the rpm to the build, adds it to the
kickstart, and passes the url to Anaconda. The dnf repo with the rpms is
created under the results directory so it will be included when
downloading the build's results.
2019-03-15 11:18:12 -07:00
Brian C. Lane f6f2308765 Add pylorax.api.gitrpm module and tests
This handles creating the rpm from the dictionary describing the
repository and rpm. Also adds tests for archive and rpm creation.
2019-03-15 11:18:12 -07:00
Brian C. Lane d7b96c8f0f Add support for [[repos.git]] section to blueprints
This adds support, documentation, and testing for a [[repos.git]]
blueprint section that can be used to install files from a git
repository. It will create an rpm that will be added to the build,
and included in the metadata that can be downloaded. This allows you to
accurately keep track of the source of configuration files and extra
metadata that is added to the build.

The source repo and reference will be listed in the rpm's summary making
it easy to discover on the installed system.
2019-03-15 11:18:12 -07:00
Brian C. Lane d32f477e0b lorax-composer: Return UnknownBlueprint errors when using deleted blueprints
Reading a blueprint wasn't checking to see if it had been deleted so it
was returning the most recent commit before it had been deleted. This
allowed things like starting a compose with a blueprint that technically
doesn't exist.

One exception to this is the /changes/ route, it must be available so
that you can use the commit hash to undo a delete.

This also adds tests for the various operations.

Resolves: rhbz#1682113
2019-03-01 14:39:21 -08:00
Brian C. Lane 26bd2c1378 lorax-composer: Delete workspace copy when deleting blueprint
Also extends the blueprint delete test to also check the workspace.
2019-03-01 14:39:21 -08:00
Brian C. Lane 3b8de2a233 Move the package requirements for live-iso setup out of the template
In order to support iso creation on multiple arches with the templates
we need to be able to select different packages based on arch.
lorax-composer uses the arch-specific Lorax templates in order to
generate the output iso so this patch:

1. Creates a new template and type to parse it, live-install.tmpl
   which contains only installpkg commands and #if clauses for arch
2. Removes bootloader related packages from the live-iso.ks
3. Remove dracut-config-rescue exclusion because it can cause problems
   with some blueprints.
4. Switch logo requirement to system-logos which is satisfied by
   generic-logos or fedora-logos. This prevents conflicts when a blueprint
   installs fedora-release-workstation.

So in the future, if x86.tmpl, etc. need a new package to support
creating the iso it should be added to the correct section in
./share/live/live-install.tmpl
2019-02-25 13:49:12 -08:00
Brian C. Lane 6c5e89ed14 Add LiveTemplateRunner to parse per-arch live-iso package requirements
This template runner only uses the installpkg command to gather a list
of package NEVRA to be installed to support live iso creation.
2019-02-25 13:49:12 -08:00
Brian C. Lane 8eaad3bc5e Move the run part of LoraxTemplateRunner into new TemplateRunner class
This will make it easier to add a new subclass that only handles
installpkg for use with livemedia-creator and lorax-composer iso
creation.
2019-02-25 13:49:12 -08:00
Brian C. Lane b47554d716 livemedia-creator: Add support for reqpart kickstart command
reqpart can be used to make kickstarts more platform agnostic, creating
needed partitions without lmc having to keep track of the arch-specific
needs. eg. ppc64 needs prepboot and /boot

This increases the size of the disk based on whether reqpart or
reqpart --add-boot is in the kickstart.

Note that this is only valid for partitioned disk output types, not
for filesystem images or live iso output.
2019-02-25 13:49:12 -08:00
Brian C. Lane 6443f34337 Drop _unique_dicts function
It is not actually needed. projects_info deduplicates the package list,
placing other builds into the builds list instead of making a new
package entry. So it returns a sorted and deduped list of packages, as
expected.
2019-02-21 08:59:29 -08:00
Brian C. Lane 9041174142 Add some extra cancel_func protection to QEMUInstall
In livemedia-creator's usage of this it can never pass in None, but if
someone were to import the library and use it, it would crash with
NoneType. So add the extra checks to make sure cancel_func isn't None,
just in case.
2019-02-21 08:54:46 -08:00
Yuval Turgeman 1c731b5618 installer: make sure cancel_func has a value (#612)
When using LMC to virt-install a system to an image, cancel_func is not
provided in run_creator, causing a TypeError (NoneType object is not
callable).

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
2019-02-19 08:09:36 -08:00
Brian C. Lane 7e78dc368f Remove 3G minimum from lorax-composer
The reason for the 3G minimum was because anaconda had a bug with how it
calculated minimum disk size when using kickstart. The gix for this has
been in Anaconda since 29.19-1, so we can now remove our limit and
create somewhat smaller disk images.
2019-02-14 15:20:50 -08:00
Lars Karlitski 6961dce08b installer: Run anaconda in a mount and pid namespace
Anaconda can leave child processes and mounts around when it crashes or
is canceled before finishing. It also sometimes unmounts unrelated file
systems (https://github.com/rhinstaller/anaconda/issues/1791).

Run it in a mount and pid namespace to clean up after it.
2019-02-12 10:30:58 +01:00
Chris Lumens 022e9eba3e Run as root/weldr by default.
We need to be root to read the certificates that give access to the
package repos.  Right now, the alternative seems to be changing
permissions on the certs themselves, which seems less good.  We're
running anaconda as root anyway.
2019-02-11 16:20:39 -08:00
Lars Karlitski e194b5926c Pass ssl certificate options to anaconda
If a repository has `sslcacert`, `sslclientcert`, or `ssclientkey` set,
pass them to anaconda through the kickstart file. This is mostly the
case when using RHEL repositories that are accessed through a
subscription.
2019-02-11 16:20:39 -08:00
Brian C. Lane 98482e444d Remove duplicate repositories from the sources list
In some cases when the host has, for whatever reason, multiple copies of
the same repo listed the build may fail with an error about running out
of space.

So this commit removes duplicate entries after the host's repos have been
loaded. It also adjusts some of the test repos to use different
temporary repo names for the tests.
2019-01-30 08:30:40 -08:00
Brian C. Lane 3676cb65bb Clarify the ks repo only error message
This also moves the run_creator kickstart checks into check_kickstart
so that tests may be added.

This will close #164
2019-01-29 14:03:11 -08:00
Brian C. Lane e4fe1aab32 lorax: Move default tmp dir to /var/tmp/lorax
If systemd's tmpfiles.d timer is executed while lorax is running it will
remove any files and directories older than 30 days. This is what has
been causing the occasional error where /proc/ would seem to vanish
during the install.

Upstream has proposed this solution, https://github.com/systemd/systemd/pull/11482
but until that is released we need a work-around to protect the lorax
files.

This commit does several things:

* Move the default tmpdir from /var/tmp/ to /var/tmp/lorax/
* Add a lorax.conf tmpfiles.d file that prevents systemd-tmpfiles from
  removing anything under /var/tmp/lorax/
* Add an exit handler to lorax so that temporary directories are removed on
  exit or on a python traceback.
* Use flock to lock access to the tempdir while lorax is running.
* Remove any unlocked tempdirs named /var/tmp/lorax/lorax.* at startup

Note that the exit handler will not remove the tempdir if lorax is
killed with a signal -- those are being caught by dnf and prevent the
exit handler from running.

systemd-tmpfiles cannot clean up the tempdirs at boot time because they
contain files labeled as shadow_t, so we have to remove those when lorax
runs. It uses the flock to prevent removing any directories created by
parallel instances of lorax and only removes ones that are unlocked.
Worst case they will be around until the first run of lorax after a
reboot.

If you want to keep the working directory around for debugging purposes
use --workdir /var/tmp/lorax/my-workdir and it won't be removed by
lorax.
2019-01-29 13:57:12 -08:00
Adam Williamson 2d3f266373 Don't exclude /dev from the `setfiles` in `novirt_install`
After a novirt disk image install, we run `setfiles` in the
install root to ensure some SELinux contexts are correct. /dev
is currently excluded from this run. However, as reported and
discussed in https://bugzilla.redhat.com/show_bug.cgi?id=1663040
it seems that with a recent systemd change, startup of many
services will fail if /dev itself is incorrectly labelled, and
in current Rawhide live images, it *is* incorrectly labelled.
Including `/dev` in this setfiles command appears to resolve the
problem in my testing.

Resolves: rhbz#1663040

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2019-01-18 09:13:52 -08:00
Brian C. Lane 4fe21135e3 Turn off pylint warning about docstring with backslash 2019-01-08 11:37:56 -08:00
Anthony F McInerney f66bff5aa7 fixes #543 qemu -nodefconfig deprecated 2019-01-08 10:19:00 -08:00
Anthony F McInerney 6bb64f94ff fix spinx build warnings 2019-01-08 10:17:44 -08:00
Brian C. Lane 7ee3b3d661 Revert "lorax-composer: Cancel running Anaconda process"
Drop running pkill. This causes problems if more than one is running on
a system (eg. in parallel using mock). It can kill off other processes
unrelated to this instance of anaconda.

This reverts commit 6b5c4df8b5.
2019-01-08 10:14:29 -08:00
David Shea 35ab6a1336 Allow customizations to be specified as a toml list
Support both

  [customizations]
  hostname = "whatever"

and

  [[customizations]]
  hostname = "whatever"

in the blueprint data. The [[ syntax matches the other customization
directives (user, group, sshkey), and as such it's easy to accidentally
use it for the hostname without even realizing it's specifying something
different.

Add some tests for converting customizations to kickstarts.
2019-01-08 10:21:07 -05:00
Brian C. Lane ca2c3d9e77 Make sure cancel_func is not None 2019-01-07 13:53:44 -08:00
Dan Horák a477db33d6 drop big endian ppc/ppc64 support 2019-01-07 13:52:46 -08:00
Dan Horák 34ab13b68a add qemu command mapping for ppc64le 2019-01-07 13:52:46 -08:00
Dan Horák d34acfc14c don't reduce initrd size on ppc64/ppc64le 2019-01-07 13:52:46 -08:00
Brian C. Lane fd173f7265 Add timestamps to program.log and dnf.log
In lorax, lorax-composer, and livemedia-creator.
2018-12-19 12:13:24 -08:00
Brian C. Lane 6f141658d4 lorax: Save information about rootfs filesystem size and usage
Run df on the filesystem image after it has been created.
Output will be in program.log, eg:

Running... df /var/tmp/lorax.imgutils.wm04pg_v
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/loop0       1998672 1619508    362780  82% /var/tmp/lorax.imgutils.wm04pg_v
Return code: 0
2018-12-17 16:24:23 -08:00
Brian C. Lane 6b5c4df8b5 lorax-composer: Cancel running Anaconda process
It ends up that this isn't as easy as you'd think. Anaconda sets up some
signal handlers to handle cleanly exiting, but they are not being run
when sent a TERM after package installation has started. I think DNF
resets them causing it to get ignored.

When the cancel is sent it can take several minutes for it to have an
effect. In my testing it usually takes around 2 minutes for anaconda to
notice and exit.

This sends a TERM to the process and then waits for it to exit. When it
returns it then removed any device-mapper devices that were setup for
image installations, removes any hanging loop devices.

It then kills off any process with pyanaconda. in the cmdline, and
anaconda-bus.conf (because anaconda starts a bunch of helpers and if it
doesn't shut down cleanly they remain running).

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane 26a684932b Add cancel_func to virt and novirt_install functions
In addition to monitoring the logs for errors, call a function (or
functions) that tell it to cancel the anaconda process and cleanup.

Also check for a cancel after creating the squashfs image for live-iso
since that's a long running process.

This required adding a new argument to a number of existing functions,
passing it down to QEMUInstall and novirt_install where the function is
called.

Resolves: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane 3762264f56 lorax-composer: Check for STATUS before deleting
When there is no run or new symlink do one last check to make sure no
STATUS file was written. If it is missing, go ahead and remove the
results directory.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane fa461dc837 Check for existing CANCEL request, and exit on FINISHED
If another CANCEL request has already been made just exit from
uuid_cancel. If the build is FINISHED before it times out just exit,
don't remove the finished results.

Related: rhbz#1656691
2018-12-17 16:10:04 -08:00
Brian C. Lane d18934775c lorax-composer: Handle packages with multiple builds
When the repository has multiple arches, eg. i686 and x86_64, it should
add a new entry to the project's builds list, not create a new project
in the list.

This handles that by adding a modified insort_left function and
examining the packages returned from dnf to make sure they aren't
already listed in the results. It also handles adding them in sorted
order so that no further sorting needs to be done on the results.

Resolves: rhbz#1656642
2018-12-06 15:48:58 -08:00
Brian C. Lane 4dd9004d13 lorax-composer: Check the queue and results at startup
If the system ran out of space, or was rebooted unexpectedly, the state
of the queue symlinks, or the results STATUS files may be inconsistent.
This checks them and:
 * Removes broken symlinks from queue/new and queue/run
 * Removes symlinks from run and sets the build to FAILED
 * Sets builds w/o a STATUS to FAILED
 * Sets builds with STATUS of RUNNING to FAILED
 * Creates missing queue/new symlinks to results with STATUS of WAITING

So, any builds that were running during the reboot will be FAILED, and
any that were waiting to be started will be started upon rebooting.

Resolves: rhbz#1647985
2018-12-06 15:13:28 -08:00
Brian C. Lane 8b11705ea0 Remove setfiles from mkrootfsimage
SELinux applies the correct labels, setfiles is no longer needed.
This allows lorax to run with SELinux in Enforcing mode.
2018-11-29 11:06:27 -08:00
Brian C. Lane 35b8957f12 Remove SELinux Permissive checks
Anaconda, Lorax, lorax-composer, and livemedia-creator can all now run
with SELinux in Enforcing mode. It does not need to be disabled and if
there are denials they should be reported as a bug.

Log the current state of SELinux when starting, update the
documentation.
2018-11-29 11:06:26 -08:00
Brian C. Lane c11bffc357 Add --no-system-repos to lorax-composer
Running lorax-composer --no-system-repos will prevent it from copying
the dnf repositories from /etc/yum.repos.d/ into the lorax-composer repo
directory. It will *only* use repositories setup using the sources api
or written to /var/lib/lorax/composer/repos.d/

If lorax-composer has previously been run without this switch the system
repos will need to be removed from the composer/repos.d/ directory. It
would also be a good idea to remove the cached metadata in
/var/tmp/composer/

Resolves: rhbz#1650363
2018-11-26 17:05:08 -08:00
Brian C. Lane d69f01d4a8 Create a kpartx_disk_img function
So that it can be used in tests as well as in imgutils
2018-11-12 11:13:37 -08:00
Brian C. Lane 8e749efbbf Fix make_appliance and the libvirt.tmpl
Apparently nobody has used these since the switch to py3, xrange is now
range and it needs to read the file in binary mode when generating the
sha256.
2018-11-12 11:13:37 -08:00
Brian C. Lane a4783ba29f Add tests for ltmpl.py
This covers things like installing globbed package names from multiple
repos, pinned package versions, and ltmpl functions

Related: rhbz#1548586
2018-10-29 13:55:19 -07:00
Brian C. Lane befa3128d7 Move get_dnf_base_object into a module
This allows it to be imported by tests.
2018-10-29 13:49:54 -07:00
Marek Marczykowski-Górecki 27e611629f Add --squashfs-only option to drop inner rootfs.img layer
Make runtime directly into squashfs image. This reduces largely
unreproducible ext4 layer, but requires anaconda's dracut module
modification to properly mount the image.
2018-10-29 13:16:38 -07:00
Brian C. Lane e9e5139750 dnf changed the type of gpgkey to a tuple
And in an intermediate version it returns a VectorString object which
isn't serializable by the json or toml modules.

So convert it to a list so that the type is consistent in the sources
code.
2018-10-24 10:22:40 -07:00
Brian C. Lane 5409748e75 lorax: Log when SOURCE_DATE_EPOCH is used for the current time 2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki de8124366e Use SOURCE_DATE_EPOCH for volumeid of efi boot image
By default mkfs.mksdos choose volume id based on current time. If
SOURCE_DATE_EPOCH is set, use that instead.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki e7f45d333f Preserve timestamps when building fs image
Even when FS do not support owner/modes, preserve timestamps.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Marek Marczykowski-Górecki 876ec52215 Use SOURCE_DATE_EPOCH for metadata timestamps
This include .buildinfo, .treeinfo and .discinfo.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
2018-10-19 14:43:29 -07:00
Brian C. Lane 7e0a288f5b Update depsolving with suggestions from dnf (#1636239)
The previous method worked, but wasn't exactly idiomatic. This is more
correct, and appears to work the same (templates depsolve, version globs
work, multiple repos work).

Note that this does use a private dnf attribute ._goal, but the word is
that this is going to become a public api soon, so yes it is there on
purpose.
2018-10-12 12:00:25 -07:00
Brian C. Lane faa65bca3c Disable false context-manager pylint error 2018-10-12 12:00:25 -07:00
Brian C. Lane d7951e3a10 Fix directory creation for blueprints
Depending on how lorax-composer is run setting up an empty blueprints
directory can fail. So this moves checking/creation until after the
other directories are created and uses make_owned_dir to make sure
ownership is correct.
2018-10-12 11:59:32 -07:00
Brian C. Lane 65b769984b Change make_dnf_dirs to be run as root
It needs to be root in order to set the ownership and permissions on the
directories that are under /var/lib/lorax/composer/

Refactor the directory creation into a utility function, and use a umask
of 0o006 to ensure that the parent directories created do not have o+rw
set on them (makedirs behavior is different between Python 3.6 and 3.7
so umask of 0 doesn't work consistently).
2018-10-12 11:59:32 -07:00
Brian C. Lane 98f8b23129 Add an openstack image type
This is a qcow2 image with cloud-init in the template.
2018-10-09 10:17:14 -07:00
Brian C. Lane c017300ad6 Revert "Rename composer-cli to composer"
This reverts commit c9be91c879.

It conflicts with the PHP dependency manager project named 'composer'
2018-10-08 09:23:47 -07:00
Brian C. Lane 3b41faae2c Work around dnf problem with multiple repos
If a package is in multiple repos dnf may return more than 1 of them
when using best...glob so we pick the highest NEVRA one and install
that.

Related: rhbz#1636239
2018-10-05 11:23:41 -07:00
David Shea 9717b3fd98 Make no-virt generated images sparser
At the end of disk image installs, use fstrim on the generated filesystem to
discard any blocks that were allocated during the install and are now unused.
This will allow tools such as qemu-img to create images that do not include
deleted data.

For raw disk images that do not go through qemu-img, use fallocate --dig-holes
to create sparse holes in place of the unused blocks.
2018-10-05 11:24:18 -04:00
Brian C. Lane 6da3079349 Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-10-03 16:41:49 -07:00
Lars Karlitski 6a37f80f0f cli: Clarify error message for unprivileged access
`os.path.exists("/run/weldr/api.socket")` returns False for users which have no
access. This leads to composer printing that the file does not exist, which is
misleading.

Since it's no possible to distinguish the two cases, fix this problem by
combining them and showing a single error message.
2018-10-03 16:36:51 -07:00
Brian C. Lane e7bfab8b4a Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-03 08:30:31 -07:00
Adam Williamson 37f264c010 Don't try to append to DNF config value that can't take it
See https://bugzilla.redhat.com/show_bug.cgi?id=1595917 and
https://github.com/rpm-software-management/dnf/pull/1200 for
more on this. Briefly, DNF before 3.0 presented this config
value as a list...and mutating it worked. DNF from 3.0 until
3.6 presented it as a list...mutating it didn't work, but also
didn't *fail*, so this has actually not been doing anything on
DNF 3.x but we haven't noticed.

In DNF 3.6 values like this are presented as tuples instead of
lists, to try and catch usages like this, and it worked! We
need to change this one.

There is an additional weirdness here. tsflags is actually, in
libdnf terms, an OptionStringListAppend option: that means that
when something tries to *set* its value, the new value is just
appended to the existing list of values. This is very weird
behaviour when you're interacting with it like this, but
happens to be quite useful, as we can just 'set' the value to
a list like this and it will actually get appended (which is
what we want), and this one syntax happens to work correctly in
DNF 2.x, 3.0 through 3.5.1, and 3.6.

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-10-02 14:07:06 -07:00
Brian C. Lane 51c73f7570 Always update repo metadata when building an image
When the kickstart is handed off to Anaconda for building it will
download its own copy of the metadata and re-run the depsolve. So if the
dnf cache isn't current there will be a mismatch and the build will
fail to find some of the versions in final-kickstart.ks

This adds a new context to DNFLock, .lock_check, that will force a check
of the metadata. It also implements its own timeout and forces a
refresh of the metadata when that expires because the dnf expiration
doesn't always work as expected.

Resolves: rhbz#1631561
2018-10-02 13:59:42 -07:00
Brian C. Lane b2eadff77c Use rootpw for setting the root password instead of user
Ends up you cannot use the kickstart user command on root, since it
already exists, so we have to translate that into a rootpw command.

So [[customizations.user]] with name = "root" only support key, which
will set the ssh key, and password which will use rootpw to set the
password. plain text or encrypted are supported.

Related: rhbz#1626122
2018-10-02 13:59:42 -07:00
David Shea 1056bfc25b Add a vmdk compose type.
This is similar to the AMI type, but also adds open-vm-tools and does not do
anything special to the partitioning
2018-10-02 11:54:41 -04:00
David Shea e0c236ff36 Add a vhd compose type for Azure images
This does pretty much the same things as the AMI compose type, but also
replaces NetworkManager with the Azure linux agent.
2018-10-02 11:54:41 -04:00
David Shea 18188bf6cf Add an ami compose type for AWS images
This differs from lmc's --make-ami in that creates a full disk image instead of
an fsimage. Create a raw disk image with a / and /boot partitions, and enable
sshd, chronyd, and cockpit by default.
2018-10-02 11:54:38 -04:00
David Shea 847fff4e11 Remove --fstype from the generated part line
Instead of specifying the fstype, just let anaconda use the default.
2018-09-28 15:27:15 -04:00
Lars Karlitski 160044ba9d Fix pylint errors and warnings
Remove `except` block which immediately raises the same exception again (it's
not a subclass of another caught exception, so this is safe).

Remove a false positive, because it is not emitted from the code base.

Disable subprocess-popen-preexec-fn in startProgram, which is not used
internally.
2018-09-25 13:49:40 +03:00
Lars Karlitski c9be91c879 Rename composer-cli to composer 2018-09-20 12:02:03 +03:00
David Shea c00036251e Fix a DeprecationWarning
SafeConfigParser is just a deprecated version of ConfigParser in
python3, so use ConfigParser.
2018-09-07 13:34:30 -04:00
Brian C. Lane a0aab15311 Ignore a pylint warning about UnquotingConfigParser get args
The args differ, but we are accepting and passing through all args so
it's ok.

Related: rhbz#1613058
2018-09-04 11:16:39 -07:00
Adam Williamson 5c0dfb74e6 Ditch all use of pyanaconda's simpleconfig
lorax uses pyanaconda's SimpleConfigParser in three different
places (twice with a copy that's been dumped into pylorax, once
by importing it), just to do a fairly simple job: read some
values out of /etc/os-release. The only value SimpleConfigParser
is adding over Python's own ConfigParser here is to read a file
with no section headers, and to unquote the values. The cost is
either a dependency on pyanaconda, or needing to copy the whole
of simpleparser plus some other utility bits from pyanaconda
into lorax. This seems like a bad trade-off.

This changes the approach: we copy one very simple utility
function from pyanaconda (`unquote`), and do some very simple
wrapping of ConfigParser to handle reading a file without any
section headers, and returning unquoted values. This way we can
read what we need out of os-release without needing a dep on
pyanaconda or to copy lots of things from it into pylorax.

Resolves: #449
Resolves: #450

Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-08-30 16:21:09 -07:00
Brian C. Lane 4d9ef60460 Add support for DNF 3.2 module_platform_id config value
This borrows simpleconfig.py from Anaconda to make parsing os-release
easier.

It defaults to platform:el8
2018-08-29 11:04:50 -07:00
Brian C. Lane 07acd2e780 lorax: Only run depmod on the installed kernels
In the near-future there may be /lib/modules/ directories for older
kernels with weak dependencies listed. These may not match the installed
kernel(s) so we cannot depend on them to drive generate_module_data.

Instead use the existing findkernels() function to get the list of
installed kernels and iterate those, running depmod on them.

Resolves: rhbz#1622213
2018-08-28 16:56:50 -07:00
Brian C. Lane 57674c9a1a Fix composer-cli blueprints changes to get correct total
blueprints/changes is different, each blueprint has it's own total,
limited by the call's limit. So it needs to find the max total of all
the requested blueprints.
2018-08-27 11:08:04 -07:00
Brian C. Lane ec908fcd2a Fix blueprints/list and blueprints/changes to return the correct total 2018-08-27 11:08:01 -07:00
Brian C. Lane 0a76d635ca Add a function to get_url_json_unlimited to retrieve the total
The blueprints/changes API is a bit different from the others, the total
that it includes is for each blueprint, not one total for all of them,
since there will be a different number of commits for each.

The function is passed the dict, and it can be used to select the total
to use for retrieving all of the results. If it isn't included it will
use data["total"] which works fine in most cases.
2018-08-27 11:03:59 -07:00
Brian C. Lane 86d556e87c Use urllib.parse instead of urlparse
python3 moved this to a new module.
2018-08-21 16:49:41 -07:00
David Shea ee98d87cea In composer-cli, request all results
Add a limit argument to all potentially paginated results, equal to
whatever the composer backend is the total number of results. This still
has the potential to provide truncated data if the number of results
increases between the two HTTP requests.

Resolves: #404
2018-08-21 16:43:44 -07:00
David Shea deff4d325f Allow '*' as a uuid in /compose/status/<uuid>
This will display all UUIDs that match the filter arguments
2018-08-21 16:42:18 -07:00
David Shea 40f23f093d Add filter arguments to /compose/status
This adds the following optional arguments to the /compose/status route:

  - type, matches the compose_type field
  - status, matches the queue_status field
  - blueprint, matches the blueprint field
2018-08-21 16:42:07 -07:00
Brian C. Lane 844ff9998b composer-cli should not log to a file by default
The user can pass --log /path/to/logfile.log if they want logging
enabled.
2018-08-21 11:49:28 -07:00
Lars Karlitski be5d50e6f3 Set TCP listen backlog for API socket to SOMAXCONN
A value of 1 is too low for heavy users of the API, such as the weldr-web
interface.

This is also systemd's default for sockets it opens. Using lorax-composer with
socket activation already results in a backlog of SOMAXCONN connections.
2018-08-20 14:54:07 -07:00
Chris Lumens 47d3a72014 Fix a little bug in running "modules list".
(cherry picked from commit 5dddef6f49)
2018-08-10 12:00:23 -04:00
Brian C. Lane c8283adc17 Move disklabel and UEFI support to compose.py
Currently we are making MBR disk images for qcow2 and partitioned disk,
so the UEFI packages aren't required at this point.

Move the clearpart command into compose.py so that in the futute it can
use clearpart --disklabel to create a GPT image, and add the required
packages to the package set.
2018-08-09 15:17:27 -07:00
Chris Lumens faba254be5 Change INVALID_NAME to INVALID_CHARS.
It's getting used in a whole lot more places now, so make it a bit more
generic sounding.

(cherry picked from commit bf0318eab6)
2018-08-09 16:20:23 -04:00
Chris Lumens 473ee99eb9 Update composer-cli for the new error return types.
(cherry picked from commit bc96f75992)
2018-08-09 16:20:17 -04:00
Chris Lumens d03a198a8c Add default error IDs everywhere else.
The idea here is to make sure all return points have the same type for
the error cases.  There's not really all that many, so they just go in
one patch.  Some of these could potentially turn into more specialized
errors later.

(cherry picked from commit fd901c5e3f)
2018-08-09 16:20:01 -04:00
Chris Lumens 6a3ccd6ade Add error IDs to things that can go wrong when running a compose.
Note the exception string checking around compose_type.  I didn't really
want to introduce a new exception type just for this, but also didn't
want to duplicate strings.  I'd be open to other suggestions for how to
do this.

(cherry picked from commit b3bb438254)
2018-08-09 16:19:19 -04:00
Chris Lumens c1af7dd69d Add error IDs for common source-related errors.
(cherry picked from commit e43adfc7af)
2018-08-09 16:19:12 -04:00
Chris Lumens f67e2c5d52 Add error IDs for unknown modules and unknown projects.
(cherry picked from commit 2adcfc9563)
2018-08-09 16:19:08 -04:00
Chris Lumens 785ee7bcc3 Add error IDs for when an unknown commit is requested.
(cherry picked from commit 07528a431e)
2018-08-09 16:19:03 -04:00
Chris Lumens 2f8f076b1f Add error IDs for when an unknown blueprint is requested.
This adds some fairly redundant code to the beginning of all the
blueprint routes to attempt reading a commit from git for the
blueprint's recipe.  If it succeeds, the blueprint exists and the route
can continue.  Otherwise, return an error.  Hopefully this doesn't slow
things down too much.

(cherry picked from commit a925cc7ddb)
2018-08-09 16:18:49 -04:00
Chris Lumens a2ce0686ca Add error IDs for when an unknown build UUID is requested.
Note that this also changes the return type of uuid_info to return None
when an unknown ID is given.  The other uuid_* functions are fine
because they are checked ahead of time.

(cherry picked from commit 6497b4fb65)
2018-08-09 16:18:40 -04:00
Chris Lumens b26d12e457 Add error IDs for bad state conditions.
(cherry picked from commit d76e24053a)
2018-08-09 16:18:23 -04:00
Chris Lumens 2efbd7cae5 Change the error return type for bad limit= and offset=.
Each element in the errors value is now a dict, with a msg field and an
id field.  The id field contains a value out of errors.py that can be
used by the front end to key on.  The msg field is the same as what's
been there.

The idea is to keep the number of IDs somewhat limited so there's not a
huge number of things for the front end to know.

(cherry picked from commit 9677b012da)
2018-08-09 16:18:17 -04:00
Chris Lumens 6c3b0b3cd2 Don't sort error messages.
This should make it easier to return more complex error structures.  It
also doesn't appear to matter - tests still pass without changes.

(cherry picked from commit 4c3f93e329)
2018-08-09 16:18:11 -04:00
Brian C. Lane 0e97afdbd6 Add + to the allowed API string character set 2018-08-07 16:47:54 -07:00
Brian C. Lane 316a829ff7 Add job_* timestamp support to compose status 2018-08-07 15:21:10 -07:00
Brian C. Lane ec4c555174 Add input string checks to the branch and format arguments
Make sure no UTF8 characters are allowed and return an error if they
are.

Also includes tests to make sure the correct error is returned.

(cherry picked from commit 86d79cd8a6)
2018-08-07 10:12:02 -07:00
Brian C. Lane f93f64a061 Add a test for invalid characters in the API route
Currently the code is not UTF8 safe, so we need to return a clear error
when invalid characters are passed in.

This also adds tests for the routes to confirm that an error is
correctly returned.

(cherry picked from commit 74f5def3d4)
2018-08-07 10:12:02 -07:00
Will Woods 439a894947 composer-cli: clean up "list" commands
This patch does two things:

1) Add "compose list", which lists compose UUIDs and other basic info,
2) Fix up "blueprints list", "modules list", "sources list", and
   "compose types" so their output is just a plain list of identifiers
2018-08-07 13:00:56 -04:00
Brian C. Lane 741e73265c Fix logging argument 2018-08-07 09:24:11 -07:00
Brian C. Lane b4f9fb5c1f Fix write_timestamp for py3
toml.dumps() returns a string for write() so .encode is not needed.
2018-08-07 09:22:18 -07:00
Chris Lumens 8a2a43be99 Return a JSON error instead of a 404 on certain malformed URLs.
This handles the case where a route is requested, but without a required
parameter.  So, /blueprints/info is requested instead of
/blueprints/info/http-server.  It accomplishes this via a decorator, so
a lot of these route-related functions now have quite a few decorators
attached to them.

Typo'd URLs (/blueprints/nfo for instance) will still return a 404.  I
think this is a reasonable thing to do.

(cherry picked from commit 5daf2d416a)
2018-08-06 16:57:49 -07:00
Chris Lumens 76f714eab5 Return an error if /modules/info doesn't return anything.
Unfortunately, this isn't very useful if /modules/info is provided with
multiple modules.  yum doesn't traceback when doPackageLists is given
something that doesn't exist.  It just returns an empty list.  If
/modules/info is given just one module and yum gives us an empty list,
it's easy to say what happened.  If /modules/info is given several
modules and just one does not exist, we will not be able to detect that.

Fixing this would require doing more yum operations, which is likely to
slow things down and isn't the direction I want to be going.

(cherry picked from commit 8e948e4a4d)
2018-08-06 16:57:40 -07:00
Chris Lumens ea7b22a8f5 Update documentation (#409).
(cherry picked from commit 6193a7c9d8)
2018-08-06 16:57:34 -07:00
Chris Lumens 022734ab01 Use constants instead of strings (#409).
(cherry picked from commit fdf4d63f3b)
2018-08-06 16:57:26 -07:00
Chris Lumens 495dcd13ec Write timestamps when important events happen during the compose (#409).
Right now, this is when the compose is queued up, when it is started by
anaconda, and when it is finished (whether that's success or not).

(cherry picked from commit 3ba9d53b8b)
2018-08-06 16:57:19 -07:00
Chris Lumens e42b797295 Return multiple timestamps in API results (#409).
If one of the timestamps isn't present (for instance, the finished
timestamp for a job that is still running), null is returned.

(cherry picked from commit 17c40ef271)
2018-08-06 16:57:13 -07:00
Chris Lumens ab76c6170e Add a new timestamp.py file to the API directory (#409).
This is responsible for writing out a new times.toml file, containing
important timestamps in the life of a compose.  This seems a little more
reliable than attempting to infer things from the filesystem, especially
in light of the fact that we can't ever really know when a file was
created.

(cherry picked from commit b59d59b124)
2018-08-06 16:57:07 -07:00
Brian C. Lane 0c19c3304e composer-cli: Add support for Group to blueprints diff
(cherry picked from commit 2edd7a995b)
2018-08-06 16:53:39 -07:00
Brian C. Lane 08fb780bb9 Update status.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 642f117d54)
2018-08-06 16:53:07 -07:00
Brian C. Lane db587afb16 Update sources.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 3205e47a13)
2018-08-06 16:53:02 -07:00
Brian C. Lane f0dd50fbae Update projects.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 77767cd93a)
2018-08-06 16:52:56 -07:00
Brian C. Lane 42ad6ad0cc Update modules.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit b55b86d0b3)
2018-08-06 16:52:51 -07:00
Brian C. Lane 4e3044de20 Update compose.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit 760aeb61a7)
2018-08-06 16:52:43 -07:00
Brian C. Lane b40c3b2ce3 Update blueprints.py to use new handle_api_result
Use the new function to properly handle error responses for all the
commands.

(cherry picked from commit c1c5536ec8)
2018-08-06 16:52:37 -07:00
Brian C. Lane 08c2965fab Modify handle_api_result so it can be used in more places
Some results have errors and no status, others have status and errors.
Update the function to return the final rc to exit with, and a bool
indicating whether or not to continue processing the other fields.

Add a bunch of tests for the new function to make sure I have the logic
correct.

(cherry picked from commit 35fa067219)
2018-08-06 16:52:28 -07:00
Chris Lumens 2d5f434b12 Fix help output on the compose subcommand.
(cherry picked from commit a3572c9fdd)
2018-07-26 10:43:01 -04:00
Chris Lumens 2251c540c8 Add timestamps to "compose-cli compose status" output.
(cherry picked from commit 7bcb61849d)
2018-07-26 10:42:57 -04:00
Chris Lumens 01582e227d And then add real output to the status command.
(cherry picked from commit eb0939d967)
2018-07-26 10:42:08 -04:00
Chris Lumens 80e2e8e6db Add the beginnings of a new status subcommand.
(cherry picked from commit 9eafc60fa0)
2018-07-26 10:42:04 -04:00
Brian C. Lane 18521cec1a composer-cli: Fix non-zero epoch in projets info
The json value is an int, so convert it to a str.
2018-07-25 11:35:03 -07:00
Brian C. Lane b1dd22afa6 Log and exit on metadata update errors at startup
A bad system repo can cause lorax-composer to fail to start. Instead of
a traceback log the error and exit.

(note that the exit still results in an OSError traceback due to part of
it running as root, this needs to be addressed in another commit).
2018-07-18 16:09:03 -07:00
Brian C. Lane 83680d92b0 Check /projects responses for null values.
Some values can be null/None so check for that instead of crashing.

(cherry picked from commit f916e41d00)
2018-07-18 16:01:18 -07:00
Brian C. Lane 816f1658db Clarify error message from /source/new
(cherry picked from commit b20aa4609c)
2018-07-18 16:01:06 -07:00
Chris Lumens fbda20b8b7 Support loading groups from the kickstart template files.
(cherry picked from commit c8d2045f89)
2018-07-18 15:38:40 -07:00
Chris Lumens 8c4804eb15 Include groups in depsolving.
This adds a new argument to projects_depsolve and
projects_depsolve_with_size that contains the group list, unfortunately.
I would have prefered adding a function that just returns a list of all
the contents of a group and then add that to what was being passed into
projects_depsolve.  However, there does not appear to be any good way to
do that in yum aside from a lot of grubbing around in the comps object,
which I am unwilling to do.

(cherry picked from commit 5fe4b47072)
2018-07-18 14:36:28 -07:00
Chris Lumens 76d376fe18 Add support for groups to blueprints.
Nothing is currently being done with this information, but it will be
soon.

(cherry picked from commit 0f69d2084c)
2018-07-18 13:48:25 -07:00
Chris Lumens bb8fdcb854 Add help output to each subcommand.
This is the same as the output at the top level, just trimmed down to
only the options for a single subcommand.  It's trigged by providing
"help" or "--help" as a subcommand option.

(cherry picked from commit 954f330ace)
2018-07-18 11:58:56 -07:00
Chris Lumens eba5658a71 Split the help output into its own module.
This means I can reuse it for help output for individual subcommands.

(cherry picked from commit 70e4211ad1)
2018-07-18 11:57:45 -07:00
Chris Lumens 3c6478e179 If the help subcommand is given, print the help output.
This isn't a real subcommand like the others.  The option processing
just intercepts it and prints the output.  Given that we're subcommand
based, it makes sense to support this in addition to --help.

(cherry picked from commit 3743d6d208)
2018-07-18 11:49:16 -07:00
Brian C. Lane 460a277d4e Check the compose templates at startup
Depsolve the packages included in the templates and report any errors
using the /api/status 'msgs' field. This should help narrow down
problems with package sources not being setup correctly.
2018-07-13 10:18:07 -07:00
Adam Williamson 4a3f9e12aa DNF 3: progress callback constants moved to dnf.transaction
Signed-off-by: Adam Williamson <awilliam@redhat.com>
2018-06-27 16:12:25 -07:00
Brian C. Lane 56926baaec Add the ability to enable DNF plugins for lorax
Pass --dnfplugin='*' to enable all of them.
Pass --dnfplugin='plugin-name' to enable one fo them. You can use it
multiple times to enable multiple plugins. Globs work as well.
2018-06-22 09:07:01 -07:00
Chris Lumens e8894b7266 It's /compose/cancel, not /blueprints/cancel. 2018-06-18 10:51:12 -04:00
Brian C. Lane 936d193413 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
(cherry picked from commit c746e8b0c3)
2018-06-11 14:00:05 -07:00
Brian C. Lane 594e762d63 Increase default ram used with lmc and virt to 2048
1024 is a bit small in some situations, double it.

Resolves: rhbz#1538747
2018-06-08 11:44:09 -07:00
Brian C. Lane d8ce013a2b Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
2018-06-06 17:10:35 -07:00
Martin Kolman d5d3dd3be3 Make LogRequestHandler configurable
Make it possible to manipulate the simple and regexp
tests the LogRequestHandler class uses to check error
messages for potential error states.

This is accomplished by moving the simple and regexp test
strings to class members, where they can be easily
manipulated by users of the pylorax module.

It's also now possible to set the log request handler class
for a LogMonitor.

This functionality can then be used for example like this:

customized_log_request_handler = monitor.LogRequestHandler
customized_log_request_handler.simple_tests.remove("Call Trace:")
log_monitor = monitor.LogMonitor(install_log,
                                 timeout=opts.timeout,
                                 log_request_handler_class = customized_log_request_handler)

This way installation will continue even if there was a call
trace in the logs. In a similar way additional tests and regexps can be
also added.
2018-06-06 14:30:28 -07:00
Brian C. Lane c9ca451568 gevent has deprecated .wsgi, should use .pywsgi instead
https://github.com/gevent/gevent/blob/master/doc/api/gevent.wsgi.rst
2018-06-05 14:57:21 -07:00
Brian C. Lane 6f6ce410c0 Add support for sources to composer-cli
This adds the sources command which can be used to list, add, change,
and delete sources using the TOML formatted source file.
2018-06-04 15:45:36 -07:00
Brian C. Lane afa89ea657 Fix DNF related issues with source selection
DNF Repo.dump() function cannot be used as a .repo file for dnf due to
it writing baseurl and gpgkey as a list instead of a string. Add a new
function to write this in the correct format, and limited to the fields
we use.

Add a test for the new function.

Fix /projects/source/info to return an error 400 if a nonexistant TOML
source is requested. If JSON is used the error is part of the standard
response.

Update test_server.py to check for the correct error code.
2018-06-04 15:45:36 -07:00
Brian C. Lane dd8e4d9e99 Fix handling bad source repos and add a test
When adding a source failed it wasn't being removed from the dnf object.
This fixes that, and returns an error when setting up the source fails.
Also adds a test for it.
This also includes detecting rawhide vs. non-rawhide releases and
adjusting the tests accordingly (some of the source names change).
2018-06-04 15:45:36 -07:00
Brian C. Lane 23f4b2a3ec Fix make_dnf_dirs
It was chopping off an extra directory level due to realpath removing
the trailing / from the paths when they are setup.
2018-06-04 15:45:36 -07:00
Brian C. Lane 6d677b2207 Add support for user defined package sources API
This lives under /api/v0/projects/source/*

See the documentation for details
2018-06-04 15:45:36 -07:00
Colin Walters bb3d8edd06 templates: Stop using gconfset
We had only been indirectly pulling in GConf, and anyways
nothing was listening to these keys.

<kalev> I still think it's a fallout from 27a90d973f?branch=master

Really in general, if we wanted to make changes like this
it'd probably be a lot simpler to do them on boot or so.

https://bugzilla.redhat.com/show_bug.cgi?id=1581838
2018-05-23 14:20:44 -07:00
Brian C. Lane 095829171a Add support for version globs to blueprints
This uses dnf's version__glob filter to implement it. It amounts to '*'
wildcards and '?' for single character matching.
2018-05-18 12:03:26 -07:00
Brian C. Lane 48e318b391 Add support for systemd socket activation
Instead of enabling lorax-composer.service enable lorax-composer.socket
and it will start lorax-composer on first access to
/run/weldr/api.socket
2018-05-14 13:57:44 -07:00
Brian C. Lane c726f19434 Update the generated html docs 2018-05-14 13:00:15 -07:00
Brian C. Lane db67c1267a Move lorax-composer and composer-cli argument parsing into modules
This allows sphinx-argparse to document them automatically.
2018-05-14 13:00:15 -07:00
Brian C. Lane 6e5ca1f062 Add new cmdline args to compose_args settings 2018-05-14 13:00:15 -07:00
Brian C. Lane b1319c7bc1 Remove temporary files after run_compose
A crash can also leave temporary lmc-* files, remove them as well.
2018-05-14 13:00:15 -07:00
Brian C. Lane 0455a5d740 Add --proxy to lorax-composer cmdline
Overrides the [dnf] proxy setting in the config file.
2018-05-14 13:00:15 -07:00
Brian C. Lane 12cb2736ac Pass the --tmp value into run_creator and cleanup after a crash
Crashing can sometimes leave directories in /var/tmp/lmc-* so clean
those up after run_creator is finished.
2018-05-14 13:00:15 -07:00
Brian C. Lane 3c31e52b3c Add --tmp to lorax-composer and set default tempdir
It was using /tmp/ which can fill up quickly when building images.
Default to /var/tmp unless the user passes --tmp /tmp/
2018-05-14 13:00:15 -07:00
Brian C. Lane 9a1dce5619 Set lorax_templates to the correct directory
Templates are under the share dir, so use find_templates() to find them.
2018-05-14 13:00:15 -07:00
Brian C. Lane 3974a4e479 Adjust the disk size estimates to match Anaconda
First is Anaconda uses 6k blocks per file for its estimate, and it
fudges by 10% so adjust for those with an extra 10% of headroom just in
case.

Second is an Anaconda bug that won't allow it to do a kickstart install
to a disk smaller than 3000 MB. There is a PR to fix it upstream, but
for now the minimum size has to be 3000e9
2018-05-14 13:00:15 -07:00
Brian C. Lane a363aee971 Skip creating groups with the same name as a user
Otherwise the user creation fails when anaconda sees there is already a
group with that name. Log a warning and continue on.
2018-05-14 13:00:15 -07:00
Brian C. Lane 07ea61be77 Add user and group creation to blueprint
[[customize.user]] and [[customize.group]]
2018-05-14 13:00:15 -07:00
Brian C. Lane 25bec0b50d Add blueprint customization support for hostname and ssh key
This adds support for the optional blueprint section [customizations].

Use it like this:

[customizations]
hostname = yourhostnamehere

[[customiations.sshkey]]
user = root
key = root user key
2018-05-14 13:00:15 -07:00
Brian C. Lane 825d1931e4 Add composer-cli and tests 2018-05-14 13:00:15 -07:00
Brian C. Lane a9b626a706 Fix the compose arguments for the Fedora version of Anaconda
This adds make_oci, and moves qcow2 to the more generic image_type
(which just uses qemu-img disk names).
2018-05-14 13:00:15 -07:00
Brian C. Lane 5e0246d4a9 Add selinux check to lorax-composer
anaconda needs to have SELinux set to disabled or permissive in order to
run correctly. Check at startup and exit with an error.
2018-05-14 13:00:15 -07:00
Brian C. Lane c9187ce4ca Convert Yum usage to DNF
The DNF api is similar, but not the same, as Yum. Make the needed
changes, and rename yum references to dnf to avoid confustion later.
2018-05-14 13:00:15 -07:00
Brian C. Lane 48671b682a workspace read and write needs UTF-8 conversion
reading needs to decode the bytes to str and writing needs to encode the
str to bytes.
2018-05-14 13:00:15 -07:00
Brian C. Lane d478864fb3 Return an empty list if depsolve results are empty 2018-05-14 13:00:15 -07:00
Brian C. Lane 8ed039107b The git blob needs to be bytes
In py3 str is unicode so it needs to be encoded as UTF-8 when writing to
the blob.
2018-05-14 13:00:15 -07:00
Brian C. Lane 719aceac82 Use default commit sort order instead of TIME
Different versions of libgit2 act differently. Using TIME results in
some commits (like a revert) being listed correctly, but the rest being
listed in reverse order. Leaving it at the default works for
libgit2-0.26.3
2018-05-14 13:00:15 -07:00
Brian C. Lane 2fa1da0f5e Add lorax-composer and the composer kickstart templates 2018-05-14 13:00:15 -07:00
Brian C. Lane 9b9b627fe5 Update pylorax.api.projects for DNF usage
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane e86f4f9a36 Update dnfbase (formerly yumbase) for DNF support
And adjust tests to match.
2018-05-14 13:00:14 -07:00
Brian C. Lane 06c227598c Move core of livemedia-creator into pylorax.creator
This moves everything except the cmdline checking into run_creator in
pylorax.creator

It also rearranges some functions to prevent import loops, and adds a
utility function to imgutils (mkfsimage_from_disk for copying a
partition into a filesystem image).
2018-05-14 13:00:14 -07:00
Brian C. Lane 63b473d4ba Convert lorax-composer yum base object to DNF
This no longer uses the enabled configuration setting to select repos to
use. It uses everything in the repo_dir, and if system repos have not
been disabled it copies them into the repo_dir at startup, overwriting
the previous copy.
2018-05-14 13:00:14 -07:00
Brian C. Lane d31bea7bd3 Use 2to3 to convert the python2 lorax-composer code to python3 2018-05-14 13:00:14 -07:00
Brian C. Lane 89050f068d livemedia-creator: Move core functions into pylorax modules
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
2018-05-14 13:00:14 -07:00
Brian C. Lane dc348fac25 Check selinux state before creating output directory
This closes issue #163
2018-04-26 11:12:24 -07:00
Jiri Konecny 2f3de6060b Update default releasever to Fedora 29 (rawhide)
There was Fedora 25 which is now unsupported.
2018-03-15 12:07:34 -07:00
Brian C. Lane e1c839de1c Fix _install_branding
filter(provides=...) doesn't work with paths. The release packages
provide system-release so just look for that instead of a file.

Now it finds the release package and selects it along with the
corresponding logos package.

Note, this has been broken since commit 431ca6ce
2018-02-23 12:50:07 -08:00
Yuval Turgeman 68d1f4208d LMC: Make the QEMU RNG device optional
Added a new QEMU argument '--with-rng' to specify a RNG device other
than /dev/random or "none".

Signed-off-by: Yuval Turgeman <yturgema@redhat.com>
2018-01-23 08:22:38 -08:00
Brian C. Lane dc635a97f6 Write the --variant string to .buildstamp as 'Variant=' 2018-01-17 09:38:16 -08:00
Brian C. Lane 629d04dc68 Fix installpkg exclude operation
Commit 8edaefd4d1 added the ability to install specific NVR's of
packages, but it did not adjust the exclude operation to account for
this.

This patch fixes that, applying the exclude only to the name part of the
package NVR, and changes some variable names to pkgnvr/pkgnvrs to make
it more clear that the content has changed to <name>-<version>-<release>
2018-01-15 08:38:43 -08:00
Brian C. Lane 73628a6e07 Log a more descriptive error when setfiles fails (#1499771)
Some lorax users run it from inside mock, which isn't able to detect
whether the host is in Permissive mode. This can lead to confusing
error messages, so this points them in the right direction.
2017-11-27 12:03:45 -08:00
Claudio Zumbo 8edaefd4d1 Allow installpkgs to do version pinning through globbing 2017-11-10 09:56:27 -08:00
Brian C. Lane 6147d58ccf Use bytes when writing strings in mk-s390-cdboot (#1504026)
python3 needs bytes when calling .write()
2017-10-19 14:37:12 -07:00
Brian C. Lane aa0067e32c Convert mk-s390-cdboot to python3 (#1497141)
It was pulling in py2 along with py3. Fix #! and make sure code works
with py3.
2017-10-11 10:27:52 -07:00
Alexander Todorov fdb7593cb0 Rename parameters to match names that dnf uses
avoids 'Parameters differ from overridden method' warnings
2017-10-09 13:38:56 -07:00