Commit Graph

1884 Commits

Author SHA1 Message Date
David Shea
955631b872 Make no-virt generated images sparser
At the end of disk image installs, use fstrim on the generated filesystem to
discard any blocks that were allocated during the install and are now unused.
This will allow tools such as qemu-img to create images that do not include
deleted data.

For raw disk images that do not go through qemu-img, use fallocate --dig-holes
to create sparse holes in place of the unused blocks.
2018-10-09 15:16:38 -07:00
Brian C. Lane
fb1dfc9488 Report an error if the blueprint doesn't exist
composer-cli uses TOML for 'blueprints save' which was returning an
empty 200 response if the blueprint didn't exist. Change this to return
a standard 400 error response if the blueprint doesn't exist.

composer-cli is already setup to handle receiving json when an error is
returned so just the toml API response for `blueprints/save` needed to
be changed.
2018-10-09 15:16:20 -07:00
Lars Karlitski
bb40856f3e cli: Clarify error message for unprivileged access
`os.path.exists("/run/weldr/api.socket")` returns False for users which have no
access. This leads to composer printing that the file does not exist, which is
misleading.

Since it's no possible to distinguish the two cases, fix this problem by
combining them and showing a single error message.
2018-10-09 15:15:57 -07:00
Brian C. Lane
c2b0e8a8bb Write a rootpw line if no root customizations in the blueprint
Anaconda requires the root password to be set or locked, so if there
isn't anything setting it we write out 'rootpw --lock'

Also adds tests for this.

Resolves: rhbz#1626122
2018-10-09 15:15:42 -07:00
Brian C. Lane
558fdecde5 Add beakerlib to Dockerfile.test
Also kill the lorax-composer process and remove /run/weldr/api.socket
so that when this is run with podman you don't get an error about
attempting to tar up the socket.
2018-10-09 15:15:23 -07:00
Brian C. Lane
0a71478ac4 Adjust the new templates for locked root
also remote the auth line so that it uses the defaults.

Related: rhbz#1628645
Related: rhbz#1628646
Related: rhbz#1628647
Related: rhbz#1628648
2018-10-09 15:14:53 -07:00
Brian C. Lane
13e6a68154 Always update repo metadata when building an image
When the kickstart is handed off to Anaconda for building it will
download its own copy of the metadata and re-run the depsolve. So if the
dnf cache isn't current there will be a mismatch and the build will
fail to find some of the versions in final-kickstart.ks

This adds a new context to DNFLock, .lock_check, that will force a check
of the metadata. It also implements its own timeout and forces a
refresh of the metadata when that expires because the dnf expiration
doesn't always work as expected.

Resolves: rhbz#1631561
2018-10-09 15:13:48 -07:00
Brian C. Lane
f5679f61b1 Add a test for repo metadata expiration
This tests to make sure that the metadata timer is working (by setting
it to 10s and adding a new package to the repo), and that
DNFLock.lock_check immediately picks up a new package.

This depends on rpmfluff which is available from Fedora or EPEL repos.

Related: rhbz#1631561
2018-10-09 15:12:21 -07:00
Brian C. Lane
8479319621 Add tests for setting root password and ssh key with blueprints
Related: rhbz#1626120
2018-10-09 15:12:09 -07:00
Brian C. Lane
a1f11f943d Use rootpw for setting the root password instead of user
Ends up you cannot use the kickstart user command on root, since it
already exists, so we have to translate that into a rootpw command.

So [[customizations.user]] with name = "root" only support key, which
will set the ssh key, and password which will use rootpw to set the
password. plain text or encrypted are supported.

Related: rhbz#1626122
2018-10-09 15:12:01 -07:00
Brian C. Lane
0cb4d04479 Lock the root account, except on live-iso
If we leave the root account w/o a password people will use it that way,
leading to insecure images. Also if we use a default password. So lock
the root account in the templates.

Users will need to do one of these things:
 1. Use [[customizations.user]] in their blueprint to configure root or
    another user.
 2. Use [[customizations.sshkey]] to set a key for root
 2. Install a package that configures a user at install time
 3. Install a package that sets up a user at boot time (eg. cloud-init)

This also drops the auth line from the kickstart templates, allowing it
to use the default password algoritm instead of md5.

Resolves: rhbz#1626122
2018-10-09 15:11:51 -07:00
David Shea
3ea07ed44a Add new compose types to compose sanity test 2018-10-09 15:11:36 -07:00
Lars Karlitski
9ae690e3b6 Also run make check on travis 2018-10-09 15:10:45 -07:00
Lars Karlitski
d8cb1a19f8 Fix pylint errors and warnings
Remove `except` block which immediately raises the same exception again (it's
not a subclass of another caught exception, so this is safe).

Remove a false positive, because it is not emitted from the code base.

Disable subprocess-popen-preexec-fn in startProgram, which is not used
internally.
2018-10-09 15:04:43 -07:00
Alexander Todorov
ca1bf01b03 New cli test covering basic compose commands
- need to specify --sharedir so lorax-composer can find its
  kickstart files

- each test script writes results into a separate directory to
  avoid a passing test overwriting the results from a failing one.
  To avoid reporting failures in case of previously failing tests
  (e.g. during development) remove the temporary directories holding
  tets results before execution!
2018-10-09 15:03:24 -07:00
Alexander Todorov
ee62425388 Execute bash tests for composer-cli
these are built on top of beakerlib and we use its internal
protocol to figure out the result without relying on the full
test runner that is tipically used inside of a RHEL environment!

Includes a disabled test snippet for Issue #460
2018-10-09 15:02:57 -07:00
Stef Walter
e95c09dd7e Start a HACKING.md file and document how to run the tests 2018-10-09 15:02:38 -07:00
Stef Walter
5cb1748908 Ignore files created by tests 2018-10-09 15:01:26 -07:00
Stef Walter
5891879ae4 Makefile: Fix the 'make install' target
This fixes the 'make install' target to work on a typical RHEL or
Fedora system. We now by default install to a prefix of /usr instead
of /usr/local

The prefix is overridable like so:

     $ make install PREFIX=/opt/
2018-10-09 15:01:14 -07:00
Brian C. Lane
48f7ad780d lorax: Only run depmod on the installed kernels
In the near-future there may be /lib/modules/ directories for older
kernels with weak dependencies listed. These may not match the installed
kernel(s) so we cannot depend on them to drive generate_module_data.

Instead use the existing findkernels() function to get the list of
installed kernels and iterate those, running depmod on them.

Resolves: rhbz#1622213
2018-10-09 14:51:05 -07:00
David Shea
2304a73676 Add virt guest agents to the qcow2 compose
(cherry picked from commit d5a1993640)
2018-10-02 12:56:56 -04:00
David Shea
de6e1d027e Add a vmdk compose type.
This is similar to the AMI type, but also adds open-vm-tools and does not do
anything special to the partitioning

(cherry picked from commit 1056bfc25b)
2018-10-02 12:56:52 -04:00
David Shea
68c1a7aa96 Add a vhd compose type for Azure images
This does pretty much the same things as the AMI compose type, but also
replaces NetworkManager with the Azure linux agent.

(cherry picked from commit e0c236ff36)
2018-10-02 12:56:47 -04:00
David Shea
f79fd46f1f Add an ami compose type for AWS images
This differs from lmc's --make-ami in that creates a full disk image instead of
an fsimage. Create a raw disk image with a / and /boot partitions, and enable
sshd, chronyd, and cockpit by default.

(cherry picked from commit 18188bf6cf)
2018-10-02 12:56:33 -04:00
David Shea
09b34889bc Remove --fstype from the generated part line
Instead of specifying the fstype, just let anaconda use the default.

(cherry picked from commit 847fff4e11)
2018-10-02 12:56:03 -04:00
Brian C. Lane
7823d46747 New lorax documentation - 28.18 2018-09-06 10:41:38 -07:00
Brian C. Lane
6eb357e71e Fix /compose/cancel API documentation
It said /blueprints/cancel which is incorrect.
2018-08-29 10:08:15 -07:00
Brian C. Lane
a951bf083d Automatic commit of package [lorax] release [28.18-1].
Created by command:

/usr/bin/tito tag
2018-08-27 15:47:16 -07:00
Brian C. Lane
a286e9b3dc Fix composer-cli blueprints changes to get correct total
blueprints/changes is different, each blueprint has it's own total,
limited by the call's limit. So it needs to find the max total of all
the requested blueprints.

(cherry picked from commit 57674c9a1a)
2018-08-27 12:02:39 -07:00
Brian C. Lane
630df546e6 Fix blueprints/list and blueprints/changes to return the correct total
(cherry picked from commit ec908fcd2a)
2018-08-27 12:02:39 -07:00
Brian C. Lane
8838ee12bc Add tests for limit=0 routes
Passing ?limit=0 to the blueprints/list, blueprints/changes,
projects/list, modules/list should always return the total possible
results, not 0.

Also move the composer-cli test_diff to the end so that it will work
consistently. Do this by naming it test_z_diff.

(cherry picked from commit 972b5c4142)
2018-08-27 12:02:39 -07:00
Brian C. Lane
187cd935df Add a function to get_url_json_unlimited to retrieve the total
The blueprints/changes API is a bit different from the others, the total
that it includes is for each blueprint, not one total for all of them,
since there will be a different number of commits for each.

The function is passed the dict, and it can be used to select the total
to use for retrieving all of the results. If it isn't included it will
use data["total"] which works fine in most cases.

(cherry picked from commit 0a76d635ca)
2018-08-27 12:02:39 -07:00
Brian C. Lane
7773be2bda Fix tests related to blueprint name changes 2018-08-23 13:46:39 -07:00
Brian C. Lane
b198c6e498 Add 'example' to the example blueprint names 2018-08-23 13:45:32 -07:00
Brian C. Lane
9249d21d6f Use urllib.parse instead of urlparse
python3 moved this to a new module.

(cherry picked from commit 86d556e87c)
2018-08-21 16:50:53 -07:00
David Shea
e3ef3d9a29 In composer-cli, request all results
Add a limit argument to all potentially paginated results, equal to
whatever the composer backend is the total number of results. This still
has the potential to provide truncated data if the number of results
increases between the two HTTP requests.

Resolves: #404
(cherry picked from commit ee98d87cea)
2018-08-21 16:50:53 -07:00
David Shea
eedf1a9243 Add tests for /compose/status filter arguments
(cherry picked from commit c52ba4236a)
2018-08-21 16:50:53 -07:00
David Shea
bba0a85aaf Allow '*' as a uuid in /compose/status/<uuid>
This will display all UUIDs that match the filter arguments

(cherry picked from commit deff4d325f)
2018-08-21 16:50:53 -07:00
David Shea
5272be3127 Add filter arguments to /compose/status
This adds the following optional arguments to the /compose/status route:

  - type, matches the compose_type field
  - status, matches the queue_status field
  - blueprint, matches the blueprint field

(cherry picked from commit 40f23f093d)
2018-08-21 16:50:53 -07:00
Brian C. Lane
8694c7a7dc composer-cli should not log to a file by default
The user can pass --log /path/to/logfile.log if they want logging
enabled.

(cherry picked from commit 844ff9998b)
2018-08-21 11:59:12 -07:00
Brian C. Lane
93b207e752 Add documentation for using a DVD as the package source 2018-08-21 11:28:59 -07:00
Lars Karlitski
d55ff41870 Set TCP listen backlog for API socket to SOMAXCONN
A value of 1 is too low for heavy users of the API, such as the weldr-web
interface.

This is also systemd's default for sockets it opens. Using lorax-composer with
socket activation already results in a backlog of SOMAXCONN connections.

(cherry picked from commit be5d50e6f3)
2018-08-20 14:54:57 -07:00
Brian C. Lane
cef073d832 New lorax documentation - 28.17 2018-08-13 16:49:52 -07:00
Brian C. Lane
c2f96b3d3b Add a note about using lorax-composer.service
Update the docs to make it clear that .socket will be slow to respond to
the first request, and to use .service to speed it up.

(cherry picked from commit 83e5f3f885)
2018-08-13 16:47:12 -07:00
Brian C. Lane
64efb8b415 Ignore dnf.logging when building docs
(cherry picked from commit 07de4c371a)
2018-08-13 16:47:10 -07:00
Chris Lumens
312faf6275 Fix a little bug in running "modules list".
(cherry picked from commit 5dddef6f49)
2018-08-10 12:00:54 -04:00
Brian C. Lane
3a89828faf Automatic commit of package [lorax] release [28.17-1].
Created by command:

/usr/bin/tito tag
2018-08-09 16:38:21 -07:00
Brian C. Lane
77dc017c78 Fix bash_completion.d typo 2018-08-09 16:25:26 -07:00
Brian C. Lane
02dc3404a8 Move disklabel and UEFI support to compose.py
Currently we are making MBR disk images for qcow2 and partitioned disk,
so the UEFI packages aren't required at this point.

Move the clearpart command into compose.py so that in the futute it can
use clearpart --disklabel to create a GPT image, and add the required
packages to the package set.
2018-08-09 15:26:09 -07:00
Chris Lumens
a08c16158d Fix more tests.
These were broken due to me rebasing before sending a PR off.  They
should work now.

(cherry picked from commit 4cb15e0a0f)
2018-08-09 16:33:30 -04:00