Commit Graph

608 Commits

Author SHA1 Message Date
Brian C. Lane
a44e499eee Close the rpmdb after every API operation.
rpm blocks all of the signals until closeRpmDB() is called.
We need to call it after all of the API calls.
2018-06-11 16:54:58 -07:00
Brian C. Lane
cf42e1a480 Fix error string when there is a problem listing projects 2018-06-11 16:54:58 -07:00
Brian C. Lane
efd15aaad7 Add --releasever option to lorax-composer
If it isn't set the releasever value will use yum and the system / to
find the releasever.
2018-06-11 16:54:58 -07:00
Brian C. Lane
bae551b7ab Add filtering and glob support to /modules/list route 2018-06-11 16:54:58 -07:00
Brian C. Lane
6b4bf92e7f Add /recipes/freeze route and tests. 2018-06-11 16:54:58 -07:00
Brian C. Lane
b8212358bc Add /recipes/depsolve route and test 2018-06-11 16:54:58 -07:00
Brian C. Lane
a85d0ced8b Add /projects and /modules API tests 2018-06-11 16:54:58 -07:00
Brian C. Lane
164232ba4d Modify pylorax.api.config.configure so it can also be used for tests. 2018-06-11 16:54:58 -07:00
Brian C. Lane
ad5ab71551 Move ComposerConfig into pylorax.api.config module 2018-06-11 16:54:58 -07:00
Brian C. Lane
8fee86a5db Catch ProjectsError and return an error 400 with a message. 2018-06-11 16:54:58 -07:00
Brian C. Lane
a1e7e4831d Catch Yum errors in the projects functions 2018-06-11 16:54:58 -07:00
Brian C. Lane
23d90ec066 Add /modules/list and /modules/info routes 2018-06-11 16:54:58 -07:00
Brian C. Lane
8635a59ac7 Add modules functions and update function documentation 2018-06-11 16:54:58 -07:00
Brian C. Lane
c3d495fe53 Add /projects/depsolve route 2018-06-11 16:54:58 -07:00
Brian C. Lane
99140b98b9 Add /projects/info route
Elements that have no match in yum (metadata) are set to empty dicts.
2018-06-11 16:54:58 -07:00
Brian C. Lane
0819e97646 Add /projects/list route 2018-06-11 16:54:58 -07:00
Brian C. Lane
105679d79d Add /api/v0/test route
Returns a simple string to indicate that the API server is running.

/api/v0/status should be used instead, it provides more detailed info in
JSON format.
2018-06-11 16:54:58 -07:00
Brian C. Lane
569ce1d3dc Add support for yum to lorax-composer
This includes a new configuration file at /etc/lorax/composer.conf with
built-in defaults. It also adds a YUMLOCK server config object so that
request handlers can access the yum base object without interfering
with each other.
2018-06-11 16:54:58 -07:00
Brian C. Lane
5d6c236af6 Add /api/docs to serve up the documentation
This requires that the docs be at /usr/share/doc/lorax-*/html/ or if
running from the source tree, at ./docs/html/

They can be re-created by running 'make docs'
2018-06-11 16:54:58 -07:00
Brian C. Lane
1d53d00532 Add limit/offset to recipes/list 2018-06-11 16:54:58 -07:00
Brian C. Lane
2cd4fcfd88 Add error message for offset/limit type errors 2018-06-11 16:54:58 -07:00
Brian C. Lane
9ca9bfe46f Add error logging to api/v0.py
These will end up in the /var/log/lorax-composer/composer.log
2018-06-11 16:54:58 -07:00
Brian C. Lane
d701120d90 Add /recipes/diff route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
052a8ba991 Add recipe_diff function and helpers.
This takes a pair of Recipe objects and returns a list of diff dicts
that include what was changed between the two recipes.
2018-06-11 16:54:57 -07:00
Brian C. Lane
a76e95dcb5 Add POST /recipes/tag/ route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
7f1adf120c Add tag_recipe_commit helper function
And change tests to use it so that both it and tag_file_commit are used.
2018-06-11 16:54:57 -07:00
Brian C. Lane
3f4140d5d3 Add POST /recipes/undo route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
639d325a0c Change read_recipe_commit to use the recipe name
Callers really shouldn't need to know the details of the filenames, so
change it to convert it internally.
2018-06-11 16:54:57 -07:00
Brian C. Lane
ab6decec0b Add revert_recipe function
And adjust the tests to use it so that it covers both revert_recipe and
revert_file.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c294b7747d Add DELETE /recipes/delete/<recipe_name> route and tests
Also add numbering to the tests so that they are executed in the
expected order.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c937e69d3b Add delete_recipe helper function and test
Use delete_recipe in the test so that both it and delete_file are
covered.
2018-06-11 16:54:57 -07:00
Brian C. Lane
6e453e70dc Add DELETE /recipes/workspace/<recipe_name> route and tests
This will delete the workspace copy of the recipe.
2018-06-11 16:54:57 -07:00
Brian C. Lane
72b4c4fe28 Add POST /recipes/workspace route
Also fix use of workspace in /recipes/info (was using filename instead
of recipe name and logic for changed was backwards).
2018-06-11 16:54:57 -07:00
Brian C. Lane
711cdf53b3 Add /recipes/new route and tests
This handles json data when the mime type is "application/json" and toml
recipes when it is "text/x-toml"
2018-06-11 16:54:57 -07:00
Brian C. Lane
7b5115d19c Split recipe_from_toml into recipe_from_dict helper.
This will be useful for creating Recipe objects from json created dicts.
2018-06-11 16:54:57 -07:00
Brian C. Lane
3c75711b30 Fix the recipe version bumping
Recipe should have its version bumped based on the version from the
previous commit, and not be bumped on the first commit. Fix the code and
the tests.
2018-06-11 16:54:57 -07:00
Brian C. Lane
02db2afb2b Add /recipes/changes route with tests. 2018-06-11 16:54:57 -07:00
Brian C. Lane
1e7335a506 Add /recipes/info route and tests 2018-06-11 16:54:57 -07:00
Brian C. Lane
b7f651a631 Add workspace module and tests
This handles the recipe workspace, which is a temporary location to
store recipes before committing them to git. It will be used by the
/recipes/ API.
2018-06-11 16:54:57 -07:00
Brian C. Lane
b6fb22133c Add /recipes/list route and tests
Includes adding a lock for access to the git repo from the API.
2018-06-11 16:54:57 -07:00
Brian C. Lane
90a8798f4c Move the git repo into a subdirectory
And fix the import of recipes (needs to have full path passed in).
2018-06-11 16:54:57 -07:00
Brian C. Lane
596f06d5c0 Fix list_commits sort order.
It appears that with libgit2 v0.24.6 reverse causes it to list them
newest first. In 0.25.1 it lists them oldest first. On both versions
just using SortMode.TIME gives the desired result of oldest first.
2018-06-11 16:54:57 -07:00
Brian C. Lane
bfc9c6913f Add pylorax.api.recipes code for handling the Recipe's Git repository 2018-06-11 16:54:57 -07:00
Alexander Todorov
2cd4f73177 Don't do wildcard imports 2018-06-11 16:54:57 -07:00
Alexander Todorov
e9e6e5701e Misc pylint fixes that are reported usually once 2018-06-11 16:54:57 -07:00
Alexander Todorov
bf9ed50d51 Fix dangerous-default value warnings
when default value is list or dict the default arguments are
instantiated as objects at the time of definition. This is significant
(exposing visible semantics) when the object is mutable. There’s no way
of re-binding that default argument name in the function’s closure. When
function is executed multiple times with its default value the value
will change between executions, possibly leading to strange side effects.

For more information see:
http://satran.in/2012/01/12/python-dangerous-default-value-as-argument.html
2018-06-11 16:54:57 -07:00
Alexander Todorov
a0766b1d70 Don't redefine variables from outer scope 2018-06-11 16:54:57 -07:00
Alexander Todorov
4d8e2b5356 Define all class attributes inside __init__ 2018-06-11 16:54:57 -07:00
Alexander Todorov
663b0f8378 Fix logging formatting 2018-06-11 16:54:57 -07:00
Alexander Todorov
18528b4f47 Don't redefine builtins 2018-06-11 16:54:57 -07:00
Alexander Todorov
b15df5a0af Silence relative import warnings 2018-06-11 16:54:57 -07:00
Alexander Todorov
7f5b492198 pylint fix: unused variable warning 2018-06-11 16:54:57 -07:00
Alexander Todorov
15de3e1687 pylint fix: remove unused imports 2018-06-11 16:54:57 -07:00
Brian C. Lane
2008a413b6 lorax-composer initial commit
The lorax-composer program will launch a BDCS compatible API server
using Flask and Gevent. Currently this is a skeleton application with
only one active route (/api/v0/status).

The API code lives in ./src/pylorax/api/v0.py with related code in other
pylorax/api/* modules.
2018-06-11 16:54:57 -07:00
Brian C. Lane
bf8be43c90 livemedia-creator: Move core functions into pylorax modules
This reduces the amount of code in livemedia-creator to the cmdline
parsing and calling of the installer functions. Moving them into other
modules will allow them to be used by other projects, like the
lorax-composer API server.
2018-06-11 16:54:57 -07:00
Brian C. Lane
c746e8b0c3 Retry losetup if loop_attach fails
It appears that sometimes the loop device doesn't get setup properly,
this may be a race with other users of loop devices on the system, or
some other mechanism that isn't understood.

To try and prevent total failure when this happens this patch retries
the loop setup 3 times before giving up. Previously it would wait for
the loop device to appear (checking 5 times), that operation is now
executed 3 times with a new losetup attempt each time.

Resolves: rhbz#1589084
2018-06-11 13:56:53 -07:00
Brian C. Lane
22392b64fc Add --dracut-arg support to lorax
Use it to override the default dracut arguments (displayed as part of
the --help output). If you want to extend the default arguments they
all need to be passed in on the cmdline as well. eg.

--dracut-arg='--xz' --dracut-arg='--install /.buildstamp' ...

Resolves: rhbz#1452220
2018-06-06 17:38:55 -07:00
Brian C. Lane
54d973e059 Write a list of installed packages to /root/lorax-packages.log (#1416155)
This can't be done the same way as on master because there is no rpm
database inside the installroot to run rpm -qa against. Do it at the end
of the yum transaction.

Resolves: rhbz#1416155
2017-09-27 08:30:04 -07:00
Brian C. Lane
5238b9cd18 Fix systemctl command (#1478247)
When multiple units are passed to systemctl and one fails it doesn't
finish the others. Change the template command to call systemctl for
each unit individually.

This also removes the lvm2-activation-generator in runtime-cleanup.tmpl

Resolves: rhbz#1478247
2017-08-11 15:49:10 -07:00
Brian C. Lane
1b29041089 Add the version to the log (#1335456)
Log the version number as INFO and add -V version display to
livemedia-creator.

Resolves: rhbz#1335456
2017-08-11 15:42:16 -07:00
Brian C. Lane
7aa71188b9 Include the dracut fips module in the initrd (#1341280)
This will allow anaconda to fetch kickstarts using https when installing
with fips=1

Leave vmlinuz and .vmlinuz.hmac in /boot

dracut-fips module needs the vmlinuz.hmac file in order to boot.

Resolves: rhbz#1341280
2017-08-11 15:22:50 -07:00
Brian C. Lane
a883412ccc Fix loop_wait (#1462150)
The previous code used losetup --list -O to return the backing store
associated with the loop device. This can fail due to losetup truncating
the output filename if sysfs isn't setup. Instead of printing the full
path it will truncate it to 64 characters with a * at the end.

See util-linux lib/loopdev.c for the code that does this.

This commit changes it to use the existing get_loop_name function, which
uses losetup -j to lookup the loop device associated with the backing
store which avoids the truncation problem.

Resolves: rhbz#1462150
2017-08-11 15:18:00 -07:00
Brian C. Lane
aaf1cc3894 Fix waiting for loop devices (#1462150)
Missing change for the previous commit.

Resolves: rhbz#1462150
2017-06-23 08:28:32 -07:00
Brian C. Lane
e0d9e309ae Make sure loop device is setup (#1462150)
It seems that on rare occasions losetup can return before the /dev/loopX
is ready for use, causing problems with mkfs. This tries to make sure
that the loop device really is associated with the backing file before
continuing.

Resolves: rhbz#1462150
2017-06-21 14:31:35 -07:00
Brian C. Lane
391f1a7b57 Copying same file shouldn't crash (#1269213)
When using the template install command copying the same file to itself
shouldn't crash. Just log the error and continue.

Also copy the s390 configuration files for use with livemedia-creator

Resolves: rhbz#1269213
(cherry picked from commit 701ab02619)
2016-04-06 17:02:41 -07:00
Colin Walters
7c4d7b0c42 Add --installpkgs argument (#1272222)
Recently, Fedora has been trying to do a 3 product split.  As part of
that, lorax was changed to do "installpkg lorax-product-*" via
provides.

I think that approach is awkward; a much simpler approach is to simply
specify the product package as input to lorax on the command line, via
external rel-eng scripts.

This patch therefore adds --installpkgs (and we should probably add an
option to remove the implicit lorax-product-* glob).

(cherry picked from commit 52d962d613)

Resolves: rhbz#1272222
2016-02-19 15:58:50 -08:00
Brian C. Lane
62cfefd718 Don't include early microcode in initramfs (#1258498)
The system the image boots on will likely not match the host where lorax
was run, and in some cases this can cause systems to hang.

Resolves: rhbz#1258498
2015-09-01 09:50:05 -07:00
Brian C. Lane
d48d0edd9c Add installimg command for use in the templates
installimg SRCDIR DESTFILE
  Create a compressed cpio archive of the contents of SRCDIR and place
  it in DESTFILE.

  If SRCDIR doesn't exist or is empty nothing is created.

Examples:
  installimg ${LORAXDIR}/product/ images/product.img

(cherry picked from commit b064ae6166)

Related: rhbz#1202278
2015-07-14 12:19:49 -07:00
Colin Walters
50f9f48cc1 Add ability for external templates to graft content into boot.iso (#1202278)
I originally added --add-template to support doing something similar
to pungi, which injects content into the system to be used by default.
However, this causes the content to be part of the squashfs, which
means PXE installations have to download significantly more data that
they may not need (if they actually want to pull the tree data from
the network, which is not an unusual case).

What I actually need is to be able to modify *both* the runtime image
and the arch-specific content.  For the runtime, I need to change
/usr/share/anaconda/interactive-defaults.ks to point to the new
content.  (Although, potentially we could patch Anaconda itself to
auto-detect an ostree repository configured in disk image, similar to
what it does for yum repositories)

For the arch-specfic image, I want to drop my content into the ISO
root.

So this patch adds --add-arch-template and --add-arch-template-var
in order to do the latter, while preserving the --add-template
to affect the runtime image.

Further, the templates will automatically graft in a directory named
"iso-graft/" from the working directory (if it exists).

(I suggest that external templates create a subdirectory named
 "content" to avoid clashes with any future lorax work)

Thus, this will be used by the Atomic Host lorax templates to inject
content/repo, but could be used by e.g. pungi to add content/rpms as
well.

I tried to avoid code deduplication by creating a new template for the
product.img bits and this, but that broke because the parent boot.iso
code needs access to the `${imggraft}` variable.  I think a real fix
here would involve turning the product.img, content/, *and* boot.iso
into a new template.

Resolves: rhbz#1202278
2015-06-25 09:18:25 +02:00
Brian C. Lane
58929b550b Add support for creating qcow2 images (#1210413)
(cherry picked from commit 07d5cb176d)

Related: rhbz#1210413
2015-06-19 15:32:38 -07:00
Brian C. Lane
a37a59b3eb Add removekmod template command (#1230356)
removekmod GLOB [GLOB...] --allbut KEEPGLOB [KEEPGLOB...]

This can be used to remove kernel modules from under
/lib/modules/*/kernel/ while keeping specific items. This should be
easier than constructing find arguments to select the right things to
save.

(cherry picked from commit 11c9e0e8ee)

Resolves: rhbz#1230356
2015-06-15 15:04:06 -07:00
Radek Vykydal
9d8aabded6 Add --live-rootfs-size option.
Resolves: rhbz#1184021
2015-04-13 10:05:17 +02:00
Radek Vykydal
28801d8d42 Add --make-pxe-live and --make-ostree-live (for Atomic) targets.
Resolves: rhbz#1184021

--make-pxe-live target generate live squashfs and initrd for pxe boot.
Also generates pxe config template.

--make-ostree-live is used for installations of Atomic Host. Additionally to
--make-pxe-live it ensures using deployment root instead of physical root of
installed disk image where needed. Atomic installation needs to be virt
installation with /boot on separate partition (the only way supported by
Anaconda currently). Content of boot partition is added to live root fs so that
ostree can find deployment by boot configuration.
2015-04-13 10:05:17 +02:00
Colin Walters
1ef49fca1e Add --add-template{,-var}
What I need is to make something like the traditional DVD which also
includes packages.  At present this is apparently handled by the
entirely separate pungi tool.

At the moment for me, it's the least bad option to modify lorax to
inject data from an external source than to create a new tool, or
attempt to also modify pungi to do this.

This would also allow pungi's DVD creation to eventually be a set of
external templates for Lorax.

(cherry picked from commit 66359415be)

Resolves: rhbz#1157777
2014-11-20 14:41:21 -08:00
Brian C. Lane
0e6ea6be85 RHEL7 doesn't include pigz or pbzip2 (#1144140)
Don't try to speed up gzip or bzip2 compression with them.

Related: rhbz#1144140
2014-09-22 17:03:43 -07:00
Brian C. Lane
a8681aca4e livemedia-creator: Add --no-recursion to mktar (#1144140)
tar recurses into directories by default, but find is feeding it all the
files and directories so the tar it produces is considerably larger than
it needs to be due to duplicate files. Add --no-recursion flag so that
tar will only add the specific files and directories piped to it by find.

Related: rhbz#1144140
2014-09-22 17:03:30 -07:00
Brian C. Lane
526988651d livemedia-creator: Add support for making tarfiles (#1144140)
This adds the --make-tar option which will produce a xz compressed tar
of the root filesystem. This works with either virt-install or no-virt
modes. Use --image-name to set the output filename.

--compression is used to set the compression type to use, which defaults
to xz. Supported types are xz, lzma, gzip and bzip2.

--compress-arg is used to pass arguments to the compression utility.

(cherry picked from commit d04a99e8f4)

Resolves: rhbz#1144140
2014-09-22 15:25:22 -07:00
David Shea
74b1877f1b Let the plymouth dracut module back into the ppc64 upgrade.img
The 32MiB size limit does not apply to upgrade.img since it's installed
to /boot by redhat-upgrade-tool instead of downloaded through TFTP. The
warning in rebuild_initrds will still be triggered by an upgrade.img
over the limit, but this doesn't halt the compose and it's probably not
a bad thing to know about.

Resolves: rhbz#1069671
2014-09-11 17:07:24 -04:00
Brian C. Lane
84af4b08f5 Add ppc64le arch (#1136490)
basearch is ppc64le so we needs to check for that in addition to ppc64.

Resolves:rhbz#1136490
(cherry picked from commit 72357bf96b6b016c3a39b2af51eaf5cf724a0928)
2014-09-03 10:43:00 -07:00
Brian C. Lane
39e4da7ce8 utf-8 encode yum actions before displaying them (#1072362)
Otherwise running lorax in other locales will crash.

Resolves: rhbz#1072362
2014-07-30 17:05:59 -07:00
Brian C. Lane
9da051cd78 Check initrd size on ppc64 and warn (#1060691)
ppc64 initrd cannot be larger than 32MiB, so log a warning if it is too
large.

Related: rhbz#1060691
2014-02-13 09:41:53 -08:00
Brian C. Lane
e108883247 Remove drivers and modules on ppc64 (#1060691)
Remove more drivers and remove plymouth and drm dracut modules. Only on
PPC64 initrd, all other arches have the full set of drivers and modules.

Resolves: rhbz#1060691
2014-02-13 09:29:38 -08:00
Brian C. Lane
88863a63d7 Remove floppy and scsi_debug from initrd (#1060691)
ppc64 can't PXE boot a initrd that is > 32M. This removes 2 drivers from
initrd creation to reduce the size.

Resolves: rhbz#1060691
2014-02-05 12:33:16 -08:00
Brian C. Lane
d65ca8d789 flush data to disk after mkfsimage (#1052175)
Make sure the data is written before we do anything else with the disk
image. This shouldn't be needed, umount should take care of it, but it
also can't hurt.

Resolves: rhbz#1052175
2014-01-24 09:44:37 -08:00
d.marlin
3a5e515d4f Add initial 64-bit ARM (aarch64) support (#1034432)
Signed-off-by: Brian C. Lane <bcl@redhat.com>

Resolves: rhbz#1034432
2013-12-17 10:48:59 -08:00
David Shea
26d8c14796 Create upgrade.img using redhat-upgrade-dracut
Use redhat-upgrade-dracut for the RHEL version of the system-upgrade
dracut module. upgrade.img also needs the convertfs module for upgrades
from RHEL6.

Resolves: rhbz#1029999
2013-11-13 13:07:20 -05:00
Brian C. Lane
ed5803bae8 macboot defaults to no on rhel7 (#1012529)
The tools needed are not shipped with RHEL7 so default to no macboot.

Resolves: rhbz#1012529
2013-09-26 16:59:13 -07:00
Brian C. Lane
abf27add37 Bump image size up to 2G (#967556) 2013-05-28 16:29:48 -07:00
Dennis Gilmore
d751ac82db remove no longer supported arm kernel variants add the new lpae one 2013-04-17 16:21:23 -07:00
Mark Hamzy
52070f045c fix version query and add one to the log file
Fix "lorax -V" and add a "version is ???" to the log file so we can easily know
what version of lorax is used to build an iso.
Changed to try/import for versioning because the version file is autogenerated.
2013-04-03 15:43:09 -07:00
Will Woods
26c77b4fba Print & log messages on scriptlet/transaction errors
We should probably let the user know if something goes wrong with the
transaction or if a scriptlet fails. So: log the messages so we can find
(and, one hopes, fix) them later.
2013-03-19 15:03:49 -07:00
Will Woods
3a1e176cbb sysutils: add -x to cp in linktree
Since you can't make cross-device links, you'd think that cp -l (make
links) would imply -x (don't cross devices).

You'd be wrong.
2013-03-19 00:47:58 -04:00
Will Woods
ad3d1e6278 treebuilder: fix "Can't stat exclude path "/selinux"..." message
/selinux has been gone for *years*. Drop the exclude.
2013-03-19 00:47:58 -04:00
Brian C. Lane
93c4a8b830 Raise an error when there are no initrds 2013-03-12 15:39:25 -07:00
Dennis Gilmore
a4cf3c3c7c remove sparc support
Signed-off-by: Dennis Gilmore <dennis@ausil.us>
2013-03-04 14:18:45 +01:00
Brian C. Lane
bcdd43997b Remove some env variables (#907692)
Some package scripts may call utilities using dbus. Since this is just a
chroot that will fail. This unsets DESKTOP and DBUS_SESSION_BUS_ADDRESS
to keep them from crashing.
2013-02-15 05:47:11 -08:00
Brian C. Lane
d2ae92b4b3 yum changed the callback info
It used to give us (x/y) packagename, but now it only sends package
name. This was changed in yum commit 7e8c76173. For now default to
showing the zeros along with the package name so that it doesn't look
like an error.
2013-01-31 10:06:14 -08:00
Martin Gracik
62920272c7 Fix rexists 2012-12-19 12:39:39 +01:00
Martin Gracik
a0791d3074 Only run installupgradeinitrd if upgrade 2012-12-18 14:31:45 +01:00
Martin Gracik
263618e12a Add --noupgrade option 2012-12-18 12:04:03 +01:00