srbala/createhdds
1
0
Fork 0
mirror of https://pagure.io/fedora-qa/createhdds.git synced 2024-11-21 23:03:08 +00:00
Code Issues Releases Activity

rewrite createhdds in python, make it shinier

Browse Source 
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!

OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:

* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images

It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).

The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.

I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.

Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.

Reviewers: jskladan, garretraziel

Reviewed By: jskladan, garretraziel

Subscribers: tflink

Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
This commit is contained in:
Adam Williamson 2015-12-17 13:25:30 -08:00
parent 0b3c6beab9
commit 326dd08e25
6 changed files with 908 additions and 218 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

110
README.createhdds Normal file
View File

@ -0,0 +1,110 @@
# createhdds.py
createhdds.py creates and maintains the set of pre-rolled hard disk images needed for some of the Fedora openQA tests.
## Usage
Most usage information can be seen in the help text - just run `createhdds.py -h` for an overview of the subcommands available, and `createhdds.py (subcommand) -h` for help on a subcommand. To put it simply, the most common usage is simply to run `createhdds.py all -c`. This will create all the currently-expected images that have not already been created, and recreate any that need recreating (images can have a 'maximum age' causing them to be rebuilt by `all` when they're older than that age, and images also have a 'version' - if the image's 'version' is bumped by the maintainers, `all` will rebuild it). It will also remove any image files that are present that aren't expected to be present - usually images for old releases that are no longer tested, or images we've simply stopped using. In a typical deployment of a Fedora openQA instance, the admin should set things up so the `openqa_fedora_tools` git checkout is updated and `createhdds.py all -c` is run regularly - say, once a day (and probably not while tests are being run).
`createhdds.py check` will just check whether all expected images are present and up-to-date; if they are, it will exit 0, if they are not, it will exit 1 and print a message. This can be handy for use with things like Ansible (so you can run the check to decide whether you need to run the creation, and thus avoid spurious 'changed' statuses).
There are also individual subcommands for each of the named 'image groups', allowing you to create just the image(s) from that group. For image groups which usually generate multiple images, the subcommand will have arguments that let you restrict creation to just a subset of those images (and for virt-builder type images, you can create the image(s) for a different release than would usually be the case, too).
In `all` mode, and in single-image mode if you do not pass `--release`, createhdds can decide what releases to build images for, for those image groups that include an installed Fedora release (the virt-builder type images). A virt-builder type image group can specify the releases to build images for absolutely (by giving the release numbers as positive integers), or relative to the next pending release (by giving the release numbers as negative integers). When it encounters one of these 'relative' release numbers, `createhdds` uses [fedfind](https://www.happyassassin.net/fedfind) to discover the 'current' release, and adds 1 to that (to find the 'pending' release). Just in case anything goes wrong with this, or you need to override it for some reason, the `--nextrel` argument is available for relevant subcommands to explicitly specify the 'next release'.
## Specifying images / 'image groups': `hdds.json` and `.commands` files
All the information on what images can/should be created comes from the `hdds.json` file and some `virt-builder` commands files. You can add, modify and remove image definitions without touching `createhdds.py`. `hdds.json` should define a single dictionary with three keys: `guestfs`, `virtbuilder`, and `renames`. The meat is `guestfs` and `virtbuilder`, which define 'image groups': for each image group, `createhdds all` will create one or more images (multiple images produced from a single group are referred to as 'variants'). Groups and variants are provided for so that if you want to create, say, three images that are identical but for their disk label, you don't have to create a whole new almost-identical entry for each one. The rules about what particular attributes of an image can be implemented as 'variants' are somewhat arbitrary and actually just taken from the old `createhdds.sh`; each function in that implementation became an 'image group' in this rewrite, and the attributes that can vary between variants are the same ones that could be set as function arguments in `createhdds.sh`. The `.commands` files allow for customization of `virt-builder` images; more on this later. `hdds.json` and any `.commands` files must always be in the same folder as `createhdds.py` (not necessarily the same folder the disk images reside in).
### renames
The value of `renames` is a simple list-of-lists. Each item is a pair of strings. In `all` mode, and optionally in `check` mode, createhdds will read in all the items from the `renames` list. For each item it will look for a file (in the working directory - createhdds always works on the working directory) named the same as the first string, and if it finds such a file, change its name to the second string. So for instance, the value of `renames` could be `[['disk_foo.img', 'disk_bar.img'], ['disk_monkey.img', 'disk_fish.img']]`; that would result in createhdds renaming `disk_foo.img` to `disk_bar.img` and `disk_monkey.img` to `disk_fish.img`. This mechanism is provided to aid in the situation where an image's expected name changes, but the existing image file is still valid; instead of forcing the user to rename it manually or re-generate it, we can list it in `renames` and it will get renamed automatically. Of course, if the image's content changes too, we shouldn't use this mechanism.
### guestfs and virtbuilder
The value for both `guestfs` and `virtbuilder` is a list of dicts. Each dict defines a single 'image group'; an image group can produce just a single image, or several variants - we will learn what determines the possible variants for an image group below. The images in `guestfs` are produced using libguestfs - these are images that just contain a particular partition layout and perhaps some files we seed directly, but no installed operating system. The images in `virtbuilder` are produced using `virt-builder` - these are images containing an installed operating system. Some keys are common to both image group types:
#### `imgver`
This is the 'image version'. It is **optional** for both image types. By convention, it should be an integer digit string, but its practical effect is simply to be included in the image filename(s), so it *can* be any string valid in a filename. If omitted or set to the empty string, no imgver component will be included in the filename. This means that by changing the version you can change the expected name - which will cause the `check` mode to report the old file as 'unknown' and the new file as 'missing', and will cause the `all` mode to build the new file. Thus if you're maintaining the image set, and you make some change to an image group which would mean that existing image files for that group can no longer be used, you can change `imgver` to cause the images to be rebuilt.
#### `name`
The image group's name (a string). **Required** for both image types. This is included in the image file names, of course, and it will also be the subcommand to create image(s) from this group.
#### `size`
This key is **required** for `guestfs`, **optional** for `virtbuilder`. It is the size of the image. A plain digit string is a size in bytes. A digit string followed by 'M', 'MB' or 'MiB' is a size in megabytes (power-of-2). A digit string followed by 'G', 'GB' or 'GiB' is a size in gigabytes (power-of-2). For `virtbuilder`, if `size` is not set, the image will be whatever size the `virt-builder` base image it's created from is.
Some keys are specific to each type of image. These are the `guestfs`-specific keys:
#### `parts`
This key is **required**. This key's value is a list of dicts. Each dict represents a single partition that should be created on the disk. Required keys are:
* `type` - the partition type: 'p' for primary, 'l' for logical, 'e' for extended
* `start` - start sector
* `end` - end sector
These values are just passed straight to libguestfs, so you can find further info on them in the libguestfs documentation, especially on various special values for `start` and `end` (negative values are relative to the end of the disk, for e.g.)
Optional keys are:
* `filesystem` - if set, this partition will *always* have this filesystem. If not set, the filesystem will be determined according to the image group's `filesystems` value (see below).
* `label` - if set, this partition will have this label
#### `writes`
This key is **optional**. Its value is a list of dicts. Each dict represents a single file that should be created on one of the partitions in the image. There are exactly three required keys for the dict:
* `part` - the number of the partition the file should be written to, starting at 1
* `path` - the path where the file should be written (root is the root of the partition it's being written to)
* `content` - the actual content to write to the file (a string)
#### `uploads`
This key is **optional**. Its value is a list of dicts. Each dict represents a single file that should be retrieved from a web site and copied to a partition in the image. There are exactly three required keys for the dict:
* `part` - the number of the partition the file should be written to, starting at 1
* `target` - the path where the file should be written (root is the root of the partition it's being written to)
* `source` - the URL of the file to download
There's currently no provision for uploading a *local* file, or any protocol besides http/https (you can use either).
#### `labels` and `filesystems`
These keys are **optional**. Each one's value is a list of strings. These keys together determine how many image variants are expected to be produced from the image group. If not set, the default value of `labels` is ['mbr'], and the default value of `filesystems` is ['ext4'] (both single-item lists). For each `guestfs` image group, the expected images will be the combinations of `labels` and `filesystems`. This means that if you don't set either key, or you set either key to a single item list, only a single image will be expected. If you set `labels` to a two-item list and `filesystems` to a single-item list, two images will be expected. If you set both keys to a two-item list, four images will be expected...and so on.
When multiple combinations are in play, the names of the images will include the relevant values. So if an image group has multiple entries in the `labels` list but not the `filesystems` list, the filenames will be `disk_(name)_(label1).img`, `disk_(name)_(label2).img`, and so on. If there are multiple entries in both lists, you'll get `disk_(name)_(filesystem1)_(label1).img`, `disk_(name)_(filesystem1)_(label2).img`, etc etc. The `all` subcommand will always create all the expected images; the image group subcommand will create all the expected images by default, but will have `--label` and `--filesystem` arguments allowing the user to restrict creation to a single item from either or both lists.
For `labels`, the values represent disk label types, and are passed to guestfs. The only values used at present are `gpt` and `mbr`. Obviously, `gpt` formats the disk with a GPT disk label, `mbr` formats the disk with an MBR label.
For `filesystems`, the values represent...filesystems. Any of the partitions defined in `parts` (see above) which does not specify a `filesystem` will be formatted with this filesystem.
Let's consider some examples!
Say an image group specifies `name : "blank"`, `labels : ["mbr", "gpt"]` and does not specify `filesystems`. There will be two expected images: `disk_blank_mbr.img` and `disk_blank_gpt.img`. The former will have an MBR disk label, the latter a GPT disk label. In all other respects, the images will be identical. If any partition does not specify a filesystem, it will be formatted as `ext4` (as the default for `filesystems` is `['ext4']`).
Say an image group specifies `name : "blank"`, `filesystems : ["ext4", "ntfs"]` and does not specify `labels`. There will be two expected images: `disk_blank_ext4.img` and `disk_blank_ntfs.img`. Both will have an MBR disk label (as the default for `labels` is `['mbr']`). On the former, all partitions which do not specify a `filesystem` will be formatted as ext4; on the latter, all partitions which do not specify a `filesystem` will be formatted as ntfs. In all other respects the images will be identical. As a special note: it is nonsense to specify multiple `filesystems`, but also explicitly specify a `filesystem` for each partition in `parts`. This will result in the creation of multiple identical images, because none of the values from `filesystems` will actually do anything. However, it's perfectly reasonable to have an image group with *some* partitions that explicitly specify a filesystem and *some* that do not, and then have multiple filesystem variants - say, you want multiple variant images with the data partitions formatted using different filesystems, but you want the `/boot` partition in each variant to be ext4.
Finally, say an image group specifies `name : "blank"`, `filesystems : ["ext4", "ntfs"]` and `labels : ["mbr", "gpt"]`. There will be *four* expected images, `disk_blank_ext4_mbr.img`, `disk_blank_ntfs_mbr.img`, `disk_blank_ext4_gpt.img`, `disk_blank_ntfs_gpt.img`. The `mbr` images will have MBR disk labels, the `gpt` images will have GPT disk labels, the `ext4` images will have partitions that don't explicitly specify a filesystem formatted as ext4, and the `ntfs` images will have partitions that don't explicitly specify a filesystem formatted as NTFS.
Whew! That was a lot of explanation, but it's not really a super-complicated concept, you'll get it easy. OK, let's move on to the `virtbuilder`-specific keys:
#### `releases`
This key is **required**. It defines the releases and arches for which images are expected; thus it determines the number of images that will be expected for this group. The value is a dict. Each key in the dict represents a release; the value for each key is a list of the arches for which images should be built for that release. The keys should be integer digit strings. **Positive** values indicate absolute release numbers. **Negative** values are relative to whatever is the pending release at the time the images are created. So a release number `-1` means 'the release one before the pending release at the time the images are built'. So if the next Fedora release will be Fedora 24 at the time the images are created, and one of the dict keys is `-1`, an image will be expected for Fedora 23.
The filename for a virt-builder type image always includes the release number and arch it's built for - `disk_f(release)_(name)_(arch).img`.
Let's look at an example! Say the `name` is `minimal` and the `releases` dict is `{ "-1" : ["i686", "x86_64"], "-2" : ["x86_64"] }`. Three images will be expected, and the expected releases will be relevant to the pending release. Say the pending release is Fedora 24, the expected images will be `disk_f23_minimal_i686.img` (Fedora 23 for i686), `disk_f23_minimal_x86_64.img` (Fedora 23 for x86_64), and `disk_f22_minimal_x86_64.img` (Fedora 22 for x86_64). When time moves on and the next pending release is F25, images will be expected for Fedora 23 and Fedora 24, and the Fedora 22 images will be considered obsolete and deleted by cleanup modes of `createhdds`.
As with the `guestfs` case, the single image group subcommand will have parameters to limit creation. So in our example, the `minimal` subcommand will have `--release` and `--arch` parameters, each allowing just a single value. For coding simplicity, passing `--arch` alone is ignored (this may be fixed later) and will just result in the 'expected' images being created. If `--release` is passed, only a single image will be created, for whatever release is specified; by default it will be the x86_64 image, you may pass `--arch (arch)` to build another arch instead.
#### `maxage`
This key is **optional**. If not set, it defaults to 14. Its value should be an integer string. This basically indicates how often (as a number of days) the image should be rebuilt. virt-builder images can go 'stale' - at build time we update the installed OS to the latest packages, but of course by the time the image is used, later updates may be available. If the test the image is used for needs all the latest packages to be installed, the test will have to install the later updates, and it's inefficient to have one or more tests doing that every day. So it makes sense to re-generate the images periodically so that the tests only have to install few if any updates. For any image group with a non-0 maxage, createhdds `check` and `any` modes will check any existing image file's age against maxage; if it exceeds the maxage `check` will consider it 'outdated', and `all` will rebuild it. To disable maxage checks for an image group, set `maxage` to 0.
### `commands` files
Customization of virt-builder type images, beyond the Fedora release/arch combination, is done with `.commands` files. virt-builder allows you to pass in any set of customization commands you like in a text file. It seemed easier to simply let maintainers create a `.commands` file for each virt-builder image group than to come up with a syntax for specifying customizations in the JSON. The logic is simple: for each virt-builder image group, if there is a file named `(name).commands`, that file will be passed to virt-builder as the customization command file. For instance, the `desktop.commands` file contains the customization commands for the 'desktop' virt-builder image group; it installs the Workstation package group, creates a regular user, and does a few other things. The virt-builder documentation explains all the customization commands. The `.commands` file is technically optional, but in most cases you will want to include at least the commands from `minimal.commands` - to set a root password, update packages, and schedule an SELinux relabel (as files touched by other customization commands will have incorrect labels).

656
createhdds.py Executable file
View File

@ -0,0 +1,656 @@
#!/usr/bin/env python3
# Copyright (C) 2015 Red Hat
#
# createhdds is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
#
# Author: Adam Williamson <awilliam@redhat.com>
"""Tool for creating hard disk images for Fedora openQA."""
import argparse
import logging
import json
import os
try:
import subprocess32 as subprocess
except ImportError:
import subprocess
import sys
import time
import tempfile
import fedfind.helpers
import guestfs
import pexpect
from six.moves.urllib.request import urlopen
# this is a bit icky, but it means you can run the script from
# anywhere - we use this to locate hdds.json and the virtbuilder
# image command files, so they must always be in the same place
# as the script itself. images are checked/created in the working
# directory.
SCRIPTDIR = os.path.abspath(os.path.dirname(sys.argv[0]))
logger = logging.getLogger('createhdds')
def handle_size(size):
"""Simple function to handle sizes like '10G' or '100MB', returns
the size in bytes as an int. Used by both image classes.
"""
size = str(size)
if size.endswith('G') or size.endswith('GB') or size.endswith('GiB'):
return int(size.split('G')[0]) * 1024 * 1024 * 1024
elif size.endswith('M') or size.endswith('MB') or size.endswith('MiB'):
return int(size.split('M')[0]) * 1024 * 1024
else:
return int(size)
class GuestfsImage(object):
"""Class representing an image created by guestfs. 'size' is the
desired image size, valid formats are a digit string (size in
bytes, digit string plus 'M', 'MB' or 'MiB' (size in power of two
megabytes), or digit string plus 'G', 'GB' or 'GiB' (size in power
of two gigabytes). 'imgver' is the image 'version' - in practice
it's simply a string that gets included in the image file name
if specified. 'filesystem' is the default filesystem for the image
- it will be used for parts that don't explicitly specify a
filesystem. 'label' is the disk label format to be used. parts,
writes, and uploads are lists of dicts that specify the partitions
that should be created and files that should be written or copied
('uploaded') to them. These are read in from hdds.json by
get_guestfs_images and passed here. 'name_extras' is an iterable
of strings which should be appended to the image file name, with _
separators (this provides a mechanism for get_guestfs_images to
include the label and/or filesystem in the image name when
creating images from a group with variants).
"""
def __init__(self, name, size, imgver='', filesystem='ext4', label='mbr', parts=None,
writes=None, uploads=None, name_extras=None):
self.name = name
self.size = handle_size(size)
# guestfs images are never outdated
self.outdated = False
self.filesystem = filesystem
self.label = label
self.parts = []
self.writes = []
self.uploads = []
if parts:
self.parts = parts
if writes:
self.writes = writes
if uploads:
self.uploads = uploads
self.filename = "disk_{0}".format(name)
if imgver:
self.filename = "{0}_{1}".format(self.filename, imgver)
if name_extras:
for item in name_extras:
self.filename = "{0}_{1}".format(self.filename, item)
self.filename = "{0}.img".format(self.filename)
def create(self):
"""Create the image."""
gfs = guestfs.GuestFS(python_return_dict=True)
try:
# Create the disk image with a temporary name
tmpfile = "{0}.tmp".format(self.filename)
gfs.disk_create(tmpfile, "raw", int(self.size))
# 'launch' guestfs with the disk attached
gfs.add_drive_opts(tmpfile, format="raw", readonly=0)
gfs.launch()
# identify the disk and create a disk label
disk = gfs.list_devices()[0]
gfs.part_init(disk, self.label)
# create and format the partitions
for part in self.parts:
# each partition can specify a filesystem, if it doesn't,
# we use the image default
if 'filesystem' not in part:
part['filesystem'] = self.filesystem
# create the partition: the dict must specify type ('p'
# for primary, 'l' for logical, 'e' for extended), and
# start and end sector numbers - more details in
# guestfs docs
gfs.part_add(disk, part['type'], int(part['start']), int(part['end']))
# identify the partition and format it
partn = gfs.list_partitions()[-1]
gfs.mkfs(part['filesystem'], partn, label=part.get('label'))
# do file 'writes' (create a file with a given string as
# its content)
for write in self.writes:
# the write dict must specify the partition to be
# written to, numbered from 1 as humans usually do;
# find that part and mount it
partn = gfs.list_partitions()[int(write['part'])-1]
gfs.mount(partn, "/")
# do the write: the dict must specify the target path
# and the string to be written ('content')
gfs.write(write['path'], write['content'])
# do file 'uploads'. in guestfs-speak that means transfer
# a file from the host to the image, we use it to mean
# download a file from an http server and transfer that
# to the image
for upload in self.uploads:
# download the file to a temp file - borrowed from
# fedora_openqa_schedule (which stole it from SO)
with tempfile.NamedTemporaryFile(prefix="createhdds") as dltmpfile:
resp = urlopen(upload['source'])
while True:
# This is the number of bytes to read between buffer
# flushes. Value taken from the SO example.
buff = resp.read(8192)
if not buff:
break
dltmpfile.write(buff)
# as with write, the dict must specify a target
# partition and location ('target')
partn = gfs.list_partitions()[int(upload['part'])-1]
gfs.mount(partn, "/")
gfs.upload(dltmpfile.name, upload['target'])
# we're all done! rename to the correct name
os.rename(tmpfile, self.filename)
except:
# if anything went wrong, we want to wipe the temp file
# then raise
os.remove(tmpfile)
raise
finally:
# whether things go right or wrong, we want to close the
# gfs instance
gfs.close()
class VirtBuilderImage(object):
"""Class representing an image created by virt-builder. 'release'
is the release the image will be built for. 'arch' is the arch.
'size' is the desired image size, valid formats are a digit string
(size in bytes, digit string plus 'M', 'MB' or 'MiB' (size in
power of two megabytes), or digit string plus 'G', 'GB' or 'GiB'
(size in power of two gigabytes). 0 (or any false-y value) means
the image will be the size of the upstream base image. 'imgver' is
the image 'version' - in practice it's simply a string that gets
included in the image file name if specified. 'maxage' is the
maximum age of the image file (in days) - if the image is older
than this, 'check' will report it as 'outdated' and 'all' will
rebuild it.
"""
def __init__(self, name, release, arch, size=0, imgver='', maxage=14):
self.name = name
self.size = handle_size(size)
self.filename = "disk_f{0}_{1}".format(str(release), name)
if imgver:
self.filename = "{0}_{1}".format(self.filename, imgver)
self.filename = "{0}_{1}.img".format(self.filename, arch)
self.release = release
self.arch = arch
self.maxage = maxage
def create(self):
"""Create the image."""
# Basic creation command with standard params
args = ["virt-builder", "fedora-{0}".format(str(self.release)), "-o", self.filename,
"--arch", self.arch]
if self.size:
args.extend(["--size", "{0}b".format(str(int(self.size)))])
# We use guestfs's ability to read customization commands from
# a file. The convention is to have a file 'name.commands' in
# SCRIPTDIR; if this file exists, we pass it to virt-builder.
if os.path.isfile("{0}/{1}.commands".format(SCRIPTDIR, self.name)):
args.extend(["--commands-from-file", "{0}/{1}.commands".format(SCRIPTDIR, self.name)])
ret = subprocess.call(args)
if ret > 0:
sys.exit("virt-builder command {0} failed!".format(' '.join(args)))
# We have to boot the disk to make SELinux relabelling happen;
# virt-builder can't do it unless the policy version on the host
# is the same as the guest(?) There are lots of bad ways to do
# this, expect is the one we're using.
child = pexpect.spawnu("qemu-kvm -m 2G -nographic {0}".format(self.filename), timeout=None)
child.expect(u"localhost login:")
child.sendline(u"root")
child.expect(u"Password:")
child.sendline(u"weakpassword")
child.expect(u"~]#")
child.sendline(u"poweroff")
child.expect(u"reboot: Power down")
child.close()
@property
def outdated(self):
"""Whether the image is outdated - if self.maxage is set and
it's older than that.
"""
if not os.path.isfile(self.filename):
return False
if self.maxage:
age = int(time.time()) - int(os.path.getmtime(self.filename))
# maxage is in days
if age > int(self.maxage) * 24 * 60 * 60:
return True
return False
def get_guestfs_images(imggrp, labels=None, filesystems=None):
"""Passed a single 'image group' dict (usually read out of hdds.
json), returns a list of GuestfsImage instances. labels and
filesystems act as overrides to the values specified in hdds.json
and the defaults in this function; multiple images will be created
for whatever combinations of labels and filesystems are specified.
If they're not passed, then we use the values from the dict; if
the dict doesn't specify, we just create a single image with the
label set to 'mbr' and the filesystem to 'ext4'. The filesystem
setting is itself a default - it's only used for 'part' entries
which don't specify a filesystem (see the GuestfsImage docs).
"""
imgs = []
# Read in the various dict values
name = imggrp['name']
size = imggrp['size']
parts = imggrp['parts']
# These are optional
writes = imggrp.get('writes')
uploads = imggrp.get('uploads')
imgver = imggrp.get('imgver')
# Here we implement the 'labels / filesystems' behaviour explained
# in the docstring
if not labels:
labels = imggrp.get('labels', ['mbr'])
if not filesystems:
filesystems = imggrp.get('filesystems', ['ext4'])
# Now we've sorted out all our settings, instantiate the images
for label in labels:
for filesystem in filesystems:
# We want to indicate filesystem/label in the filename
# when we're creating images with more than one. We check
# both the passed 'filesystems' value and the imggrp
# value, so if the caller overrode the imggrp value we
# still get the the long name (happens when using the
# single-image CLI subcommand and restricting the label/
# filesystem)
name_extras = []
if len(imggrp.get('filesystems', [])) > 1 or len(filesystems) > 1:
name_extras.append(filesystem)
if len(imggrp.get('labels', [])) > 1 or len(labels) > 1:
name_extras.append(label)
img = GuestfsImage(
name, size, imgver, filesystem, label, parts, writes, uploads, name_extras)
imgs.append(img)
return imgs
def get_virtbuilder_images(imggrp, nextrel=None, releases=None):
"""Passed a single 'image group' dict (usually read out of hdds.
json), returns a list of VirtBuilderImage instances. 'nextrel'
indicates the 'next' release of Fedora: sometimes we determine the
release to build image(s) for in relation to this, so we need to
know it. If it's not specified, we ask fedfind to figure it out
for us (this is the usual case, we just allow specifying it
in case there's an issue with fedfind or you want to test image
creation for the next-next release ahead of time or something).
The image group dict must include a dict named 'releases' which
indicates what releases to build images for and what arches to
build for each release; 'releases' can be used to override that. If
set, the image group 'releases' dict is ignored, and this dict is
used instead. The dict's keys must be release numbers or negative
integers: -1 means 'one release lower than the "next" release',
-2 means 'two releases lower than the "next" release', and so on.
The values are the arches to build for that release.
"""
imgs = []
# Set this here so if we need to calculate it, we only do it once
if not nextrel:
nextrel = 0
name = imggrp['name']
# this is the second place we set a default for maxage - bit ugly
maxage = int(imggrp.get('maxage', 14))
if not releases:
releases = imggrp['releases']
size = imggrp.get('size', 0)
imgver = imggrp.get('imgver')
# add an image for each release/arch combination
for (relnum, arches) in releases.items():
if int(relnum) < 0:
# negative relnum indicates 'relative to next release'
if not nextrel:
nextrel = fedfind.helpers.get_current_release() + 1
relnum = int(nextrel) + int(relnum)
for arch in arches:
imgs.append(
VirtBuilderImage(name, relnum, arch, size=size, imgver=imgver, maxage=maxage))
return imgs
def get_all_images(hdds, nextrel=None):
"""Simply iterates over the 'image group' dicts in hdds.json and
calls the appropriate get_foo_images() function for each, then
returns the list of all images. No overrides for labels,
filesystems, releases or arches are provided here, this function
is just for painting inside the lines - it's used to determine
what images are 'expected' to exist according to hdds.json. We
do allow passing of 'nextrel' just in case there's some issue
with the auto-discovery.
"""
imgs = []
for imggrp in hdds['guestfs']:
imgs.extend(get_guestfs_images(imggrp))
for imggrp in hdds['virtbuilder']:
imgs.extend(get_virtbuilder_images(imggrp, nextrel=nextrel))
return imgs
def do_renames(hdds):
"""Rename files according to the 'renames' list in hdds.json,
which is just a list of 'old name, new name' pairs. Say there's
an image we used to only create an mbr version of, but now we add
a gpt version; that means the expected name of the mbr version
changes. This function allows us to specify in hdds.json that the
mbr image should be renamed, which saves having to rebuild it.
'all' always does renames, 'check' can be told to do renames.
"""
for (orig, new) in hdds['renames']:
if os.path.isfile(orig) and not os.path.exists(new):
logger.info("Renaming %s to %s...", orig, new)
os.rename(orig, new)
def delete_all():
"""Remove absolutely all createhdds-controlled files; we assume
anything in the working directory starting with 'disk' and ending
with 'img' is one of our files.
"""
files = [fl for fl in os.listdir('.') if fl.startswith('disk') and fl.endswith('img')]
for _file in files:
os.remove(_file)
def clean(unknown):
"""This simply removes all the files in the list. The list is
expected to be the list of 'unknown' files returned by check();
both 'all' and 'clean' can delete 'unknown' files if requested.
'unknown' files are usually images for old releases we're no
longer testing, images we've simply stopped using, or old image
files when the image group name has been changed to indicate an
incompatible change to the image(s).
"""
for filename in unknown:
try:
logger.info("Removing unknown image %s", filename)
os.remove(filename)
except OSError:
# We don't really care if the file didn't exist for some
# reason, so just pass.
pass
def check(hdds, nextrel=None):
"""This calls get_all_images() to find out what images are expected
to exist, then compares that to the images that are actually
present and returns three lists. The first two are lists of Image
subclass instances, the last is a list of filenames. The first is a
list of the images that just aren't there at all. The second is a
list of images that are present but 'outdated', because they've
exceeded their maxage. The last is a list of image files that are
present but don't match any of the expected images - 'unknown'
images. The 'clean()' function can be used to remove these.
'nextrel' is passed through to get_all_images(), and is available
in case auto-detection via fedfind fails.
"""
current = []
missing = []
outdated = []
unknown = []
# Get the list of all 'expected' images
expected = get_all_images(hdds, nextrel=nextrel)
# Get the list of all present image files
files = set(fl for fl in os.listdir('.') if fl.startswith('disk') and fl.endswith('img'))
# Compare present images vs. expected images to produce 'unknown'
expnames = set(img.filename for img in expected)
unknown = list(files.difference(expnames))
# Now determine if images are absent or outdated
for img in expected:
if not os.path.isfile(img.filename):
missing.append(img)
continue
if img.outdated:
outdated.append(img)
continue
current.append(img)
logger.debug("Current images: %s", ', '.join([img.filename for img in current]))
logger.debug("Missing images: %s", ', '.join([img.filename for img in missing]))
logger.debug("Outdated images: %s", ', '.join([img.filename for img in outdated]))
logger.debug("Unknown images: %s", ', '.join(unknown))
return (missing, outdated, unknown)
def cli_all(args, hdds):
"""Function for the CLI 'all' subcommand. Creates all images. If
args.delete is set, blows all existing images away and recreates
them; otherwise it will just fill in missing or outdated images.
If args.clean is set, also wipes 'unknown' images.
"""
if args.delete:
logger.info("Removing all images...")
delete_all()
# handle renamed images (see do_renames docstring)
do_renames(hdds)
# call check() to find out what we need to do
(missing, outdated, unknown) = check(hdds, nextrel=args.nextrel)
# wipe 'unknown' images if requested
if args.clean:
clean(unknown)
# 'missing' plus 'outdated' is all the images we need to build; if
# args.delete was set, all images will be in this list
missing.extend(outdated)
for (num, img) in enumerate(missing, 1):
logger.info("Creating image %s...[%s/%s]", img.filename, str(num), str(len(missing)))
img.create()
def cli_check(args, hdds):
"""Function for the CLI 'check' subcommand. Basically just calls
check() and prints the results. Does renames before checking if
args.rename was set, and wipes 'unknown' images after checking if
args.clean was set. Exits with status 1 if any missing and/or
outdated files are found (handy for scripting/automation).
"""
if args.rename:
do_renames(hdds)
(missing, outdated, unknown) = check(hdds, nextrel=args.nextrel)
if missing:
print("Missing images: {0}".format(', '.join([img.filename for img in missing])))
if outdated:
print("Outdated images: {0}".format(', '.join([img.filename for img in outdated])))
if unknown:
print("Unknown images: {0}".format(', '.join(unknown)))
if args.clean:
clean(unknown)
if missing or outdated:
sys.exit("Missing and/or outdated images found!")
else:
sys.exit()
def cli_image(args, *_):
"""Function for CLI image group subcommands (a subcommand is added
for each image group in hdds.json). Will create the image(s) from
the specified group. For guestfs image groups with multiple labels
and/or filesystems, the user can pass args.label and/or args.
filesystem to limit creation to a single label and/or filesystem.
Note this function does no checking; the image will always be
recreated, even if it already exists and is current.
"""
# Note that on this path, the parsing of hdds is done by
# parse_args(). It passes us the image type and the image group
# dict from hdds as a tuple; we need to know the type so we know
# what fiddling to do with the args and what function to call
imggrp = args.imggrp[1]
imgtype = args.imggrp[0]
if imgtype == 'guestfs':
# If the user passed in label or filesystem, we pass them on
# to get_guestfs_images as single-item lists, otherwise we
# just pass 'None', causing it to use the values from imggrp
# (which come from hdds.json)
labels = None
filesystems = None
if args.label:
labels = [args.label]
if args.filesystem:
filesystems = [args.filesystem]
imgs = get_guestfs_images(imggrp, labels=labels, filesystems=filesystems)
elif imgtype == 'virtbuilder':
# If the user passed args.release, we construct a releases
# dict to pass to get_virtbuilder_images to override the dict
# from imggrp. If they passed args.arch, we use that arch,
# otherwise we default to x86_64. If args.release isn't set,
# we just pass None as release, and the releases dict from
# imggrp will be used, and images created for all release/
# arch combinations listed there. FIXME: if the user passes
# args.arch but not args.release, we just ignore it...
releases = None
if args.release:
if args.arch:
arches = [args.arch]
else:
arches = ['x86_64']
releases = {args.release: arches}
imgs = get_virtbuilder_images(imggrp, releases=releases)
for (num, img) in enumerate(imgs, 1):
logger.info("Creating image %s...[%s/%s]", img.filename, str(num), str(len(imgs)))
img.create()
def parse_args(hdds):
"""Parse arguments with argparse."""
parser = argparse.ArgumentParser(description=(
"Tool for creating hard disk images for Fedora openQA."))
parser.add_argument(
'-l', '--loglevel', help="The level of log messages to show",
choices=('debug', 'info', 'warning', 'error', 'critical'),
default='info')
# This is a workaround for a somewhat infamous argparse bug
# in Python 3. See:
# https://stackoverflow.com/questions/23349349/argparse-with-required-subparser
# http://bugs.python.org/issue16308
subparsers = parser.add_subparsers(dest='subcommand')
subparsers.required = True
parser_all = subparsers.add_parser(
'all', description="Ensure all images are present and up-to-date.")
parser_all.add_argument(
'-d', '--delete', help="Delete and re-build all images",
action='store_true')
parser_all.add_argument(
'-c', '--clean', help="Remove unknown (usually old) images",
action='store_true')
parser_all.add_argument(
'-n', '--nextrel', help="The release to treat as the 'next' release "
"- this determines what releases some images will be built for. If "
"not set or set to 0, createhdds will try to discover it when needed",
type=int, default=0)
parser_all.set_defaults(func=cli_all)
parser_check = subparsers.add_parser(
'check', description="Check status of existing image files.")
parser_check.add_argument(
'-n', '--nextrel', help="The release to treat as the 'next' release "
"- this determines what releases some images are expected to exist for. If "
"not set or set to 0, createhdds will try to discover it when needed",
type=int, default=0)
parser_check.add_argument(
'-r', '--rename', help="Whether to rename images when the expected name "
"has changed or not", action="store_true")
parser_check.add_argument(
'-c', '--clean', help="Remove unknown (usually old) images",
action='store_true')
parser_check.set_defaults(func=cli_check)
# This here is somewhat clever-clever: we generate a subcommand for
# each image group listed in hdds.json, which can be used to build
# image(s) from that group. For guestfs image groups, we also check
# if the group has multiple labels and/or filesystems by default,
# and add arguments to limit image creation to just a single label
# and/or filesystem.
for imggrp in hdds['guestfs']:
imgparser = subparsers.add_parser(
imggrp['name'], description="Create {0} image(s)".format(imggrp['name']))
labels = imggrp.get('labels', [])
if len(labels) > 1:
imgparser.add_argument(
'-l', '--label', help="Create only images with this disk label",
choices=labels)
filesystems = imggrp.get('filesystems', [])
if len(filesystems) > 1:
imgparser.add_argument(
'-f', '--filesystem', help="Create only images with this filesystem",
choices=filesystems)
imgparser.set_defaults(func=cli_image, label=None, filesystem=None)
# Here we're stuffing the type of the image and the dict from
# hdds into args for cli_image() to use.
imgparser.set_defaults(imggrp=('guestfs', imggrp))
# For libvirt images, we provide args to override the release/arch
# combination; using args.release will always result in just a
# single image being built, for x86_64 unless args.arch is set to
# i686.
for imggrp in hdds['virtbuilder']:
imgparser = subparsers.add_parser(
imggrp['name'], description="Create {0} image(s)".format(imggrp['name']))
imgparser.add_argument(
'-r', '--release', help="The release to build the image(s) for. If not "
"set or set to 0, createhdds will attempt to determine the current "
"release and build for appropriate releases relative to that",
type=int, default=0)
imgparser.add_argument(
'-a', '--arch', help="The arch to build the image(s) for. If neither "
"this nor --release is set, createhdds will decide the appropriate "
"arch(es) to build for each release. If this is not set but --release "
"is set, only x86_64 image(s) will be built.",
choices=('x86_64', 'i686'))
imgparser.set_defaults(func=cli_image)
# Here we're stuffing the type of the image and the dict from
# hdds into args for cli_image() to use.
imgparser.set_defaults(imggrp=('virtbuilder', imggrp))
return parser.parse_args()
def main():
"""Main loop - set up logging, parse args, run subcommand
function.
"""
try:
with open('{0}/hdds.json'.format(SCRIPTDIR), 'r') as fout:
hdds = json.load(fout)
args = parse_args(hdds)
loglevel = getattr(
logging, args.loglevel.upper(), logging.INFO)
logging.basicConfig(level=loglevel)
args.func(args, hdds)
except KeyboardInterrupt:
sys.stderr.write("Interrupted, exiting...\n")
# there may be a guestfs temp image file lying around...
tmps = [fl for fl in os.listdir('.') if fl.startswith('disk') and fl.endswith('.tmp')]
for tmp in tmps:
os.remove(tmp)
sys.exit(1)
if __name__ == '__main__':
main()

218
createhdds.sh
View File

@ -1,218 +0,0 @@
#!/bin/bash
function disk_full {
part_table_type=$1
diskname="disk_full_${part_table_type}"
echo "Creating ${diskname}.img..."
guestfish <<_EOF_
sparse ${diskname}.img 10G
run
part-init /dev/sda ${part_table_type}
part-add /dev/sda p 4096 10485760
part-add /dev/sda p 10485761 -4097
mkfs ext4 /dev/sda1
mkfs ext4 /dev/sda2
mount /dev/sda1 /
write /testfile "Hello, world!"
umount /
mount /dev/sda2 /
write /testfile "Oh, hi Mark"
umount /
_EOF_
}
function disk_freespace {
part_table_type=$1
diskname="disk_freespace_${part_table_type}"
echo "Creating ${diskname}.img..."
guestfish <<_EOF_
sparse ${diskname}.img 10G
run
part-init /dev/sda ${part_table_type}
part-add /dev/sda p 4096 2097152
mkfs ext4 /dev/sda1
mount /dev/sda1 /
write /testfile "Hello, world!"
_EOF_
}
function disk_minimal {
version=$1
arch=$2
echo "Creating disk_f${version}_minimal_${arch}.img..."
virt-builder fedora-${version} -o disk_f${version}_minimal_${arch}.img --arch ${arch} --update --selinux-relabel \
--root-password password:weakpassword > /dev/null
expect <<_EOF_
log_user 0
set timeout -1
spawn qemu-kvm -m 2G -nographic disk_f${version}_minimal_${arch}.img
expect "localhost login:"
send "root\r"
expect "Password:"
send "weakpassword\r"
expect "~]#"
send "poweroff\r"
expect "reboot: Power down"
_EOF_
}
function disk_desktop {
version=$1
arch=$2
if [ ${version} -lt 22 ]
then
cmd="yum -y remove firewalld* && yum -y groupinstall 'Fedora Workstation'"
else
cmd="dnf -y groupinstall 'Fedora Workstation'"
fi
echo "Creating disk_f${version}_desktop_${arch}.img..."
# these steps are required
# 1. update fedora
# 2. (F<22) remove firewalld - firewalld configuration in minimal and desktop are conflicting
# 3. install Fedora Workstation group
# 4. add new user on first boot
# 5. use expect to set graphical boot target and set password for user
virt-builder fedora-${version} -o disk_f${version}_desktop_${arch}.img --size 20G --arch ${arch} --update \
--run-command "${cmd}" --selinux-relabel \
--root-password password:weakpassword --firstboot-command 'useradd -m -p "" ejohn' > /dev/null
expect <<_EOF_
log_user 0
set timeout -1
spawn qemu-kvm -m 2G -nographic disk_f${version}_desktop_${arch}.img
expect "localhost login:"
send "root\r"
expect "Password:"
send "weakpassword\r"
expect "~]#"
send "systemctl set-default graphical.target\r"
send "echo 'ejohn:weakpassword' | chpasswd\r"
send "poweroff\r"
expect "reboot: Power down"
_EOF_
}
function disk_ks {
diskname="disk_ks"
echo "Creating ${diskname}.img..."
curl --silent -o "/tmp/root-user-crypted-net.ks" "https://jskladan.fedorapeople.org/kickstarts/root-user-crypted-net.ks" > /dev/null
guestfish <<_EOF_
sparse ${diskname}.img 100MB
run
part-init /dev/sda mbr
part-add /dev/sda p 4096 -1
mkfs ext4 /dev/sda1
mount /dev/sda1 /
upload /tmp/root-user-crypted-net.ks /root-user-crypted-net.ks
_EOF_
}
function disk_updates_img {
diskname="disk_updates_img"
echo "Creating ${diskname}.img..."
curl --silent -o "/tmp/updates.img" "https://fedorapeople.org/groups/qa/updates/updates-unipony.img" > /dev/null
guestfish <<_EOF_
sparse ${diskname}.img 100MB
run
part-init /dev/sda mbr
part-add /dev/sda p 4096 -1
mkfs ext4 /dev/sda1 label:UPDATES_IMG
mount /dev/sda1 /
upload /tmp/updates.img /updates.img
_EOF_
}
function disk_shrink {
fstype=$1
part_table_type=$2
diskname="disk_shrink_${fstype}_${part_table_type}"
echo "Creating ${diskname}.img..."
guestfish <<_EOF_
sparse ${diskname}.img 10G
run
part-init /dev/sda ${part_table_type}
part-add /dev/sda p 4096 -4097
mkfs $fstype /dev/sda1
mount /dev/sda1 /
write /testfile "Hello, world!"
_EOF_
}
if [[ "$1" != "" ]]; then
VERSION="$1"
shift
if [[ "$#" -eq 0 ]]; then
disk_full "mbr"
disk_full "gpt"
disk_freespace "mbr"
disk_freespace "gpt"
disk_minimal ${VERSION} "x86_64"
disk_minimal ${VERSION} "i686"
disk_desktop ${VERSION} "x86_64"
disk_desktop ${VERSION} "i686"
disk_ks
disk_updates_img
disk_shrink "ext4" "mbr"
disk_shrink "ntfs" "mbr"
disk_shrink "ext4" "gpt"
disk_shrink "ntfs" "gpt"
else
# default type of partition table is mbr
PART_TABLE_TYPE="mbr"
if [[ "$#" -gt 1 ]]; then
case $2 in
mbr)
;;
gpt)
PART_TABLE_TYPE="gpt"
;;
*)
echo "partition table type should be gpt or mbr (default)"
exit 1
;;
esac
fi
case $1 in
full)
disk_full ${PART_TABLE_TYPE}
;;
freespace)
disk_freespace ${PART_TABLE_TYPE}
;;
minimal_64bit)
disk_minimal ${VERSION} "x86_64"
;;
minimal_32bit)
disk_minimal ${VERSION} "i686"
;;
desktop_64bit)
disk_desktop ${VERSION} "x86_64"
;;
desktop_32bit)
disk_desktop ${VERSION} "i686"
;;
ks)
disk_ks
;;
updates)
disk_updates_img
;;
shrink_ext4)
disk_shrink "ext4" ${PART_TABLE_TYPE}
;;
shrink_ntfs)
disk_shrink "ntfs" ${PART_TABLE_TYPE}
;;
*)
echo "name not in [full|freespace|minimal_64bit|minimal_32bit|desktop_64bit|desktop_32bit|ks|updates|shrink_ext4|shrink_ntfs]"
exit 1
;;
esac
fi
else
echo "USAGE: $0 VERSION [full|freespace|minimal_64bit|minimal_32bit|desktop_64bit|desktop_32bit|ks|updates|shrink_ext4|shrink_ntfs] [mbr|gpt]"
exit 1
fi

7
desktop.commands Normal file
View File

@ -0,0 +1,7 @@
root-password password:weakpassword
update
selinux-relabel
install @workstation-product-environment
link /usr/lib/systemd/system/graphical.target:/etc/systemd/system/default.target
firstboot-command useradd -m -p '' ejohn
firstboot-command echo 'ejohn:weakpassword' | chpasswd

132
hdds.json Normal file
View File

@ -0,0 +1,132 @@
{
"guestfs" : [
{
"name" : "full",
"size" : "10G",
"labels" : ["mbr", "gpt"],
"parts" : [
{
"filesystem" : "ext4",
"type" : "p",
"start" : "4096",
"end" : "10485760"
},
{
"filesystem" : "ext4",
"type" : "p",
"start" : "10485761",
"end" : "-4097"
}
],
"writes" : [
{
"part" : "1",
"path" : "/testfile",
"content" : "Hello, world!"
},
{
"part" : "2",
"path" : "/testfile",
"content" : "Oh, hi Mark"
}
]
},
{
"name" : "freespace",
"size" : "10G",
"labels" : ["mbr", "gpt"],
"parts" : [
{
"filesystem" : "ext4",
"type" : "p",
"start" : "4096",
"end" : "2097152"
}
],
"writes" : [
{
"part" : "1",
"path" : "/testfile",
"content" : "Hello, world!"
}
]
},
{
"name" : "ks",
"size" : "100M",
"parts" : [
{
"filesystem" : "ext4",
"type" : "p",
"start" : "4096",
"end" : "-1"
}
],
"uploads" : [
{
"part" : "1",
"target" : "/root-user-crypted-net.ks",
"source" : "https://jskladan.fedorapeople.org/kickstarts/root-user-crypted-net.ks"
}
]
},
{
"name" : "updates_img",
"size" : "100M",
"parts" : [
{
"filesystem" : "ext4",
"label" : "UPDATES_IMG",
"type" : "p",
"start" : "4096",
"end" : "-1"
}
],
"uploads" : [
{
"part" : "1",
"target" : "/updates.img",
"source" : "https://fedorapeople.org/groups/qa/updates/updates-unipony.img"
}
]
},
{
"name" : "shrink",
"size" : "10G",
"labels" : ["mbr", "gpt"],
"filesystems" : ["ext4", "ntfs"],
"parts" : [
{
"type" : "p",
"start" : "4096",
"end" : "-4097"
}
],
"writes" : [
{
"part" : "1",
"path" : "/testfile",
"content" : "Hello, world!"
}
]
}
],
"virtbuilder" : [
{
"name" : "minimal",
"releases" : {
"-1" : ["x86_64"],
"-2" : ["x86_64"]
}
},
{
"name" : "desktop",
"releases" : {
"-1" : ["x86_64", "i686"],
"-2" : ["x86_64", "i686"]
},
"size" : "20G"
}
],
"renames" : []
}

3
minimal.commands Normal file
View File

@ -0,0 +1,3 @@
root-password password:weakpassword
update
selinux-relabel