rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
{
|
|
|
|
|
"guestfs" : [
|
|
|
|
|
{
|
|
|
|
|
"name" : "full",
|
|
|
|
|
"size" : "10G",
|
|
|
|
|
"labels" : ["mbr", "gpt"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "10485760"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "10485761",
|
|
|
|
|
"end" : "-4097"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "2",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Oh, hi Mark"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "freespace",
|
|
|
|
|
"size" : "10G",
|
|
|
|
|
"labels" : ["mbr", "gpt"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "2097152"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "ks",
|
|
|
|
|
"size" : "100M",
|
2016-05-04 18:55:22 +00:00
|
|
|
"imgver" : "2",
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-1"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"uploads" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/root-user-crypted-net.ks",
|
|
|
|
|
"source" : "https://jskladan.fedorapeople.org/kickstarts/root-user-crypted-net.ks"
|
2016-05-04 18:55:22 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/freeipa.ks",
|
|
|
|
|
"source" : "https://www.happyassassin.net/ks/oqipas.ks"
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/freeipaclient.ks",
|
|
|
|
|
"source" : "https://www.happyassassin.net/ks/oqipac.ks"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "updates_img",
|
|
|
|
|
"size" : "100M",
|
2016-04-01 15:01:12 +00:00
|
|
|
"imgver" : "2",
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"filesystem" : "ext4",
|
|
|
|
|
"label" : "UPDATES_IMG",
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-1"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"uploads" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"target" : "/updates.img",
|
2016-04-01 15:01:12 +00:00
|
|
|
"source" : "https://fedorapeople.org/groups/qa/updates/updates-openqa.img"
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "shrink",
|
|
|
|
|
"size" : "10G",
|
|
|
|
|
"labels" : ["mbr", "gpt"],
|
|
|
|
|
"filesystems" : ["ext4", "ntfs"],
|
|
|
|
|
"parts" : [
|
|
|
|
|
{
|
|
|
|
|
"type" : "p",
|
|
|
|
|
"start" : "4096",
|
|
|
|
|
"end" : "-4097"
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"writes" : [
|
|
|
|
|
{
|
|
|
|
|
"part" : "1",
|
|
|
|
|
"path" : "/testfile",
|
|
|
|
|
"content" : "Hello, world!"
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"virtbuilder" : [
|
|
|
|
|
{
|
|
|
|
|
"name" : "minimal",
|
|
|
|
|
"releases" : {
|
|
|
|
|
"-1" : ["x86_64"],
|
|
|
|
|
"-2" : ["x86_64"]
|
|
|
|
|
}
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "desktop",
|
|
|
|
|
"releases" : {
|
|
|
|
|
"-1" : ["x86_64", "i686"],
|
|
|
|
|
"-2" : ["x86_64", "i686"]
|
|
|
|
|
},
|
2016-05-05 23:43:10 +00:00
|
|
|
"size" : "20G",
|
|
|
|
|
"imgver": "2"
|
2016-04-25 07:59:21 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "server",
|
|
|
|
|
"releases" : {
|
|
|
|
|
"-1" : ["x86_64", "i686"],
|
|
|
|
|
"-2" : ["x86_64", "i686"]
|
2016-05-05 23:43:10 +00:00
|
|
|
},
|
|
|
|
|
"imgver": "2"
|
2016-04-25 07:59:21 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "kde",
|
|
|
|
|
"releases" : {
|
|
|
|
|
"-1" : ["x86_64", "i686"],
|
|
|
|
|
"-2" : ["x86_64", "i686"]
|
|
|
|
|
},
|
2016-05-05 23:43:10 +00:00
|
|
|
"size" : "20G",
|
|
|
|
|
"imgver": "2"
|
createhdds: add a disk image for use by a 'support server'
Summary:
this will be used for tests like iSCSI and NFS which need a
server end, but where (unlike e.g. FreeIPA) we don't want set
up of the server end to be considered a test and run on the
compose being tested, but instead we just consider the server
end a 'test asset' and expect it to be as reliable as possible.
For now we just set it up to provide an iSCSI target, as that's
what I want to write a test for first.
Note: I was initially writing the actual iSCSI target config
file as part of image creation, but that turns out to be a bit
tricky because we can't safely write a relative or absolute path
for the source file into the commands file. We'd have to instead
allow specifying file injections in the JSON, similar to how we
do for guestfs images, then construct an absolute path in the
Python code using SCRIPTDIR, but it was easier to just write the
file in the openQA test instead, so I did that.
Test Plan:
check that creating the support image works (and if
you're being really thorough, boot it and check the package is
there).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D883
2016-06-08 22:45:54 +00:00
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"name" : "support",
|
2016-06-13 15:42:52 +00:00
|
|
|
"imgver" : "2",
|
createhdds: add a disk image for use by a 'support server'
Summary:
this will be used for tests like iSCSI and NFS which need a
server end, but where (unlike e.g. FreeIPA) we don't want set
up of the server end to be considered a test and run on the
compose being tested, but instead we just consider the server
end a 'test asset' and expect it to be as reliable as possible.
For now we just set it up to provide an iSCSI target, as that's
what I want to write a test for first.
Note: I was initially writing the actual iSCSI target config
file as part of image creation, but that turns out to be a bit
tricky because we can't safely write a relative or absolute path
for the source file into the commands file. We'd have to instead
allow specifying file injections in the JSON, similar to how we
do for guestfs images, then construct an absolute path in the
Python code using SCRIPTDIR, but it was easier to just write the
file in the openQA test instead, so I did that.
Test Plan:
check that creating the support image works (and if
you're being really thorough, boot it and check the package is
there).
Reviewers: jskladan, garretraziel
Reviewed By: garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D883
2016-06-08 22:45:54 +00:00
|
|
|
"releases" : {
|
|
|
|
|
"-1" : ["x86_64"]
|
|
|
|
|
}
|
rewrite createhdds in python, make it shinier
Summary:
createhdds.sh was just too damn simple and understandable, so
I thought I'd make it three times longer, object oriented,
and hard to understand!
OK, OK, that's not great sales. Alright. The main thing was to
make it smarter. This rewrite lets it do these things:
* Only create the images that are missing (not rebuild all)
* Work out the releases to build images for
* Rename images when appropriate
* Rebuild images when they need rebuilding
* Remove old / abandoned images
It can figure out what images ought to be present - including
working out the 'next' release and figuring out from that what
releases it needs images for - and build only the missing ones.
There's a 'version' concept for images; if the existing image
is older than the version given in the data file, it'll be
rebuilt. The data file can list 'rename' pairs, allowing images
to be renamed (like when we move from a single image to multiple
label/filesystem variants). This code uses fedfind's ability
to find the current release version to figure out what releases
we need virtbuilder images for (so you don't have to pass it
in). And it can find image files that aren't in the 'currently
expected' set and wipe them. Images can also have a 'maxage',
triggering a rebuild when they exceed it - this is intended
for the virtbuilder images, so we get a rebuild with the
latest updates every so often (default is two weeks).
The point of all this is to help with unattended deployment/
maintenance, i.e. the ansible deployment we have in infra;
the idea is that we can just set that up to run the 'all'
subcommand every so often, and it'll remove old images, create
new ones, and rebuild ones that are outdated.
I kept the ability to build a single image (or a whole image
'group'), and included the ability to just run a check without
actually doing a rebuild. There's a few little weird things
and holes here as it's not really the focus of the tool.
Test Plan:
Build all images, do a full test run, and see if
it works OK. Test out all the variations of building single
images / image groups, and using the 'check' command.
Reviewers: jskladan, garretraziel
Reviewed By: jskladan, garretraziel
Subscribers: tflink
Differential Revision: https://phab.qadevel.cloud.fedoraproject.org/D687
2015-12-17 21:25:30 +00:00
|
|
|
}
|
|
|
|
|
],
|
|
|
|
|
"renames" : []
|
|
|
|
|
}
|
