81 lines
2.8 KiB
Diff
81 lines
2.8 KiB
Diff
From bfa0b1238f6318d06930ec4d389a27932fc90a16 Mon Sep 17 00:00:00 2001
|
|
From: "Richard W.M. Jones" <rjones@redhat.com>
|
|
Date: Sat, 4 Feb 2023 12:34:57 +0000
|
|
Subject: [PATCH] curl: Use the parallel thread model
|
|
|
|
After previous changes, it is now safe to use the parallel thread
|
|
model in this plugin. The locking in pool.c protects a single curl
|
|
handle from being used from multiple threads.
|
|
|
|
An advantage of this is we can now combine the curl plugin with
|
|
filters such as readahead and scan.
|
|
|
|
This pessimizes some workloads and improves others. See my earlier
|
|
comments here:
|
|
|
|
https://listman.redhat.com/archives/libguestfs/2023-February/030610.html
|
|
https://listman.redhat.com/archives/libguestfs/2023-February/030618.html
|
|
https://listman.redhat.com/archives/libguestfs/2023-February/030619.html
|
|
https://listman.redhat.com/archives/libguestfs/2023-February/030620.html
|
|
|
|
Tests below use this basic command:
|
|
|
|
$ time nbdkit -r -U - curl https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 \
|
|
--run '$COPY $uri /var/tmp/out'
|
|
|
|
where $COPY is either nbdcopy or qemu-img convert.
|
|
|
|
Before this change:
|
|
|
|
nbdkit + nbdcopy 0m55.397s
|
|
nbdkit + qemu-img convert [over 20 minutes]
|
|
|
|
After this change:
|
|
|
|
nbdkit + nbdcopy 1m1.235s
|
|
nbdkit + qemu-img convert 6m3.262s
|
|
nbdkit --filter=readahead --filter=cache + qemu-img convert
|
|
8m16.488s
|
|
nbdkit connections=8 + qemu-img convert
|
|
3m48.502s
|
|
|
|
Previously [see first link above] we noted that file: URLs are
|
|
impacted, and indeed they are, with this command:
|
|
|
|
$ time nbdkit -r -U - curl file:/var/tmp/fedora-36.img \
|
|
--run 'nbdcopy --no-extents -p "$uri" null:'
|
|
|
|
nearly doubling in run-time (0.78 -> 1.34). However I think this may
|
|
be a peculiarity of curl's handling of file. (Use nbdkit-file-plugin
|
|
instead).
|
|
|
|
Note the fundamental issue here is still lack of multiconn support in
|
|
qemu's NBD client.
|
|
|
|
(cherry picked from commit f2163754860d041c4cb12dace90591c280eccae8)
|
|
---
|
|
plugins/curl/curl.c | 7 +------
|
|
1 file changed, 1 insertion(+), 6 deletions(-)
|
|
|
|
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
|
|
index 58ffb662..425f1d90 100644
|
|
--- a/plugins/curl/curl.c
|
|
+++ b/plugins/curl/curl.c
|
|
@@ -537,12 +537,7 @@ curl_close (void *handle)
|
|
free (h);
|
|
}
|
|
|
|
-/* This plugin could support the parallel thread model. It currently
|
|
- * uses serialize_requests because parallel has the unfortunate effect
|
|
- * of pessimising common workloads. See:
|
|
- * https://listman.redhat.com/archives/libguestfs/2023-February/030618.html
|
|
- */
|
|
-#define THREAD_MODEL NBDKIT_THREAD_MODEL_SERIALIZE_REQUESTS
|
|
+#define THREAD_MODEL NBDKIT_THREAD_MODEL_PARALLEL
|
|
|
|
/* Calls get_handle() ... put_handle() to get a handle for the length
|
|
* of the current scope.
|
|
--
|
|
2.39.3
|
|
|