Compare commits

...

No commits in common. "imports/c8s-stream-rhel/nbdkit-1.16.2-2.module+el8.3.0+6423+e4cb6418" and "c8-stream-rhel" have entirely different histories.

25 changed files with 1278 additions and 3211 deletions

2
.gitignore vendored
View File

@ -1,2 +1,2 @@
SOURCES/libguestfs.keyring
SOURCES/nbdkit-1.16.2.tar.gz
SOURCES/nbdkit-1.24.0.tar.gz

View File

@ -1,2 +1,2 @@
1bbc40f501a7fef9eef2a39b701a71aee2fea7c4 SOURCES/libguestfs.keyring
42a5761cd3403c02c43cdf7d541ff3faaf8b4769 SOURCES/nbdkit-1.16.2.tar.gz
069720cc0d1502b007652101d293a57d7b4d7c41 SOURCES/nbdkit-1.24.0.tar.gz

View File

@ -0,0 +1,82 @@
From 99788909d9ec36e3210cf85976fe5b18da690ddd Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Wed, 4 Aug 2021 20:24:59 +0100
Subject: [PATCH] cache, cow: Fix data corruption in zero and trim on unaligned
tail
Commit eb6009b092 ("cache, cow: Reduce use of bounce-buffer") first
introduced in nbdkit 1.14 added an optimization of the
read-modify-write mechanism used for unaligned heads and tails when
zeroing in the cache layer.
Unfortunately the part applied to the tail contained a mistake: It
zeroes the end of the buffer rather than the beginning. This causes
data corruption when you use the zero or trim function with an offset
and count which is not aligned to the block size.
Although the bug has been around for years, a recent change made it
more likely to happen. Commit c1905b0a28 ("cache, cow: Use a 64K
block size by default") increased the default block size from 4K to
64K. Most filesystems use a 4K block size so operations like fstrim
will make 4K-aligned requests, and with a 4K block size also in the
cache or cow filter the unaligned case would never have been hit
before.
We can demonstrate the bug simply by filling a buffer with data
(100000 bytes in the example), and then trimming that data, which
ought to zero it out.
Before this commit there is data visible after the trim:
$ nbdkit --filter=cow data "0x21 * 100000" --run 'nbdsh -u $uri -c "h.trim(100000, 0)" ; nbdcopy $uri - | hexdump -C'
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00018000 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 |!!!!!!!!!!!!!!!!|
*
000186a0
After this commit the trim completely clears the data:
$ nbdkit --filter=cow data "0x21 * 100000" --run 'nbdsh -u $uri -c "h.trim(100000, 0)" ; nbdcopy $uri - | hexdump -C'
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000186a0
Thanks: Ming Xie for finding the bug
Fixes: commit eb6009b092ae642ed25f133d487dd40ef7bf70f8
(cherry picked from commit a0ae7b2158598ce48ac31706319007f716d01c87)
(cherry picked from commit c0b15574647672cb5c48178333acdd07424692ef)
---
filters/cache/cache.c | 2 +-
filters/cow/cow.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/filters/cache/cache.c b/filters/cache/cache.c
index 91dcc43d..0616cc7b 100644
--- a/filters/cache/cache.c
+++ b/filters/cache/cache.c
@@ -493,7 +493,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
r = blk_read (next_ops, nxdata, blknum, block, err);
if (r != -1) {
- memset (&block[count], 0, blksize - count);
+ memset (block, 0, count);
r = blk_write (next_ops, nxdata, blknum, block, flags, err);
}
if (r == -1)
diff --git a/filters/cow/cow.c b/filters/cow/cow.c
index 51ca64a4..1cfcc4e7 100644
--- a/filters/cow/cow.c
+++ b/filters/cow/cow.c
@@ -419,7 +419,7 @@ cow_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
r = blk_read (next_ops, nxdata, blknum, block, err);
if (r != -1) {
- memset (&block[count], 0, BLKSIZE - count);
+ memset (block, 0, count);
r = blk_write (blknum, block, err);
}
if (r == -1)
--
2.31.1

View File

@ -1,74 +0,0 @@
From d7836fb0a7131c725e3c02be7e48e99c671637c3 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 08:57:15 +0000
Subject: [PATCH] server: Allow -D nbdkit.* debug flags for the core server.
These work like plugin/filter debug flags, but apply to the internals
of the server.
(cherry picked from commit 3b45db234a691f8ff926a6fef583e11c3601f112)
---
docs/nbdkit.pod | 7 +++++++
docs/synopsis.txt | 2 +-
server/main.c | 3 +++
server/nbdkit.syms | 2 ++
4 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index a2e72b1..346d833 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -177,6 +177,13 @@ Display brief command line usage information and exit.
Set the plugin or filter Debug Flag called C<FLAG> to the integer
value C<N>. See L<nbdkit-plugin(3)/Debug Flags>.
+=item B<-D> nbdkit.FLAG=N
+
+=item B<--debug> nbdkit.FLAG=N
+
+Set the nbdkit server Debug Flag called C<FLAG> to the integer value
+C<N>.
+
=item B<--dump-config>
Dump out the compile-time configuration values and exit.
diff --git a/docs/synopsis.txt b/docs/synopsis.txt
index 3c23937..c367542 100644
--- a/docs/synopsis.txt
+++ b/docs/synopsis.txt
@@ -1,4 +1,4 @@
-nbdkit [-D|--debug PLUGIN|FILTER.FLAG=N]
+nbdkit [-D|--debug PLUGIN|FILTER|nbdkit.FLAG=N]
[-e|--exportname EXPORTNAME] [--exit-with-parent]
[--filter FILTER ...] [-f|--foreground]
[-g|--group GROUP] [-i|--ipaddr IPADDR]
diff --git a/server/main.c b/server/main.c
index d39941b..11ba1e6 100644
--- a/server/main.c
+++ b/server/main.c
@@ -563,6 +563,9 @@ main (int argc, char *argv[])
free (t);
}
+ /* Apply nbdkit.* flags for the server. */
+ apply_debug_flags (NULL, "nbdkit");
+
/* Check all debug flags were used, and free them. */
free_debug_flags ();
diff --git a/server/nbdkit.syms b/server/nbdkit.syms
index 390972e..96c22c0 100644
--- a/server/nbdkit.syms
+++ b/server/nbdkit.syms
@@ -67,6 +67,8 @@
nbdkit_vdebug;
nbdkit_verror;
+ nbdkit_debug_*;
+
# Everything else is hidden.
local: *;
};
--
2.18.2

View File

@ -1,66 +0,0 @@
From e5d2d44fff9214725506cbc84e7b3c035ec0eae9 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 11:06:36 +0000
Subject: [PATCH] server: Allow -D debug flags to contain dots for namespacing.
This is just a convenience. Either of:
-D myplugin.foo_bar=1
-D myplugin.foo.bar=1
correspond to the same plugin variable "myplugin_debug_foo_bar".
(cherry picked from commit a895fa84aaa50f52af68319523020046394c789f)
---
docs/nbdkit-plugin.pod | 8 ++++++++
server/debug-flags.c | 10 +++++++++-
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/docs/nbdkit-plugin.pod b/docs/nbdkit-plugin.pod
index b69cb82..879ddf0 100644
--- a/docs/nbdkit-plugin.pod
+++ b/docs/nbdkit-plugin.pod
@@ -1298,6 +1298,14 @@ You should only use this feature for debug settings. For general
settings use ordinary plugin parameters. Debug Flags can only be C
ints. They are not supported by non-C language plugins.
+For convenience C<'.'> characters are replaced with C<'_'> characters
+in the variable name, so both of these parameters:
+
+ -D myplugin.foo_bar=1
+ -D myplugin.foo.bar=1
+
+correspond to the plugin variable C<myplugin_debug_foo_bar>.
+
=head1 INSTALLING THE PLUGIN
The plugin is a C<*.so> file and possibly a manual page. You can of
diff --git a/server/debug-flags.c b/server/debug-flags.c
index 9344d85..5e06f5e 100644
--- a/server/debug-flags.c
+++ b/server/debug-flags.c
@@ -56,12 +56,20 @@ static char *
symbol_of_debug_flag (const char *name, const char *flag)
{
char *var;
+ size_t i;
+ int len;
- if (asprintf (&var, "%s_debug_%s", name, flag) == -1) {
+ if ((len = asprintf (&var, "%s_debug_%s", name, flag)) == -1) {
perror ("asprintf");
exit (EXIT_FAILURE);
}
+ /* If there are any '.'s remaining in the name, convert them to '_'. */
+ for (i = 0; i < (size_t) len; ++i) {
+ if (var[i] == '.')
+ var[i] = '_';
+ }
+
return var; /* caller frees */
}
--
2.18.2

View File

@ -0,0 +1,94 @@
From 6b9d4380df9bd0be91f49aad8c4f47b4e672adde Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Mon, 16 Aug 2021 13:43:29 -0500
Subject: [PATCH] server: CVE-2021-3716 reset structured replies on starttls
https://nostarttls.secvuln.info/ pointed out a series of CVEs in
common implementation flaw in various SMTP and IMAP clients and
servers, all with a common thread of improperly caching plaintext
state across the STARTTLS encryption boundary; and recommended that
other protocols with a STARTTLS operation perform a similar audit.
It turns out that nbdkit has the same vulnerability in regards to the
NBD protocol: when nbdkit is run in opportunistic TLS mode, an
attacker is able to inject a plaintext NBD_OPT_STRUCTURED_REPLY before
proxying everything else a client sends to the server; if the server
then acts on that plaintext request (as nbdkit did before this patch),
then the server ends up sending structured replies to at least
NBD_CMD_READ, even though the client was assuming that the transition
to TLS has ruled out a MitM attack.
On the bright side, nbdkit's behavior on a second
NBD_OPT_STRUCTURED_REPLY was to still reply with success, so a client
that always requests structured replies after starting TLS sees no
difference in behavior (that is, qemu 2.12 and later are immune) (had
nbdkit given an error to the second request, that may have caused
confusion to more clients). And there is always the mitigation of
using --tls=require, which lets nbdkit reject the MitM message
pre-encryption. However, nbd-client 3.15 to the present do not
understand structured replies, and I have confirmed that a MitM
attacker can thus cause a denial-of-service attack that does not
trigger until the client does its first encrypted NBD_CMD_READ.
The NBD spec has been recently tightened to declare the nbdkit
behavior to be a security hole:
https://github.com/NetworkBlockDevice/nbd/commit/77e55378096aa
Fixes: eaa4c6e9a2c4bd (server: Minimal implementation of NBD Structured Replies.)
(cherry picked from commit 09a13dafb7bb3a38ab52eb5501cba786365ba7fd)
(cherry picked from commit 6185b15a81e6915734d678f0781e31d45a7941a1)
---
docs/nbdkit-security.pod | 11 +++++++++--
server/protocol-handshake-newstyle.c | 3 ++-
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/docs/nbdkit-security.pod b/docs/nbdkit-security.pod
index 3a28e54d..5a4e6da8 100644
--- a/docs/nbdkit-security.pod
+++ b/docs/nbdkit-security.pod
@@ -10,7 +10,7 @@ For how to report new security issues, see the C<SECURITY> file in the
top level source directory, also available online here:
L<https://github.com/libguestfs/nbdkit/blob/master/SECURITY>
-=head2 CVE-2019-14850
+=head2 CVE-2019-14850
denial of service due to premature opening of back-end connection
See the full announcement and links to mitigation, tests and fixes
@@ -26,6 +26,13 @@ See the full announcement and links to mitigation, tests and fixes
here:
https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html
+=head2 CVE-2021-3716
+structured read denial of service attack against starttls
+
+See the full announcement and links to mitigation, tests and fixes
+here:
+https://www.redhat.com/archives/libguestfs/2021-August/msg00083.html
+
=head1 SEE ALSO
L<nbdkit(1)>.
@@ -38,4 +45,4 @@ Richard W.M. Jones
=head1 COPYRIGHT
-Copyright (C) 2013-2020 Red Hat Inc.
+Copyright (C) 2013-2021 Red Hat Inc.
diff --git a/server/protocol-handshake-newstyle.c b/server/protocol-handshake-newstyle.c
index 0a76a814..b94950e2 100644
--- a/server/protocol-handshake-newstyle.c
+++ b/server/protocol-handshake-newstyle.c
@@ -495,7 +495,8 @@ negotiate_handshake_newstyle_options (void)
return -1;
conn->using_tls = true;
debug ("using TLS on this connection");
- /* Wipe out any cached default export name. */
+ /* Wipe out any cached state. */
+ conn->structured_replies = false;
for_each_backend (b) {
struct handle *h = get_handle (conn, b->i);
free (h->default_exportname);
--
2.31.1

View File

@ -1,451 +0,0 @@
From 83c72d9bf9d6a9ccf6939b4ebd0028b62673a78a Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 10:57:52 +0000
Subject: [PATCH] server: Add -D nbdkit.backend.controlpath and -D
nbdkit.backend.datapath.
These can be used to suppress verbose debugging messages from the
backend.
BugLink: https://bugzilla.redhat.com/1782868
Cherry picked from commit 231717e8cd5f27d76631be6651062d5a5ccf7fdc.
Remove use of nofilter from the test.
---
docs/nbdkit.pod | 35 ++++++++++++-
server/backend.c | 83 ++++++++++++++++++------------
tests/Makefile.am | 4 ++
tests/test-nbdkit-backend-debug.sh | 70 +++++++++++++++++++++++++
4 files changed, 158 insertions(+), 34 deletions(-)
create mode 100755 tests/test-nbdkit-backend-debug.sh
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index 346d833..38e6bfc 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -182,7 +182,7 @@ value C<N>. See L<nbdkit-plugin(3)/Debug Flags>.
=item B<--debug> nbdkit.FLAG=N
Set the nbdkit server Debug Flag called C<FLAG> to the integer value
-C<N>.
+C<N>. See L</SERVER DEBUG FLAGS> below.
=item B<--dump-config>
@@ -527,6 +527,39 @@ languages. The file should be executable. For example:
(see L<nbdkit-perl-plugin(3)> for a full example).
+=head1 SERVER DEBUG FLAGS
+
+As well as enabling or disabling debugging in the server using
+I<--verbose> you can control extra debugging in the server using the
+C<-D nbdkit.*> flags listed in this section. Note these flags are an
+internal implementation detail of the server and may be changed or
+removed at any time in the future.
+
+=over 4
+
+=item B<-D nbdkit.backend.controlpath=0>
+
+=item B<-D nbdkit.backend.controlpath=1>
+
+=item B<-D nbdkit.backend.datapath=0>
+
+=item B<-D nbdkit.backend.datapath=1>
+
+These flags control the verbosity of nbdkit backend debugging messages
+(the ones which show every request processed by the server). The
+default for both settings is C<1> (normal debugging) but you can set
+them to C<0> to suppress these messages.
+
+C<-D nbdkit.backend.datapath=0> is the more useful setting which lets you
+suppress messages about pread, pwrite, zero, trim, etc. commands.
+When transferring large amounts of data these messages are numerous
+and not usually very interesting.
+
+C<-D nbdkit.backend.controlpath=0> suppresses the non-datapath
+commands (config, open, close, can_write, etc.)
+
+=back
+
=head1 SIGNALS
nbdkit responds to the following signals:
diff --git a/server/backend.c b/server/backend.c
index b9fe2a2..208c07b 100644
--- a/server/backend.c
+++ b/server/backend.c
@@ -46,6 +46,22 @@
/* Helpers for registering a new backend. */
+/* Use:
+ * -D nbdkit.backend.controlpath=0 to suppress control path debugging.
+ * -D nbdkit.backend.datapath=0 to suppress data path debugging.
+ */
+int nbdkit_debug_backend_controlpath = 1;
+int nbdkit_debug_backend_datapath = 1;
+
+#define controlpath_debug(fs, ...) \
+ do { \
+ if (nbdkit_debug_backend_controlpath) debug ((fs), ##__VA_ARGS__); \
+ } while (0)
+#define datapath_debug(fs, ...) \
+ do { \
+ if (nbdkit_debug_backend_datapath) debug ((fs), ##__VA_ARGS__); \
+ } while (0)
+
void
backend_init (struct backend *b, struct backend *next, size_t index,
const char *filename, void *dl, const char *type)
@@ -108,7 +124,7 @@ backend_load (struct backend *b, const char *name, void (*load) (void))
apply_debug_flags (b->dl, name);
/* Call the on-load callback if it exists. */
- debug ("%s: load", name);
+ controlpath_debug ("%s: load", name);
if (load)
load ();
}
@@ -121,7 +137,7 @@ backend_unload (struct backend *b, void (*unload) (void))
*/
lock_unload ();
- debug ("%s: unload %s", b->name, b->type);
+ controlpath_debug ("%s: unload %s", b->name, b->type);
if (unload)
unload ();
@@ -139,7 +155,7 @@ backend_open (struct backend *b, struct connection *conn, int readonly)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: open readonly=%d", b->name, readonly);
+ controlpath_debug ("%s: open readonly=%d", b->name, readonly);
assert (h->handle == NULL);
assert ((h->state & HANDLE_OPEN) == 0);
@@ -151,7 +167,7 @@ backend_open (struct backend *b, struct connection *conn, int readonly)
* inner-to-outer ordering.
*/
h->handle = b->open (b, conn, readonly);
- debug ("%s: open returned handle %p", b->name, h->handle);
+ controlpath_debug ("%s: open returned handle %p", b->name, h->handle);
if (h->handle == NULL) {
if (b->i) /* Do not strand backend if this layer failed */
@@ -179,7 +195,7 @@ backend_prepare (struct backend *b, struct connection *conn)
if (b->i && backend_prepare (b->next, conn) == -1)
return -1;
- debug ("%s: prepare readonly=%d", b->name, h->can_write == 0);
+ controlpath_debug ("%s: prepare readonly=%d", b->name, h->can_write == 0);
if (b->prepare (b, conn, h->handle, h->can_write == 0) == -1)
return -1;
@@ -196,7 +212,7 @@ backend_finalize (struct backend *b, struct connection *conn)
* filter furthest away from the plugin, and matching .close order.
*/
- debug ("%s: finalize", b->name);
+ controlpath_debug ("%s: finalize", b->name);
/* Once finalize fails, we can do nothing further on this connection */
if (h->state & HANDLE_FAILED)
@@ -223,7 +239,7 @@ backend_close (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
/* outer-to-inner order, opposite .open */
- debug ("%s: close", b->name);
+ controlpath_debug ("%s: close", b->name);
if (h->handle) {
assert (h->state & HANDLE_OPEN);
@@ -252,7 +268,7 @@ backend_valid_range (struct backend *b, struct connection *conn,
int
backend_reopen (struct backend *b, struct connection *conn, int readonly)
{
- debug ("%s: reopen readonly=%d", b->name, readonly);
+ controlpath_debug ("%s: reopen readonly=%d", b->name, readonly);
if (backend_finalize (b, conn) == -1)
return -1;
@@ -274,7 +290,7 @@ backend_get_size (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: get_size", b->name);
+ controlpath_debug ("%s: get_size", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->exportsize == -1)
@@ -287,7 +303,7 @@ backend_can_write (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_write", b->name);
+ controlpath_debug ("%s: can_write", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_write == -1)
@@ -300,7 +316,7 @@ backend_can_flush (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_flush", b->name);
+ controlpath_debug ("%s: can_flush", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_flush == -1)
@@ -313,7 +329,7 @@ backend_is_rotational (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: is_rotational", b->name);
+ controlpath_debug ("%s: is_rotational", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->is_rotational == -1)
@@ -327,7 +343,7 @@ backend_can_trim (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_trim", b->name);
+ controlpath_debug ("%s: can_trim", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_trim == -1) {
@@ -347,7 +363,7 @@ backend_can_zero (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_zero", b->name);
+ controlpath_debug ("%s: can_zero", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_zero == -1) {
@@ -367,7 +383,7 @@ backend_can_fast_zero (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_fast_zero", b->name);
+ controlpath_debug ("%s: can_fast_zero", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_fast_zero == -1) {
@@ -386,7 +402,7 @@ backend_can_extents (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_extents", b->name);
+ controlpath_debug ("%s: can_extents", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_extents == -1)
@@ -400,7 +416,7 @@ backend_can_fua (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_fua", b->name);
+ controlpath_debug ("%s: can_fua", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_fua == -1) {
@@ -420,7 +436,7 @@ backend_can_multi_conn (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
assert (h->handle && (h->state & HANDLE_CONNECTED));
- debug ("%s: can_multi_conn", b->name);
+ controlpath_debug ("%s: can_multi_conn", b->name);
if (h->can_multi_conn == -1)
h->can_multi_conn = b->can_multi_conn (b, conn, h->handle);
@@ -432,7 +448,7 @@ backend_can_cache (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_cache", b->name);
+ controlpath_debug ("%s: can_cache", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_cache == -1)
@@ -451,8 +467,8 @@ backend_pread (struct backend *b, struct connection *conn,
assert (h->handle && (h->state & HANDLE_CONNECTED));
assert (backend_valid_range (b, conn, offset, count));
assert (flags == 0);
- debug ("%s: pread count=%" PRIu32 " offset=%" PRIu64,
- b->name, count, offset);
+ datapath_debug ("%s: pread count=%" PRIu32 " offset=%" PRIu64,
+ b->name, count, offset);
r = b->pread (b, conn, h->handle, buf, count, offset, flags, err);
if (r == -1)
@@ -475,8 +491,8 @@ backend_pwrite (struct backend *b, struct connection *conn,
assert (!(flags & ~NBDKIT_FLAG_FUA));
if (fua)
assert (h->can_fua > NBDKIT_FUA_NONE);
- debug ("%s: pwrite count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
- b->name, count, offset, fua);
+ datapath_debug ("%s: pwrite count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
+ b->name, count, offset, fua);
r = b->pwrite (b, conn, h->handle, buf, count, offset, flags, err);
if (r == -1)
@@ -494,7 +510,7 @@ backend_flush (struct backend *b, struct connection *conn,
assert (h->handle && (h->state & HANDLE_CONNECTED));
assert (h->can_flush == 1);
assert (flags == 0);
- debug ("%s: flush", b->name);
+ datapath_debug ("%s: flush", b->name);
r = b->flush (b, conn, h->handle, flags, err);
if (r == -1)
@@ -518,8 +534,8 @@ backend_trim (struct backend *b, struct connection *conn,
assert (!(flags & ~NBDKIT_FLAG_FUA));
if (fua)
assert (h->can_fua > NBDKIT_FUA_NONE);
- debug ("%s: trim count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
- b->name, count, offset, fua);
+ datapath_debug ("%s: trim count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
+ b->name, count, offset, fua);
r = b->trim (b, conn, h->handle, count, offset, flags, err);
if (r == -1)
@@ -547,9 +563,10 @@ backend_zero (struct backend *b, struct connection *conn,
assert (h->can_fua > NBDKIT_FUA_NONE);
if (fast)
assert (h->can_fast_zero == 1);
- debug ("%s: zero count=%" PRIu32 " offset=%" PRIu64
- " may_trim=%d fua=%d fast=%d",
- b->name, count, offset, !!(flags & NBDKIT_FLAG_MAY_TRIM), fua, fast);
+ datapath_debug ("%s: zero count=%" PRIu32 " offset=%" PRIu64
+ " may_trim=%d fua=%d fast=%d",
+ b->name, count, offset,
+ !!(flags & NBDKIT_FLAG_MAY_TRIM), fua, fast);
r = b->zero (b, conn, h->handle, count, offset, flags, err);
if (r == -1) {
@@ -572,8 +589,8 @@ backend_extents (struct backend *b, struct connection *conn,
assert (h->can_extents >= 0);
assert (backend_valid_range (b, conn, offset, count));
assert (!(flags & ~NBDKIT_FLAG_REQ_ONE));
- debug ("%s: extents count=%" PRIu32 " offset=%" PRIu64 " req_one=%d",
- b->name, count, offset, !!(flags & NBDKIT_FLAG_REQ_ONE));
+ datapath_debug ("%s: extents count=%" PRIu32 " offset=%" PRIu64 " req_one=%d",
+ b->name, count, offset, !!(flags & NBDKIT_FLAG_REQ_ONE));
if (h->can_extents == 0) {
/* By default it is safe assume that everything in the range is
@@ -602,8 +619,8 @@ backend_cache (struct backend *b, struct connection *conn,
assert (h->can_cache > NBDKIT_CACHE_NONE);
assert (backend_valid_range (b, conn, offset, count));
assert (flags == 0);
- debug ("%s: cache count=%" PRIu32 " offset=%" PRIu64,
- b->name, count, offset);
+ datapath_debug ("%s: cache count=%" PRIu32 " offset=%" PRIu64,
+ b->name, count, offset);
if (h->can_cache == NBDKIT_CACHE_EMULATE) {
static char buf[MAX_REQUEST_SIZE]; /* data sink, never read */
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 0134197..d225cc6 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -135,6 +135,7 @@ EXTRA_DIST = \
test-nbd-extents.sh \
test-nbd-tls.sh \
test-nbd-tls-psk.sh \
+ test-nbdkit-backend-debug.sh \
test-nozero.sh \
test-null-extents.sh \
test_ocaml_plugin.ml \
@@ -746,6 +747,9 @@ endif HAVE_VDDK
# zero plugin test.
TESTS += test-zero.sh
+# -D nbdkit.backend.* settings.
+TESTS += test-nbdkit-backend-debug.sh
+
#----------------------------------------------------------------------
# Tests of language plugins.
diff --git a/tests/test-nbdkit-backend-debug.sh b/tests/test-nbdkit-backend-debug.sh
new file mode 100755
index 0000000..69a69a7
--- /dev/null
+++ b/tests/test-nbdkit-backend-debug.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+source ./functions.sh
+set -x
+set -e
+
+requires qemu-img --version
+
+out="test-nbdkit-backend-debug.out"
+debug="test-nbdkit-backend-debug.debug"
+files="$out $debug"
+rm -f $files
+cleanup_fn rm -f $files
+
+nbdkit -U - \
+ -v \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain all debugging messages.
+grep '^nbdkit:.*debug: memory: open' $debug
+grep '^nbdkit:.*debug: memory: pread' $debug
+
+nbdkit -U - \
+ -v -D nbdkit.backend.controlpath=0 \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain only datapath messages.
+grep -v '^nbdkit:.*debug: memory: open' $debug
+grep '^nbdkit:.*debug: memory: pread' $debug
+
+nbdkit -U - \
+ -v -D nbdkit.backend.datapath=0 \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain only controlpath messages.
+grep '^nbdkit:.*debug: memory: open' $debug
+grep -v '^nbdkit:.*debug: memory: pread' $debug
--
2.18.2

View File

@ -0,0 +1,40 @@
From add9b794b9dc697a1b52115c997fcfb6e06bf64c Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Mon, 16 Aug 2021 13:43:29 -0500
Subject: [PATCH] server: reset meta context replies on starttls
Related to CVE-2021-3716, but not as severe. No compliant client will
send NBD_CMD_BLOCK_STATUS unless it first negotiates
NBD_OPT_SET_META_CONTEXT. If an attacker injects a premature
SET_META_CONTEXT, either the client will never notice (because it
never uses BLOCK_STATUS), or the client will overwrite the attacker's
attempt with the client's own SET_META_CONTEXT request after
encryption is enabled. So I don't class this as having the potential
to trigger denial-of-service due to any protocol mismatch between
compliant client and server (I don't care what happens with
non-compliant clients).
Fixes: 26455d45 (server: protocol: Implement Block Status "base:allocation".)
(cherry picked from commit 6c5faac6a37077cf2366388a80862bb00616d0d8)
(cherry picked from commit 814d8103fb4b581dc01dfd25d2cd81596576f211)
---
server/protocol-handshake-newstyle.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/server/protocol-handshake-newstyle.c b/server/protocol-handshake-newstyle.c
index b94950e2..eb0f3961 100644
--- a/server/protocol-handshake-newstyle.c
+++ b/server/protocol-handshake-newstyle.c
@@ -497,6 +497,9 @@ negotiate_handshake_newstyle_options (void)
debug ("using TLS on this connection");
/* Wipe out any cached state. */
conn->structured_replies = false;
+ free (conn->exportname_from_set_meta_context);
+ conn->exportname_from_set_meta_context = NULL;
+ conn->meta_context_base_allocation = false;
for_each_backend (b) {
struct handle *h = get_handle (conn, b->i);
free (h->default_exportname);
--
2.31.1

View File

@ -0,0 +1,59 @@
From 3c2879a38c299b725091cea45329879e3f46fc99 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 31 Aug 2021 11:23:27 +0100
Subject: [PATCH] cow: Fix for qemu 6.1 which requires backing format
The diffing example in the manual created a qcow2 file with a backing
file but did not specify the backing format. However qemu 6.1 now
requires this and fails with:
qemu-img: cow-diff.qcow2: Backing file specified without backing format
or:
qemu-img: Could not change the backing file to 'cow-base.img': backing format must be specified
Fix the example by adding the -F option to the command line.
Also there was a test of this rebasing sequence which failed, so this
commit updates the test too.
(cherry picked from commit 618290ef33ce13b75c1a79fea1f1ffb327b5ba07)
---
filters/cow/nbdkit-cow-filter.pod | 4 ++--
tests/test-cow.sh | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/filters/cow/nbdkit-cow-filter.pod b/filters/cow/nbdkit-cow-filter.pod
index 4d5ae856..510bdd40 100644
--- a/filters/cow/nbdkit-cow-filter.pod
+++ b/filters/cow/nbdkit-cow-filter.pod
@@ -101,8 +101,8 @@ At the end, disconnect the client.
Run these C<qemu-img> commands to construct a qcow2 file containing
the differences:
- qemu-img create -f qcow2 -b nbd:localhost diff.qcow2
- qemu-img rebase -b disk.img diff.qcow2
+ qemu-img create -F raw -b nbd:localhost -f qcow2 diff.qcow2
+ qemu-img rebase -F raw -b disk.img -f qcow2 diff.qcow2
F<diff.qcow2> now contains the differences between the base
(F<disk.img>) and the changes stored in nbdkit-cow-filter. C<nbdkit>
diff --git a/tests/test-cow.sh b/tests/test-cow.sh
index 8772afd7..edc4c223 100755
--- a/tests/test-cow.sh
+++ b/tests/test-cow.sh
@@ -72,8 +72,8 @@ fi
# If we have qemu-img, try the hairy rebase operation documented
# in the nbdkit-cow-filter manual.
if qemu-img --version >/dev/null 2>&1; then
- qemu-img create -f qcow2 -b nbd:unix:$sock cow-diff.qcow2
- time qemu-img rebase -b cow-base.img cow-diff.qcow2
+ qemu-img create -F raw -b nbd:unix:$sock -f qcow2 cow-diff.qcow2
+ time qemu-img rebase -F raw -b cow-base.img -f qcow2 cow-diff.qcow2
qemu-img info cow-diff.qcow2
# This checks the file we created exists.
--
2.31.1

View File

@ -1,65 +0,0 @@
From b646050b8da51c39cf21f95fa847c12784a1169c Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 15:02:44 +0000
Subject: [PATCH] python: Add various constants to the API.
These are accessible from the plugin by:
import nbdkit
if flags & nbdkit.FLAG_MAY_TRIM:
&c.
Many (all?) of these are not yet useful for plugins, some will never
be useful, but they only consume a tiny bit of memory and it's nice to
have the complete set available for future use.
(cherry picked from commit 14b7fe2e0de881e3dfc8803484ade29a61e323c9)
---
plugins/python/python.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 7052aac..47da083 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -231,6 +231,36 @@ create_nbdkit_module (void)
nbdkit_error ("could not create the nbdkit API module");
exit (EXIT_FAILURE);
}
+
+ /* Constants corresponding to various flags. */
+#define ADD_INT_CONSTANT(name) \
+ if (PyModule_AddIntConstant (m, #name, NBDKIT_##name) == -1) { \
+ nbdkit_error ("could not add constant %s to nbdkit API module", \
+ #name); \
+ exit (EXIT_FAILURE); \
+ }
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_CONNECTIONS);
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_ALL_REQUESTS);
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_REQUESTS);
+ ADD_INT_CONSTANT (THREAD_MODEL_PARALLEL);
+
+ ADD_INT_CONSTANT (FLAG_MAY_TRIM);
+ ADD_INT_CONSTANT (FLAG_FUA);
+ ADD_INT_CONSTANT (FLAG_REQ_ONE);
+ ADD_INT_CONSTANT (FLAG_FAST_ZERO);
+
+ ADD_INT_CONSTANT (FUA_NONE);
+ ADD_INT_CONSTANT (FUA_EMULATE);
+ ADD_INT_CONSTANT (FUA_NATIVE);
+
+ ADD_INT_CONSTANT (CACHE_NONE);
+ ADD_INT_CONSTANT (CACHE_EMULATE);
+ ADD_INT_CONSTANT (CACHE_NATIVE);
+
+ ADD_INT_CONSTANT (EXTENT_HOLE);
+ ADD_INT_CONSTANT (EXTENT_ZERO);
+#undef ADD_INT_CONSTANT
+
return m;
}
--
2.18.2

View File

@ -1,558 +0,0 @@
From 49ef7e7d7c3602cc8e53d2052fce9d3a12840ea2 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 15:44:39 +0000
Subject: [PATCH] python: Implement nbdkit API version 2.
To avoid breaking existing plugins, Python plugins wishing to use
version 2 of the API must opt in by declaring:
API_VERSION = 2
(Plugins which do not do this are assumed to want API version 1).
For v2 API, we also avoid a copy by passing a buffer into pread.
It's more efficient if we pass the C buffer directly to Python code.
In some cases the Python code will be able to write directly into the
C buffer using functions like file.readinto and socket.recv_into.
This avoids an extra copy.
Thanks: Nir Soffer
https://www.redhat.com/archives/libguestfs/2019-November/thread.html#00220
(cherry picked from commit a9b2637cf4f00fb8a25ffaf31ee83be5fe019ae2)
---
plugins/python/example.py | 20 +++-
plugins/python/nbdkit-python-plugin.pod | 69 +++++++-----
plugins/python/python.c | 139 +++++++++++++++++++-----
tests/python-exception.py | 4 +-
tests/shebang.py | 5 +-
tests/test.py | 28 +++--
6 files changed, 190 insertions(+), 75 deletions(-)
diff --git a/plugins/python/example.py b/plugins/python/example.py
index 60f9d7f..c04b7e2 100644
--- a/plugins/python/example.py
+++ b/plugins/python/example.py
@@ -34,6 +34,12 @@ import errno
disk = bytearray(1024 * 1024)
+# There are several variants of the API. nbdkit will call this
+# function first to determine which one you want to use. This is the
+# latest version at the time this example was written.
+API_VERSION = 2
+
+
# This just prints the extra command line parameters, but real plugins
# should parse them and reject any unknown parameters.
def config(key, value):
@@ -54,20 +60,22 @@ def get_size(h):
return len(disk)
-def pread(h, count, offset):
+def pread(h, buf, offset, flags):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
+ # or if reading from a file you can use:
+ #f.readinto(buf)
-
-def pwrite(h, buf, offset):
+def pwrite(h, buf, offset, flags):
global disk
end = offset + len(buf)
disk[offset:end] = buf
-def zero(h, count, offset, may_trim):
+def zero(h, count, offset, flags):
global disk
- if may_trim:
+ if flags & nbdkit.FLAG_MAY_TRIM:
disk[offset:offset+count] = bytearray(count)
else:
nbdkit.set_error(errno.EOPNOTSUPP)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 3680fd6..4923d9d 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -33,11 +33,12 @@ To write a Python nbdkit plugin, you create a Python file which
contains at least the following required functions (in the top level
C<__main__> module):
+ API_VERSION = 2
def open(readonly):
# see below
def get_size(h):
# see below
- def pread(h, count, offset):
+ def pread(h, buf, offset, flags):
# see below
Note that the subroutines must have those literal names (like C<open>),
@@ -82,6 +83,18 @@ I<--dump-plugin> option, eg:
python_version=3.7.0
python_pep_384_abi_version=3
+=head2 API versions
+
+The nbdkit API has evolved and new versions are released periodically.
+To ensure backwards compatibility plugins have to opt in to the new
+version. From Python you do this by declaring a constant in your
+module:
+
+ API_VERSION = 2
+
+(where 2 is the latest version at the time this documentation was
+written). All newly written Python modules must have this constant.
+
=head2 Executable script
If you want you can make the script executable and include a "shebang"
@@ -199,16 +212,12 @@ contents will be garbage collected.
(Required)
- def pread(h, count, offset):
- # construct a buffer of length count bytes and return it
+ def pread(h, buf, offset, flags):
+ # read into the buffer
-The body of your C<pread> function should construct a buffer of length
-(at least) C<count> bytes. You should read C<count> bytes from the
-disk starting at C<offset>.
-
-The returned buffer can be any type compatible with the Python 3
-buffer protocol, such as bytearray, bytes or memoryview
-(L<https://docs.python.org/3/c-api/buffer.html>)
+The body of your C<pread> function should read exactly C<len(buf)>
+bytes of data starting at disk C<offset> and write it into the buffer
+C<buf>. C<flags> is always 0.
NBD only supports whole reads, so your function should try to read
the whole region (perhaps requiring a loop). If the read fails or
@@ -219,13 +228,13 @@ C<nbdkit.set_error> first.
(Optional)
- def pwrite(h, buf, offset):
+ def pwrite(h, buf, offset, flags):
length = len (buf)
# no return value
The body of your C<pwrite> function should write the buffer C<buf> to
the disk. You should write C<count> bytes to the disk starting at
-C<offset>.
+C<offset>. C<flags> may contain C<nbdkit.FLAG_FUA>.
NBD only supports whole writes, so your function should try to
write the whole region (perhaps requiring a loop). If the write
@@ -236,11 +245,12 @@ fails or is partial, your function should throw an exception,
(Optional)
- def flush(h):
+ def flush(h, flags):
# no return value
The body of your C<flush> function should do a L<sync(2)> or
L<fdatasync(2)> or equivalent on the backing store.
+C<flags> is always 0.
If the flush fails, your function should throw an exception, optionally
using C<nbdkit.set_error> first.
@@ -249,32 +259,35 @@ using C<nbdkit.set_error> first.
(Optional)
- def trim(h, count, offset):
+ def trim(h, count, offset, flags):
# no return value
-The body of your C<trim> function should "punch a hole" in the
-backing store. If the trim fails, your function should throw an
-exception, optionally using C<nbdkit.set_error> first.
+The body of your C<trim> function should "punch a hole" in the backing
+store. C<flags> may contain C<nbdkit.FLAG_FUA>. If the trim fails,
+your function should throw an exception, optionally using
+C<nbdkit.set_error> first.
=item C<zero>
(Optional)
- def zero(h, count, offset, may_trim):
+ def zero(h, count, offset, flags):
# no return value
-The body of your C<zero> function should ensure that C<count> bytes
-of the disk, starting at C<offset>, will read back as zero. If
-C<may_trim> is true, the operation may be optimized as a trim as long
-as subsequent reads see zeroes.
+The body of your C<zero> function should ensure that C<count> bytes of
+the disk, starting at C<offset>, will read back as zero. C<flags> is
+a bitmask which may include C<nbdkit.FLAG_MAY_TRIM>,
+C<nbdkit.FLAG_FUA>, C<nbdkit.FLAG_FAST_ZERO>.
NBD only supports whole writes, so your function should try to
-write the whole region (perhaps requiring a loop). If the write
-fails or is partial, your function should throw an exception,
-optionally using C<nbdkit.set_error> first. In particular, if
-you would like to automatically fall back to C<pwrite> (perhaps
-because there is nothing to optimize if C<may_trim> is false),
-use C<nbdkit.set_error(errno.EOPNOTSUPP)>.
+write the whole region (perhaps requiring a loop).
+
+If the write fails or is partial, your function should throw an
+exception, optionally using C<nbdkit.set_error> first. In particular,
+if you would like to automatically fall back to C<pwrite> (perhaps
+because there is nothing to optimize if
+S<C<flags & nbdkit.FLAG_MAY_TRIM>> is false), use
+S<C<nbdkit.set_error (errno.EOPNOTSUPP)>>.
=back
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 47da083..0f28595 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -46,6 +46,8 @@
#define PY_SSIZE_T_CLEAN 1
#include <Python.h>
+#define NBDKIT_API_VERSION 2
+
#include <nbdkit-plugin.h>
#include "cleanup.h"
@@ -60,6 +62,7 @@
*/
static const char *script;
static PyObject *module;
+static int py_api_version = 1;
static int last_error;
@@ -285,9 +288,14 @@ py_dump_plugin (void)
PyObject *fn;
PyObject *r;
+ /* Python version and ABI. */
printf ("python_version=%s\n", PY_VERSION);
printf ("python_pep_384_abi_version=%d\n", PYTHON_ABI_VERSION);
+ /* Maximum nbdkit API version supported. */
+ printf ("nbdkit_python_maximum_api_version=%d\n", NBDKIT_API_VERSION);
+
+ /* If the script has a dump_plugin function, call it. */
if (script && callback_defined ("dump_plugin", &fn)) {
PyErr_Clear ();
@@ -297,6 +305,30 @@ py_dump_plugin (void)
}
}
+static int
+get_py_api_version (void)
+{
+ PyObject *obj;
+ long value;
+
+ obj = PyObject_GetAttrString (module, "API_VERSION");
+ if (obj == NULL)
+ return 1; /* Default to API version 1. */
+
+ value = PyLong_AsLong (obj);
+ Py_DECREF (obj);
+
+ if (value < 1 || value > NBDKIT_API_VERSION) {
+ nbdkit_error ("%s: API_VERSION requested unknown version: %ld. "
+ "This plugin supports API versions between 1 and %d.",
+ script, value, NBDKIT_API_VERSION);
+ return -1;
+ }
+
+ nbdkit_debug ("module requested API_VERSION %ld", value);
+ return (int) value;
+}
+
static int
py_config (const char *key, const char *value)
{
@@ -359,6 +391,11 @@ py_config (const char *key, const char *value)
"nbdkit requires these callbacks.", script);
return -1;
}
+
+ /* Get the API version. */
+ py_api_version = get_py_api_version ();
+ if (py_api_version == -1)
+ return -1;
}
else if (callback_defined ("config", &fn)) {
/* Other parameters are passed to the Python .config callback. */
@@ -469,8 +506,8 @@ py_get_size (void *handle)
}
static int
-py_pread (void *handle, void *buf,
- uint32_t count, uint64_t offset)
+py_pread (void *handle, void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -485,24 +522,40 @@ py_pread (void *handle, void *buf,
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "ONLI", obj,
+ PyMemoryView_FromMemory ((char *)buf, count, PyBUF_WRITE),
+ offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("pread") == -1)
return ret;
- if (PyObject_GetBuffer (r, &view, PyBUF_SIMPLE) == -1) {
- nbdkit_error ("%s: value returned from pread does not support the "
- "buffer protocol",
- script);
- goto out;
- }
+ if (py_api_version == 1) {
+ /* In API v1 the Python pread function had to return a buffer
+ * protocol compatible function. In API v2+ it writes directly to
+ * the C buffer so this code is not used.
+ */
+ if (PyObject_GetBuffer (r, &view, PyBUF_SIMPLE) == -1) {
+ nbdkit_error ("%s: value returned from pread does not support the "
+ "buffer protocol",
+ script);
+ goto out;
+ }
- if (view.len < count) {
- nbdkit_error ("%s: buffer returned from pread is too small", script);
- goto out;
- }
+ if (view.len < count) {
+ nbdkit_error ("%s: buffer returned from pread is too small", script);
+ goto out;
+ }
- memcpy (buf, view.buf, count);
+ memcpy (buf, view.buf, count);
+ }
ret = 0;
out:
@@ -515,8 +568,8 @@ out:
}
static int
-py_pwrite (void *handle, const void *buf,
- uint32_t count, uint64_t offset)
+py_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -525,9 +578,19 @@ py_pwrite (void *handle, const void *buf,
if (callback_defined ("pwrite", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "ONL", obj,
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "ONL", obj,
PyMemoryView_FromMemory ((char *)buf, count, PyBUF_READ),
offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "ONLI", obj,
+ PyMemoryView_FromMemory ((char *)buf, count, PyBUF_READ),
+ offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("pwrite") == -1)
return -1;
@@ -542,7 +605,7 @@ py_pwrite (void *handle, const void *buf,
}
static int
-py_flush (void *handle)
+py_flush (void *handle, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -551,7 +614,15 @@ py_flush (void *handle)
if (callback_defined ("flush", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "OI", obj, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("flush") == -1)
return -1;
@@ -566,7 +637,7 @@ py_flush (void *handle)
}
static int
-py_trim (void *handle, uint32_t count, uint64_t offset)
+py_trim (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -575,7 +646,15 @@ py_trim (void *handle, uint32_t count, uint64_t offset)
if (callback_defined ("trim", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("trim") == -1)
return -1;
@@ -590,7 +669,7 @@ py_trim (void *handle, uint32_t count, uint64_t offset)
}
static int
-py_zero (void *handle, uint32_t count, uint64_t offset, int may_trim)
+py_zero (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -600,9 +679,19 @@ py_zero (void *handle, uint32_t count, uint64_t offset, int may_trim)
PyErr_Clear ();
last_error = 0;
- r = PyObject_CallFunction (fn, "OiLO",
- obj, count, offset,
- may_trim ? Py_True : Py_False);
+ switch (py_api_version) {
+ case 1: {
+ int may_trim = flags & NBDKIT_FLAG_MAY_TRIM;
+ r = PyObject_CallFunction (fn, "OiLO",
+ obj, count, offset,
+ may_trim ? Py_True : Py_False);
+ break;
+ }
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (last_error == EOPNOTSUPP || last_error == ENOTSUP) {
/* When user requests this particular error, we want to
diff --git a/tests/python-exception.py b/tests/python-exception.py
index d0c79bb..ee4a3f3 100644
--- a/tests/python-exception.py
+++ b/tests/python-exception.py
@@ -62,5 +62,5 @@ def get_size(h):
return 0
-def pread(h, count, offset):
- return ""
+def pread(h, buf, offset):
+ buf[:] = bytearray(len(buf))
diff --git a/tests/shebang.py b/tests/shebang.py
index 6f33623..0634589 100755
--- a/tests/shebang.py
+++ b/tests/shebang.py
@@ -13,6 +13,7 @@ def get_size(h):
return len(disk)
-def pread(h, count, offset):
+def pread(h, buf, offset):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
diff --git a/tests/test.py b/tests/test.py
index 9a2e947..4db5662 100644
--- a/tests/test.py
+++ b/tests/test.py
@@ -3,6 +3,9 @@ import nbdkit
disk = bytearray(1024*1024)
+API_VERSION = 2
+
+
def config_complete():
print ("set_error = %r" % nbdkit.set_error)
@@ -32,25 +35,26 @@ def can_trim(h):
return True
-def pread(h, count, offset):
+def pread(h, buf, offset, flags):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
-def pwrite(h, buf, offset):
+def pwrite(h, buf, offset, flags):
global disk
end = offset + len(buf)
disk[offset:end] = buf
-def zero(h, count, offset, may_trim=False):
+def flush(h, flags):
+ pass
+
+
+def trim(h, count, offset, flags):
+ pass
+
+
+def zero(h, count, offset, flags):
global disk
disk[offset:offset+count] = bytearray(count)
-
-
-def flush(h):
- pass
-
-
-def trim(h, count, offset):
- pass
--
2.18.2

View File

@ -0,0 +1,141 @@
From 9e20e2696fdb68008c9b4f1c36298f813320e381 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 23 Oct 2021 16:16:39 +0100
Subject: [PATCH] vddk: Include VDDK major library version in --dump-plugin
output
Although it doesn't seem to be possible to get the precise VDDK
version, With a relatively simple change we can at least return the
VDDK major version. Currently this can be 5, 6 or 7.
(cherry picked from commit 8700649d147948897f3b97810a1dff37924bdd6e)
---
plugins/vddk/nbdkit-vddk-plugin.pod | 4 ++++
plugins/vddk/vddk.c | 29 +++++++++++++++++++----------
tests/test-vddk-real-dump-plugin.sh | 2 ++
3 files changed, 25 insertions(+), 10 deletions(-)
diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod b/plugins/vddk/nbdkit-vddk-plugin.pod
index 8b14eda0..822b96be 100644
--- a/plugins/vddk/nbdkit-vddk-plugin.pod
+++ b/plugins/vddk/nbdkit-vddk-plugin.pod
@@ -417,6 +417,10 @@ at runtime.
If this is printed then the C<nfchostport=PORT> parameter is supported
by this build.
+=item C<vddk_library_version=...>
+
+The VDDK major library version: 5, 6, 7, ...
+
=item C<vddk_dll=...>
Prints the full path to the VDDK shared library. Since this requires
diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c
index 69193504..291283f4 100644
--- a/plugins/vddk/vddk.c
+++ b/plugins/vddk/vddk.c
@@ -77,6 +77,7 @@ int vddk_debug_datapath = 1;
static void *dl; /* dlopen handle */
static bool init_called; /* was InitEx called */
static __thread int error_suppression; /* threadlocal error suppression */
+static int library_version; /* VDDK major: 5, 6, 7, ... */
static enum { NONE = 0, ZLIB, FASTLZ, SKIPZ } compression; /* compression */
static char *config; /* config */
@@ -297,7 +298,10 @@ vddk_config (const char *key, const char *value)
static void
load_library (bool load_error_is_fatal)
{
- static const char *sonames[] = {
+ static struct {
+ const char *soname;
+ int library_version;
+ } libs[] = {
/* Prefer the newest library in case multiple exist. Check two
* possible directories: the usual VDDK installation puts .so
* files in an arch-specific subdirectory of $libdir (our minimum
@@ -305,12 +309,13 @@ load_library (bool load_error_is_fatal)
* but our testsuite is easier to write if we point libdir
* directly to a stub .so.
*/
- "lib64/libvixDiskLib.so.7",
- "libvixDiskLib.so.7",
- "lib64/libvixDiskLib.so.6",
- "libvixDiskLib.so.6",
- "lib64/libvixDiskLib.so.5",
- "libvixDiskLib.so.5",
+ { "lib64/libvixDiskLib.so.7", 7 },
+ { "libvixDiskLib.so.7", 7 },
+ { "lib64/libvixDiskLib.so.6", 6 },
+ { "libvixDiskLib.so.6", 6 },
+ { "lib64/libvixDiskLib.so.5", 5 },
+ { "libvixDiskLib.so.5", 5 },
+ { NULL }
};
size_t i;
CLEANUP_FREE char *orig_error = NULL;
@@ -323,19 +328,20 @@ load_library (bool load_error_is_fatal)
}
}
- for (i = 0; i < sizeof sonames / sizeof sonames[0]; ++i) {
+ for (i = 0; libs[i].soname != NULL; ++i) {
CLEANUP_FREE char *path;
/* Set the full path so that dlopen will preferentially load the
* system libraries from the same directory.
*/
- if (asprintf (&path, "%s/%s", libdir, sonames[i]) == -1) {
+ if (asprintf (&path, "%s/%s", libdir, libs[i].soname) == -1) {
nbdkit_error ("asprintf: %m");
exit (EXIT_FAILURE);
}
dl = dlopen (path, RTLD_NOW);
if (dl != NULL) {
+ library_version = libs[i].library_version;
/* Now that we found the library, ensure that LD_LIBRARY_PATH
* includes its directory for all future loads. This may modify
* path in-place and/or re-exec nbdkit, but that's okay.
@@ -356,10 +362,12 @@ load_library (bool load_error_is_fatal)
"If '%s' is located on a non-standard path you may need to\n"
"set libdir=/path/to/vmware-vix-disklib-distrib.\n\n"
"See nbdkit-vddk-plugin(1) man page section \"LIBRARY LOCATION\" for details.",
- orig_error ? : "(unknown error)", sonames[0]);
+ orig_error ? : "(unknown error)", libs[0].soname);
exit (EXIT_FAILURE);
}
+ assert (library_version >= 5);
+
/* Load symbols. */
#define STUB(fn,ret,args) \
do { \
@@ -474,6 +482,7 @@ vddk_dump_plugin (void)
printf ("vddk_default_libdir=%s\n", VDDK_LIBDIR);
printf ("vddk_has_nfchostport=1\n");
+ printf ("vddk_library_version=%d\n", library_version);
#if defined(HAVE_DLADDR)
/* It would be nice to print the version of VDDK from the shared
diff --git a/tests/test-vddk-real-dump-plugin.sh b/tests/test-vddk-real-dump-plugin.sh
index 1479e416..59c79693 100755
--- a/tests/test-vddk-real-dump-plugin.sh
+++ b/tests/test-vddk-real-dump-plugin.sh
@@ -51,10 +51,12 @@ rm -f $files
cleanup_fn rm -f $files
nbdkit -f -v vddk libdir="$vddkdir" --dump-plugin > $out
+cat $out
# Check the vddk_* entries are set.
grep ^vddk_default_libdir= $out
grep ^vddk_has_nfchostport= $out
+grep ^vddk_library_version= $out
grep ^vddk_dll= $out
dll="$(grep ^vddk_dll $out | cut -d= -f2)"
--
2.31.1

View File

@ -1,98 +0,0 @@
From c5b1fac4c67078f0164bd23eab6d4d2b8c9830b0 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:42:02 +0000
Subject: [PATCH] python: Implement cache.
However this does not implement can_cache, since that is not a simple
boolean.
(cherry picked from commit e61ffb73c7a0af0c383184fdb8f08d30784a195e)
---
plugins/python/nbdkit-python-plugin.pod | 14 ++++++++++-
plugins/python/python.c | 31 +++++++++++++++++++++++++
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 4923d9d..0ea8dee 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -289,6 +289,19 @@ because there is nothing to optimize if
S<C<flags & nbdkit.FLAG_MAY_TRIM>> is false), use
S<C<nbdkit.set_error (errno.EOPNOTSUPP)>>.
+=item C<cache>
+
+(Optional)
+
+ def cache(h, count, offset, flags):
+ # no return value
+
+The body of your C<cache> function should prefetch data in the
+indicated range.
+
+If the cache operation fails, your function should throw an exception,
+optionally using C<nbdkit.set_error> first.
+
=back
=head2 Missing callbacks
@@ -317,7 +330,6 @@ C<can_zero>,
C<can_fast_zero>,
C<can_extents>,
C<can_multi_conn>,
-C<cache>,
C<extents>.
These are not yet supported.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 0f28595..c5cf38e 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -714,6 +714,36 @@ py_zero (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
return -1;
}
+static int
+py_cache (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+
+ if (callback_defined ("cache", &fn)) {
+ PyErr_Clear ();
+
+ switch (py_api_version) {
+ case 1:
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags, NULL);
+ break;
+ default: abort ();
+ }
+ Py_DECREF (fn);
+ if (check_python_failure ("cache") == -1)
+ return -1;
+ Py_DECREF (r);
+ }
+ else {
+ nbdkit_error ("%s not implemented", "cache");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
boolean_callback (void *handle, const char *can_fn, const char *plain_fn)
{
@@ -799,6 +829,7 @@ static struct nbdkit_plugin plugin = {
.flush = py_flush,
.trim = py_trim,
.zero = py_zero,
+ .cache = py_cache,
};
NBDKIT_REGISTER_PLUGIN (plugin)
--
2.18.2

View File

@ -0,0 +1,55 @@
From b8b376cf39d97c9f523a9867612126088b43c523 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 23 Oct 2021 19:50:52 +0100
Subject: [PATCH] vddk: Only print vddk_library_version when we managed to load
the library
Because --dump-plugin calls load_library (false) it won't fail if we
didn't manage to load the library. This results in library_version
being 0, which we printed incorrectly.
Resolve this problem by not printing the vddk_library_version entry in
this case.
Fixes: commit 8700649d147948897f3b97810a1dff37924bdd6e
(cherry picked from commit a3fba12c3e9c2113009f556360ae0bd04c45f6bb)
---
plugins/vddk/nbdkit-vddk-plugin.pod | 1 +
plugins/vddk/vddk.c | 9 ++++++++-
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod b/plugins/vddk/nbdkit-vddk-plugin.pod
index 822b96be..c56faddc 100644
--- a/plugins/vddk/nbdkit-vddk-plugin.pod
+++ b/plugins/vddk/nbdkit-vddk-plugin.pod
@@ -420,6 +420,7 @@ by this build.
=item C<vddk_library_version=...>
The VDDK major library version: 5, 6, 7, ...
+If this is omitted it means the library could not be loaded.
=item C<vddk_dll=...>
diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c
index 291283f4..96615749 100644
--- a/plugins/vddk/vddk.c
+++ b/plugins/vddk/vddk.c
@@ -482,7 +482,14 @@ vddk_dump_plugin (void)
printf ("vddk_default_libdir=%s\n", VDDK_LIBDIR);
printf ("vddk_has_nfchostport=1\n");
- printf ("vddk_library_version=%d\n", library_version);
+
+ /* Because load_library (false) we might not have loaded VDDK, in
+ * which case we didn't set library_version. Note this cannot
+ * happen in the normal (non-debug-plugin) path because there we use
+ * load_library (true).
+ */
+ if (library_version > 0)
+ printf ("vddk_library_version=%d\n", library_version);
#if defined(HAVE_DLADDR)
/* It would be nice to print the version of VDDK from the shared
--
2.31.1

View File

@ -1,80 +0,0 @@
From 17721b316dd66b0a1ed792eeccd2489fb97828df Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:42:59 +0000
Subject: [PATCH] python: Implement can_zero, can_fast_zero.
(cherry picked from commit 039f600d2ad7a9ff04523a165eb2fe41b9c87c01)
---
plugins/python/nbdkit-python-plugin.pod | 16 ++++++++++++++--
plugins/python/python.c | 14 ++++++++++++++
2 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 0ea8dee..1f1c30f 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -208,6 +208,20 @@ contents will be garbage collected.
def can_trim(h):
# return a boolean
+=item C<can_zero>
+
+(Optional)
+
+ def can_zero(h):
+ # return a boolean
+
+=item C<can_fast_zero>
+
+(Optional)
+
+ def can_fast_zero(h):
+ # return a boolean
+
=item C<pread>
(Required)
@@ -326,8 +340,6 @@ C<config_help>,
C<magic_config_key>,
C<can_fua>,
C<can_cache>,
-C<can_zero>,
-C<can_fast_zero>,
C<can_extents>,
C<can_multi_conn>,
C<extents>.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index c5cf38e..38fc119 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -797,6 +797,18 @@ py_can_trim (void *handle)
return boolean_callback (handle, "can_trim", "trim");
}
+static int
+py_can_zero (void *handle)
+{
+ return boolean_callback (handle, "can_zero", "zero");
+}
+
+static int
+py_can_fast_zero (void *handle)
+{
+ return boolean_callback (handle, "can_fast_zero", NULL);
+}
+
#define py_config_help \
"script=<FILENAME> (required) The Python plugin to run.\n" \
"[other arguments may be used by the plugin that you load]"
@@ -823,6 +835,8 @@ static struct nbdkit_plugin plugin = {
.can_write = py_can_write,
.can_flush = py_can_flush,
.can_trim = py_can_trim,
+ .can_zero = py_can_zero,
+ .can_fast_zero = py_can_fast_zero,
.pread = py_pread,
.pwrite = py_pwrite,
--
2.18.2

View File

@ -0,0 +1,53 @@
From e850f65053d89ad54c27280f48506da5eb631a68 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Fri, 18 Nov 2022 09:43:19 +0000
Subject: [PATCH] vddk: Add support for VDDK 8.0.0
There are no changes in any of the structures or enums that we rely on.
Reported-by: Ming Xie
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2143889
(cherry picked from commit dbe12ed499baeea94d603db55cad9e971e0ebcf0)
---
plugins/vddk/nbdkit-vddk-plugin.pod | 2 +-
plugins/vddk/vddk.c | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/plugins/vddk/nbdkit-vddk-plugin.pod b/plugins/vddk/nbdkit-vddk-plugin.pod
index c56faddc..c94c41eb 100644
--- a/plugins/vddk/nbdkit-vddk-plugin.pod
+++ b/plugins/vddk/nbdkit-vddk-plugin.pod
@@ -419,7 +419,7 @@ by this build.
=item C<vddk_library_version=...>
-The VDDK major library version: 5, 6, 7, ...
+The VDDK major library version: 5, 6, 7, 8, ...
If this is omitted it means the library could not be loaded.
=item C<vddk_dll=...>
diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c
index 96615749..2140789a 100644
--- a/plugins/vddk/vddk.c
+++ b/plugins/vddk/vddk.c
@@ -77,7 +77,7 @@ int vddk_debug_datapath = 1;
static void *dl; /* dlopen handle */
static bool init_called; /* was InitEx called */
static __thread int error_suppression; /* threadlocal error suppression */
-static int library_version; /* VDDK major: 5, 6, 7, ... */
+static int library_version; /* VDDK major: 5, 6, 7, 8, ... */
static enum { NONE = 0, ZLIB, FASTLZ, SKIPZ } compression; /* compression */
static char *config; /* config */
@@ -309,6 +309,8 @@ load_library (bool load_error_is_fatal)
* but our testsuite is easier to write if we point libdir
* directly to a stub .so.
*/
+ { "lib64/libvixDiskLib.so.8", 8 },
+ { "libvixDiskLib.so.8", 8 },
{ "lib64/libvixDiskLib.so.7", 7 },
{ "libvixDiskLib.so.7", 7 },
{ "lib64/libvixDiskLib.so.6", 6 },
--
2.31.1

View File

@ -1,65 +0,0 @@
From 2a85ce81ad95eb2f9b2f29666480b814ea0f80d9 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:46:11 +0000
Subject: [PATCH] python: Implement can_multi_conn.
(cherry picked from commit 21dd7bf49d3238c7e75918d4bf324b617f458d83)
---
plugins/python/nbdkit-python-plugin.pod | 8 +++++++-
plugins/python/python.c | 7 +++++++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 1f1c30f..b92bb56 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -187,6 +187,13 @@ contents will be garbage collected.
def is_rotational(h):
# return a boolean
+=item C<can_multi_conn>
+
+(Optional)
+
+ def can_multi_conn(h):
+ # return a boolean
+
=item C<can_write>
(Optional)
@@ -341,7 +348,6 @@ C<magic_config_key>,
C<can_fua>,
C<can_cache>,
C<can_extents>,
-C<can_multi_conn>,
C<extents>.
These are not yet supported.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 38fc119..b186b99 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -779,6 +779,12 @@ py_is_rotational (void *handle)
return boolean_callback (handle, "is_rotational", NULL);
}
+static int
+py_can_multi_conn (void *handle)
+{
+ return boolean_callback (handle, "can_multi_conn", NULL);
+}
+
static int
py_can_write (void *handle)
{
@@ -832,6 +838,7 @@ static struct nbdkit_plugin plugin = {
.get_size = py_get_size,
.is_rotational = py_is_rotational,
+ .can_multi_conn = py_can_multi_conn,
.can_write = py_can_write,
.can_flush = py_can_flush,
.can_trim = py_can_trim,
--
2.18.2

View File

@ -1,126 +0,0 @@
From 38124a137974e1433d68732640ca7f88664557da Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Fri, 22 Nov 2019 19:25:53 +0000
Subject: [PATCH] python: Implement can_fua and can_cache.
(cherry picked from commit 97c46f885edec5a61a96ac86eccb9d8c874c602e)
---
plugins/python/nbdkit-python-plugin.pod | 18 +++++++-
plugins/python/python.c | 58 +++++++++++++++++++++++++
2 files changed, 74 insertions(+), 2 deletions(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index b92bb56..4065ec7 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -229,6 +229,22 @@ contents will be garbage collected.
def can_fast_zero(h):
# return a boolean
+=item C<can_fua>
+
+(Optional)
+
+ def can_fua(h):
+ # return nbdkit.FUA_NONE or nbdkit.FUA_EMULATE
+ # or nbdkit.FUA_NATIVE
+
+=item C<can_cache>
+
+(Optional)
+
+ def can_cache(h):
+ # return nbdkit.CACHE_NONE or nbdkit.CACHE_EMULATE
+ # or nbdkit.CACHE_NATIVE
+
=item C<pread>
(Required)
@@ -345,8 +361,6 @@ C<longname>,
C<description>,
C<config_help>,
C<magic_config_key>,
-C<can_fua>,
-C<can_cache>,
C<can_extents>,
C<extents>.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index b186b99..5e2e526 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -815,6 +815,62 @@ py_can_fast_zero (void *handle)
return boolean_callback (handle, "can_fast_zero", NULL);
}
+static int
+py_can_fua (void *handle)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+ int ret;
+
+ if (callback_defined ("can_fua", &fn)) {
+ PyErr_Clear ();
+
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ Py_DECREF (fn);
+ if (check_python_failure ("can_fua") == -1)
+ return -1;
+ ret = PyLong_AsLong (r);
+ Py_DECREF (r);
+ return ret;
+ }
+ /* No Python can_fua, but check if there's a Python flush
+ * callback defined. (In C modules, nbdkit would do this).
+ */
+ else if (callback_defined ("flush", NULL))
+ return NBDKIT_FUA_EMULATE;
+ else
+ return NBDKIT_FUA_NONE;
+}
+
+static int
+py_can_cache (void *handle)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+ int ret;
+
+ if (callback_defined ("can_cache", &fn)) {
+ PyErr_Clear ();
+
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ Py_DECREF (fn);
+ if (check_python_failure ("can_cache") == -1)
+ return -1;
+ ret = PyLong_AsLong (r);
+ Py_DECREF (r);
+ return ret;
+ }
+ /* No Python can_cache, but check if there's a Python cache
+ * callback defined. (In C modules, nbdkit would do this).
+ */
+ else if (callback_defined ("cache", NULL))
+ return NBDKIT_CACHE_NATIVE;
+ else
+ return NBDKIT_CACHE_NONE;
+}
+
#define py_config_help \
"script=<FILENAME> (required) The Python plugin to run.\n" \
"[other arguments may be used by the plugin that you load]"
@@ -844,6 +900,8 @@ static struct nbdkit_plugin plugin = {
.can_trim = py_can_trim,
.can_zero = py_can_zero,
.can_fast_zero = py_can_fast_zero,
+ .can_fua = py_can_fua,
+ .can_cache = py_can_cache,
.pread = py_pread,
.pwrite = py_pwrite,
--
2.18.2

View File

@ -1,597 +0,0 @@
From 7cb79aef2a12f29f1286caf3858001e47214f871 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 20:54:41 +0000
Subject: [PATCH] tests: Test the Python plugin thoroughly.
This tests the Python plugin thoroughly by issuing client commands
through libnbd and checking we get the expected results.
(cherry picked from commit 8ead4a82ec3227dbecb6cbfc419f1a18f2817d62)
---
.gitignore | 1 +
README | 2 +
tests/Makefile.am | 15 +--
tests/test-lang-plugins.c | 3 +-
tests/test-python-plugin.py | 133 +++++++++++++++++++++
tests/test-python.sh | 49 ++++++++
tests/test.py | 60 ----------
tests/test_python.py | 222 ++++++++++++++++++++++++++++++++++++
8 files changed, 413 insertions(+), 72 deletions(-)
create mode 100644 tests/test-python-plugin.py
create mode 100755 tests/test-python.sh
delete mode 100644 tests/test.py
create mode 100755 tests/test_python.py
diff --git a/.gitignore b/.gitignore
index b25ac7f..e25bd99 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,7 @@ Makefile.in
/server/synopsis.c
/server/test-public
/stamp-h1
+/tests/__pycache__/
/tests/disk
/tests/disk.gz
/tests/disk.xz
diff --git a/README b/README
index 40f4cd3..05f1e06 100644
--- a/README
+++ b/README
@@ -130,6 +130,8 @@ For the Python plugin:
- python development libraries
+ - python unittest to run the test suite
+
For the OCaml plugin:
- OCaml >= 4.02.2
diff --git a/tests/Makefile.am b/tests/Makefile.am
index d225cc6..09103fb 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -67,6 +67,7 @@ EXTRA_PROGRAMS =
TESTS_ENVIRONMENT = \
PATH=$(abs_top_builddir):$(PATH) \
SRCDIR=$(srcdir) \
+ PYTHON=$(PYTHON) \
LIBGUESTFS_ATTACH_METHOD=appliance \
LIBGUESTFS_DEBUG=1 \
LIBGUESTFS_TRACE=1 \
@@ -160,7 +161,9 @@ EXTRA_DIST = \
test-probe-plugin.sh \
test-python-exception.sh \
test.pl \
- test.py \
+ test_python.py \
+ test-python-plugin.py \
+ test-python.sh \
test-rate.sh \
test-rate-dynamic.sh \
test.rb \
@@ -801,18 +804,10 @@ endif HAVE_PERL
if HAVE_PYTHON
TESTS += \
+ test-python.sh \
test-python-exception.sh \
test-shebang-python.sh \
$(NULL)
-LIBGUESTFS_TESTS += test-python
-
-test_python_SOURCES = test-lang-plugins.c test.h
-test_python_CFLAGS = \
- -DLANG='"python"' -DSCRIPT='"$(srcdir)/test.py"' \
- $(WARNINGS_CFLAGS) \
- $(LIBGUESTFS_CFLAGS) \
- $(NULL)
-test_python_LDADD = libtest.la $(LIBGUESTFS_LIBS)
endif HAVE_PYTHON
diff --git a/tests/test-lang-plugins.c b/tests/test-lang-plugins.c
index ffb1918..93f9938 100644
--- a/tests/test-lang-plugins.c
+++ b/tests/test-lang-plugins.c
@@ -56,8 +56,7 @@ main (int argc, char *argv[])
*/
s = getenv ("NBDKIT_VALGRIND");
if (s && strcmp (s, "1") == 0 &&
- (strcmp (LANG, "python") == 0 ||
- strcmp (LANG, "ruby") == 0 ||
+ (strcmp (LANG, "ruby") == 0 ||
strcmp (LANG, "tcl") == 0)) {
fprintf (stderr, "%s test skipped under valgrind.\n", LANG);
exit (77); /* Tells automake to skip the test. */
diff --git a/tests/test-python-plugin.py b/tests/test-python-plugin.py
new file mode 100644
index 0000000..8e90bc2
--- /dev/null
+++ b/tests/test-python-plugin.py
@@ -0,0 +1,133 @@
+# nbdkit test plugin
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+"""See test-python.py."""
+
+import nbdkit
+import sys
+import pickle
+import base64
+
+API_VERSION = 2
+
+cfg = {}
+
+def config (k, v):
+ global cfg
+ if k == "cfg":
+ cfg = pickle.loads (base64.b64decode (v.encode()))
+
+def config_complete ():
+ print ("set_error = %r" % nbdkit.set_error)
+
+def open (readonly):
+ return {
+ 'disk': bytearray (cfg.get ('size', 0))
+ }
+
+def get_size (h):
+ return len (h['disk'])
+
+def is_rotational (h):
+ return cfg.get ('is_rotational', False)
+
+def can_multi_conn (h):
+ return cfg.get ('can_multi_conn', False)
+
+def can_write (h):
+ return cfg.get ('can_write', True)
+
+def can_flush (h):
+ return cfg.get ('can_flush', False)
+
+def can_trim (h):
+ return cfg.get ('can_trim', False)
+
+def can_zero (h):
+ return cfg.get ('can_zero', False)
+
+def can_fast_zero (h):
+ return cfg.get ('can_fast_zero', False)
+
+def can_fua (h):
+ fua = cfg.get ('can_fua', "none")
+ if fua == "none":
+ return nbdkit.FUA_NONE
+ elif fua == "emulate":
+ return nbdkit.FUA_EMULATE
+ elif fua == "native":
+ return nbdkit.FUA_NATIVE
+
+def can_cache (h):
+ cache = cfg.get ('can_cache', "none")
+ if cache == "none":
+ return nbdkit.CACHE_NONE
+ elif cache == "emulate":
+ return nbdkit.CACHE_EMULATE
+ elif cache == "native":
+ return nbdkit.CACHE_NATIVE
+
+def pread (h, buf, offset, flags):
+ assert flags == 0
+ end = offset + len(buf)
+ buf[:] = h['disk'][offset:end]
+
+def pwrite (h, buf, offset, flags):
+ expect_fua = cfg.get ('pwrite_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ end = offset + len(buf)
+ h['disk'][offset:end] = buf
+
+def flush (h, flags):
+ assert flags == 0
+
+def trim (h, count, offset, flags):
+ expect_fua = cfg.get ('trim_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ h['disk'][offset:offset+count] = bytearray(count)
+
+def zero (h, count, offset, flags):
+ expect_fua = cfg.get ('zero_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ expect_may_trim = cfg.get ('zero_expect_may_trim', False)
+ actual_may_trim = bool (flags & nbdkit.FLAG_MAY_TRIM)
+ assert expect_may_trim == actual_may_trim
+ expect_fast_zero = cfg.get ('zero_expect_fast_zero', False)
+ actual_fast_zero = bool (flags & nbdkit.FLAG_FAST_ZERO)
+ assert expect_fast_zero == actual_fast_zero
+ h['disk'][offset:offset+count] = bytearray(count)
+
+def cache (h, count, offset, flags):
+ assert flags == 0
+ # do nothing
diff --git a/tests/test-python.sh b/tests/test-python.sh
new file mode 100755
index 0000000..50324d0
--- /dev/null
+++ b/tests/test-python.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+source ./functions.sh
+set -e
+set -x
+
+requires $PYTHON --version
+requires $PYTHON -c 'import unittest'
+requires $PYTHON -c 'import nbd'
+requires test -f test_python.py
+requires test -f test-python-plugin.py
+
+# Python has proven very difficult to valgrind, therefore it is disabled.
+if [ "$NBDKIT_VALGRIND" = "1" ]; then
+ echo "$0: skipping Python test under valgrind."
+ exit 77
+fi
+
+$PYTHON -m unittest test_python
diff --git a/tests/test.py b/tests/test.py
deleted file mode 100644
index 4db5662..0000000
--- a/tests/test.py
+++ /dev/null
@@ -1,60 +0,0 @@
-import nbdkit
-
-disk = bytearray(1024*1024)
-
-
-API_VERSION = 2
-
-
-def config_complete():
- print ("set_error = %r" % nbdkit.set_error)
-
-
-def open(readonly):
- return 1
-
-
-def get_size(h):
- global disk
- return len(disk)
-
-
-def can_write(h):
- return True
-
-
-def can_flush(h):
- return True
-
-
-def is_rotational(h):
- return False
-
-
-def can_trim(h):
- return True
-
-
-def pread(h, buf, offset, flags):
- global disk
- end = offset + len(buf)
- buf[:] = disk[offset:end]
-
-
-def pwrite(h, buf, offset, flags):
- global disk
- end = offset + len(buf)
- disk[offset:end] = buf
-
-
-def flush(h, flags):
- pass
-
-
-def trim(h, count, offset, flags):
- pass
-
-
-def zero(h, count, offset, flags):
- global disk
- disk[offset:offset+count] = bytearray(count)
diff --git a/tests/test_python.py b/tests/test_python.py
new file mode 100755
index 0000000..6b9f297
--- /dev/null
+++ b/tests/test_python.py
@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+"""
+This tests the Python plugin thoroughly by issuing client commands
+through libnbd and checking we get the expected results. It uses an
+associated plugin (test-python-plugin.sh).
+"""
+
+import os
+import sys
+import nbd
+import unittest
+import pickle
+import base64
+
+class Test (unittest.TestCase):
+ def setUp (self):
+ self.h = nbd.NBD ()
+
+ def tearDown (self):
+ del self.h
+
+ def connect (self, cfg):
+ cfg = base64.b64encode (pickle.dumps (cfg)).decode()
+ cmd = ["nbdkit", "-v", "-s", "--exit-with-parent",
+ "python", "test-python-plugin.py", "cfg=" + cfg]
+ self.h.connect_command (cmd)
+
+ def test_none (self):
+ """
+ Test we can send an empty pickled test configuration and do
+ nothing else. This is just to ensure the machinery of the
+ test works.
+ """
+ self.connect ({})
+
+ def test_size_512 (self):
+ """Test the size."""
+ self.connect ({"size": 512})
+ assert self.h.get_size() == 512
+
+ def test_size_1m (self):
+ """Test the size."""
+ self.connect ({"size": 1024*1024})
+ assert self.h.get_size() == 1024*1024
+
+ # Test each flag call.
+ def test_is_rotational_true (self):
+ self.connect ({"size": 512, "is_rotational": True})
+ assert self.h.is_rotational()
+
+ def test_is_rotational_false (self):
+ self.connect ({"size": 512, "is_rotational": False})
+ assert not self.h.is_rotational()
+
+ def test_can_multi_conn_true (self):
+ self.connect ({"size": 512, "can_multi_conn": True})
+ assert self.h.can_multi_conn()
+
+ def test_can_multi_conn_false (self):
+ self.connect ({"size": 512, "can_multi_conn": False})
+ assert not self.h.can_multi_conn()
+
+ def test_read_write (self):
+ self.connect ({"size": 512, "can_write": True})
+ assert not self.h.is_read_only()
+
+ def test_read_only (self):
+ self.connect ({"size": 512, "can_write": False})
+ assert self.h.is_read_only()
+
+ def test_can_flush_true (self):
+ self.connect ({"size": 512, "can_flush": True})
+ assert self.h.can_flush()
+
+ def test_can_flush_false (self):
+ self.connect ({"size": 512, "can_flush": False})
+ assert not self.h.can_flush()
+
+ def test_can_trim_true (self):
+ self.connect ({"size": 512, "can_trim": True})
+ assert self.h.can_trim()
+
+ def test_can_trim_false (self):
+ self.connect ({"size": 512, "can_trim": False})
+ assert not self.h.can_trim()
+
+ # nbdkit can always zero because it emulates it.
+ #self.connect ({"size": 512, "can_zero": True})
+ #assert self.h.can_zero()
+ #self.connect ({"size": 512, "can_zero": False})
+ #assert not self.h.can_zero()
+
+ def test_can_fast_zero_true (self):
+ self.connect ({"size": 512, "can_fast_zero": True})
+ assert self.h.can_fast_zero()
+
+ def test_can_fast_zero_false (self):
+ self.connect ({"size": 512, "can_fast_zero": False})
+ assert not self.h.can_fast_zero()
+
+ def test_can_fua_none (self):
+ self.connect ({"size": 512, "can_fua": "none"})
+ assert not self.h.can_fua()
+
+ def test_can_fua_emulate (self):
+ self.connect ({"size": 512, "can_fua": "emulate"})
+ assert self.h.can_fua()
+
+ def test_can_fua_native (self):
+ self.connect ({"size": 512, "can_fua": "native"})
+ assert self.h.can_fua()
+
+ def test_can_cache_none (self):
+ self.connect ({"size": 512, "can_cache": "none"})
+ assert not self.h.can_cache()
+
+ def test_can_cache_emulate (self):
+ self.connect ({"size": 512, "can_cache": "emulate"})
+ assert self.h.can_cache()
+
+ def test_can_cache_native (self):
+ self.connect ({"size": 512, "can_cache": "native"})
+ assert self.h.can_cache()
+
+ # Not yet implemented: can_extents.
+
+ def test_pread (self):
+ """Test pread."""
+ self.connect ({"size": 512})
+ buf = self.h.pread (512, 0)
+ assert buf == bytearray (512)
+
+ # Test pwrite + flags.
+ def test_pwrite (self):
+ self.connect ({"size": 512})
+ buf = bytearray (512)
+ self.h.pwrite (buf, 0)
+
+ def test_pwrite_fua (self):
+ self.connect ({"size": 512,
+ "can_fua": "native",
+ "pwrite_expect_fua": True})
+ buf = bytearray (512)
+ self.h.pwrite (buf, 0, nbd.CMD_FLAG_FUA)
+
+ def test_flush (self):
+ """Test flush."""
+ self.connect ({"size": 512, "can_flush": True})
+ self.h.flush ()
+
+ # Test trim + flags.
+ def test_trim (self):
+ self.connect ({"size": 512, "can_trim": True})
+ self.h.trim (512, 0)
+
+ def test_trim_fua (self):
+ self.connect ({"size": 512,
+ "can_trim": True,
+ "can_fua": "native",
+ "trim_expect_fua": True})
+ self.h.trim (512, 0, nbd.CMD_FLAG_FUA)
+
+ # Test zero + flags.
+ def test_zero (self):
+ self.connect ({"size": 512, "can_zero": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE)
+
+ def test_zero_fua (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "can_fua": "native",
+ "zero_expect_fua": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE | nbd.CMD_FLAG_FUA)
+
+ def test_zero_may_trim (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "zero_expect_may_trim": True})
+ self.h.zero (512, 0, 0) # absence of nbd.CMD_FLAG_NO_HOLE
+
+ def test_zero_fast_zero (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "can_fast_zero": True,
+ "zero_expect_fast_zero": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE | nbd.CMD_FLAG_FAST_ZERO)
+
+ def test_cache (self):
+ """Test cache."""
+ self.connect ({"size": 512, "can_cache": "native"})
+ self.h.cache (512, 0)
--
2.18.2

View File

@ -1,790 +0,0 @@
From e744dcb38cc52cbe64977efcdd4bc60e802d1b17 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 23 Jan 2020 19:52:00 +0000
Subject: [PATCH] New filter: extentlist.
Allows a list of extents to be placed on top of an existing plugin.
(cherry picked from commit 3e770b6d6620a62546849a2863638041c0b00640)
---
TODO | 4 +
configure.ac | 2 +
.../nbdkit-cacheextents-filter.pod | 1 +
filters/extentlist/Makefile.am | 67 ++++
filters/extentlist/extentlist.c | 326 ++++++++++++++++++
.../extentlist/nbdkit-extentlist-filter.pod | 90 +++++
filters/noextents/nbdkit-noextents-filter.pod | 1 +
tests/Makefile.am | 4 +
tests/test-extentlist.sh | 175 ++++++++++
9 files changed, 670 insertions(+)
create mode 100644 filters/extentlist/Makefile.am
create mode 100644 filters/extentlist/extentlist.c
create mode 100644 filters/extentlist/nbdkit-extentlist-filter.pod
create mode 100755 tests/test-extentlist.sh
diff --git a/TODO b/TODO
index d2aca44..2a3e89d 100644
--- a/TODO
+++ b/TODO
@@ -187,6 +187,10 @@ Suggestions for filters
MBs of extra data)
https://github.com/facebook/zstd/issues/395#issuecomment-535875379
+* nbdkit-extentlist-filter could read the extents generated by
+ qemu-img map, allowing extents to be ported from a qemu block
+ device.
+
nbdkit-rate-filter:
* allow other kinds of traffic shaping such as VBR
diff --git a/configure.ac b/configure.ac
index fde498b..41e68de 100644
--- a/configure.ac
+++ b/configure.ac
@@ -896,6 +896,7 @@ filters="\
cow \
delay \
error \
+ extentlist \
fua \
log \
nocache \
@@ -979,6 +980,7 @@ AC_CONFIG_FILES([Makefile
filters/cow/Makefile
filters/delay/Makefile
filters/error/Makefile
+ filters/extentlist/Makefile
filters/fua/Makefile
filters/log/Makefile
filters/nocache/Makefile
diff --git a/filters/cacheextents/nbdkit-cacheextents-filter.pod b/filters/cacheextents/nbdkit-cacheextents-filter.pod
index fdd2285..bb2514a 100644
--- a/filters/cacheextents/nbdkit-cacheextents-filter.pod
+++ b/filters/cacheextents/nbdkit-cacheextents-filter.pod
@@ -52,6 +52,7 @@ C<nbdkit-cacheextents-filter> first appeared in nbdkit 1.14.
L<nbdkit(1)>,
L<nbdkit-cache-filter(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-vddk-plugin(1)>,
L<nbdkit-filter(3)>,
diff --git a/filters/extentlist/Makefile.am b/filters/extentlist/Makefile.am
new file mode 100644
index 0000000..88a9afe
--- /dev/null
+++ b/filters/extentlist/Makefile.am
@@ -0,0 +1,67 @@
+# nbdkit
+# Copyright (C) 2019-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = nbdkit-extentlist-filter.pod
+
+filter_LTLIBRARIES = nbdkit-extentlist-filter.la
+
+nbdkit_extentlist_filter_la_SOURCES = \
+ extentlist.c \
+ $(top_srcdir)/include/nbdkit-filter.h \
+ $(NULL)
+
+nbdkit_extentlist_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/include \
+ -I$(top_srcdir)/common/utils \
+ $(NULL)
+nbdkit_extentlist_filter_la_CFLAGS = $(WARNINGS_CFLAGS)
+nbdkit_extentlist_filter_la_LDFLAGS = \
+ -module -avoid-version -shared \
+ -Wl,--version-script=$(top_srcdir)/filters/filters.syms \
+ $(NULL)
+nbdkit_extentlist_filter_la_LIBADD = \
+ $(top_builddir)/common/utils/libutils.la \
+ $(NULL)
+
+if HAVE_POD
+
+man_MANS = nbdkit-extentlist-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-extentlist-filter.1: nbdkit-extentlist-filter.pod
+ $(PODWRAPPER) --section=1 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
diff --git a/filters/extentlist/extentlist.c b/filters/extentlist/extentlist.c
new file mode 100644
index 0000000..5f4990b
--- /dev/null
+++ b/filters/extentlist/extentlist.c
@@ -0,0 +1,326 @@
+/* nbdkit
+ * Copyright (C) 2019-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <errno.h>
+#include <assert.h>
+
+#include <nbdkit-filter.h>
+
+#include "cleanup.h"
+#include "minmax.h"
+
+#define HOLE (NBDKIT_EXTENT_HOLE|NBDKIT_EXTENT_ZERO)
+
+static const char *extentlist;
+
+/* List of extents. Once we've finally parsed them this will be
+ * ordered, non-overlapping and have no gaps.
+ */
+struct extent {
+ uint64_t offset, length;
+ uint32_t type;
+};
+static struct extent *extents;
+static size_t nr_extents, allocated;
+
+/* Insert an extent before i. If i = nr_extents, inserts at the end. */
+static void
+insert_extent (size_t i, struct extent new_extent)
+{
+ if (nr_extents >= allocated) {
+ allocated = allocated == 0 ? 1 : allocated * 2;
+ extents = realloc (extents, (sizeof (struct extent) * allocated));
+ if (extents == NULL) {
+ nbdkit_error ("realloc: %m");
+ exit (EXIT_FAILURE);
+ }
+ }
+ memmove (&extents[i+1], &extents[i],
+ sizeof (struct extent) * (nr_extents-i));
+ extents[i] = new_extent;
+ nr_extents++;
+}
+
+static void
+extentlist_unload (void)
+{
+ free (extents);
+}
+
+/* Called for each key=value passed on the command line. */
+static int
+extentlist_config (nbdkit_next_config *next, void *nxdata,
+ const char *key, const char *value)
+{
+ if (strcmp (key, "extentlist") == 0) {
+ if (extentlist != NULL) {
+ nbdkit_error ("extentlist cannot appear twice");
+ exit (EXIT_FAILURE);
+ }
+ extentlist = value;
+ return 0;
+ }
+ else
+ return next (nxdata, key, value);
+}
+
+static int
+compare_offsets (const void *ev1, const void *ev2)
+{
+ const struct extent *e1 = ev1;
+ const struct extent *e2 = ev2;
+
+ if (e1->offset < e2->offset)
+ return -1;
+ else if (e1->offset > e2->offset)
+ return 1;
+ else
+ return 0;
+}
+
+static int
+compare_ranges (const void *ev1, const void *ev2)
+{
+ const struct extent *e1 = ev1;
+ const struct extent *e2 = ev2;
+
+ if (e1->offset < e2->offset)
+ return -1;
+ else if (e1->offset >= e2->offset + e2->length)
+ return 1;
+ else
+ return 0;
+}
+
+/* Similar to parse_extents in plugins/sh/methods.c */
+static void
+parse_extentlist (void)
+{
+ FILE *fp;
+ CLEANUP_FREE char *line = NULL;
+ size_t linelen = 0;
+ ssize_t len;
+ size_t i;
+ uint64_t end;
+
+ assert (extentlist != NULL);
+ assert (extents == NULL);
+ assert (nr_extents == 0);
+
+ fp = fopen (extentlist, "r");
+ if (!fp) {
+ nbdkit_error ("open: %s: %m", extentlist);
+ exit (EXIT_FAILURE);
+ }
+
+ while ((len = getline (&line, &linelen, fp)) != -1) {
+ const char *delim = " \t";
+ char *sp, *p;
+ int64_t offset, length;
+ uint32_t type;
+
+ if (len > 0 && line[len-1] == '\n') {
+ line[len-1] = '\0';
+ len--;
+ }
+
+ if ((p = strtok_r (line, delim, &sp)) == NULL) {
+ parse_error:
+ nbdkit_error ("%s: cannot parse %s", extentlist, line);
+ exit (EXIT_FAILURE);
+ }
+ offset = nbdkit_parse_size (p);
+ if (offset == -1)
+ exit (EXIT_FAILURE);
+
+ if ((p = strtok_r (NULL, delim, &sp)) == NULL)
+ goto parse_error;
+ length = nbdkit_parse_size (p);
+ if (length == -1)
+ exit (EXIT_FAILURE);
+
+ /* Skip zero length extents. Makes the rest of the code easier. */
+ if (length == 0)
+ continue;
+
+ if ((p = strtok_r (NULL, delim, &sp)) == NULL)
+ /* empty type field means allocated data (0) */
+ type = 0;
+ else if (sscanf (p, "%" SCNu32, &type) == 1)
+ ;
+ else {
+ type = 0;
+ if (strstr (p, "hole") != NULL)
+ type |= NBDKIT_EXTENT_HOLE;
+ if (strstr (p, "zero") != NULL)
+ type |= NBDKIT_EXTENT_ZERO;
+ }
+
+ insert_extent (nr_extents,
+ (struct extent){.offset = offset, .length=length,
+ .type=type});
+ }
+
+ fclose (fp);
+
+ /* Sort the extents by offset. */
+ qsort (extents, nr_extents, sizeof (struct extent), compare_offsets);
+
+ /* There must not be overlaps at this point. */
+ end = 0;
+ for (i = 0; i < nr_extents; ++i) {
+ if (extents[i].offset < end ||
+ extents[i].offset + extents[i].length < extents[i].offset) {
+ nbdkit_error ("extents in the extent list are overlapping");
+ exit (EXIT_FAILURE);
+ }
+ end = extents[i].offset + extents[i].length;
+ }
+
+ /* If there's a gap at the beginning, insert a hole|zero extent. */
+ if (nr_extents == 0 || extents[0].offset > 0) {
+ end = nr_extents == 0 ? UINT64_MAX : extents[0].offset;
+ insert_extent (0, (struct extent){.offset = 0, .length = end,
+ .type = HOLE});
+ }
+
+ /* Now insert hole|zero extents after every extent where there
+ * is a gap between that extent and the next one.
+ */
+ for (i = 0; i < nr_extents-1; ++i) {
+ end = extents[i].offset + extents[i].length;
+ if (end < extents[i+1].offset)
+ insert_extent (i+1, (struct extent){.offset = end,
+ .length = extents[i+1].offset - end,
+ .type = HOLE});
+ }
+
+ /* If there's a gap at the end, insert a hole|zero extent. */
+ end = extents[nr_extents-1].offset + extents[nr_extents-1].length;
+ if (end < UINT64_MAX)
+ insert_extent (nr_extents, (struct extent){.offset = end,
+ .length = UINT64_MAX-end,
+ .type = HOLE});
+
+ /* Debug the final list. */
+ for (i = 0; i < nr_extents; ++i) {
+ nbdkit_debug ("extentlist: "
+ "extent[%zu] = %" PRIu64 "-%" PRIu64 " (length %" PRIu64 ")"
+ " type %" PRIu32,
+ i, extents[i].offset,
+ extents[i].offset + extents[i].length - 1,
+ extents[i].length,
+ extents[i].type);
+ }
+}
+
+static int
+extentlist_config_complete (nbdkit_next_config_complete *next, void *nxdata)
+{
+ if (extentlist == NULL) {
+ nbdkit_error ("you must supply the extentlist parameter "
+ "on the command line");
+ return -1;
+ }
+
+ parse_extentlist ();
+
+ return next (nxdata);
+}
+
+static int
+extentlist_can_extents (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle)
+{
+ return 1;
+}
+
+/* Use -D extentlist.lookup=1 to debug the function below. */
+int extentlist_debug_lookup = 0;
+
+/* Read extents. */
+static int
+extentlist_extents (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offset,
+ uint32_t flags,
+ struct nbdkit_extents *ret_extents,
+ int *err)
+{
+ const struct extent eoffset = { .offset = offset };
+ struct extent *p;
+ ssize_t i;
+ uint64_t end;
+
+ /* Find the starting point in the extents list. */
+ p = bsearch (&eoffset, extents,
+ nr_extents, sizeof (struct extent), compare_ranges);
+ assert (p != NULL);
+ i = p - extents;
+
+ /* Add extents to the output. */
+ while (count > 0) {
+ if (extentlist_debug_lookup)
+ nbdkit_debug ("extentlist lookup: "
+ "loop i=%zd count=%" PRIu32 " offset=%" PRIu64,
+ i, count, offset);
+
+ end = extents[i].offset + extents[i].length;
+ if (nbdkit_add_extent (ret_extents, offset, end - offset,
+ extents[i].type) == -1)
+ return -1;
+
+ count -= MIN (count, end-offset);
+ offset = end;
+ i++;
+ }
+
+ return 0;
+}
+
+static struct nbdkit_filter filter = {
+ .name = "extentlist",
+ .longname = "nbdkit extentlist filter",
+ .unload = extentlist_unload,
+ .config = extentlist_config,
+ .config_complete = extentlist_config_complete,
+ .can_extents = extentlist_can_extents,
+ .extents = extentlist_extents,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
diff --git a/filters/extentlist/nbdkit-extentlist-filter.pod b/filters/extentlist/nbdkit-extentlist-filter.pod
new file mode 100644
index 0000000..adfb4ad
--- /dev/null
+++ b/filters/extentlist/nbdkit-extentlist-filter.pod
@@ -0,0 +1,90 @@
+=head1 NAME
+
+nbdkit-extentlist-filter - place extent list over a plugin
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=extentlist plugin extentlist=FILENAME
+
+=head1 DESCRIPTION
+
+C<nbdkit-extentlist-filter> is an nbdkit filter lets you place a
+static list of extents on top of an existing plugin. Extents record
+whether or not specific parts of the disk are allocated or sparse.
+
+You can use this with plugins which cannot get extent information
+themselves, but you can get this information from another source. One
+place where it is useful is with L<nbdkit-ssh-plugin(1)> because the
+sftp protocol does not support reading sparseness information, but you
+may be able to get this information directly from the source disk on
+the remote server.
+
+=head1 FILE FORMAT
+
+The list of extents is specified in a text file. There is one extent
+specified per line. Each line has the format:
+
+ offset length type
+
+The C<offset> and C<length> fields may use any format understood by
+C<nbdkit_parse_size>. The optional C<type> field may be an integer,
+missing (same as 0), or a comma-separated list of the words C<hole>
+and C<zero>. (The fields correspond to the inputs of the
+C<nbdkit_add_extent> function, see L<nbdkit-plugin(3)>).
+
+An example of a valid set of extents covering a C<10M> disk where the
+first megabyte only is allocated data:
+
+ 0 1M
+ 1M 9M hole,zero
+
+Or you could omit the C<hole,zero> extent since any gaps are assumed
+to be holes with that type:
+
+ 0 1M
+
+The extent list need not cover the whole disk, and does not need to be
+in ascending order, but it must I<not> contain overlapping extents.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<extentlist=>FILENAME
+
+Specify the file containing the extent list, in the format described
+in L</FILE FORMAT> above.
+
+=back
+
+=head1 FILES
+
+=over 4
+
+=item F<$filterdir/nbdkit-extentlist-filter.so>
+
+The filter.
+
+Use C<nbdkit --dump-config> to find the location of C<$filterdir>.
+
+=back
+
+=head1 VERSION
+
+C<nbdkit-extentlist-filter> first appeared in nbdkit 1.18.
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-cacheextents-filter(1)>,
+L<nbdkit-noextents-filter(1)>,
+L<nbdkit-filter(3)>,
+L<nbdkit-plugin(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2020 Red Hat Inc.
diff --git a/filters/noextents/nbdkit-noextents-filter.pod b/filters/noextents/nbdkit-noextents-filter.pod
index 991ecfe..0260a5c 100644
--- a/filters/noextents/nbdkit-noextents-filter.pod
+++ b/filters/noextents/nbdkit-noextents-filter.pod
@@ -47,6 +47,7 @@ C<nbdkit-noextents-filter> first appeared in nbdkit 1.14.
L<nbdkit(1)>,
L<nbdkit-filter(3)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-fua-filter(1)>,
L<nbdkit-nocache-filter(1)>,
L<nbdkit-noparallel-filter(1)>,
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 09103fb..b99952f 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -110,6 +110,7 @@ EXTRA_DIST = \
test-error100.sh \
test-error-triggered.sh \
test-export-name.sh \
+ test-extentlist.sh \
test-file-extents.sh \
test-floppy.sh \
test-foreground.sh \
@@ -1009,6 +1010,9 @@ TESTS += \
test-error-triggered.sh \
$(NULL)
+# extentlist filter test.
+TESTS += test-extentlist.sh
+
# fua filter test.
TESTS += test-fua.sh
diff --git a/tests/test-extentlist.sh b/tests/test-extentlist.sh
new file mode 100755
index 0000000..7d05de4
--- /dev/null
+++ b/tests/test-extentlist.sh
@@ -0,0 +1,175 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2016-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Test the extentlist filter.
+
+source ./functions.sh
+set -e
+set -x
+
+requires jq --version
+requires qemu-img --version
+requires qemu-img map --help
+
+out=test-extentlist.out
+input=test-extentlist.in
+expected=test-extentlist.expected
+files="$out $input $expected"
+rm -f $files
+cleanup_fn rm $files
+
+test ()
+{
+ nbdkit -v -D extentlist.lookup=1 \
+ -U - \
+ --filter=extentlist \
+ null size=$1 extentlist=$input \
+ --run 'qemu-img map -f raw --output=json $nbd' |
+ jq -c '.[] | {start:.start, length:.length, data:.data, zero:.zero}' \
+ > $out
+ diff -u $out $expected
+}
+
+# Empty extent list.
+cat > $input <<'EOF'
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+
+# Extent list covering 0-1M with data.
+cat > $input <<'EOF'
+0 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+EOF
+test 1M
+
+# Extent list covering 1-2M with data.
+cat > $input <<'EOF'
+1M 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+{"start":2097152,"length":1048576,"data":false,"zero":true}
+EOF
+test 3M
+
+# Extent list covering 1-2M with data, but in a more fragmented
+# way than the above.
+cat > $input <<'EOF'
+1024K 512K
+1536K 512K
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+{"start":2097152,"length":1048576,"data":false,"zero":true}
+EOF
+test 3M
+
+# Adjacent data and holes.
+cat > $input <<'EOF'
+0 1M
+2M 1M
+4M 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+{"start":2097152,"length":1048576,"data":true,"zero":false}
+EOF
+test 3M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+{"start":2097152,"length":1048576,"data":true,"zero":false}
+{"start":3145728,"length":1048576,"data":false,"zero":true}
+{"start":4194304,"length":1048576,"data":true,"zero":false}
+{"start":5242880,"length":1048576,"data":false,"zero":true}
+EOF
+test 6M
--
2.18.2

View File

@ -1,125 +0,0 @@
From 2a3e909e9e1ccb608bde75b76524acd753b33889 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 25 Jan 2020 11:38:14 +0000
Subject: [PATCH] extentlist: Documentation and test fixes.
Updates commit 3e770b6d6620a62546849a2863638041c0b00640.
(cherry picked from commit c16709ef663a5ed9fd9ddef4e379f316d84c9a07)
---
TODO | 12 +++++++----
.../extentlist/nbdkit-extentlist-filter.pod | 21 +++++++++++++------
plugins/curl/nbdkit-curl-plugin.pod | 1 +
plugins/ssh/nbdkit-ssh-plugin.pod | 1 +
tests/test-extentlist.sh | 2 +-
5 files changed, 26 insertions(+), 11 deletions(-)
diff --git a/TODO b/TODO
index 2a3e89d..e1ac71c 100644
--- a/TODO
+++ b/TODO
@@ -187,10 +187,6 @@ Suggestions for filters
MBs of extra data)
https://github.com/facebook/zstd/issues/395#issuecomment-535875379
-* nbdkit-extentlist-filter could read the extents generated by
- qemu-img map, allowing extents to be ported from a qemu block
- device.
-
nbdkit-rate-filter:
* allow other kinds of traffic shaping such as VBR
@@ -216,6 +212,14 @@ nbdkit-retry-filter:
* subsecond times
+nbdkit-extentlist-filter:
+
+* read the extents generated by qemu-img map, allowing extents to be
+ ported from a qemu block device
+
+* make non-read-only access safe by updating the extent list when the
+ filter sees writes and trims
+
Filters for security
--------------------
diff --git a/filters/extentlist/nbdkit-extentlist-filter.pod b/filters/extentlist/nbdkit-extentlist-filter.pod
index adfb4ad..5d1a38a 100644
--- a/filters/extentlist/nbdkit-extentlist-filter.pod
+++ b/filters/extentlist/nbdkit-extentlist-filter.pod
@@ -4,7 +4,7 @@ nbdkit-extentlist-filter - place extent list over a plugin
=head1 SYNOPSIS
- nbdkit --filter=extentlist plugin extentlist=FILENAME
+ nbdkit -r --filter=extentlist plugin extentlist=FILENAME
=head1 DESCRIPTION
@@ -13,11 +13,20 @@ static list of extents on top of an existing plugin. Extents record
whether or not specific parts of the disk are allocated or sparse.
You can use this with plugins which cannot get extent information
-themselves, but you can get this information from another source. One
-place where it is useful is with L<nbdkit-ssh-plugin(1)> because the
-sftp protocol does not support reading sparseness information, but you
-may be able to get this information directly from the source disk on
-the remote server.
+themselves, but where you can get this information from another
+source. One place where it is useful is with L<nbdkit-ssh-plugin(1)>
+because the sftp protocol does not support reading sparseness
+information, but you may be able to get this information directly from
+the source disk on the remote server using commands such as
+L<xfs_bmap(8)>. A similar situation applies to
+L<nbdkit-curl-plugin(1)>.
+
+Note that the extent list is read-only. This filter does not monitor
+writes and trims in order to update the extent list. What can happen
+is that you would write to a “hole” in the disk, but would not be able
+to read it back because the NBD client would still think that part of
+the disk is a hole. So it is generally only safe to use this filter
+in read-only mode (I<-r> option).
=head1 FILE FORMAT
diff --git a/plugins/curl/nbdkit-curl-plugin.pod b/plugins/curl/nbdkit-curl-plugin.pod
index 827e0bd..d3c8524 100644
--- a/plugins/curl/nbdkit-curl-plugin.pod
+++ b/plugins/curl/nbdkit-curl-plugin.pod
@@ -182,6 +182,7 @@ L<libcurl(3)>,
L<CURLOPT_COOKIE(3)>
L<CURLOPT_VERBOSE(3)>,
L<nbdkit(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-retry-filter(1)>,
L<nbdkit-ssh-plugin(1)>,
diff --git a/plugins/ssh/nbdkit-ssh-plugin.pod b/plugins/ssh/nbdkit-ssh-plugin.pod
index 0a0421d..3fc3146 100644
--- a/plugins/ssh/nbdkit-ssh-plugin.pod
+++ b/plugins/ssh/nbdkit-ssh-plugin.pod
@@ -316,6 +316,7 @@ C<nbdkit-ssh-plugin> first appeared in nbdkit 1.12.
L<nbdkit(1)>,
L<nbdkit-curl-plugin(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-retry-filter(1)>,
L<nbdkit-plugin(3)>,
diff --git a/tests/test-extentlist.sh b/tests/test-extentlist.sh
index 7d05de4..73ce3ca 100755
--- a/tests/test-extentlist.sh
+++ b/tests/test-extentlist.sh
@@ -50,7 +50,7 @@ cleanup_fn rm $files
test ()
{
nbdkit -v -D extentlist.lookup=1 \
- -U - \
+ -r -U - \
--filter=extentlist \
null size=$1 extentlist=$input \
--run 'qemu-img map -f raw --output=json $nbd' |
--
2.18.2

55
SOURCES/copy-patches.sh Executable file
View File

@ -0,0 +1,55 @@
#!/bin/bash -
set -e
# Maintainer script to copy patches from the git repo to the current
# directory. Use it like this:
# ./copy-patches.sh
rhel_version=8.8
# Check we're in the right directory.
if [ ! -f nbdkit.spec ]; then
echo "$0: run this from the directory containing 'nbdkit.spec'"
exit 1
fi
git_checkout=$HOME/d/nbdkit-rhel-$rhel_version
if [ ! -d $git_checkout ]; then
echo "$0: $git_checkout does not exist"
echo "This script is only for use by the maintainer when preparing a"
echo "nbdkit release on RHEL."
exit 1
fi
# Get the base version of nbdkit.
version=`grep '^Version:' nbdkit.spec | awk '{print $2}'`
tag="v$version"
# Remove any existing patches.
git rm -f [0-9]*.patch ||:
rm -f [0-9]*.patch
# Get the patches.
(cd $git_checkout; rm -f [0-9]*.patch; git format-patch -N $tag)
mv $git_checkout/[0-9]*.patch .
# Remove any not to be applied.
rm -f *NOT-FOR-RPM*.patch
# Add the patches.
git add [0-9]*.patch
# Print out the patch lines.
echo
echo "--- Copy the following text into nbdkit.spec file"
echo
echo "# Patches."
for f in [0-9]*.patch; do
n=`echo $f | awk -F- '{print $1}'`
echo "Patch$n: $f"
done
echo
echo "--- End of text"

View File

@ -1,17 +0,0 @@
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEE93dPsa0HSn6Mh2fqkXOPc+G3aKAFAl4wFvoRHHJpY2hAYW5u
ZXhpYS5vcmcACgkQkXOPc+G3aKASexAAmpZw61rCI7SY8zm4O0gb+pIx7oLYx0Lq
2puIftzxUUw9Q6pFJJyXSvlsvHy3qUF7HiMVdpW61ItIChV1xBDVKEPAacNzsZh4
30CI7kfJMfj6u+hpOCVlLk4uJFjZkmIpEKkDpEBemxLMME4JsLJdawKzKhjT2PI7
dWMjYkOeD4NkAzQLQGskEswoIgZQ0twuyPUErjEL9fcXw4OjxFvQJG85FsIF2lR6
FUDQg5y9YLzeMJMsjW4rO+LAz2c1mJwYR1EgYP43avm/pJfd1mVQLGRoLb7NwMSw
6mkwhJ4Kvq6BN0PSqpKqQtXZrDoElWN8cVJVf+dAjONcvzYi0gsHWDL+FZ731Q2M
s4nq0aRscBTL2DOaE9DzBY2AO1jKUB/+02qRpidWTYBmsmL2QQI8n33Q7JuDuEXX
bVm1RDA4ike4PUXXY5KJ6MZhKID5453SVFausFse+u4MCQHQPFYspkXmaNWRhjgs
yu2zPc9jHdBkpzNov/CCZoFketFRz/BKexBeH2vcfTYfREVf9lEZi7qEa0kQHDn9
EMTFsCqmGat9TEVbt9t8c/tODTeRE00MFx4gPspzy+m4YP+Gl3ySHsAbPQ90uBGX
c8xggwqWXr1GAP5HbAhs/Bs7USrWMMgqii1ppnzoAkHh+j4rsdL4dS2dmhxX756u
IKP/JC2oA8U=
=mV8z
-----END PGP SIGNATURE-----

View File

@ -0,0 +1,17 @@
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEE93dPsa0HSn6Mh2fqkXOPc+G3aKAFAl/3RBgRHHJpY2hAYW5u
ZXhpYS5vcmcACgkQkXOPc+G3aKBIIRAAmgoGrmJ8aYO7z+kKgNFjd/p0QxRTZhS/
ol59ojG6jIzN2x/C2PFbRmPB6HJTEg4anrDX04WrP6R+lID1RrH9pTFQabv0YDQC
z49oeXAqINYHvAqgFUJCwlymd7BHEYUudLlK3yu7gQKxMM+J/2v0glpxrtLM7KlD
vvSZkVfbvHlCWIbMWLWIaRHeoWZIXNOjsAp3uEWN2YgikDoxbXVKoh07JoQx5tJ5
2U+a/zo4BQuRspjnhmWc252ZF/8d954/L8J+2mKvbRRf2iAmsqPgS+MNi7WKWO4K
w7/urKn0osuOaArs5xYHJnApmJ9U88CzZpoHQkYhcGgnDOipW9ByJRzT41vVQPW5
IluQODpZUuawWtRIwV/Eoi+LaV2gINAL48Afr02UFYj4gmYQ5TeayLP7NKRQO0VL
jwL4Z3a0cDyUX4i1OArn2ll8THfiog38HfLb70AG1l3P1BVoVVBYWCYbs4xgC9IK
LWkjPKuGXvkGVfZi0nCGdPTOoB1CqCXUvKHXm52FCHg12uJMrBQEivodBoCTbtl0
fSjULQcfrovUEb4d/rDAX7EgJbFS+1jDnodaFHsmNToo3CqfkMBdhLkxG3XExwjy
OOR34wZssjTLsLlWH/RPucWD25RDy1vdPBska9QvvO7W0p+aOtFbnttkTh5cqs45
rHg/sDEiaLA=
=OrsS
-----END PGP SIGNATURE-----

File diff suppressed because it is too large Load Diff