rpms/nbdkit
7
0
Fork 0
Code Issues Pull Requests Releases Activity

import nbdkit-1.24.0-4.module+el8.6.0+14480+c0a3aa0f

Browse Source
This commit is contained in:
CentOS Sources 2022-05-10 03:08:30 -04:00 committed by Stepan Oksanichenko
parent 003a44a973
commit f07d6b31a4
29 changed files with 1017 additions and 4568 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitignore vendored
View File

@ -1,2 +1,2 @@
SOURCES/libguestfs.keyring
SOURCES/nbdkit-1.16.2.tar.gz
SOURCES/nbdkit-1.24.0.tar.gz

2
.nbdkit.metadata
View File

@ -1,2 +1,2 @@
1bbc40f501a7fef9eef2a39b701a71aee2fea7c4 SOURCES/libguestfs.keyring
42a5761cd3403c02c43cdf7d541ff3faaf8b4769 SOURCES/nbdkit-1.16.2.tar.gz
069720cc0d1502b007652101d293a57d7b4d7c41 SOURCES/nbdkit-1.24.0.tar.gz

82
SOURCES/0001-cache-cow-Fix-data-corruption-in-zero-and-trim-on-un.patch Normal file
View File

@ -0,0 +1,82 @@
From 99788909d9ec36e3210cf85976fe5b18da690ddd Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Wed, 4 Aug 2021 20:24:59 +0100
Subject: [PATCH] cache, cow: Fix data corruption in zero and trim on unaligned
tail
Commit eb6009b092 ("cache, cow: Reduce use of bounce-buffer") first
introduced in nbdkit 1.14 added an optimization of the
read-modify-write mechanism used for unaligned heads and tails when
zeroing in the cache layer.
Unfortunately the part applied to the tail contained a mistake: It
zeroes the end of the buffer rather than the beginning. This causes
data corruption when you use the zero or trim function with an offset
and count which is not aligned to the block size.
Although the bug has been around for years, a recent change made it
more likely to happen. Commit c1905b0a28 ("cache, cow: Use a 64K
block size by default") increased the default block size from 4K to
64K. Most filesystems use a 4K block size so operations like fstrim
will make 4K-aligned requests, and with a 4K block size also in the
cache or cow filter the unaligned case would never have been hit
before.
We can demonstrate the bug simply by filling a buffer with data
(100000 bytes in the example), and then trimming that data, which
ought to zero it out.
Before this commit there is data visible after the trim:
$ nbdkit --filter=cow data "0x21 * 100000" --run 'nbdsh -u $uri -c "h.trim(100000, 0)" ; nbdcopy $uri - | hexdump -C'
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00018000 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 |!!!!!!!!!!!!!!!!|
*
000186a0
After this commit the trim completely clears the data:
$ nbdkit --filter=cow data "0x21 * 100000" --run 'nbdsh -u $uri -c "h.trim(100000, 0)" ; nbdcopy $uri - | hexdump -C'
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000186a0
Thanks: Ming Xie for finding the bug
Fixes: commit eb6009b092ae642ed25f133d487dd40ef7bf70f8
(cherry picked from commit a0ae7b2158598ce48ac31706319007f716d01c87)
(cherry picked from commit c0b15574647672cb5c48178333acdd07424692ef)
---
filters/cache/cache.c | 2 +-
filters/cow/cow.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/filters/cache/cache.c b/filters/cache/cache.c
index 91dcc43d..0616cc7b 100644
--- a/filters/cache/cache.c
+++ b/filters/cache/cache.c
@@ -493,7 +493,7 @@ cache_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
r = blk_read (next_ops, nxdata, blknum, block, err);
if (r != -1) {
- memset (&block[count], 0, blksize - count);
+ memset (block, 0, count);
r = blk_write (next_ops, nxdata, blknum, block, flags, err);
}
if (r == -1)
diff --git a/filters/cow/cow.c b/filters/cow/cow.c
index 51ca64a4..1cfcc4e7 100644
--- a/filters/cow/cow.c
+++ b/filters/cow/cow.c
@@ -419,7 +419,7 @@ cow_zero (struct nbdkit_next_ops *next_ops, void *nxdata,
ACQUIRE_LOCK_FOR_CURRENT_SCOPE (&lock);
r = blk_read (next_ops, nxdata, blknum, block, err);
if (r != -1) {
- memset (&block[count], 0, BLKSIZE - count);
+ memset (block, 0, count);
r = blk_write (blknum, block, err);
}
if (r == -1)
--
2.31.1

75
SOURCES/0001-server-Allow-D-nbdkit.-debug-flags-for-the-core-serv.patch
View File

@ -1,75 +0,0 @@
From d7836fb0a7131c725e3c02be7e48e99c671637c3 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 08:57:15 +0000
Subject: [PATCH 01/19] server: Allow -D nbdkit.* debug flags for the core
server.
These work like plugin/filter debug flags, but apply to the internals
of the server.
(cherry picked from commit 3b45db234a691f8ff926a6fef583e11c3601f112)
---
docs/nbdkit.pod | 7 +++++++
docs/synopsis.txt | 2 +-
server/main.c | 3 +++
server/nbdkit.syms | 2 ++
4 files changed, 13 insertions(+), 1 deletion(-)
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index a2e72b13..346d8332 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -177,6 +177,13 @@ Display brief command line usage information and exit.
Set the plugin or filter Debug Flag called C<FLAG> to the integer
value C<N>. See L<nbdkit-plugin(3)/Debug Flags>.
+=item B<-D> nbdkit.FLAG=N
+
+=item B<--debug> nbdkit.FLAG=N
+
+Set the nbdkit server Debug Flag called C<FLAG> to the integer value
+C<N>.
+
=item B<--dump-config>
Dump out the compile-time configuration values and exit.
diff --git a/docs/synopsis.txt b/docs/synopsis.txt
index 3c239373..c3675422 100644
--- a/docs/synopsis.txt
+++ b/docs/synopsis.txt
@@ -1,4 +1,4 @@
-nbdkit [-D|--debug PLUGIN|FILTER.FLAG=N]
+nbdkit [-D|--debug PLUGIN|FILTER|nbdkit.FLAG=N]
[-e|--exportname EXPORTNAME] [--exit-with-parent]
[--filter FILTER ...] [-f|--foreground]
[-g|--group GROUP] [-i|--ipaddr IPADDR]
diff --git a/server/main.c b/server/main.c
index d39941b1..11ba1e6d 100644
--- a/server/main.c
+++ b/server/main.c
@@ -563,6 +563,9 @@ main (int argc, char *argv[])
free (t);
}
+ /* Apply nbdkit.* flags for the server. */
+ apply_debug_flags (NULL, "nbdkit");
+
/* Check all debug flags were used, and free them. */
free_debug_flags ();
diff --git a/server/nbdkit.syms b/server/nbdkit.syms
index 390972e2..96c22c07 100644
--- a/server/nbdkit.syms
+++ b/server/nbdkit.syms
@@ -67,6 +67,8 @@
nbdkit_vdebug;
nbdkit_verror;
+ nbdkit_debug_*;
+
# Everything else is hidden.
local: *;
};
--
2.18.2

67
SOURCES/0002-server-Allow-D-debug-flags-to-contain-dots-for-names.patch
View File

@ -1,67 +0,0 @@
From e5d2d44fff9214725506cbc84e7b3c035ec0eae9 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 11:06:36 +0000
Subject: [PATCH 02/19] server: Allow -D debug flags to contain dots for
namespacing.
This is just a convenience. Either of:
-D myplugin.foo_bar=1
-D myplugin.foo.bar=1
correspond to the same plugin variable "myplugin_debug_foo_bar".
(cherry picked from commit a895fa84aaa50f52af68319523020046394c789f)
---
docs/nbdkit-plugin.pod | 8 ++++++++
server/debug-flags.c | 10 +++++++++-
2 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/docs/nbdkit-plugin.pod b/docs/nbdkit-plugin.pod
index b69cb825..879ddf09 100644
--- a/docs/nbdkit-plugin.pod
+++ b/docs/nbdkit-plugin.pod
@@ -1298,6 +1298,14 @@ You should only use this feature for debug settings. For general
settings use ordinary plugin parameters. Debug Flags can only be C
ints. They are not supported by non-C language plugins.
+For convenience C<'.'> characters are replaced with C<'_'> characters
+in the variable name, so both of these parameters:
+
+ -D myplugin.foo_bar=1
+ -D myplugin.foo.bar=1
+
+correspond to the plugin variable C<myplugin_debug_foo_bar>.
+
=head1 INSTALLING THE PLUGIN
The plugin is a C<*.so> file and possibly a manual page. You can of
diff --git a/server/debug-flags.c b/server/debug-flags.c
index 9344d85c..5e06f5ed 100644
--- a/server/debug-flags.c
+++ b/server/debug-flags.c
@@ -56,12 +56,20 @@ static char *
symbol_of_debug_flag (const char *name, const char *flag)
{
char *var;
+ size_t i;
+ int len;
- if (asprintf (&var, "%s_debug_%s", name, flag) == -1) {
+ if ((len = asprintf (&var, "%s_debug_%s", name, flag)) == -1) {
perror ("asprintf");
exit (EXIT_FAILURE);
}
+ /* If there are any '.'s remaining in the name, convert them to '_'. */
+ for (i = 0; i < (size_t) len; ++i) {
+ if (var[i] == '.')
+ var[i] = '_';
+ }
+
return var; /* caller frees */
}
--
2.18.2

94
SOURCES/0002-server-CVE-2021-3716-reset-structured-replies-on-sta.patch Normal file
View File

@ -0,0 +1,94 @@
From 6b9d4380df9bd0be91f49aad8c4f47b4e672adde Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Mon, 16 Aug 2021 13:43:29 -0500
Subject: [PATCH] server: CVE-2021-3716 reset structured replies on starttls
https://nostarttls.secvuln.info/ pointed out a series of CVEs in
common implementation flaw in various SMTP and IMAP clients and
servers, all with a common thread of improperly caching plaintext
state across the STARTTLS encryption boundary; and recommended that
other protocols with a STARTTLS operation perform a similar audit.
It turns out that nbdkit has the same vulnerability in regards to the
NBD protocol: when nbdkit is run in opportunistic TLS mode, an
attacker is able to inject a plaintext NBD_OPT_STRUCTURED_REPLY before
proxying everything else a client sends to the server; if the server
then acts on that plaintext request (as nbdkit did before this patch),
then the server ends up sending structured replies to at least
NBD_CMD_READ, even though the client was assuming that the transition
to TLS has ruled out a MitM attack.
On the bright side, nbdkit's behavior on a second
NBD_OPT_STRUCTURED_REPLY was to still reply with success, so a client
that always requests structured replies after starting TLS sees no
difference in behavior (that is, qemu 2.12 and later are immune) (had
nbdkit given an error to the second request, that may have caused
confusion to more clients). And there is always the mitigation of
using --tls=require, which lets nbdkit reject the MitM message
pre-encryption. However, nbd-client 3.15 to the present do not
understand structured replies, and I have confirmed that a MitM
attacker can thus cause a denial-of-service attack that does not
trigger until the client does its first encrypted NBD_CMD_READ.
The NBD spec has been recently tightened to declare the nbdkit
behavior to be a security hole:
https://github.com/NetworkBlockDevice/nbd/commit/77e55378096aa
Fixes: eaa4c6e9a2c4bd (server: Minimal implementation of NBD Structured Replies.)
(cherry picked from commit 09a13dafb7bb3a38ab52eb5501cba786365ba7fd)
(cherry picked from commit 6185b15a81e6915734d678f0781e31d45a7941a1)
---
docs/nbdkit-security.pod | 11 +++++++++--
server/protocol-handshake-newstyle.c | 3 ++-
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/docs/nbdkit-security.pod b/docs/nbdkit-security.pod
index 3a28e54d..5a4e6da8 100644
--- a/docs/nbdkit-security.pod
+++ b/docs/nbdkit-security.pod
@@ -10,7 +10,7 @@ For how to report new security issues, see the C<SECURITY> file in the
top level source directory, also available online here:
L<https://github.com/libguestfs/nbdkit/blob/master/SECURITY>
-=head2 CVE-2019-14850
+=head2 CVE-2019-14850
denial of service due to premature opening of back-end connection
See the full announcement and links to mitigation, tests and fixes
@@ -26,6 +26,13 @@ See the full announcement and links to mitigation, tests and fixes
here:
https://www.redhat.com/archives/libguestfs/2019-September/msg00272.html
+=head2 CVE-2021-3716
+structured read denial of service attack against starttls
+
+See the full announcement and links to mitigation, tests and fixes
+here:
+https://www.redhat.com/archives/libguestfs/2021-August/msg00083.html
+
=head1 SEE ALSO
L<nbdkit(1)>.
@@ -38,4 +45,4 @@ Richard W.M. Jones
=head1 COPYRIGHT
-Copyright (C) 2013-2020 Red Hat Inc.
+Copyright (C) 2013-2021 Red Hat Inc.
diff --git a/server/protocol-handshake-newstyle.c b/server/protocol-handshake-newstyle.c
index 0a76a814..b94950e2 100644
--- a/server/protocol-handshake-newstyle.c
+++ b/server/protocol-handshake-newstyle.c
@@ -495,7 +495,8 @@ negotiate_handshake_newstyle_options (void)
return -1;
conn->using_tls = true;
debug ("using TLS on this connection");
- /* Wipe out any cached default export name. */
+ /* Wipe out any cached state. */
+ conn->structured_replies = false;
for_each_backend (b) {
struct handle *h = get_handle (conn, b->i);
free (h->default_exportname);
--
2.31.1

451
SOURCES/0003-server-Add-D-nbdkit.backend.controlpath-and-D-nbdkit.patch
View File

@ -1,451 +0,0 @@
From 83c72d9bf9d6a9ccf6939b4ebd0028b62673a78a Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 12 Dec 2019 10:57:52 +0000
Subject: [PATCH 03/19] server: Add -D nbdkit.backend.controlpath and -D
nbdkit.backend.datapath.
These can be used to suppress verbose debugging messages from the
backend.
BugLink: https://bugzilla.redhat.com/1782868
Cherry picked from commit 231717e8cd5f27d76631be6651062d5a5ccf7fdc.
Remove use of nofilter from the test.
---
docs/nbdkit.pod | 35 ++++++++++++-
server/backend.c | 83 ++++++++++++++++++------------
tests/Makefile.am | 4 ++
tests/test-nbdkit-backend-debug.sh | 70 +++++++++++++++++++++++++
4 files changed, 158 insertions(+), 34 deletions(-)
create mode 100755 tests/test-nbdkit-backend-debug.sh
diff --git a/docs/nbdkit.pod b/docs/nbdkit.pod
index 346d8332..38e6bfca 100644
--- a/docs/nbdkit.pod
+++ b/docs/nbdkit.pod
@@ -182,7 +182,7 @@ value C<N>. See L<nbdkit-plugin(3)/Debug Flags>.
=item B<--debug> nbdkit.FLAG=N
Set the nbdkit server Debug Flag called C<FLAG> to the integer value
-C<N>.
+C<N>. See L</SERVER DEBUG FLAGS> below.
=item B<--dump-config>
@@ -527,6 +527,39 @@ languages. The file should be executable. For example:
(see L<nbdkit-perl-plugin(3)> for a full example).
+=head1 SERVER DEBUG FLAGS
+
+As well as enabling or disabling debugging in the server using
+I<--verbose> you can control extra debugging in the server using the
+C<-D nbdkit.*> flags listed in this section. Note these flags are an
+internal implementation detail of the server and may be changed or
+removed at any time in the future.
+
+=over 4
+
+=item B<-D nbdkit.backend.controlpath=0>
+
+=item B<-D nbdkit.backend.controlpath=1>
+
+=item B<-D nbdkit.backend.datapath=0>
+
+=item B<-D nbdkit.backend.datapath=1>
+
+These flags control the verbosity of nbdkit backend debugging messages
+(the ones which show every request processed by the server). The
+default for both settings is C<1> (normal debugging) but you can set
+them to C<0> to suppress these messages.
+
+C<-D nbdkit.backend.datapath=0> is the more useful setting which lets you
+suppress messages about pread, pwrite, zero, trim, etc. commands.
+When transferring large amounts of data these messages are numerous
+and not usually very interesting.
+
+C<-D nbdkit.backend.controlpath=0> suppresses the non-datapath
+commands (config, open, close, can_write, etc.)
+
+=back
+
=head1 SIGNALS
nbdkit responds to the following signals:
diff --git a/server/backend.c b/server/backend.c
index b9fe2a21..208c07b1 100644
--- a/server/backend.c
+++ b/server/backend.c
@@ -46,6 +46,22 @@
/* Helpers for registering a new backend. */
+/* Use:
+ * -D nbdkit.backend.controlpath=0 to suppress control path debugging.
+ * -D nbdkit.backend.datapath=0 to suppress data path debugging.
+ */
+int nbdkit_debug_backend_controlpath = 1;
+int nbdkit_debug_backend_datapath = 1;
+
+#define controlpath_debug(fs, ...) \
+ do { \
+ if (nbdkit_debug_backend_controlpath) debug ((fs), ##__VA_ARGS__); \
+ } while (0)
+#define datapath_debug(fs, ...) \
+ do { \
+ if (nbdkit_debug_backend_datapath) debug ((fs), ##__VA_ARGS__); \
+ } while (0)
+
void
backend_init (struct backend *b, struct backend *next, size_t index,
const char *filename, void *dl, const char *type)
@@ -108,7 +124,7 @@ backend_load (struct backend *b, const char *name, void (*load) (void))
apply_debug_flags (b->dl, name);
/* Call the on-load callback if it exists. */
- debug ("%s: load", name);
+ controlpath_debug ("%s: load", name);
if (load)
load ();
}
@@ -121,7 +137,7 @@ backend_unload (struct backend *b, void (*unload) (void))
*/
lock_unload ();
- debug ("%s: unload %s", b->name, b->type);
+ controlpath_debug ("%s: unload %s", b->name, b->type);
if (unload)
unload ();
@@ -139,7 +155,7 @@ backend_open (struct backend *b, struct connection *conn, int readonly)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: open readonly=%d", b->name, readonly);
+ controlpath_debug ("%s: open readonly=%d", b->name, readonly);
assert (h->handle == NULL);
assert ((h->state & HANDLE_OPEN) == 0);
@@ -151,7 +167,7 @@ backend_open (struct backend *b, struct connection *conn, int readonly)
* inner-to-outer ordering.
*/
h->handle = b->open (b, conn, readonly);
- debug ("%s: open returned handle %p", b->name, h->handle);
+ controlpath_debug ("%s: open returned handle %p", b->name, h->handle);
if (h->handle == NULL) {
if (b->i) /* Do not strand backend if this layer failed */
@@ -179,7 +195,7 @@ backend_prepare (struct backend *b, struct connection *conn)
if (b->i && backend_prepare (b->next, conn) == -1)
return -1;
- debug ("%s: prepare readonly=%d", b->name, h->can_write == 0);
+ controlpath_debug ("%s: prepare readonly=%d", b->name, h->can_write == 0);
if (b->prepare (b, conn, h->handle, h->can_write == 0) == -1)
return -1;
@@ -196,7 +212,7 @@ backend_finalize (struct backend *b, struct connection *conn)
* filter furthest away from the plugin, and matching .close order.
*/
- debug ("%s: finalize", b->name);
+ controlpath_debug ("%s: finalize", b->name);
/* Once finalize fails, we can do nothing further on this connection */
if (h->state & HANDLE_FAILED)
@@ -223,7 +239,7 @@ backend_close (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
/* outer-to-inner order, opposite .open */
- debug ("%s: close", b->name);
+ controlpath_debug ("%s: close", b->name);
if (h->handle) {
assert (h->state & HANDLE_OPEN);
@@ -252,7 +268,7 @@ backend_valid_range (struct backend *b, struct connection *conn,
int
backend_reopen (struct backend *b, struct connection *conn, int readonly)
{
- debug ("%s: reopen readonly=%d", b->name, readonly);
+ controlpath_debug ("%s: reopen readonly=%d", b->name, readonly);
if (backend_finalize (b, conn) == -1)
return -1;
@@ -274,7 +290,7 @@ backend_get_size (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: get_size", b->name);
+ controlpath_debug ("%s: get_size", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->exportsize == -1)
@@ -287,7 +303,7 @@ backend_can_write (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_write", b->name);
+ controlpath_debug ("%s: can_write", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_write == -1)
@@ -300,7 +316,7 @@ backend_can_flush (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_flush", b->name);
+ controlpath_debug ("%s: can_flush", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_flush == -1)
@@ -313,7 +329,7 @@ backend_is_rotational (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: is_rotational", b->name);
+ controlpath_debug ("%s: is_rotational", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->is_rotational == -1)
@@ -327,7 +343,7 @@ backend_can_trim (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_trim", b->name);
+ controlpath_debug ("%s: can_trim", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_trim == -1) {
@@ -347,7 +363,7 @@ backend_can_zero (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_zero", b->name);
+ controlpath_debug ("%s: can_zero", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_zero == -1) {
@@ -367,7 +383,7 @@ backend_can_fast_zero (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_fast_zero", b->name);
+ controlpath_debug ("%s: can_fast_zero", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_fast_zero == -1) {
@@ -386,7 +402,7 @@ backend_can_extents (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_extents", b->name);
+ controlpath_debug ("%s: can_extents", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_extents == -1)
@@ -400,7 +416,7 @@ backend_can_fua (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
int r;
- debug ("%s: can_fua", b->name);
+ controlpath_debug ("%s: can_fua", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_fua == -1) {
@@ -420,7 +436,7 @@ backend_can_multi_conn (struct backend *b, struct connection *conn)
struct b_conn_handle *h = &conn->handles[b->i];
assert (h->handle && (h->state & HANDLE_CONNECTED));
- debug ("%s: can_multi_conn", b->name);
+ controlpath_debug ("%s: can_multi_conn", b->name);
if (h->can_multi_conn == -1)
h->can_multi_conn = b->can_multi_conn (b, conn, h->handle);
@@ -432,7 +448,7 @@ backend_can_cache (struct backend *b, struct connection *conn)
{
struct b_conn_handle *h = &conn->handles[b->i];
- debug ("%s: can_cache", b->name);
+ controlpath_debug ("%s: can_cache", b->name);
assert (h->handle && (h->state & HANDLE_CONNECTED));
if (h->can_cache == -1)
@@ -451,8 +467,8 @@ backend_pread (struct backend *b, struct connection *conn,
assert (h->handle && (h->state & HANDLE_CONNECTED));
assert (backend_valid_range (b, conn, offset, count));
assert (flags == 0);
- debug ("%s: pread count=%" PRIu32 " offset=%" PRIu64,
- b->name, count, offset);
+ datapath_debug ("%s: pread count=%" PRIu32 " offset=%" PRIu64,
+ b->name, count, offset);
r = b->pread (b, conn, h->handle, buf, count, offset, flags, err);
if (r == -1)
@@ -475,8 +491,8 @@ backend_pwrite (struct backend *b, struct connection *conn,
assert (!(flags & ~NBDKIT_FLAG_FUA));
if (fua)
assert (h->can_fua > NBDKIT_FUA_NONE);
- debug ("%s: pwrite count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
- b->name, count, offset, fua);
+ datapath_debug ("%s: pwrite count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
+ b->name, count, offset, fua);
r = b->pwrite (b, conn, h->handle, buf, count, offset, flags, err);
if (r == -1)
@@ -494,7 +510,7 @@ backend_flush (struct backend *b, struct connection *conn,
assert (h->handle && (h->state & HANDLE_CONNECTED));
assert (h->can_flush == 1);
assert (flags == 0);
- debug ("%s: flush", b->name);
+ datapath_debug ("%s: flush", b->name);
r = b->flush (b, conn, h->handle, flags, err);
if (r == -1)
@@ -518,8 +534,8 @@ backend_trim (struct backend *b, struct connection *conn,
assert (!(flags & ~NBDKIT_FLAG_FUA));
if (fua)
assert (h->can_fua > NBDKIT_FUA_NONE);
- debug ("%s: trim count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
- b->name, count, offset, fua);
+ datapath_debug ("%s: trim count=%" PRIu32 " offset=%" PRIu64 " fua=%d",
+ b->name, count, offset, fua);
r = b->trim (b, conn, h->handle, count, offset, flags, err);
if (r == -1)
@@ -547,9 +563,10 @@ backend_zero (struct backend *b, struct connection *conn,
assert (h->can_fua > NBDKIT_FUA_NONE);
if (fast)
assert (h->can_fast_zero == 1);
- debug ("%s: zero count=%" PRIu32 " offset=%" PRIu64
- " may_trim=%d fua=%d fast=%d",
- b->name, count, offset, !!(flags & NBDKIT_FLAG_MAY_TRIM), fua, fast);
+ datapath_debug ("%s: zero count=%" PRIu32 " offset=%" PRIu64
+ " may_trim=%d fua=%d fast=%d",
+ b->name, count, offset,
+ !!(flags & NBDKIT_FLAG_MAY_TRIM), fua, fast);
r = b->zero (b, conn, h->handle, count, offset, flags, err);
if (r == -1) {
@@ -572,8 +589,8 @@ backend_extents (struct backend *b, struct connection *conn,
assert (h->can_extents >= 0);
assert (backend_valid_range (b, conn, offset, count));
assert (!(flags & ~NBDKIT_FLAG_REQ_ONE));
- debug ("%s: extents count=%" PRIu32 " offset=%" PRIu64 " req_one=%d",
- b->name, count, offset, !!(flags & NBDKIT_FLAG_REQ_ONE));
+ datapath_debug ("%s: extents count=%" PRIu32 " offset=%" PRIu64 " req_one=%d",
+ b->name, count, offset, !!(flags & NBDKIT_FLAG_REQ_ONE));
if (h->can_extents == 0) {
/* By default it is safe assume that everything in the range is
@@ -602,8 +619,8 @@ backend_cache (struct backend *b, struct connection *conn,
assert (h->can_cache > NBDKIT_CACHE_NONE);
assert (backend_valid_range (b, conn, offset, count));
assert (flags == 0);
- debug ("%s: cache count=%" PRIu32 " offset=%" PRIu64,
- b->name, count, offset);
+ datapath_debug ("%s: cache count=%" PRIu32 " offset=%" PRIu64,
+ b->name, count, offset);
if (h->can_cache == NBDKIT_CACHE_EMULATE) {
static char buf[MAX_REQUEST_SIZE]; /* data sink, never read */
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 01341973..d225cc63 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -135,6 +135,7 @@ EXTRA_DIST = \
test-nbd-extents.sh \
test-nbd-tls.sh \
test-nbd-tls-psk.sh \
+ test-nbdkit-backend-debug.sh \
test-nozero.sh \
test-null-extents.sh \
test_ocaml_plugin.ml \
@@ -746,6 +747,9 @@ endif HAVE_VDDK
# zero plugin test.
TESTS += test-zero.sh
+# -D nbdkit.backend.* settings.
+TESTS += test-nbdkit-backend-debug.sh
+
#----------------------------------------------------------------------
# Tests of language plugins.
diff --git a/tests/test-nbdkit-backend-debug.sh b/tests/test-nbdkit-backend-debug.sh
new file mode 100755
index 00000000..69a69a7c
--- /dev/null
+++ b/tests/test-nbdkit-backend-debug.sh
@@ -0,0 +1,70 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+source ./functions.sh
+set -x
+set -e
+
+requires qemu-img --version
+
+out="test-nbdkit-backend-debug.out"
+debug="test-nbdkit-backend-debug.debug"
+files="$out $debug"
+rm -f $files
+cleanup_fn rm -f $files
+
+nbdkit -U - \
+ -v \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain all debugging messages.
+grep '^nbdkit:.*debug: memory: open' $debug
+grep '^nbdkit:.*debug: memory: pread' $debug
+
+nbdkit -U - \
+ -v -D nbdkit.backend.controlpath=0 \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain only datapath messages.
+grep -v '^nbdkit:.*debug: memory: open' $debug
+grep '^nbdkit:.*debug: memory: pread' $debug
+
+nbdkit -U - \
+ -v -D nbdkit.backend.datapath=0 \
+ memory 10M \
+ --run "qemu-img convert \$nbd $out" |& tee $debug
+
+# Should contain only controlpath messages.
+grep '^nbdkit:.*debug: memory: open' $debug
+grep -v '^nbdkit:.*debug: memory: pread' $debug
--
2.18.2

40
SOURCES/0003-server-reset-meta-context-replies-on-starttls.patch Normal file
View File

@ -0,0 +1,40 @@
From add9b794b9dc697a1b52115c997fcfb6e06bf64c Mon Sep 17 00:00:00 2001
From: Eric Blake <eblake@redhat.com>
Date: Mon, 16 Aug 2021 13:43:29 -0500
Subject: [PATCH] server: reset meta context replies on starttls
Related to CVE-2021-3716, but not as severe. No compliant client will
send NBD_CMD_BLOCK_STATUS unless it first negotiates
NBD_OPT_SET_META_CONTEXT. If an attacker injects a premature
SET_META_CONTEXT, either the client will never notice (because it
never uses BLOCK_STATUS), or the client will overwrite the attacker's
attempt with the client's own SET_META_CONTEXT request after
encryption is enabled. So I don't class this as having the potential
to trigger denial-of-service due to any protocol mismatch between
compliant client and server (I don't care what happens with
non-compliant clients).
Fixes: 26455d45 (server: protocol: Implement Block Status "base:allocation".)
(cherry picked from commit 6c5faac6a37077cf2366388a80862bb00616d0d8)
(cherry picked from commit 814d8103fb4b581dc01dfd25d2cd81596576f211)
---
server/protocol-handshake-newstyle.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/server/protocol-handshake-newstyle.c b/server/protocol-handshake-newstyle.c
index b94950e2..eb0f3961 100644
--- a/server/protocol-handshake-newstyle.c
+++ b/server/protocol-handshake-newstyle.c
@@ -497,6 +497,9 @@ negotiate_handshake_newstyle_options (void)
debug ("using TLS on this connection");
/* Wipe out any cached state. */
conn->structured_replies = false;
+ free (conn->exportname_from_set_meta_context);
+ conn->exportname_from_set_meta_context = NULL;
+ conn->meta_context_base_allocation = false;
for_each_backend (b) {
struct handle *h = get_handle (conn, b->i);
free (h->default_exportname);
--
2.31.1

59
SOURCES/0004-cow-Fix-for-qemu-6.1-which-requires-backing-format.patch Normal file
View File

@ -0,0 +1,59 @@
From 3c2879a38c299b725091cea45329879e3f46fc99 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 31 Aug 2021 11:23:27 +0100
Subject: [PATCH] cow: Fix for qemu 6.1 which requires backing format
The diffing example in the manual created a qcow2 file with a backing
file but did not specify the backing format. However qemu 6.1 now
requires this and fails with:
qemu-img: cow-diff.qcow2: Backing file specified without backing format
or:
qemu-img: Could not change the backing file to 'cow-base.img': backing format must be specified
Fix the example by adding the -F option to the command line.
Also there was a test of this rebasing sequence which failed, so this
commit updates the test too.
(cherry picked from commit 618290ef33ce13b75c1a79fea1f1ffb327b5ba07)
---
filters/cow/nbdkit-cow-filter.pod | 4 ++--
tests/test-cow.sh | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/filters/cow/nbdkit-cow-filter.pod b/filters/cow/nbdkit-cow-filter.pod
index 4d5ae856..510bdd40 100644
--- a/filters/cow/nbdkit-cow-filter.pod
+++ b/filters/cow/nbdkit-cow-filter.pod
@@ -101,8 +101,8 @@ At the end, disconnect the client.
Run these C<qemu-img> commands to construct a qcow2 file containing
the differences:
- qemu-img create -f qcow2 -b nbd:localhost diff.qcow2
- qemu-img rebase -b disk.img diff.qcow2
+ qemu-img create -F raw -b nbd:localhost -f qcow2 diff.qcow2
+ qemu-img rebase -F raw -b disk.img -f qcow2 diff.qcow2
F<diff.qcow2> now contains the differences between the base
(F<disk.img>) and the changes stored in nbdkit-cow-filter. C<nbdkit>
diff --git a/tests/test-cow.sh b/tests/test-cow.sh
index 8772afd7..edc4c223 100755
--- a/tests/test-cow.sh
+++ b/tests/test-cow.sh
@@ -72,8 +72,8 @@ fi
# If we have qemu-img, try the hairy rebase operation documented
# in the nbdkit-cow-filter manual.
if qemu-img --version >/dev/null 2>&1; then
- qemu-img create -f qcow2 -b nbd:unix:$sock cow-diff.qcow2
- time qemu-img rebase -b cow-base.img cow-diff.qcow2
+ qemu-img create -F raw -b nbd:unix:$sock -f qcow2 cow-diff.qcow2
+ time qemu-img rebase -F raw -b cow-base.img -f qcow2 cow-diff.qcow2
qemu-img info cow-diff.qcow2
# This checks the file we created exists.
--
2.31.1

65
SOURCES/0004-python-Add-various-constants-to-the-API.patch
View File

@ -1,65 +0,0 @@
From b646050b8da51c39cf21f95fa847c12784a1169c Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 15:02:44 +0000
Subject: [PATCH 04/19] python: Add various constants to the API.
These are accessible from the plugin by:
import nbdkit
if flags & nbdkit.FLAG_MAY_TRIM:
&c.
Many (all?) of these are not yet useful for plugins, some will never
be useful, but they only consume a tiny bit of memory and it's nice to
have the complete set available for future use.
(cherry picked from commit 14b7fe2e0de881e3dfc8803484ade29a61e323c9)
---
plugins/python/python.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 7052aac0..47da0838 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -231,6 +231,36 @@ create_nbdkit_module (void)
nbdkit_error ("could not create the nbdkit API module");
exit (EXIT_FAILURE);
}
+
+ /* Constants corresponding to various flags. */
+#define ADD_INT_CONSTANT(name) \
+ if (PyModule_AddIntConstant (m, #name, NBDKIT_##name) == -1) { \
+ nbdkit_error ("could not add constant %s to nbdkit API module", \
+ #name); \
+ exit (EXIT_FAILURE); \
+ }
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_CONNECTIONS);
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_ALL_REQUESTS);
+ ADD_INT_CONSTANT (THREAD_MODEL_SERIALIZE_REQUESTS);
+ ADD_INT_CONSTANT (THREAD_MODEL_PARALLEL);
+
+ ADD_INT_CONSTANT (FLAG_MAY_TRIM);
+ ADD_INT_CONSTANT (FLAG_FUA);
+ ADD_INT_CONSTANT (FLAG_REQ_ONE);
+ ADD_INT_CONSTANT (FLAG_FAST_ZERO);
+
+ ADD_INT_CONSTANT (FUA_NONE);
+ ADD_INT_CONSTANT (FUA_EMULATE);
+ ADD_INT_CONSTANT (FUA_NATIVE);
+
+ ADD_INT_CONSTANT (CACHE_NONE);
+ ADD_INT_CONSTANT (CACHE_EMULATE);
+ ADD_INT_CONSTANT (CACHE_NATIVE);
+
+ ADD_INT_CONSTANT (EXTENT_HOLE);
+ ADD_INT_CONSTANT (EXTENT_ZERO);
+#undef ADD_INT_CONSTANT
+
return m;
}
--
2.18.2

558
SOURCES/0005-python-Implement-nbdkit-API-version-2.patch
View File

@ -1,558 +0,0 @@
From 49ef7e7d7c3602cc8e53d2052fce9d3a12840ea2 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 15:44:39 +0000
Subject: [PATCH 05/19] python: Implement nbdkit API version 2.
To avoid breaking existing plugins, Python plugins wishing to use
version 2 of the API must opt in by declaring:
API_VERSION = 2
(Plugins which do not do this are assumed to want API version 1).
For v2 API, we also avoid a copy by passing a buffer into pread.
It's more efficient if we pass the C buffer directly to Python code.
In some cases the Python code will be able to write directly into the
C buffer using functions like file.readinto and socket.recv_into.
This avoids an extra copy.
Thanks: Nir Soffer
https://www.redhat.com/archives/libguestfs/2019-November/thread.html#00220
(cherry picked from commit a9b2637cf4f00fb8a25ffaf31ee83be5fe019ae2)
---
plugins/python/example.py | 20 +++-
plugins/python/nbdkit-python-plugin.pod | 69 +++++++-----
plugins/python/python.c | 139 +++++++++++++++++++-----
tests/python-exception.py | 4 +-
tests/shebang.py | 5 +-
tests/test.py | 28 +++--
6 files changed, 190 insertions(+), 75 deletions(-)
diff --git a/plugins/python/example.py b/plugins/python/example.py
index 60f9d7f0..c04b7e29 100644
--- a/plugins/python/example.py
+++ b/plugins/python/example.py
@@ -34,6 +34,12 @@ import errno
disk = bytearray(1024 * 1024)
+# There are several variants of the API. nbdkit will call this
+# function first to determine which one you want to use. This is the
+# latest version at the time this example was written.
+API_VERSION = 2
+
+
# This just prints the extra command line parameters, but real plugins
# should parse them and reject any unknown parameters.
def config(key, value):
@@ -54,20 +60,22 @@ def get_size(h):
return len(disk)
-def pread(h, count, offset):
+def pread(h, buf, offset, flags):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
+ # or if reading from a file you can use:
+ #f.readinto(buf)
-
-def pwrite(h, buf, offset):
+def pwrite(h, buf, offset, flags):
global disk
end = offset + len(buf)
disk[offset:end] = buf
-def zero(h, count, offset, may_trim):
+def zero(h, count, offset, flags):
global disk
- if may_trim:
+ if flags & nbdkit.FLAG_MAY_TRIM:
disk[offset:offset+count] = bytearray(count)
else:
nbdkit.set_error(errno.EOPNOTSUPP)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 3680fd65..4923d9da 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -33,11 +33,12 @@ To write a Python nbdkit plugin, you create a Python file which
contains at least the following required functions (in the top level
C<__main__> module):
+ API_VERSION = 2
def open(readonly):
# see below
def get_size(h):
# see below
- def pread(h, count, offset):
+ def pread(h, buf, offset, flags):
# see below
Note that the subroutines must have those literal names (like C<open>),
@@ -82,6 +83,18 @@ I<--dump-plugin> option, eg:
python_version=3.7.0
python_pep_384_abi_version=3
+=head2 API versions
+
+The nbdkit API has evolved and new versions are released periodically.
+To ensure backwards compatibility plugins have to opt in to the new
+version. From Python you do this by declaring a constant in your
+module:
+
+ API_VERSION = 2
+
+(where 2 is the latest version at the time this documentation was
+written). All newly written Python modules must have this constant.
+
=head2 Executable script
If you want you can make the script executable and include a "shebang"
@@ -199,16 +212,12 @@ contents will be garbage collected.
(Required)
- def pread(h, count, offset):
- # construct a buffer of length count bytes and return it
+ def pread(h, buf, offset, flags):
+ # read into the buffer
-The body of your C<pread> function should construct a buffer of length
-(at least) C<count> bytes. You should read C<count> bytes from the
-disk starting at C<offset>.
-
-The returned buffer can be any type compatible with the Python 3
-buffer protocol, such as bytearray, bytes or memoryview
-(L<https://docs.python.org/3/c-api/buffer.html>)
+The body of your C<pread> function should read exactly C<len(buf)>
+bytes of data starting at disk C<offset> and write it into the buffer
+C<buf>. C<flags> is always 0.
NBD only supports whole reads, so your function should try to read
the whole region (perhaps requiring a loop). If the read fails or
@@ -219,13 +228,13 @@ C<nbdkit.set_error> first.
(Optional)
- def pwrite(h, buf, offset):
+ def pwrite(h, buf, offset, flags):
length = len (buf)
# no return value
The body of your C<pwrite> function should write the buffer C<buf> to
the disk. You should write C<count> bytes to the disk starting at
-C<offset>.
+C<offset>. C<flags> may contain C<nbdkit.FLAG_FUA>.
NBD only supports whole writes, so your function should try to
write the whole region (perhaps requiring a loop). If the write
@@ -236,11 +245,12 @@ fails or is partial, your function should throw an exception,
(Optional)
- def flush(h):
+ def flush(h, flags):
# no return value
The body of your C<flush> function should do a L<sync(2)> or
L<fdatasync(2)> or equivalent on the backing store.
+C<flags> is always 0.
If the flush fails, your function should throw an exception, optionally
using C<nbdkit.set_error> first.
@@ -249,32 +259,35 @@ using C<nbdkit.set_error> first.
(Optional)
- def trim(h, count, offset):
+ def trim(h, count, offset, flags):
# no return value
-The body of your C<trim> function should "punch a hole" in the
-backing store. If the trim fails, your function should throw an
-exception, optionally using C<nbdkit.set_error> first.
+The body of your C<trim> function should "punch a hole" in the backing
+store. C<flags> may contain C<nbdkit.FLAG_FUA>. If the trim fails,
+your function should throw an exception, optionally using
+C<nbdkit.set_error> first.
=item C<zero>
(Optional)
- def zero(h, count, offset, may_trim):
+ def zero(h, count, offset, flags):
# no return value
-The body of your C<zero> function should ensure that C<count> bytes
-of the disk, starting at C<offset>, will read back as zero. If
-C<may_trim> is true, the operation may be optimized as a trim as long
-as subsequent reads see zeroes.
+The body of your C<zero> function should ensure that C<count> bytes of
+the disk, starting at C<offset>, will read back as zero. C<flags> is
+a bitmask which may include C<nbdkit.FLAG_MAY_TRIM>,
+C<nbdkit.FLAG_FUA>, C<nbdkit.FLAG_FAST_ZERO>.
NBD only supports whole writes, so your function should try to
-write the whole region (perhaps requiring a loop). If the write
-fails or is partial, your function should throw an exception,
-optionally using C<nbdkit.set_error> first. In particular, if
-you would like to automatically fall back to C<pwrite> (perhaps
-because there is nothing to optimize if C<may_trim> is false),
-use C<nbdkit.set_error(errno.EOPNOTSUPP)>.
+write the whole region (perhaps requiring a loop).
+
+If the write fails or is partial, your function should throw an
+exception, optionally using C<nbdkit.set_error> first. In particular,
+if you would like to automatically fall back to C<pwrite> (perhaps
+because there is nothing to optimize if
+S<C<flags & nbdkit.FLAG_MAY_TRIM>> is false), use
+S<C<nbdkit.set_error (errno.EOPNOTSUPP)>>.
=back
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 47da0838..0f28595f 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -46,6 +46,8 @@
#define PY_SSIZE_T_CLEAN 1
#include <Python.h>
+#define NBDKIT_API_VERSION 2
+
#include <nbdkit-plugin.h>
#include "cleanup.h"
@@ -60,6 +62,7 @@
*/
static const char *script;
static PyObject *module;
+static int py_api_version = 1;
static int last_error;
@@ -285,9 +288,14 @@ py_dump_plugin (void)
PyObject *fn;
PyObject *r;
+ /* Python version and ABI. */
printf ("python_version=%s\n", PY_VERSION);
printf ("python_pep_384_abi_version=%d\n", PYTHON_ABI_VERSION);
+ /* Maximum nbdkit API version supported. */
+ printf ("nbdkit_python_maximum_api_version=%d\n", NBDKIT_API_VERSION);
+
+ /* If the script has a dump_plugin function, call it. */
if (script && callback_defined ("dump_plugin", &fn)) {
PyErr_Clear ();
@@ -297,6 +305,30 @@ py_dump_plugin (void)
}
}
+static int
+get_py_api_version (void)
+{
+ PyObject *obj;
+ long value;
+
+ obj = PyObject_GetAttrString (module, "API_VERSION");
+ if (obj == NULL)
+ return 1; /* Default to API version 1. */
+
+ value = PyLong_AsLong (obj);
+ Py_DECREF (obj);
+
+ if (value < 1 || value > NBDKIT_API_VERSION) {
+ nbdkit_error ("%s: API_VERSION requested unknown version: %ld. "
+ "This plugin supports API versions between 1 and %d.",
+ script, value, NBDKIT_API_VERSION);
+ return -1;
+ }
+
+ nbdkit_debug ("module requested API_VERSION %ld", value);
+ return (int) value;
+}
+
static int
py_config (const char *key, const char *value)
{
@@ -359,6 +391,11 @@ py_config (const char *key, const char *value)
"nbdkit requires these callbacks.", script);
return -1;
}
+
+ /* Get the API version. */
+ py_api_version = get_py_api_version ();
+ if (py_api_version == -1)
+ return -1;
}
else if (callback_defined ("config", &fn)) {
/* Other parameters are passed to the Python .config callback. */
@@ -469,8 +506,8 @@ py_get_size (void *handle)
}
static int
-py_pread (void *handle, void *buf,
- uint32_t count, uint64_t offset)
+py_pread (void *handle, void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -485,24 +522,40 @@ py_pread (void *handle, void *buf,
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "ONLI", obj,
+ PyMemoryView_FromMemory ((char *)buf, count, PyBUF_WRITE),
+ offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("pread") == -1)
return ret;
- if (PyObject_GetBuffer (r, &view, PyBUF_SIMPLE) == -1) {
- nbdkit_error ("%s: value returned from pread does not support the "
- "buffer protocol",
- script);
- goto out;
- }
+ if (py_api_version == 1) {
+ /* In API v1 the Python pread function had to return a buffer
+ * protocol compatible function. In API v2+ it writes directly to
+ * the C buffer so this code is not used.
+ */
+ if (PyObject_GetBuffer (r, &view, PyBUF_SIMPLE) == -1) {
+ nbdkit_error ("%s: value returned from pread does not support the "
+ "buffer protocol",
+ script);
+ goto out;
+ }
- if (view.len < count) {
- nbdkit_error ("%s: buffer returned from pread is too small", script);
- goto out;
- }
+ if (view.len < count) {
+ nbdkit_error ("%s: buffer returned from pread is too small", script);
+ goto out;
+ }
- memcpy (buf, view.buf, count);
+ memcpy (buf, view.buf, count);
+ }
ret = 0;
out:
@@ -515,8 +568,8 @@ out:
}
static int
-py_pwrite (void *handle, const void *buf,
- uint32_t count, uint64_t offset)
+py_pwrite (void *handle, const void *buf, uint32_t count, uint64_t offset,
+ uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -525,9 +578,19 @@ py_pwrite (void *handle, const void *buf,
if (callback_defined ("pwrite", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "ONL", obj,
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "ONL", obj,
PyMemoryView_FromMemory ((char *)buf, count, PyBUF_READ),
offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "ONLI", obj,
+ PyMemoryView_FromMemory ((char *)buf, count, PyBUF_READ),
+ offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("pwrite") == -1)
return -1;
@@ -542,7 +605,7 @@ py_pwrite (void *handle, const void *buf,
}
static int
-py_flush (void *handle)
+py_flush (void *handle, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -551,7 +614,15 @@ py_flush (void *handle)
if (callback_defined ("flush", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "OI", obj, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("flush") == -1)
return -1;
@@ -566,7 +637,7 @@ py_flush (void *handle)
}
static int
-py_trim (void *handle, uint32_t count, uint64_t offset)
+py_trim (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -575,7 +646,15 @@ py_trim (void *handle, uint32_t count, uint64_t offset)
if (callback_defined ("trim", &fn)) {
PyErr_Clear ();
- r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ switch (py_api_version) {
+ case 1:
+ r = PyObject_CallFunction (fn, "OiL", obj, count, offset);
+ break;
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (check_python_failure ("trim") == -1)
return -1;
@@ -590,7 +669,7 @@ py_trim (void *handle, uint32_t count, uint64_t offset)
}
static int
-py_zero (void *handle, uint32_t count, uint64_t offset, int may_trim)
+py_zero (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
{
PyObject *obj = handle;
PyObject *fn;
@@ -600,9 +679,19 @@ py_zero (void *handle, uint32_t count, uint64_t offset, int may_trim)
PyErr_Clear ();
last_error = 0;
- r = PyObject_CallFunction (fn, "OiLO",
- obj, count, offset,
- may_trim ? Py_True : Py_False);
+ switch (py_api_version) {
+ case 1: {
+ int may_trim = flags & NBDKIT_FLAG_MAY_TRIM;
+ r = PyObject_CallFunction (fn, "OiLO",
+ obj, count, offset,
+ may_trim ? Py_True : Py_False);
+ break;
+ }
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags);
+ break;
+ default: abort ();
+ }
Py_DECREF (fn);
if (last_error == EOPNOTSUPP || last_error == ENOTSUP) {
/* When user requests this particular error, we want to
diff --git a/tests/python-exception.py b/tests/python-exception.py
index d0c79bb0..ee4a3f3a 100644
--- a/tests/python-exception.py
+++ b/tests/python-exception.py
@@ -62,5 +62,5 @@ def get_size(h):
return 0
-def pread(h, count, offset):
- return ""
+def pread(h, buf, offset):
+ buf[:] = bytearray(len(buf))
diff --git a/tests/shebang.py b/tests/shebang.py
index 6f336230..0634589a 100755
--- a/tests/shebang.py
+++ b/tests/shebang.py
@@ -13,6 +13,7 @@ def get_size(h):
return len(disk)
-def pread(h, count, offset):
+def pread(h, buf, offset):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
diff --git a/tests/test.py b/tests/test.py
index 9a2e947d..4db56623 100644
--- a/tests/test.py
+++ b/tests/test.py
@@ -3,6 +3,9 @@ import nbdkit
disk = bytearray(1024*1024)
+API_VERSION = 2
+
+
def config_complete():
print ("set_error = %r" % nbdkit.set_error)
@@ -32,25 +35,26 @@ def can_trim(h):
return True
-def pread(h, count, offset):
+def pread(h, buf, offset, flags):
global disk
- return disk[offset:offset+count]
+ end = offset + len(buf)
+ buf[:] = disk[offset:end]
-def pwrite(h, buf, offset):
+def pwrite(h, buf, offset, flags):
global disk
end = offset + len(buf)
disk[offset:end] = buf
-def zero(h, count, offset, may_trim=False):
+def flush(h, flags):
+ pass
+
+
+def trim(h, count, offset, flags):
+ pass
+
+
+def zero(h, count, offset, flags):
global disk
disk[offset:offset+count] = bytearray(count)
-
-
-def flush(h):
- pass
-
-
-def trim(h, count, offset):
- pass
--
2.18.2

98
SOURCES/0006-python-Implement-cache.patch
View File

@ -1,98 +0,0 @@
From c5b1fac4c67078f0164bd23eab6d4d2b8c9830b0 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:42:02 +0000
Subject: [PATCH 06/19] python: Implement cache.
However this does not implement can_cache, since that is not a simple
boolean.
(cherry picked from commit e61ffb73c7a0af0c383184fdb8f08d30784a195e)
---
plugins/python/nbdkit-python-plugin.pod | 14 ++++++++++-
plugins/python/python.c | 31 +++++++++++++++++++++++++
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 4923d9da..0ea8deef 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -289,6 +289,19 @@ because there is nothing to optimize if
S<C<flags & nbdkit.FLAG_MAY_TRIM>> is false), use
S<C<nbdkit.set_error (errno.EOPNOTSUPP)>>.
+=item C<cache>
+
+(Optional)
+
+ def cache(h, count, offset, flags):
+ # no return value
+
+The body of your C<cache> function should prefetch data in the
+indicated range.
+
+If the cache operation fails, your function should throw an exception,
+optionally using C<nbdkit.set_error> first.
+
=back
=head2 Missing callbacks
@@ -317,7 +330,6 @@ C<can_zero>,
C<can_fast_zero>,
C<can_extents>,
C<can_multi_conn>,
-C<cache>,
C<extents>.
These are not yet supported.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 0f28595f..c5cf38e5 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -714,6 +714,36 @@ py_zero (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
return -1;
}
+static int
+py_cache (void *handle, uint32_t count, uint64_t offset, uint32_t flags)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+
+ if (callback_defined ("cache", &fn)) {
+ PyErr_Clear ();
+
+ switch (py_api_version) {
+ case 1:
+ case 2:
+ r = PyObject_CallFunction (fn, "OiLI", obj, count, offset, flags, NULL);
+ break;
+ default: abort ();
+ }
+ Py_DECREF (fn);
+ if (check_python_failure ("cache") == -1)
+ return -1;
+ Py_DECREF (r);
+ }
+ else {
+ nbdkit_error ("%s not implemented", "cache");
+ return -1;
+ }
+
+ return 0;
+}
+
static int
boolean_callback (void *handle, const char *can_fn, const char *plain_fn)
{
@@ -799,6 +829,7 @@ static struct nbdkit_plugin plugin = {
.flush = py_flush,
.trim = py_trim,
.zero = py_zero,
+ .cache = py_cache,
};
NBDKIT_REGISTER_PLUGIN (plugin)
--
2.18.2

80
SOURCES/0007-python-Implement-can_zero-can_fast_zero.patch
View File

@ -1,80 +0,0 @@
From 17721b316dd66b0a1ed792eeccd2489fb97828df Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:42:59 +0000
Subject: [PATCH 07/19] python: Implement can_zero, can_fast_zero.
(cherry picked from commit 039f600d2ad7a9ff04523a165eb2fe41b9c87c01)
---
plugins/python/nbdkit-python-plugin.pod | 16 ++++++++++++++--
plugins/python/python.c | 14 ++++++++++++++
2 files changed, 28 insertions(+), 2 deletions(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 0ea8deef..1f1c30f6 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -208,6 +208,20 @@ contents will be garbage collected.
def can_trim(h):
# return a boolean
+=item C<can_zero>
+
+(Optional)
+
+ def can_zero(h):
+ # return a boolean
+
+=item C<can_fast_zero>
+
+(Optional)
+
+ def can_fast_zero(h):
+ # return a boolean
+
=item C<pread>
(Required)
@@ -326,8 +340,6 @@ C<config_help>,
C<magic_config_key>,
C<can_fua>,
C<can_cache>,
-C<can_zero>,
-C<can_fast_zero>,
C<can_extents>,
C<can_multi_conn>,
C<extents>.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index c5cf38e5..38fc1193 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -797,6 +797,18 @@ py_can_trim (void *handle)
return boolean_callback (handle, "can_trim", "trim");
}
+static int
+py_can_zero (void *handle)
+{
+ return boolean_callback (handle, "can_zero", "zero");
+}
+
+static int
+py_can_fast_zero (void *handle)
+{
+ return boolean_callback (handle, "can_fast_zero", NULL);
+}
+
#define py_config_help \
"script=<FILENAME> (required) The Python plugin to run.\n" \
"[other arguments may be used by the plugin that you load]"
@@ -823,6 +835,8 @@ static struct nbdkit_plugin plugin = {
.can_write = py_can_write,
.can_flush = py_can_flush,
.can_trim = py_can_trim,
+ .can_zero = py_can_zero,
+ .can_fast_zero = py_can_fast_zero,
.pread = py_pread,
.pwrite = py_pwrite,
--
2.18.2

65
SOURCES/0008-python-Implement-can_multi_conn.patch
View File

@ -1,65 +0,0 @@
From 2a85ce81ad95eb2f9b2f29666480b814ea0f80d9 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 16:46:11 +0000
Subject: [PATCH 08/19] python: Implement can_multi_conn.
(cherry picked from commit 21dd7bf49d3238c7e75918d4bf324b617f458d83)
---
plugins/python/nbdkit-python-plugin.pod | 8 +++++++-
plugins/python/python.c | 7 +++++++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index 1f1c30f6..b92bb56a 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -187,6 +187,13 @@ contents will be garbage collected.
def is_rotational(h):
# return a boolean
+=item C<can_multi_conn>
+
+(Optional)
+
+ def can_multi_conn(h):
+ # return a boolean
+
=item C<can_write>
(Optional)
@@ -341,7 +348,6 @@ C<magic_config_key>,
C<can_fua>,
C<can_cache>,
C<can_extents>,
-C<can_multi_conn>,
C<extents>.
These are not yet supported.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index 38fc1193..b186b991 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -779,6 +779,12 @@ py_is_rotational (void *handle)
return boolean_callback (handle, "is_rotational", NULL);
}
+static int
+py_can_multi_conn (void *handle)
+{
+ return boolean_callback (handle, "can_multi_conn", NULL);
+}
+
static int
py_can_write (void *handle)
{
@@ -832,6 +838,7 @@ static struct nbdkit_plugin plugin = {
.get_size = py_get_size,
.is_rotational = py_is_rotational,
+ .can_multi_conn = py_can_multi_conn,
.can_write = py_can_write,
.can_flush = py_can_flush,
.can_trim = py_can_trim,
--
2.18.2

126
SOURCES/0009-python-Implement-can_fua-and-can_cache.patch
View File

@ -1,126 +0,0 @@
From 38124a137974e1433d68732640ca7f88664557da Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Fri, 22 Nov 2019 19:25:53 +0000
Subject: [PATCH 09/19] python: Implement can_fua and can_cache.
(cherry picked from commit 97c46f885edec5a61a96ac86eccb9d8c874c602e)
---
plugins/python/nbdkit-python-plugin.pod | 18 +++++++-
plugins/python/python.c | 58 +++++++++++++++++++++++++
2 files changed, 74 insertions(+), 2 deletions(-)
diff --git a/plugins/python/nbdkit-python-plugin.pod b/plugins/python/nbdkit-python-plugin.pod
index b92bb56a..4065ec75 100644
--- a/plugins/python/nbdkit-python-plugin.pod
+++ b/plugins/python/nbdkit-python-plugin.pod
@@ -229,6 +229,22 @@ contents will be garbage collected.
def can_fast_zero(h):
# return a boolean
+=item C<can_fua>
+
+(Optional)
+
+ def can_fua(h):
+ # return nbdkit.FUA_NONE or nbdkit.FUA_EMULATE
+ # or nbdkit.FUA_NATIVE
+
+=item C<can_cache>
+
+(Optional)
+
+ def can_cache(h):
+ # return nbdkit.CACHE_NONE or nbdkit.CACHE_EMULATE
+ # or nbdkit.CACHE_NATIVE
+
=item C<pread>
(Required)
@@ -345,8 +361,6 @@ C<longname>,
C<description>,
C<config_help>,
C<magic_config_key>,
-C<can_fua>,
-C<can_cache>,
C<can_extents>,
C<extents>.
diff --git a/plugins/python/python.c b/plugins/python/python.c
index b186b991..5e2e5269 100644
--- a/plugins/python/python.c
+++ b/plugins/python/python.c
@@ -815,6 +815,62 @@ py_can_fast_zero (void *handle)
return boolean_callback (handle, "can_fast_zero", NULL);
}
+static int
+py_can_fua (void *handle)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+ int ret;
+
+ if (callback_defined ("can_fua", &fn)) {
+ PyErr_Clear ();
+
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ Py_DECREF (fn);
+ if (check_python_failure ("can_fua") == -1)
+ return -1;
+ ret = PyLong_AsLong (r);
+ Py_DECREF (r);
+ return ret;
+ }
+ /* No Python ‘can_fua’, but check if there's a Python ‘flush’
+ * callback defined. (In C modules, nbdkit would do this).
+ */
+ else if (callback_defined ("flush", NULL))
+ return NBDKIT_FUA_EMULATE;
+ else
+ return NBDKIT_FUA_NONE;
+}
+
+static int
+py_can_cache (void *handle)
+{
+ PyObject *obj = handle;
+ PyObject *fn;
+ PyObject *r;
+ int ret;
+
+ if (callback_defined ("can_cache", &fn)) {
+ PyErr_Clear ();
+
+ r = PyObject_CallFunctionObjArgs (fn, obj, NULL);
+ Py_DECREF (fn);
+ if (check_python_failure ("can_cache") == -1)
+ return -1;
+ ret = PyLong_AsLong (r);
+ Py_DECREF (r);
+ return ret;
+ }
+ /* No Python ‘can_cache’, but check if there's a Python ‘cache’
+ * callback defined. (In C modules, nbdkit would do this).
+ */
+ else if (callback_defined ("cache", NULL))
+ return NBDKIT_CACHE_NATIVE;
+ else
+ return NBDKIT_CACHE_NONE;
+}
+
#define py_config_help \
"script=<FILENAME> (required) The Python plugin to run.\n" \
"[other arguments may be used by the plugin that you load]"
@@ -844,6 +900,8 @@ static struct nbdkit_plugin plugin = {
.can_trim = py_can_trim,
.can_zero = py_can_zero,
.can_fast_zero = py_can_fast_zero,
+ .can_fua = py_can_fua,
+ .can_cache = py_can_cache,
.pread = py_pread,
.pwrite = py_pwrite,
--
2.18.2

597
SOURCES/0010-tests-Test-the-Python-plugin-thoroughly.patch
View File

@ -1,597 +0,0 @@
From 7cb79aef2a12f29f1286caf3858001e47214f871 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 21 Nov 2019 20:54:41 +0000
Subject: [PATCH 10/19] tests: Test the Python plugin thoroughly.
This tests the Python plugin thoroughly by issuing client commands
through libnbd and checking we get the expected results.
(cherry picked from commit 8ead4a82ec3227dbecb6cbfc419f1a18f2817d62)
---
.gitignore | 1 +
README | 2 +
tests/Makefile.am | 15 +--
tests/test-lang-plugins.c | 3 +-
tests/test-python-plugin.py | 133 +++++++++++++++++++++
tests/test-python.sh | 49 ++++++++
tests/test.py | 60 ----------
tests/test_python.py | 222 ++++++++++++++++++++++++++++++++++++
8 files changed, 413 insertions(+), 72 deletions(-)
create mode 100644 tests/test-python-plugin.py
create mode 100755 tests/test-python.sh
delete mode 100644 tests/test.py
create mode 100755 tests/test_python.py
diff --git a/.gitignore b/.gitignore
index b25ac7fe..e25bd99b 100644
--- a/.gitignore
+++ b/.gitignore
@@ -71,6 +71,7 @@ Makefile.in
/server/synopsis.c
/server/test-public
/stamp-h1
+/tests/__pycache__/
/tests/disk
/tests/disk.gz
/tests/disk.xz
diff --git a/README b/README
index 40f4cd37..05f1e060 100644
--- a/README
+++ b/README
@@ -130,6 +130,8 @@ For the Python plugin:
- python development libraries
+ - python unittest to run the test suite
+
For the OCaml plugin:
- OCaml >= 4.02.2
diff --git a/tests/Makefile.am b/tests/Makefile.am
index d225cc63..09103fbb 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -67,6 +67,7 @@ EXTRA_PROGRAMS =
TESTS_ENVIRONMENT = \
PATH=$(abs_top_builddir):$(PATH) \
SRCDIR=$(srcdir) \
+ PYTHON=$(PYTHON) \
LIBGUESTFS_ATTACH_METHOD=appliance \
LIBGUESTFS_DEBUG=1 \
LIBGUESTFS_TRACE=1 \
@@ -160,7 +161,9 @@ EXTRA_DIST = \
test-probe-plugin.sh \
test-python-exception.sh \
test.pl \
- test.py \
+ test_python.py \
+ test-python-plugin.py \
+ test-python.sh \
test-rate.sh \
test-rate-dynamic.sh \
test.rb \
@@ -801,18 +804,10 @@ endif HAVE_PERL
if HAVE_PYTHON
TESTS += \
+ test-python.sh \
test-python-exception.sh \
test-shebang-python.sh \
$(NULL)
-LIBGUESTFS_TESTS += test-python
-
-test_python_SOURCES = test-lang-plugins.c test.h
-test_python_CFLAGS = \
- -DLANG='"python"' -DSCRIPT='"$(srcdir)/test.py"' \
- $(WARNINGS_CFLAGS) \
- $(LIBGUESTFS_CFLAGS) \
- $(NULL)
-test_python_LDADD = libtest.la $(LIBGUESTFS_LIBS)
endif HAVE_PYTHON
diff --git a/tests/test-lang-plugins.c b/tests/test-lang-plugins.c
index ffb19180..93f99381 100644
--- a/tests/test-lang-plugins.c
+++ b/tests/test-lang-plugins.c
@@ -56,8 +56,7 @@ main (int argc, char *argv[])
*/
s = getenv ("NBDKIT_VALGRIND");
if (s && strcmp (s, "1") == 0 &&
- (strcmp (LANG, "python") == 0 ||
- strcmp (LANG, "ruby") == 0 ||
+ (strcmp (LANG, "ruby") == 0 ||
strcmp (LANG, "tcl") == 0)) {
fprintf (stderr, "%s test skipped under valgrind.\n", LANG);
exit (77); /* Tells automake to skip the test. */
diff --git a/tests/test-python-plugin.py b/tests/test-python-plugin.py
new file mode 100644
index 00000000..8e90bc23
--- /dev/null
+++ b/tests/test-python-plugin.py
@@ -0,0 +1,133 @@
+# nbdkit test plugin
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+"""See test-python.py."""
+
+import nbdkit
+import sys
+import pickle
+import base64
+
+API_VERSION = 2
+
+cfg = {}
+
+def config (k, v):
+ global cfg
+ if k == "cfg":
+ cfg = pickle.loads (base64.b64decode (v.encode()))
+
+def config_complete ():
+ print ("set_error = %r" % nbdkit.set_error)
+
+def open (readonly):
+ return {
+ 'disk': bytearray (cfg.get ('size', 0))
+ }
+
+def get_size (h):
+ return len (h['disk'])
+
+def is_rotational (h):
+ return cfg.get ('is_rotational', False)
+
+def can_multi_conn (h):
+ return cfg.get ('can_multi_conn', False)
+
+def can_write (h):
+ return cfg.get ('can_write', True)
+
+def can_flush (h):
+ return cfg.get ('can_flush', False)
+
+def can_trim (h):
+ return cfg.get ('can_trim', False)
+
+def can_zero (h):
+ return cfg.get ('can_zero', False)
+
+def can_fast_zero (h):
+ return cfg.get ('can_fast_zero', False)
+
+def can_fua (h):
+ fua = cfg.get ('can_fua', "none")
+ if fua == "none":
+ return nbdkit.FUA_NONE
+ elif fua == "emulate":
+ return nbdkit.FUA_EMULATE
+ elif fua == "native":
+ return nbdkit.FUA_NATIVE
+
+def can_cache (h):
+ cache = cfg.get ('can_cache', "none")
+ if cache == "none":
+ return nbdkit.CACHE_NONE
+ elif cache == "emulate":
+ return nbdkit.CACHE_EMULATE
+ elif cache == "native":
+ return nbdkit.CACHE_NATIVE
+
+def pread (h, buf, offset, flags):
+ assert flags == 0
+ end = offset + len(buf)
+ buf[:] = h['disk'][offset:end]
+
+def pwrite (h, buf, offset, flags):
+ expect_fua = cfg.get ('pwrite_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ end = offset + len(buf)
+ h['disk'][offset:end] = buf
+
+def flush (h, flags):
+ assert flags == 0
+
+def trim (h, count, offset, flags):
+ expect_fua = cfg.get ('trim_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ h['disk'][offset:offset+count] = bytearray(count)
+
+def zero (h, count, offset, flags):
+ expect_fua = cfg.get ('zero_expect_fua', False)
+ actual_fua = bool (flags & nbdkit.FLAG_FUA)
+ assert expect_fua == actual_fua
+ expect_may_trim = cfg.get ('zero_expect_may_trim', False)
+ actual_may_trim = bool (flags & nbdkit.FLAG_MAY_TRIM)
+ assert expect_may_trim == actual_may_trim
+ expect_fast_zero = cfg.get ('zero_expect_fast_zero', False)
+ actual_fast_zero = bool (flags & nbdkit.FLAG_FAST_ZERO)
+ assert expect_fast_zero == actual_fast_zero
+ h['disk'][offset:offset+count] = bytearray(count)
+
+def cache (h, count, offset, flags):
+ assert flags == 0
+ # do nothing
diff --git a/tests/test-python.sh b/tests/test-python.sh
new file mode 100755
index 00000000..50324d0f
--- /dev/null
+++ b/tests/test-python.sh
@@ -0,0 +1,49 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+source ./functions.sh
+set -e
+set -x
+
+requires $PYTHON --version
+requires $PYTHON -c 'import unittest'
+requires $PYTHON -c 'import nbd'
+requires test -f test_python.py
+requires test -f test-python-plugin.py
+
+# Python has proven very difficult to valgrind, therefore it is disabled.
+if [ "$NBDKIT_VALGRIND" = "1" ]; then
+ echo "$0: skipping Python test under valgrind."
+ exit 77
+fi
+
+$PYTHON -m unittest test_python
diff --git a/tests/test.py b/tests/test.py
deleted file mode 100644
index 4db56623..00000000
--- a/tests/test.py
+++ /dev/null
@@ -1,60 +0,0 @@
-import nbdkit
-
-disk = bytearray(1024*1024)
-
-
-API_VERSION = 2
-
-
-def config_complete():
- print ("set_error = %r" % nbdkit.set_error)
-
-
-def open(readonly):
- return 1
-
-
-def get_size(h):
- global disk
- return len(disk)
-
-
-def can_write(h):
- return True
-
-
-def can_flush(h):
- return True
-
-
-def is_rotational(h):
- return False
-
-
-def can_trim(h):
- return True
-
-
-def pread(h, buf, offset, flags):
- global disk
- end = offset + len(buf)
- buf[:] = disk[offset:end]
-
-
-def pwrite(h, buf, offset, flags):
- global disk
- end = offset + len(buf)
- disk[offset:end] = buf
-
-
-def flush(h, flags):
- pass
-
-
-def trim(h, count, offset, flags):
- pass
-
-
-def zero(h, count, offset, flags):
- global disk
- disk[offset:offset+count] = bytearray(count)
diff --git a/tests/test_python.py b/tests/test_python.py
new file mode 100755
index 00000000..6b9f2979
--- /dev/null
+++ b/tests/test_python.py
@@ -0,0 +1,222 @@
+#!/usr/bin/env python3
+# nbdkit
+# Copyright (C) 2019 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+"""
+This tests the Python plugin thoroughly by issuing client commands
+through libnbd and checking we get the expected results. It uses an
+associated plugin (test-python-plugin.sh).
+"""
+
+import os
+import sys
+import nbd
+import unittest
+import pickle
+import base64
+
+class Test (unittest.TestCase):
+ def setUp (self):
+ self.h = nbd.NBD ()
+
+ def tearDown (self):
+ del self.h
+
+ def connect (self, cfg):
+ cfg = base64.b64encode (pickle.dumps (cfg)).decode()
+ cmd = ["nbdkit", "-v", "-s", "--exit-with-parent",
+ "python", "test-python-plugin.py", "cfg=" + cfg]
+ self.h.connect_command (cmd)
+
+ def test_none (self):
+ """
+ Test we can send an empty pickled test configuration and do
+ nothing else. This is just to ensure the machinery of the
+ test works.
+ """
+ self.connect ({})
+
+ def test_size_512 (self):
+ """Test the size."""
+ self.connect ({"size": 512})
+ assert self.h.get_size() == 512
+
+ def test_size_1m (self):
+ """Test the size."""
+ self.connect ({"size": 1024*1024})
+ assert self.h.get_size() == 1024*1024
+
+ # Test each flag call.
+ def test_is_rotational_true (self):
+ self.connect ({"size": 512, "is_rotational": True})
+ assert self.h.is_rotational()
+
+ def test_is_rotational_false (self):
+ self.connect ({"size": 512, "is_rotational": False})
+ assert not self.h.is_rotational()
+
+ def test_can_multi_conn_true (self):
+ self.connect ({"size": 512, "can_multi_conn": True})
+ assert self.h.can_multi_conn()
+
+ def test_can_multi_conn_false (self):
+ self.connect ({"size": 512, "can_multi_conn": False})
+ assert not self.h.can_multi_conn()
+
+ def test_read_write (self):
+ self.connect ({"size": 512, "can_write": True})
+ assert not self.h.is_read_only()
+
+ def test_read_only (self):
+ self.connect ({"size": 512, "can_write": False})
+ assert self.h.is_read_only()
+
+ def test_can_flush_true (self):
+ self.connect ({"size": 512, "can_flush": True})
+ assert self.h.can_flush()
+
+ def test_can_flush_false (self):
+ self.connect ({"size": 512, "can_flush": False})
+ assert not self.h.can_flush()
+
+ def test_can_trim_true (self):
+ self.connect ({"size": 512, "can_trim": True})
+ assert self.h.can_trim()
+
+ def test_can_trim_false (self):
+ self.connect ({"size": 512, "can_trim": False})
+ assert not self.h.can_trim()
+
+ # nbdkit can always zero because it emulates it.
+ #self.connect ({"size": 512, "can_zero": True})
+ #assert self.h.can_zero()
+ #self.connect ({"size": 512, "can_zero": False})
+ #assert not self.h.can_zero()
+
+ def test_can_fast_zero_true (self):
+ self.connect ({"size": 512, "can_fast_zero": True})
+ assert self.h.can_fast_zero()
+
+ def test_can_fast_zero_false (self):
+ self.connect ({"size": 512, "can_fast_zero": False})
+ assert not self.h.can_fast_zero()
+
+ def test_can_fua_none (self):
+ self.connect ({"size": 512, "can_fua": "none"})
+ assert not self.h.can_fua()
+
+ def test_can_fua_emulate (self):
+ self.connect ({"size": 512, "can_fua": "emulate"})
+ assert self.h.can_fua()
+
+ def test_can_fua_native (self):
+ self.connect ({"size": 512, "can_fua": "native"})
+ assert self.h.can_fua()
+
+ def test_can_cache_none (self):
+ self.connect ({"size": 512, "can_cache": "none"})
+ assert not self.h.can_cache()
+
+ def test_can_cache_emulate (self):
+ self.connect ({"size": 512, "can_cache": "emulate"})
+ assert self.h.can_cache()
+
+ def test_can_cache_native (self):
+ self.connect ({"size": 512, "can_cache": "native"})
+ assert self.h.can_cache()
+
+ # Not yet implemented: can_extents.
+
+ def test_pread (self):
+ """Test pread."""
+ self.connect ({"size": 512})
+ buf = self.h.pread (512, 0)
+ assert buf == bytearray (512)
+
+ # Test pwrite + flags.
+ def test_pwrite (self):
+ self.connect ({"size": 512})
+ buf = bytearray (512)
+ self.h.pwrite (buf, 0)
+
+ def test_pwrite_fua (self):
+ self.connect ({"size": 512,
+ "can_fua": "native",
+ "pwrite_expect_fua": True})
+ buf = bytearray (512)
+ self.h.pwrite (buf, 0, nbd.CMD_FLAG_FUA)
+
+ def test_flush (self):
+ """Test flush."""
+ self.connect ({"size": 512, "can_flush": True})
+ self.h.flush ()
+
+ # Test trim + flags.
+ def test_trim (self):
+ self.connect ({"size": 512, "can_trim": True})
+ self.h.trim (512, 0)
+
+ def test_trim_fua (self):
+ self.connect ({"size": 512,
+ "can_trim": True,
+ "can_fua": "native",
+ "trim_expect_fua": True})
+ self.h.trim (512, 0, nbd.CMD_FLAG_FUA)
+
+ # Test zero + flags.
+ def test_zero (self):
+ self.connect ({"size": 512, "can_zero": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE)
+
+ def test_zero_fua (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "can_fua": "native",
+ "zero_expect_fua": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE | nbd.CMD_FLAG_FUA)
+
+ def test_zero_may_trim (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "zero_expect_may_trim": True})
+ self.h.zero (512, 0, 0) # absence of nbd.CMD_FLAG_NO_HOLE
+
+ def test_zero_fast_zero (self):
+ self.connect ({"size": 512,
+ "can_zero": True,
+ "can_fast_zero": True,
+ "zero_expect_fast_zero": True})
+ self.h.zero (512, 0, nbd.CMD_FLAG_NO_HOLE | nbd.CMD_FLAG_FAST_ZERO)
+
+ def test_cache (self):
+ """Test cache."""
+ self.connect ({"size": 512, "can_cache": "native"})
+ self.h.cache (512, 0)
--
2.18.2

790
SOURCES/0011-New-filter-extentlist.patch
View File

@ -1,790 +0,0 @@
From e744dcb38cc52cbe64977efcdd4bc60e802d1b17 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Thu, 23 Jan 2020 19:52:00 +0000
Subject: [PATCH 11/19] New filter: extentlist.
Allows a list of extents to be placed on top of an existing plugin.
(cherry picked from commit 3e770b6d6620a62546849a2863638041c0b00640)
---
TODO | 4 +
configure.ac | 2 +
.../nbdkit-cacheextents-filter.pod | 1 +
filters/extentlist/Makefile.am | 67 ++++
filters/extentlist/extentlist.c | 326 ++++++++++++++++++
.../extentlist/nbdkit-extentlist-filter.pod | 90 +++++
filters/noextents/nbdkit-noextents-filter.pod | 1 +
tests/Makefile.am | 4 +
tests/test-extentlist.sh | 175 ++++++++++
9 files changed, 670 insertions(+)
create mode 100644 filters/extentlist/Makefile.am
create mode 100644 filters/extentlist/extentlist.c
create mode 100644 filters/extentlist/nbdkit-extentlist-filter.pod
create mode 100755 tests/test-extentlist.sh
diff --git a/TODO b/TODO
index d2aca440..2a3e89dc 100644
--- a/TODO
+++ b/TODO
@@ -187,6 +187,10 @@ Suggestions for filters
MBs of extra data)
https://github.com/facebook/zstd/issues/395#issuecomment-535875379
+* nbdkit-extentlist-filter could read the extents generated by
+ qemu-img map, allowing extents to be ported from a qemu block
+ device.
+
nbdkit-rate-filter:
* allow other kinds of traffic shaping such as VBR
diff --git a/configure.ac b/configure.ac
index fde498b8..41e68de3 100644
--- a/configure.ac
+++ b/configure.ac
@@ -896,6 +896,7 @@ filters="\
cow \
delay \
error \
+ extentlist \
fua \
log \
nocache \
@@ -979,6 +980,7 @@ AC_CONFIG_FILES([Makefile
filters/cow/Makefile
filters/delay/Makefile
filters/error/Makefile
+ filters/extentlist/Makefile
filters/fua/Makefile
filters/log/Makefile
filters/nocache/Makefile
diff --git a/filters/cacheextents/nbdkit-cacheextents-filter.pod b/filters/cacheextents/nbdkit-cacheextents-filter.pod
index fdd2285a..bb2514a4 100644
--- a/filters/cacheextents/nbdkit-cacheextents-filter.pod
+++ b/filters/cacheextents/nbdkit-cacheextents-filter.pod
@@ -52,6 +52,7 @@ C<nbdkit-cacheextents-filter> first appeared in nbdkit 1.14.
L<nbdkit(1)>,
L<nbdkit-cache-filter(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-vddk-plugin(1)>,
L<nbdkit-filter(3)>,
diff --git a/filters/extentlist/Makefile.am b/filters/extentlist/Makefile.am
new file mode 100644
index 00000000..88a9afe1
--- /dev/null
+++ b/filters/extentlist/Makefile.am
@@ -0,0 +1,67 @@
+# nbdkit
+# Copyright (C) 2019-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+include $(top_srcdir)/common-rules.mk
+
+EXTRA_DIST = nbdkit-extentlist-filter.pod
+
+filter_LTLIBRARIES = nbdkit-extentlist-filter.la
+
+nbdkit_extentlist_filter_la_SOURCES = \
+ extentlist.c \
+ $(top_srcdir)/include/nbdkit-filter.h \
+ $(NULL)
+
+nbdkit_extentlist_filter_la_CPPFLAGS = \
+ -I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/include \
+ -I$(top_srcdir)/common/utils \
+ $(NULL)
+nbdkit_extentlist_filter_la_CFLAGS = $(WARNINGS_CFLAGS)
+nbdkit_extentlist_filter_la_LDFLAGS = \
+ -module -avoid-version -shared \
+ -Wl,--version-script=$(top_srcdir)/filters/filters.syms \
+ $(NULL)
+nbdkit_extentlist_filter_la_LIBADD = \
+ $(top_builddir)/common/utils/libutils.la \
+ $(NULL)
+
+if HAVE_POD
+
+man_MANS = nbdkit-extentlist-filter.1
+CLEANFILES += $(man_MANS)
+
+nbdkit-extentlist-filter.1: nbdkit-extentlist-filter.pod
+ $(PODWRAPPER) --section=1 --man $@ \
+ --html $(top_builddir)/html/$@.html \
+ $<
+
+endif HAVE_POD
diff --git a/filters/extentlist/extentlist.c b/filters/extentlist/extentlist.c
new file mode 100644
index 00000000..5f4990b3
--- /dev/null
+++ b/filters/extentlist/extentlist.c
@@ -0,0 +1,326 @@
+/* nbdkit
+ * Copyright (C) 2019-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdint.h>
+#include <inttypes.h>
+#include <string.h>
+#include <errno.h>
+#include <assert.h>
+
+#include <nbdkit-filter.h>
+
+#include "cleanup.h"
+#include "minmax.h"
+
+#define HOLE (NBDKIT_EXTENT_HOLE|NBDKIT_EXTENT_ZERO)
+
+static const char *extentlist;
+
+/* List of extents. Once we've finally parsed them this will be
+ * ordered, non-overlapping and have no gaps.
+ */
+struct extent {
+ uint64_t offset, length;
+ uint32_t type;
+};
+static struct extent *extents;
+static size_t nr_extents, allocated;
+
+/* Insert an extent before i. If i = nr_extents, inserts at the end. */
+static void
+insert_extent (size_t i, struct extent new_extent)
+{
+ if (nr_extents >= allocated) {
+ allocated = allocated == 0 ? 1 : allocated * 2;
+ extents = realloc (extents, (sizeof (struct extent) * allocated));
+ if (extents == NULL) {
+ nbdkit_error ("realloc: %m");
+ exit (EXIT_FAILURE);
+ }
+ }
+ memmove (&extents[i+1], &extents[i],
+ sizeof (struct extent) * (nr_extents-i));
+ extents[i] = new_extent;
+ nr_extents++;
+}
+
+static void
+extentlist_unload (void)
+{
+ free (extents);
+}
+
+/* Called for each key=value passed on the command line. */
+static int
+extentlist_config (nbdkit_next_config *next, void *nxdata,
+ const char *key, const char *value)
+{
+ if (strcmp (key, "extentlist") == 0) {
+ if (extentlist != NULL) {
+ nbdkit_error ("extentlist cannot appear twice");
+ exit (EXIT_FAILURE);
+ }
+ extentlist = value;
+ return 0;
+ }
+ else
+ return next (nxdata, key, value);
+}
+
+static int
+compare_offsets (const void *ev1, const void *ev2)
+{
+ const struct extent *e1 = ev1;
+ const struct extent *e2 = ev2;
+
+ if (e1->offset < e2->offset)
+ return -1;
+ else if (e1->offset > e2->offset)
+ return 1;
+ else
+ return 0;
+}
+
+static int
+compare_ranges (const void *ev1, const void *ev2)
+{
+ const struct extent *e1 = ev1;
+ const struct extent *e2 = ev2;
+
+ if (e1->offset < e2->offset)
+ return -1;
+ else if (e1->offset >= e2->offset + e2->length)
+ return 1;
+ else
+ return 0;
+}
+
+/* Similar to parse_extents in plugins/sh/methods.c */
+static void
+parse_extentlist (void)
+{
+ FILE *fp;
+ CLEANUP_FREE char *line = NULL;
+ size_t linelen = 0;
+ ssize_t len;
+ size_t i;
+ uint64_t end;
+
+ assert (extentlist != NULL);
+ assert (extents == NULL);
+ assert (nr_extents == 0);
+
+ fp = fopen (extentlist, "r");
+ if (!fp) {
+ nbdkit_error ("open: %s: %m", extentlist);
+ exit (EXIT_FAILURE);
+ }
+
+ while ((len = getline (&line, &linelen, fp)) != -1) {
+ const char *delim = " \t";
+ char *sp, *p;
+ int64_t offset, length;
+ uint32_t type;
+
+ if (len > 0 && line[len-1] == '\n') {
+ line[len-1] = '\0';
+ len--;
+ }
+
+ if ((p = strtok_r (line, delim, &sp)) == NULL) {
+ parse_error:
+ nbdkit_error ("%s: cannot parse %s", extentlist, line);
+ exit (EXIT_FAILURE);
+ }
+ offset = nbdkit_parse_size (p);
+ if (offset == -1)
+ exit (EXIT_FAILURE);
+
+ if ((p = strtok_r (NULL, delim, &sp)) == NULL)
+ goto parse_error;
+ length = nbdkit_parse_size (p);
+ if (length == -1)
+ exit (EXIT_FAILURE);
+
+ /* Skip zero length extents. Makes the rest of the code easier. */
+ if (length == 0)
+ continue;
+
+ if ((p = strtok_r (NULL, delim, &sp)) == NULL)
+ /* empty type field means allocated data (0) */
+ type = 0;
+ else if (sscanf (p, "%" SCNu32, &type) == 1)
+ ;
+ else {
+ type = 0;
+ if (strstr (p, "hole") != NULL)
+ type |= NBDKIT_EXTENT_HOLE;
+ if (strstr (p, "zero") != NULL)
+ type |= NBDKIT_EXTENT_ZERO;
+ }
+
+ insert_extent (nr_extents,
+ (struct extent){.offset = offset, .length=length,
+ .type=type});
+ }
+
+ fclose (fp);
+
+ /* Sort the extents by offset. */
+ qsort (extents, nr_extents, sizeof (struct extent), compare_offsets);
+
+ /* There must not be overlaps at this point. */
+ end = 0;
+ for (i = 0; i < nr_extents; ++i) {
+ if (extents[i].offset < end ||
+ extents[i].offset + extents[i].length < extents[i].offset) {
+ nbdkit_error ("extents in the extent list are overlapping");
+ exit (EXIT_FAILURE);
+ }
+ end = extents[i].offset + extents[i].length;
+ }
+
+ /* If there's a gap at the beginning, insert a hole|zero extent. */
+ if (nr_extents == 0 || extents[0].offset > 0) {
+ end = nr_extents == 0 ? UINT64_MAX : extents[0].offset;
+ insert_extent (0, (struct extent){.offset = 0, .length = end,
+ .type = HOLE});
+ }
+
+ /* Now insert hole|zero extents after every extent where there
+ * is a gap between that extent and the next one.
+ */
+ for (i = 0; i < nr_extents-1; ++i) {
+ end = extents[i].offset + extents[i].length;
+ if (end < extents[i+1].offset)
+ insert_extent (i+1, (struct extent){.offset = end,
+ .length = extents[i+1].offset - end,
+ .type = HOLE});
+ }
+
+ /* If there's a gap at the end, insert a hole|zero extent. */
+ end = extents[nr_extents-1].offset + extents[nr_extents-1].length;
+ if (end < UINT64_MAX)
+ insert_extent (nr_extents, (struct extent){.offset = end,
+ .length = UINT64_MAX-end,
+ .type = HOLE});
+
+ /* Debug the final list. */
+ for (i = 0; i < nr_extents; ++i) {
+ nbdkit_debug ("extentlist: "
+ "extent[%zu] = %" PRIu64 "-%" PRIu64 " (length %" PRIu64 ")"
+ " type %" PRIu32,
+ i, extents[i].offset,
+ extents[i].offset + extents[i].length - 1,
+ extents[i].length,
+ extents[i].type);
+ }
+}
+
+static int
+extentlist_config_complete (nbdkit_next_config_complete *next, void *nxdata)
+{
+ if (extentlist == NULL) {
+ nbdkit_error ("you must supply the extentlist parameter "
+ "on the command line");
+ return -1;
+ }
+
+ parse_extentlist ();
+
+ return next (nxdata);
+}
+
+static int
+extentlist_can_extents (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle)
+{
+ return 1;
+}
+
+/* Use ‘-D extentlist.lookup=1’ to debug the function below. */
+int extentlist_debug_lookup = 0;
+
+/* Read extents. */
+static int
+extentlist_extents (struct nbdkit_next_ops *next_ops, void *nxdata,
+ void *handle, uint32_t count, uint64_t offset,
+ uint32_t flags,
+ struct nbdkit_extents *ret_extents,
+ int *err)
+{
+ const struct extent eoffset = { .offset = offset };
+ struct extent *p;
+ ssize_t i;
+ uint64_t end;
+
+ /* Find the starting point in the extents list. */
+ p = bsearch (&eoffset, extents,
+ nr_extents, sizeof (struct extent), compare_ranges);
+ assert (p != NULL);
+ i = p - extents;
+
+ /* Add extents to the output. */
+ while (count > 0) {
+ if (extentlist_debug_lookup)
+ nbdkit_debug ("extentlist lookup: "
+ "loop i=%zd count=%" PRIu32 " offset=%" PRIu64,
+ i, count, offset);
+
+ end = extents[i].offset + extents[i].length;
+ if (nbdkit_add_extent (ret_extents, offset, end - offset,
+ extents[i].type) == -1)
+ return -1;
+
+ count -= MIN (count, end-offset);
+ offset = end;
+ i++;
+ }
+
+ return 0;
+}
+
+static struct nbdkit_filter filter = {
+ .name = "extentlist",
+ .longname = "nbdkit extentlist filter",
+ .unload = extentlist_unload,
+ .config = extentlist_config,
+ .config_complete = extentlist_config_complete,
+ .can_extents = extentlist_can_extents,
+ .extents = extentlist_extents,
+};
+
+NBDKIT_REGISTER_FILTER(filter)
diff --git a/filters/extentlist/nbdkit-extentlist-filter.pod b/filters/extentlist/nbdkit-extentlist-filter.pod
new file mode 100644
index 00000000..adfb4ad8
--- /dev/null
+++ b/filters/extentlist/nbdkit-extentlist-filter.pod
@@ -0,0 +1,90 @@
+=head1 NAME
+
+nbdkit-extentlist-filter - place extent list over a plugin
+
+=head1 SYNOPSIS
+
+ nbdkit --filter=extentlist plugin extentlist=FILENAME
+
+=head1 DESCRIPTION
+
+C<nbdkit-extentlist-filter> is an nbdkit filter lets you place a
+static list of extents on top of an existing plugin. Extents record
+whether or not specific parts of the disk are allocated or sparse.
+
+You can use this with plugins which cannot get extent information
+themselves, but you can get this information from another source. One
+place where it is useful is with L<nbdkit-ssh-plugin(1)> because the
+sftp protocol does not support reading sparseness information, but you
+may be able to get this information directly from the source disk on
+the remote server.
+
+=head1 FILE FORMAT
+
+The list of extents is specified in a text file. There is one extent
+specified per line. Each line has the format:
+
+ offset length type
+
+The C<offset> and C<length> fields may use any format understood by
+C<nbdkit_parse_size>. The optional C<type> field may be an integer,
+missing (same as 0), or a comma-separated list of the words C<hole>
+and C<zero>. (The fields correspond to the inputs of the
+C<nbdkit_add_extent> function, see L<nbdkit-plugin(3)>).
+
+An example of a valid set of extents covering a C<10M> disk where the
+first megabyte only is allocated data:
+
+ 0 1M
+ 1M 9M hole,zero
+
+Or you could omit the C<hole,zero> extent since any gaps are assumed
+to be holes with that type:
+
+ 0 1M
+
+The extent list need not cover the whole disk, and does not need to be
+in ascending order, but it must I<not> contain overlapping extents.
+
+=head1 PARAMETERS
+
+=over 4
+
+=item B<extentlist=>FILENAME
+
+Specify the file containing the extent list, in the format described
+in L</FILE FORMAT> above.
+
+=back
+
+=head1 FILES
+
+=over 4
+
+=item F<$filterdir/nbdkit-extentlist-filter.so>
+
+The filter.
+
+Use C<nbdkit --dump-config> to find the location of C<$filterdir>.
+
+=back
+
+=head1 VERSION
+
+C<nbdkit-extentlist-filter> first appeared in nbdkit 1.18.
+
+=head1 SEE ALSO
+
+L<nbdkit(1)>,
+L<nbdkit-cacheextents-filter(1)>,
+L<nbdkit-noextents-filter(1)>,
+L<nbdkit-filter(3)>,
+L<nbdkit-plugin(3)>.
+
+=head1 AUTHORS
+
+Richard W.M. Jones
+
+=head1 COPYRIGHT
+
+Copyright (C) 2020 Red Hat Inc.
diff --git a/filters/noextents/nbdkit-noextents-filter.pod b/filters/noextents/nbdkit-noextents-filter.pod
index 991ecfe8..0260a5cf 100644
--- a/filters/noextents/nbdkit-noextents-filter.pod
+++ b/filters/noextents/nbdkit-noextents-filter.pod
@@ -47,6 +47,7 @@ C<nbdkit-noextents-filter> first appeared in nbdkit 1.14.
L<nbdkit(1)>,
L<nbdkit-filter(3)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-fua-filter(1)>,
L<nbdkit-nocache-filter(1)>,
L<nbdkit-noparallel-filter(1)>,
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 09103fbb..b99952f4 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -110,6 +110,7 @@ EXTRA_DIST = \
test-error100.sh \
test-error-triggered.sh \
test-export-name.sh \
+ test-extentlist.sh \
test-file-extents.sh \
test-floppy.sh \
test-foreground.sh \
@@ -1009,6 +1010,9 @@ TESTS += \
test-error-triggered.sh \
$(NULL)
+# extentlist filter test.
+TESTS += test-extentlist.sh
+
# fua filter test.
TESTS += test-fua.sh
diff --git a/tests/test-extentlist.sh b/tests/test-extentlist.sh
new file mode 100755
index 00000000..7d05de4f
--- /dev/null
+++ b/tests/test-extentlist.sh
@@ -0,0 +1,175 @@
+#!/usr/bin/env bash
+# nbdkit
+# Copyright (C) 2016-2020 Red Hat Inc.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are
+# met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in the
+# documentation and/or other materials provided with the distribution.
+#
+# * Neither the name of Red Hat nor the names of its contributors may be
+# used to endorse or promote products derived from this software without
+# specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+# PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+# USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+# OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+
+# Test the extentlist filter.
+
+source ./functions.sh
+set -e
+set -x
+
+requires jq --version
+requires qemu-img --version
+requires qemu-img map --help
+
+out=test-extentlist.out
+input=test-extentlist.in
+expected=test-extentlist.expected
+files="$out $input $expected"
+rm -f $files
+cleanup_fn rm $files
+
+test ()
+{
+ nbdkit -v -D extentlist.lookup=1 \
+ -U - \
+ --filter=extentlist \
+ null size=$1 extentlist=$input \
+ --run 'qemu-img map -f raw --output=json $nbd' |
+ jq -c '.[] | {start:.start, length:.length, data:.data, zero:.zero}' \
+ > $out
+ diff -u $out $expected
+}
+
+# Empty extent list.
+cat > $input <<'EOF'
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+
+# Extent list covering 0-1M with data.
+cat > $input <<'EOF'
+0 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+EOF
+test 1M
+
+# Extent list covering 1-2M with data.
+cat > $input <<'EOF'
+1M 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+{"start":2097152,"length":1048576,"data":false,"zero":true}
+EOF
+test 3M
+
+# Extent list covering 1-2M with data, but in a more fragmented
+# way than the above.
+cat > $input <<'EOF'
+1024K 512K
+1536K 512K
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":false,"zero":true}
+{"start":1048576,"length":1048576,"data":true,"zero":false}
+{"start":2097152,"length":1048576,"data":false,"zero":true}
+EOF
+test 3M
+
+# Adjacent data and holes.
+cat > $input <<'EOF'
+0 1M
+2M 1M
+4M 1M
+EOF
+
+cat > $expected <<'EOF'
+{"start":0,"length":0,"data":false,"zero":false}
+EOF
+test 0
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+EOF
+test 1M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+EOF
+test 2M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+{"start":2097152,"length":1048576,"data":true,"zero":false}
+EOF
+test 3M
+cat > $expected <<'EOF'
+{"start":0,"length":1048576,"data":true,"zero":false}
+{"start":1048576,"length":1048576,"data":false,"zero":true}
+{"start":2097152,"length":1048576,"data":true,"zero":false}
+{"start":3145728,"length":1048576,"data":false,"zero":true}
+{"start":4194304,"length":1048576,"data":true,"zero":false}
+{"start":5242880,"length":1048576,"data":false,"zero":true}
+EOF
+test 6M
--
2.18.2

125
SOURCES/0012-extentlist-Documentation-and-test-fixes.patch
View File

@ -1,125 +0,0 @@
From 2a3e909e9e1ccb608bde75b76524acd753b33889 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 25 Jan 2020 11:38:14 +0000
Subject: [PATCH 12/19] extentlist: Documentation and test fixes.
Updates commit 3e770b6d6620a62546849a2863638041c0b00640.
(cherry picked from commit c16709ef663a5ed9fd9ddef4e379f316d84c9a07)
---
TODO | 12 +++++++----
.../extentlist/nbdkit-extentlist-filter.pod | 21 +++++++++++++------
plugins/curl/nbdkit-curl-plugin.pod | 1 +
plugins/ssh/nbdkit-ssh-plugin.pod | 1 +
tests/test-extentlist.sh | 2 +-
5 files changed, 26 insertions(+), 11 deletions(-)
diff --git a/TODO b/TODO
index 2a3e89dc..e1ac71cd 100644
--- a/TODO
+++ b/TODO
@@ -187,10 +187,6 @@ Suggestions for filters
MBs of extra data)
https://github.com/facebook/zstd/issues/395#issuecomment-535875379
-* nbdkit-extentlist-filter could read the extents generated by
- qemu-img map, allowing extents to be ported from a qemu block
- device.
-
nbdkit-rate-filter:
* allow other kinds of traffic shaping such as VBR
@@ -216,6 +212,14 @@ nbdkit-retry-filter:
* subsecond times
+nbdkit-extentlist-filter:
+
+* read the extents generated by qemu-img map, allowing extents to be
+ ported from a qemu block device
+
+* make non-read-only access safe by updating the extent list when the
+ filter sees writes and trims
+
Filters for security
--------------------
diff --git a/filters/extentlist/nbdkit-extentlist-filter.pod b/filters/extentlist/nbdkit-extentlist-filter.pod
index adfb4ad8..5d1a38ae 100644
--- a/filters/extentlist/nbdkit-extentlist-filter.pod
+++ b/filters/extentlist/nbdkit-extentlist-filter.pod
@@ -4,7 +4,7 @@ nbdkit-extentlist-filter - place extent list over a plugin
=head1 SYNOPSIS
- nbdkit --filter=extentlist plugin extentlist=FILENAME
+ nbdkit -r --filter=extentlist plugin extentlist=FILENAME
=head1 DESCRIPTION
@@ -13,11 +13,20 @@ static list of extents on top of an existing plugin. Extents record
whether or not specific parts of the disk are allocated or sparse.
You can use this with plugins which cannot get extent information
-themselves, but you can get this information from another source. One
-place where it is useful is with L<nbdkit-ssh-plugin(1)> because the
-sftp protocol does not support reading sparseness information, but you
-may be able to get this information directly from the source disk on
-the remote server.
+themselves, but where you can get this information from another
+source. One place where it is useful is with L<nbdkit-ssh-plugin(1)>
+because the sftp protocol does not support reading sparseness
+information, but you may be able to get this information directly from
+the source disk on the remote server using commands such as
+L<xfs_bmap(8)>. A similar situation applies to
+L<nbdkit-curl-plugin(1)>.
+
+Note that the extent list is read-only. This filter does not monitor
+writes and trims in order to update the extent list. What can happen
+is that you would write to a “hole” in the disk, but would not be able
+to read it back because the NBD client would still think that part of
+the disk is a hole. So it is generally only safe to use this filter
+in read-only mode (I<-r> option).
=head1 FILE FORMAT
diff --git a/plugins/curl/nbdkit-curl-plugin.pod b/plugins/curl/nbdkit-curl-plugin.pod
index 827e0bd1..d3c85248 100644
--- a/plugins/curl/nbdkit-curl-plugin.pod
+++ b/plugins/curl/nbdkit-curl-plugin.pod
@@ -182,6 +182,7 @@ L<libcurl(3)>,
L<CURLOPT_COOKIE(3)>
L<CURLOPT_VERBOSE(3)>,
L<nbdkit(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-retry-filter(1)>,
L<nbdkit-ssh-plugin(1)>,
diff --git a/plugins/ssh/nbdkit-ssh-plugin.pod b/plugins/ssh/nbdkit-ssh-plugin.pod
index 0a0421d5..3fc3146a 100644
--- a/plugins/ssh/nbdkit-ssh-plugin.pod
+++ b/plugins/ssh/nbdkit-ssh-plugin.pod
@@ -316,6 +316,7 @@ C<nbdkit-ssh-plugin> first appeared in nbdkit 1.12.
L<nbdkit(1)>,
L<nbdkit-curl-plugin(1)>,
+L<nbdkit-extentlist-filter(1)>,
L<nbdkit-readahead-filter(1)>,
L<nbdkit-retry-filter(1)>,
L<nbdkit-plugin(3)>,
diff --git a/tests/test-extentlist.sh b/tests/test-extentlist.sh
index 7d05de4f..73ce3ca6 100755
--- a/tests/test-extentlist.sh
+++ b/tests/test-extentlist.sh
@@ -50,7 +50,7 @@ cleanup_fn rm $files
test ()
{
nbdkit -v -D extentlist.lookup=1 \
- -U - \
+ -r -U - \
--filter=extentlist \
null size=$1 extentlist=$input \
--run 'qemu-img map -f raw --output=json $nbd' |
--
2.18.2

51
SOURCES/0013-vddk-Update-for-VDDK-7.0-RHBZ-1831969.patch
View File

@ -1,51 +0,0 @@
From bf1eabb211004f3dc74dd243e2adf52a13290377 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Wed, 6 May 2020 09:33:32 +0100
Subject: [PATCH 13/19] vddk: Update for VDDK 7.0 (RHBZ#1831969).
This version of VDDK changes the soname to libvixDiskLib.so.7.
Thanks: Ming Xie
(cherry picked from commit 7f53999179af98aa47569c6771517f7dfa08c5d0)
---
plugins/vddk/vddk-structs.h | 4 +++-
plugins/vddk/vddk.c | 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/plugins/vddk/vddk-structs.h b/plugins/vddk/vddk-structs.h
index 86087871..fff7201b 100644
--- a/plugins/vddk/vddk-structs.h
+++ b/plugins/vddk/vddk-structs.h
@@ -32,7 +32,7 @@
/* Types and structs that we pass to or return from the VDDK API.
*
- * Updated to VDDK 6.7
+ * Updated to VDDK 7.0
*/
#ifndef NBDKIT_VDDK_STRUCTS_H
@@ -127,6 +127,8 @@ typedef struct VixDiskLibInfo {
int numLinks;
char *parentFileNameHint;
char *uuid;
+ uint32_t logicalSectorSize; /* Added in 7.0. */
+ uint32_t physicalSectorSize; /* Added in 7.0. */
} VixDiskLibInfo;
typedef struct {
diff --git a/plugins/vddk/vddk.c b/plugins/vddk/vddk.c
index 5d3764d6..97ef5297 100644
--- a/plugins/vddk/vddk.c
+++ b/plugins/vddk/vddk.c
@@ -149,6 +149,7 @@ vddk_load (void)
{
static const char *sonames[] = {
/* Prefer the newest library in case multiple exist. */
+ "libvixDiskLib.so.7",
"libvixDiskLib.so.6",
"libvixDiskLib.so.5",
};
--
2.18.2

354
SOURCES/0014-common-include-Add-ASCII-only-ctype-header-and-ascii.patch
View File

@ -1,354 +0,0 @@
From cb3d83d0606d5267752895151bb3c229c48d6fb6 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 19 May 2020 12:03:23 +0100
Subject: [PATCH 14/19] common/include: Add ASCII-only ctype header and
ascii_is* functions.
Our existing uses of <ctype.h> were not necessarily safe if the locale
was changed.
Also I removed the unnecessary use of isascii, deprecated by POSIX.1-2008.
(cherry picked from commit 9f34db74786fdc92b290a7d47e4b003bd84fec69)
---
.gitignore | 1 +
common/include/Makefile.am | 6 +++
common/include/ascii-ctype.h | 60 ++++++++++++++++++++++++++
common/include/test-ascii-ctype.c | 63 ++++++++++++++++++++++++++++
plugins/partitioning/partition-gpt.c | 12 +++---
plugins/sh/Makefile.am | 1 +
plugins/sh/call.c | 6 +--
server/backend.c | 7 ++--
server/public.c | 4 +-
9 files changed, 146 insertions(+), 14 deletions(-)
create mode 100644 common/include/ascii-ctype.h
create mode 100644 common/include/test-ascii-ctype.c
diff --git a/.gitignore b/.gitignore
index e25bd99b..523894b7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -26,6 +26,7 @@ Makefile.in
/aclocal.m4
/autom4te.cache
/common/bitmap/test-bitmap
+/common/include/test-ascii-ctype
/common/include/test-byte-swapping
/common/include/test-current-dir-name
/common/include/test-isaligned
diff --git a/common/include/Makefile.am b/common/include/Makefile.am
index 4482de37..d7b0d7a8 100644
--- a/common/include/Makefile.am
+++ b/common/include/Makefile.am
@@ -34,6 +34,7 @@ include $(top_srcdir)/common-rules.mk
# These headers contain only common code shared by the core server,
# plugins and/or filters. They are not installed.
EXTRA_DIST = \
+ ascii-ctype.h \
byte-swapping.h \
exit-with-parent.h \
get-current-dir-name.h \
@@ -50,6 +51,7 @@ EXTRA_DIST = \
# Unit tests.
TESTS = \
+ test-ascii-ctype \
test-byte-swapping \
test-current-dir-name \
test-isaligned \
@@ -62,6 +64,10 @@ TESTS = \
$(NULL)
check_PROGRAMS = $(TESTS)
+test_ascii_ctype_SOURCES = test-ascii-ctype.c ascii-ctype.h
+test_ascii_ctype_CPPFLAGS = -I$(srcdir)
+test_ascii_ctype_CFLAGS = $(WARNINGS_CFLAGS)
+
test_byte_swapping_SOURCES = test-byte-swapping.c byte-swapping.h
test_byte_swapping_CPPFLAGS = -I$(srcdir)
test_byte_swapping_CFLAGS = $(WARNINGS_CFLAGS)
diff --git a/common/include/ascii-ctype.h b/common/include/ascii-ctype.h
new file mode 100644
index 00000000..5e8bf237
--- /dev/null
+++ b/common/include/ascii-ctype.h
@@ -0,0 +1,60 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Normal ctype functions are affected by the current locale. For
+ * example isupper() might recognize Ä in some but not all locales.
+ * These functions match only 7 bit ASCII characters.
+ */
+
+#ifndef NBDKIT_ASCII_CTYPE_H
+#define NBDKIT_ASCII_CTYPE_H
+
+#define ascii_isalnum(c) (ascii_isalpha (c) || ascii_isdigit (c))
+
+#define ascii_isalpha(c) \
+ (((c) >= 'a' && (c) <= 'z') || ((c) >= 'A' && (c) <= 'Z'))
+
+#define ascii_isdigit(c) \
+ ((c) >= '0' && (c) <= '9')
+
+#define ascii_isspace(c) \
+ ((c) == '\t' || (c) == '\n' || (c) == '\f' || (c) == '\r' || (c) == ' ')
+
+#define ascii_isxdigit(c) \
+ ((c) == '0' || (c) == '1' || (c) == '2' || (c) == '3' || (c) == '4' || \
+ (c) == '5' || (c) == '6' || (c) == '7' || (c) == '8' || (c) == '9' || \
+ (c) == 'a' || (c) == 'b' || (c) == 'c' || \
+ (c) == 'd' || (c) == 'e' || (c) == 'f' || \
+ (c) == 'A' || (c) == 'B' || (c) == 'C' || \
+ (c) == 'D' || (c) == 'E' || (c) == 'F')
+
+#endif /* NBDKIT_ASCII_CTYPE_H */
diff --git a/common/include/test-ascii-ctype.c b/common/include/test-ascii-ctype.c
new file mode 100644
index 00000000..edf27aa6
--- /dev/null
+++ b/common/include/test-ascii-ctype.c
@@ -0,0 +1,63 @@
+/* nbdkit
+ * Copyright (C) 2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+#include "ascii-ctype.h"
+
+int
+main (void)
+{
+ assert (ascii_isspace (' '));
+ assert (ascii_isspace ('\t'));
+ assert (ascii_isspace ('\n'));
+ assert (! ascii_isspace ('a'));
+
+ assert (ascii_isalpha ('a'));
+ assert (ascii_isalpha ('Z'));
+ assert (ascii_isalpha ('z'));
+ assert (! ascii_isalpha (' '));
+ assert (! ascii_isalpha ('0'));
+ { const char *s = "Ä"; assert (! ascii_isalpha (s[0])); }
+ { const char *s = "®"; assert (! ascii_isalpha (s[0])); }
+
+ assert (ascii_isdigit ('0'));
+ assert (ascii_isdigit ('9'));
+ { const char *s = "Ø"; assert (! ascii_isdigit (s[0])); } /* U+00D8 */
+ { const char *s = "９"; assert (! ascii_isdigit (s[0])); } /* U+FF19 */
+
+ exit (EXIT_SUCCESS);
+}
diff --git a/plugins/partitioning/partition-gpt.c b/plugins/partitioning/partition-gpt.c
index 75b4643a..819e9abe 100644
--- a/plugins/partitioning/partition-gpt.c
+++ b/plugins/partitioning/partition-gpt.c
@@ -36,12 +36,12 @@
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
-#include <ctype.h>
#include <inttypes.h>
#include <assert.h>
#include <nbdkit-plugin.h>
+#include "ascii-ctype.h"
#include "byte-swapping.h"
#include "efi-crc32.h"
@@ -244,19 +244,19 @@ parse_guid (const char *str, char *out)
return -1;
for (i = 0; i < 8; ++i)
- if (!isxdigit (str[i]))
+ if (!ascii_isxdigit (str[i]))
return -1;
for (i = 9; i < 13; ++i)
- if (!isxdigit (str[i]))
+ if (!ascii_isxdigit (str[i]))
return -1;
for (i = 14; i < 18; ++i)
- if (!isxdigit (str[i]))
+ if (!ascii_isxdigit (str[i]))
return -1;
for (i = 19; i < 23; ++i)
- if (!isxdigit (str[i]))
+ if (!ascii_isxdigit (str[i]))
return -1;
for (i = 24; i < 36; ++i)
- if (!isxdigit (str[i]))
+ if (!ascii_isxdigit (str[i]))
return -1;
/* The first, second and third blocks are parsed as little endian,
diff --git a/plugins/sh/Makefile.am b/plugins/sh/Makefile.am
index 445cdcd5..1f42b64c 100644
--- a/plugins/sh/Makefile.am
+++ b/plugins/sh/Makefile.am
@@ -48,6 +48,7 @@ nbdkit_sh_plugin_la_SOURCES = \
nbdkit_sh_plugin_la_CPPFLAGS = \
-I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/include \
-I$(top_srcdir)/common/utils \
$(NULL)
nbdkit_sh_plugin_la_CFLAGS = $(WARNINGS_CFLAGS)
diff --git a/plugins/sh/call.c b/plugins/sh/call.c
index 2d99a120..ae0cc0ac 100644
--- a/plugins/sh/call.c
+++ b/plugins/sh/call.c
@@ -44,10 +44,10 @@
#include <poll.h>
#include <sys/types.h>
#include <sys/wait.h>
-#include <ctype.h>
#include <nbdkit-plugin.h>
+#include "ascii-ctype.h"
#include "cleanup.h"
#include "utils.h"
@@ -392,7 +392,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
}
if (skip && ebuf[skip]) {
- if (!isspace ((unsigned char) ebuf[skip])) {
+ if (!ascii_isspace ((unsigned char) ebuf[skip])) {
/* Treat 'EINVALID' as EIO, not EINVAL */
err = EIO;
skip = 0;
@@ -400,7 +400,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
else
do
skip++;
- while (isspace ((unsigned char) ebuf[skip]));
+ while (ascii_isspace ((unsigned char) ebuf[skip]));
}
while (len > 0 && ebuf[len-1] == '\n')
diff --git a/server/backend.c b/server/backend.c
index 208c07b1..9add341f 100644
--- a/server/backend.c
+++ b/server/backend.c
@@ -37,13 +37,14 @@
#include <inttypes.h>
#include <string.h>
#include <assert.h>
-#include <ctype.h>
#include <dlfcn.h>
-#include "internal.h"
+#include "ascii-ctype.h"
#include "minmax.h"
+#include "internal.h"
+
/* Helpers for registering a new backend. */
/* Use:
@@ -100,7 +101,7 @@ backend_load (struct backend *b, const char *name, void (*load) (void))
for (i = 0; i < len; ++i) {
unsigned char c = name[i];
- if (!(isascii (c) && isalnum (c))) {
+ if (! ascii_isalnum (c)) {
fprintf (stderr,
"%s: %s: %s.name ('%s') field "
"must contain only ASCII alphanumeric characters\n",
diff --git a/server/public.c b/server/public.c
index 418945fe..98b78482 100644
--- a/server/public.c
+++ b/server/public.c
@@ -45,13 +45,13 @@
#include <string.h>
#include <unistd.h>
#include <limits.h>
-#include <ctype.h>
#include <termios.h>
#include <errno.h>
#include <poll.h>
#include <signal.h>
#include <sys/socket.h>
+#include "ascii-ctype.h"
#include "get-current-dir-name.h"
#include "internal.h"
@@ -210,7 +210,7 @@ nbdkit_parse_int64_t (const char *what, const char *str, int64_t *rp)
*/
#define PARSE_ERROR_IF_NEGATIVE \
do { \
- while (isspace (*str)) \
+ while (ascii_isspace (*str)) \
str++; \
if (*str == '-') { \
nbdkit_error ("%s: negative numbers are not allowed", what); \
--
2.18.2

46
SOURCES/0015-curl-Remove-some-useless-debug-messages.patch
View File

@ -1,46 +0,0 @@
From 0aa8e873e626c8af5f47e2e9896f33dcff4d7bb6 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 28 Mar 2020 16:59:55 +0000
Subject: [PATCH 15/19] curl: Remove some useless debug messages.
You can get the same information by setting -D curl.verbose=1
(cherry picked from commit e3539d55241adcbf1bc8102c019a0dd0ae8a3407)
---
plugins/curl/curl.c | 6 ------
1 file changed, 6 deletions(-)
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index 007449bc..8b341ae0 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -311,8 +311,6 @@ curl_open (int readonly)
goto err;
}
- nbdkit_debug ("opened libcurl easy handle");
-
/* Note this writes the output to stderr directly. We should
* consider using CURLOPT_DEBUGFUNCTION so we can handle it with
* nbdkit_debug.
@@ -340,8 +338,6 @@ curl_open (int readonly)
goto err;
}
- nbdkit_debug ("set libcurl URL: %s", url);
-
curl_easy_setopt (h->c, CURLOPT_AUTOREFERER, 1);
curl_easy_setopt (h->c, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt (h->c, CURLOPT_FAILONERROR, 1);
@@ -436,8 +432,6 @@ curl_open (int readonly)
curl_easy_setopt (h->c, CURLOPT_READDATA, h);
}
- nbdkit_debug ("returning new handle %p", h);
-
return h;
err:
--
2.18.2

182
SOURCES/0016-curl-Fix-D-curl.verbose-1-option.patch
View File

@ -1,182 +0,0 @@
From f421e599d3507f22d3d06b2dab070811e7e4f41c Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Sat, 28 Mar 2020 18:22:42 +0000
Subject: [PATCH 16/19] curl: Fix -D curl.verbose=1 option.
It didn't work previously for various reasons:
- Passed int instead of long to curl_easy_setopt.
- No CURLOPT_DEBUGFUNCTION callback was supplied, so messages were
sent to stderr, which meant they were never logged for the majority
of use cases.
This also removes extra debugging in the regular header callback.
This is no longer needed as the now-working -D curl.verbose=1 option
will log the headers.
Fixes commit 2ba11ee8f154ad1c84e10b43479b265fca2e996b.
(cherry picked from commit 6791c69bddf76577b65fa3ddfde652c0594ce340)
---
plugins/curl/Makefile.am | 2 ++
plugins/curl/curl.c | 73 ++++++++++++++++++++++++++++++----------
tests/test-curl.c | 3 +-
3 files changed, 60 insertions(+), 18 deletions(-)
diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am
index 6595eb95..024ddb6d 100644
--- a/plugins/curl/Makefile.am
+++ b/plugins/curl/Makefile.am
@@ -44,6 +44,7 @@ nbdkit_curl_plugin_la_SOURCES = \
nbdkit_curl_plugin_la_CPPFLAGS = \
-I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/utils \
$(NULL)
nbdkit_curl_plugin_la_CFLAGS = \
$(WARNINGS_CFLAGS) \
@@ -51,6 +52,7 @@ nbdkit_curl_plugin_la_CFLAGS = \
$(NULL)
nbdkit_curl_plugin_la_LIBADD = \
$(CURL_LIBS) \
+ $(top_builddir)/common/utils/libutils.la \
$(NULL)
nbdkit_curl_plugin_la_LDFLAGS = \
-module -avoid-version -shared \
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index 8b341ae0..b1693dc0 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -56,6 +56,8 @@
#include <nbdkit-plugin.h>
+#include "cleanup.h"
+
static const char *url = NULL;
static const char *user = NULL;
static char *password = NULL;
@@ -283,6 +285,8 @@ struct curl_handle {
curl_easy_strerror ((r)), (h)->errbuf); \
} while (0)
+static int debug_cb (CURL *handle, curl_infotype type,
+ const char *data, size_t size, void *);
static size_t header_cb (void *ptr, size_t size, size_t nmemb, void *opaque);
static size_t write_cb (char *ptr, size_t size, size_t nmemb, void *opaque);
static size_t read_cb (void *ptr, size_t size, size_t nmemb, void *opaque);
@@ -311,11 +315,13 @@ curl_open (int readonly)
goto err;
}
- /* Note this writes the output to stderr directly. We should
- * consider using CURLOPT_DEBUGFUNCTION so we can handle it with
- * nbdkit_debug.
- */
- curl_easy_setopt (h->c, CURLOPT_VERBOSE, curl_debug_verbose);
+ if (curl_debug_verbose) {
+ /* NB: Constants must be explicitly long because the parameter is
+ * varargs.
+ */
+ curl_easy_setopt (h->c, CURLOPT_VERBOSE, 1L);
+ curl_easy_setopt (h->c, CURLOPT_DEBUGFUNCTION, debug_cb);
+ }
curl_easy_setopt (h->c, CURLOPT_ERRORBUFFER, h->errbuf);
@@ -441,12 +447,56 @@ curl_open (int readonly)
return NULL;
}
+/* When using CURLOPT_VERBOSE, this callback is used to redirect
+ * messages to nbdkit_debug (instead of stderr).
+ */
+static int
+debug_cb (CURL *handle, curl_infotype type,
+ const char *data, size_t size, void *opaque)
+{
+ size_t origsize = size;
+ CLEANUP_FREE char *str;
+
+ /* The data parameter passed is NOT \0-terminated, but also it may
+ * have \n or \r\n line endings. The only sane way to deal with
+ * this is to copy the string. (The data strings may also be
+ * multi-line, but we don't deal with that here).
+ */
+ str = malloc (size + 1);
+ if (str == NULL)
+ goto out;
+ memcpy (str, data, size);
+ str[size] = '\0';
+
+ while (size > 0 && (str[size-1] == '\n' || str[size-1] == '\r')) {
+ str[size-1] = '\0';
+ size--;
+ }
+
+ switch (type) {
+ case CURLINFO_TEXT:
+ nbdkit_debug ("%s", str);
+ break;
+ case CURLINFO_HEADER_IN:
+ nbdkit_debug ("S: %s", str);
+ break;
+ case CURLINFO_HEADER_OUT:
+ nbdkit_debug ("C: %s", str);
+ break;
+ default:
+ /* Assume everything else is binary data that we cannot print. */
+ nbdkit_debug ("<data with size=%zu>", origsize);
+ }
+
+ out:
+ return 0;
+}
+
static size_t
header_cb (void *ptr, size_t size, size_t nmemb, void *opaque)
{
struct curl_handle *h = opaque;
size_t realsize = size * nmemb;
- size_t len;
const char *accept_line = "Accept-Ranges: bytes";
const char *line = ptr;
@@ -454,17 +504,6 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque)
strncmp (line, accept_line, strlen (accept_line)) == 0)
h->accept_range = true;
- /* Useful to print the server headers when debugging. However we
- * must strip off trailing \r?\n from each line.
- */
- len = realsize;
- if (len > 0 && line[len-1] == '\n')
- len--;
- if (len > 0 && line[len-1] == '\r')
- len--;
- if (len > 0)
- nbdkit_debug ("S: %.*s", (int) len, line);
-
return realsize;
}
diff --git a/tests/test-curl.c b/tests/test-curl.c
index 2b7e3beb..165edb35 100644
--- a/tests/test-curl.c
+++ b/tests/test-curl.c
@@ -74,9 +74,10 @@ main (int argc, char *argv[])
exit (EXIT_FAILURE);
}
if (test_start_nbdkit ("curl",
+ "-D", "curl.verbose=1",
+ "http://localhost/disk",
"cookie=foo=bar; baz=1;",
usp_param, /* unix-socket-path=... */
- "http://localhost/disk",
NULL) == -1)
exit (EXIT_FAILURE);
--
2.18.2

116
SOURCES/0017-curl-Case-insensitive-check-for-accept-ranges-RHBZ-1.patch
View File

@ -1,116 +0,0 @@
From aa62596afcc9143aa663bf834d305441cdd4cc70 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 19 May 2020 11:15:07 +0100
Subject: [PATCH 17/19] curl: Case insensitive check for accept-ranges
(RHBZ#1837337).
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When accessing an HTTP/2 server we read lowercase headers so the
existing test for byte range support did not work. You would see an
error like this:
nbdkit: curl[1]: error: server does not support 'range' (byte range) requests
This commit copies the bug fix which was recently added to qemu’s
block/curl.c.
qemu commits:
commit 69032253c33ae1774233c63cedf36d32242a85fc
Author: David Edmondson <david.edmondson@oracle.com>
Date: Mon Feb 24 10:13:10 2020 +0000
block/curl: HTTP header field names are case insensitive
commit 7788a319399f17476ff1dd43164c869e320820a2
Author: David Edmondson <david.edmondson@oracle.com>
Date: Mon Feb 24 10:13:09 2020 +0000
block/curl: HTTP header fields allow whitespace around values
Thanks: David Edmondson, Pino Toscano, Zi Liu
(cherry picked from commit c1260ec1f6538831e10f164567b53054a2ec0c2a)
---
plugins/curl/Makefile.am | 1 +
plugins/curl/curl.c | 29 ++++++++++++++++++++++++-----
tests/web-server.c | 2 +-
3 files changed, 26 insertions(+), 6 deletions(-)
diff --git a/plugins/curl/Makefile.am b/plugins/curl/Makefile.am
index 024ddb6d..3dbe3ca8 100644
--- a/plugins/curl/Makefile.am
+++ b/plugins/curl/Makefile.am
@@ -44,6 +44,7 @@ nbdkit_curl_plugin_la_SOURCES = \
nbdkit_curl_plugin_la_CPPFLAGS = \
-I$(top_srcdir)/include \
+ -I$(top_srcdir)/common/include \
-I$(top_srcdir)/common/utils \
$(NULL)
nbdkit_curl_plugin_la_CFLAGS = \
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index b1693dc0..ac30cbdd 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -57,6 +57,7 @@
#include <nbdkit-plugin.h>
#include "cleanup.h"
+#include "ascii-ctype.h"
static const char *url = NULL;
static const char *user = NULL;
@@ -497,12 +498,30 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque)
{
struct curl_handle *h = opaque;
size_t realsize = size * nmemb;
- const char *accept_line = "Accept-Ranges: bytes";
- const char *line = ptr;
+ const char *header = ptr;
+ const char *end = header + realsize;
+ const char *accept_ranges = "accept-ranges:";
+ const char *bytes = "bytes";
- if (realsize >= strlen (accept_line) &&
- strncmp (line, accept_line, strlen (accept_line)) == 0)
- h->accept_range = true;
+ if (realsize >= strlen (accept_ranges) &&
+ strncasecmp (header, accept_ranges, strlen (accept_ranges)) == 0) {
+ const char *p = strchr (header, ':') + 1;
+
+ /* Skip whitespace between the header name and value. */
+ while (p < end && *p && ascii_isspace (*p))
+ p++;
+
+ if (end - p >= strlen (bytes)
+ && strncmp (p, bytes, strlen (bytes)) == 0) {
+ /* Check that there is nothing but whitespace after the value. */
+ p += strlen (bytes);
+ while (p < end && *p && ascii_isspace (*p))
+ p++;
+
+ if (p == end || !*p)
+ h->accept_range = true;
+ }
+ }
return realsize;
}
diff --git a/tests/web-server.c b/tests/web-server.c
index f27ee70d..f9f10917 100644
--- a/tests/web-server.c
+++ b/tests/web-server.c
@@ -235,7 +235,7 @@ handle_request (int s, bool headers_only)
const char response1_ok[] = "HTTP/1.1 200 OK\r\n";
const char response1_partial[] = "HTTP/1.1 206 Partial Content\r\n";
const char response2[] =
- "Accept-Ranges: bytes\r\n"
+ "Accept-rANGES: bytes\r\n" /* See RHBZ#1837337 */
"Connection: keep-alive\r\n"
"Content-Type: application/octet-stream\r\n";
char response3[64];
--
2.18.2

41
SOURCES/0018-sh-Don-t-need-to-cast-parameter-of-ascii_is-to-unsig.patch
View File

@ -1,41 +0,0 @@
From d5947881c2918196d61d7795adba0abb881a86b9 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 19 May 2020 15:29:55 +0100
Subject: [PATCH 18/19] sh: Don't need to cast parameter of ascii_is* to
(unsigned char).
Our replacements for these functions are not undefined for negative
values.
Thanks: Eric Blake
Fixes: commit 9f34db74786fdc92b290a7d47e4b003bd84fec69.
(cherry picked from commit 06a79b8bb8cfd97a272223c967601d8858acb817)
---
plugins/sh/call.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/plugins/sh/call.c b/plugins/sh/call.c
index ae0cc0ac..ba9f055f 100644
--- a/plugins/sh/call.c
+++ b/plugins/sh/call.c
@@ -392,7 +392,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
}
if (skip && ebuf[skip]) {
- if (!ascii_isspace ((unsigned char) ebuf[skip])) {
+ if (!ascii_isspace (ebuf[skip])) {
/* Treat 'EINVALID' as EIO, not EINVAL */
err = EIO;
skip = 0;
@@ -400,7 +400,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
else
do
skip++;
- while (ascii_isspace ((unsigned char) ebuf[skip]));
+ while (ascii_isspace (ebuf[skip]));
}
while (len > 0 && ebuf[len-1] == '\n')
--
2.18.2

559
SOURCES/0019-common-include-Add-locale-safe-ascii_strcasecmp-and-.patch
View File

@ -1,559 +0,0 @@
From 9a99549f5df6ef69dd1d2b509c13aaff4e431742 Mon Sep 17 00:00:00 2001
From: "Richard W.M. Jones" <rjones@redhat.com>
Date: Tue, 19 May 2020 16:11:28 +0100
Subject: [PATCH 19/19] common/include: Add locale-safe ascii_strcasecmp and
ascii_strncasecmp.
These are derived from the FreeBSD functions here:
https://github.com/freebsd/freebsd/blob/master/sys/libkern/strcasecmp.c
Thanks: Eric Blake.
(cherry picked from commit 46a29b8e91d69e812d78df53e91b526a560000fe)
---
.gitignore | 1 +
common/include/Makefile.am | 6 +++
common/include/ascii-ctype.h | 6 +++
common/include/ascii-string.h | 77 ++++++++++++++++++++++++++++
common/include/test-ascii-string.c | 79 +++++++++++++++++++++++++++++
plugins/curl/curl.c | 7 +--
plugins/info/info.c | 17 ++++---
plugins/nbd/nbd.c | 8 +--
plugins/partitioning/partitioning.c | 10 ++--
plugins/sh/call.c | 25 ++++-----
server/main.c | 8 +--
server/public.c | 21 ++++----
12 files changed, 222 insertions(+), 43 deletions(-)
create mode 100644 common/include/ascii-string.h
create mode 100644 common/include/test-ascii-string.c
diff --git a/.gitignore b/.gitignore
index 523894b7..aa148f04 100644
--- a/.gitignore
+++ b/.gitignore
@@ -27,6 +27,7 @@ Makefile.in
/autom4te.cache
/common/bitmap/test-bitmap
/common/include/test-ascii-ctype
+/common/include/test-ascii-string
/common/include/test-byte-swapping
/common/include/test-current-dir-name
/common/include/test-isaligned
diff --git a/common/include/Makefile.am b/common/include/Makefile.am
index d7b0d7a8..eff71863 100644
--- a/common/include/Makefile.am
+++ b/common/include/Makefile.am
@@ -35,6 +35,7 @@ include $(top_srcdir)/common-rules.mk
# plugins and/or filters. They are not installed.
EXTRA_DIST = \
ascii-ctype.h \
+ ascii-string.h \
byte-swapping.h \
exit-with-parent.h \
get-current-dir-name.h \
@@ -52,6 +53,7 @@ EXTRA_DIST = \
TESTS = \
test-ascii-ctype \
+ test-ascii-string \
test-byte-swapping \
test-current-dir-name \
test-isaligned \
@@ -68,6 +70,10 @@ test_ascii_ctype_SOURCES = test-ascii-ctype.c ascii-ctype.h
test_ascii_ctype_CPPFLAGS = -I$(srcdir)
test_ascii_ctype_CFLAGS = $(WARNINGS_CFLAGS)
+test_ascii_string_SOURCES = test-ascii-string.c ascii-string.h
+test_ascii_string_CPPFLAGS = -I$(srcdir)
+test_ascii_string_CFLAGS = $(WARNINGS_CFLAGS)
+
test_byte_swapping_SOURCES = test-byte-swapping.c byte-swapping.h
test_byte_swapping_CPPFLAGS = -I$(srcdir)
test_byte_swapping_CFLAGS = $(WARNINGS_CFLAGS)
diff --git a/common/include/ascii-ctype.h b/common/include/ascii-ctype.h
index 5e8bf237..a700563e 100644
--- a/common/include/ascii-ctype.h
+++ b/common/include/ascii-ctype.h
@@ -49,6 +49,9 @@
#define ascii_isspace(c) \
((c) == '\t' || (c) == '\n' || (c) == '\f' || (c) == '\r' || (c) == ' ')
+#define ascii_isupper(c) \
+ ((c) >= 'A' && (c) <= 'Z')
+
#define ascii_isxdigit(c) \
((c) == '0' || (c) == '1' || (c) == '2' || (c) == '3' || (c) == '4' || \
(c) == '5' || (c) == '6' || (c) == '7' || (c) == '8' || (c) == '9' || \
@@ -57,4 +60,7 @@
(c) == 'A' || (c) == 'B' || (c) == 'C' || \
(c) == 'D' || (c) == 'E' || (c) == 'F')
+#define ascii_tolower(c) \
+ (ascii_isupper ((c)) ? (c) - 'A' + 'a' : (c))
+
#endif /* NBDKIT_ASCII_CTYPE_H */
diff --git a/common/include/ascii-string.h b/common/include/ascii-string.h
new file mode 100644
index 00000000..0a60d5f4
--- /dev/null
+++ b/common/include/ascii-string.h
@@ -0,0 +1,77 @@
+/* nbdkit
+ * Copyright (C) 2013-2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+/* Case insensitive string comparison functions (like strcasecmp,
+ * strncasecmp) which work correctly in any locale. They can only be
+ * used for comparison when one or both strings is 7 bit ASCII.
+ */
+
+#ifndef NBDKIT_ASCII_STRING_H
+#define NBDKIT_ASCII_STRING_H
+
+#include "ascii-ctype.h"
+
+static inline int
+ascii_strcasecmp (const char *s1, const char *s2)
+{
+ const unsigned char *us1 = (const unsigned char *)s1;
+ const unsigned char *us2 = (const unsigned char *)s2;
+
+ while (ascii_tolower (*us1) == ascii_tolower (*us2)) {
+ if (*us1++ == '\0')
+ return 0;
+ us2++;
+ }
+
+ return ascii_tolower (*us1) - ascii_tolower (*us2);
+}
+
+static inline int
+ascii_strncasecmp (const char *s1, const char *s2, size_t n)
+{
+ if (n != 0) {
+ const unsigned char *us1 = (const unsigned char *)s1;
+ const unsigned char *us2 = (const unsigned char *)s2;
+
+ do {
+ if (ascii_tolower (*us1) != ascii_tolower (*us2))
+ return ascii_tolower (*us1) - ascii_tolower (*us2);
+ if (*us1++ == '\0')
+ break;
+ us2++;
+ } while (--n != 0);
+ }
+
+ return 0;
+}
+
+#endif /* NBDKIT_ASCII_STRING_H */
diff --git a/common/include/test-ascii-string.c b/common/include/test-ascii-string.c
new file mode 100644
index 00000000..0fa4a483
--- /dev/null
+++ b/common/include/test-ascii-string.c
@@ -0,0 +1,79 @@
+/* nbdkit
+ * Copyright (C) 2020 Red Hat Inc.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ *
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ *
+ * * Redistributions in binary form must reproduce the above copyright
+ * notice, this list of conditions and the following disclaimer in the
+ * documentation and/or other materials provided with the distribution.
+ *
+ * * Neither the name of Red Hat nor the names of its contributors may be
+ * used to endorse or promote products derived from this software without
+ * specific prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY RED HAT AND CONTRIBUTORS ''AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
+ * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL RED HAT OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include <config.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+
+#include "ascii-string.h"
+
+int
+main (void)
+{
+ assert (ascii_strcasecmp ("", "") == 0);
+ assert (ascii_strcasecmp ("a", "a") == 0);
+ assert (ascii_strcasecmp ("abc", "abc") == 0);
+ assert (ascii_strcasecmp ("a", "b") < 0);
+ assert (ascii_strcasecmp ("b", "a") > 0);
+ assert (ascii_strcasecmp ("aa", "a") > 0);
+
+ /* Second string contains Turkish dotless lowercase letter ı. */
+ assert (ascii_strcasecmp ("hi", "hı") != 0);
+
+ /* Check that we got our rounding behaviour correct. */
+ assert (ascii_strcasecmp ("\x1", "\x7f") < 0);
+ assert (ascii_strcasecmp ("\x1", "\x80") < 0);
+ assert (ascii_strcasecmp ("\x1", "\x81") < 0);
+ assert (ascii_strcasecmp ("\x1", "\xff") < 0);
+
+ assert (ascii_strncasecmp ("", "", 0) == 0);
+ assert (ascii_strncasecmp ("a", "a", 1) == 0);
+ assert (ascii_strncasecmp ("abc", "abc", 3) == 0);
+ assert (ascii_strncasecmp ("abc", "def", 0) == 0);
+ assert (ascii_strncasecmp ("abc", "abd", 2) == 0);
+ assert (ascii_strncasecmp ("a", "b", 1) < 0);
+ assert (ascii_strncasecmp ("b", "a", 1) > 0);
+ assert (ascii_strncasecmp ("aa", "a", 2) > 0);
+ assert (ascii_strncasecmp ("aa", "a", 100) > 0);
+
+ assert (ascii_strncasecmp ("hi", "hı", 1) == 0);
+ assert (ascii_strncasecmp ("hi", "hı", 2) != 0);
+
+ assert (ascii_strncasecmp ("\x1", "\x7f", 1) < 0);
+ assert (ascii_strncasecmp ("\x1", "\x80", 1) < 0);
+ assert (ascii_strncasecmp ("\x1", "\x81", 1) < 0);
+ assert (ascii_strncasecmp ("\x1", "\xff", 1) < 0);
+
+ exit (EXIT_SUCCESS);
+}
diff --git a/plugins/curl/curl.c b/plugins/curl/curl.c
index ac30cbdd..00f0628a 100644
--- a/plugins/curl/curl.c
+++ b/plugins/curl/curl.c
@@ -58,6 +58,7 @@
#include "cleanup.h"
#include "ascii-ctype.h"
+#include "ascii-string.h"
static const char *url = NULL;
static const char *user = NULL;
@@ -419,8 +420,8 @@ curl_open (int readonly)
#endif
nbdkit_debug ("content length: %" PRIi64, h->exportsize);
- if (strncasecmp (url, "http://", strlen ("http://")) == 0 ||
- strncasecmp (url, "https://", strlen ("https://")) == 0) {
+ if (ascii_strncasecmp (url, "http://", strlen ("http://")) == 0 ||
+ ascii_strncasecmp (url, "https://", strlen ("https://")) == 0) {
if (!h->accept_range) {
nbdkit_error ("server does not support 'range' (byte range) requests");
goto err;
@@ -504,7 +505,7 @@ header_cb (void *ptr, size_t size, size_t nmemb, void *opaque)
const char *bytes = "bytes";
if (realsize >= strlen (accept_ranges) &&
- strncasecmp (header, accept_ranges, strlen (accept_ranges)) == 0) {
+ ascii_strncasecmp (header, accept_ranges, strlen (accept_ranges)) == 0) {
const char *p = strchr (header, ':') + 1;
/* Skip whitespace between the header name and value. */
diff --git a/plugins/info/info.c b/plugins/info/info.c
index 329a3684..e04d672b 100644
--- a/plugins/info/info.c
+++ b/plugins/info/info.c
@@ -49,6 +49,7 @@
#include <nbdkit-plugin.h>
+#include "ascii-string.h"
#include "byte-swapping.h"
#include "tvdiff.h"
@@ -76,12 +77,12 @@ static int
info_config (const char *key, const char *value)
{
if (strcmp (key, "mode") == 0) {
- if (strcasecmp (value, "exportname") == 0 ||
- strcasecmp (value, "export-name") == 0) {
+ if (ascii_strcasecmp (value, "exportname") == 0 ||
+ ascii_strcasecmp (value, "export-name") == 0) {
mode = MODE_EXPORTNAME;
}
- else if (strcasecmp (value, "base64exportname") == 0 ||
- strcasecmp (value, "base64-export-name") == 0) {
+ else if (ascii_strcasecmp (value, "base64exportname") == 0 ||
+ ascii_strcasecmp (value, "base64-export-name") == 0) {
#ifdef HAVE_BASE64
mode = MODE_BASE64EXPORTNAME;
#else
@@ -89,13 +90,13 @@ info_config (const char *key, const char *value)
return -1;
#endif
}
- else if (strcasecmp (value, "address") == 0)
+ else if (ascii_strcasecmp (value, "address") == 0)
mode = MODE_ADDRESS;
- else if (strcasecmp (value, "time") == 0)
+ else if (ascii_strcasecmp (value, "time") == 0)
mode = MODE_TIME;
- else if (strcasecmp (value, "uptime") == 0)
+ else if (ascii_strcasecmp (value, "uptime") == 0)
mode = MODE_UPTIME;
- else if (strcasecmp (value, "conntime") == 0)
+ else if (ascii_strcasecmp (value, "conntime") == 0)
mode = MODE_CONNTIME;
else {
nbdkit_error ("unknown mode: '%s'", value);
diff --git a/plugins/nbd/nbd.c b/plugins/nbd/nbd.c
index d020beec..980ce8ec 100644
--- a/plugins/nbd/nbd.c
+++ b/plugins/nbd/nbd.c
@@ -52,6 +52,8 @@
#define NBDKIT_API_VERSION 2
#include <nbdkit-plugin.h>
+
+#include "ascii-string.h"
#include "byte-swapping.h"
#include "cleanup.h"
#include "utils.h"
@@ -152,9 +154,9 @@ nbdplug_config (const char *key, const char *value)
shared = r;
}
else if (strcmp (key, "tls") == 0) {
- if (strcasecmp (value, "require") == 0 ||
- strcasecmp (value, "required") == 0 ||
- strcasecmp (value, "force") == 0)
+ if (ascii_strcasecmp (value, "require") == 0 ||
+ ascii_strcasecmp (value, "required") == 0 ||
+ ascii_strcasecmp (value, "force") == 0)
tls = LIBNBD_TLS_REQUIRE;
else {
r = nbdkit_parse_bool (value);
diff --git a/plugins/partitioning/partitioning.c b/plugins/partitioning/partitioning.c
index 6e426b93..e35764dc 100644
--- a/plugins/partitioning/partitioning.c
+++ b/plugins/partitioning/partitioning.c
@@ -48,6 +48,7 @@
#include <nbdkit-plugin.h>
+#include "ascii-string.h"
#include "byte-swapping.h"
#include "isaligned.h"
#include "iszero.h"
@@ -176,9 +177,10 @@ partitioning_config (const char *key, const char *value)
nr_files++;
}
else if (strcmp (key, "partition-type") == 0) {
- if (strcasecmp (value, "mbr") == 0 || strcasecmp (value, "dos") == 0)
+ if (ascii_strcasecmp (value, "mbr") == 0 ||
+ ascii_strcasecmp (value, "dos") == 0)
parttype = PARTTYPE_MBR;
- else if (strcasecmp (value, "gpt") == 0)
+ else if (ascii_strcasecmp (value, "gpt") == 0)
parttype = PARTTYPE_GPT;
else {
nbdkit_error ("unknown partition-type: %s", value);
@@ -209,13 +211,13 @@ partitioning_config (const char *key, const char *value)
alignment = r;
}
else if (strcmp (key, "mbr-id") == 0) {
- if (strcasecmp (value, "default") == 0)
+ if (ascii_strcasecmp (value, "default") == 0)
mbr_id = DEFAULT_MBR_ID;
else if (nbdkit_parse_uint8_t ("mbr-id", value, &mbr_id) == -1)
return -1;
}
else if (strcmp (key, "type-guid") == 0) {
- if (strcasecmp (value, "default") == 0)
+ if (ascii_strcasecmp (value, "default") == 0)
parse_guid (DEFAULT_TYPE_GUID, type_guid);
else if (parse_guid (value, type_guid) == -1) {
nbdkit_error ("could not validate GUID: %s", value);
diff --git a/plugins/sh/call.c b/plugins/sh/call.c
index ba9f055f..554e2f78 100644
--- a/plugins/sh/call.c
+++ b/plugins/sh/call.c
@@ -48,6 +48,7 @@
#include <nbdkit-plugin.h>
#include "ascii-ctype.h"
+#include "ascii-string.h"
#include "cleanup.h"
#include "utils.h"
@@ -332,48 +333,48 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
}
/* Recognize the errno values that match NBD protocol errors */
- if (strncasecmp (ebuf, "EPERM", 5) == 0) {
+ if (ascii_strncasecmp (ebuf, "EPERM", 5) == 0) {
err = EPERM;
skip = 5;
}
- else if (strncasecmp (ebuf, "EIO", 3) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EIO", 3) == 0) {
err = EIO;
skip = 3;
}
- else if (strncasecmp (ebuf, "ENOMEM", 6) == 0) {
+ else if (ascii_strncasecmp (ebuf, "ENOMEM", 6) == 0) {
err = ENOMEM;
skip = 6;
}
- else if (strncasecmp (ebuf, "EINVAL", 6) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EINVAL", 6) == 0) {
err = EINVAL;
skip = 6;
}
- else if (strncasecmp (ebuf, "ENOSPC", 6) == 0) {
+ else if (ascii_strncasecmp (ebuf, "ENOSPC", 6) == 0) {
err = ENOSPC;
skip = 6;
}
- else if (strncasecmp (ebuf, "EOVERFLOW", 9) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EOVERFLOW", 9) == 0) {
err = EOVERFLOW;
skip = 9;
}
- else if (strncasecmp (ebuf, "ESHUTDOWN", 9) == 0) {
+ else if (ascii_strncasecmp (ebuf, "ESHUTDOWN", 9) == 0) {
err = ESHUTDOWN;
skip = 9;
}
- else if (strncasecmp (ebuf, "ENOTSUP", 7) == 0) {
+ else if (ascii_strncasecmp (ebuf, "ENOTSUP", 7) == 0) {
err = ENOTSUP;
skip = 7;
}
- else if (strncasecmp (ebuf, "EOPNOTSUPP", 10) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EOPNOTSUPP", 10) == 0) {
err = EOPNOTSUPP;
skip = 10;
}
/* Other errno values that server/protocol.c treats specially */
- else if (strncasecmp (ebuf, "EROFS", 5) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EROFS", 5) == 0) {
err = EROFS;
skip = 5;
}
- else if (strncasecmp (ebuf, "EDQUOT", 6) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EDQUOT", 6) == 0) {
#ifdef EDQUOT
err = EDQUOT;
#else
@@ -381,7 +382,7 @@ handle_script_error (const char *argv0, char *ebuf, size_t len)
#endif
skip = 6;
}
- else if (strncasecmp (ebuf, "EFBIG", 5) == 0) {
+ else if (ascii_strncasecmp (ebuf, "EFBIG", 5) == 0) {
err = EFBIG;
skip = 5;
}
diff --git a/server/main.c b/server/main.c
index 11ba1e6d..08116d74 100644
--- a/server/main.c
+++ b/server/main.c
@@ -55,6 +55,8 @@
#include <dlfcn.h>
+#include "ascii-string.h"
+
#include "internal.h"
#include "nbd-protocol.h"
#include "options.h"
@@ -282,9 +284,9 @@ main (int argc, char *argv[])
case TLS_OPTION:
tls_set_on_cli = true;
- if (strcasecmp (optarg, "require") == 0 ||
- strcasecmp (optarg, "required") == 0 ||
- strcasecmp (optarg, "force") == 0)
+ if (ascii_strcasecmp (optarg, "require") == 0 ||
+ ascii_strcasecmp (optarg, "required") == 0 ||
+ ascii_strcasecmp (optarg, "force") == 0)
tls = 2;
else {
tls = nbdkit_parse_bool (optarg);
diff --git a/server/public.c b/server/public.c
index 98b78482..919f082d 100644
--- a/server/public.c
+++ b/server/public.c
@@ -52,6 +52,7 @@
#include <sys/socket.h>
#include "ascii-ctype.h"
+#include "ascii-string.h"
#include "get-current-dir-name.h"
#include "internal.h"
@@ -385,19 +386,19 @@ int
nbdkit_parse_bool (const char *str)
{
if (!strcmp (str, "1") ||
- !strcasecmp (str, "true") ||
- !strcasecmp (str, "t") ||
- !strcasecmp (str, "yes") ||
- !strcasecmp (str, "y") ||
- !strcasecmp (str, "on"))
+ !ascii_strcasecmp (str, "true") ||
+ !ascii_strcasecmp (str, "t") ||
+ !ascii_strcasecmp (str, "yes") ||
+ !ascii_strcasecmp (str, "y") ||
+ !ascii_strcasecmp (str, "on"))
return 1;
if (!strcmp (str, "0") ||
- !strcasecmp (str, "false") ||
- !strcasecmp (str, "f") ||
- !strcasecmp (str, "no") ||
- !strcasecmp (str, "n") ||
- !strcasecmp (str, "off"))
+ !ascii_strcasecmp (str, "false") ||
+ !ascii_strcasecmp (str, "f") ||
+ !ascii_strcasecmp (str, "no") ||
+ !ascii_strcasecmp (str, "n") ||
+ !ascii_strcasecmp (str, "off"))
return 0;
nbdkit_error ("could not decipher boolean (%s)", str);
--
2.18.2

55
SOURCES/copy-patches.sh Executable file
View File

@ -0,0 +1,55 @@
#!/bin/bash -
set -e
# Maintainer script to copy patches from the git repo to the current
# directory. Use it like this:
# ./copy-patches.sh
rhel_version=8.6
# Check we're in the right directory.
if [ ! -f nbdkit.spec ]; then
echo "$0: run this from the directory containing 'nbdkit.spec'"
exit 1
fi
git_checkout=$HOME/d/nbdkit-rhel-$rhel_version
if [ ! -d $git_checkout ]; then
echo "$0: $git_checkout does not exist"
echo "This script is only for use by the maintainer when preparing a"
echo "nbdkit release on RHEL."
exit 1
fi
# Get the base version of nbdkit.
version=`grep '^Version:' nbdkit.spec | awk '{print $2}'`
tag="v$version"
# Remove any existing patches.
git rm -f [0-9]*.patch ||:
rm -f [0-9]*.patch
# Get the patches.
(cd $git_checkout; rm -f [0-9]*.patch; git format-patch -N $tag)
mv $git_checkout/[0-9]*.patch .
# Remove any not to be applied.
rm -f *NOT-FOR-RPM*.patch
# Add the patches.
git add [0-9]*.patch
# Print out the patch lines.
echo
echo "--- Copy the following text into nbdkit.spec file"
echo
echo "# Patches."
for f in [0-9]*.patch; do
n=`echo $f | awk -F- '{print $1}'`
echo "Patch$n: $f"
done
echo
echo "--- End of text"

17
SOURCES/nbdkit-1.16.2.tar.gz.sig
View File

@ -1,17 +0,0 @@
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEE93dPsa0HSn6Mh2fqkXOPc+G3aKAFAl4wFvoRHHJpY2hAYW5u
ZXhpYS5vcmcACgkQkXOPc+G3aKASexAAmpZw61rCI7SY8zm4O0gb+pIx7oLYx0Lq
2puIftzxUUw9Q6pFJJyXSvlsvHy3qUF7HiMVdpW61ItIChV1xBDVKEPAacNzsZh4
30CI7kfJMfj6u+hpOCVlLk4uJFjZkmIpEKkDpEBemxLMME4JsLJdawKzKhjT2PI7
dWMjYkOeD4NkAzQLQGskEswoIgZQ0twuyPUErjEL9fcXw4OjxFvQJG85FsIF2lR6
FUDQg5y9YLzeMJMsjW4rO+LAz2c1mJwYR1EgYP43avm/pJfd1mVQLGRoLb7NwMSw
6mkwhJ4Kvq6BN0PSqpKqQtXZrDoElWN8cVJVf+dAjONcvzYi0gsHWDL+FZ731Q2M
s4nq0aRscBTL2DOaE9DzBY2AO1jKUB/+02qRpidWTYBmsmL2QQI8n33Q7JuDuEXX
bVm1RDA4ike4PUXXY5KJ6MZhKID5453SVFausFse+u4MCQHQPFYspkXmaNWRhjgs
yu2zPc9jHdBkpzNov/CCZoFketFRz/BKexBeH2vcfTYfREVf9lEZi7qEa0kQHDn9
EMTFsCqmGat9TEVbt9t8c/tODTeRE00MFx4gPspzy+m4YP+Gl3ySHsAbPQ90uBGX
c8xggwqWXr1GAP5HbAhs/Bs7USrWMMgqii1ppnzoAkHh+j4rsdL4dS2dmhxX756u
IKP/JC2oA8U=
=mV8z
-----END PGP SIGNATURE-----

17
SOURCES/nbdkit-1.24.0.tar.gz.sig Normal file
View File

@ -0,0 +1,17 @@
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEE93dPsa0HSn6Mh2fqkXOPc+G3aKAFAl/3RBgRHHJpY2hAYW5u
ZXhpYS5vcmcACgkQkXOPc+G3aKBIIRAAmgoGrmJ8aYO7z+kKgNFjd/p0QxRTZhS/
ol59ojG6jIzN2x/C2PFbRmPB6HJTEg4anrDX04WrP6R+lID1RrH9pTFQabv0YDQC
z49oeXAqINYHvAqgFUJCwlymd7BHEYUudLlK3yu7gQKxMM+J/2v0glpxrtLM7KlD
vvSZkVfbvHlCWIbMWLWIaRHeoWZIXNOjsAp3uEWN2YgikDoxbXVKoh07JoQx5tJ5
2U+a/zo4BQuRspjnhmWc252ZF/8d954/L8J+2mKvbRRf2iAmsqPgS+MNi7WKWO4K
w7/urKn0osuOaArs5xYHJnApmJ9U88CzZpoHQkYhcGgnDOipW9ByJRzT41vVQPW5
IluQODpZUuawWtRIwV/Eoi+LaV2gINAL48Afr02UFYj4gmYQ5TeayLP7NKRQO0VL
jwL4Z3a0cDyUX4i1OArn2ll8THfiog38HfLb70AG1l3P1BVoVVBYWCYbs4xgC9IK
LWkjPKuGXvkGVfZi0nCGdPTOoB1CqCXUvKHXm52FCHg12uJMrBQEivodBoCTbtl0
fSjULQcfrovUEb4d/rDAX7EgJbFS+1jDnodaFHsmNToo3CqfkMBdhLkxG3XExwjy
OOR34wZssjTLsLlWH/RPucWD25RDy1vdPBska9QvvO7W0p+aOtFbnttkTh5cqs45
rHg/sDEiaLA=
=OrsS
-----END PGP SIGNATURE-----

771
SPECS/nbdkit.spec
View File

File diff suppressed because it is too large Load Diff