From ac59d3b2310511b1537d408b675b19ec9a5d384e Mon Sep 17 00:00:00 2001 From: "Richard W.M. Jones" Date: Tue, 7 Dec 2021 12:53:41 +0000 Subject: [PATCH] output: Don't use nbdkit-file-plugin cache=none when writing nbdkit-file-plugin flag cache=none is meant to ensure that the file plugin does not pollute the page cache. However it doesn't yet work very well, especially for writing. As you can see here: https://gitlab.com/nbdkit/nbdkit/-/blob/048d5b9818c88355e596824355269773e5c4f6ad/plugins/file/file.c#L594 it is currently implemented by flushing the whole file (including parallel requests), and then evicting the data from the page cache. This implementation is naive. (See nbdcopy copy/file-ops.c for a much better implementation.) This causes quite a considerable slow down when writing to a local file (eg -os local). The penalty varies between machines and possibly kernels. On my Fedora Rawhide AMD server with 12 cores and SSDs, the slow down is under 5%. But on my RHEL 9 Intel NUC server with 2 cores and a hard disk, it makes a really huge difference, nearly doubling the total time taken. I left a note in the code that we should fix this in nbdkit and re-add this option or similar when it is working. I also removed the fadvise=sequential option. This is associated with cache=none in the nbdkit documentation, and anyway we are not doing sequential writes. (This makes no measurable difference in my testing) --- output/output.ml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/output/output.ml b/output/output.ml index 7e976508..f829fe43 100644 --- a/output/output.ml +++ b/output/output.ml @@ -98,8 +98,14 @@ let output_to_local_file ?(changeuid = fun f -> f ()) let cmd = Nbdkit.add_arg cmd "file" filename in let cmd = if Nbdkit.version nbdkit_config >= (1, 22, 0) then ( - let cmd = Nbdkit.add_arg cmd "fadvise" "sequential" in - let cmd = Nbdkit.add_arg cmd "cache" "none" in + (* nbdkit 1.28 has a very naive implementation of + * page cache eviction. We need to copy the one from + * nbdcopy copy/file-ops.c. Until then do not use + * this flag because it causes a large slow down on + * some machines. XXX + *) + (*let cmd = Nbdkit.add_arg cmd "fadvise" "sequential" in + let cmd = Nbdkit.add_arg cmd "cache" "none" in*) cmd ) else cmd in -- 2.31.1