From b75fe3d680e0295303b0b256afe546d23d744265 Mon Sep 17 00:00:00 2001 From: Miroslav Rezanina Date: Mon, 30 Mar 2026 16:45:07 +0200 Subject: [PATCH] * Mon Mar 30 2026 Miroslav Rezanina - 10.1.0-16 - kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch [RHEL-158224] - kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch [RHEL-158224] - kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch [RHEL-158224] - kvm-io-uring-Resubmit-tails-of-short-writes.patch [RHEL-158224] - Resolves: RHEL-158224 (qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2]) --- ...void-potentially-getting-stuck-after.patch | 167 +++++++++++ ...uring-Resubmit-tails-of-short-writes.patch | 278 ++++++++++++++++++ ...-Put-all-parameters-into-qemu_laiocb.patch | 127 ++++++++ ...Resubmit-tails-of-short-reads-writes.patch | 215 ++++++++++++++ qemu-kvm.spec | 18 +- 5 files changed, 804 insertions(+), 1 deletion(-) create mode 100644 kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch create mode 100644 kvm-io-uring-Resubmit-tails-of-short-writes.patch create mode 100644 kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch create mode 100644 kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch diff --git a/kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch b/kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch new file mode 100644 index 0000000..918862f --- /dev/null +++ b/kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch @@ -0,0 +1,167 @@ +From 4ad89aa6a9efcbc0420e332acab2dd06e55be2fa Mon Sep 17 00:00:00 2001 +From: Fiona Ebner +Date: Tue, 25 Nov 2025 14:31:03 +0100 +Subject: [PATCH 3/4] block/io_uring: avoid potentially getting stuck after + resubmit at the end of ioq_submit() + +RH-Author: Hanna Czenczek +RH-MergeRequest: 479: linux-aio/io-uring: Resubmit tails of short requests +RH-Jira: RHEL-158224 +RH-Acked-by: Kevin Wolf +RH-Acked-by: Stefan Hajnoczi +RH-Commit: [3/4] fd599da3ffcd6b37ceff35587ae9dbc1698b0f57 (hreitz/qemu-kvm-c-9-s) + +Note that this issue seems already fixed as a consequence of the large +io_uring rework with 047dabef97 ("block/io_uring: use aio_add_sqe()") +in current master, so this is purely for QEMU stable branches. + +At the end of ioq_submit(), there is an opportunistic call to +luring_process_completions(). This is the single caller of +luring_process_completions() that doesn't use the +luring_process_completions_and_submit() wrapper. + +Other callers use the wrapper, because luring_process_completions() +might require a subsequent call to ioq_submit() after resubmitting a +request. As noted for luring_resubmit(): + +> Resubmit a request by appending it to submit_queue. The caller must ensure +> that ioq_submit() is called later so that submit_queue requests are started. + +So the caller at the end of ioq_submit() violates the contract and can +in fact be problematic if no other requests come in later. In such a +case, the request intended to be resubmitted will never be actually be +submitted via io_uring_submit(). + +A reproducer exposing this issue is [0], which is based on user +reports from [1]. Another reproducer is iotest 109 with '-i io_uring'. + +I had the most success to trigger the issue with [0] when using a +BTRFS RAID 1 storage. With tmpfs, it can take quite a few iterations, +but also triggers eventually on my machine. With iotest 109 with '-i +io_uring' the issue triggers reliably on my ext4 file system. + +Have ioq_submit() submit any resubmitted requests after calling +luring_process_completions(). The return value from io_uring_submit() +is checked to be non-negative before the opportunistic processing of +completions and going for the new resubmit logic, to ensure that a +failure of io_uring_submit() is not missed. Also note that the return +value already was not necessarily the total number of submissions, +since the loop might've been iterated more than once even before the +current change. + +Only trigger the resubmission logic if it is actually necessary to +avoid changing behavior more than necessary. For example iotest 109 +would produce more 'mirror ready' events if always resubmitting after +luring_process_completions() at the end of ioq_submit(). + +Note iotest 109 still does not pass as is when run with '-i io_uring', +because of two offset values for BLOCK_JOB_COMPLETED events being zero +instead of non-zero as in the expected output. Note that the two +affected test cases are expected failures and still fail, so they just +fail "faster". The test cases are actually not triggering the resubmit +logic, so the reason seems to be different ordering of requests and +completions of the current aio=io_uring implementation versus +aio=threads. + +[0]: + +> #!/bin/bash -e +> #file=/mnt/btrfs/disk.raw +> file=/tmp/disk.raw +> filesize=256 +> readsize=512 +> rm -f $file +> truncate -s $filesize $file +> ./qemu-system-x86_64 --trace '*uring*' --qmp stdio \ +> --blockdev raw,node-name=node0,file.driver=file,file.cache.direct=off,file.filename=$file,file.aio=io_uring \ +> < {"execute": "qmp_capabilities"} +> {"execute": "human-monitor-command", "arguments": { "command-line": "qemu-io node0 \"read 0 $readsize \"" }} +> {"execute": "quit"} +> EOF + +[1]: https://forum.proxmox.com/threads/170045/ + +Cc: qemu-stable@nongnu.org +Signed-off-by: Fiona Ebner +Reviewed-by: Stefan Hajnoczi +Signed-off-by: Michael Tokarev +(cherry picked from commit 2bb0153cd806b8f6b4f82b353bd0113cd1c488a5) +Signed-off-by: Hanna Czenczek +--- + block/io_uring.c | 16 +++++++++++++--- + 1 file changed, 13 insertions(+), 3 deletions(-) + +diff --git a/block/io_uring.c b/block/io_uring.c +index dd4f304910..5dbafc8f7b 100644 +--- a/block/io_uring.c ++++ b/block/io_uring.c +@@ -120,11 +120,14 @@ static void luring_resubmit_short_read(LuringState *s, LuringAIOCB *luringcb, + * event loop. When there are no events left to complete the BH is being + * canceled. + * ++ * Returns whether ioq_submit() must be called again afterwards since requests ++ * were resubmitted via luring_resubmit(). + */ +-static void luring_process_completions(LuringState *s) ++static bool luring_process_completions(LuringState *s) + { + struct io_uring_cqe *cqes; + int total_bytes; ++ bool resubmit = false; + + defer_call_begin(); + +@@ -182,6 +185,7 @@ static void luring_process_completions(LuringState *s) + */ + if (ret == -EINTR || ret == -EAGAIN) { + luring_resubmit(s, luringcb); ++ resubmit = true; + continue; + } + } else if (!luringcb->qiov) { +@@ -194,6 +198,7 @@ static void luring_process_completions(LuringState *s) + if (luringcb->is_read) { + if (ret > 0) { + luring_resubmit_short_read(s, luringcb, ret); ++ resubmit = true; + continue; + } else { + /* Pad with zeroes */ +@@ -224,6 +229,8 @@ end: + qemu_bh_cancel(s->completion_bh); + + defer_call_end(); ++ ++ return resubmit; + } + + static int ioq_submit(LuringState *s) +@@ -231,6 +238,7 @@ static int ioq_submit(LuringState *s) + int ret = 0; + LuringAIOCB *luringcb, *luringcb_next; + ++resubmit: + while (s->io_q.in_queue > 0) { + /* + * Try to fetch sqes from the ring for requests waiting in +@@ -260,12 +268,14 @@ static int ioq_submit(LuringState *s) + } + s->io_q.blocked = (s->io_q.in_queue > 0); + +- if (s->io_q.in_flight) { ++ if (ret >= 0 && s->io_q.in_flight) { + /* + * We can try to complete something just right away if there are + * still requests in-flight. + */ +- luring_process_completions(s); ++ if (luring_process_completions(s)) { ++ goto resubmit; ++ } + } + return ret; + } +-- +2.47.3 + diff --git a/kvm-io-uring-Resubmit-tails-of-short-writes.patch b/kvm-io-uring-Resubmit-tails-of-short-writes.patch new file mode 100644 index 0000000..378e5af --- /dev/null +++ b/kvm-io-uring-Resubmit-tails-of-short-writes.patch @@ -0,0 +1,278 @@ +From 1bdac2a8c4ac77133cb0c2b4d40819bff1a35fc4 Mon Sep 17 00:00:00 2001 +From: Hanna Czenczek +Date: Tue, 24 Mar 2026 09:43:36 +0100 +Subject: [PATCH 4/4] io-uring: Resubmit tails of short writes + +RH-Author: Hanna Czenczek +RH-MergeRequest: 479: linux-aio/io-uring: Resubmit tails of short requests +RH-Jira: RHEL-158224 +RH-Acked-by: Kevin Wolf +RH-Acked-by: Stefan Hajnoczi +RH-Commit: [4/4] 7ff66622acbb5dbdbacafe71bffd0a277ac919d9 (hreitz/qemu-kvm-c-9-s) + +Short writes can happen, too, not just short reads. The difference to +aio=native is that the kernel will actually retry the tail of short +requests internally already -- so it is harder to reproduce. But if the +tail of a short request returns an error to the kernel, we will see it +in userspace still. To reproduce this, apply the following patch on top +of the one shown in HEAD^ (again %s/escaped // to apply): + +escaped diff --git a/block/export/fuse.c b/block/export/fuse.c +escaped index 67dc50a412..2b98489a32 100644 +escaped --- a/block/export/fuse.c +escaped +++ b/block/export/fuse.c +@@ -1059,8 +1059,15 @@ fuse_co_read(FuseExport *exp, void **bufptr, uint64_t offset, uint32_t size) + int64_t blk_len; + void *buf; + int ret; ++ static uint32_t error_size; + +- size = MIN(size, 4096); ++ if (error_size == size) { ++ error_size = 0; ++ return -EIO; ++ } else if (size > 4096) { ++ error_size = size - 4096; ++ size = 4096; ++ } + + /* Limited by max_read, should not happen */ + if (size > FUSE_MAX_READ_BYTES) { +@@ -1111,8 +1118,15 @@ fuse_co_write(FuseExport *exp, struct fuse_write_out *out, + { + int64_t blk_len; + int ret; ++ static uint32_t error_size; + +- size = MIN(size, 4096); ++ if (error_size == size) { ++ error_size = 0; ++ return -EIO; ++ } else if (size > 4096) { ++ error_size = size - 4096; ++ size = 4096; ++ } + + QEMU_BUILD_BUG_ON(FUSE_MAX_WRITE_BYTES > BDRV_REQUEST_MAX_BYTES); + /* Limited by max_write, should not happen */ + +I know this is a bit artificial because to produce this, there must be +an I/O error somewhere anyway, but if it does happen, qemu will +understand it to mean ENOSPC for short writes, which is incorrect. So I +believe we need to resubmit the tail to maybe have it succeed now, or at +least get the correct error code. + +Reproducer as before: +$ ./qemu-img create -f raw test.raw 8k +Formatting 'test.raw', fmt=raw size=8192 +$ ./qemu-io -f raw -c 'write -P 42 0 8k' test.raw +wrote 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (64.804 MiB/sec and 8294.9003 ops/sec) +$ hexdump -C test.raw +00000000 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a |****************| +* +00002000 +$ storage-daemon/qemu-storage-daemon \ + --blockdev file,node-name=test,filename=test.raw \ + --export fuse,id=exp,node-name=test,mountpoint=test.raw,writable=true + +$ ./qemu-io --image-opts -c 'read -P 23 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=io_uring +read 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (58.481 MiB/sec and 7485.5342 ops/sec) +$ ./qemu-io --image-opts -c 'write -P 23 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=io_uring +write failed: No space left on device +$ hexdump -C test.raw +00000000 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 |................| +* +00001000 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a |****************| +* +00002000 + +So short reads already work (because there is code for that), but short +writes incorrectly produce ENOSPC. This patch fixes that by +resubmitting not only the tail of short reads but short writes also. + +(And this patch uses the opportunity to make it so qemu_iovec_destroy() +is called only if req->resubmit_qiov.iov is non-NULL. Functionally a +non-op, but this is how the code generally checks whether the +resubmit_qiov has been set up or not.) + +Reviewed-by: Kevin Wolf +Signed-off-by: Hanna Czenczek +Message-ID: <20260324084338.37453-4-hreitz@redhat.com> +Signed-off-by: Kevin Wolf +(cherry picked from commit cf9cdaea6e24d13dfdf8402f6829d2ca4dca864b) + +Conflicts: +- block/io_uring.c, block/trace-events + Missing 047dabef97bd0c4af3c3dc453b19e20345de3602 ("block/io_uring: use + aio_add_sqe()") downstream, which changed quite a few things about the + io-uring code. Backporting it seems excessive, though, especially + given that it would pull in even more dependencies (general aio and + aio-posix changes that come right before 047dabef). + +Signed-off-by: Hanna Czenczek +--- + block/io_uring.c | 77 +++++++++++++++++++++++++--------------------- + block/trace-events | 2 +- + 2 files changed, 43 insertions(+), 36 deletions(-) + +diff --git a/block/io_uring.c b/block/io_uring.c +index 5dbafc8f7b..582550f8e9 100644 +--- a/block/io_uring.c ++++ b/block/io_uring.c +@@ -31,14 +31,14 @@ typedef struct LuringAIOCB { + struct io_uring_sqe sqeq; + ssize_t ret; + QEMUIOVector *qiov; +- bool is_read; ++ int type; + QSIMPLEQ_ENTRY(LuringAIOCB) next; + + /* +- * Buffered reads may require resubmission, see +- * luring_resubmit_short_read(). ++ * Short reads/writes require resubmission, see ++ * luring_resubmit_short_io(). + */ +- int total_read; ++ int total_done; + QEMUIOVector resubmit_qiov; + } LuringAIOCB; + +@@ -73,22 +73,27 @@ static void luring_resubmit(LuringState *s, LuringAIOCB *luringcb) + } + + /** +- * luring_resubmit_short_read: ++ * luring_resubmit_short_io: + * +- * Short reads are rare but may occur. The remaining read request needs to be +- * resubmitted. ++ * Short reads and writes are rare but may occur. The remaining request needs ++ * to be resubmitted. ++ * ++ * For example, short reads can be reproduced by a FUSE export deliberately ++ * executing short reads. The tail of short writes is generally resubmitted by ++ * io-uring in the kernel, but if that resubmission encounters an I/O error, the ++ * already submitted portion will be returned as a short write. + */ +-static void luring_resubmit_short_read(LuringState *s, LuringAIOCB *luringcb, +- int nread) ++static void luring_resubmit_short_io(LuringState *s, LuringAIOCB *luringcb, ++ int ndone) + { + QEMUIOVector *resubmit_qiov; + size_t remaining; + +- trace_luring_resubmit_short_read(s, luringcb, nread); ++ trace_luring_resubmit_short_io(s, luringcb, ndone); + +- /* Update read position */ +- luringcb->total_read += nread; +- remaining = luringcb->qiov->size - luringcb->total_read; ++ /* Update I/O position */ ++ luringcb->total_done += ndone; ++ remaining = luringcb->qiov->size - luringcb->total_done; + + /* Shorten qiov */ + resubmit_qiov = &luringcb->resubmit_qiov; +@@ -97,11 +102,11 @@ static void luring_resubmit_short_read(LuringState *s, LuringAIOCB *luringcb, + } else { + qemu_iovec_reset(resubmit_qiov); + } +- qemu_iovec_concat(resubmit_qiov, luringcb->qiov, luringcb->total_read, ++ qemu_iovec_concat(resubmit_qiov, luringcb->qiov, luringcb->total_done, + remaining); + + /* Update sqe */ +- luringcb->sqeq.off += nread; ++ luringcb->sqeq.off += ndone; + luringcb->sqeq.addr = (uintptr_t)luringcb->resubmit_qiov.iov; + luringcb->sqeq.len = luringcb->resubmit_qiov.niov; + +@@ -165,8 +170,8 @@ static bool luring_process_completions(LuringState *s) + s->io_q.in_flight--; + trace_luring_process_completion(s, luringcb, ret); + +- /* total_read is non-zero only for resubmitted read requests */ +- total_bytes = ret + luringcb->total_read; ++ /* total_done is non-zero only for resubmitted requests */ ++ total_bytes = ret + luringcb->total_done; + + if (ret < 0) { + /* +@@ -192,27 +197,29 @@ static bool luring_process_completions(LuringState *s) + goto end; + } else if (total_bytes == luringcb->qiov->size) { + ret = 0; +- /* Only read/write */ ++ } else if (ret > 0 && (luringcb->type == QEMU_AIO_READ || ++ luringcb->type == QEMU_AIO_WRITE)) { ++ luring_resubmit_short_io(s, luringcb, ret); ++ resubmit = true; ++ continue; ++ } else if (luringcb->type == QEMU_AIO_READ) { ++ /* Read ret == 0: EOF, pad with zeroes */ ++ qemu_iovec_memset(luringcb->qiov, total_bytes, 0, ++ luringcb->qiov->size - total_bytes); ++ ret = 0; + } else { +- /* Short Read/Write */ +- if (luringcb->is_read) { +- if (ret > 0) { +- luring_resubmit_short_read(s, luringcb, ret); +- resubmit = true; +- continue; +- } else { +- /* Pad with zeroes */ +- qemu_iovec_memset(luringcb->qiov, total_bytes, 0, +- luringcb->qiov->size - total_bytes); +- ret = 0; +- } +- } else { +- ret = -ENOSPC; +- } ++ /* ++ * Normal write ret == 0 means ENOSPC. ++ * For zone-append, we treat any 0 <= ret < qiov->size as ENOSPC, ++ * too, because resubmitting the tail seems a little unsafe. ++ */ ++ ret = -ENOSPC; + } + end: + luringcb->ret = ret; +- qemu_iovec_destroy(&luringcb->resubmit_qiov); ++ if (luringcb->resubmit_qiov.iov) { ++ qemu_iovec_destroy(&luringcb->resubmit_qiov); ++ } + + /* + * If the coroutine is already entered it must be in ioq_submit() +@@ -409,7 +416,7 @@ int coroutine_fn luring_co_submit(BlockDriverState *bs, int fd, uint64_t offset, + .co = qemu_coroutine_self(), + .ret = -EINPROGRESS, + .qiov = qiov, +- .is_read = (type == QEMU_AIO_READ), ++ .type = type, + }; + trace_luring_co_submit(bs, s, &luringcb, fd, offset, qiov ? qiov->size : 0, + type); +diff --git a/block/trace-events b/block/trace-events +index 8e789e1f12..99b8c12bc8 100644 +--- a/block/trace-events ++++ b/block/trace-events +@@ -70,7 +70,7 @@ luring_do_submit_done(void *s, int ret) "LuringState %p submitted to kernel %d" + luring_co_submit(void *bs, void *s, void *luringcb, int fd, uint64_t offset, size_t nbytes, int type) "bs %p s %p luringcb %p fd %d offset %" PRId64 " nbytes %zd type %d" + luring_process_completion(void *s, void *aiocb, int ret) "LuringState %p luringcb %p ret %d" + luring_io_uring_submit(void *s, int ret) "LuringState %p ret %d" +-luring_resubmit_short_read(void *s, void *luringcb, int nread) "LuringState %p luringcb %p nread %d" ++luring_resubmit_short_io(void *s, void *luringcb, int ndone) "LuringState %p luringcb %p ndone %d" + + # qcow2.c + qcow2_add_task(void *co, void *bs, void *pool, const char *action, int cluster_type, uint64_t host_offset, uint64_t offset, uint64_t bytes, void *qiov, size_t qiov_offset) "co %p bs %p pool %p: %s: cluster_type %d file_cluster_offset %" PRIu64 " offset %" PRIu64 " bytes %" PRIu64 " qiov %p qiov_offset %zu" +-- +2.47.3 + diff --git a/kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch b/kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch new file mode 100644 index 0000000..b458e78 --- /dev/null +++ b/kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch @@ -0,0 +1,127 @@ +From 30c3962bbef6ce083f326988f35741dd21aaf09d Mon Sep 17 00:00:00 2001 +From: Hanna Czenczek +Date: Tue, 24 Mar 2026 09:43:34 +0100 +Subject: [PATCH 1/4] linux-aio: Put all parameters into qemu_laiocb + +RH-Author: Hanna Czenczek +RH-MergeRequest: 479: linux-aio/io-uring: Resubmit tails of short requests +RH-Jira: RHEL-158224 +RH-Acked-by: Kevin Wolf +RH-Acked-by: Stefan Hajnoczi +RH-Commit: [1/4] f449ea0c49a093bbec59b24fb44308e7f58c9ed2 (hreitz/qemu-kvm-c-9-s) + +Put all request parameters into the qemu_laiocb struct, which will allow +re-submitting the tail of short reads/writes. + +Reviewed-by: Kevin Wolf +Signed-off-by: Hanna Czenczek +Message-ID: <20260324084338.37453-2-hreitz@redhat.com> +Signed-off-by: Kevin Wolf +(cherry picked from commit cc03b62df47a09c507e199cc043f57bdc941cc67) +Signed-off-by: Hanna Czenczek +--- + block/linux-aio.c | 34 ++++++++++++++++++++++------------ + 1 file changed, 22 insertions(+), 12 deletions(-) + +diff --git a/block/linux-aio.c b/block/linux-aio.c +index c200e7ad20..c2c5e11946 100644 +--- a/block/linux-aio.c ++++ b/block/linux-aio.c +@@ -41,9 +41,15 @@ struct qemu_laiocb { + LinuxAioState *ctx; + struct iocb iocb; + ssize_t ret; ++ off_t offset; + size_t nbytes; + QEMUIOVector *qiov; +- bool is_read; ++ ++ int fd; ++ int type; ++ BdrvRequestFlags flags; ++ ++ uint64_t dev_max_batch; + QSIMPLEQ_ENTRY(qemu_laiocb) next; + }; + +@@ -87,7 +93,7 @@ static void qemu_laio_process_completion(struct qemu_laiocb *laiocb) + ret = 0; + } else if (ret >= 0) { + /* Short reads mean EOF, pad with zeros. */ +- if (laiocb->is_read) { ++ if (laiocb->type == QEMU_AIO_READ) { + qemu_iovec_memset(laiocb->qiov, ret, 0, + laiocb->qiov->size - ret); + } else { +@@ -367,23 +373,23 @@ static void laio_deferred_fn(void *opaque) + } + } + +-static int laio_do_submit(int fd, struct qemu_laiocb *laiocb, off_t offset, +- int type, BdrvRequestFlags flags, +- uint64_t dev_max_batch) ++static int laio_do_submit(struct qemu_laiocb *laiocb) + { + LinuxAioState *s = laiocb->ctx; + struct iocb *iocbs = &laiocb->iocb; + QEMUIOVector *qiov = laiocb->qiov; ++ int fd = laiocb->fd; ++ off_t offset = laiocb->offset; + +- switch (type) { ++ switch (laiocb->type) { + case QEMU_AIO_WRITE: + #ifdef HAVE_IO_PREP_PWRITEV2 + { +- int laio_flags = (flags & BDRV_REQ_FUA) ? RWF_DSYNC : 0; ++ int laio_flags = (laiocb->flags & BDRV_REQ_FUA) ? RWF_DSYNC : 0; + io_prep_pwritev2(iocbs, fd, qiov->iov, qiov->niov, offset, laio_flags); + } + #else +- assert(flags == 0); ++ assert(laiocb->flags == 0); + io_prep_pwritev(iocbs, fd, qiov->iov, qiov->niov, offset); + #endif + break; +@@ -399,7 +405,7 @@ static int laio_do_submit(int fd, struct qemu_laiocb *laiocb, off_t offset, + /* Currently Linux kernel does not support other operations */ + default: + fprintf(stderr, "%s: invalid AIO request type 0x%x.\n", +- __func__, type); ++ __func__, laiocb->type); + return -EIO; + } + io_set_eventfd(&laiocb->iocb, event_notifier_get_fd(&s->e)); +@@ -407,7 +413,7 @@ static int laio_do_submit(int fd, struct qemu_laiocb *laiocb, off_t offset, + QSIMPLEQ_INSERT_TAIL(&s->io_q.pending, laiocb, next); + s->io_q.in_queue++; + if (!s->io_q.blocked) { +- if (s->io_q.in_queue >= laio_max_batch(s, dev_max_batch)) { ++ if (s->io_q.in_queue >= laio_max_batch(s, laiocb->dev_max_batch)) { + ioq_submit(s); + } else { + defer_call(laio_deferred_fn, s); +@@ -425,14 +431,18 @@ int coroutine_fn laio_co_submit(int fd, uint64_t offset, QEMUIOVector *qiov, + AioContext *ctx = qemu_get_current_aio_context(); + struct qemu_laiocb laiocb = { + .co = qemu_coroutine_self(), ++ .offset = offset, + .nbytes = qiov ? qiov->size : 0, + .ctx = aio_get_linux_aio(ctx), + .ret = -EINPROGRESS, +- .is_read = (type == QEMU_AIO_READ), + .qiov = qiov, ++ .fd = fd, ++ .type = type, ++ .flags = flags, ++ .dev_max_batch = dev_max_batch, + }; + +- ret = laio_do_submit(fd, &laiocb, offset, type, flags, dev_max_batch); ++ ret = laio_do_submit(&laiocb); + if (ret < 0) { + return ret; + } +-- +2.47.3 + diff --git a/kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch b/kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch new file mode 100644 index 0000000..2da3b76 --- /dev/null +++ b/kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch @@ -0,0 +1,215 @@ +From ad3a6c9b3487226d9622120eaea8218bd6050a04 Mon Sep 17 00:00:00 2001 +From: Hanna Czenczek +Date: Tue, 24 Mar 2026 09:43:35 +0100 +Subject: [PATCH 2/4] linux-aio: Resubmit tails of short reads/writes + +RH-Author: Hanna Czenczek +RH-MergeRequest: 479: linux-aio/io-uring: Resubmit tails of short requests +RH-Jira: RHEL-158224 +RH-Acked-by: Kevin Wolf +RH-Acked-by: Stefan Hajnoczi +RH-Commit: [2/4] f97271e609a150acc04a15ffc85c40e7bcb00060 (hreitz/qemu-kvm-c-9-s) + +Short reads/writes can happen. One way to reproduce them is via our +FUSE export, with the following diff applied (%s/escaped // to apply -- +if you put plain diffs in commit messages, git-am will apply them, and I +would rather avoid breaking FUSE accidentally via this patch): + +escaped diff --git a/block/export/fuse.c b/block/export/fuse.c +escaped index a2a478d293..67dc50a412 100644 +escaped --- a/block/export/fuse.c +escaped +++ b/block/export/fuse.c +@@ -828,7 +828,7 @@ static ssize_t coroutine_fn GRAPH_RDLOCK + fuse_co_init(FuseExport *exp, struct fuse_init_out *out, + const struct fuse_init_in_compat *in) + { +- const uint32_t supported_flags = FUSE_ASYNC_READ | FUSE_ASYNC_DIO; ++ const uint32_t supported_flags = FUSE_ASYNC_READ; + + if (in->major != 7) { + error_report("FUSE major version mismatch: We have 7, but kernel has %" +@@ -1060,6 +1060,8 @@ fuse_co_read(FuseExport *exp, void **bufptr, uint64_t offset, uint32_t size) + void *buf; + int ret; + ++ size = MIN(size, 4096); ++ + /* Limited by max_read, should not happen */ + if (size > FUSE_MAX_READ_BYTES) { + return -EINVAL; +@@ -1110,6 +1112,8 @@ fuse_co_write(FuseExport *exp, struct fuse_write_out *out, + int64_t blk_len; + int ret; + ++ size = MIN(size, 4096); ++ + QEMU_BUILD_BUG_ON(FUSE_MAX_WRITE_BYTES > BDRV_REQUEST_MAX_BYTES); + /* Limited by max_write, should not happen */ + if (size > FUSE_MAX_WRITE_BYTES) { + +Then: +$ ./qemu-img create -f raw test.raw 8k +Formatting 'test.raw', fmt=raw size=8192 +$ ./qemu-io -f raw -c 'write -P 42 0 8k' test.raw +wrote 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (64.804 MiB/sec and 8294.9003 ops/sec) +$ hexdump -C test.raw +00000000 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a |****************| +* +00002000 + +With aio=threads, short I/O works: +$ storage-daemon/qemu-storage-daemon \ + --blockdev file,node-name=test,filename=test.raw \ + --export fuse,id=exp,node-name=test,mountpoint=test.raw,writable=true + +Other shell: +$ ./qemu-io --image-opts -c 'read -P 42 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=threads +read 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (36.563 MiB/sec and 4680.0923 ops/sec) +$ ./qemu-io --image-opts -c 'write -P 23 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=threads +wrote 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (35.995 MiB/sec and 4607.2970 ops/sec) +$ hexdump -C test.raw +00000000 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 |................| +* +00002000 + +But with aio=native, it does not: +$ ./qemu-io --image-opts -c 'read -P 23 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=native +Pattern verification failed at offset 0, 8192 bytes +read 8192/8192 bytes at offset 0 +8 KiB, 1 ops; 00.00 sec (86.155 MiB/sec and 11027.7900 ops/sec) +$ ./qemu-io --image-opts -c 'write -P 42 0 8k' \ + driver=file,filename=test.raw,cache.direct=on,aio=native +write failed: No space left on device +$ hexdump -C test.raw +00000000 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a 2a |****************| +* +00001000 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 17 |................| +* +00002000 + +This patch fixes that. + +Reviewed-by: Kevin Wolf +Signed-off-by: Hanna Czenczek +Message-ID: <20260324084338.37453-3-hreitz@redhat.com> +Signed-off-by: Kevin Wolf +(cherry picked from commit 7eca3d4883be8d328377001a9ea7ae9882b00f3c) +Signed-off-by: Hanna Czenczek +--- + block/linux-aio.c | 56 ++++++++++++++++++++++++++++++++++++++++++----- + 1 file changed, 50 insertions(+), 6 deletions(-) + +diff --git a/block/linux-aio.c b/block/linux-aio.c +index c2c5e11946..84397de54c 100644 +--- a/block/linux-aio.c ++++ b/block/linux-aio.c +@@ -45,6 +45,10 @@ struct qemu_laiocb { + size_t nbytes; + QEMUIOVector *qiov; + ++ /* For handling short reads/writes */ ++ size_t total_done; ++ QEMUIOVector resubmit_qiov; ++ + int fd; + int type; + BdrvRequestFlags flags; +@@ -74,28 +78,61 @@ struct LinuxAioState { + }; + + static void ioq_submit(LinuxAioState *s); ++static int laio_do_submit(struct qemu_laiocb *laiocb); + + static inline ssize_t io_event_ret(struct io_event *ev) + { + return (ssize_t)(((uint64_t)ev->res2 << 32) | ev->res); + } + ++/** ++ * Retry tail of short requests. ++ */ ++static int laio_resubmit_short_io(struct qemu_laiocb *laiocb, size_t done) ++{ ++ QEMUIOVector *resubmit_qiov = &laiocb->resubmit_qiov; ++ ++ laiocb->total_done += done; ++ ++ if (!resubmit_qiov->iov) { ++ qemu_iovec_init(resubmit_qiov, laiocb->qiov->niov); ++ } else { ++ qemu_iovec_reset(resubmit_qiov); ++ } ++ qemu_iovec_concat(resubmit_qiov, laiocb->qiov, ++ laiocb->total_done, laiocb->nbytes - laiocb->total_done); ++ ++ return laio_do_submit(laiocb); ++} ++ + /* + * Completes an AIO request. + */ + static void qemu_laio_process_completion(struct qemu_laiocb *laiocb) + { +- int ret; ++ ssize_t ret; + + ret = laiocb->ret; + if (ret != -ECANCELED) { +- if (ret == laiocb->nbytes) { ++ if (ret == laiocb->nbytes - laiocb->total_done) { + ret = 0; ++ } else if (ret > 0 && (laiocb->type == QEMU_AIO_READ || ++ laiocb->type == QEMU_AIO_WRITE)) { ++ ret = laio_resubmit_short_io(laiocb, ret); ++ if (!ret) { ++ return; ++ } + } else if (ret >= 0) { +- /* Short reads mean EOF, pad with zeros. */ ++ /* ++ * For normal reads and writes, we only get here if ret == 0, which ++ * means EOF for reads and ENOSPC for writes. ++ * For zone-append, we get here with any ret >= 0, which we just ++ * treat as ENOSPC, too (safer than resubmitting, probably, but not ++ * 100 % clear). ++ */ + if (laiocb->type == QEMU_AIO_READ) { +- qemu_iovec_memset(laiocb->qiov, ret, 0, +- laiocb->qiov->size - ret); ++ qemu_iovec_memset(laiocb->qiov, laiocb->total_done, 0, ++ laiocb->qiov->size - laiocb->total_done); + } else { + ret = -ENOSPC; + } +@@ -103,6 +140,9 @@ static void qemu_laio_process_completion(struct qemu_laiocb *laiocb) + } + + laiocb->ret = ret; ++ if (laiocb->resubmit_qiov.iov) { ++ qemu_iovec_destroy(&laiocb->resubmit_qiov); ++ } + + /* + * If the coroutine is already entered it must be in ioq_submit() and +@@ -379,7 +419,11 @@ static int laio_do_submit(struct qemu_laiocb *laiocb) + struct iocb *iocbs = &laiocb->iocb; + QEMUIOVector *qiov = laiocb->qiov; + int fd = laiocb->fd; +- off_t offset = laiocb->offset; ++ off_t offset = laiocb->offset + laiocb->total_done; ++ ++ if (laiocb->resubmit_qiov.iov) { ++ qiov = &laiocb->resubmit_qiov; ++ } + + switch (laiocb->type) { + case QEMU_AIO_WRITE: +-- +2.47.3 + diff --git a/qemu-kvm.spec b/qemu-kvm.spec index 40ac1d0..7cd5d33 100644 --- a/qemu-kvm.spec +++ b/qemu-kvm.spec @@ -143,7 +143,7 @@ Obsoletes: %{name}-block-ssh <= %{epoch}:%{version} \ Summary: QEMU is a machine emulator and virtualizer Name: qemu-kvm Version: 10.1.0 -Release: 15%{?rcrel}%{?dist}%{?cc_suffix} +Release: 16%{?rcrel}%{?dist}%{?cc_suffix} # Epoch because we pushed a qemu-1.0 package. AIUI this can't ever be dropped # Epoch 15 used for RHEL 8 # Epoch 17 used for RHEL 9 (due to release versioning offset in RHEL 8.5) @@ -414,6 +414,14 @@ Patch128: kvm-hw-uefi-add-variable-digest-to-vmstate.patch Patch129: kvm-block-Never-drop-BLOCK_IO_ERROR-with-action-stop-for.patch # For RHEL-155601 - Mirror job can miss writes during startup, corrupting the copy [rhel-10.2] Patch130: kvm-mirror-Fix-missed-dirty-bitmap-writes-during-startup.patch +# For RHEL-158224 - qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2] +Patch131: kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch +# For RHEL-158224 - qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2] +Patch132: kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch +# For RHEL-158224 - qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2] +Patch133: kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch +# For RHEL-158224 - qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2] +Patch134: kvm-io-uring-Resubmit-tails-of-short-writes.patch %if %{have_clang} BuildRequires: clang @@ -1493,6 +1501,14 @@ useradd -r -u 107 -g qemu -G kvm -d / -s /sbin/nologin \ %endif %changelog +* Mon Mar 30 2026 Miroslav Rezanina - 10.1.0-16 +- kvm-linux-aio-Put-all-parameters-into-qemu_laiocb.patch [RHEL-158224] +- kvm-linux-aio-Resubmit-tails-of-short-reads-writes.patch [RHEL-158224] +- kvm-block-io_uring-avoid-potentially-getting-stuck-after.patch [RHEL-158224] +- kvm-io-uring-Resubmit-tails-of-short-writes.patch [RHEL-158224] +- Resolves: RHEL-158224 + (qemu-kvm: disk writes of fewer bytes than requested is a retry condition, not necessarily an indication of ENOSPC [rhel-10.2]) + * Thu Mar 26 2026 Miroslav Rezanina - 10.1.0-15 - kvm-mirror-Fix-missed-dirty-bitmap-writes-during-startup.patch [RHEL-155601] - Resolves: RHEL-155601