From fa6a07a8fd5cc3216eb53cd2ad54e9e0dea42036 Mon Sep 17 00:00:00 2001 From: Nir Soffer Date: Sun, 13 Apr 2025 23:39:15 +0000 Subject: [PATCH] copy: Enable zero optimization for allocated extents We optimized zero extents but computed the hash for all data blocks, including data blocks full of zeros. Detecting a zero block is 20-100 times faster than computing a hash, depending on the machine and the hash algorithm. When adding a completed block, detect zero blocks and mark the block as zero block, saving the computation of the hash and the allocation of the digest buffer. This optimization is already implemented for incomplete blocks. Testing shows that computing a hash for image full of zeros is up to 7.4 times faster, and memory usage is up to 40% lower. | size | content | tool | source | version | memory | time | |--------|---------|------------|--------|---------|----------|----------| | 10g | zero | nbdcopy | file | before | 20236k | 1.33s | | 10g | zero | nbdcopy | file | after | 13212k | 0.33s | | 10g | zero | nbdcopy | nbd | before | 32648k | 8.21s | | 10g | zero | nbdcopy | nbd | after | 24996k | 3.32s | | 10g | zero | nbdcopy | pipe | before | 19052k | 4.56s | | 10g | zero | nbdcopy | pipe | after | 11244k | 0.61s | | 10g | zero | blksum | nbd | - | 13948k | 3.90s | | 10g | zero | blksum | pipe | - | 10340k | 0.55s | | 10g | zero | sha256sum | file | - | 2796k | 4.45s | |--------|---------|------------|--------|---------|----------|----------| | 10g | data | nbdcopy | file | before | 20224k | 1.28s | | 10g | data | nbdcopy | file | after | 20400k | 1.28s | | 10g | data | nbdcopy | nbd | before | 32792k | 8.02s | | 10g | data | nbdcopy | nbd | after | 32536k | 8.01s | | 10g | data | nbdcopy | pipe | before | 19052k | 4.56s | | 10g | data | nbdcopy | pipe | after | 19048k | 4.55s | | 10g | data | blksum | nbd | - | 13888k | 3.88s | | 10g | data | blksum | pipe | - | 12512k | 1.10s | | 10g | data | sha256sum | file | - | 2788k | 4.49s | (cherry picked from commit efbe283f9fcfc8b4e57370f71356b1bfe7ffd0a4) --- copy/blkhash.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/copy/blkhash.c b/copy/blkhash.c index 41253ec8..92ffafbd 100644 --- a/copy/blkhash.c +++ b/copy/blkhash.c @@ -213,7 +213,10 @@ set_complete_block (uint64_t blknum, const char *buf) /* Assert that we haven't seen this block before. */ assert (b.type == block_unknown); - if (buf) { + /* Detecting a zero block is 20-100 times faster than computing a hash + * depending on the machine and the algorithm. + */ + if (buf && !is_zero (buf, blkhash_size)) { b.type = block_data; /* Compute the hash of the whole block now. */ -- 2.47.1