Compare commits

..

No commits in common. "c9-beta" and "c8" have entirely different histories.
c9-beta ... c8

3 changed files with 281 additions and 119 deletions

View File

@ -1,2 +1,2 @@
cf5c3d8a15c0666cc980a8cf7227ae711664f5a3 SOURCES/ldb-2.8.0.tar.gz cf5c3d8a15c0666cc980a8cf7227ae711664f5a3 SOURCES/ldb-2.8.0.tar.gz
8ac6d09878c4218fb8e365fcf5a877a621dd40f9 SOURCES/ldb.keyring 5d2957f5d63a72a6fc196af3e45242f3d321f6cf SOURCES/ldb.keyring

View File

@ -0,0 +1,221 @@
From 1944fcf4b7e5ab4cf580e17031918ba5f441902b Mon Sep 17 00:00:00 2001
From: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Date: Wed, 26 Jun 2024 11:05:49 +1200
Subject: [PATCH 1/2] ldb_kv_index: dn_list load sub transaction can re-use
keys
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
We don't want to modify the original list, but we can reuse the keys
if we treat them as immutable and don't free them. That makes it a lot
quicker if there are many keys (i.e. where an index is useful) and may
sub-transactions. In particular, it avoids O(n²) talloc_memdups.
A removed comment that says "We have to free the top level index
memory otherwise we would leak", and this will be addressed in the
next commit.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15590
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 5f0198d69843c864f2b98a7c0c6305ad789a68a0)
---
lib/ldb/ldb_key_value/ldb_kv_index.c | 96 +++++++++++++++++-----------
1 file changed, 57 insertions(+), 39 deletions(-)
diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
index 3f1a847f2b6..fed1033f492 100644
--- a/lib/ldb/ldb_key_value/ldb_kv_index.c
+++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
@@ -446,34 +446,39 @@ static int ldb_kv_dn_list_load(struct ldb_module *module,
* There is an active index sub transaction, and the record was
* found in the primary index transaction cache. A copy of the
* record needs be taken to prevent the original entry being
- * altered, until the index sub transaction is committed.
+ * altered, until the index sub transaction is committed, but we
+ * don't copy the actual values, just the array of struct ldb_val
+ * that points to the values (which are offsets into a GUID array).
+ *
+ * As a reminder, our primary cache is an in-memory tdb that
+ * maps attributes to struct dn_list objects, which point to
+ * the actual index, which is an array of struct ldb_val, the
+ * contents of which are {.data = <binary GUID>, .length =
+ * 16}. The array is sorted by GUID data, and these GUIDs are
+ * used to look up index entries in the main database. There
+ * are more layers of indirection than necessary, but what
+ * makes the index useful is we can use a binary search to
+ * find if the array contains a GUID.
+ *
+ * What we do in a sub-transaction is make a copy of the struct
+ * dn_list and the array of struct ldb_val, but *not* of the
+ * .data that they point to. This copy is put into a new
+ * in-memory tdb which masks the primary cache for the duration
+ * of the sub-transaction.
+ *
+ * In an add operation in a sub-transaction, the new ldb_val
+ * is a child of the sub-transaction dn_list, which will
+ * become the main dn_list if the transaction succeeds.
+ *
+ * These acrobatics do not affect read-only operations.
*/
-
- {
- struct ldb_val *dns = NULL;
- size_t x = 0;
-
- dns = talloc_array(
- list,
- struct ldb_val,
- list2->count);
- if (dns == NULL) {
- return LDB_ERR_OPERATIONS_ERROR;
- }
- for (x = 0; x < list2->count; x++) {
- dns[x].length = list2->dn[x].length;
- dns[x].data = talloc_memdup(
- dns,
- list2->dn[x].data,
- list2->dn[x].length);
- if (dns[x].data == NULL) {
- TALLOC_FREE(dns);
- return LDB_ERR_OPERATIONS_ERROR;
- }
- }
- list->dn = dns;
- list->count = list2->count;
+ list->dn = talloc_memdup(list,
+ list2->dn,
+ talloc_get_size(list2->dn));
+ if (list->dn == NULL) {
+ return LDB_ERR_OPERATIONS_ERROR;
}
+ list->count = list2->count;
return LDB_SUCCESS;
/*
@@ -3852,9 +3857,7 @@ int ldb_kv_reindex(struct ldb_module *module)
* Copy the contents of the nested transaction index cache record to the
* transaction index cache.
*
- * During this 'commit' of the subtransaction to the main transaction
- * (cache), care must be taken to free any existing index at the top
- * level because otherwise we would leak memory.
+ * This is a 'commit' of the subtransaction to the main transaction cache.
*/
static int ldb_kv_sub_transaction_traverse(
struct tdb_context *tdb,
@@ -3883,8 +3886,7 @@ static int ldb_kv_sub_transaction_traverse(
/*
* Do we already have an entry in the primary transaction cache
- * If so free it's dn_list and replace it with the dn_list from
- * the secondary cache
+ * If so replace dn_list with the one from the subtransaction.
*
* The TDB and so the fetched rec contains NO DATA, just a
* pointer to data held in memory.
@@ -3897,21 +3899,37 @@ static int ldb_kv_sub_transaction_traverse(
abort();
}
/*
- * We had this key at the top level. However we made a copy
- * at the sub-transaction level so that we could possibly
- * roll back. We have to free the top level index memory
- * otherwise we would leak
+ * We had this key at the top level, and made a copy
+ * of the dn list for this sub-transaction level that
+ * borrowed the top level GUID data. We can't free the
+ * original dn list just yet.
+ *
+ * In this diagram, ... is the C pointer structure
+ * and --- is the talloc structure (::: is both).
+ *
+ * index_in_top_level ::: dn orig ..............
+ * | | :
+ * | `--GUID array :
+ * | |----- val1 data
+ * ldb_kv `----- val2 data
+ * | :
+ * index_in_subtransaction :: dn copy ..........:
+ * | :
+ * `------------ new val3 data
+ *
+ * So we don't free the index_in_top_level dn list yet,
+ * because we are (probably) borrowing most of its
+ * children.
*/
- if (index_in_top_level->count > 0) {
- TALLOC_FREE(index_in_top_level->dn);
- }
index_in_top_level->dn
= talloc_steal(index_in_top_level,
index_in_subtransaction->dn);
index_in_top_level->count = index_in_subtransaction->count;
return 0;
}
-
+ /*
+ * We found no top level index in the cache, so we put one in.
+ */
index_in_top_level = talloc(ldb_kv->idxptr, struct dn_list);
if (index_in_top_level == NULL) {
ldb_kv->idxptr->error = LDB_ERR_OPERATIONS_ERROR;
--
2.46.0
From 70d8b1b2f87cbb16b671d334e46244ba001fbd31 Mon Sep 17 00:00:00 2001
From: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Date: Mon, 22 Jul 2024 22:22:15 +1200
Subject: [PATCH 2/2] ldb:kv_index: realloc away old dn list
We can't just free it, because has the GUID index list as a child, and
these are shared by the new dn list (from the subtransaction we are
committing). But if the dn list is long and the main transaction is
long-lived, we can save a lot of memory by turning this dn list into
an almost empty node in the talloc tree. This returns us to roughly
the situation we had prior to the last commit.
For example, with the repro.sh script on bug 15590 in indexes mode
with 10000 rules, The last 3 commits use this much memory at the end
of an unusually large transaction:
full talloc report on 'struct ldb_context' (total 4012222 bytes in 90058 blocks)
full talloc report on 'struct ldb_context' (total 2405482219 bytes in 90058 blocks)
full talloc report on 'struct ldb_context' (total 4282195 bytes in 90058 blocks)
That is, the last commit increased usage 500 fold, and this commit
brings it back to normal.
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15590
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
(cherry picked from commit 1bf9ede94f0a6b41fb18e880e59a8e390f8c21d3)
---
lib/ldb/ldb_key_value/ldb_kv_index.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/lib/ldb/ldb_key_value/ldb_kv_index.c b/lib/ldb/ldb_key_value/ldb_kv_index.c
index fed1033f492..11bdf00dc08 100644
--- a/lib/ldb/ldb_key_value/ldb_kv_index.c
+++ b/lib/ldb/ldb_key_value/ldb_kv_index.c
@@ -3919,8 +3919,12 @@ static int ldb_kv_sub_transaction_traverse(
*
* So we don't free the index_in_top_level dn list yet,
* because we are (probably) borrowing most of its
- * children.
+ * children. But we can save memory by discarding the
+ * values and keeping it as an almost empty talloc
+ * node.
*/
+ talloc_realloc(index_in_top_level,
+ index_in_top_level->dn, struct ldb_val *, 1);
index_in_top_level->dn
= talloc_steal(index_in_top_level,
index_in_subtransaction->dn);
--
2.46.0

View File

@ -38,6 +38,7 @@ Source1: https://www.samba.org/ftp/ldb/ldb-%{version}.tar.asc
Source2: ldb.keyring Source2: ldb.keyring
# Patches # Patches
Patch0: libldb-fix-indexes-performance.patch
BuildRequires: gcc BuildRequires: gcc
BuildRequires: libtalloc-devel >= %{talloc_version} BuildRequires: libtalloc-devel >= %{talloc_version}
@ -124,7 +125,7 @@ Development files for the Python bindings for the LDB library
%prep %prep
zcat %{SOURCE0} | gpgv2 --quiet --keyring %{SOURCE2} %{SOURCE1} - zcat %{SOURCE0} | gpgv2 --quiet --keyring %{SOURCE2} %{SOURCE1} -
%autosetup -n ldb-%{version} -p1 %autosetup -n ldb-%{version} -p3
%build %build
# workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1217376 # workaround for https://bugzilla.redhat.com/show_bug.cgi?id=1217376
@ -221,148 +222,88 @@ rm -f $RPM_BUILD_ROOT/%{_mandir}/man3/_*
%endif %endif
%changelog %changelog
* Mon Dec 04 2023 Andreas Schneider <asn@redhat.com> - 2.8.0-1 * Tue Aug 13 2024 Andreas Schneider <asn@redhat.com> - 2.8.0-1
- resolves: RHEL-16482 - Rebase version to 2.8.0 - resolves: RHEL-12109 - Fix performance regression with indexes
* Mon Jun 05 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.7.2-2 * Fri Nov 17 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.8.0-0
- resolves: rhbz#2190426 - Rebuilt to retrigger brew build - resolves: RHEL-16509 - Update to libldb-2.8.0
* Tue Jun 06 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.7.2-3
- resolves: rhbz#2190427 - Rebuild to trigger distrobaker sync
* Wed May 24 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.7.2-2
- resolves: rhbz#2190427 - Add missing tests to fix osci.brew-build.tier0.functional
* Thu May 18 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.7.2-1 * Thu May 18 2023 Pavel Filipenský <pfilipen@redhat.com> - 2.7.2-1
- resolves: rhbz#2190426 - Rebase to version 2.7.2 - resolves: rhbz#2190427 - Update to version 2.7.2
* Fri Oct 21 2022 Andreas Schneider <asn@redhat.com> - 2.6.1-1 * Mon Oct 24 2022 Andreas Schneider <asn@redhat.com> - 2.6.1-1
- resolves: rhbz#2131995 - Rebase to version 2.6.1 - resolves: rhbz#2132052 - Update to version 2.6.1
* Thu Aug 11 2022 Andreas Schneider <asn@redhat.com> - 2.5.2-2
- resolves: rhbz#2108998 - Rebuild to include python3-ldb-devel in CRB
* Wed Jul 27 2022 Andreas Schneider <asn@redhat.com> - 2.5.2-1 * Wed Jul 27 2022 Andreas Schneider <asn@redhat.com> - 2.5.2-1
- Rebase to version 2.5.2 - Rebase to version 2.5.2
- resolves: rhbz#2109017 - Fix CVE-2022-32746 - resolves: rhbz#2109016 - Fix CVE-2022-32746
* Mon Jun 13 2022 Pavel Filipenský <pfilipen@redhat.com> - 2.5.1-1 * Mon Jun 13 2022 Pavel Filipenský <pfilipen@redhat.com> - 2.5.1-1
- related: rhbz#2077490 - Rebase to version 2.5.1 - related: rhbz#2077484 - Rebase to version 2.5.1
* Tue May 03 2022 Pavel Filipenský <pfilipen@redhat.com> - 2.5.0-1 * Mon May 02 2022 Pavel Filipenský <pfilipen@redhat.com> - 2.5.0-1
- resolves: rhbz#2077490 - Rebase to version 2.5.0 - resolves: rhbz#2077484 - Rebase to version 2.5.0
* Mon Nov 29 2021 Pavel Filipenský <pfilipen@redhat.com> - 2.4.1-1 * Thu Nov 25 2021 Pavel Filipenský <pfilipen@redhat.com> - 2.4.1-1
- resolves: rhbz#2013588 - Rebase to version 2.4.1 - resolves: rhbz#2013605 - Rebase to version 2.4.1
* Mon Aug 09 2021 Mohan Boddu <mboddu@redhat.com> - 2.3.0-6 * Tue May 25 2021 Andreas Schneider <asn@redhat.com> - 2.3.0-2
- Rebuilt for IMA sigs, glibc 2.34, aarch64 flags - related: rhbz#1897082 - Fix libldb tests on ppc64le
Related: rhbz#1991688
* Tue May 25 2021 Andreas Schneider <asn@redhat.com> - 2.4.0-5 * Tue May 11 2021 Andreas Schneider <asn@redhat.com> - 2.3.0-1
- resolves rhbz#1962812 - Added gating tests - resolves: rhbz#1945021 - Update to version 2.3.0
- resolves: rhbz#1897082 - Fix libldb tests on aarch64
* Thu May 20 2021 Andreas Schneider <asn@redhat.com> - 2.3.0-4 * Wed Mar 24 2021 Andreas Schneider <asn@redhat.com> - 2.2.0-2
- related: #1951285 - Enable lmdb support on ppc64le again * resolves: rhbz#1941516 - Fixed CVE-2021-20277
* Wed Apr 21 2021 Andreas Schneider <asn@redhat.com> - 2.3.0-3 * Mon Nov 9 2020 Isaac Boukris <iboukris@redhat.com> - 2.2.0-1
- resolves: #1951285 - Fix ldb tests on aarch64 - Resolves: rhbz#1878114 - Rebase libldb to the version required by Samba
- Resolves: rhbz#1794349 - Build libldb with lmdb support
* Fri Apr 16 2021 Mohan Boddu <mboddu@redhat.com> - 2.3.0-2 * Wed Jun 24 2020 Isaac Boukris <iboukris@redhat.com> - 2.1.3-2
- Rebuilt for RHEL 9 BETA on Apr 15th 2021. Related: rhbz#1947937 - Resolves: rhbz#1849615 - Fix CVE-2020-10730 use-after-free
* Wed Mar 24 2021 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.3.0-1 * Tue Jun 2 2020 Isaac Boukris <iboukris@redhat.com> - 2.1.3-1
- libldb-2.3.0 is required for new samba - Resolves: rhbz#1817567 - Rebase libldb to 2.1.3 for samba
* Tue Jan 26 2021 Fedora Release Engineering <releng@fedoraproject.org> - 2.2.0-8 * Tue Nov 26 2019 Isaac Boukris <iboukris@redhat.com> - 2.0.7-3
- Rebuilt for https://fedoraproject.org/wiki/Fedora_34_Mass_Rebuild - Resolves: rhbz#1754423 - Rebase libldb to 2.0.7 version for samba
- Related: rhbz#1754423 - Fix sssd tests (ldb)
* Thu Oct 29 2020 Andreas Schneider <asn@redhat.com> - 2.2.0-7 * Tue May 7 2019 Jakub Hrozek <jhrozek@redhat.com> - 1.5.4-2
- Fix FTBFS / Increase the over-estimation for sparse files in tests - Fix some python2-related upgrade issues
- Related: rhbz#1567115 - libldb: Drop Python 2 subpackages from RHEL 8
* Tue Oct 27 2020 Andreas Schneider <asn@redhat.com> - 2.2.0-6 * Wed Apr 24 2019 Jakub Hrozek <jhrozek@redhat.com> - 1.5.4-1
- Spec file cleanup and improvements - Resolves: rhbz#1684582 - Rebase libldb to version 1.5.4 for Samba
- Resolves: rhbz#1567115 - libldb: Drop Python 2 subpackages from RHEL 8
- Resolves: rhbz#1597243 - libldb uses Python 2 to build.
* Thu Oct 22 2020 Alexander Bokovoy <abokovoy@redhat.com> - 2.2.0-5 * Thu Sep 20 2018 Jakub Hrozek <jhrozek@redhat.com> - 1.4.2-2
- Rebuild for Python 3.9 - Resolves: rhbz#1624132 - Review annocheck distro flag failures in libldb
* Tue Jul 28 2020 Fedora Release Engineering <releng@fedoraproject.org> - 2.2.0-4 * Fri Aug 17 2018 Alexander Bokovoy <abokovoy@redhat.com> - 1.4.2-1
- Rebuilt for https://fedoraproject.org/wiki/Fedora_33_Mass_Rebuild
* Mon Jul 13 2020 Tom Stellard <tstellar@redhat.com> - 2.2.0-3
- Use make macros
- https://fedoraproject.org/wiki/Changes/UseMakeBuildInstallMacro
* Thu Jul 09 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.2.0-2
- libldb-2.2.0 is required for samba 4.13rc1
* Thu Jul 02 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.1.4-1
- rhbz#1837364 - libldb-2.1.4 is available
* Tue May 26 2020 Miro Hrončok <mhroncok@redhat.com> - 2.1.3-2
- Rebuilt for Python 3.9
* Wed May 20 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.1.3-1
- rhbz#1837364 New: libldb-2.1.3 is available
* Tue Apr 28 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.1.2-1
- rhbz#1828721 New: libldb-2.1.2 is available
* Wed Feb 26 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.1.1-1
- rhbz#1807066 - libldb-2.1.1 is available
* Wed Jan 29 2020 Fedora Release Engineering <releng@fedoraproject.org> - 2.1.0-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_32_Mass_Rebuild
* Wed Jan 22 2020 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.1.0-1
- rhbz#1748422 - libldb-2.1.0 is available
* Mon Dec 16 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.0.8-1
- rhbz#1748422 - libldb-2.0.8 is available
* Wed Sep 11 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.0.7-1
- rhbz#1748422 - libldb-2.0.7 is available
* Tue Sep 03 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 2.0.6-1
- rhbz#1748422 - New upstream release 2.0.6
* Mon Aug 26 2019 Guenther Deschner <gdeschner@redhat.com> - 2.0.5-1
- rhbz#1683147 - libldb-2.0.5 is available
- rhbz#1737644 - libldb, libtalloc, libtevent, libtdb: Remove Python 2 subpackages from Fedora 31+
* Mon Aug 19 2019 Miro Hrončok <mhroncok@redhat.com> - 1.5.5-3
- Rebuilt for Python 3.8
* Thu Jul 25 2019 Fedora Release Engineering <releng@fedoraproject.org> - 1.5.5-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_31_Mass_Rebuild
* Sat Jul 13 2019 Guenther Deschner <gdeschner@redhat.com> - 1.5.5-1
- New upstream release 1.5.5
* Fri Jun 14 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.5.4-3
- rhbz#1718113 - samba fail to build with Python 3.8
AttributeError: module 'time' has no attribute 'clock'
* Mon Jun 03 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.5.4-2
- rhbz#1711638 - fails to build with Python 3.8.0a4
* Wed Mar 06 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.5.4-1
- New upstream release 1.5.4
* Thu Feb 14 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.5.3-1
- New upstream release 1.5.3
* Fri Feb 01 2019 Fedora Release Engineering <releng@fedoraproject.org> - 1.5.2-2
- Rebuilt for https://fedoraproject.org/wiki/Fedora_30_Mass_Rebuild
* Thu Jan 17 2019 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.5.2-1
- New upstream release 1.5.2
* Thu Nov 8 2018 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.4.3-1
- New upstream release 1.4.3
* Thu Aug 16 2018 Lukas Slebodnik <lslebodn@fedoraproject.org> - 1.4.2-1
- New upstream release 1.4.2 - New upstream release 1.4.2
- Resolves: rhbz#1615989
* Fri Jul 13 2018 Fedora Release Engineering <releng@fedoraproject.org> - 1.4.1-2 * Fri Jul 13 2018 Jakub Hrozek <jhrozek@redhat.com> - 1.4.1-1
- Rebuilt for https://fedoraproject.org/wiki/Fedora_29_Mass_Rebuild
* Thu Jul 12 2018 Jakub Hrozek <jhrozek@redhat.com> - 1.4.1-1
- New upstream release 1.4.1 - New upstream release 1.4.1
- Apply a patch to hide local ABI symbols to avoid issues with new binutils - Obsoletes 0001-ldb-Fix-memory-leak-on-module-context.patch
- Patch the waf script to explicitly call python2 as "env python" doesn't
yield py2 anymore * Mon Jul 02 2018 Petr Viktorin <pviktori@redhat.com> - 1.4.0-3
- Use %%{__python2}, not "python", as the Python2 interpreter
- Add workaround to allow building with Python 2
- Remove the lmdb dependency in RHEL
* Tue Jun 19 2018 Miro Hrončok <mhroncok@redhat.com> - 1.4.0-2 * Tue Jun 19 2018 Miro Hrončok <mhroncok@redhat.com> - 1.4.0-2
- Rebuilt for Python 3.7 - Rebuilt for Python 3.7