From 194e476ec178638454736468a210835f38b95971 Mon Sep 17 00:00:00 2001 From: Chris Lumens Date: Mon, 20 Oct 2025 16:46:13 -0400 Subject: [PATCH] Don't overwhelm the FSA queue with repeated CIB queries - Related: RHEL-76276 --- 016-fewer-messages.patch | 88 ++++++++++++++++++++++++++++++++++++++++ pacemaker.spec | 7 +++- 2 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 016-fewer-messages.patch diff --git a/016-fewer-messages.patch b/016-fewer-messages.patch new file mode 100644 index 0000000..2d37039 --- /dev/null +++ b/016-fewer-messages.patch @@ -0,0 +1,88 @@ +From 7be9cdca98217d002497714aaafcfc292c02555b Mon Sep 17 00:00:00 2001 +From: Chris Lumens +Date: Fri, 31 Oct 2025 11:24:14 -0400 +Subject: [PATCH] Med: daemons: Don't add repeated I_PE_CALC messages to the + fsa queue. + +Let's say you have a two node cluster, node1 and node2. For purposes of +testing, it's easiest if you use fence_dummy instead of a real fencing +agent as this will fake fencing happen but without rebooting the node so +you can see all the log files. + +Assume the DC is node1. Now do the following: + +- pcs node standby node1 +- pcs resource defaults update resource-stickiness=1 +- for i in $(seq 1 300); do echo $i; pcs resource create dummy$i ocf:heartbeat:Dummy --group dummy-group; done +- pcs node unstandby node1 + +It will take a long time to create that many resources. After node1 +comes out of standby, it'll take a minute or two but eventually you'll +see that node1 was fenced. On node1, you'll see a lot of transition +abort messages happen. Each of these transition aborts causes an +I_PE_CALC message to be generated and added to the fsa queue. In my +testing, I've seen the queue grow to ~ 600 messages, all of which are +exactly the same thing. + +The FSA is triggered at G_PRIORITY_HIGH, and once it is triggered, it +will run until its queue is empty. With so many messages being added so +quickly, we've basically ensured it won't be empty any time soon. While +controld is processing the FSA messages, it will be unable to read +anything out of the IPC backlog. + +based continues to attempt to send IPC events to controld but is unable +to do so, so the backlog continues to grow. Eventually, the backlog +reaches that 500 message threshold without anything having been read by +controld, which triggers the eviction process. + +There doesn't seem to be any reason for all these I_PE_CALC messages to +be generated. They're all exactly the same, and they don't appear to be +tagged with any unique data tying them to a specific query, and their +presence just slows everything down. + +Thus, the fix here is very simple: if the latest message in the queue is +an I_PE_CALC message, just don't add another one. We could also make +sure there's only ever one I_PE_CALC message in the queue, but there +could potentially be valid reasons for there to be multiple interleaved +with other message types. I am erring on the side of caution with this +minimal fix. + +Related: RHEL-76276 +--- + daemons/controld/controld_messages.c | 20 ++++++++++++++++++++ + 1 file changed, 20 insertions(+) + +diff --git a/daemons/controld/controld_messages.c b/daemons/controld/controld_messages.c +index 0b0f25b..30af707 100644 +--- a/daemons/controld/controld_messages.c ++++ b/daemons/controld/controld_messages.c +@@ -73,6 +73,26 @@ register_fsa_input_adv(enum crmd_fsa_cause cause, enum crmd_fsa_input input, + return; + } + ++ if (input == I_PE_CALC) { ++ GList *ele = NULL; ++ ++ if (prepend) { ++ ele = g_list_first(controld_globals.fsa_message_queue); ++ } else { ++ ele = g_list_last(controld_globals.fsa_message_queue); ++ } ++ ++ if (ele != NULL) { ++ fsa_data_t *message = (fsa_data_t *) ele->data; ++ ++ if (message->fsa_input == I_PE_CALC) { ++ crm_debug("%s item in fsa queue is I_PE_CALC, not adding another", ++ (prepend ? "First" : "Last")); ++ return; ++ } ++ } ++ } ++ + if (input == I_WAIT_FOR_EVENT) { + controld_set_global_flags(controld_fsa_is_stalled); + crm_debug("Stalling the FSA pending further input: source=%s cause=%s data=%p queue=%d", +-- +2.43.0 + diff --git a/pacemaker.spec b/pacemaker.spec index 6289e12..7fbbb7e 100644 --- a/pacemaker.spec +++ b/pacemaker.spec @@ -244,7 +244,7 @@ Name: pacemaker Summary: Scalable High-Availability cluster resource manager Version: %{pcmkversion} -Release: %{pcmk_release}.4%{?dist} +Release: %{pcmk_release}.5%{?dist} %if %{defined _unitdir} License: GPL-2.0-or-later AND LGPL-2.1-or-later %else @@ -280,6 +280,7 @@ Patch012: 012-dont-set-as-xml-id.patch Patch013: 013-crm_node-i-initialize.patch Patch014: 014-remote-fencing.patch Patch015: 015-ipc-disconnect.patch +Patch016: 016-fewer-messages.patch Requires: resource-agents Requires: %{pkgname_pcmk_libs}%{?_isa} = %{version}-%{release} @@ -1031,6 +1032,10 @@ exit 0 %license %{nagios_name}-%{nagios_hash}/COPYING %changelog +* Mon Nov 17 2025 Chris Lumens - 2.1.7-5.5 +- Don't overwhelm the FSA queue with repeated CIB queries +- Related: RHEL-76276 + * Tue Sep 30 2025 Chris Lumens - 2.1.7-5.4 - Be more lenient in evicting IPC clients - Resolves: RHEL-76276