ConfD 8.0.6 confd.smp RSS leak under sustained SNMP polling -- powerpc/arm64 only, not x86_64

[ricardo.martincoski]

Hello,

We are seeing a slow but steady RSS leak in confd.smp on our products under sustained SNMP polling load. The anonymous RSS (RssAnon) keeps climbing while the load is active, and does not recover after the load stops.

We can reproduce this in the lab using a load script that sends SNMP GET requests from multiple pollers for an hour or longer.

We use:

• ConfD 8.0.6
• Linux kernel 5.4.47
• processors: powerpc (32-bit), x86_64, arm64
• glibc 2.25
• gcc >= 6.3

With the setup above, we observe the growth of RssAnon and VmRSS on powerpc (32-bit) and on arm64, but not on x86_64.

Using the same setup but with ConfD 7.2.6 (same kernel, glibc, gcc), the growth does NOT occur.

To reproduce in the lab, we run the SNMP GET load for an hour or longer, then stop the load and leave the system idle for a while.
Following plot shows confd.smp RSS over time, during and after the load. You can see the steady climb, and that not all of the memory is reclaimed after the load stops.

We would appreciate any pointers from anyone who has seen a similar pattern.

Best Regards,
Ricardo

[fnchooft]

Can you provide a standalone reproduction that demonstrates the issue?
We can try it in later releases to see if it still behaves the same way.

[ittaloss]

Hi @fnchooft,

Thanks for picking this up. Some follow-up details on our reproduction
setup, and an update: we have now reproduced the same leak on ConfD
8.0.11 with no other changes to the setup.

ConfD versions reproduced on

• 8.0.6 (original report).
• 8.0.11 – same growth shape, same architecture sensitivity.

Architecture matrix (identical workload everywhere)

• arm64: leak reproduces consistently.
• powerpc: leak reproduces consistently.
• x86_64: no growth. confd.smp RSS plateaus within minutes and
  stays flat.

Reproduction image

To rule out anything in our product stack, we stripped our integration
image down to “raw ConfD only”:

• Removed every local patch we apply on top of the upstream installer (14
  quilt patches in our case), so the daemon is the unmodified upstream
  BEAM binary (confd.smp).
• Removed our product YANG / .fxs / init XML / confd_dyncfg.fxs –
  none of our application schemas are loaded.
• The only schema in the loadPath is the upstream
  examples.confd/snmpa/1-simple/ example (simple.fxs and
  TAIL-F-TEST-MIB.bin), built from the example’s stock sources with
  confdc at image build time.

confd.conf (essentials)

Minimal config – just what the daemon needs plus the SNMP agent. No
NETCONF, no CLI, no HA, no rollback, no runtimeReconfiguration, and
candidate datastore disabled:

<confdConfig xmlns="http://tail-f.com/ns/confd_cfg/1.0">
    <aaa>
        <sshServerKeyDir>/etc/ssh</sshServerKeyDir>
    </aaa>
    <cdb>
        <enabled>true</enabled>
        <dbDir>/var/confd/cdb</dbDir>
        <initPath>
            <dir>/etc/confd/init-xml</dir>
        </initPath>
    </cdb>
    <datastores>
        <candidate><enabled>false</enabled></candidate>
    </datastores>
    <stateDir>/var/confd/state</stateDir>
    <snmpAgent>
        <enabled>true</enabled>
        <ip>0.0.0.0</ip>
        <port>161</port>
        <mibs><fromLoadPath>true</fromLoadPath></mibs>
        <snmpEngine><snmpEngineID>80:00:61:81:05:01</snmpEngineID></snmpEngine>
        <system>
            <sysDescr>Tail-f ConfD agent</sysDescr>
            <sysObjectID>1.3.6.1.4.1.24961</sysObjectID>
        </system>
    </snmpAgent>
    <loadPath>
        <dir>/opt/confd/etc/confd</dir>
        <dir>/opt/confd/etc/confd/snmp</dir>
        <dir>/etc/confd</dir>
    </loadPath>
    <snmpLog>
        <enabled>true</enabled>
        <file>
            <enabled>true</enabled>
            <name>@CONFD_LOG_DIR@/confd_snmp.log</name>
        </file>
        <syslog>
            <enabled>true</enabled>
            <facility>local4</facility>
        </syslog>
    </snmpLog>
    <!-- logs: confdLog, developerLog, auditLog, netconfLog -->
</confdConfig>

The example’s simple_init.xml, community_init.xml and vacm_init.xml
are shipped under /etc/confd/init-xml/ so the CDB picks them up on
first boot (two hosts, two servers, two services under
TAIL-F-TEST-MIB).

Load

A small Python harness drives net-snmp’s snmpget in parallel. The exact
parameters for the runs in the graph below:

• Per-PDU invocation (one snmpget subprocess per PDU, Zabbix-style):

  snmpget -v 2c -c public -t 5 -r 0 -On <host> <oid>
  

  -- SNMPv2c, community public, single OID per PDU, 5 s timeout, no
  retries.

• 15 parallel worker threads, fed by a queue.

• Back-to-back scheduling: the scheduler enqueues one full pass
  through the OID pool, blocks until all workers have drained the queue,
  then enqueues the next pass – so the daemon sees sustained
  GetRequest traffic with no idle gap and no Zabbix-style poll-interval
  pacing (this corresponds to POLLING_INTERVAL_SEC=0 in our harness).

• OID pool: walked once at startup against the populated subtree of
  TAIL-F-TEST-MIB, then held constant. Concretely:

  • the 5 scalars (numberOfServers.0, numberOfHosts.0,
    maxNumberOfServers.0, maxNumberOfHosts.0, extraDescr.0),
  • the columns of hostTable, serverTable, serviceTable for every
    row created by simple_init.xml (~20 OIDs total for the canned
    two-host / two-server / two-service data).

• Sampling: every 30 s the harness also captures confd.smp VmRSS,
  MemAvailable, slab, and the top-3 mappings from
  /proc/<pid>/smaps, plus per-PDU latency percentiles (p50/p95/p99)
  and the running get_ok / get_err / get_timeouts counters. That
  is the data feeding the graph below.

What we see

• confd.smp RSS grows monotonically – no plateau in our longest runs.
• The growth is mirrored in /proc/<pid>/smaps: the top-3 mappings
  (BEAM carriers + process heap) all grow together.
• System MemAvailable decreases at the matching rate.
• Stopping the SNMP load freezes RSS but the memory is never reclaimed.

Happy to share more info if necessary.

Thanks,
Ittalo

[fnchooft]

Good Morning,
I have tested this scenario on an arm64 machine:

Kernel version…: 6.12.87+rpt-rpi-v8
Full kernel info.: Linux raspberrypi 6.12.87+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.87-1+rpt1~bookworm (2026-05-12) aarch64 GNU/Linux
Architecture…: aarch64
glibc version…: glibc 2.36
GCC version…: gcc (Debian 12.2.0-14+deb12u1) 12.2.0
OS release…: Debian GNU/Linux 12 (bookworm)

I could not see the same behavior and I tested our versions 8.0.21, 8.4 and 8.7.
None of the versions on that kernel- and glibc-version have unbounded RSS-growth.

Kind regards,

Fabian

[ittaloss]

Thanks for getting back to me, @fnchooft!

Just a few questions.

• Have you reproduced the issue with version 8.0.11 or earlier? Could you share the results, please?
• Are there any fixes or improvements documented on RN for versions between 8.0.11 and 8.0.21?
• Have you observed any ConfD configurations or usage patterns that make the leak more likely to happen? Something you recommend not using?
• Is there a chance we can get the 8.0.21 version temporarily for a test?

Thanks in advance,

Best regards,

Ittalo

[fnchooft]

Afternoon, I will try to obtain the 8.0.11 and re-run the tests on my target.
This might take some days.

Kind regards,

Fabian

[ittaloss]

Good morning,

Thank you for doing the tests. Were you able to reproduce the issue with the 8.0.11 or earlier?

Thanks,

Ittalo

[ittaloss]

Good morning,

Any updates on this?

Thanks

[fnchooft]

Good Evening, I was able to locate the 8.0.11 installer for arm64 and the machine which was used for these tests.

On the same machine, with the 8.0.11, the RSS memory is not growing on this setup.

elapsed_s,vmrss_kb,get_ok,get_err,get_timeouts
0,60680,0,0,0
30,53948,3339,0,0
60,54592,6636,0,0
90,53684,9933,0,0
120,53576,13272,0,0
150,53540,16632,0,0
180,53388,19929,0,0
210,53676,23247,0,0
240,53456,26565,0,0
270,53420,29883,0,0
301,53744,33138,0,0

It came even down a bit from the original value of 60680 to about 53744.

Kind regards,

Fabian

[ittaloss]

Hi, fnchooft

Thank you for answering.

How long have you tested for?

Would you be able to test for two hours?

Thanks again,

Best regards

[fnchooft]

Evening @ittaloss , I left it running for 7200 secs:

elapsed_s,vmrss_kb,get_ok,get_err,get_timeouts
0,59496,0,0,0
30,53028,3360,0,0
60,53480,6678,0,0
90,53572,9933,0,0
120,53604,13272,0,0
150,53352,16611,0,0
180,53076,19950,0,0
210,53316,23289,0,0
240,53248,26649,0,0
...
6952,52836,761166,0,0
6982,53096,764505,0,0
7012,53048,767823,0,0
7042,52784,771162,0,0
7072,53016,774480,0,0
7102,53364,777819,0,0
7133,52984,781179,0,0
7163,53104,784518,0,0
7193,53176,787857,0,0
7201,53184,788613,0,0

But in the end the numbers are very much inline with the previous measurements.

Kind regards,
Fabian

[ittaloss]

Good afternoon, @fnchooft!

I’m sorry it took me too long to answer you.

We ran the same load-generation approach as the standalone reproducer we shared earlier (30 SNMP GET workers, ~70k OIDs/min), this time on x86_64 hardware, for an extended duration.

Phase	Duration	RssAnon slope
Idle baseline	6h	+0.018 MiB/h (noise)
Under load, 0-12h	12h	+0.125 MiB/h (noise)
Under load, 12h onward	36.3h	+74.0 MiB/h (95% CI ±0.32)

There is an onset delay of roughly 12-13 hours before growth becomes visible on x86_64. That’s longer than any x86_64 test we have run before now, which is likely why it looked absent previously.

Once growth starts, it does not stop: the process hit an actual OOM kill at t=48.3h (RssAnon 2811.6 MiB, VmPeak 7841.6 MiB, MemAvailable=0).

Given what we’re seeing here, it seems the leak is not exclusive to powerpc/arm64 after all. It could be just a matter of time. So a short negative result on arm64 wouldn’t rule this out either.

Could you please re-run the test for at least 48h? I’m very confident there is a memory leak in ConfD versions up to and including 8.0.11. We’ve fully isolated this from our own application layer here.

Your test result is vital for us in deciding whether to purchase another ConfD binary or not.

Thank you.

Best regards,

Ittalo

[fnchooft]

Good Afternoon @ittaloss ,

I took my setup and am running a 24h test on the arm64 machine from the previous tests using ConfD version 8.0.11.

I will keep you posted on the results.

Kind regards,

Fabian