ConfD 8.0.6 system processes consuming a lot of CPU

General
1

Hello DMAP community, I hope someone can help me with a issue regarding ConfD consuming a lot of the system CPU.

We have noted that since we updated our ConfD from 7.2.6 to 8.0.6, two new processes are now running on our systems: 1_scheduler and 2_scheduler.

PID PPID VIRT RES %CPU %MEM TIME+ S COMMAND
4964 4894 125.1m 86.9m 66.7 4.4 0:21.75 R 2_scheduler
4960 4894 125.1m 86.9m 58.3 4.4 0:36.11 R 1_scheduler

We checked whether these processes already exist on ConfD 7.2.6, and it did not exist.

We ran the system with the same configuration database on both ConfD version, with 7.2.6 we did not note any CPU issue, but when running in ConfD 8.0.6, the CPU was often at 100% of usage, due mainly to these two processes.

I would like to know if this is a known problem in ConfD 8.0.6, if there is something that can be configured in order to keep the old behavior (i.e not running these processes), or an explanation on what are those two processes.

The system in which the problem happened is the following:

2x e5500 (32bit powerpc)

/ # lscpu
Architecture: ppc
CPU op-mode(s): 32-bit
Byte Order: Big Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 2
Model: 2.1 (pvr 8024 1021)
Model name: e5500
BogoMIPS: 50.00
L1d cache: 64 KiB
L1i cache: 64 KiB
L2 cache: 512 KiB
L3 cache: 256 KiB

2

Evening Hugo,

Can you please check what smp-flags are being set for ConfD 7.2.6?

Between version 7.x and 8.x some changes were introduced in this subsystem.

If you want to explicitly only run one scheduler - try passing --smp 1 to the script which
starts ConfD, see the -h option for confd-executable script.

Kind regards,

Fabian

3

Below are the smp-flags being set for ConfD 7.2.6

./confd.smp -S 2 -K false -MHe true -- -root /opt/confd/lib/confd
    -progname confd -- -home /root -- -smp enable
    -boot confd -noshell -noinput -foreground
    -yaws embedded true -ignore_initial_validation
    -stacktrace_depth 24 -shutdown_time 30000
    -conffile <path_to_confd.conf> -start_phase0 -max_fds 8192

Below are the flags being set for ConfD 8.0.6:

./confd.smp -S 2 -K false -MHe true -- -root /opt/confd/lib/confd
    -progname confd -- -home /root -- -boot confd -noshell
    -noinput -foreground -yaws embedded true -ignore_initial_validation
    -stacktrace_depth 24 -shutdown_time 30000
    -conffile <path_to_confd.conf> -start_phase0 -max_fds 8192 --

Now that you asked for these information that i noticed the absence of -smp parameter on the execution of the confd 8.0.6, we will check it.

Despite the difference, there is any known issue regarding cpu usage on ConfD 8.0.6 scheduler?

4

Morning Hugo,
Currently there is not a known issue.

Were you able to try the system with --smp 1?

As a side note: The executables are the Erlang-processes and these are standard.

Could you describe what is happening in the system while you see the high-cpu-load? Is it
during boot-phase, while loading a configuration, or constant?

Kind regards,

Fabian

5

Hi Fabian,

Thank you for your reply.

The two processes consuming significant CPU resources were observed not in our testing environment but in a customer environment. While attempting to reproduce the issue, we noticed that when using either --smp 1 or --smp 2 on a 2-core CPU, the scheduler processes occasionally spike to 100% CPU usage. However, this doesn’t raise major concerns—it’s just brief spikes that, before the update (I assume), would have been attributed to the confd.smp process directly.

Regarding the --smp parameter:

  • If we use --smp 5, for example, would it create five processes (scheduler_1 through scheduler_5)?
  • If so, is it recommended to align the --smp value with the number of CPU cores available on the system running ConfD?

Additionally, what are the performance implications of using --smp 1, --smp 2, or higher values like --smp 5? For instance:

  • Would lower values create a bottleneck in systems with many scheduling tasks?
  • Would higher values increase CPU overhead for task execution?

I ask because the core issue may not be CPU usage itself but rather my limited understanding of these processes and the actual impact of the --smp parameter. The user guide provides a description of the parameter, but I couldn’t find details about the changes in scheduling behavior between ConfD 7.2.6 and 8.2.6.

Thank you for your insights!

Best regards,
Hugo

6

Afternoon Hugo,

Normally you would instantiate one instance per core.

There is the following description in the documentation:

SMP Run-Time System
The Erlang run-time system takes advantage of a multi-core or multi-CPU computer by running several Erlang scheduler threads (typically, the same number of threads as the number of cores).
To gain performance from a multi-core computer, your application must have more than one runnable Erlang process most of the time. Otherwise, the Erlang emulator can still only run one Erlang process at the time.

This is normally not an issue, however it seems that you could not reproduce the issue in the same manner as it is running at the customer?

In this case you could profile a bit more to see when the high-load occurs, this is normally the case when initial configuration is loaded - or - if you are committing large NETCONF-configurations.

Kind regards,

Fabian