[BUG] Memory leak in TLS connection ERROR when HEP is enabled for TLS

[zooptwopointone]

OpenSIPS version you are running

flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 78cdea979
main.c compiled on  with gcc 12

Describe the bug
There appears to be a Memory leak in PKG memory for TLS connections when HEP is enabled for TLS.

To Reproduce

Using Wolf SSL

modparam("tls_mgm", "server_domain", "REDACTED")
modparam("tls_mgm", "certificate", "[cert1]/etc/opensips/tls/cert1.pem")
modparam("tls_mgm", "private_key", "[cert1]/etc/opensips/tls/cert1.key")
modparam("tls_mgm", "tls_method", "[cert1]TLSv1_2")
modparam("tls_mgm", "ca_dir", "[cert1]/etc/ssl/certs")
modparam("tls_mgm", "require_cert", "[cert1]0")
modparam("tls_mgm", "verify_cert", "[cert1]0")
modparam("tls_mgm", "client_domain", "client1")
modparam("tls_mgm", "certificate", "[client1]/etc/opensips/tls/cert1.pem")
modparam("tls_mgm", "private_key", "[client1]/etc/opensips/tls/cert1.key")
modparam("tls_mgm", "tls_method", "[client1]TLSv1_2")
modparam("tls_mgm", "ca_dir", "[client1]/etc/ssl/certs")
modparam("tls_mgm", "require_cert", "[client1]0")
modparam("tls_mgm", "verify_cert", "[client1]0")
modparam("proto_tls", "tls_max_msg_chunks", 8)

loadmodule "proto_hep.so"
loadmodule "tracer.so"
modparam("proto_hep", "hep_id", "[hep_dst]1.1.1.1:9060;transport=udp;version=3")
modparam("proto_hep", "homer5_on", 1)
modparam("proto_hep", "hep_capture_id", 2870)
modparam("tracer", "trace_on", 1)
modparam("tracer", "trace_id", "[tid]uri=hep:hep_dst")
modparam("proto_tls", "trace_on", 1)
modparam("proto_tls", "trace_destination", "hep_dst")

Running this in a loop, which connects sends nothing over and closes the connection. So that TLS fails.

while true ; do  nc SIPSERVER 5061 >/dev/null < /dev/zero; done

Will result in this error message, which also starts to cause problems with real calls.

2024-10-16T20:24:56.256222+00:00 ID3270LA /usr/sbin/opensips[2500793]: WARNING:core:fm_malloc: not enough contiguous free pkg memory (8 bytes left, need 168), attempting defragmentation... please increase the "-M" command line parameter!
2024-10-16T20:24:56.256222+00:00 ID3270LA /usr/sbin/opensips[2500793]: WARNING:core:fm_malloc: not enough contiguous free pkg memory (8 bytes left, need 168), attempting defragmentation... please increase the "-M" command line parameter!
2024-10-16T20:24:56.256331+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:fm_malloc: not enough free pkg memory (8 bytes left, need 168), please increase the "-M" command line parameter!
2024-10-16T20:24:56.256331+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:fm_malloc: not enough free pkg memory (8 bytes left, need 168), please increase the "-M" command line parameter!
2024-10-16T20:24:56.256429+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:proto_hep:create_hep3_message: no more pkg mem!
2024-10-16T20:24:56.256429+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:proto_hep:create_hep3_message: no more pkg mem!
2024-10-16T20:24:56.256525+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:proto_hep:add_hep_chunk: invalid call! bad input params!
2024-10-16T20:24:56.256525+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:proto_hep:add_hep_chunk: invalid call! bad input params!
2024-10-16T20:24:56.256621+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:create_trace_message: failed to add correlation id! aborting trace...!
2024-10-16T20:24:56.256621+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:create_trace_message: failed to add correlation id! aborting trace...!
2024-10-16T20:24:56.256716+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:trace_message_atonce: failed to create the message!
2024-10-16T20:24:56.256716+00:00 ID3270LA /usr/sbin/opensips[2500793]: ERROR:core:trace_message_atonce: failed to create the message!

Commenting out

modparam("proto_tls", "trace_on", 1)
modparam("proto_tls", "trace_destination", "hep_dst")

and doing the same test never results in failure. I used -m 64 -M 2 options to test on a system not taking any calls just to narrow down the problem. I tried looking into the code to see if I could spot where it was missing when a TLS connection fails, but it is a bit more involved than I can spend right now.

Expected behavior
TLS connection failures should cleanup memory used by HEP(speculation) just like successful connection end.

Relevant System Logs
Provided above.

OS/environment information

Debian 12
Opensips installed from Opensips package installation process provided by Opensips.org

Additional context

The TLS errors are high on our network as there is a LoadBalancer that makes a TCP connection to the port to validate reachability. It is currently not capable of negotiating TLS. This was not a problem for 2.4 version of Opensips.

[bogdan-iancu]

@zooptwopointone , please see here some info on how to deal with OOM issues - https://opensips.org/Documentation/TroubleShooting-OutOfMem. Try to get a pkg mem dump from the proc reporting the pkg OOM.

[zooptwopointone]

dump.log

Here is the output of a server configured with 4MB of pkg memory after it started to have the problem.

Followed the link you provided and this is the result of running the opensips-cli .... command.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[zooptwopointone]

This is still an issue.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[zooptwopointone]

No updates from me, just waiting to see if there is more information I can provide.

[github-actions]

Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days.

[zooptwopointone]

This is still an issue.

[zooptwopointone]

Adding comment to not let it go stale.