Zabbix and semaphore use

[tlacroix]

Hey there,

I've recently moved my Zabbix server to a new physical server and took the time to upgrade Zabbix version (from 4.0.2 to 4.2.1) and debian version, of course.
I'm having a new problem i didn't had before, concerning semaphores

I'm running a few scripts (with crontabs under "root" user) to get different values from remote servers
Typical script is to connect to remote server (ssh, psql, etc), get back value, and send it to a defined server/key with zabbix_sender

This morning i'm having a lot of alerts because of no value has been sent since a few hours
Digging everywhere, i've found the reason to be around semaphores that are full

ipcs -u
------ Semaphore Status --------
used arrays = 32000

(max is 32000)

Digging again, i can see that used semaphores are linked to non-existing processes

ipcs -spc

------ Semaphore Arrays Creators/Owners --------
semid      perms      cuid       cgid       uid        gid       
1048903680 600        zabbix     zabbix     zabbix     zabbix    
1048870913 600        root       root       www-data   www-data  
65538      600        root       root       www-data   www-data  
1048936451 600        root       root       root       root      
1048969220 600        root       root       root       root      
1049001989 600        root       root       root       root      
1049034758 600        root       root       root       root      
1049067527 600        root       root       root       root      
1049100296 600        root       root       root       root      
[.....]

ipcs -i 1048969220 -s

Semaphore Array semid=1048969220
uid=0     gid=0     cuid=0     cgid=0
mode=0600, access_perms=0600
nsems = 14
otime = Not set                   
ctime = Mon May 13 10:00:24 2019  
semnum     value      ncount     zcount     pid       
0          1          0          0          8327      
1          1          0          0          8327      
2          1          0          0          8327      
3          1          0          0          8327      
4          1          0          0          8327      
5          1          0          0          8327      
6          1          0          0          8327      
7          1          0          0          8327      
8          1          0          0          8327      
9          1          0          0          8327      
10         1          0          0          8327      
11         1          0          0          8327      
12         1          0          0          8327      
13         1          0          0          8327

ps aux |grep 8327
root      7355  0.0  0.0  12780   944 pts/1    S+   11:34   0:00 grep 8327

When I manually start my script, i can see the number of semaphores growing one by one (= each new value sent through zabbix_sender)
This didn't occur with the old Zabbix version

Am I doing someting wrong ? (except running scripts as root, yeah)

Feedback appreciated!

Thanks!

[laocius]

I am having exactly the same issue on FreeBSD 11 since upgrading to zabbix 4.2.
I have to cleanup the semaphores once in a while with `ipcrm -W`

[SirDice]

Getting the same problem on FreeBSD 11.2 and Zabbix 4.2.1. Every time zabbix_sender runs a new semaphore is added to the list and at some point you simply run out. Increasing kern.ipc.semmns and kern.ipc.semmsl only postpones the inevitable. It looks like zabbix_sender creates a new semaphore with every run and never clears it.

Opened ZBX-16218 for this.

[SirDice]

The semaphore issue is fixed in Zabbix 4.2.4. The FreeBSD ports tree was updated a few days ago.

https://docs.freebsd.org/cgi/getmsg.cgi?fetch=2441003+0+/usr/local/www/mailindex/archive/2019/svn-ports-head/20190707.svn-ports-head

[mike3sons]

I have the Zabbix agent 4.2.1 semaphore issue on AIX 7.1 and 7.2, but there is no pre-compiled download for 4.2.4 on AIX. The 4.2.1 version is the latest for AIX. Our system guys gave up on trying to compile Zabbix on AIX, so I guess the options are to either revert to the prior version or write a semaphore-cleanup script to run once an hour.