Monitor linux SYSV semaphore?

[aschild]

We have a application which does sometime leak semaphores.

Every ~100 days, we see this error on the server:

Insufficient system resources - failed to allocate a SYSV semaphore

Is there a way to monitor such SYSV semaphores in linux?

[jan.garaj]

Convert output from

ipcs -s

to required Zabbix metric by using UserParameter.

https://www.zabbix.com/documentation/2.4/manual/config/items/userparameters

[aschild]

Convert output from

ipcs -s

to required Zabbix metric by using UserParameter.

https://www.zabbix.com/documentation...userparameters

Yep,

but this only lists the semaphores,
we will then need to look inside the semaphores and the number of free slots on certain semaphores.

ipcs -s

to list all semaphores, then

ipcs -s -i <id>

to list the slots per semaphore and finally look how many are free or used up.
This is where I'm currently struggling...

[jan.garaj]

I'm lost. Please define metrics (items), which do you need to monitor. "I need semaphore monitoring" is vague definition.

[aschild]

We have a process which has dbsrv11 as name.
From this one I can get the PID it is running under, lets say it's PID 1911.

In the list of sempahore (ipcs -s), I see about 30 semaphores

When I then inspect each of these via ipcs -s -i <id>, I receive such tables:

Semaphore Array semid=11534409
uid=0    gid=0   cuid=0  cgid=0
mode=0777, access_perms=0777
nsems = 250
otime = Mon Feb  2 05:24:40 2015
ctime = Mon Feb  2 05:24:44 2015
semnum     value      ncount     zcount     pid
0          1          0          0          1911
1          1          0          0          1911
2          1          0          0          1911
3          1          0          0          1911
4          1          0          0          1911
5          1          0          0          1911
6          1          0          0          1911
7          1          0          0          1911
8          1          0          0          1911
9          1          0          0          1911
10         1          0          0          1911
11         1          0          0          1911
12         1          0          0          1911
13         1          0          0          1911
14         1          0          0          1911
15         1          0          0          1911
16         1          0          0          1911
17         1          0          0          1911
18         1          0          0          1911
19         1          0          0          1911
20         1          0          0          1911
21         1          0          0          1911
...
199        1          0          0          1911
200        1          0          0          1911
201        1          0          0          1911
202        0          0          0          0
203        0          0          0          0
204        0          0          0          0
...
246        0          0          0          0
247        0          0          0          0
248        0          0          0          0
249        0          0          0          0

As you see in the example above, the semaphore 11534409 has 250 slots,
of which currently 201 are used up.
I would need to get notified, when the slots are used up 80%.

So the step to monitor would be:

1. Get the PID of the dbsrv11 process
2. Find all semaphores which are used by PID (There are usually many of them)
3. In these semaphores, look if there are still 20% free slots

Sound like a task for some bash wizard...

[jan.garaj]

Proper Zabbix monitoring solution:
Zabbix LLD discovery which return semaphore id and related process name of this semaphore ID. Then you can filter which processes do you want to monitor in zabbix by using regexp and for these semaphore IDs will be created metric (item) "Semaphore utilization" and trigger, which raise event problem if metric value will more than defined threshold. And then event will be transformed into action (email, jabber message, ...)

Is it complicated? Yes, but it's proper (Zabbix) monitoring concept: item -> trigger -> event -> action.

But you are talking only about notification. So IMHO you need only bash script, which will send analyze/calculate/evaluate (create alert email) everything in one go. To be honest I don't see any advantage to use Zabbix for this task for you. It'll be easier for you to create just this one (probably) bash script :-)