Shared Memory stuck in DEST state for Zabbix Agent

[Yekim]

I'm seeing an issue across basically any server that I have a zabbix agent or zabbix server on, where if you check the shared memory you can see that there are a ton of shared memory keys that are stuck in the status destroy state.

What is causing this issue and is this potentially harmful for my system? From the screenshot below it is a production machine of mine that has been up for 453 days, and is using zabbix agent 4.0.7



MyDataServer>uptime
15:51:17 up 453 days, 15:52, 1 user, load average: 0.05, 0.13, 0.18

MyDataServer>rpm -qa |grep zabbix
zabbix-release-4.0-1.el7.noarch
zabbix-sender-4.0.7-1.el7.x86_64
zabbix-agent-4.0.7-1.el7.x86_64

[tim.mooney]

I don't see this specific problem mentioned in the support.zabbix.com bug tracker, but there have been something like 30 bug fix releases in the 4.0 series since 4.0.7 was released.

I would pick one system running the agent, maybe a less-important server, stop the existing zabbix agent, delete the SHMs if they don't go away automatically when the agent is stopped, and then upgrade the agent and sender to something like 4.0.35 or later. Then see if the problem continues or if it's resolved.

[Yekim]

Thanks Tim, I had the same issue on a RHEL 8 box running Zabbix 5.0.14, they do all get destroyed when the process finally ends. However these servers can be up sometimes for long periods of time as seen by the box above that has an uptime of 453 days. Do you think this is a concern?

[tim.mooney]

I guess I would be a little concerned about it, yeah. The thing is, we have RHEL systems too that (some of them) go for long periods without being rebooted, and I'm not aware of lots of SHM segments getting stuck on any of our systems.

$ uptime
02:13:52 up 871 days, 21:02, 1 user, load average: 0.00, 0.01, 0.05
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
$ sudo ipcs -m

------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 1212416 zabbix 600 576 5 dest
0x00000000 1245185 zabbix 600 1825056 5 dest

2 segments is what I'm used to seeing. Your server with 361, that would definitely make me dig deeper. I haven't checked all my hosts but I spot checked about a dozen of them (various versions of RHEL & various kernel versions) and they all have exactly 2 SHM segments created by zabbix (agentd).

If you use the '-p' option with "ipcs", the pid of the creator is probably (?) always going to be the zabbix_agentd, but I wonder if looking at the pid of the last "operator" would give you any insight into what might be going on? That's just a guess, but it's probably where I would start investigating. That is, after updating some of the agents to the latest in the series. Still, if you're also seeing it with 5.0.14, which is almost up to date for the 5.0 series, it's probably not an agent bug.

Are you using any third-party templates or locally-defined UserParameter's?

[Yekim]

So looking through the ipcs -p, it does point back to zabbix_agentd, We use custom made templates and we do use locally defined UserParameter's, curious if it's trying to pull data that is not working correctly and that is what is causing it.

[tim.mooney]

That was my theory. Please do follow up once you've figured out what's causing it. It's an interesting issue.

[tim.mooney]

So my earlier search in the Zabbix bug tracker didn't find anything similar to the issue you're running into, but today I was searching for something else and happened to find this:

https://support.zabbix.com/browse/ZBX-18886

https://support.zabbix.com/browse/ZBX-19274