Zabbix_Server logs display : In zbx_ipc_service_recv() timeout:1

[admin_Djo]

Hello zabbix community

I am trying to setup zabbix in order to monitor our servers : rhel 7

I follow this setup : https://www.zabbix.com/documentation...lation/install

I can't manage to fix some errors from server log :

12917:20200427:101855.183 In zbx_ipc_service_recv() timeout:1
12925:20200427:101855.183 In zbx_ipc_service_recv() timeout:1
2925:20200427:101855.183 End of zbx_ipc_service_recv():2^C

I try to increase timeout setting in both *.conf from server and agentd but same errors, can't find any solutions yet

Did someone can help me ?

thank you

[tim.mooney]

What version of Zabbix have you installed?

Is SELinux enabled on your Zabbix server? I believe the default packages assume that SELinux is either disabled or changed to "permissive" mode. It is possible to run Zabbix server with SELinux in "enforcing" mode, but it may require additonal configuration that isn't covered in the (basic) installation guide.

Is there a host-based firewall enabled on your Zabbix server, and if so, have you opened the necessary ports in the firewall?

[admin_Djo]

Hi tim, thanks for your reply,

Actually SELinux is disabled,
I have check ports : 10050 , also 161 and 199 for snmp, everyones are openned and connection are succeed when testing with netcat...

I trying to figure out from what zbx_ipc_service received timeout ?

[tim.mooney]

What about port 10051? That's the server port. 10050 is the agent port.

[admin_Djo]

i had checked 10051 also, TCP connection succeed.

[admin_Djo]

Hi everyone

Still struggling with this error.
Does someone know what zbx_ipc_service is used for ?

thanks

[tim.mooney]

I haven't looked at this part of the Zabbix source code before, so this analysis is just a quick first impression, but it looks like this is part of the communication that happens between the various separate Zabbix-related processes that run on the server. It appears that there is a robust mechanism for the processes to pass messages back and forth using the various UNIX domain sockets that Zabbix creates.

On the Zabbix server I run on RHEL 7.8, Zabbix creates its socket files in /tmp/ :

$ ls -al /tmp/ | egrep zabbix
srwx------   1 zabbix zabbix     0 Apr 13 16:08 zabbix_server_alerter.sock=
srwx------   1 zabbix zabbix     0 Apr 13 16:08 zabbix_server_lld.sock=
srwx------   1 zabbix zabbix     0 Apr 13 16:08 zabbix_server_preprocessing.sock=

You may have more or fewer sockets, depending on what Zabbix subsystems you're using.

You've already indicated that SELinux is not running, so it can't be SELinux that's preventing communication.

Does the zbx_ipc_service_recv() timeout start as soon as your Zabbix server is restarted, or does it take a while after a restart before the messages start appearing in the logs?

[admin_Djo]

Thx again for your reply,

I've got Timeout immediately after server start.

I've got exactly the same sockets as you :

151085493 srwx------ 1 zabbix zabbix 0 May 5 09:26 zabbix_server_alerter.sock
151085494 srwx------ 1 zabbix zabbix 0 May 5 09:26 zabbix_server_lld.sock
151085495 srwx------ 1 zabbix zabbix 0 May 5 09:26 zabbix_server_preprocessing.sock

does sockets need right on folder or system ?

[Stanlety]

Sometimes, monitored devices need more time to respond to the Zabbix server because they may have a large number of monitored metrics, or their CPU is overloaded, SNMP agent is badly implemented, or there is a huge lag on a network, etc MCDVOICE

[tim.mooney]

Hmmm, Ok. If there had been a delay between server startup and when the messages started, I thought it might indicate that something was going wrong in one of the sockets some time after it's created. I've seen poorly-written scripts that clean out "old files" in /tmp/ that aren't careful to skip sockets, so the sockets got deleted after some # of days. That's not the problem here, though.

I'm not sure what you mean by your last question. UNIX domain sockets work in a similar fashion to a network connection, but they exist only on the local system and because they are a special type of file, they use the Linux filesystem permission model. Your sockets are owned by the zabbix user and zabbix group and have the permissions I would expect. Processes running as 'zabbix' should have the necessary permission to read and write to them, using the message passing scheme the Zabbix developers have created. If SELinux were running it could definitely prevent Zabbix from being able to pass messages via the sockets, but you've already reported that SELinux is not enabled, so that can't be the problem.

The output you show includes the inode number for each socket (the first column, before the type and permissions), and I guess I'm a little surprised at how large those inode numbers are. It's possible whatever filesystem you're using isn't assigning inodes contiguously, but having used 151 million inodes on a Zabbix server is a little surprising.

The volume containing /tmp isn't full or nearly full, is it? As long as the sockets are created I don't think a full volume should prevent communication via the sockets, but I'm running out of other ideas.

Are there any suspicious messages in your server's kernel log buffer (look at the output of the 'dmesg' command). Sometimes (but definitely not always) kernel issues or resource issues get logged there. It's one place I look when I've exhausted the more obvious ideas for what might be causing a problem.

[admin_Djo]

Ok, I guess socket permissions are good. I don't totally understand what do you mean about inodes number, but i haven't got volume issue ( everyone has got at least 20% free). As past days have been free for me i haven't check kernel log buffer as you mentionned i try this and came bback to you.
Thanks again for you help

[Hackcraze]

This type of error usually indicates an error in the Zabbix key although your format matches the documentation. I would do the following to try and narrow down the cause:

• When an item/trigger isn't working start with the bare minimum setup. Disable/remove the trigger and use the simplest item key possible (in this case just log[/tmp/scribetest/test3/test3_current]).
• Make sure the log file exists and is readable by Zabbix on the client.
• Check the log files both on the Zabbix server and client. For example, on my setup I was getting "No active checks found on server: host not found" errors which indicates some sort of Zabbix configuration issue preventing all active checks from working.
  
  For more information please visit : HackCraze

[SeekLamp]

Steps:

1. Download Zabbix source files that matches your version of Zabbix server.
2. Extract files, enter newly extracted directory and with text editor open file “src/zabbix_server/server.c“
3. Change the maximum timeout value of 30 to higher value
   {"Timeout", &CONFIG_TIMEOUT, TYPE_INT, PARM_OPT, 1, 30},
4. Recompile Zabbix server
5. Change “Timeout” value in Zabbix configuration file (“zabbix_server.conf“) to match the one you defined in source file
6. Restart Zabbix Server

For more informtaion visit : https://www.seeklamp.com/