Zabbix says agents not reporting.. But they are..

[joshuamcdo]

Virtualbox hosted
CentO/S 7
Server - Zabbix 3.0.7. © 2001–2016, Zabbix SIA


Client
VMWare hosted
Red Hat Enterprise Linux Server release 6.6 (Santiago)

Agents - zabbix_agentd (daemon) (Zabbix) 3.0.5
Revision 62889 30 September 2016, compilation time: Oct 2 2016 11:54:08

I am experiencing a bizarre problem.

I had to reboot the VM that runs the stopgap zabbix server I have been running. When it came back up I didn't realize that it wasn't "reporting". I woke up this morning and realized I was missing around 6 hours worth of data. So I started looking into this and I am baffled.
The server reports that all the agents have been unreachable. However, that's not true... If I telnet to <serverip> 10051 I connect as it should. When i put the agent in debug level 4 I see that it says it's succeeding in sending the data.
4399:20170224:081345.879 __zbx_zbx_setproctitle() title:'collector [idle 1 sec]'
4409:20170224:081346.360 In send_buffer() host:'10.2.2.1' port:10051 entries:0/100
4409:20170224:081346.360 End of send_buffer():SUCCEED

I see no failures in the log files of any kind. However, when I go to the server and run iptraf-ng. I don't see any connections coming in on 10051. The server ip is reserved so it has not changed. iptables is flushed. It's so bizarre to me because the client says it's succeeding in sending the data. Yet the dashboard has no data of any kind for over six hours. It gets even stranger..

If I go to 'latest data'. There appears to be new data arriving. There are values that have changed anyway.

I have stopped the agent, stopped/started the server.. Check the IPs, checked for connectivity and it's all there yet it doesn't work. If I stop the zabbix-sever service and try to telnet to port 10051 I get a connection refused as expected.

Please help I am baffled!

Thanks in advance,
J

[vsimoes]

We need more info

Can you post here your agent and server configuration?

It seems like to be a misconfiguration issue.

[kloczek]

FTR: whole VmWare monitoring has nothing to do with zabbix agent.

[joshuamcdo]

I wanted to update this issue so people weren't left wondering..

My zabbix configurations were good as they had not changed. The only thing that had changed was the fact that I had restarted the zabbix server.

As I started digging.. I realized that the zabbix server was no longer DNS aware. So I looked closer at the resolv.conf. As it turns out. The resolv.conf was missing a search domain.. ?Who knew? Once I added that back in and set peerdns=no in order to stop that behavior from happening abain.

Thanks,

J

Can you post here your agent and server configuration?

It seems like to be a misconfiguration issue.