New Server v4.4: Host Availability ("ZBX") is red, but data is coming in

[J_Adatasol]

Hi all,

I recently created/installed a Gentoo-based Zabbix server on AWS (v4.4); it is described as an "Appliance" (not sure how that differs from a regular Zabbix server install). I have added a couple of Hosts (Agents) to the machine we want to monitor, and ran across an issue with the most recently added one:

• The Host 'Availability' indicator (Configuration -> Hosts -> 'Availability' column), for a new Host I set up yesterday shows that it is red, with an error = "Get value from agent failed: cannot connect to [[support.adatasol.com]:10050]: [4] Interrupted system call". But I appear to be getting valid current data from this Host.

There are 2 passive checks, and 1 active check (Log file) configured for this Host. This Host SHOULD be configured the same as another host I recently set up, which is showing a green indicator. (see screenshot) But there is a fare bit of manual configuration going on - I could have missed something.


Configuration -> Hosts -> 'Availabilty' column:



From Monitoring -> Latest Data (showing both Hosts); this data does refresh every minute. One difference - the Green Host has information for the 'Disk space' check, whereas the Red Host does not. (I don't know why the Diskspace check might be failing - the other passive checks are working.) And the 'Latest Data' for the Green Host looks nearly identical to the data from the Red Host:




The active check is supposed to communicate over port 10051 to the Agent, right? It doesn't appear that the Agent is listening on that port. But when I do a port scan of the Host that is showing a green indicator, it shows the same results (10050 = Zabbix Agent, 10051 = (no result)). Doing a port scan (to ensure that firewall openings are working) does only show 1 of the two ports (this is the same result for both Hosts):




So what should I look into to figure out why this indicator is red? How do I determine if the monitors are really working correctly?

Thanks,
J

[Ed.M]

Hi, J!
I guess first you should check firewall on new host. Seems that Zabbix agent is not reachable from server. Those data you are receiving seems to be web scenarios and they are not performed via Zabbix agent on remote host.
Also port 10051 for active checks should be opened on the Zabbix server side. Active check means that Zabbix agent sends data to Zabbix server. When Zabbix server asks for data from hosts - it is passive check, or poll request.
Regards,
Ed

[J_Adatasol]

Thanks for the hints, Ed.

Good point - I hadn't drawn the line that all of the data that is updating is from the Web Scenario check.

Both of the servers shown are hosted on AWS; they are both in the same security group (I think...someone else is administering the AWS stuff); I have opened the ports (10050 and 10051) on this new Host...but you are saying that port 10051 only needs to be opened on the Zabbix Server machine? (I will double check that.)

[Ed.M]

Here how it looks like on Zabbix server with installed Zabbix agent:

root@zabbix:~# netstat -nlp | grep zabbix
tcp 0 0 0.0.0.0:10050 0.0.0.0:* LISTEN 1282/zabbix_agentd
tcp 0 0 0.0.0.0:10051 0.0.0.0:* LISTEN 1308/zabbix_server
tcp6 0 0 :::10050 :::* LISTEN 1282/zabbix_agentd
tcp6 0 0 :::10051 :::* LISTEN 1308/zabbix_server

Check if you set correct Zabbix server address in zabbix_agent.conf, otherwise agent won't accept packets from another address. Also check ServerActive= is pointing to correct address. You'll need this for active checks.

[J_Adatasol]

I checked my AWS Security group configuration, and I had the wrong port specified. I updated the ports that it would allow through to my server (10050 and 10051).

Doing the command you suggested about ... mine didn't quite look like that. (My server is installed on Gentoo, using a Zabbix 4.4 Appliance AMI from COG.)

But, since then, I have installed the Agent on a new server; at first it was showing up an gray... I never have figured out what the lever is that turns it green - I poked and prodded at it, but it stayed gray. I went away to eat lunch, and this new Host was now showing Green. So that's a start.

So I have 3 Hosts now: 2 Green, 1 red.
- Only 1 host (one of the green ones) seems to be correctly reporting the Disk Space information.
- I noticed I had a configuration issue with my active Log File Trigger, so am updating that. But even after updating it doesn't appear to be reporting data.

[J_Adatasol]

Now the two Green Hosts are correctly reporting disk information. So I think they are working correctly.

For the Red host, though...I have tried a number of additional steps, but nothing has helped yet.
• Reinstalled Agent a couple of times
• Double checked Firewall rules
• Tested connection from new Zabbix Server to Host: linux command from the Server - "nc -zv myhost.com 10050" : that worked, said the port was open
• Left default configuration in place, after doing a reinstall (did define the Server during the install)
• I have a custom config file I am using - went through it again, and it all looks right
• Verified that the Server and ServerActive values were accurate.
Not sure what else to check.

[Ed.M]

When Activity is grey it means that Zabbix server don't have a status of a host yet. On new hosts it takes some time. Server should contact host, get information from host, ensure its status. So it was OK.

Now let's look at problem host. As you mentioned before error was "[4] Interrupted system call". This means Zabbix server can not communicate with agent. This can be not only local firewall. It can be SELINUX on RedHat/Centos/Fedora or APPARMOR on Debian/Ubuntu. Also you may check agent log file on a problem host.

About active checks: check Latest data. Does server receive any from hosts? If there is no data, trigger is useless. Ensure data is coming from hosts, then I could help to setup a trigger.

[J_Adatasol]

Thanks getting back to me again, Ed.

RE: Red status: Zabbix server is running on Gentoo, Zabbix 4.4 Appliance image.
• I have 3 agents, all talking to the same server; two of these are Green. So, I would guess that there isn't a firewall issue with the server itself, otherwise the 2 Hosts wouldn't be green.
• I have checked the firewall on the Red Host; it appears to be correctly configured: Port 10050 (inbound) is open on that Host. I am able to do a Port scan from both the Zabbix Server and from my own machine, and this Red Host correctly responds that the Port is open. Thus I don't believe that there is a firewall issue at the Host, either.

RE: Active Checks: I am not, as of yet, getting data back for my 1 Active test, but I haven't had time to actually troubleshoot that part yet. Been working on other things. So let's backburner that issue for now.

[J_Adatasol]

Is there a manual command I can run from the Host (Agent) to test connection to the Zabbix Server? E.g. 'zabbix_get -s myhost.com -V' (to try and get the version of the server). Or would it be Zabbix_sender? (Doing research on these, now.)

[Ed.M]

As I mentioned before there may be SeLinux or APPARMOR on the "red" host. What OS does it run?

For testing connectivity (which I guess will be OK, as the port is open), yiu can try zabbix_sender -z "zabbix server host" -s "red host name as it is in Zabbix server" -k "key to send" -o "value"

[J_Adatasol]

The Zabbix server is running on Gentoo.

I believe that I found methods of doing a port scan from each entity (Server -> Host, Host -> Server), and I believe the results of those tests indicates that the ports are both open, in each direction.

Thanks for the tip on the Sender. I have tried a couple of versions of the Zabbix_Sender, but not having much luck yet. It says it processes the command, but the server responds with a Failed = 1. I'm probably not passing the right Key; I'll have to try some more later.

[Ed.M]

No, not Zabbix servers OS! That "red" hosts OS! What is on that host? If RedHat/Centos - disable Selinux and it will work.

[J_Adatasol]

Doh! Sorry about that.

All of my Hosts are running Windows (Server 2016 mostly).