No reaction from host behind firewall

[thomasmathiesen]

Hi folks,

I am having a host behind a firewall that does not want to play with Zabbix (1.4.4). Here is my setup:

mylocation.mycompany.com = public ip (e.g. 82.55.99.88)
zabbix = NAT'ed ip (e.g. 192.168.1.2)
server = NAT'ed ip on same subnet (e.g. 192.168.1.3).

remotelocation.mycompany.com = public ip (e.g. 85.22.33.44)
remoteserver = NAT'ed ip (e.g. 192.168.0.4)
port 10050 forwarded to 192.168.0.4 in ADSL modem.
firewall accept tcp traffic on port 10050

I can successfully run passive checks on server and zabbix server, but not on remoteserver.

A simple check is:

[myuser@zabbix ~]$ telnet 192.168.1.3 10050
Trying 192.168.1.11...
Connected to 192.168.1.11 (192.168.1.11).
Escape character is '^]'.
helo
ZBXDZBX_NOTSUPPORTEDConnection closed by foreign host.

This seems to work.. but when I do the same to the ip of the remotelocation,
the following happens:

[thomas@server ~]$ telnet remotelocation.mycompany.com 10050
Trying 85.22.33.44...
Connected to mailmaster.linprofs.com (85.22.33.44).
Escape character is '^]'.
helo


^]
telnet> quit
Connection closed.


I've setup the agent to do full debugging, but there's nothing in the log file, except for just saying that it's starting up. I think it has something to do with the ip that the request is coming from (from the server to the agent)? I've used both the FQDN and the public ip address of mylocation.mycompany.com in the agent's zabbix_agentd.conf file.

Can anyone help me debug this? What else can I try?

[thomasmathiesen]

nobody knows?

[cbidwell]

I think this is similar if not the same problem that I'm having. Does your client behind the firewall come up active at first but after time lose connectivity w/ the server? That's what mine is doing so when I restart the zabbix client it reestablishes the connection.

[thomasmathiesen]

I've actually solved it. I dunno exactly which change fixed it, but here's a log of what I did/found:

1. The server was actually running 1.3.5.. even though 1.4.4 was installed. I had to change this, and reset the db (mysql).

2. I made sure that port 10050 and 10051 is portforwarded to the client through the firewall.

3. I made sure that the /etc/services file lists 10050 and 10051

4. I setup a fresh /etc/zabbix/zabbix_agentd.conf file, which lists the ip address of the server, debug=4, disable active checking and spawns 5 agents.

then I restarted the zabbix client.

A very good test, is to telnet from the zabbix server to port 10050 on the client.
If you type "helo" and press enter (when telnet connection is open), you should get an error message, whcih says something like "ZABBIX*(&*(@#". If you don't get anything, the zabbix agent does not answer.. and it might be a firewall issue. The agent is listening, but it does not authorize you to get anything.

[cbidwell]

Thanks for the reply on this. I do have a dumb question:

Is it REALLY necessary to put the port numbers in /etc/services? Isn't that just used to convert the port number to a readable name?

Nevertheless, I noticed I didn't have them in my services file on the client that was intermittently losing connectivity. That's about the only thing different that I hadn't do that you mentioned. I don't think it'll fix my problem, but I'm hoping it will.

[thomasmathiesen]

Thanks for the reply on this. I do have a dumb question:

Is it REALLY necessary to put the port numbers in /etc/services? Isn't that just used to convert the port number to a readable name?

Nevertheless, I noticed I didn't have them in my services file on the client that was intermittently losing connectivity. That's about the only thing different that I hadn't do that you mentioned. I don't think it'll fix my problem, but I'm hoping it will.

I don't know if the services file needs those two lines, but it's better to be safe than sorry.

I had the host showing up with:

Cannot connect to [<ip address>:10050] [Interrupted system call]

in the the configuration for hosts. I do believe that it's related to how often the server checks the client. I've been playing with the settings, and I'll see if it helps. Right now I am fighting vmware, as the clock seems to go wild on the zabbix server. The clock is extremely important, and that's why you should make sure that it's ok.

[cbidwell]

Ah! VMWare! I've had that problem also with VMWare. I'm not sure which version you're using, but there was a known problem that the times weren't syncing properly for the vmware client; they subsequently released a patch that was supposed to have fixed it. I don't think I even got the patch to work either. I wound up putting this in my crontab:

0 * * * * /usr/bin/rdate -s time-a.timefreq.bldrdoc.gov > /dev/null 2>&1

to sync the clock every hour.

[thomasmathiesen]

My clock is running away every minute.. reckon I'll be taking zabbix off the vmware box. Xen smells better when it comes to these things.

Thanks for the tip though

[cbidwell]

One more thing. I have a host that claims is unreachable, however, I can telnet to tcp/10050 but when I type helo, I get no response:

[root@zabbix_server scripts]# telnet zabbix.client 10050
Trying 192.168.1.5...
Connected to zabbix.client (192.168.1.5).
Escape character is '^]'.
helo

In the configuration Hosts section it says:

Monitored Not available ZBX_TCP_READ() failed [Interrupted system call]

But I can restart the zabbix client service and it would miraculously reestablish communication again.

[thomasmathiesen]

One more thing. I have a host that claims is unreachable, however, I can telnet to tcp/10050 but when I type helo, I get no response:

[root@zabbix_server scripts]# telnet zabbix.client 10050
Trying 192.168.1.5...
Connected to zabbix.client (192.168.1.5).
Escape character is '^]'.
helo

In the configuration Hosts section it says:

Monitored Not available ZBX_TCP_READ() failed [Interrupted system call]

But I can restart the zabbix client service and it would miraculously reestablish communication again.

When it reconnects (after restarting the client), what happens if you login through telnet and type "whatever"?

I think that you are checking your hosts too often, and that's why it's no longer answering. I'll do some more research on my end to see if I can find out more.