Can't get proxies to work (yes the names are the same) -- Zabbix 2.2

[aib]

From your picture I can see that your proxy also unreachable.
Is it correct?
And if your proxy is unreachable how you can reach any device which talk with proxy?

Could you confirm that your proxy reachable and has any different host which is Active and Monitored?

[clearclaw]

The proxy is showing as unreachable as I also have the proxy system running an agent, which in turn is also configured to report through the proxy...which isn't working. As that configuration is as wee bit more confusing, I've not been talking about it, and instead have been talking only about a third system, prodBETA-lb-0, which is also configured through the same proxy.

The actual setup in detail with names:

-- ops-monitor-master: Runs Zabbix Server. Successfully monitors itself.

-- prodBETA-mon-master: Runs zabbix Proxy and Zabbix Agent. Configured to be monitored by itself as a proxy.

-- prodBETA-lb-0: Runs Zabbix Agent. Configured to be monitored by the proxy on prodBETA-mon-master

You can see all of this here:

https://www.dropbox.com/s/3ly94nck6jvs1un/Screenshot%20from%202014-02-28%2012%3A42%3A35.png

Notice that only ops-monitor-master, the only one which is isn't going through the proxy, is showing as being monitored.

[aib]

Did you try to monitor proxy from server directly, without proxy?
If yes - try to use "Zabbix agent(active)" type to check that connection is working in both ways.
Just to be sure, that your server can talk to proxy correctly.

After your proxy will work we can remove it from the list of potential obstacles. Right?

[clearclaw]

Did you try to monitor proxy from server directly, without proxy?

Yes, and whenever I skip using a proxy, that node is happy:

https://www.dropbox.com/s/btfbmxoerwqssjq/Screenshot%20from%202014-02-28%2014%3A18%3A23.png

But that doesn't change the state of prodBETA-lb-0 -- which is configured to be monitored via the proxy. (This is some hours later BTW).

If yes - try to use "Zabbix agent(active)" type to check that connection is working in both ways.
Just to be sure, that your server can talk to proxy correctly.

After your proxy will work we can remove it from the list of potential obstacles. Right?

I'd like to think so, yes, but so far nothing that touches the Zabbix proxy process works for me.

The driver behind this all BTW is that all those prodBETA machines (and prodALPHA) need to move into a VPC...and soon. And when that happens the Zabbix installation won't have IP visibility to the boxes it is monitoring any more, but the proxies can act as a bridge and keep my on-call life almost happy. Thus all my excitement about proxies and you know, having them work.

[clearclaw]

Zabbix server logs in debug mode: http://pastebin.com/JCA8dDNn

Zabbix agent logs in debug mode: http://pastebin.com/brg61UW2

I see nothing in the server logs that's clearly actionable/fixable. Mostly it seems like complaints that could be explained by "not getting data from the agent".

In the agent logs however I see:

23021:20140303:195702.097 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23017:20140303:195702.097 In update_cpustats()
 23017:20140303:195702.100 End of update_cpustats()
 23021:20140303:195702.100 End of send_buffer():SUCCEED
 23021:20140303:195702.101 refresh_active_checks('10.185.3.248',10051)
 23021:20140303:195702.102 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23021:20140303:195702.102 Before read
 23021:20140303:195702.104 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]
 23021:20140303:195702.105 In parse_list_of_checks()
 23021:20140303:195702.105 In disable_all_metrics()
 23021:20140303:195702.105 No active checks on server: host [prodBETA-lb-0] not found
 23021:20140303:195702.106 In process_active_checks('10.185.3.248',10051)
 23021:20140303:195702.106 End of process_active_checks()
 23021:20140303:195702.106 In get_min_nextcheck()

I've no idea what to do about that, but it seems the heart of the problem.

[aib]

Your assumptions are correct. Done.

zabbix_agent.conf:

Server=10.185.3.248,10.168.24.153
ServerActive=10.185.3.248

Hello, ClearClaw!
As I can see from your citation, you configured only one server in the list of ServerActive.
When you switch to Zabbix agetn(active), the agent send the answers only to address(es) which he found after ServerActive keyword.

Because of that you have a lot of complains in zabbix_agent.log file

23021:20140303:195702.101 refresh_active_checks('10.185.3.248',10051)
 23021:20140303:195702.102 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23021:20140303:195702.102 Before read
 23021:20140303:195702.104 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]

When you add the proxy IP address in the list of ServerActive, the zabbix agent will start the second instance which will send the answers to second server(which is actually your proxy) Multiple servers for active agent? Sure!.

Do you mind to try this setting and tell us how it's work?

zabbix_agent.conf:

Server=10.185.3.248,10.168.24.153
ServerActive=10.185.3.248[B],10.168.24.153[/B]

Please, correct me if I am wrong. I reread all thread and still not sure about IP addresses.
In the first message you showed that :

zabbix_proxy.conf:

Server=10.168.24.153

Then, in next message when you provide zabbix_agent.conf file, you told that your Server has this address and proxy has addres 10.185.3.248

Because of misunderstanding of IP addresses we push you in (possible) wrong way to change the configuration of zabbix agent...

Sorry about that.

[clearclaw]

When you add the proxy IP address in the list of ServerActive, the zabbix agent will start the second instance which will send the answers to second server(which is actually your proxy) Multiple servers for active agent? Sure!.

Do you mind to try this setting and tell us how it's work?

That's actually the configuration I had at the start of this thread (see the copies of the configs there). I then removed one of the addresses in the ServerActive value per request, and found that it made no difference.

Reverting the value of Server Active to:

ServerActive=10.168.24.153,10.185.3.248

Results in the following debug log:

23212:20140303:210739.393 Sleeping for 1 second(s)
 23213:20140303:210740.393 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23212:20140303:210740.394 In send_buffer() host:'10.168.24.153' port:10051 values:0/100
 23208:20140303:210740.394 In update_cpustats()
 23212:20140303:210740.394 End of send_buffer():SUCCEED
 23208:20140303:210740.395 End of update_cpustats()
 23213:20140303:210740.395 End of send_buffer():SUCCEED
 23212:20140303:210740.396 Sleeping for 1 second(s)
 23213:20140303:210740.396 Sleeping for 1 second(s)
 23212:20140303:210741.396 In send_buffer() host:'10.168.24.153' port:10051 values:0/100
 23208:20140303:210741.397 In update_cpustats()
 23213:20140303:210741.397 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23208:20140303:210741.398 End of update_cpustats()
 23213:20140303:210741.398 End of send_buffer():SUCCEED
 23212:20140303:210741.399 End of send_buffer():SUCCEED
 23213:20140303:210741.399 Sleeping for 1 second(s)
 23212:20140303:210741.399 Sleeping for 1 second(s)
 23213:20140303:210742.400 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23208:20140303:210742.400 In update_cpustats()
 23212:20140303:210742.401 In send_buffer() host:'10.168.24.153' port:10051 values:0/100
 23212:20140303:210742.401 End of send_buffer():SUCCEED
 23213:20140303:210742.402 End of send_buffer():SUCCEED
 23208:20140303:210742.402 End of update_cpustats()
 23212:20140303:210742.402 refresh_active_checks('10.168.24.153',10051)
 23213:20140303:210742.403 refresh_active_checks('10.185.3.248',10051)
 23213:20140303:210742.404 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23212:20140303:210742.404 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23212:20140303:210742.405 Before read
 23213:20140303:210742.405 Before read
 23213:20140303:210742.408 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]
 23213:20140303:210742.408 In parse_list_of_checks()
 23213:20140303:210742.409 In disable_all_metrics()
 23213:20140303:210742.409 No active checks on server: host [prodBETA-lb-0] not found
 23213:20140303:210742.409 In process_active_checks('10.185.3.248',10051)
 23213:20140303:210742.410 End of process_active_checks()
 23213:20140303:210742.410 In get_min_nextcheck()
 23213:20140303:210742.410 Sleeping for 1 second(s)
 23212:20140303:210742.429 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]
 23212:20140303:210742.430 In parse_list_of_checks()
 23212:20140303:210742.430 In disable_all_metrics()
 23212:20140303:210742.431 No active checks on server: host [prodBETA-lb-0] not found
 23212:20140303:210742.431 In process_active_checks('10.168.24.153',10051)
 23212:20140303:210742.431 End of process_active_checks()
 23212:20140303:210742.432 In get_min_nextcheck()
 23212:20140303:210742.432 Sleeping for 1 second(s)

Please, correct me if I am wrong. I reread all thread and still not sure about IP addresses.

ops-monitor-master: 10.168.24.153 # Zabbix server
prodBETA-mon-master: 10.185.3.248 # Zabbix server and agent
prodBETA-lb-0: 10.164.38.249 # Zabbix agent

Sorry about that.

Not a worry.

[aib]

OK, now it's more clear to me.

As I can see from debug, your agent requesting the list of active checks from both sources - from server and from proxy.

I assume that you configured only passive check for the host prodBETA-lb-0, right?

Let's change one Item from "Zabbix agent" type of check to "Zabbix agent(active)" type of check.

Then check zabbix_agent.log file to see the conversation between agent and server. (the same part as you have before in your log file)

23212:20140303:210742.402 refresh_active_checks('10.168.24.153',10051)
 23213:20140303:210742.403 refresh_active_checks('10.185.3.248',10051)
 23213:20140303:210742.404 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23212:20140303:210742.404 Sending [{
        "request":"active checks",
        "host":"prodBETA-lb-0"}]
 23212:20140303:210742.405 Before read
 23213:20140303:210742.405 Before read
 23213:20140303:210742.408 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]
 23213:20140303:210742.408 In parse_list_of_checks()
 23213:20140303:210742.409 In disable_all_metrics()
 23213:20140303:210742.409 No active checks on server: host [prodBETA-lb-0] not found
 23213:20140303:210742.409 In process_active_checks('10.185.3.248',10051)
 23213:20140303:210742.410 End of process_active_checks()
 23213:20140303:210742.410 In get_min_nextcheck()
 23213:20140303:210742.410 Sleeping for 1 second(s)
 23212:20140303:210742.429 Got [{
        "response":"failed",
        "info":"host [prodBETA-lb-0] not found"}]
 23212:20140303:210742.430 In parse_list_of_checks()

[clearclaw]

As I can see from debug, your agent requesting the list of active checks from both sources - from server and from proxy.

Yes, and this was determined to be in error, and thus the change to zabbix_agent.conf to have only one address for ServerActive.

I assume that you configured only passive check for the host prodBETA-lb-0, right?

No, the configuration of the proxy is for active monitoring. Long term, when I get this working, the Zabbix server will not have IP visibility to the proxy, but the proxy will have IP visibility to the server.

Let's change one Item from "Zabbix agent" type of check to "Zabbix agent(active)" type of check.

Sorry, I don't understand. What exactly do you want me to do?

https://www.dropbox.com/s/hi29avgwajp0hri/Screenshot%20from%202014-03-03%2013%3A34%3A58.png

Then check zabbix_agent.log file to see the conversation between agent and server. (the same part as you have before in your log file)

If I remove the proxy, if I have prodBETA-lb-0 connect directly to the Zabbix server, then everything works perfectly -- as I showed already with prodBETA-mon-master when I moved its agent from the proxy back to the Zabbix Server. But that's useless to me for prodBETA-lb-0. I need to move all the prodBETA hosts (there are many more) into a VPC (think firewalled private network) and thus need to use a proxy to bridge between the VPC and the external Zabbix server.

[aib]

I have only one last hope: try to rename manually Hostname (on server side and in agent config) from prodBETA-lb-0 to some simple like "ppppp" or "12345".

[aib]

Sorry, I don't understand. What exactly do you want me to do?

Host prodBETA-lb-0 -> Configuration of items -> Item "Agent ping"
Field "Type" can be any from the long list of checks:
- Zabbix agent
- Zabbix agent(active)
- simple check
etc.....

I would like to ask you just change from Zabbix agent to Zabbix agent(active) check.
In this case your proxy will make a list of "Active check" and send it to agent when the agent ask about that list.

PS Sorry, English is not my language.

[clearclaw]

Renamed prodBETA-lb-0 to "mango" in the Zabbix UI:

https://www.dropbox.com/s/fvw0wi9z3thhr2w/Screenshot%20from%202014-03-03%2013%3A39%3A49.png

New zabbix_agent.conf:

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
EnableRemoteCommands=1
LogRemoteCommands=1
Server=10.185.3.248,10.168.24.153
ServerActive=10.168.24.153,10.185.3.248
Hostname=mango
Include=/etc/zabbix/zabbix_agentd.d/

Zabbix proxy restarted just to get things moving faster.

Resulting zabbix_agents.log:

23261:20140303:213907.934 No active checks on server: host [mango] not found
 23260:20140303:213907.956 No active checks on server: host [mango] not found
 23261:20140303:214107.951 No active checks on server: host [mango] not found
 23260:20140303:214107.998 No active checks on server: host [mango] not found

Debug log:

 23291:20140303:214503.030 Sleeping for 1 second(s)
 23292:20140303:214504.031 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23291:20140303:214504.031 In send_buffer() host:'10.168.24.153' port:10051 values:0/10
0
 23287:20140303:214504.031 In update_cpustats()
 23287:20140303:214504.032 End of update_cpustats()
 23291:20140303:214504.032 End of send_buffer():SUCCEED
 23292:20140303:214504.032 End of send_buffer():SUCCEED
 23291:20140303:214504.033 Sleeping for 1 second(s)
 23292:20140303:214504.033 Sleeping for 1 second(s)
 23291:20140303:214505.033 In send_buffer() host:'10.168.24.153' port:10051 values:0/10
0
 23287:20140303:214505.034 In update_cpustats()
 23292:20140303:214505.034 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23292:20140303:214505.034 End of send_buffer():SUCCEED
 23287:20140303:214505.035 End of update_cpustats()
 23291:20140303:214505.035 End of send_buffer():SUCCEED
 23292:20140303:214505.035 refresh_active_checks('10.185.3.248',10051)
 23291:20140303:214505.035 refresh_active_checks('10.168.24.153',10051)
 23292:20140303:214505.036 Sending [{
        "request":"active checks",
        "host":"mango"}]
 23292:20140303:214505.037 Before read
 23291:20140303:214505.037 Sending [{
        "request":"active checks",
        "host":"mango"}]
 23291:20140303:214505.037 Before read
 23292:20140303:214505.039 Got [{
        "response":"failed",
        "info":"host [mango] not found"}]
 23292:20140303:214505.039 In parse_list_of_checks()
 23292:20140303:214505.039 In disable_all_metrics()
 23292:20140303:214505.039 No active checks on server: host [mango] not found
 23292:20140303:214505.040 In process_active_checks('10.185.3.248',10051)
 23292:20140303:214505.040 End of process_active_checks()
 23292:20140303:214505.040 In get_min_nextcheck()
 23292:20140303:214505.041 Sleeping for 1 second(s)
 23291:20140303:214505.076 Got [{
        "response":"failed",
        "info":"host [mango] not found"}]
 23291:20140303:214505.077 In parse_list_of_checks()
 23291:20140303:214505.077 In disable_all_metrics()
 23291:20140303:214505.077 No active checks on server: host [mango] not found
 23291:20140303:214505.078 In process_active_checks('10.168.24.153',10051)
 23291:20140303:214505.078 End of process_active_checks()
 23291:20140303:214505.078 In get_min_nextcheck()
 23291:20140303:214505.078 Sleeping for 1 second(s)
 23287:20140303:214506.036 In update_cpustats()
 23287:20140303:214506.036 End of update_cpustats()
 23292:20140303:214506.041 In send_buffer() host:'10.185.3.248' port:10051 values:0/100
 23292:20140303:214506.042 End of send_buffer():SUCCEED
 23292:20140303:214506.042 Sleeping for 1 second(s)
 23291:20140303:214506.079 In send_buffer() host:'10.168.24.153' port:10051 values:0/100
 23291:20140303:214506.079 End of send_buffer():SUCCEED
 23291:20140303:214506.079 Sleeping for 1 second(s)

[clearclaw]

Host prodBETA-lb-0 -> Configuration of items -> Item "Agent ping"
Field "Type" can be any from the long list of checks:
- Zabbix agent
- Zabbix agent(active)
- simple check
etc.....

I would like to ask you just change from Zabbix agent to Zabbix agent(active) check.
In this case your proxy will make a list of "Active check" and send it to agent when the agent ask about that list.

Sorry, I don't understand. I don't see that option on this screen:

https://www.dropbox.com/s/xxnnmkqxklgk7hb/Screenshot%20from%202014-03-03%2013%3A57%3A22.png

Should I look somewhere else?

PS Sorry, English is not my language.

Not a problem at all. Your English is far better than my Latvian!

[aib]

When you still have your template linked - you cannot change Type of agent.
If you Unlink the template from host you will have that possibility of editing the field which is now Grey.

As a result of empty list of active checks, you are getting the message about it.

 23291:20140303:214505.037 Sending [{
        "request":"active checks",
        "host":"mango"}]
 23291:20140303:214505.037 Before read
 23292:20140303:214505.039 Got [{
        "response":"failed",
        "info":"host [mango] not found"}]

I think is better to draw a diagram to explain how I understand the active/passive situation.

1) Zabbix server -> zabbix proxy (passive) -> zabbix agent (passive)
This is the typical situation when :
- all check is "Zabbix agent" or SNMP or "simple check"
- Zabbis server(ZS) request the value from Zabbix proxy (ZP) and ZP request Zabbix Agent (ZA). ZA send reply to ZP, ZP send reply to ZS. Done

2) Zabbix Server -> Zabbix proxy (passive) <- zabbix agent (active)
Zabbix server(ZS) request the value from Zabbix proxy (ZP) and ZP Make a list of check for ZA. When ZA request of list of checks from ZP, he gets it and start to collect information for all checks from that list. Then ZA sends data to ZP which save the data until ZS asks ZP about results. When ZS asks ZP, it sends data from internal database to ZS. Done

3) Zabbix Server <- Zabbix proxy (active) <- zabbix agent (active)
ZS make a list of checks and wait for ZP request. When ZP request the list of checks, ZS just send it to ZP. Then ZP wait for request from ZA. When ZA request a list of check, he gets it from ZP and starts to collect data for all checks from that list. When data is collected, ZA initiate the connection to ZP. After all data sent/received, ZP initiate the connection to ZS and send data to server. Done.

4) Zabbix Server <- Zabbix proxy (active) -> zabbix agent (passive)
ZS make a list of checks and wait for ZP request. When ZP request the list of checks, ZS just send it to ZP. Then ZP request the data from ZA. When data is collected, ZP initiate the connection to ZS and send data to server. Done.

The configuration (1) is useful for typical LAN when we have no restriction, no firewalls, no rules about traffic.

The configuration (2) is good for the situation when you have NAT/firewall between the agent and proxy.

The configuration (3) is the way to deal with double NAT/Firewall which was installed between ZS and ZP, and also between ZP and ZA's. Just imagine the case when all your server are sitting in internal DMZ, and your ZP is in external DMZ, and your ZS (surprise!) is outside of your company.

Your situation as I can understand by now is switching between (1) and (4).
When you switch the role of ZP, you change the side which initiate the session between ZS and ZP.

If you Unlink template from the host, all Items will be on the same place but you will have the opportunity to change the type of checks from "Zabbix agent" to "Zabbix agent(active)". In that case you can use the full spectrum of possibilities - from case (1) up to case (4).

Good luck!

[clearclaw]

Thanks, that's a useful explanation. Great.

Yes, we're in #1 now and want to get to #4. You are correct.

I've had to recycle the prodBETA-* systems and move to a new test environment. Sorry for any confusion! The new systems are:

ops-monitor-master: 10.168.24.153 # Not changed -- Zabbix Server
test-mon-master: 10.185.20.42 # Zabbix proxy and agent
test-lb-0: 10.147.197.184 # Zabbix agent

zabbix_proxy.conf:

Server=10.168.24.153
Hostname=test-mon-master
LogFile=/var/log/zabbix/zabbix_proxy.log
LogFileSize=0
PidFile=/var/run/zabbix/zabbix_proxy.pid
DBName=/var/lib/zabbix/zabbix_proxy.db
ProxyOfflineBuffer=24
ExternalScripts=/usr/lib/zabbix/externalscripts
FpingLocation=/usr/bin/fping
Fping6Location=/usr/bin/fping6

zabbix_agentd.conf:

PidFile=/var/run/zabbix/zabbix_agentd.pid
LogFile=/var/log/zabbix/zabbix_agentd.log
LogFileSize=0
EnableRemoteCommands=1
LogRemoteCommands=1
# 10.185.20.42 is the IP of the proxy
Server=10.185.20.42
ServerActive=10.185.20.42
Hostname=test-lb-0
Include=/etc/zabbix/zabbix_agentd.d/

I have changed the "Agent ping" item for the Zabbix Agent template to "Zabbix agent (active)" as you suggest:

https://www.dropbox.com/s/ow7uiomefksmi9i/Screenshot%20from%202014-03-04%2019%3A37%3A36.png

But that doesn't seem to have helped?

zabbix_agentd.log:

 17726:20140305:034419.908 In send_buffer() host:'10.185.20.42' port:10051 values:0/100
 17726:20140305:034419.908 End of send_buffer():SUCCEED
 17726:20140305:034419.908 refresh_active_checks('10.185.20.42',10051)
 17722:20140305:034419.908 In update_cpustats()
 17722:20140305:034419.908 End of update_cpustats()
 17726:20140305:034419.909 Sending [{
        "request":"active checks",
        "host":"test-lb-0"}]
 17726:20140305:034419.909 Before read
 17726:20140305:034419.911 Got [{
        "response":"failed",
        "info":"host [test-lb-0] not found"}]
 17726:20140305:034419.911 In parse_list_of_checks()
 17726:20140305:034419.911 In disable_all_metrics()
 17726:20140305:034419.911 No active checks on server: host [test-lb-0] not found
 17726:20140305:034419.911 In process_active_checks('10.185.20.42',10051)
 17726:20140305:034419.912 End of process_active_checks()
 17726:20140305:034419.912 In get_min_nextcheck()
 17726:20140305:034419.912 Sleeping for 1 second(s)
 17722:20140305:034420.909 In update_cpustats()
 17722:20140305:034420.909 End of update_cpustats()
 17726:20140305:034420.912 In send_buffer() host:'10.185.20.42' port:10051 values:0/100
 17726:20140305:034420.912 End of send_buffer():SUCCEED
 17726:20140305:034420.912 Sleeping for 1 second(s)

Zabbix proxy log: http://pastebin.com/bUeuSCge

What should I try now? Are my configs above correct?