Monitoring VM Clones with same hostname

[tmroberts]

Apologies if this is a topic discussed in the past as I'm new to the forum. I'll try to make this as brief as possible. (loaded statement right?)

Big Picture First:
We have an environment that is 99% virtual with a mixture of Windows and Linux (differing versions therein as well) that we are monitoring with Zabbix. Recently we created a carbon copy (we call it zones) of the systems and installed a Global Traffic Manager in front of it with out changing any host names. The only difference is which VLAN the host(s) use. All of the hosts use internal static IP's (i.e. 172.16.10.xxx) with are then NAT'd and show up as external IP's (i.e. 20.200.yy.xxx where yy is the subnet of the VLAN and xxx is the same as the internal IP).

To make things a little more complicated, every other week, we refresh the VM's from a single gold copy to both zones, without changing the hostname, only the VLAN. So in the end we end up with two vm clones that are identical that sit on different VLANs (the FQDN's might look something like: vm.zonea.lan and vm.zoneb.lan)

Problem:
In zabbix we are able to monitor one set of the VM's by default. No changes required to the template or the agent config file. But the other zone we are only able to check the ping and that's it. I know that we can go on to each agent in one zone or the other and physically change the hostname in the agent config file (i.e. use the FQDN), but that is not ideal as this would have to happen every other week. What we are finding out is even though we have each system set up to use the external IP and is still using the hostname for some metrics. The result is metrics are collected for one system (including auto-discovery items), but not the other.

Question:
Is there some trick in the templates or someplace else that we can get around this issue? I'd even be open to creating a Zone A and Zone B template if that's what it comes down to.

If this isn't clear I'd be happy to provide more clarification. Thanks in advance

[guzzijason]

I don't know about Windows (don't use it), but on my linux hosts, I set the following in my zabbix_agentd.conf to ensure that each one automatically gets it's hostname set to the FQDN, rather than short hostname:

# Get the FQDN rather than short hostname:
# Hostname=
HostnameItem=system.run[hostname -f]

If 'hostname -f' doesn't work for you, you could also bake some custom script into your golden image which returns the FQDN that you expect for each host, and then call that using system.run[].

__Jason

[tmroberts]

I thought I posted this earlier, but it hasn't shown up yet...

Therein lies the problem. Using the global traffic manager, we have a couple different layers in our network. Here's an example:

Server Host name: testvm
Internal IP: 172.16.10.10
Zone A External IP: 10.220.66.10
Zone B External IP: 10.220.68.10

Zone A DNS A Record: testvm.zonea.lan points to 10.220.66.10
Zone B DNS A Record: testvm.zoneb.lan points to 10.220.68.10
CName in Global Traffic Manager: testvm.test.lan

The CName in the global traffic manager (using F5 Big-IP) is testvm.test.lan and points to both. In some cases there is only one A Record, but we still have the CName using the test.lan domain.

At the Host level, because they are pointing to the DNS and/or Big-IP, if I do a hostname -f as you suggested, I get testvm.test.lan

I stumbled on some documentation about macros and found HOST.DNS Is there anyway to force the template to look up the dns fqdn using the IP that is already coming through?

[guzzijason]

I think I'm beginning to understand (forgive me if I'm not!)

Are all of your hosts actually configured with the hostname like "testvm.test.lan"? I was initially under the impression that their local hostnames were either in the zonea.lan or zoneb.lan domains. If they are all under "test.lan", then I can see how that might be tricky.

Next question: does your server connect to the target hosts using the GTM CNAME, or by using their "*.zonea.lan" or "*.zoneb.lan" addresses?

Third question: do you have reverse DNS set up for the internal IP addresses? If so, what do those resolve to?

__Jason

[tmroberts]

I'll try to answer your questions here:

I'll use some real examples and change a few things just to obfuscate them a bit.

In GTM:
say we have a CName called mysql.test.tsi.lan. That CName points to two A-Records mysql.zonea.tsi.lan (10.220.20.10) and mysql.zoneb.tsi.lan (10.220.30.10).

In vCenter/Hosts:
we have two linux hosts named mysql on separate VLANs (for organizational purposes they are in separate folder in vCenter). On the actual hosts, the hostname is mysql.test.tsi.lan with identical internal IP's of 172.16.10.10.

Agent Conf File:
The agent config files are populated with short name of the host (mysql).

In Zabbix:
In zabbix we have two host entries, one of which we had to manually change to host name and IP to match the clone.

As far as how zabbix server connects to each host, I would assume that its using the IP address since that is how the hosts are configured right?

For the reverse DNS Lookup for interal IP's, no we do not have that set up.

[guzzijason]

So the clone servers will both have the same internal IP (on separate VLANs) but different external IPs, is that right?

Are the individual hosts "aware" of what their respective public IPs are, or have some way to look it up, or is that completely transparent to them?

__Jason

[tmroberts]

that is correct and yes, it is completely transparent to the host(s)

[guzzijason]

And are your zabbix agents active or passive?

__Jason

[tmroberts]

They are active.

[guzzijason]

I'm pretty sure you *must* have a unique Hostname (or HostnameItem) in your zabbix_agentd.conf. I'm not really sure the best way to go about this in your situation, though.

Perhaps you can build something into your cloning process so that on first boot (and only first boot) after the clone, a timestamp gets written to a file someplace, like:

date '+%Y%m%d' >/var/lib/zabbix/born-on

And then in your zabbix_agentd.conf, you could set:

#Hostname=
HostnameItem=system.run[printf $(hostname)-$(cat /var/lib/zabbix/born-on)]

Which would result in an agent.hostname value something like 'mysql-20170114'.

I'm really not sure if this would ultimately solve your problem or not - just throwing some ideas out there. Would need to do some testing to see if this actually solves anything or just introduces new problems.

__Jason

[tmroberts]

I thought as much. I'm actually leaning towards some sort of a script that runs after the clone process that appends the appropriate FQDN to the host name in the conf file. Not ideal, but I don't see any other way around this. thanks for your input.

One last question.... I did run across a macro item, or what ever they are called, HOST.DNS. I'm not really sure how or where to use it, but would that force zabbix to take the IP and go check DNS for the hostname?

[guzzijason]

I believe the HOST.DNS macro expands to whatever the "DNS name" is you have configured for a host on the zabbix server (i.e. when you are configuring a new host to monitor in the zabbix UI). Likewise, HOST.IP expands to whatever IP you have configured on the agent interface in the configuration, and HOST.CONN will expand to either the IP *or* the DNS name, depending on whether you have the "IP" or "DNS" option selected on the agent interface.

None of these will help you in configuring the actual agent on the target host, though, unfortunately.

__Jason