2 Zabbix Servers & 1 Zabbix Agent Communication

[ingus.vilnis]

I read it already :-)

Maybe you will have any suggestion for me regarding network discovery in my post

https://www.zabbix.com/forum/showthread.php?t=45667

Thanks anyway for your quick response !

Natalia,

I wrote my thoughts about discovery in that thread for you.

Regarding High availability - what can I say, you have to collect all the ideas and practices available in many resources, draw a plan and then stick to it. We can discuss the ideas here as well, if needed.

Best Regards,
Ingus

[Mukesh]

2 Zabbix Servers & 1 Zabbix Agent Communication

Hi Ingus,


Thanks for support,

i have added 2 zabbix server ip in window zabbix agent host(Server 2008)

now zabbix agent properly communication with 2 zabbix server and telnet also

but triggers not come simultaneously you can check my screen shot with attachment. please see the different both server not update triggers .


Please find the attachment screen shot

Please help me give me solution

Thanks
Regards
Mukesh

[natalia]

Regarding High availability - what can I say, you have to collect all the ideas and practices available in many resources, draw a plan and then stick to it. We can discuss the ideas here as well, if needed.

Best Regards,
Ingus

This is exactly what I did already :-)
I have a plan:


It's still just a plan ...
Let me know if it's clear

Thanks a lot for your help !

[ingus.vilnis]

Mukesh,

What are the update intervals for those three items you showed in the screenshots?

What template do you use for monitoring this host? Is it some default one or custom modified?

The problem in your case might be in the fact that both servers check items at different time moments, counting from the start of server. Even if the templates on both servers are identical, the servers have been started at different times thus checking the items at different points of timeline. You could probably get to closer values if you start both your servers at the exact same second but you can never expect them to happen simultaneously.

Best Regards,
Ingus

[ingus.vilnis]

Natalia,

First of all compliments for the nice picture you have created.

However I am not sure how could you possibly build your environment in such way. I am afraid that you will not be able to synchronize your servers and databases horizontally like in your picture plus having both Zabbix servers writing data into both databases.

Would you consider building your system like in the article mentioned before? http://blog.zabbix.com/wp-content/up...tecture-HA.png

It would be far more simple and reliable to do it like this.

Proxy1  Proxy2  ProxyX
  \       |       /
  Zabbix Server VIP
          |
Server 1 <==> Server2
          |
     Database VIP
          |
     DB1 <==> DB2

How about that?

Best Regards,
Ingus

[natalia]

Natalia,

First of all compliments for the nice picture you have created.

However I am not sure how could you possibly build your environment in such way. I am afraid that you will not be able to synchronize your servers and databases horizontally like in your picture plus having both Zabbix servers writing data into both databases.

Would you consider building your system like in the article mentioned before? http://blog.zabbix.com/wp-content/up...tecture-HA.png

It would be far more simple and reliable to do it like this.

Proxy1  Proxy2  ProxyX
  \       |       /
  Zabbix Server VIP
          |
Server 1 <==> Server2
          |
     Database VIP
          |
     DB1 <==> DB2

How about that?

Best Regards,
Ingus

your plan is more simple ☺ we plan more complex ..
how you plan to sync Server1 and Server2?

I will explain my picture:
we have 7 sites in 7 different locations.
one on this sites is Main and one is Backup.
In each site I will install proxy cluster for HA with 2 VM's that will monitoring all hosts in this site
each proxy will talking with VIP Zabbix server that point to VIP Zabbix server in main site, in case on main site is unreachable , we will switch VIP Zabbix server point to VIP Zabbix server in Backup site, until this all zabbix processes on backup site are stopped.
DB is mysql in configuration 2 master/slave in main site and 2 slave in backup site, no process are writing to slave in backup site until db in main site will be unreachable, if it's happend slave in backup site start be master.
let me know if it's clear (sorry for my English :-) )
what do you think ?
Thanks !!!

[ingus.vilnis]

Natalia,

Don't be sorry for your English! It is perfectly fine and understandable. It's not my native either but the main thing here is to understand each other.

I will probably talk to some HA experts later and discuss your picture.

What I can tell you now is that I don't think you will get such DB replication at the very millisecond the failover happens which will result in either loss of data or system configuration failures. You have a huge environment. What is your server performance now, by the way? (New values per second?) With so many hosts you have the DB written constantly and any failures will cause data gaps which could possibly end up crashing the whole system.

Separate your picture. Draw a horizontal line below the database level. Build HA for Zabbix servers. Set up one main DB which will be used by both servers. Set up HA for DB separately.

How about that?

Best Regards,
Ingus

[natalia]

Thanks you :-)

I don't think you will get such DB replication at the very millisecond the failover happens which will result in either loss of data or system configuration failures.

DB replication is a solution that I got from our DB experts and the sync will be very fast (DB servers with SSD and network also very fast).
There are several scenarios of failover:
1. DB master server in main site crushed - the second DB server will start be master (looks the same like cluster configuration)
2. both DB servers in main site crush - the slave on backup site start be master and zabbix server/web server reconfig manually to working with DB VIP in backup site or stop zabbix server in main site and start zabbix server on backup site with change Zabbix server VIP point to backup zabbix server (this will be done on via DNS or external F5 that not located in any of site (?) )

you are right, some of data can be missing during switch between servers but I think it will be happened in any configuration if DB server crushed ...

You have a huge environment. What is your server performance now, by the way? (New values per second?)

for now, only one site (500 hosts) is monitoring by Zabbix proxy and nothing monitoring by Zabbix server itself .
we config only ~20% of all checks (we are migrate all monitoring from HPOV to zabbix)
New values per second: 68.69
Items : 20000
Triggers: 25500
server and proxy performance are ok, I am monitoring them and increase some parameters if need.

With so many hosts you have the DB written constantly and any failures will cause data gaps which could possibly end up crashing the whole system.

I agree that we will have some gaps maybe but why it will can crashing the whole system ?

Separate your picture. Draw a horizontal line below the database level. Build HA for Zabbix servers. Set up one main DB which will be used by both servers. Set up HA for DB separately.

How about that?

what do you mean HA for Zabbix servers ? cluster ?
I have HA for Zabbix servers in main site and also the same in backup site,
do you think that it's too much ?
I need also solution in case if main site will be unreachable ...
btw I don't think that it's possible to config cluster on 2 servers in different location, am I wrong ?

Thanks a lot again for your help!

[Mukesh]

[QUOTE=ingus.vilnis;151920]Mukesh,

What are the update intervals for those three items you showed in the screenshots?



Really Thanks for Support ingus

Ingus ..

This is my currently network diagram in production.

I want to zabbix server migration current building to another building with NO DOWNTIME.

Thats the my mean 2 zabbix Server communicate with agents because i want to run zabbix server both side (current building to another building) during the migration after migration I will plan zabbix HA with full redundancy.

so please give me your expertise suggestion.
How to make successfully zabbix migration without downtime.

Thanks

Regards
Mukesh

[ingus.vilnis]

Mukesh,

You're welcome!

However it is a bit hard for me to troubleshoot your situation because you don't give back any useful answers to the questions I ask you, for example about update intervals etc. I cannot give you any more useful solutions about that.

Regarding HA. Again the same. I gave you plenty of links to resources for building HA. It is impossible to write a step by step guide for your environment only. Please take your own time to investigate your situation.

Regarding migration.
Here is how I see it.

• Move one fully functional and updated Zabbix server to new building.
• Set up all agents to send data to both old and new servers.
• Forget about delays in trigger intervals. You will not get them at the same second for the reasons I told you already yesterday.
• Shut down the old Zabicom server.
• Build HA from the new Zabbix 2.2.4 server.

Note that this is the easiest way, no real downtime, only the old Zabicom history will be lost. But I don't see this as a problem since you are writing data to both servers already.

Best Regards,
Ingus

[ingus.vilnis]

Natalia,

Thank you for the detailed information and reasonable answers!

As promised, I discussed your HA schema with some other experts.
The comments were all positive about the serious scale oh HA in your environment. You are doing a great job there.

Some notes from us regarding failover scenarios:
You should have DB replication available both ways. In case main DB site crash, you will switch over to backup DB and write your data there. Since you have fast sync, no data loss will be in such case. However, at one point your master DB will turn on again and then you should require all data that have been written to backup available at the master as well to avoid any data loss. Talk to your DB team about this.

what do you mean HA for Zabbix servers ? cluster ?
I have HA for Zabbix servers in main site and also the same in backup site,
do you think that it's too much ?
I need also solution in case if main site will be unreachable ...
btw I don't think that it's possible to config cluster on 2 servers in different location, am I wrong ?

Two Zabbix clusters in two different sites is not a problem. That would be a really bulletproof concept. I don't see a problem in building that. It will not be simple but it can be done. Is it too much? It depends on your needs and money.

2. both DB servers in main site crush - the slave on backup site start be master and zabbix server/web server reconfig manually to working with DB VIP in backup site or stop zabbix server in main site and start zabbix server on backup site with change Zabbix server VIP point to backup zabbix server (this will be done on via DNS or external F5 that not located in any of site (?) )

Avoid manual actions which you told me about switching servers. It can be configurable to have all these parts working automatically. Again, this is more about networking and servers, less about Zabbix. I believe you also have experts on your team about these issues.

Natalia, I think you can stick with this idea and go forward to implement this. What improvements can you make to this idea? Well, as I said already before, make the clusters level by level - DB - Zabbix server - Zabbix frontend. In the worst case scenario you will switch to backup location. However you can set it up also in a way that if Zabbix server crash at main site, it switches to backup location, however that does not mean that DB at main site is down. So think of ways to make this even more elastic.

I really hope I did not miss any of your questions from previous posts. If there is anything else unclear, please ask or simply repeat the missed questions.

Best Regards,
Ingus

[natalia]

You should have DB replication available both ways. In case main DB site crash, you will switch over to backup DB and write your data there. Since you have fast sync, no data loss will be in such case. However, at one point your master DB will turn on again and then you should require all data that have been written to backup available at the master as well to avoid any data loss. Talk to your DB team about this.

you raise a good point !!!!! I checked with DB team and we don't have "real" solution to go back when DB master in main site turn on again.
as a solution, create dump (very large) in Backup site, copy it to Main site (it will effect network), recreate both DB servers in main site with upload dump ... maybe we should think about solution but for now we have no idea (:- there is no built-in solution in mysql, just if to use any other tools ...
Do you have any suggestions ?

as I said already before, make the clusters level by level - DB - Zabbix server - Zabbix frontend. In the worst case scenario you will switch to backup location. However you can set it up also in a way that if Zabbix server crash at main site, it switches to backup location, however that does not mean that DB at main site is down. So think of ways to make this even more elastic.

it sounds more correct and easy than my configuration. I will think again :-)
could you give me more details what do you mean cluster zabbix server ? if I have main and backup site.

Thanks a lot !

[ingus.vilnis]

Natalia,

For full DB replication you can try Galera cluster. http://galeracluster.com/
Dump backup and recreate main? No, please, no...

For Zabbix server cluster you can simply use failover IP scenario.

Data to Zabbix server goes in to VIP which
   goes to main site cluster server 1
      if not available then to main site cluster server 2
         if not available then to backup site cluster server 1
            if not available then to backup site cluster server 2
               if not available then to main site cluster server 1
and so on

Something like that.

Ingus

[natalia]

Natalia,

For full DB replication you can try Galera cluster. http://galeracluster.com/
Dump backup and recreate main? No, please, no...

For Zabbix server cluster you can simply use failover IP scenario.

Data to Zabbix server goes in to VIP which
   goes to main site cluster server 1
      if not available then to main site cluster server 2
         if not available then to backup site cluster server 1
            if not available then to backup site cluster server 2
               if not available then to main site cluster server 1
and so on

Something like that.

Ingus

Thanks a lot ! will check and prepere the new questions :-)

[aib]

also ask you DB team about MySQL HA cluster solution.
for example, MySQL Cluster: High Availability

it that case you will have HA Zabbix cluster + HA MySQL cluster and you will be on safe side.

btw, both clusters can have remote sites in different (geographically) places.