Hi All,
I'm experiencing somewhat of an issue with child nodes sending/receiving configuration changes to the master node.
My DM setup:
2 x Dedicated MySQL servers and a Master Node server at a 'home' data center
--> Master Node uses the DB servers
2 x Child Node servers (each located at a remote data center)
--> Each child has its own DB instance running locally
(Side note: the 3 DCs are connected together via dedicated 1G MPLS circuits)
The problem I am experiencing is the lag time through the child to master configuration synchronization. While the synchronization process is taking place, communication about alerts and events is suspended temporarily. When triggers fire off from one of the child nodes, they may take over a minute to show up on the master node's GUI. This is leading to issues where techs receive alarms but don't see them on their dashboard right away.
When it is time for the child node to synchronize, this is what I see from the server logs:
I am hoping that someone can give me some pointers in helping speed up the configuration process with these nodes. I'm not sure if I should be looking further into database performance or otherwise. The lag does seem to be dependent on the datalen of the configuration event. (ie: the higher the datalength, the longer it takes)
Or is it possible for the events/alerts to continue sending to the master node while the config is syncing?
I would really appreciate anyone's insight or recommendation on this.
More info:
Zabbix 2.2.2 (on all)
MySQL 5.6 (on all)
Master Node: 240,000 items (491vps)
Child Nodes: 40,000 items (75vps)
Thanks !!
I'm experiencing somewhat of an issue with child nodes sending/receiving configuration changes to the master node.
My DM setup:
2 x Dedicated MySQL servers and a Master Node server at a 'home' data center
--> Master Node uses the DB servers
2 x Child Node servers (each located at a remote data center)
--> Each child has its own DB instance running locally
(Side note: the 3 DCs are connected together via dedicated 1G MPLS circuits)
The problem I am experiencing is the lag time through the child to master configuration synchronization. While the synchronization process is taking place, communication about alerts and events is suspended temporarily. When triggers fire off from one of the child nodes, they may take over a minute to show up on the master node's GUI. This is leading to issues where techs receive alarms but don't see them on their dashboard right away.
When it is time for the child node to synchronize, this is what I see from the server logs:
- Sending of history/events/alerts temporarily stops (Im assuming by design)
- A period of about 20-30 seconds passes before the log on the child node shows "sending configuration changes"
- 5 - 10 seconds pass before master node shows "received configuration changes"
- Another 20-30 seconds before master node shows "sending configuration changes" back to child node
- 1-2 seconds for child node to show "received configuration changes" from master
- Sending of history/events/alerts resumes
I am hoping that someone can give me some pointers in helping speed up the configuration process with these nodes. I'm not sure if I should be looking further into database performance or otherwise. The lag does seem to be dependent on the datalen of the configuration event. (ie: the higher the datalength, the longer it takes)
Or is it possible for the events/alerts to continue sending to the master node while the config is syncing?
I would really appreciate anyone's insight or recommendation on this.
More info:
Zabbix 2.2.2 (on all)
MySQL 5.6 (on all)
Master Node: 240,000 items (491vps)
Child Nodes: 40,000 items (75vps)
Thanks !!
Comment