Distributed Monitoring Not Working

[side_control]

I have three nodes, two works which work and one that just refuses to work.

Issues with non working node.
1. After adding the node, after a while it automatically removes itself, it's also missing from the zabbix.nodes table
2. The first data transfer it will send *some* data over but stops right after that.
3. New nodes stops logging completely but still sends alerts.

grep NODE /tmp/* (master node)

zabbix_server.log:  2801:20090421:133756 NODE 1: Received history from node 2 for node 2 datalen 350595
zabbix_server.log:  2802:20090421:133819 NODE 1: Received data from slave node 2 for node 2 datalen 110
zabbix_server.log:  2804:20090421:133908 NODE 1: Received history from node 2 for node 2 datalen 350595
zabbix_server.log:  2802:20090421:133928 NODE 1: Received history_uint from node 2 for node 2 datalen 336608
zabbix_server.log:  2804:20090421:133944 NODE 1: Received history_str from node 2 for node 2 datalen 464
zabbix_server.log:  2803:20090421:133948 NODE 1: Received events from node 2 for node 2 datalen 222
zabbix_server.log:  2803:20090421:133950 NODE 1: Received history from node 2 for node 2 datalen 351798
zabbix_server.log:  2801:20090421:134012 NODE 1: Received data from slave node 2 for node 2 datalen 11626
zabbix_server.log:  2805:20090421:134116 NODE 1: Received history from node 2 for node 2 datalen 351798
zabbix_server.log:  2805:20090421:134140 NODE 1: Received history_uint from node 2 for node 2 datalen 336727
zabbix_server.log:  2804:20090421:134151 NODE 1: Received history_str from node 2 for node 2 datalen 1520
zabbix_server.log:  2805:20090421:134152 NODE 1: Received events from node 2 for node 2 datalen 14553
zabbix_server.log:  2805:20090421:134154 NODE 1: Received history from node 2 for node 2 datalen 351067
zabbix_server.log:  2804:20090421:134213 NODE 1: Received history_uint from node 2 for node 2 datalen 336667
zabbix_server.log:  2803:20090421:134223 NODE 1: Received history_str from node 2 for node 2 datalen 554
zabbix_server.log:  2801:20090421:134232 NODE 1: Received data from slave node 2 for node 2 datalen 7148
zabbix_server.log:  2801:20090421:134321 NODE 1: Received history from node 2 for node 2 datalen 351663
zabbix_server.log:  2803:20090421:134331 NODE 1: Received history_uint from node 2 for node 2 datalen 335475
zabbix_server.log:  2801:20090421:134341 NODE 1: Received history_str from node 2 for node 2 datalen 458
zabbix_server.log:  2801:20090421:134345 NODE 1: Received history from node 2 for node 2 datalen 351546
zabbix_server.log:  2802:20090421:134400 NODE 1: Received history_uint from node 2 for node 2 datalen 329297
zabbix_server.log:  2805:20090421:134408 NODE 1: Received history_str from node 2 for node 2 datalen 248
zabbix_server.log:  2804:20090421:134414 NODE 1: Received history from node 2 for node 2 datalen 351729
zabbix_server.log:  2805:20090421:134429 NODE 1: Received history_uint from node 2 for node 2 datalen 330067
zabbix_server.log:  2801:20090421:134438 NODE 1: Received history_str from node 2 for node 2 datalen 386
zabbix_server.log:  2805:20090421:134438 NODE 1: Received events from node 2 for node 2 datalen 69
zabbix_server.log:  2805:20090421:134446 NODE 1: Received data from slave node 2 for node 2 datalen 78
zabbix_server.log:  2803:20090421:134508 NODE 1: Received history from node 2 for node 2 datalen 352172
zabbix_server.log:  2801:20090421:134520 NODE 1: Received history_uint from node 2 for node 2 datalen 334517
zabbix_server.log:  2805:20090421:134529 NODE 1: Received history_str from node 2 for node 2 datalen 248
zabbix_server.log:  2802:20090421:134535 NODE 1: Received history from node 2 for node 2 datalen 351670
zabbix_server.log:  2804:20090421:134559 NODE 1: Received history_uint from node 2 for node 2 datalen 336728
zabbix_server.log:  2805:20090421:134608 NODE 1: Received history_str from node 2 for node 2 datalen 600
zabbix_server.log:  2803:20090421:134619 NODE 1: Received history from node 2 for node 2 datalen 350855
zabbix_server.log:  2805:20090421:134633 NODE 1: Received history_uint from node 2 for node 2 datalen 336554
zabbix_server.log:  2803:20090421:134646 NODE 1: Received data from slave node 2 for node 2 datalen 8
zabbix_server.log:  2802:20090421:134709 NODE 1: Received history from node 2 for node 2 datalen 352072
zabbix_server.log:  2804:20090421:134710 NODE 1: Received data from slave node 4 for node 4 datalen 4511806
zabbix_server.log:  2801:20090421:134720 NODE 1: Received history_uint from node 2 for node 2 datalen 336853
zabbix_server.log:  2805:20090421:134731 NODE 1: Received history_str from node 2 for node 2 datalen 464
zabbix_server.log:  2805:20090421:134746 NODE 1: Received history from node 2 for node 2 datalen 351534
zabbix_server.log:  2801:20090421:134758 NODE 1: Received history_uint from node 2 for node 2 datalen 336629
zabbix_server.log:  2802:20090421:134808 NODE 1: Received history_str from node 2 for node 2 datalen 248
zabbix_server.log:  2803:20090421:134812 NODE 1: Received history from node 2 for node 2 datalen 351256
zabbix_server.log:  2801:20090421:134826 NODE 1: Received history_uint from node 2 for node 2 datalen 336683
zabbix_server.log:  2803:20090421:134836 NODE 1: Received history_str from node 2 for node 2 datalen 238
zabbix_server.log:  2803:20090421:134844 NODE 1: Received history from node 2 for node 2 datalen 351253
zabbix_server.log:  2805:20090421:134855 NODE 1: Received history_uint from node 2 for node 2 datalen 336525
zabbix_server.log:  2802:20090421:135343 NODE 1: Received data from slave node 4 for node 4 datalen 4511806
zabbix_server.log:  2801:20090421:135412 NODE 1: Received data from slave node 2 for node 2 datalen 8
zabbix_server.log:  2805:20090421:135432 NODE 1: Received history from node 2 for node 2 datalen 351719
zabbix_server.log:  2803:20090421:135452 NODE 1: Received history_uint from node 2 for node 2 datalen 336588
zabbix_server.log:  2805:20090421:135503 NODE 1: Received history_str from node 2 for node 2 datalen 2196
zabbix_server.log:  2801:20090421:135509 NODE 1: Received data from slave node 2 for node 2 datalen 8
zabbix_server.log:  2803:20090421:135549 NODE 1: Received history from node 2 for node 2 datalen 351074
zabbix_server.log:  2803:20090421:135626 NODE 1: Received history_uint from node 2 for node 2 datalen 336622
zabbix_server.log.old:  2147:20090421:133746 NODE 1: Received history from node 2 for node 2 datalen 350595

Broken Node

tail -f zabbix_server.log
  2642:20090421:134957 Executing housekeeper
  2645:20090421:134957 server #15 started [Poller for unreachable hosts. SNMP:YES]
  2647:20090421:134957 server #16 started [Node watcher. Node ID:4]
  2650:20090421:134957 server #17 started [HTTP Poller]
  2652:20090421:134957 server #18 started [Discoverer. SNMP:YES]
  2653:20090421:134957 server #19 started [Escalator]
  2621:20090421:134957 server #0 started [Watchdog]
  2621:20090421:134957 In main_watchdog_loop()
  2642:20090421:135002 Deleted 0 records from history and trends
  2647:20090421:135005 NODE 4: Sending configuration changes to master node 1 for node 4 datalen 4511807

Any help is appreciated.

[xs-]

What zabbix/database version.

[side_control]

All three nodes are 1.6.4 and mysql is 5.0.45.

[Senator]

Hi

maybe you have same problem as we - http://www.zabbix.com/forum/showthre...ghlight=events

Regards
Tobias