Table history_uint does not sync at the same speed as the other tables
I was wondering if anyone was experiencing the same thing here. I reported it as Zabbix bug ZBX-5413: https://support.zabbix.com/browse/ZBX-5413
I'm encountering an issue with the history_uint tables not sync'ing to the master node at the same speed as the other tables, causing the integer values to lag/run behind, all other values are fine. This problem may only occur on very large installations, as the tables are actually sync'ing, just one table is sync'ing too slowly for us.
I suspect that there is a limit in the code (didn't check this) somewhere, because all tables go over 400K bytes every now and then, especially when something big happens, except the history_uint table. I've never seen any sync of that table with a size hitting 400K, it's always something like 395K or so, while that table is running behind several days, and it's getting worse (it's a week at the moment and increasing, but the values ARE updating, just too slowly to keep up). Why isn't the sync sync'ing more data than 395K, when it's running behind that much? I should sync all missing data at once! If you look at the sync speeds, this is actually quite high, it takes about 2 minutes for a host to sync.
Don't pay attention to nodes 5 and 6, they are not monitoring at the moment (temporarily suspended), that's why they are only sync'ing config, but no history tables.
Very large distributed setup, 1 master node, 5 child nodes, all on CentOS 6.2 64-bit, MySQL 5.5:
Number of hosts (monitored/not monitored/templates) 12043 11664 / 116 / 263
Number of items (monitored/disabled/not supported) 688965 647656 / 2571 / 38738
Number of triggers (enabled/disabled)[problem/unknown/ok] 194100 159297 / 34803 [709 / 0 / 158588]
Number of users (online) 197 5
Required server performance, new values per second 225.83 -
Node 2 is the MASTER.
5531:20120803:111521.326 NODE 2: Received history from node 3 for node 3 datalen 443005
5531:20120803:111536.455 NODE 2: Received history_uint from node 3 for node 3 datalen 396645
5531:20120803:111555.261 NODE 2: Received history_str from node 3 for node 3 datalen 132367
5531:20120803:111555.755 NODE 2: Received history_text from node 3 for node 3 datalen 19195
5531:20120803:111556.048 NODE 2: Received events from node 3 for node 3 datalen 2194
5521:20120803:111658.997 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:111702.549 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5517:20120803:111715.534 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11350
5517:20120803:111734.200 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5531:20120803:111813.913 NODE 2: Received history from node 1 for node 1 datalen 100370
5531:20120803:111816.536 NODE 2: Received history_uint from node 1 for node 1 datalen 214517
5531:20120803:111831.605 NODE 2: Received history_str from node 1 for node 1 datalen 48827
5531:20120803:111831.774 NODE 2: Received history_text from node 1 for node 1 datalen 843
5531:20120803:111831.820 NODE 2: Received events from node 1 for node 1 datalen 2518
5518:20120803:111855.844 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13684
5518:20120803:111947.278 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:112023.346 NODE 2: Received history from node 4 for node 4 datalen 323253
5542:20120803:112032.608 NODE 2: Received history_uint from node 4 for node 4 datalen 392062
5529:20120803:112055.617 NODE 2: Received history_str from node 4 for node 4 datalen 130125
5514:20120803:112055.930 NODE 2: Received history_text from node 4 for node 4 datalen 7032
5536:20120803:112206.922 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:112207.403 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5520:20120803:112209.701 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 183258
5520:20120803:112303.249 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:112335.860 NODE 2: Received history from node 3 for node 3 datalen 443058
5542:20120803:112350.530 NODE 2: Received history_uint from node 3 for node 3 datalen 396611
5517:20120803:112408.939 NODE 2: Received history_str from node 3 for node 3 datalen 130895
5517:20120803:112409.265 NODE 2: Received history_text from node 3 for node 3 datalen 18562
5529:20120803:112409.584 NODE 2: Received events from node 3 for node 3 datalen 2579
5521:20120803:112511.723 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:112514.787 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:112528.235 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12582
5519:20120803:112550.751 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5525:20120803:112629.988 NODE 2: Received history from node 1 for node 1 datalen 112019
5537:20120803:112633.314 NODE 2: Received history_uint from node 1 for node 1 datalen 231831
5537:20120803:112650.023 NODE 2: Received history_str from node 1 for node 1 datalen 70526
5537:20120803:112650.216 NODE 2: Received history_text from node 1 for node 1 datalen 1183
5537:20120803:112650.284 NODE 2: Received events from node 1 for node 1 datalen 2352
5514:20120803:112709.772 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13514
5514:20120803:112804.221 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5525:20120803:112839.229 NODE 2: Received history from node 4 for node 4 datalen 326743
5525:20120803:112848.894 NODE 2: Received history_uint from node 4 for node 4 datalen 391996
5525:20120803:112911.255 NODE 2: Received history_str from node 4 for node 4 datalen 121776
5528:20120803:112911.579 NODE 2: Received history_text from node 4 for node 4 datalen 5349
5536:20120803:113019.849 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:113020.329 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:113022.114 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 25711
5543:20120803:113118.650 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5513:20120803:113151.167 NODE 2: Received history from node 3 for node 3 datalen 443548
5513:20120803:113208.302 NODE 2: Received history_uint from node 3 for node 3 datalen 395505
5513:20120803:113225.817 NODE 2: Received history_str from node 3 for node 3 datalen 128866
5513:20120803:113226.311 NODE 2: Received history_text from node 3 for node 3 datalen 26341
5513:20120803:113226.678 NODE 2: Received events from node 3 for node 3 datalen 2802
5521:20120803:113328.522 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:113331.553 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:113345.389 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 10252
5519:20120803:113408.591 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5513:20120803:113449.012 NODE 2: Received history from node 1 for node 1 datalen 112909
5513:20120803:113452.530 NODE 2: Received history_uint from node 1 for node 1 datalen 262472
5513:20120803:113511.365 NODE 2: Received history_str from node 1 for node 1 datalen 68640
5515:20120803:113511.615 NODE 2: Received history_text from node 1 for node 1 datalen 1001
5539:20120803:113511.678 NODE 2: Received events from node 1 for node 1 datalen 3398
5528:20120803:113528.370 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14449
5528:20120803:113622.746 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5525:20120803:113657.802 NODE 2: Received history from node 4 for node 4 datalen 323329
5537:20120803:113707.471 NODE 2: Received history_uint from node 4 for node 4 datalen 391869
5525:20120803:113730.552 NODE 2: Received history_str from node 4 for node 4 datalen 125779
5520:20120803:113730.846 NODE 2: Received history_text from node 4 for node 4 datalen 5035
5536:20120803:113838.352 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:113838.847 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:113841.612 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 26063
5543:20120803:113940.059 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:114012.332 NODE 2: Received history from node 3 for node 3 datalen 443429
5531:20120803:114028.629 NODE 2: Received history_uint from node 3 for node 3 datalen 397121
5523:20120803:114046.755 NODE 2: Received history_str from node 3 for node 3 datalen 141948
5523:20120803:114047.182 NODE 2: Received history_text from node 3 for node 3 datalen 48697
5523:20120803:114047.547 NODE 2: Received events from node 3 for node 3 datalen 3900
5521:20120803:114149.438 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:114152.485 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:114204.783 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 14756
5519:20120803:114229.600 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5542:20120803:114308.694 NODE 2: Received history from node 1 for node 1 datalen 109585
5542:20120803:114312.525 NODE 2: Received history_uint from node 1 for node 1 datalen 243135
5542:20120803:114329.426 NODE 2: Received history_str from node 1 for node 1 datalen 47932
5542:20120803:114329.652 NODE 2: Received history_text from node 1 for node 1 datalen 846
5542:20120803:114329.695 NODE 2: Received events from node 1 for node 1 datalen 3296
5539:20120803:114348.166 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15523
5539:20120803:114440.661 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5542:20120803:114515.411 NODE 2: Received history from node 4 for node 4 datalen 312201
5542:20120803:114525.828 NODE 2: Received history_uint from node 4 for node 4 datalen 392273
5542:20120803:114547.734 NODE 2: Received history_str from node 4 for node 4 datalen 123651
5542:20120803:114548.044 NODE 2: Received history_text from node 4 for node 4 datalen 4743
5536:20120803:114658.078 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:114658.559 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:114700.239 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29921
5543:20120803:114759.609 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:114833.786 NODE 2: Received history from node 3 for node 3 datalen 443004
5523:20120803:114849.640 NODE 2: Received history_uint from node 3 for node 3 datalen 396099
5523:20120803:114907.383 NODE 2: Received history_str from node 3 for node 3 datalen 134588
5518:20120803:114907.801 NODE 2: Received history_text from node 3 for node 3 datalen 56545
5516:20120803:114908.204 NODE 2: Received events from node 3 for node 3 datalen 3715
5520:20120803:115011.428 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5520:20120803:115015.084 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:115027.386 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12627
5519:20120803:115049.961 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5515:20120803:115130.610 NODE 2: Received history from node 1 for node 1 datalen 121779
5515:20120803:115134.618 NODE 2: Received history_uint from node 1 for node 1 datalen 224083
5525:20120803:115152.669 NODE 2: Received history_str from node 1 for node 1 datalen 47794
5539:20120803:115152.798 NODE 2: Received history_text from node 1 for node 1 datalen 1004
5539:20120803:115152.843 NODE 2: Received events from node 1 for node 1 datalen 2792
5521:20120803:115209.677 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14310
5521:20120803:115301.825 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:115334.683 NODE 2: Received history from node 4 for node 4 datalen 320118
5525:20120803:115344.476 NODE 2: Received history_uint from node 4 for node 4 datalen 391732
5513:20120803:115406.588 NODE 2: Received history_str from node 4 for node 4 datalen 132676
5515:20120803:115406.896 NODE 2: Received history_text from node 4 for node 4 datalen 5026
5536:20120803:115515.958 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:115516.441 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:115519.261 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 53369
5543:20120803:115613.294 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5523:20120803:115648.665 NODE 2: Received history from node 3 for node 3 datalen 442680
5523:20120803:115706.704 NODE 2: Received history_uint from node 3 for node 3 datalen 397391
5514:20120803:115731.302 NODE 2: Received history_str from node 3 for node 3 datalen 163601
5528:20120803:115731.835 NODE 2: Received history_text from node 3 for node 3 datalen 43015
5525:20120803:115732.169 NODE 2: Received events from node 3 for node 3 datalen 6488
5520:20120803:115828.939 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5520:20120803:115832.013 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:115846.696 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11397
5519:20120803:115908.260 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5542:20120803:115946.914 NODE 2: Received history from node 1 for node 1 datalen 104991
5529:20120803:115949.804 NODE 2: Received history_uint from node 1 for node 1 datalen 222859
5529:20120803:120005.399 NODE 2: Received history_str from node 1 for node 1 datalen 39662
5529:20120803:120005.503 NODE 2: Received history_text from node 1 for node 1 datalen 1506
5529:20120803:120005.561 NODE 2: Received events from node 1 for node 1 datalen 2618
5515:20120803:120025.920 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14525
5515:20120803:120135.351 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5536:20120803:120212.637 NODE 2: Received history from node 4 for node 4 datalen 330907
5543:20120803:120222.291 NODE 2: Received history_uint from node 4 for node 4 datalen 391685
5543:20120803:120244.512 NODE 2: Received history_str from node 4 for node 4 datalen 145511
5513:20120803:120244.877 NODE 2: Received history_text from node 4 for node 4 datalen 14526
5539:20120803:120353.335 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:120353.823 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5520:20120803:120355.960 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 39383
5520:20120803:120450.879 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:120526.796 NODE 2: Received history from node 3 for node 3 datalen 442859
5515:20120803:120543.733 NODE 2: Received history_uint from node 3 for node 3 datalen 396221
5515:20120803:120600.855 NODE 2: Received history_str from node 3 for node 3 datalen 158744
5515:20120803:120601.330 NODE 2: Received history_text from node 3 for node 3 datalen 45534
5515:20120803:120601.694 NODE 2: Received events from node 3 for node 3 datalen 2964
5521:20120803:120700.830 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:120703.845 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:120716.982 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11532
5519:20120803:120735.828 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:120814.520 NODE 2: Received history from node 1 for node 1 datalen 108863
5543:20120803:120817.361 NODE 2: Received history_uint from node 1 for node 1 datalen 225009
5543:20120803:120834.586 NODE 2: Received history_str from node 1 for node 1 datalen 51035
5543:20120803:120834.701 NODE 2: Received history_text from node 1 for node 1 datalen 2500
5543:20120803:120834.775 NODE 2: Received events from node 1 for node 1 datalen 3466
5513:20120803:120853.872 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15127
5513:20120803:120945.765 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5520:20120803:121021.364 NODE 2: Received history from node 4 for node 4 datalen 308145
5533:20120803:121030.371 NODE 2: Received history_uint from node 4 for node 4 datalen 391715
5533:20120803:121052.367 NODE 2: Received history_str from node 4 for node 4 datalen 143454
5529:20120803:121052.739 NODE 2: Received history_text from node 4 for node 4 datalen 17819
5539:20120803:121203.734 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:121204.218 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:121206.334 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 284636
5521:20120803:121311.227 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5520:20120803:121345.794 NODE 2: Received history from node 3 for node 3 datalen 443110
5520:20120803:121402.412 NODE 2: Received history_uint from node 3 for node 3 datalen 396096
5520:20120803:121419.478 NODE 2: Received history_str from node 3 for node 3 datalen 144233
5520:20120803:121419.979 NODE 2: Received history_text from node 3 for node 3 datalen 49993
5520:20120803:121420.378 NODE 2: Received events from node 3 for node 3 datalen 3452
5515:20120803:121523.697 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:121526.703 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:121539.898 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13715
5519:20120803:121558.949 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5514:20120803:121643.481 NODE 2: Received history from node 1 for node 1 datalen 105929
5514:20120803:121646.124 NODE 2: Received history_uint from node 1 for node 1 datalen 231451
5514:20120803:121702.902 NODE 2: Received history_str from node 1 for node 1 datalen 50292
5514:20120803:121703.014 NODE 2: Received history_text from node 1 for node 1 datalen 2016
5514:20120803:121703.074 NODE 2: Received events from node 1 for node 1 datalen 2741
5529:20120803:121722.926 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 12259
5529:20120803:121818.856 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5514:20120803:121855.227 NODE 2: Received history from node 4 for node 4 datalen 332895
5514:20120803:121905.265 NODE 2: Received history_uint from node 4 for node 4 datalen 391387
5523:20120803:121927.572 NODE 2: Received history_str from node 4 for node 4 datalen 149823
5523:20120803:121928.600 NODE 2: Received history_text from node 4 for node 4 datalen 11034
5539:20120803:122036.692 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:122037.173 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:122039.520 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29947
5521:20120803:122131.397 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5514:20120803:122204.369 NODE 2: Received history from node 3 for node 3 datalen 443178
5514:20120803:122220.576 NODE 2: Received history_uint from node 3 for node 3 datalen 395831
5514:20120803:122238.234 NODE 2: Received history_str from node 3 for node 3 datalen 134196
5514:20120803:122238.706 NODE 2: Received history_text from node 3 for node 3 datalen 49058
5514:20120803:122239.118 NODE 2: Received events from node 3 for node 3 datalen 3192
5514:20120803:122239.153 NODE 2: Received acknowledges from node 3 for node 3 datalen 124
5515:20120803:122341.701 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:122344.727 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:122357.509 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12989
5519:20120803:122418.858 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5514:20120803:122456.832 NODE 2: Received history from node 1 for node 1 datalen 112497
5514:20120803:122500.100 NODE 2: Received history_uint from node 1 for node 1 datalen 237948
5514:20120803:122517.399 NODE 2: Received history_str from node 1 for node 1 datalen 57979
5514:20120803:122517.575 NODE 2: Received history_text from node 1 for node 1 datalen 1671
5514:20120803:122517.631 NODE 2: Received events from node 1 for node 1 datalen 2952
5523:20120803:122536.214 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15025
5523:20120803:122631.633 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5529:20120803:122705.775 NODE 2: Received history from node 4 for node 4 datalen 316808
5529:20120803:122715.688 NODE 2: Received history_uint from node 4 for node 4 datalen 392075
5529:20120803:122738.970 NODE 2: Received history_str from node 4 for node 4 datalen 143217
5529:20120803:122739.290 NODE 2: Received history_text from node 4 for node 4 datalen 19563
5539:20120803:122846.841 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:122847.322 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:122849.474 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 27114
5521:20120803:122948.157 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:123019.900 NODE 2: Received history from node 3 for node 3 datalen 443132
5539:20120803:123035.123 NODE 2: Received history_uint from node 3 for node 3 datalen 395533
5539:20120803:123054.304 NODE 2: Received history_str from node 3 for node 3 datalen 150855
5539:20120803:123054.744 NODE 2: Received history_text from node 3 for node 3 datalen 48957
5539:20120803:123055.166 NODE 2: Received events from node 3 for node 3 datalen 3544
5515:20120803:123156.956 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:123159.972 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:123212.888 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12619
5519:20120803:123231.778 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5539:20120803:123313.576 NODE 2: Received history from node 1 for node 1 datalen 106490
5539:20120803:123316.712 NODE 2: Received history_uint from node 1 for node 1 datalen 242031
5539:20120803:123333.989 NODE 2: Received history_str from node 1 for node 1 datalen 56525
5539:20120803:123334.191 NODE 2: Received history_text from node 1 for node 1 datalen 2053
5539:20120803:123334.243 NODE 2: Received events from node 1 for node 1 datalen 2461
5529:20120803:123352.927 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15372
5529:20120803:123443.082 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:123518.408 NODE 2: Received history from node 4 for node 4 datalen 325252
5531:20120803:123530.089 NODE 2: Received history_uint from node 4 for node 4 datalen 392019
5539:20120803:123552.392 NODE 2: Received history_str from node 4 for node 4 datalen 149438
5539:20120803:123552.813 NODE 2: Received history_text from node 4 for node 4 datalen 4111
5523:20120803:123659.389 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:123659.870 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:123701.725 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 30557
5521:20120803:123802.975 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:123835.138 NODE 2: Received history from node 3 for node 3 datalen 443071
5537:20120803:123851.859 NODE 2: Received history_uint from node 3 for node 3 datalen 396377
5537:20120803:123908.993 NODE 2: Received history_str from node 3 for node 3 datalen 151620
5537:20120803:123909.518 NODE 2: Received history_text from node 3 for node 3 datalen 72626
5537:20120803:123909.878 NODE 2: Received events from node 3 for node 3 datalen 3187
5515:20120803:124011.658 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:124014.672 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:124028.830 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 9432
5519:20120803:124047.102 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5537:20120803:124128.143 NODE 2: Received history from node 1 for node 1 datalen 119410
5528:20120803:124132.915 NODE 2: Received history_uint from node 1 for node 1 datalen 278229
5528:20120803:124151.528 NODE 2: Received history_str from node 1 for node 1 datalen 57823
5514:20120803:124151.763 NODE 2: Received history_text from node 1 for node 1 datalen 1633
5514:20120803:124151.817 NODE 2: Received events from node 1 for node 1 datalen 2614
5529:20120803:124207.983 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14696
5529:20120803:124257.894 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:124331.787 NODE 2: Received history from node 4 for node 4 datalen 309336
5539:20120803:124341.113 NODE 2: Received history_uint from node 4 for node 4 datalen 392183
5525:20120803:124404.480 NODE 2: Received history_str from node 4 for node 4 datalen 141133
5525:20120803:124404.804 NODE 2: Received history_text from node 4 for node 4 datalen 18420
5523:20120803:124513.247 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:124513.733 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:124515.798 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 21855
5513:20120803:124620.941 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5521:20120803:124652.937 NODE 2: Received history from node 3 for node 3 datalen 443105
5521:20120803:124709.937 NODE 2: Received history_uint from node 3 for node 3 datalen 396404
5521:20120803:124726.887 NODE 2: Received history_str from node 3 for node 3 datalen 149411
5521:20120803:124727.325 NODE 2: Received history_text from node 3 for node 3 datalen 65594
5521:20120803:124727.669 NODE 2: Received events from node 3 for node 3 datalen 3542
5515:20120803:124830.452 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:124833.560 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:124847.107 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12272
5514:20120803:124906.130 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5525:20120803:124946.451 NODE 2: Received history from node 1 for node 1 datalen 122938
5525:20120803:124950.116 NODE 2: Received history_uint from node 1 for node 1 datalen 246729
5517:20120803:125007.424 NODE 2: Received history_str from node 1 for node 1 datalen 62666
5517:20120803:125008.051 NODE 2: Received history_text from node 1 for node 1 datalen 2738
5517:20120803:125008.195 NODE 2: Received events from node 1 for node 1 datalen 2100
5531:20120803:125025.614 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13283
5531:20120803:125123.257 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5517:20120803:125156.420 NODE 2: Received history from node 4 for node 4 datalen 327200
5517:20120803:125207.434 NODE 2: Received history_uint from node 4 for node 4 datalen 392154
5525:20120803:125230.151 NODE 2: Received history_str from node 4 for node 4 datalen 142021
5525:20120803:125230.498 NODE 2: Received history_text from node 4 for node 4 datalen 4915
5523:20120803:125337.349 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:125337.843 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:125339.851 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29935
5513:20120803:125437.929 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:125510.027 NODE 2: Received history from node 3 for node 3 datalen 443199
5531:20120803:125526.269 NODE 2: Received history_uint from node 3 for node 3 datalen 396729
5533:20120803:125543.972 NODE 2: Received history_str from node 3 for node 3 datalen 144821
5533:20120803:125544.498 NODE 2: Received history_text from node 3 for node 3 datalen 61374
5533:20120803:125544.853 NODE 2: Received events from node 3 for node 3 datalen 1959
5515:20120803:125647.357 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:125650.399 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:125704.235 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 9425
5514:20120803:125723.595 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5531:20120803:125804.967 NODE 2: Received history from node 1 for node 1 datalen 107392
5531:20120803:125808.070 NODE 2: Received history_uint from node 1 for node 1 datalen 247474
5531:20120803:125824.563 NODE 2: Received history_str from node 1 for node 1 datalen 59823
5517:20120803:125824.797 NODE 2: Received history_text from node 1 for node 1 datalen 2299
5517:20120803:125824.838 NODE 2: Received events from node 1 for node 1 datalen 2353
5525:20120803:125844.321 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15431
5525:20120803:125937.254 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5517:20120803:130010.980 NODE 2: Received history from node 4 for node 4 datalen 314266
5517:20120803:130019.909 NODE 2: Received history_uint from node 4 for node 4 datalen 391544
5517:20120803:130042.969 NODE 2: Received history_str from node 4 for node 4 datalen 126653
5531:20120803:130043.254 NODE 2: Received history_text from node 4 for node 4 datalen 5054
5523:20120803:130151.169 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:130151.676 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:130153.761 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 28538
5513:20120803:130248.010 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5516:20120803:130321.007 NODE 2: Received history from node 3 for node 3 datalen 442493
5516:20120803:130336.302 NODE 2: Received history_uint from node 3 for node 3 datalen 395485
5516:20120803:130353.891 NODE 2: Received history_str from node 3 for node 3 datalen 147806
5521:20120803:130354.374 NODE 2: Received history_text from node 3 for node 3 datalen 49102
5518:20120803:130354.738 NODE 2: Received events from node 3 for node 3 datalen 6092
5515:20120803:130459.021 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:130502.073 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:130515.275 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11699
5514:20120803:130537.155 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5521:20120803:130619.077 NODE 2: Received history from node 1 for node 1 datalen 110691
5539:20120803:130622.289 NODE 2: Received history_uint from node 1 for node 1 datalen 231512
5539:20120803:130638.681 NODE 2: Received history_str from node 1 for node 1 datalen 65096
5539:20120803:130638.846 NODE 2: Received history_text from node 1 for node 1 datalen 3441
5539:20120803:130638.922 NODE 2: Received events from node 1 for node 1 datalen 2242
5531:20120803:130658.734 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13983
5531:20120803:130751.400 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:130825.079 NODE 2: Received history from node 4 for node 4 datalen 315142
5539:20120803:130834.689 NODE 2: Received history_uint from node 4 for node 4 datalen 391855
5539:20120803:130858.460 NODE 2: Received history_str from node 4 for node 4 datalen 125297
5539:20120803:130858.739 NODE 2: Received history_text from node 4 for node 4 datalen 3241
5523:20120803:131007.313 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:131007.801 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:131009.643 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29746
5513:20120803:131107.595 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:131139.089 NODE 2: Received history from node 3 for node 3 datalen 443466
5531:20120803:131154.696 NODE 2: Received history_uint from node 3 for node 3 datalen 395841
5521:20120803:131212.754 NODE 2: Received history_str from node 3 for node 3 datalen 144832
5531:20120803:131213.268 NODE 2: Received history_text from node 3 for node 3 datalen 58197
5531:20120803:131213.720 NODE 2: Received events from node 3 for node 3 datalen 1651
5515:20120803:131316.379 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:131319.397 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:131331.284 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11649
5514:20120803:131351.276 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:131426.041 NODE 2: Received history from node 1 for node 1 datalen 102440
5543:20120803:131428.609 NODE 2: Received history_uint from node 1 for node 1 datalen 243410
5543:20120803:131445.621 NODE 2: Received history_str from node 1 for node 1 datalen 58448
5543:20120803:131445.812 NODE 2: Received history_text from node 1 for node 1 datalen 1959
5543:20120803:131445.874 NODE 2: Received events from node 1 for node 1 datalen 2288
5539:20120803:131505.768 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15382
5539:20120803:131558.001 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:131631.640 NODE 2: Received history from node 4 for node 4 datalen 302567
5543:20120803:131641.075 NODE 2: Received history_uint from node 4 for node 4 datalen 392343
5543:20120803:131704.255 NODE 2: Received history_str from node 4 for node 4 datalen 127963
5543:20120803:131704.670 NODE 2: Received history_text from node 4 for node 4 datalen 3410
5523:20120803:131812.188 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:131812.691 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5519:20120803:131814.414 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 21974
5519:20120803:131906.044 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:131937.540 NODE 2: Received history from node 3 for node 3 datalen 443029
5539:20120803:131954.540 NODE 2: Received history_uint from node 3 for node 3 datalen 396291
5539:20120803:132011.985 NODE 2: Received history_str from node 3 for node 3 datalen 130469
5539:20120803:132012.511 NODE 2: Received history_text from node 3 for node 3 datalen 54410
5518:20120803:132012.896 NODE 2: Received events from node 3 for node 3 datalen 1644
5515:20120803:132115.295 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:132118.310 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:132129.779 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 10376
5514:20120803:132151.404 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5555:20120803:132158.281 executing housekeeper
5518:20120803:132227.471 NODE 2: Received history from node 1 for node 1 datalen 106883
5518:20120803:132231.182 NODE 2: Received history_uint from node 1 for node 1 datalen 242394
5518:20120803:132246.804 NODE 2: Received history_str from node 1 for node 1 datalen 54571
5518:20120803:132246.916 NODE 2: Received history_text from node 1 for node 1 datalen 1228
5518:20120803:132246.960 NODE 2: Received events from node 1 for node 1 datalen 2440
5543:20120803:132307.682 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13805
5543:20120803:132412.605 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5518:20120803:132446.920 NODE 2: Received history from node 4 for node 4 datalen 327367
5518:20120803:132500.985 NODE 2: Received history_uint from node 4 for node 4 datalen 392103
5518:20120803:132526.288 NODE 2: Received history_str from node 4 for node 4 datalen 140107
5528:20120803:132526.788 NODE 2: Received history_text from node 4 for node 4 datalen 4567
5523:20120803:132628.905 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:132629.452 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5519:20120803:132631.857 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 25279
5519:20120803:132754.681 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5517:20120803:132826.354 NODE 2: Received history from node 3 for node 3 datalen 443234
5536:20120803:132850.429 NODE 2: Received history_uint from node 3 for node 3 datalen 395852
5536:20120803:132907.489 NODE 2: Received history_str from node 3 for node 3 datalen 165644
5513:20120803:132908.193 NODE 2: Received history_text from node 3 for node 3 datalen 61028
5513:20120803:132908.871 NODE 2: Received events from node 3 for node 3 datalen 2462
5515:20120803:133005.274 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:133011.850 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:133025.656 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11048
5514:20120803:133056.786 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5515:20120803:133137.062 NODE 2: Received history from node 1 for node 1 datalen 122593
5515:20120803:133141.743 NODE 2: Received history_uint from node 1 for node 1 datalen 291383
5515:20120803:133154.651 NODE 2: Received history_str from node 1 for node 1 datalen 61388
5515:20120803:133154.824 NODE 2: Received history_text from node 1 for node 1 datalen 1458
5515:20120803:133154.882 NODE 2: Received events from node 1 for node 1 datalen 2625
5528:20120803:133217.185 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14896
5528:20120803:133346.158 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:133420.005 NODE 2: Received history from node 4 for node 4 datalen 389765
5543:20120803:133452.653 NODE 2: Received history_uint from node 4 for node 4 datalen 391927
5543:20120803:133519.873 NODE 2: Received history_str from node 4 for node 4 datalen 179584
5543:20120803:133520.756 NODE 2: Received history_text from node 4 for node 4 datalen 12291
5523:20120803:133603.898 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:133605.149 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:133607.539 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 22251
5513:20120803:133736.666 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:133808.484 NODE 2: Received history from node 3 for node 3 datalen 442978
5516:20120803:133820.708 NODE 2: Received history_uint from node 3 for node 3 datalen 396286
5536:20120803:133830.423 NODE 2: Received history_str from node 3 for node 3 datalen 167079
5536:20120803:133830.922 NODE 2: Received history_text from node 3 for node 3 datalen 70425
5536:20120803:133831.266 NODE 2: Received events from node 3 for node 3 datalen 1929
5521:20120803:133946.938 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:133952.324 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:134005.620 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13639
5514:20120803:134036.519 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:134117.460 NODE 2: Received history from node 1 for node 1 datalen 130576
5516:20120803:134126.185 NODE 2: Received history_uint from node 1 for node 1 datalen 310193
5543:20120803:134153.254 NODE 2: Received history_str from node 1 for node 1 datalen 69149
5515:20120803:134153.509 NODE 2: Received history_text from node 1 for node 1 datalen 1445
5518:20120803:134153.660 NODE 2: Received events from node 1 for node 1 datalen 2982
5533:20120803:134157.353 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5533:20120803:134200.982 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5539:20120803:134204.182 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 16738
5539:20120803:134338.229 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5518:20120803:134412.225 NODE 2: Received history from node 4 for node 4 datalen 377923
5531:20120803:134426.483 NODE 2: Received history_uint from node 4 for node 4 datalen 391603
5525:20120803:134440.421 NODE 2: Received history_str from node 4 for node 4 datalen 163888
5516:20120803:134440.852 NODE 2: Received history_text from node 4 for node 4 datalen 7275
5517:20120803:134555.775 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 28750
5517:20120803:134709.140 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5529:20120803:134741.046 NODE 2: Received history from node 3 for node 3 datalen 443517
5539:20120803:134810.405 NODE 2: Received history_uint from node 3 for node 3 datalen 395738
5539:20120803:134832.441 NODE 2: Received history_str from node 3 for node 3 datalen 169223
5525:20120803:134836.096 NODE 2: Received history_text from node 3 for node 3 datalen 55557
5516:20120803:134837.025 NODE 2: Received events from node 3 for node 3 datalen 2493
5521:20120803:134920.547 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:134926.068 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5515:20120803:134940.116 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5515:20120803:134940.654 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5533:20120803:134943.135 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13913
5533:20120803:135013.070 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5555:20120803:135016.554 housekeeper deleted: 452022 records from history and trends, 0 records of deleted items, 56949 events, 0 alerts, 0 sessions
5542:20120803:135055.229 NODE 2: Received history from node 1 for node 1 datalen 140841
5542:20120803:135102.525 NODE 2: Received history_uint from node 1 for node 1 datalen 290446
5521:20120803:135127.308 NODE 2: Received history_str from node 1 for node 1 datalen 64688
5536:20120803:135127.603 NODE 2: Received history_text from node 1 for node 1 datalen 2507
5516:20120803:135127.679 NODE 2: Received events from node 1 for node 1 datalen 3446
5523:20120803:135134.407 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13641
5523:20120803:135244.103 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5514:20120803:135317.721 NODE 2: Received history from node 4 for node 4 datalen 362708
5521:20120803:135340.958 NODE 2: Received history_uint from node 4 for node 4 datalen 391842
5539:20120803:135402.040 NODE 2: Received history_str from node 4 for node 4 datalen 169887
5536:20120803:135403.428 NODE 2: Received history_text from node 4 for node 4 datalen 7458
5529:20120803:135502.684 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 26464
5529:20120803:135556.627 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5542:20120803:135628.245 NODE 2: Received history from node 3 for node 3 datalen 442993
5521:20120803:135647.142 NODE 2: Received history_uint from node 3 for node 3 datalen 397425
5539:20120803:135706.526 NODE 2: Received history_str from node 3 for node 3 datalen 144661
5525:20120803:135710.098 NODE 2: Received history_text from node 3 for node 3 datalen 46116
5525:20120803:135710.472 NODE 2: Received events from node 3 for node 3 datalen 2346
5519:20120803:135806.716 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5519:20120803:135807.204 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5537:20120803:135809.539 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5537:20120803:135812.797 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5533:20120803:135827.218 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 15880
5533:20120803:135851.607 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
There are no errors in the logs.
I'm encountering an issue with the history_uint tables not sync'ing to the master node at the same speed as the other tables, causing the integer values to lag/run behind, all other values are fine. This problem may only occur on very large installations, as the tables are actually sync'ing, just one table is sync'ing too slowly for us.
I suspect that there is a limit in the code (didn't check this) somewhere, because all tables go over 400K bytes every now and then, especially when something big happens, except the history_uint table. I've never seen any sync of that table with a size hitting 400K, it's always something like 395K or so, while that table is running behind several days, and it's getting worse (it's a week at the moment and increasing, but the values ARE updating, just too slowly to keep up). Why isn't the sync sync'ing more data than 395K, when it's running behind that much? I should sync all missing data at once! If you look at the sync speeds, this is actually quite high, it takes about 2 minutes for a host to sync.
Don't pay attention to nodes 5 and 6, they are not monitoring at the moment (temporarily suspended), that's why they are only sync'ing config, but no history tables.
Very large distributed setup, 1 master node, 5 child nodes, all on CentOS 6.2 64-bit, MySQL 5.5:
Number of hosts (monitored/not monitored/templates) 12043 11664 / 116 / 263
Number of items (monitored/disabled/not supported) 688965 647656 / 2571 / 38738
Number of triggers (enabled/disabled)[problem/unknown/ok] 194100 159297 / 34803 [709 / 0 / 158588]
Number of users (online) 197 5
Required server performance, new values per second 225.83 -
Node 2 is the MASTER.
5531:20120803:111521.326 NODE 2: Received history from node 3 for node 3 datalen 443005
5531:20120803:111536.455 NODE 2: Received history_uint from node 3 for node 3 datalen 396645
5531:20120803:111555.261 NODE 2: Received history_str from node 3 for node 3 datalen 132367
5531:20120803:111555.755 NODE 2: Received history_text from node 3 for node 3 datalen 19195
5531:20120803:111556.048 NODE 2: Received events from node 3 for node 3 datalen 2194
5521:20120803:111658.997 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:111702.549 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5517:20120803:111715.534 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11350
5517:20120803:111734.200 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5531:20120803:111813.913 NODE 2: Received history from node 1 for node 1 datalen 100370
5531:20120803:111816.536 NODE 2: Received history_uint from node 1 for node 1 datalen 214517
5531:20120803:111831.605 NODE 2: Received history_str from node 1 for node 1 datalen 48827
5531:20120803:111831.774 NODE 2: Received history_text from node 1 for node 1 datalen 843
5531:20120803:111831.820 NODE 2: Received events from node 1 for node 1 datalen 2518
5518:20120803:111855.844 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13684
5518:20120803:111947.278 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:112023.346 NODE 2: Received history from node 4 for node 4 datalen 323253
5542:20120803:112032.608 NODE 2: Received history_uint from node 4 for node 4 datalen 392062
5529:20120803:112055.617 NODE 2: Received history_str from node 4 for node 4 datalen 130125
5514:20120803:112055.930 NODE 2: Received history_text from node 4 for node 4 datalen 7032
5536:20120803:112206.922 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:112207.403 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5520:20120803:112209.701 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 183258
5520:20120803:112303.249 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:112335.860 NODE 2: Received history from node 3 for node 3 datalen 443058
5542:20120803:112350.530 NODE 2: Received history_uint from node 3 for node 3 datalen 396611
5517:20120803:112408.939 NODE 2: Received history_str from node 3 for node 3 datalen 130895
5517:20120803:112409.265 NODE 2: Received history_text from node 3 for node 3 datalen 18562
5529:20120803:112409.584 NODE 2: Received events from node 3 for node 3 datalen 2579
5521:20120803:112511.723 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:112514.787 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:112528.235 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12582
5519:20120803:112550.751 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5525:20120803:112629.988 NODE 2: Received history from node 1 for node 1 datalen 112019
5537:20120803:112633.314 NODE 2: Received history_uint from node 1 for node 1 datalen 231831
5537:20120803:112650.023 NODE 2: Received history_str from node 1 for node 1 datalen 70526
5537:20120803:112650.216 NODE 2: Received history_text from node 1 for node 1 datalen 1183
5537:20120803:112650.284 NODE 2: Received events from node 1 for node 1 datalen 2352
5514:20120803:112709.772 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13514
5514:20120803:112804.221 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5525:20120803:112839.229 NODE 2: Received history from node 4 for node 4 datalen 326743
5525:20120803:112848.894 NODE 2: Received history_uint from node 4 for node 4 datalen 391996
5525:20120803:112911.255 NODE 2: Received history_str from node 4 for node 4 datalen 121776
5528:20120803:112911.579 NODE 2: Received history_text from node 4 for node 4 datalen 5349
5536:20120803:113019.849 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:113020.329 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:113022.114 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 25711
5543:20120803:113118.650 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5513:20120803:113151.167 NODE 2: Received history from node 3 for node 3 datalen 443548
5513:20120803:113208.302 NODE 2: Received history_uint from node 3 for node 3 datalen 395505
5513:20120803:113225.817 NODE 2: Received history_str from node 3 for node 3 datalen 128866
5513:20120803:113226.311 NODE 2: Received history_text from node 3 for node 3 datalen 26341
5513:20120803:113226.678 NODE 2: Received events from node 3 for node 3 datalen 2802
5521:20120803:113328.522 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:113331.553 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:113345.389 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 10252
5519:20120803:113408.591 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5513:20120803:113449.012 NODE 2: Received history from node 1 for node 1 datalen 112909
5513:20120803:113452.530 NODE 2: Received history_uint from node 1 for node 1 datalen 262472
5513:20120803:113511.365 NODE 2: Received history_str from node 1 for node 1 datalen 68640
5515:20120803:113511.615 NODE 2: Received history_text from node 1 for node 1 datalen 1001
5539:20120803:113511.678 NODE 2: Received events from node 1 for node 1 datalen 3398
5528:20120803:113528.370 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14449
5528:20120803:113622.746 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5525:20120803:113657.802 NODE 2: Received history from node 4 for node 4 datalen 323329
5537:20120803:113707.471 NODE 2: Received history_uint from node 4 for node 4 datalen 391869
5525:20120803:113730.552 NODE 2: Received history_str from node 4 for node 4 datalen 125779
5520:20120803:113730.846 NODE 2: Received history_text from node 4 for node 4 datalen 5035
5536:20120803:113838.352 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:113838.847 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:113841.612 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 26063
5543:20120803:113940.059 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:114012.332 NODE 2: Received history from node 3 for node 3 datalen 443429
5531:20120803:114028.629 NODE 2: Received history_uint from node 3 for node 3 datalen 397121
5523:20120803:114046.755 NODE 2: Received history_str from node 3 for node 3 datalen 141948
5523:20120803:114047.182 NODE 2: Received history_text from node 3 for node 3 datalen 48697
5523:20120803:114047.547 NODE 2: Received events from node 3 for node 3 datalen 3900
5521:20120803:114149.438 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:114152.485 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:114204.783 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 14756
5519:20120803:114229.600 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5542:20120803:114308.694 NODE 2: Received history from node 1 for node 1 datalen 109585
5542:20120803:114312.525 NODE 2: Received history_uint from node 1 for node 1 datalen 243135
5542:20120803:114329.426 NODE 2: Received history_str from node 1 for node 1 datalen 47932
5542:20120803:114329.652 NODE 2: Received history_text from node 1 for node 1 datalen 846
5542:20120803:114329.695 NODE 2: Received events from node 1 for node 1 datalen 3296
5539:20120803:114348.166 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15523
5539:20120803:114440.661 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5542:20120803:114515.411 NODE 2: Received history from node 4 for node 4 datalen 312201
5542:20120803:114525.828 NODE 2: Received history_uint from node 4 for node 4 datalen 392273
5542:20120803:114547.734 NODE 2: Received history_str from node 4 for node 4 datalen 123651
5542:20120803:114548.044 NODE 2: Received history_text from node 4 for node 4 datalen 4743
5536:20120803:114658.078 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:114658.559 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:114700.239 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29921
5543:20120803:114759.609 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:114833.786 NODE 2: Received history from node 3 for node 3 datalen 443004
5523:20120803:114849.640 NODE 2: Received history_uint from node 3 for node 3 datalen 396099
5523:20120803:114907.383 NODE 2: Received history_str from node 3 for node 3 datalen 134588
5518:20120803:114907.801 NODE 2: Received history_text from node 3 for node 3 datalen 56545
5516:20120803:114908.204 NODE 2: Received events from node 3 for node 3 datalen 3715
5520:20120803:115011.428 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5520:20120803:115015.084 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:115027.386 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12627
5519:20120803:115049.961 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5515:20120803:115130.610 NODE 2: Received history from node 1 for node 1 datalen 121779
5515:20120803:115134.618 NODE 2: Received history_uint from node 1 for node 1 datalen 224083
5525:20120803:115152.669 NODE 2: Received history_str from node 1 for node 1 datalen 47794
5539:20120803:115152.798 NODE 2: Received history_text from node 1 for node 1 datalen 1004
5539:20120803:115152.843 NODE 2: Received events from node 1 for node 1 datalen 2792
5521:20120803:115209.677 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14310
5521:20120803:115301.825 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:115334.683 NODE 2: Received history from node 4 for node 4 datalen 320118
5525:20120803:115344.476 NODE 2: Received history_uint from node 4 for node 4 datalen 391732
5513:20120803:115406.588 NODE 2: Received history_str from node 4 for node 4 datalen 132676
5515:20120803:115406.896 NODE 2: Received history_text from node 4 for node 4 datalen 5026
5536:20120803:115515.958 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5536:20120803:115516.441 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5543:20120803:115519.261 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 53369
5543:20120803:115613.294 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5523:20120803:115648.665 NODE 2: Received history from node 3 for node 3 datalen 442680
5523:20120803:115706.704 NODE 2: Received history_uint from node 3 for node 3 datalen 397391
5514:20120803:115731.302 NODE 2: Received history_str from node 3 for node 3 datalen 163601
5528:20120803:115731.835 NODE 2: Received history_text from node 3 for node 3 datalen 43015
5525:20120803:115732.169 NODE 2: Received events from node 3 for node 3 datalen 6488
5520:20120803:115828.939 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5520:20120803:115832.013 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:115846.696 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11397
5519:20120803:115908.260 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5542:20120803:115946.914 NODE 2: Received history from node 1 for node 1 datalen 104991
5529:20120803:115949.804 NODE 2: Received history_uint from node 1 for node 1 datalen 222859
5529:20120803:120005.399 NODE 2: Received history_str from node 1 for node 1 datalen 39662
5529:20120803:120005.503 NODE 2: Received history_text from node 1 for node 1 datalen 1506
5529:20120803:120005.561 NODE 2: Received events from node 1 for node 1 datalen 2618
5515:20120803:120025.920 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14525
5515:20120803:120135.351 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5536:20120803:120212.637 NODE 2: Received history from node 4 for node 4 datalen 330907
5543:20120803:120222.291 NODE 2: Received history_uint from node 4 for node 4 datalen 391685
5543:20120803:120244.512 NODE 2: Received history_str from node 4 for node 4 datalen 145511
5513:20120803:120244.877 NODE 2: Received history_text from node 4 for node 4 datalen 14526
5539:20120803:120353.335 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:120353.823 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5520:20120803:120355.960 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 39383
5520:20120803:120450.879 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:120526.796 NODE 2: Received history from node 3 for node 3 datalen 442859
5515:20120803:120543.733 NODE 2: Received history_uint from node 3 for node 3 datalen 396221
5515:20120803:120600.855 NODE 2: Received history_str from node 3 for node 3 datalen 158744
5515:20120803:120601.330 NODE 2: Received history_text from node 3 for node 3 datalen 45534
5515:20120803:120601.694 NODE 2: Received events from node 3 for node 3 datalen 2964
5521:20120803:120700.830 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:120703.845 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:120716.982 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11532
5519:20120803:120735.828 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:120814.520 NODE 2: Received history from node 1 for node 1 datalen 108863
5543:20120803:120817.361 NODE 2: Received history_uint from node 1 for node 1 datalen 225009
5543:20120803:120834.586 NODE 2: Received history_str from node 1 for node 1 datalen 51035
5543:20120803:120834.701 NODE 2: Received history_text from node 1 for node 1 datalen 2500
5543:20120803:120834.775 NODE 2: Received events from node 1 for node 1 datalen 3466
5513:20120803:120853.872 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15127
5513:20120803:120945.765 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5520:20120803:121021.364 NODE 2: Received history from node 4 for node 4 datalen 308145
5533:20120803:121030.371 NODE 2: Received history_uint from node 4 for node 4 datalen 391715
5533:20120803:121052.367 NODE 2: Received history_str from node 4 for node 4 datalen 143454
5529:20120803:121052.739 NODE 2: Received history_text from node 4 for node 4 datalen 17819
5539:20120803:121203.734 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:121204.218 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:121206.334 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 284636
5521:20120803:121311.227 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5520:20120803:121345.794 NODE 2: Received history from node 3 for node 3 datalen 443110
5520:20120803:121402.412 NODE 2: Received history_uint from node 3 for node 3 datalen 396096
5520:20120803:121419.478 NODE 2: Received history_str from node 3 for node 3 datalen 144233
5520:20120803:121419.979 NODE 2: Received history_text from node 3 for node 3 datalen 49993
5520:20120803:121420.378 NODE 2: Received events from node 3 for node 3 datalen 3452
5515:20120803:121523.697 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:121526.703 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:121539.898 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13715
5519:20120803:121558.949 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5514:20120803:121643.481 NODE 2: Received history from node 1 for node 1 datalen 105929
5514:20120803:121646.124 NODE 2: Received history_uint from node 1 for node 1 datalen 231451
5514:20120803:121702.902 NODE 2: Received history_str from node 1 for node 1 datalen 50292
5514:20120803:121703.014 NODE 2: Received history_text from node 1 for node 1 datalen 2016
5514:20120803:121703.074 NODE 2: Received events from node 1 for node 1 datalen 2741
5529:20120803:121722.926 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 12259
5529:20120803:121818.856 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5514:20120803:121855.227 NODE 2: Received history from node 4 for node 4 datalen 332895
5514:20120803:121905.265 NODE 2: Received history_uint from node 4 for node 4 datalen 391387
5523:20120803:121927.572 NODE 2: Received history_str from node 4 for node 4 datalen 149823
5523:20120803:121928.600 NODE 2: Received history_text from node 4 for node 4 datalen 11034
5539:20120803:122036.692 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:122037.173 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:122039.520 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29947
5521:20120803:122131.397 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5514:20120803:122204.369 NODE 2: Received history from node 3 for node 3 datalen 443178
5514:20120803:122220.576 NODE 2: Received history_uint from node 3 for node 3 datalen 395831
5514:20120803:122238.234 NODE 2: Received history_str from node 3 for node 3 datalen 134196
5514:20120803:122238.706 NODE 2: Received history_text from node 3 for node 3 datalen 49058
5514:20120803:122239.118 NODE 2: Received events from node 3 for node 3 datalen 3192
5514:20120803:122239.153 NODE 2: Received acknowledges from node 3 for node 3 datalen 124
5515:20120803:122341.701 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:122344.727 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:122357.509 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12989
5519:20120803:122418.858 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5514:20120803:122456.832 NODE 2: Received history from node 1 for node 1 datalen 112497
5514:20120803:122500.100 NODE 2: Received history_uint from node 1 for node 1 datalen 237948
5514:20120803:122517.399 NODE 2: Received history_str from node 1 for node 1 datalen 57979
5514:20120803:122517.575 NODE 2: Received history_text from node 1 for node 1 datalen 1671
5514:20120803:122517.631 NODE 2: Received events from node 1 for node 1 datalen 2952
5523:20120803:122536.214 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15025
5523:20120803:122631.633 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5529:20120803:122705.775 NODE 2: Received history from node 4 for node 4 datalen 316808
5529:20120803:122715.688 NODE 2: Received history_uint from node 4 for node 4 datalen 392075
5529:20120803:122738.970 NODE 2: Received history_str from node 4 for node 4 datalen 143217
5529:20120803:122739.290 NODE 2: Received history_text from node 4 for node 4 datalen 19563
5539:20120803:122846.841 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5539:20120803:122847.322 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:122849.474 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 27114
5521:20120803:122948.157 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:123019.900 NODE 2: Received history from node 3 for node 3 datalen 443132
5539:20120803:123035.123 NODE 2: Received history_uint from node 3 for node 3 datalen 395533
5539:20120803:123054.304 NODE 2: Received history_str from node 3 for node 3 datalen 150855
5539:20120803:123054.744 NODE 2: Received history_text from node 3 for node 3 datalen 48957
5539:20120803:123055.166 NODE 2: Received events from node 3 for node 3 datalen 3544
5515:20120803:123156.956 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:123159.972 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:123212.888 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12619
5519:20120803:123231.778 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5539:20120803:123313.576 NODE 2: Received history from node 1 for node 1 datalen 106490
5539:20120803:123316.712 NODE 2: Received history_uint from node 1 for node 1 datalen 242031
5539:20120803:123333.989 NODE 2: Received history_str from node 1 for node 1 datalen 56525
5539:20120803:123334.191 NODE 2: Received history_text from node 1 for node 1 datalen 2053
5539:20120803:123334.243 NODE 2: Received events from node 1 for node 1 datalen 2461
5529:20120803:123352.927 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15372
5529:20120803:123443.082 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:123518.408 NODE 2: Received history from node 4 for node 4 datalen 325252
5531:20120803:123530.089 NODE 2: Received history_uint from node 4 for node 4 datalen 392019
5539:20120803:123552.392 NODE 2: Received history_str from node 4 for node 4 datalen 149438
5539:20120803:123552.813 NODE 2: Received history_text from node 4 for node 4 datalen 4111
5523:20120803:123659.389 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:123659.870 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5521:20120803:123701.725 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 30557
5521:20120803:123802.975 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:123835.138 NODE 2: Received history from node 3 for node 3 datalen 443071
5537:20120803:123851.859 NODE 2: Received history_uint from node 3 for node 3 datalen 396377
5537:20120803:123908.993 NODE 2: Received history_str from node 3 for node 3 datalen 151620
5537:20120803:123909.518 NODE 2: Received history_text from node 3 for node 3 datalen 72626
5537:20120803:123909.878 NODE 2: Received events from node 3 for node 3 datalen 3187
5515:20120803:124011.658 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:124014.672 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5519:20120803:124028.830 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 9432
5519:20120803:124047.102 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5537:20120803:124128.143 NODE 2: Received history from node 1 for node 1 datalen 119410
5528:20120803:124132.915 NODE 2: Received history_uint from node 1 for node 1 datalen 278229
5528:20120803:124151.528 NODE 2: Received history_str from node 1 for node 1 datalen 57823
5514:20120803:124151.763 NODE 2: Received history_text from node 1 for node 1 datalen 1633
5514:20120803:124151.817 NODE 2: Received events from node 1 for node 1 datalen 2614
5529:20120803:124207.983 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14696
5529:20120803:124257.894 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:124331.787 NODE 2: Received history from node 4 for node 4 datalen 309336
5539:20120803:124341.113 NODE 2: Received history_uint from node 4 for node 4 datalen 392183
5525:20120803:124404.480 NODE 2: Received history_str from node 4 for node 4 datalen 141133
5525:20120803:124404.804 NODE 2: Received history_text from node 4 for node 4 datalen 18420
5523:20120803:124513.247 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:124513.733 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:124515.798 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 21855
5513:20120803:124620.941 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5521:20120803:124652.937 NODE 2: Received history from node 3 for node 3 datalen 443105
5521:20120803:124709.937 NODE 2: Received history_uint from node 3 for node 3 datalen 396404
5521:20120803:124726.887 NODE 2: Received history_str from node 3 for node 3 datalen 149411
5521:20120803:124727.325 NODE 2: Received history_text from node 3 for node 3 datalen 65594
5521:20120803:124727.669 NODE 2: Received events from node 3 for node 3 datalen 3542
5515:20120803:124830.452 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:124833.560 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:124847.107 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 12272
5514:20120803:124906.130 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5525:20120803:124946.451 NODE 2: Received history from node 1 for node 1 datalen 122938
5525:20120803:124950.116 NODE 2: Received history_uint from node 1 for node 1 datalen 246729
5517:20120803:125007.424 NODE 2: Received history_str from node 1 for node 1 datalen 62666
5517:20120803:125008.051 NODE 2: Received history_text from node 1 for node 1 datalen 2738
5517:20120803:125008.195 NODE 2: Received events from node 1 for node 1 datalen 2100
5531:20120803:125025.614 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13283
5531:20120803:125123.257 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5517:20120803:125156.420 NODE 2: Received history from node 4 for node 4 datalen 327200
5517:20120803:125207.434 NODE 2: Received history_uint from node 4 for node 4 datalen 392154
5525:20120803:125230.151 NODE 2: Received history_str from node 4 for node 4 datalen 142021
5525:20120803:125230.498 NODE 2: Received history_text from node 4 for node 4 datalen 4915
5523:20120803:125337.349 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:125337.843 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:125339.851 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29935
5513:20120803:125437.929 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:125510.027 NODE 2: Received history from node 3 for node 3 datalen 443199
5531:20120803:125526.269 NODE 2: Received history_uint from node 3 for node 3 datalen 396729
5533:20120803:125543.972 NODE 2: Received history_str from node 3 for node 3 datalen 144821
5533:20120803:125544.498 NODE 2: Received history_text from node 3 for node 3 datalen 61374
5533:20120803:125544.853 NODE 2: Received events from node 3 for node 3 datalen 1959
5515:20120803:125647.357 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:125650.399 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:125704.235 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 9425
5514:20120803:125723.595 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5531:20120803:125804.967 NODE 2: Received history from node 1 for node 1 datalen 107392
5531:20120803:125808.070 NODE 2: Received history_uint from node 1 for node 1 datalen 247474
5531:20120803:125824.563 NODE 2: Received history_str from node 1 for node 1 datalen 59823
5517:20120803:125824.797 NODE 2: Received history_text from node 1 for node 1 datalen 2299
5517:20120803:125824.838 NODE 2: Received events from node 1 for node 1 datalen 2353
5525:20120803:125844.321 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15431
5525:20120803:125937.254 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5517:20120803:130010.980 NODE 2: Received history from node 4 for node 4 datalen 314266
5517:20120803:130019.909 NODE 2: Received history_uint from node 4 for node 4 datalen 391544
5517:20120803:130042.969 NODE 2: Received history_str from node 4 for node 4 datalen 126653
5531:20120803:130043.254 NODE 2: Received history_text from node 4 for node 4 datalen 5054
5523:20120803:130151.169 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:130151.676 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:130153.761 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 28538
5513:20120803:130248.010 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5516:20120803:130321.007 NODE 2: Received history from node 3 for node 3 datalen 442493
5516:20120803:130336.302 NODE 2: Received history_uint from node 3 for node 3 datalen 395485
5516:20120803:130353.891 NODE 2: Received history_str from node 3 for node 3 datalen 147806
5521:20120803:130354.374 NODE 2: Received history_text from node 3 for node 3 datalen 49102
5518:20120803:130354.738 NODE 2: Received events from node 3 for node 3 datalen 6092
5515:20120803:130459.021 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:130502.073 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:130515.275 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11699
5514:20120803:130537.155 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5521:20120803:130619.077 NODE 2: Received history from node 1 for node 1 datalen 110691
5539:20120803:130622.289 NODE 2: Received history_uint from node 1 for node 1 datalen 231512
5539:20120803:130638.681 NODE 2: Received history_str from node 1 for node 1 datalen 65096
5539:20120803:130638.846 NODE 2: Received history_text from node 1 for node 1 datalen 3441
5539:20120803:130638.922 NODE 2: Received events from node 1 for node 1 datalen 2242
5531:20120803:130658.734 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13983
5531:20120803:130751.400 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5539:20120803:130825.079 NODE 2: Received history from node 4 for node 4 datalen 315142
5539:20120803:130834.689 NODE 2: Received history_uint from node 4 for node 4 datalen 391855
5539:20120803:130858.460 NODE 2: Received history_str from node 4 for node 4 datalen 125297
5539:20120803:130858.739 NODE 2: Received history_text from node 4 for node 4 datalen 3241
5523:20120803:131007.313 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:131007.801 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:131009.643 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 29746
5513:20120803:131107.595 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5531:20120803:131139.089 NODE 2: Received history from node 3 for node 3 datalen 443466
5531:20120803:131154.696 NODE 2: Received history_uint from node 3 for node 3 datalen 395841
5521:20120803:131212.754 NODE 2: Received history_str from node 3 for node 3 datalen 144832
5531:20120803:131213.268 NODE 2: Received history_text from node 3 for node 3 datalen 58197
5531:20120803:131213.720 NODE 2: Received events from node 3 for node 3 datalen 1651
5515:20120803:131316.379 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:131319.397 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:131331.284 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11649
5514:20120803:131351.276 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:131426.041 NODE 2: Received history from node 1 for node 1 datalen 102440
5543:20120803:131428.609 NODE 2: Received history_uint from node 1 for node 1 datalen 243410
5543:20120803:131445.621 NODE 2: Received history_str from node 1 for node 1 datalen 58448
5543:20120803:131445.812 NODE 2: Received history_text from node 1 for node 1 datalen 1959
5543:20120803:131445.874 NODE 2: Received events from node 1 for node 1 datalen 2288
5539:20120803:131505.768 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 15382
5539:20120803:131558.001 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:131631.640 NODE 2: Received history from node 4 for node 4 datalen 302567
5543:20120803:131641.075 NODE 2: Received history_uint from node 4 for node 4 datalen 392343
5543:20120803:131704.255 NODE 2: Received history_str from node 4 for node 4 datalen 127963
5543:20120803:131704.670 NODE 2: Received history_text from node 4 for node 4 datalen 3410
5523:20120803:131812.188 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:131812.691 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5519:20120803:131814.414 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 21974
5519:20120803:131906.044 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5539:20120803:131937.540 NODE 2: Received history from node 3 for node 3 datalen 443029
5539:20120803:131954.540 NODE 2: Received history_uint from node 3 for node 3 datalen 396291
5539:20120803:132011.985 NODE 2: Received history_str from node 3 for node 3 datalen 130469
5539:20120803:132012.511 NODE 2: Received history_text from node 3 for node 3 datalen 54410
5518:20120803:132012.896 NODE 2: Received events from node 3 for node 3 datalen 1644
5515:20120803:132115.295 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:132118.310 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:132129.779 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 10376
5514:20120803:132151.404 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5555:20120803:132158.281 executing housekeeper
5518:20120803:132227.471 NODE 2: Received history from node 1 for node 1 datalen 106883
5518:20120803:132231.182 NODE 2: Received history_uint from node 1 for node 1 datalen 242394
5518:20120803:132246.804 NODE 2: Received history_str from node 1 for node 1 datalen 54571
5518:20120803:132246.916 NODE 2: Received history_text from node 1 for node 1 datalen 1228
5518:20120803:132246.960 NODE 2: Received events from node 1 for node 1 datalen 2440
5543:20120803:132307.682 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13805
5543:20120803:132412.605 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5518:20120803:132446.920 NODE 2: Received history from node 4 for node 4 datalen 327367
5518:20120803:132500.985 NODE 2: Received history_uint from node 4 for node 4 datalen 392103
5518:20120803:132526.288 NODE 2: Received history_str from node 4 for node 4 datalen 140107
5528:20120803:132526.788 NODE 2: Received history_text from node 4 for node 4 datalen 4567
5523:20120803:132628.905 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:132629.452 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5519:20120803:132631.857 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 25279
5519:20120803:132754.681 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5517:20120803:132826.354 NODE 2: Received history from node 3 for node 3 datalen 443234
5536:20120803:132850.429 NODE 2: Received history_uint from node 3 for node 3 datalen 395852
5536:20120803:132907.489 NODE 2: Received history_str from node 3 for node 3 datalen 165644
5513:20120803:132908.193 NODE 2: Received history_text from node 3 for node 3 datalen 61028
5513:20120803:132908.871 NODE 2: Received events from node 3 for node 3 datalen 2462
5515:20120803:133005.274 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5515:20120803:133011.850 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:133025.656 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 11048
5514:20120803:133056.786 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5515:20120803:133137.062 NODE 2: Received history from node 1 for node 1 datalen 122593
5515:20120803:133141.743 NODE 2: Received history_uint from node 1 for node 1 datalen 291383
5515:20120803:133154.651 NODE 2: Received history_str from node 1 for node 1 datalen 61388
5515:20120803:133154.824 NODE 2: Received history_text from node 1 for node 1 datalen 1458
5515:20120803:133154.882 NODE 2: Received events from node 1 for node 1 datalen 2625
5528:20120803:133217.185 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 14896
5528:20120803:133346.158 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5543:20120803:133420.005 NODE 2: Received history from node 4 for node 4 datalen 389765
5543:20120803:133452.653 NODE 2: Received history_uint from node 4 for node 4 datalen 391927
5543:20120803:133519.873 NODE 2: Received history_str from node 4 for node 4 datalen 179584
5543:20120803:133520.756 NODE 2: Received history_text from node 4 for node 4 datalen 12291
5523:20120803:133603.898 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5523:20120803:133605.149 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5513:20120803:133607.539 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 22251
5513:20120803:133736.666 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5515:20120803:133808.484 NODE 2: Received history from node 3 for node 3 datalen 442978
5516:20120803:133820.708 NODE 2: Received history_uint from node 3 for node 3 datalen 396286
5536:20120803:133830.423 NODE 2: Received history_str from node 3 for node 3 datalen 167079
5536:20120803:133830.922 NODE 2: Received history_text from node 3 for node 3 datalen 70425
5536:20120803:133831.266 NODE 2: Received events from node 3 for node 3 datalen 1929
5521:20120803:133946.938 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:133952.324 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5514:20120803:134005.620 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13639
5514:20120803:134036.519 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5543:20120803:134117.460 NODE 2: Received history from node 1 for node 1 datalen 130576
5516:20120803:134126.185 NODE 2: Received history_uint from node 1 for node 1 datalen 310193
5543:20120803:134153.254 NODE 2: Received history_str from node 1 for node 1 datalen 69149
5515:20120803:134153.509 NODE 2: Received history_text from node 1 for node 1 datalen 1445
5518:20120803:134153.660 NODE 2: Received events from node 1 for node 1 datalen 2982
5533:20120803:134157.353 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5533:20120803:134200.982 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5539:20120803:134204.182 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 16738
5539:20120803:134338.229 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5518:20120803:134412.225 NODE 2: Received history from node 4 for node 4 datalen 377923
5531:20120803:134426.483 NODE 2: Received history_uint from node 4 for node 4 datalen 391603
5525:20120803:134440.421 NODE 2: Received history_str from node 4 for node 4 datalen 163888
5516:20120803:134440.852 NODE 2: Received history_text from node 4 for node 4 datalen 7275
5517:20120803:134555.775 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 28750
5517:20120803:134709.140 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5529:20120803:134741.046 NODE 2: Received history from node 3 for node 3 datalen 443517
5539:20120803:134810.405 NODE 2: Received history_uint from node 3 for node 3 datalen 395738
5539:20120803:134832.441 NODE 2: Received history_str from node 3 for node 3 datalen 169223
5525:20120803:134836.096 NODE 2: Received history_text from node 3 for node 3 datalen 55557
5516:20120803:134837.025 NODE 2: Received events from node 3 for node 3 datalen 2493
5521:20120803:134920.547 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5521:20120803:134926.068 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5515:20120803:134940.116 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5515:20120803:134940.654 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5533:20120803:134943.135 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 13913
5533:20120803:135013.070 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
5555:20120803:135016.554 housekeeper deleted: 452022 records from history and trends, 0 records of deleted items, 56949 events, 0 alerts, 0 sessions
5542:20120803:135055.229 NODE 2: Received history from node 1 for node 1 datalen 140841
5542:20120803:135102.525 NODE 2: Received history_uint from node 1 for node 1 datalen 290446
5521:20120803:135127.308 NODE 2: Received history_str from node 1 for node 1 datalen 64688
5536:20120803:135127.603 NODE 2: Received history_text from node 1 for node 1 datalen 2507
5516:20120803:135127.679 NODE 2: Received events from node 1 for node 1 datalen 3446
5523:20120803:135134.407 NODE 2: Received configuration changes from slave node 4 for node 4 datalen 13641
5523:20120803:135244.103 NODE 2: Sending configuration changes to slave node 4 for node 4 datalen 8
5514:20120803:135317.721 NODE 2: Received history from node 4 for node 4 datalen 362708
5521:20120803:135340.958 NODE 2: Received history_uint from node 4 for node 4 datalen 391842
5539:20120803:135402.040 NODE 2: Received history_str from node 4 for node 4 datalen 169887
5536:20120803:135403.428 NODE 2: Received history_text from node 4 for node 4 datalen 7458
5529:20120803:135502.684 NODE 2: Received configuration changes from slave node 3 for node 3 datalen 26464
5529:20120803:135556.627 NODE 2: Sending configuration changes to slave node 3 for node 3 datalen 8
5542:20120803:135628.245 NODE 2: Received history from node 3 for node 3 datalen 442993
5521:20120803:135647.142 NODE 2: Received history_uint from node 3 for node 3 datalen 397425
5539:20120803:135706.526 NODE 2: Received history_str from node 3 for node 3 datalen 144661
5525:20120803:135710.098 NODE 2: Received history_text from node 3 for node 3 datalen 46116
5525:20120803:135710.472 NODE 2: Received events from node 3 for node 3 datalen 2346
5519:20120803:135806.716 NODE 2: Received configuration changes from slave node 5 for node 5 datalen 8
5519:20120803:135807.204 NODE 2: Sending configuration changes to slave node 5 for node 5 datalen 8
5537:20120803:135809.539 NODE 2: Received configuration changes from slave node 6 for node 6 datalen 8
5537:20120803:135812.797 NODE 2: Sending configuration changes to slave node 6 for node 6 datalen 8
5533:20120803:135827.218 NODE 2: Received configuration changes from slave node 1 for node 1 datalen 15880
5533:20120803:135851.607 NODE 2: Sending configuration changes to slave node 1 for node 1 datalen 8
There are no errors in the logs.
Attached Files