Ad Widget

Collapse

History syncer processes obviously blocked - but by what?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • BDiE8VNy
    Senior Member
    • Apr 2010
    • 680

    #1

    History syncer processes obviously blocked - but by what?

    This post is mainly for documentation purposes and well, to maybe gain some interesting information from potential fellow sufferers

    I've no evidence of what causes that issue. All I know so far is, if it happens, then it happens mostly at night around 2-6 o'clock am.
    For now I could avoid false-positive alerts caused through time based trigger functions by introducing an appropriate trigger which monitors the Zabbix server health and making related triggers dependent from this one. While mitigating the issue in terms of falsely triggered action operations it still means flying almost blind for quite some time.

    Click image for larger version

Name:	ZbxSrvInternalProcessBusyZbxSrvPerformance.jpg
Views:	1
Size:	71.7 KB
ID:	317371
    Click image for larger version

Name:	ZbxSrvCacheUsageZbxSrvValueCacheEffectiveness.jpg
Views:	1
Size:	58.3 KB
ID:	317370
    Click image for larger version

Name:	ZbxDbTuplesZbxDbTupleInserts.jpg
Views:	1
Size:	49.4 KB
ID:	317372
    Click image for larger version

Name:	ZbxDbLocksZbxDbXact.jpg
Views:	1
Size:	65.7 KB
ID:	317373
    Click image for larger version

Name:	ZbxSrvCpuUtilizationZbxDbCpuUtilizationZbxSrvValueCacheMisses.jpg
Views:	1
Size:	46.4 KB
ID:	317374

    To me it looks pretty much like ZBX-9201. But I can't identify significant additional load to log items, what I would expect in such a case.

    When looking at the total values per hour of all items, then there are even less values synchronized in the questioned period of time:
    Code:
    zabbix=# SELECT date_trunc('hour', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447545600
    zabbix-# GROUP  BY 1
    zabbix-# ORDER  BY 1 ASC;
           date_trunc       |  count
    ------------------------+---------
     2015-11-13 01:00:00+01 | 1708755
     2015-11-13 02:00:00+01 | 1709124
     2015-11-13 03:00:00+01 | 1709314
     2015-11-13 04:00:00+01 | 1709604
     2015-11-13 05:00:00+01 | 1708992
     2015-11-13 06:00:00+01 | 1709129
     2015-11-13 07:00:00+01 | 1709716
     2015-11-13 08:00:00+01 | 1708976
     2015-11-13 09:00:00+01 | 1708617
     2015-11-13 10:00:00+01 | 1708947
     2015-11-13 11:00:00+01 | 1709269
     2015-11-13 12:00:00+01 | 1709407
     2015-11-13 13:00:00+01 | 1708857
     2015-11-13 14:00:00+01 | 1708889
     2015-11-13 15:00:00+01 | 1710206
     2015-11-13 16:00:00+01 | 1709694
     2015-11-13 17:00:00+01 | 1709572
     2015-11-13 18:00:00+01 | 1711227
     2015-11-13 19:00:00+01 | 1710895
     2015-11-13 20:00:00+01 | 1711000
     2015-11-13 21:00:00+01 | 1710976
     2015-11-13 22:00:00+01 | 1710950
     2015-11-13 23:00:00+01 | 1711578
     2015-11-14 00:00:00+01 | 1711366
     2015-11-14 01:00:00+01 | 1710721
     2015-11-14 02:00:00+01 | 1710930
     2015-11-14 03:00:00+01 | 1710764
     2015-11-14 04:00:00+01 | 1711406
     2015-11-14 05:00:00+01 | 1708589
     2015-11-14 06:00:00+01 | 1708214
     2015-11-14 07:00:00+01 | 1707961
     2015-11-14 08:00:00+01 | 1707741
     2015-11-14 09:00:00+01 | 1707695
     2015-11-14 10:00:00+01 | 1707991
     2015-11-14 11:00:00+01 | 1707748
     2015-11-14 12:00:00+01 | 1707859
     2015-11-14 13:00:00+01 | 1707875
     2015-11-14 14:00:00+01 | 1707700
     2015-11-14 15:00:00+01 | 1707796
     2015-11-14 16:00:00+01 | 1707386
     2015-11-14 17:00:00+01 | 1707620
     2015-11-14 18:00:00+01 | 1707725
     2015-11-14 19:00:00+01 | 1707979
     2015-11-14 20:00:00+01 | 1707624
     2015-11-14 21:00:00+01 | 1707738
     2015-11-14 22:00:00+01 | 1707969
     2015-11-14 23:00:00+01 | 1707476
     2015-11-15 00:00:00+01 | 1708055
    (48 rows)
    
    zabbix=# SELECT date_trunc('hour', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_uint
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447545600
    zabbix-# GROUP  BY 1
    zabbix-# ORDER  BY 1 ASC;
           date_trunc       |  count
    ------------------------+---------
     2015-11-13 01:00:00+01 | 3379266
     2015-11-13 02:00:00+01 | 3379703
     2015-11-13 03:00:00+01 | 3378473
     2015-11-13 04:00:00+01 | 3380569
     2015-11-13 05:00:00+01 | 3377806
     2015-11-13 06:00:00+01 | 3379306
     2015-11-13 07:00:00+01 | 3380283
     2015-11-13 08:00:00+01 | 3380090
     2015-11-13 09:00:00+01 | 3379115
     2015-11-13 10:00:00+01 | 3378317
     2015-11-13 11:00:00+01 | 3379619
     2015-11-13 12:00:00+01 | 3378967
     2015-11-13 13:00:00+01 | 3375900
     2015-11-13 14:00:00+01 | 3377003
     2015-11-13 15:00:00+01 | 3379585
     2015-11-13 16:00:00+01 | 3380145
     2015-11-13 17:00:00+01 | 3380139
     2015-11-13 18:00:00+01 | 3382097
     2015-11-13 19:00:00+01 | 3383785
     2015-11-13 20:00:00+01 | 3383414
     2015-11-13 21:00:00+01 | 3381046
     2015-11-13 22:00:00+01 | 3382832
     2015-11-13 23:00:00+01 | 3383440
     2015-11-14 00:00:00+01 | 3382123
     2015-11-14 01:00:00+01 | 3381796
     2015-11-14 02:00:00+01 | 3381401
     2015-11-14 03:00:00+01 | 3382060
     2015-11-14 04:00:00+01 | 3383890
     2015-11-14 05:00:00+01 | 3379148
     2015-11-14 06:00:00+01 | 3378250
     2015-11-14 07:00:00+01 | 3378963
     2015-11-14 08:00:00+01 | 3378471
     2015-11-14 09:00:00+01 | 3377148
     2015-11-14 10:00:00+01 | 3378257
     2015-11-14 11:00:00+01 | 3378423
     2015-11-14 12:00:00+01 | 3377880
     2015-11-14 13:00:00+01 | 3377636
     2015-11-14 14:00:00+01 | 3378441
     2015-11-14 15:00:00+01 | 3377179
     2015-11-14 16:00:00+01 | 3378204
     2015-11-14 17:00:00+01 | 3377643
     2015-11-14 18:00:00+01 | 3377226
     2015-11-14 19:00:00+01 | 3378534
     2015-11-14 20:00:00+01 | 3378792
     2015-11-14 21:00:00+01 | 3376873
     2015-11-14 22:00:00+01 | 3379069
     2015-11-14 23:00:00+01 | 3378414
     2015-11-15 00:00:00+01 | 3377531
    (48 rows)
    
    zabbix=# SELECT date_trunc('hour', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_log
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447545600
    zabbix-# GROUP  BY 1
    zabbix-# ORDER  BY 1 ASC;
           date_trunc       | count
    ------------------------+--------
     2015-11-13 01:00:00+01 |  17256
     2015-11-13 02:00:00+01 |  10235
     2015-11-13 03:00:00+01 |  10245
     2015-11-13 04:00:00+01 |  10897
     2015-11-13 05:00:00+01 |   9151
     2015-11-13 06:00:00+01 |  12318
     2015-11-13 07:00:00+01 |  21741
     2015-11-13 08:00:00+01 |  40169
     2015-11-13 09:00:00+01 |  64447
     2015-11-13 10:00:00+01 |  81163
     2015-11-13 11:00:00+01 |  83733
     2015-11-13 12:00:00+01 |  80605
     2015-11-13 13:00:00+01 |  88795
     2015-11-13 14:00:00+01 |  90636
     2015-11-13 15:00:00+01 |  92678
     2015-11-13 16:00:00+01 | 103498
     2015-11-13 17:00:00+01 | 103028
     2015-11-13 18:00:00+01 |  96390
     2015-11-13 19:00:00+01 |  97112
     2015-11-13 20:00:00+01 |  92188
     2015-11-13 21:00:00+01 |  82307
     2015-11-13 22:00:00+01 |  65486
     2015-11-13 23:00:00+01 |  41005
     2015-11-14 00:00:00+01 |  27824
     2015-11-14 01:00:00+01 |  19751
     2015-11-14 02:00:00+01 |  22391
     2015-11-14 03:00:00+01 |  17314
     2015-11-14 04:00:00+01 |  13102
     2015-11-14 05:00:00+01 |  12330
     2015-11-14 06:00:00+01 |   8185
     2015-11-14 07:00:00+01 |  13253
     2015-11-14 08:00:00+01 |  24214
     2015-11-14 09:00:00+01 |  40533
     2015-11-14 10:00:00+01 |  59038
     2015-11-14 11:00:00+01 |  69332
     2015-11-14 12:00:00+01 |  69233
     2015-11-14 13:00:00+01 |  74332
     2015-11-14 14:00:00+01 |  82409
     2015-11-14 15:00:00+01 |  81014
     2015-11-14 16:00:00+01 |  81578
     2015-11-14 17:00:00+01 |  86561
     2015-11-14 18:00:00+01 |  74298
     2015-11-14 19:00:00+01 |  71515
     2015-11-14 20:00:00+01 |  67004
     2015-11-14 21:00:00+01 |  61002
     2015-11-14 22:00:00+01 |  50654
     2015-11-14 23:00:00+01 |  37179
     2015-11-15 00:00:00+01 |  25514
    (48 rows)
    
    zabbix=# SELECT date_trunc('hour', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_text
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447545600
    zabbix-# GROUP  BY 1
    zabbix-# ORDER  BY 1 ASC;
           date_trunc       | count
    ------------------------+--------
     2015-11-13 01:00:00+01 | 184593
     2015-11-13 02:00:00+01 | 184596
     2015-11-13 03:00:00+01 | 184981
     2015-11-13 04:00:00+01 | 184838
     2015-11-13 05:00:00+01 | 185006
     2015-11-13 06:00:00+01 | 184628
     2015-11-13 07:00:00+01 | 184624
     2015-11-13 08:00:00+01 | 184735
     2015-11-13 09:00:00+01 | 185011
     2015-11-13 10:00:00+01 | 184482
     2015-11-13 11:00:00+01 | 184691
     2015-11-13 12:00:00+01 | 184771
     2015-11-13 13:00:00+01 | 184687
     2015-11-13 14:00:00+01 | 184556
     2015-11-13 15:00:00+01 | 185169
     2015-11-13 16:00:00+01 | 184907
     2015-11-13 17:00:00+01 | 184816
     2015-11-13 18:00:00+01 | 184981
     2015-11-13 19:00:00+01 | 184613
     2015-11-13 20:00:00+01 | 184933
     2015-11-13 21:00:00+01 | 184674
     2015-11-13 22:00:00+01 | 184845
     2015-11-13 23:00:00+01 | 184714
     2015-11-14 00:00:00+01 | 184740
     2015-11-14 01:00:00+01 | 184879
     2015-11-14 02:00:00+01 | 185151
     2015-11-14 03:00:00+01 | 185027
     2015-11-14 04:00:00+01 | 185075
     2015-11-14 05:00:00+01 | 184935
     2015-11-14 06:00:00+01 | 184720
     2015-11-14 07:00:00+01 | 184878
     2015-11-14 08:00:00+01 | 184684
     2015-11-14 09:00:00+01 | 184951
     2015-11-14 10:00:00+01 | 184597
     2015-11-14 11:00:00+01 | 184671
     2015-11-14 12:00:00+01 | 184855
     2015-11-14 13:00:00+01 | 184742
     2015-11-14 14:00:00+01 | 184674
     2015-11-14 15:00:00+01 | 185340
     2015-11-14 16:00:00+01 | 184723
     2015-11-14 17:00:00+01 | 184835
     2015-11-14 18:00:00+01 | 184760
     2015-11-14 19:00:00+01 | 184881
     2015-11-14 20:00:00+01 | 184658
     2015-11-14 21:00:00+01 | 184732
     2015-11-14 22:00:00+01 | 184911
     2015-11-14 23:00:00+01 | 184537
     2015-11-15 00:00:00+01 | 184918
    (48 rows)
    
    zabbix=# SELECT date_trunc('hour', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_str
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447545600
    zabbix-# GROUP  BY 1
    zabbix-# ORDER  BY 1 ASC;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 01:00:00+01 | 47987
     2015-11-13 02:00:00+01 | 47900
     2015-11-13 03:00:00+01 | 47869
     2015-11-13 04:00:00+01 | 47872
     2015-11-13 05:00:00+01 | 47882
     2015-11-13 06:00:00+01 | 47882
     2015-11-13 07:00:00+01 | 47945
     2015-11-13 08:00:00+01 | 47958
     2015-11-13 09:00:00+01 | 47969
     2015-11-13 10:00:00+01 | 47877
     2015-11-13 11:00:00+01 | 47919
     2015-11-13 12:00:00+01 | 47811
     2015-11-13 13:00:00+01 | 47794
     2015-11-13 14:00:00+01 | 47769
     2015-11-13 15:00:00+01 | 47797
     2015-11-13 16:00:00+01 | 47903
     2015-11-13 17:00:00+01 | 47898
     2015-11-13 18:00:00+01 | 48318
     2015-11-13 19:00:00+01 | 47833
     2015-11-13 20:00:00+01 | 47829
     2015-11-13 21:00:00+01 | 47882
     2015-11-13 22:00:00+01 | 48012
     2015-11-13 23:00:00+01 | 47857
     2015-11-14 00:00:00+01 | 47959
     2015-11-14 01:00:00+01 | 47911
     2015-11-14 02:00:00+01 | 47864
     2015-11-14 03:00:00+01 | 47841
     2015-11-14 04:00:00+01 | 47836
     2015-11-14 05:00:00+01 | 47815
     2015-11-14 06:00:00+01 | 47789
     2015-11-14 07:00:00+01 | 47945
     2015-11-14 08:00:00+01 | 47924
     2015-11-14 09:00:00+01 | 47936
     2015-11-14 10:00:00+01 | 47988
     2015-11-14 11:00:00+01 | 48011
     2015-11-14 12:00:00+01 | 48030
     2015-11-14 13:00:00+01 | 48026
     2015-11-14 14:00:00+01 | 48018
     2015-11-14 15:00:00+01 | 47988
     2015-11-14 16:00:00+01 | 48018
     2015-11-14 17:00:00+01 | 48094
     2015-11-14 18:00:00+01 | 48510
     2015-11-14 19:00:00+01 | 48020
     2015-11-14 20:00:00+01 | 47987
     2015-11-14 21:00:00+01 | 47930
     2015-11-14 22:00:00+01 | 48100
     2015-11-14 23:00:00+01 | 48076
     2015-11-15 00:00:00+01 | 48098
    (48 rows)
    
    zabbix=#
    When looking at top values received per minute of single items, then there is none in the questioned time neither:
    Code:
    zabbix=# SELECT date_trunc('minute', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447408800
    zabbix-# GROUP  BY 1,
    zabbix-#           itemid
    zabbix-# ORDER  BY 2 DESC
    zabbix-# LIMIT  50;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 01:17:00+01 |     4
     2015-11-13 01:23:00+01 |     4
     2015-11-13 01:10:00+01 |     4
     2015-11-13 01:16:00+01 |     4
     2015-11-13 01:20:00+01 |     4
     2015-11-13 01:23:00+01 |     4
     2015-11-13 01:06:00+01 |     4
     2015-11-13 01:09:00+01 |     4
     2015-11-13 01:13:00+01 |     4
     2015-11-13 01:16:00+01 |     4
     2015-11-13 01:19:00+01 |     4
     2015-11-13 01:20:00+01 |     4
     2015-11-13 01:22:00+01 |     4
     2015-11-13 01:22:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:07:00+01 |     4
     2015-11-13 01:09:00+01 |     4
     2015-11-13 01:11:00+01 |     4
     2015-11-13 01:13:00+01 |     4
     2015-11-13 01:14:00+01 |     4
     2015-11-13 01:15:00+01 |     4
     2015-11-13 01:18:00+01 |     4
     2015-11-13 01:18:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:19:00+01 |     4
     2015-11-13 01:21:00+01 |     4
     2015-11-13 01:21:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:06:00+01 |     4
     2015-11-13 01:07:00+01 |     4
     2015-11-13 01:08:00+01 |     4
     2015-11-13 01:08:00+01 |     4
     2015-11-13 01:10:00+01 |     4
     2015-11-13 01:11:00+01 |     4
     2015-11-13 01:12:00+01 |     4
     2015-11-13 01:12:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:14:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:15:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:17:00+01 |     4
     2015-11-13 01:24:00+01 |     4
     2015-11-13 01:24:00+01 |     4
    (50 rows)
    
    zabbix=# SELECT date_trunc('minute', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_uint
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447408800
    zabbix-# GROUP  BY 1,
    zabbix-#           itemid
    zabbix-# ORDER  BY 2 DESC
    zabbix-# LIMIT  50;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:04:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:00:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:01:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:02:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:03:00+01 |     4
     2015-11-13 01:05:00+01 |     4
     2015-11-13 01:05:00+01 |     4
    (50 rows)
    
    zabbix=# SELECT date_trunc('minute', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_log
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447408800
    zabbix-# GROUP  BY 1,
    zabbix-#           itemid
    zabbix-# ORDER  BY 2 DESC
    zabbix-# LIMIT  50;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 10:10:00+01 |   363
     2015-11-13 10:10:00+01 |   358
     2015-11-13 10:21:00+01 |   358
     2015-11-13 10:21:00+01 |   356
     2015-11-13 10:30:00+01 |   340
     2015-11-13 10:06:00+01 |   340
     2015-11-13 10:30:00+01 |   339
     2015-11-13 10:06:00+01 |   334
     2015-11-13 10:51:00+01 |   331
     2015-11-13 10:50:00+01 |   331
     2015-11-13 10:11:00+01 |   330
     2015-11-13 10:11:00+01 |   328
     2015-11-13 10:33:00+01 |   328
     2015-11-13 10:55:00+01 |   326
     2015-11-13 10:33:00+01 |   325
     2015-11-13 10:55:00+01 |   325
     2015-11-13 10:49:00+01 |   322
     2015-11-13 10:27:00+01 |   321
     2015-11-13 10:50:00+01 |   321
     2015-11-13 10:52:00+01 |   319
     2015-11-13 10:12:00+01 |   318
     2015-11-13 10:49:00+01 |   317
     2015-11-13 10:03:00+01 |   317
     2015-11-13 10:00:00+01 |   316
     2015-11-13 10:25:00+01 |   315
     2015-11-13 10:01:00+01 |   315
     2015-11-13 10:18:00+01 |   315
     2015-11-13 10:16:00+01 |   315
     2015-11-13 10:27:00+01 |   314
     2015-11-13 10:57:00+01 |   314
     2015-11-13 10:15:00+01 |   314
     2015-11-13 10:56:00+01 |   313
     2015-11-13 10:28:00+01 |   313
     2015-11-13 10:57:00+01 |   313
     2015-11-13 10:31:00+01 |   313
     2015-11-13 10:09:00+01 |   313
     2015-11-13 10:16:00+01 |   313
     2015-11-13 10:56:00+01 |   313
     2015-11-13 10:45:00+01 |   313
     2015-11-13 10:40:00+01 |   312
     2015-11-13 10:32:00+01 |   312
     2015-11-13 10:01:00+01 |   312
     2015-11-13 10:25:00+01 |   312
     2015-11-13 10:41:00+01 |   312
     2015-11-13 10:12:00+01 |   312
     2015-11-13 10:28:00+01 |   311
     2015-11-13 10:15:00+01 |   311
     2015-11-13 10:02:00+01 |   310
     2015-11-13 10:22:00+01 |   310
     2015-11-13 10:31:00+01 |   310
    (50 rows)
    
    zabbix=# SELECT date_trunc('minute', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_text
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447408800
    zabbix-# GROUP  BY 1,
    zabbix-#           itemid
    zabbix-# ORDER  BY 2 DESC
    zabbix-# LIMIT  50;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 04:23:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 08:05:00+01 |     4
     2015-11-13 07:23:00+01 |     4
     2015-11-13 05:10:00+01 |     4
     2015-11-13 09:59:00+01 |     4
     2015-11-13 08:41:00+01 |     4
     2015-11-13 08:22:00+01 |     4
     2015-11-13 08:22:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 04:16:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 04:16:00+01 |     4
     2015-11-13 02:06:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 05:10:00+01 |     4
     2015-11-13 10:17:00+01 |     4
     2015-11-13 04:12:00+01 |     4
     2015-11-13 04:12:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 05:58:00+01 |     4
     2015-11-13 02:10:00+01 |     4
     2015-11-13 04:00:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 01:33:00+01 |     4
     2015-11-13 04:13:00+01 |     4
     2015-11-13 07:23:00+01 |     4
     2015-11-13 08:22:00+01 |     4
     2015-11-13 04:12:00+01 |     4
     2015-11-13 07:41:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 08:34:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 03:11:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 08:32:00+01 |     4
     2015-11-13 10:07:00+01 |     4
     2015-11-13 06:50:00+01 |     4
     2015-11-13 07:25:00+01 |     4
     2015-11-13 02:45:00+01 |     4
     2015-11-13 09:21:00+01 |     4
     2015-11-13 10:10:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 08:29:00+01 |     4
     2015-11-13 08:13:00+01 |     4
     2015-11-13 04:26:00+01 |     4
     2015-11-13 07:21:00+01 |     4
    (50 rows)
    zabbix=# SELECT date_trunc('minute', to_timestamp(clock)),
    zabbix-#        count(*)
    zabbix-# FROM   history_str
    zabbix-# WHERE  clock >= 1447372800
    zabbix-#        AND clock < 1447408800
    zabbix-# GROUP  BY 1,
    zabbix-#           itemid
    zabbix-# ORDER  BY 2 DESC
    zabbix-# LIMIT  50;
           date_trunc       | count
    ------------------------+-------
     2015-11-13 10:03:00+01 |     3
     2015-11-13 02:56:00+01 |     3
     2015-11-13 01:35:00+01 |     3
     2015-11-13 07:16:00+01 |     3
     2015-11-13 08:25:00+01 |     3
     2015-11-13 09:56:00+01 |     3
     2015-11-13 10:17:00+01 |     3
     2015-11-13 05:05:00+01 |     3
     2015-11-13 05:01:00+01 |     3
     2015-11-13 10:30:00+01 |     3
     2015-11-13 05:59:00+01 |     3
     2015-11-13 08:40:00+01 |     3
     2015-11-13 09:38:00+01 |     3
     2015-11-13 02:03:00+01 |     3
     2015-11-13 09:40:00+01 |     3
     2015-11-13 04:58:00+01 |     3
     2015-11-13 08:27:00+01 |     3
     2015-11-13 09:28:00+01 |     3
     2015-11-13 09:20:00+01 |     3
     2015-11-13 05:25:00+01 |     3
     2015-11-13 06:03:00+01 |     3
     2015-11-13 03:08:00+01 |     3
     2015-11-13 01:30:00+01 |     3
     2015-11-13 06:34:00+01 |     2
     2015-11-13 10:30:00+01 |     2
     2015-11-13 10:47:00+01 |     2
     2015-11-13 05:01:00+01 |     2
     2015-11-13 02:51:00+01 |     2
     2015-11-13 04:46:00+01 |     2
     2015-11-13 05:47:00+01 |     2
     2015-11-13 05:33:00+01 |     2
     2015-11-13 03:14:00+01 |     2
     2015-11-13 01:15:00+01 |     2
     2015-11-13 05:34:00+01 |     2
     2015-11-13 04:29:00+01 |     2
     2015-11-13 02:16:00+01 |     2
     2015-11-13 01:56:00+01 |     2
     2015-11-13 03:12:00+01 |     2
     2015-11-13 02:50:00+01 |     2
     2015-11-13 08:29:00+01 |     2
     2015-11-13 03:42:00+01 |     2
     2015-11-13 10:15:00+01 |     2
     2015-11-13 10:06:00+01 |     2
     2015-11-13 04:37:00+01 |     2
     2015-11-13 07:37:00+01 |     2
     2015-11-13 06:56:00+01 |     2
     2015-11-13 02:49:00+01 |     2
     2015-11-13 04:55:00+01 |     2
     2015-11-13 03:29:00+01 |     2
     2015-11-13 03:28:00+01 |     2
    (50 rows)
    
    zabbix=#
    Quick'n'dirty check of suspicious log file messages:
    Code:
    [root@zabbix-server ~]# grep -c ':20151114:' /var/log/zabbix/zabbix_server.log
    39205
    [root@zabbix-server ~]# grep ':20151114:' /var/log/zabbix/zabbix_server.log | grep -v 'sending configuration data to proxy' | grep -v 'cannot connect to proxy' | grep -v 'cannot process step' | grep -v 'became supported' | grep -v 'became not supported' | grep -v 'is not suitable for value type' | grep -v housekeepe | wc -l
    0
    [root@zabbix-server ~]#
    Zabbx server configuration at time of the issue:

    Code:
    [root@zabbix-server ~]# grep -Ev '^[^=]+(IP|File|Script)|^DB|^#|^$' /etc/zabbix/zabbix_server.conf
    StartTrappers=50
    StartPingers=4
    StartDiscoverers=5
    StartTimers=2
    CacheSize=350M
    HistoryCacheSize=2G
    TrendCacheSize=128M
    HistoryTextCacheSize=2G
    ValueCacheSize=600M
    Timeout=4
    TrapperTimeout=60
    StartProxyPollers=4
    ProxyConfigFrequency=120
    [root@zabbix-server ~]#
    Last edited by BDiE8VNy; 17-11-2015, 19:16. Reason: Added Zabbix server configuration
  • zalex_ua
    Senior Member
    Zabbix Certified Trainer
    Zabbix Certified SpecialistZabbix Certified Professional
    • Oct 2009
    • 1286

    #2
    As we can see text write cache did not drop too much. It allows to think that incoming values could be from numeric items too, or from logs, but pasred on agent side ad stored as numeric values.
    In this case you need to check numeric tables too using similar SQL query.

    If your text cache configured to be big - I'd decrease it to see better drop downs on graph to be able perform correlation.

    Comment

    • zalex_ua
      Senior Member
      Zabbix Certified Trainer
      Zabbix Certified SpecialistZabbix Certified Professional
      • Oct 2009
      • 1286

      #3
      make also sure that incoming data have actual time stamps

      Comment

      • BDiE8VNy
        Senior Member
        • Apr 2010
        • 680

        #4
        Root cause identified in ZBXNEXT-3051

        Comment

        Working...