Ad Widget

Collapse

All agents unreachable for 5 minutes everyday at 3:36am

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • abdalaziz.janem
    Junior Member
    • Feb 2020
    • 6

    #1

    All agents unreachable for 5 minutes everyday at 3:36am

    Hi All,

    All agents that connected with zabbix-server sent emails everyday at 3:36am with title "agents unreachable for 5 minutes"
    This is the template: {AGENT_NAME:agent.ping.nodata(300)}=1

    there is a slow query in the zabbix server log file in that time:


    ----------------------------------------------------------------
    1883:20200205:033537.012 slow query: 276.178750 sec, "delete from event_suppress where suppress_until<1580866260"
    1989:20200205:033537.014 slow query: 332.165419 sec, "update item_discovery set lastcheck=1580866204 where itemid in (35168,35169,35170,35171,35172,35173,35174,35175,3 5176,35177);
    1950:20200205:033537.016 slow query: 14.793457 sec, "update hosts set errors_from=1580866522,disable_until=1580866537 where hostid=10353"
    1957:20200205:033537.032 slow query: 315.788106 sec, "update hosts set snmp_disable_until=1580866281 where hostid=10267"
    1892:20200205:033537.037 slow query: 331.853711 sec, "insert into history_uint (itemid,clock,ns,value) values (36304,1580866205,121028142,1);
    1990:20200205:033537.037 slow query: 333.758244 sec, "update item_discovery set lastcheck=1580866203 where itemid in (34811,34812,34813,34814,34815,34816,34817,34818,3 4819,34820,34821,34822,34823,34824,34825,34826,348 27,34828,34829,34830,34831,34832,34833,34834,34835 ,34836,34837,34838,34839,34840,34841,34842,34843,3 4844,34845,34846,34847,34848,34849,34850,34851,348 52,34853,34854,34855,34856,34857,34858,34859,34860 ,34861,34862,34863,34864,34865,34866,34867,34868,3 4869,34870,34871,34872,34873,34874,34875,34876,348 77,34878,34879,34880,34881,34882,34883,34884,34885 ,34886,34887,34888,34889,34890,34891,34892,34893,3 4894,34895,34896,34897,34898,34899,34900,34901,349 02,34903,34904,34905,34906,34907,34908,34909,34910 ,34911,34912,34913,34914,34915,34916,34917,34918,3 4919,34920,34921,34922,34923,34924,34925,34926);
    1889:20200205:033537.053 slow query: 331.931698 sec, "insert into history (itemid,clock,ns,value) values (34753,1580866202,814875402,0.000000),(34754,15808 66202,814875402,0.001523),(35404,1580866204,665485 746,99.582007),(33544,1580866204,846570469,0.00000 0),(35224,1580866204,960713646,0.000000),(33784,15 80866204,967618763,41.319732),(35764,1580866205,11 735349,0.000000),(36665,1580866205,25613648,84.636 230),(35344,1580866205,31906529,47.113662),(35765, 1580866205,79446100,0.000000);
    1891:20200205:033537.054 slow query: 332.936342 sec, "insert into history (itemid,clock,ns,value) values (33995,1580866201,794954189,0.000100),(33994,15808 66201,794954189,0.000000),(34751,1580866201,794954 189,0.000640),(34750,1580866201,794954189,0.000000 ),(35403,1580866203,570105937,0.000000),(34263,158 0866203,573233510,41.260867),(36243,1580866203,579 231337,19.569406),(33063,1580866203,664517957,54.2 84390);
    1888:20200205:033537.063 slow query: 333.948823 sec, "insert into history (itemid,clock,ns,value) values (34699,1580866200,803772898,0.000720),(34698,15808 66200,803772898,0.000000),(35222,1580866202,657864 407,1.040625),(35402,1580866202,958690131,0.000000 ),(36122,1580866202,968829914,99.993322),(36062,15 80866202,984757952,99.931250);
    1892:20200205:034500.117 slow query: 3.166743 sec, "commit;"
    1891:20200205:034500.117 slow query: 3.298416 sec, "commit;"
    1883:20200206:033612.242 slow query: 311.306443 sec, "delete from event_suppress where suppress_until<1580952660"
    1957:20200206:033612.246 slow query: 329.330588 sec, "update hosts set snmp_disable_until=1580952702 where hostid=10267"
    1990:20200206:033612.268 slow query: 367.673300 sec, "update item_discovery set lastcheck=1580952604 where itemid in (35168,35169,35170,35171,35172,35173,35174,35175,3 5176,35177);
    1989:20200206:033612.273 slow query: 360.536352 sec, "update item_discovery set lastcheck=1580952611 where itemid in (34811,34812,34813,34814,34815,34816,34817,34818,3 4819,34820,34821,34822,34823,34824,34825,34826,348 27,34828,34829,34830,34831,34832,34833,34834,34835 ,34836,34837,34838,34839,34840,34841,34842,34843,3 4844,34845,34846,34847,34848,34849,34850,34851,348 52,34853,34854,34855,34856,34857,34858,34859,34860 ,34861,34862,34863,34864,34865,34866,34867,34868,3 4869,34870,34871,34872,34873,34874,34875,34876,348 77,34878,34879,34880,34881,34882,34883,34884,34885 ,34886,34887,34888,34889,34890,34891,34892,34893,3 4894,34895,34896,34897,34898,34899,34900,34901,349 02,34903,34904,34905,34906,34907,34908,34909,34910 ,34911,34912,34913,34914,34915,34916,34917,34918,3 4919,34920,34921,34922,34923,34924,34925,34926);
    1892:20200206:033612.297 slow query: 368.237224 sec, "insert into history (itemid,clock,ns,value) values (33994,1580952601,810860594,0.000000),(33995,15809 52601,810860594,0.000100),(34750,1580952601,810860 594,0.000000),(34751,1580952601,810860594,0.000497 ),(33063,1580952603,330888673,53.840547),(34263,15 80952603,396138061,41.260529),(35763,1580952603,40 7384092,81.375286),(35403,1580952603,412116948,0.0 00000),(36243,1580952603,413398446,26.771749);
    1889:20200206:033612.297 slow query: 369.238683 sec, "insert into history (itemid,clock,ns,value) values (34698,1580952600,848150931,0.000000),(34699,15809 52600,848150931,0.000550),(36122,1580952602,268586 651,99.993322),(35402,1580952602,277445159,0.00000 0),(36062,1580952602,304728312,99.931250),(35762,1 580952602,318373094,38.725195),(35222,1580952602,4 11432423,1.027500);
    1888:20200206:033612.297 slow query: 367.236340 sec, "insert into history (itemid,clock,ns,value) values (34753,1580952602,859782709,0.000000),(34754,15809 52602,859782709,0.001383),(35404,1580952604,419068 534,99.660570),(33784,1580952604,424091573,40.8641 31),(33544,1580952604,458171992,0.000000),(35764,1 580952604,492110685,0.000000),(35224,1580952604,56 5526961,0.000000),(35344,1580952604,590090107,82.6 03037);
    1891:20200206:033612.305 slow query: 366.243045 sec, "insert into history (itemid,clock,ns,value) values (34756,1580952603,832757993,0.000000),(34757,15809 52603,832757993,0.001127),(35765,1580952605,506764 372,0.000000),(36665,1580952605,568297883,84.56390 4),(36065,1580952605,591411754,20.849133),(33245,1 580952605,591553893,0.161875),(33545,1580952605,59 3739131,0.016012),(35405,1580952605,597605470,0.00 0000),(35225,1580952605,631749601,0.000000),(35345 ,1580952605,711755549,21.616057);
    1883:20200207:033454.094 slow query: 233.233829 sec, "delete from event_suppress where suppress_until<1581039060"
    1957:20200207:033454.099 slow query: 254.258724 sec, "update hosts set snmp_disable_until=1581039099 where hostid=10267"
    1989:20200207:033454.102 slow query: 289.730347 sec, "update item_discovery set lastcheck=1581039004 where itemid in (35168,35169,35170,35171,35172,35173,35174,35175,3 5176,35177);
    1990:20200207:033454.102 slow query: 281.981234 sec, "update item_discovery set lastcheck=1581039012 where itemid in (29921,29922,29923,29924,29925,29927,29928,29929,2 9930,29932,32739,32740,32741,32749,32751,32755,328 24,32951,32953,32956,32970);
    1891:20200207:033454.127 slow query: 288.619208 sec, "insert into history (itemid,clock,ns,value) values (34756,1581039003,74525420,0.000000),(34757,158103 9003,74525420,0.002090),(35344,1581039004,51467708 7,82.459696),(35224,1581039004,525224024,0.000000) ,(33544,1581039004,537343200,0.012332),(35764,1581 039004,552804989,0.000000);
    1892:20200207:033454.130 slow query: 290.628261 sec, "insert into history (itemid,clock,ns,value) values (33994,1581039001,53409902,0.000000),(33995,158103 9001,53409902,0.000100),(34750,1581039001,53409902 ,0.000000),(34751,1581039001,53409902,0.000500),(3 5222,1581039002,542604035,1.003750),(36962,1581039 002,576580975,0.000000),(36062,1581039002,64254136 7,99.931250);
    1888:20200207:033454.152 slow query: 289.647623 sec, "insert into history (itemid,clock,ns,value) values (34754,1581039002,77225674,0.001663),(34753,158103 9002,77225674,0.000000),(34263,1581039003,66066572 6,41.260215),(36963,1581039003,779785188,0.398690) ,(36243,1581039003,797326365,26.861442),(36903,158 1039003,797411348,0.000000),(36723,1581039003,7986 79843,4917.993250),(35403,1581039003,951851899,0.0 00000),(35763,1581039003,960513396,80.167297),(369 64,1581039004,160154336,0.067172),(35404,158103900 4,164025258,99.539298),(33784,1581039004,202587260 ,37.373957),(33063,1581039004,406951644,52.932749) ;
    1882:20200207:033454.206 slow query: 36.204415 sec, "delete from history where itemid=10073 and clock<1580434457"
    1889:20200207:033454.216 slow query: 291.699097 sec, "insert into history_uint (itemid,clock,ns,value) values (34814,1581038997,166963255,0),(34817,1581038997,1 78321313,0),(34820,1581038997,186532351,1),(34829, 1581038997,201832330,1),(34832,1581038997,20967934 7,2),(34836,1581038997,217414962,1),(34840,1581038 997,225576329,1),(34845,1581038997,236244645,0),(3 4924,1581038997,244155155,2),(34848,1581038997,251 949670,2),(34852,1581038997,259897901,1),(34856,15 81038997,267927507,1),(34860,1581038997,276337215, 1),(34864,1581038997,283769586,0),(34868,158103899 7,291689175,2),(34873,1581038997,299014088,1),(348 76,1581038997,307536742,1),(34880,1581038997,31519 7653,0),(34884,1581038997,323372707,1),(34888,1581 038997,331571499,1),(34894,1581038997,339226931,0) ,(34898,1581038997,347247353,0),(34900,1581038997, 359562870,0),(34906,1581038997,368523728,0),(34910 ,1581038997,376170435,0),(34914,1581038997,3830203 83,1),(34918,1581038997,390479130,0),(34922,158103 8997,397034395,1);
    1883:20200208:033346.243 slow query: 165.674780 sec, "delete from event_suppress where suppress_until<1581125460"
    1957:20200208:033346.244 slow query: 211.211809 sec, "update hosts set snmp_disable_until=1581125475 where hostid=10267"
    1990:20200208:033346.244 slow query: 221.156411 sec, "update item_discovery set lastcheck=1581125405 where itemid in (35168,35169,35170,35171,35172,35173,35174,35175,3 5176,35177);
    1989:20200208:033346.247 slow query: 217.908307 sec, "update item_discovery set lastcheck=1581125408 where itemid in (34811,34812,34813,34814,34815,34816,34817,34818,3 4819,34820,34821,34822,34823,34824,34825,34826,348 27,34828,34829,34830,34831,34832,34833,34834,34835 ,34836,34837,34838,34839,34840,34841,34842,34843,3 4844,34845,34846,34847,34848,34849,34850,34851,348 52,34853,34854,34855,34856,34857,34858,34859,34860 ,34861,34862,34863,34864,34865,34866,34867,34868,3 4869,34870,34871,34872,34873,34874,34875,34876,348 77,34878,34879,34880,34881,34882,34883,34884,34885 ,34886,34887,34888,34889,34890,34891,34892,34893,3 4894,34895,34896,34897,34898,34899,34900,34901,349 02,34903,34904,34905,34906,34907,34908,34909,34910 ,34911,34912,34913,34914,34915,34916,34917,34918,3 4919,34920,34921,34922,34923,34924,34925,34926);
    1889:20200208:033346.248 slow query: 223.718706 sec, "insert into history (itemid,clock,ns,value) values (34696,1581125399,582145884,0.000563),(34695,15811 25399,582145884,0.000000),(36722,1581125402,454244 497,2366.847593);
    1892:20200208:033346.260 slow query: 222.729598 sec, "insert into history (itemid,clock,ns,value) values (34698,1581125400,809755197,0.000000),(34699,15811 25400,809755197,0.000533),(36062,1581125402,549814 266,99.931250),(35222,1581125402,590065765,1.01375 0),(35402,1581125402,594528498,0.000000),(35762,15 81125402,599569037,38.581391),(36962,1581125402,60 1434823,0.000000),(36122,1581125402,602534186,99.9 93322),(36723,1581125403,343097680,16228.802481),( 36903,1581125403,343162646,0.000000),(35403,158112 5403,360578739,0.000000);
    1891:20200208:033346.264 slow query: 221.732490 sec, "insert into history (itemid,clock,ns,value) values (33995,1581125401,590601921,0.000100),(34751,15811 25401,590601921,0.000433),(33994,1581125401,590601 921,0.000000),(34750,1581125401,590601921,0.000000 ),(36963,1581125403,609940380,0.429364),(36243,158 1125403,619152703,22.378741),(34263,1581125403,737 588636,41.258092),(35763,1581125403,943060371,80.3 31810),(33063,1581125404,12384474,52.991820),(3534 4,1581125404,449287268,81.951404),(35764,158112540 4,465567917,0.000000);
    1888:20200208:033346.283 slow query: 220.748845 sec, "insert into history (itemid,clock,ns,value) values (34754,1581125402,817327800,0.001753),(34753,15811 25402,817327800,0.000000),(35224,1581125404,597304 182,0.000000),(33544,1581125404,601551163,0.000000 ),(36964,1581125404,959292304,0.083305),(33784,158 1125404,959918628,37.179062),(35404,1581125404,960 060390,99.470139),(36905,1581125405,4043385,1.4048 15),(36665,1581125405,96676936,84.559326),(35225,1 581125405,210828270,0.000000),(33245,1581125405,32 9312641,0.065625),(33545,1581125405,351714960,0.00 0000),(35345,1581125405,467469439,21.489688),(3540 5,1581125405,469835065,0.000000),(35765,1581125405 ,490585459,0.000000);
    1891:20200208:034433.128 slow query: 3.049387 sec, "commit;"
    1889:20200208:103511.529 slow query: 5.420568 sec, "commit;"
    1957:20200209:033312.402 slow query: 159.038666 sec, "update hosts set snmp_disable_until=1581211893 where hostid=10267"
    1989:20200209:033312.403 slow query: 187.534647 sec, "update item_discovery set lastcheck=1581211804 where itemid in (35168,35169,35170,35171,35172,35173,35174,35175,3 5176,35177);
    1891:20200209:033312.404 slow query: 187.137288 sec, "insert into history (itemid,clock,ns,value) values (34754,1581211802,987844929,0.001587),(34753,15812 11802,987844929,0.000000),(34756,1581211803,178300 589,0.000000),(34757,1581211803,178300589,0.001113 ),(33784,1581211804,432805176,37.288600),(33544,15 81211804,546168460,0.000000),(35344,1581211804,666 831754,81.270500),(35404,1581211804,679775601,99.5 36980),(36964,1581211804,769157203,0.066644),(3522 4,1581211804,781495737,0.000000),(35764,1581211804 ,788375574,0.000000);
    1892:20200209:033312.405 slow query: 189.139812 sec, "insert into history (itemid,clock,ns,value) values (34698,1581211800,981173198,0.000000),(34699,15812 11800,981173198,0.000510),(34751,1581211801,160666 926,0.000477),(34750,1581211801,160666926,0.000000 ),(33994,1581211801,160666926,0.000000),(33995,158 1211801,160666926,0.000100),(35402,1581211802,9401 67159,0.000000),(36062,1581211802,964878787,99.931 250),(36122,1581211802,965999065,99.993322),(36722 ,1581211802,970984616,78.760964),(36962,1581211802 ,972602358,0.000000),(35762,1581211802,981671061,3 7.982233),(35222,1581211803,73902124,0.970625);
    1889:20200209:033312.407 slow query: 190.143194 sec, "insert into history (itemid,clock,ns,value) values (34696,1581211799,978137438,0.000663),(34695,15812 11799,978137438,0.000000),(36421,1581211801,445383 530,99.939703),(35341,1581211801,560700819,98.5532 33),(36901,1581211801,560884448,1.006374),(36721,1 581211801,588880475,0.029798),(35221,1581211801,73 0009237,0.903125),(36061,1581211801,733500010,98.3 87456),(29161,1581211801,734877109,66.841952),(331 21,1581211801,739645231,9.999572),(34261,158121180 1,939099818,98.840261);
    1888:20200209:033312.408 slow query: 188.141347 sec, "insert into history (itemid,clock,ns,value) values (34263,1581211803,319055047,41.258299),(36243,1581 211803,327313524,23.361360),(35403,1581211803,3778 39326,0.000000),(36963,1581211803,384538925,0.4012 42),(36723,1581211803,386214324,1386.192967),(3690 3,1581211803,386294136,0.000000),(35763,1581211803 ,399294898,80.850441),(33063,1581211803,623343702, 54.032330);
    1990:20200209:033312.409 slow query: 183.357504 sec, "update item_discovery set lastcheck=1581211809 where itemid in (34811,34812,34813,34814,34815,34816,34817,34818,3 4819,34820,34821,34822,34823,34824,34825,34826,348 27,34828,34829,34830,34831,34832,34833,34834,34835 ,34836,34837,34838,34839,34840,34841,34842,34843,3 4844,34845,34846,34847,34848,34849,34850,34851,348 52,34853,34854,34855,34856,34857,34858,34859,34860 ,34861,34862,34863,34864,34865,34866,34867,34868,3 4869,34870,34871,34872,34873,34874,34875,34876,348 77,34878,34879,34880,34881,34882,34883,34884,34885 ,34886,34887,34888,34889,34890,34891,34892,34893,3 4894,34895,34896,34897,34898,34899,34900,34901,349 02,34903,34904,34905,34906,34907,34908,34909,34910 ,34911,34912,34913,34914,34915,34916,34917,34918,3 4919,34920,34921,34922,34923,34924,34925,34926);
    1883:20200209:033312.411 slow query: 132.244413 sec, "delete from event_suppress where suppress_until<1581211860"
    1891:20200209:034505.890 slow query: 3.620666 sec, "commit;"
    ----------------------------------------------------------------

    Please advice how can we solve this iissue ??

    Regards




  • abdalaziz.janem
    Junior Member
    • Feb 2020
    • 6

    #2
    Any update please ?

    Comment

    • tim.mooney
      Senior Member
      • Dec 2012
      • 1427

      #3
      It's good that you included information about the many slow queries that are happening at the same time. They are very likely the source of your problem.

      Is you Zabbix database on the same server as your Zabbix server, or is it on a separate system?

      Is there some very I/O intensive operation happening on the database server at that time? Backups?

      Does the database server have its own storage, or does it share storage bandwidth with other hosts? For example, if your database storage is on the same group of disks on a SAN, and some other host has a very I/O intensive task taking place on those disks.

      Comment

      Working...