Ad Widget

Collapse

Zabbix service start and stop randomly

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • zux
    Member
    • Sep 2018
    • 93

    #16
    it seems that Zabbix is running, check with "ps aux | grep zabbix_server"
    Also it seems you have debug level set very high, it's hard to notice serious problems with this much spam in there. Try to decrease the debug level to 3. But you need to first understand where is zabbix running from.

    Comment

    • DSI_agglo
      Member
      • Jul 2019
      • 35

      #17
      After "ps aux | grep zabbix_server" i have this :

      zabbix 52327 6.7 3.5 6800012 435560 ? S 14:38 0:30 /usr/sbin/zabbix_server -c /etc/zabbix/zabbix_server.conf
      zabbix 52408 25.1 3.4 6800012 424776 ? R 14:38 1:47 /usr/sbin/zabbix_server: configuration syncer [synced configuration in 95.116419 sec, syncing configuration]
      zabbix 52409 0.0 2.3 6800012 287704 ? S 14:38 0:00 /usr/sbin/zabbix_server: housekeeper [startup idle for 30 minutes]
      zabbix 52410 0.0 2.3 6800012 293124 ? S 14:38 0:00 /usr/sbin/zabbix_server: timer #1 [updated 0 hosts, suppressed 0 events in 0.005079 sec, idle 59 sec]
      zabbix 52411 0.0 2.3 6800012 292128 ? S 14:38 0:00 /usr/sbin/zabbix_server: http poller #1 [got 0 values in 0.006357 sec, idle 5 sec]
      zabbix 52412 0.0 2.3 6804184 293276 ? S 14:38 0:00 /usr/sbin/zabbix_server: discoverer #1 [processed 0 rules in 0.004464 sec, idle 60 sec]
      zabbix 52413 11.2 3.2 6800072 402236 ? S 14:38 0:47 /usr/sbin/zabbix_server: history syncer #1 [processed 1076 values, 1286 triggers in 8.371571 sec, idle 1 sec]
      zabbix 52421 11.4 3.2 6800072 402400 ? R 14:38 0:48 /usr/sbin/zabbix_server: history syncer #2 [processed 1707 values, 2151 triggers in 11.858034 sec, syncing history]
      zabbix 52422 11.3 3.2 6800072 402064 ? R 14:38 0:48 /usr/sbin/zabbix_server: history syncer #3 [processed 0 values, 0 triggers in 0.000386 sec, syncing history]
      zabbix 52426 12.3 3.2 6800072 402352 ? S 14:38 0:52 /usr/sbin/zabbix_server: history syncer #4 [processed 1182 values, 1554 triggers in 10.329647 sec, syncing history]
      zabbix 52431 0.0 2.3 6800012 292000 ? S 14:38 0:00 /usr/sbin/zabbix_server: escalator #1 [processed 0 escalations in 0.004957 sec, idle 3 sec]
      zabbix 52432 0.0 2.3 6800012 292000 ? S 14:38 0:00 /usr/sbin/zabbix_server: proxy poller #1 [exchanged data with 0 proxies in 0.001236 sec, idle 5 sec]
      zabbix 52433 0.0 2.3 6800012 289604 ? S 14:38 0:00 /usr/sbin/zabbix_server: self-monitoring [processed data in 0.000082 sec, idle 1 sec]
      zabbix 52434 0.0 2.3 6800012 292272 ? S 14:38 0:00 /usr/sbin/zabbix_server: task manager [processed 0 task(s) in 0.006066 sec, idle 5 sec]
      zabbix 52435 7.2 2.8 6807892 352860 ? S 14:38 0:31 /usr/sbin/zabbix_server: poller #1 [got 127 values in 0.234994 sec, idle 1 sec]
      zabbix 52437 7.6 2.8 6807892 353100 ? S 14:38 0:32 /usr/sbin/zabbix_server: poller #2 [got 108 values in 0.191285 sec, idle 1 sec]
      zabbix 52440 8.2 2.8 6807892 353092 ? S 14:38 0:35 /usr/sbin/zabbix_server: poller #3 [got 0 values in 0.000417 sec, idle 1 sec]
      zabbix 52441 7.5 2.8 6807892 353092 ? S 14:38 0:32 /usr/sbin/zabbix_server: poller #4 [got 109 values in 0.260443 sec, idle 1 sec]
      zabbix 52444 7.4 2.8 6807892 353108 ? S 14:38 0:31 /usr/sbin/zabbix_server: poller #5 [got 197 values in 0.245100 sec, idle 1 sec]
      zabbix 52447 0.0 2.4 6807760 302456 ? S 14:38 0:00 /usr/sbin/zabbix_server: unreachable poller #1 [got 0 values in 0.009997 sec, idle 5 sec]
      zabbix 52453 0.0 2.3 6800060 291904 ? S 14:38 0:00 /usr/sbin/zabbix_server: trapper #1 [processed data in 0.000610 sec, waiting for connection]
      zabbix 52456 0.0 2.3 6800060 291904 ? S 14:38 0:00 /usr/sbin/zabbix_server: trapper #2 [processed data in 0.001704 sec, waiting for connection]
      zabbix 52459 0.0 2.3 6800060 291904 ? S 14:38 0:00 /usr/sbin/zabbix_server: trapper #3 [processed data in 0.009570 sec, waiting for connection]
      zabbix 52463 0.0 2.3 6800060 291904 ? S 14:38 0:00 /usr/sbin/zabbix_server: trapper #4 [processed data in 0.000819 sec, waiting for connection]
      zabbix 52466 0.0 2.3 6800060 291904 ? S 14:38 0:00 /usr/sbin/zabbix_server: trapper #5 [processed data in 0.000572 sec, waiting for connection]
      zabbix 52468 0.5 2.4 6803384 297960 ? S 14:38 0:02 /usr/sbin/zabbix_server: icmp pinger #1 [pinging hosts]
      zabbix 52471 0.0 2.3 6800012 292836 ? S 14:38 0:00 /usr/sbin/zabbix_server: alert manager #1 [sent 0, failed 0 alerts, idle 5.038142 sec during 5.039439 sec]
      zabbix 52472 0.0 2.3 6800012 287704 ? S 14:38 0:00 /usr/sbin/zabbix_server: alerter #1 started
      zabbix 52473 0.0 2.3 6800012 287704 ? S 14:38 0:00 /usr/sbin/zabbix_server: alerter #2 started
      zabbix 52474 0.0 2.3 6800012 287704 ? S 14:38 0:00 /usr/sbin/zabbix_server: alerter #3 started
      zabbix 52475 11.6 3.0 6800012 371480 ? S 14:38 0:49 /usr/sbin/zabbix_server: preprocessing manager #1 [queued 6266, processed 2138 values, idle 0.493292 sec during 5.003150 sec]
      zabbix 52476 1.8 2.3 6800012 289504 ? S 14:38 0:07 /usr/sbin/zabbix_server: preprocessing worker #1 started
      zabbix 52477 1.8 2.3 6800012 289504 ? S 14:38 0:07 /usr/sbin/zabbix_server: preprocessing worker #2 started
      zabbix 52478 1.8 2.3 6800012 289504 ? S 14:38 0:07 /usr/sbin/zabbix_server: preprocessing worker #3 started
      zabbix 52479 0.0 2.3 6800012 289592 ? S 14:38 0:00 /usr/sbin/zabbix_server: lld manager #1 [processed 0 LLD rules during 5.016070 sec]
      zabbix 52480 0.0 2.3 6800012 292376 ? S 14:38 0:00 /usr/sbin/zabbix_server: lld worker #1 [processed 1 LLD rules, idle 67.358379 sec during 67.368319 sec]
      zabbix 52481 0.0 2.3 6800012 292376 ? S 14:38 0:00 /usr/sbin/zabbix_server: lld worker #2 [processed 1 LLD rules, idle 17.266869 sec during 17.270454 sec]
      zabbix 54257 0.0 0.0 4628 876 ? S 14:46 0:00 sh -c /usr/sbin/fping -C3 2>&1 </tmp/zabbix_server_52468.pinger;
      root 54262 0.0 0.0 15664 1036 pts/1 S+ 14:46 0:00 grep --color=auto zabbix_server

      I succeded to change debug level. I'll send you the logs when the zabbix server will be down.

      Comment

      • DSI_agglo
        Member
        • Jul 2019
        • 35

        #18
        here is the logs in level 3 of verbosity :
        62668:20190806:160331.833 __mem_malloc: skipped 0 asked 48 skip_min 18446744073709551615 skip_max 0
        62668:20190806:160331.833 [file:dbcache.c,line:3502] __zbx_mem_malloc(): out of memory (requested 48 bytes)
        62668:20190806:160331.833 [file:dbcache.c,line:3502] __zbx_mem_malloc(): please increase HistoryIndexCacheSize configuration parameter
        62668:20190806:160331.833 === memory statistics for history index cache ===
        62668:20190806:160331.833 free chunks of size 32 bytes: 1
        62668:20190806:160331.833 free chunks of size 40 bytes: 1
        62668:20190806:160331.833 min chunk size: 32 bytes
        62668:20190806:160331.833 max chunk size: 40 bytes
        62668:20190806:160331.833 memory of total size 4193912 bytes fragmented into 46362 chunks
        62668:20190806:160331.833 of those, 72 bytes are in 2 free chunks
        62668:20190806:160331.833 of those, 3452064 bytes are in 46360 used chunks
        62668:20190806:160331.833 ================================
        62668:20190806:160331.833 === Backtrace: ===
        62668:20190806:160332.268 14: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](zbx_backtrace+0x4e) [0x564989a02d60]
        62668:20190806:160332.268 13: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](__zbx_mem_malloc+0x180) [0x5649899fe05a]
        62668:20190806:160332.268 12: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](+0x17a94a) [0x5649899c394a]
        62668:20190806:160332.268 11: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](zbx_hashset_insert_ext+0x12f) [0x564989a084ac]
        62668:20190806:160332.268 10: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](zbx_hashset_insert+0x2d) [0x564989a0837b]
        62668:20190806:160332.268 9: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](+0x17ac97) [0x5649899c3c97]
        62668:20190806:160332.268 8: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](+0x17b2ec) [0x5649899c42ec]
        62668:20190806:160332.268 7: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](dc_flush_history+0x49) [0x5649899c38b6]
        62668:20190806:160332.268 6: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](preprocessing_manager_thread+0x499) [0x5649898e465b]
        62668:20190806:160332.268 5: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](zbx_thread_start+0x37) [0x564989a10e66]
        62668:20190806:160332.268 4: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](MAIN_ZABBIX_ENTRY+0xe18) [0x5649898885cd]
        62668:20190806:160332.268 3: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](daemon_start+0x2f0) [0x564989a0249f]
        62668:20190806:160332.268 2: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](main+0x305) [0x56498988779f]
        62668:20190806:160332.268 1: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ff64e27cb97]
        62668:20190806:160332.268 0: /usr/sbin/zabbix_server: preprocessing manager #1 [queued 69, processed 3434 values, idle 5.672253 sec during 5.715457 sec](_start+0x2a) [0x56498988684a]
        62671:20190806:160332.759 cannot read preprocessing service request
        62573:20190806:160332.760 One child process died (PID:62668,exitcode/signal:1). Exiting ...
        62670:20190806:160332.800 cannot read preprocessing service request

        Comment

        • zux
          Member
          • Sep 2018
          • 93

          #19
          You should check this line:
          62668:20190806:160331.833 [file:dbcache.c,line:3502] __zbx_mem_malloc(): please increase HistoryIndexCacheSize configuration parameter

          Comment

          • DSI_agglo
            Member
            • Jul 2019
            • 35

            #20
            Thank you so much zux i changed the HistoryIndexCacheSize the problem is fixed !

            Comment

            Working...