Ad Widget

Collapse

zabbix-server takes a long time to stop due to syncing trends data

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

    zabbix-server takes a long time to stop due to syncing trends data

    The zabbix-server process on our master server takes about 4 minutes to fully shut down. in zabbix-server.log I can see that most of this time is spent on operations for syncing trend data. Has anyone seen a similar shutdown time and have you done anything to reduce that? I've run a strace on the main zabbix-server process during that time and I mostly see select and insert DB operations into the trends_uint table.
    A quick summary of our environment: 8 proxy servers across multiple data centers feed into a single master, which connects to a postgres database. We're monitoring about 7000 hosts with NVPS around 3000. We keep 1 year of trend data. In zabbix-server.conf I've set TrendCacheSize=256M. I've looked into the trend write cache graphs and I can see it's pretty stable in using about 150MB of the cache (% free is usually around 45%). At one point we had a higher value for TrendCacheSize and I noticed the time it takes to sync the trends data also increases (and thus shutdown time also was longer) with a larger value. (At one point during testing someone had set to to 1GB and it took 12 minutes to sync the trend cache, even though usage was still at around 150MB.) So in theory I have some wiggle room to reduce the value further but I want to keep at least 30% free. Ideally I'd like the process to stop pretty quickly, i.e. within 15 seconds. But with an environment at our scale I'm not sure what would be a reasonable shutdown time for zabbix-server. But 4 minutes still seems excessive. I appreciate any feedback.

    #2
    the server does really shuts down in some computers but i don't this is universal problem. mathwayIts best recommended to consult a professional .


    Last edited by violetevergarden; 01-05-2019, 23:43.

    Comment


      #3
      I have a similar issue, mine started after the upgrade from 3.2.11 to 4.2.1 we started having same issue, it does stop but takes up to 2 hours some times. I have played multiple times with he history cache as I see in the log it usually shows syncing history. I have not found a fix for it yet
      4 Zabbix Frontend Servers (Load balanced)
      2 Zabbix App Servers (HA)
      2 Zabbix Database Servers (HA)
      18 Zabbix Proxy Servers (HA)
      3897 Deployed Zabbix Agents
      6161 Values per second
      X-Layer Integration
      Jaspersoft report Servers (HA)

      Comment


      • iamnotarobot
        iamnotarobot commented
        Editing a comment
        I had this problem and it was due to table/index fragmentation in the MySQL database. Your root issue might be different, but running an index rebuild fixed my problem and the server now takes a few seconds to stop/start again. You can do it both from inside the MySQL server or via the mysqlcheck command line tool. You can also check your Zabbix server logs to see if there are any long running queries being logged and then run traces on those queries.

    Announcement

    Collapse
    No announcement yet.
    Working...
    X