Ad Widget

Collapse

LLD Worker Processes Constantly Running Hot on CPU? Is this Normal?

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • alexw-z
    Member
    • Dec 2021
    • 36

    #1

    LLD Worker Processes Constantly Running Hot on CPU? Is this Normal?

    In the course of investigating both a sudden increase in memory usage and a "LLD Poller over 75% Utilisation" alert that popped up on my Zabbix 6.0 server (which I don't necessarily think are directly related, judging by timeframe), I noticed that the LLD processes seem to run unusually hot on CPU, they're pretty much using an entire core each near constantly, and this persists even after adding a couple more pollers to clear the 75% utilisation alert. Most poller types don't run anywhere near this hot on resource, so I'm not sure fi this is normal or if there's an issue or misconfiguration somewhere I need to try and troubleshoot. There doesn't seem to be a lot in the Zabbix server logs that references LLD, even with debug levels turned up that might point to a problem.

    Code:
    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    1694793 zabbix 20 0 4737756 119052 41384 R 99.7 0.0 1978:09 zabbix_server
    1694790 zabbix 20 0 4752620 132400 40116 R 99.3 0.1 7821:00 zabbix_server
    1694792 zabbix 20 0 4755760 139364 43668 R 81.8 0.1 5534:13 zabbix_server
    Nobody with access to the server has come forward when asked about adding any new templates or batches of hosts etc.

    Is anybody able to advise whether this is normal behavior or not, and how I might investigate further if this is not expected?

    Many thanks.
  • MRedbourne
    Senior Member
    • Feb 2023
    • 103

    #2

    Which version of 6.0 LTS specifically? There were some breaking changes in Zabbix 7.2.8 and 7.0.14 that are right up your alley.

    [ZBX-26537] Large number of redundant allocations in preprocessing manager per request - ZABBIX SUPPORT
    [ZBX-26588] Zabbix 7.0.14 preprocessing manager overload - ZABBIX SUPPORT

    If you want to see if anyone's been up to any funny business in Zabbix...

    Dashboard -> Reports -> Audit Log. Set your resource filter to the following (one by one) and look for any substantial changes:
    • Template (Actions: Add, Update)
    • Host (Actions: Add, Update)
    • ​LLD Rule (Actions: Add, Update)
    • Discovery RUle (Actions (Add, Update)

    Comment

    • alexw-z
      Member
      • Dec 2021
      • 36

      #3
      Hey MRedbourne, thank for the reply.

      I'm running 6.0.40-release1.el8 on an Alma 8 box, latest version. I don't think I see any reference in those links as to whether the bug affects Zabbix 6.0,but I have also just noted it's sneaked into EOL, so I guess that could be why. I was planning on making the jumping up to 7.0LTS as soon as I can make the time for it, hopefully in the next month or two.

      Thanks for the Auditing info as well, for some reason I had it in my head that auditing template changes wasn't possible through the GUI and I'd need to go digging with SQL queries, but that's clearly a load of crap. The page to review the audit logs fell completely off my radar some time ago - maybe partially because I thought all Auditing options were held in Admin > Audit logs, and that just has a handful of bare bones retention settings, I never really have a need to/go near the Reports section where that page lives

      Comment

      Working...