Zabbix 3.4 - Apache with high CPU usage

[lpossamai]

Hi all...

I am having some issues with the Zabbix performance and was wondering if you guys could help.

My setup is pretty simple and small:

• Zabbix Server 3.4 with Mysql (VM)
  • 16GB RAM
  • SSDs
  • CentOS 7.5
  • Xeon X5650 @ 2.67GHz - 2 vCores
  • CacheSize=2G
  • HistoryCacheSize=1G
  • Timeout=4
  • LogSlowQueries=3000
• Zabbix server, Apache, and Mysql are in the same VM.
• Mysql has replication enabled
• Mysql is installed in a dedicated Raid SSD volume
• Apache version is 2.4.6
• Monitoring 34 hosts
• Apache server-status is enabled
• 25 web scenarios
• 4 Bash scripts being used constantly as external check service
• 4k triggers
• NVPS: 51.78
• There is 1 Zabbi Proxy in another country connected via a Site-to-Site VPN to this Zabbix Server

The Zabbix server is reporting CPU spikes (it happens twice, three times a day... it goes back to "normal" by itself), as follow:




On the apache side, I don't see any issues apart from lots and lots of connections. Those connections are from IP's that I know/trust, so it is not spam or an attack.
Using strace I am able to see some weird stuff like (Attention to the "No such file or directory" part)
:

 stat("/usr/share/zabbix/include/classes/export/writers/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/export/elements/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/graphdraw/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/converters/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/importers/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/preprocessors/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/readers/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/import/validators/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/items/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/triggers/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/server/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/screens/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/services/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/sysmaps/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/helpers/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/helpers/trigger/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/macros/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/tree/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/html/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/html/pageheader/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

stat("/usr/share/zabbix/include/classes/html/widget/CWidgetFieldTags.php", 0x7ffd6dcee280) = -1 ENOENT (No such file or directory)

Apache logs seems normal. What could be happening? or this is just Zabbix being busy? I don't think this is normal... otherwise imagine a setup with +8k hosts? not even a supercomputer would handle Zabbix :P

Thanks in advance!

[LenR]

Which 3.4.x version? See https://support.zabbix.com/browse/ZBX-12578, a bug fixed in 3.4.2 maybe.

Do a locate on one of the file names, this may be normal, php chasing paths to find the include file.

Check your server log at times of high CPU and see if housekeeping is running. What is your housekeeping frequency?

If you can catch it happening, try to find what service is using the cpu. You could also create items for the various service cpu usage.

How frequently for how many items and hosts are the external scripts called? There are warnings about the overhead of external scripts. One for each host every 5 minutes is a lot less that 10 for each host every 10 seconds.

[lpossamai]

Which 3.4.x version? See https://support.zabbix.com/browse/ZBX-12578, a bug fixed in 3.4.2 maybe.

Do a locate on one of the file names, this may be normal, php chasing paths to find the include file.

Check your server log at times of high CPU and see if housekeeping is running. What is your housekeeping frequency?

If you can catch it happening, try to find what service is using the cpu. You could also create items for the various service cpu usage.

How frequently for how many items and hosts are the external scripts called? There are warnings about the overhead of external scripts. One for each host every 5 minutes is a lot less that 10 for each host every 10 seconds.

• Zabbix Server zabbix-server-mysql-3.4.10-1.el7.x86_64
• I can find the files but they are in different location.
• Services using it:
  • 26% CPU - monitor.example.com:80 POST /zabbix/jsrpc.php?output=json-rpc HTTP/1.1
  • 23% CPU - monitor.example.com:80 GET /map.php?sysmapid=1&severity_min=0&sid=0c34f2e1903a 461b&cur
  • 13% CPU - monitor.example.com:80 OPTIONS * HTTP/1.0
• The external scripts are being called every 1m for 5 hosts.

You could also create items for the various service cpu usage.

ummm.. how could I do that?

[LenR]

proc.cpu.util[<name>,<user>,<type>,<cmdline>,<mode>,<zone>]

For example but not tested :-)
top or ps will show the proc names

proc.cpu.util[httpd] <-- should be apache on centos
proc.cpu.util[mysql]
proc.cpu.util[zabbix_server]
proc.cpu.util[,elasticsearch] <--- all CPU for user elasticsearch

[lpossamai]

proc.cpu.util[<name>,<user>,<type>,<cmdline>,<mode>,<zone>]

For example but not tested :-)
top or ps will show the proc names

proc.cpu.util[httpd] <-- should be apache on centos
proc.cpu.util[mysql]
proc.cpu.util[zabbix_server]
proc.cpu.util[,elasticsearch] <--- all CPU for user elasticsearch

cool... so from the Documentation; I've put it like this:

proc.cpu.util[httpd,apache,user,httpd]

But all that the graph tells me is what I already knew.... ;(

EDIT:

• Housekeeper process is run every 1h. I will increase this now to every 6h.
• I can see from the graphs that the housekeeper process takes about 20% of CPU each time it runs... so this is not the issue.
• I am running this Zabbix-server on a VM, and have increased the vCPUs from 2 to 6 now.

[LenR]

Apache access logs should show requests during the time. Do you have anything using the Zabbix API?

[lpossamai]

Apache access logs should show requests during the time. Do you have anything using the Zabbix API?

yeah... but the requests are coming from my network... my big screen on the wall, my laptop... nothing weird....

but once I increased the number of vCPUs, I believe it fixed the issue...