Problems with New Installation

[seanfulton]

I've used Zabbix 4.0 for years so I'm not a novice with this. There are a number of templates we want to use that are not supported so we decided to start using Zabbix 7.0. Rather than upgrade, we built a brand new host with Alma9, PHP, MySQL, etc. and installed fresh. Twice. Both times Zabbix turns completely unreliable. I have maybe 20 hosts configured, and there are constant gaps in data collection. I made changes to five of our kafka hosts and re-enabled the JMX interface--I saw the boxes on Zabbix turn green that it was seeing the JMX connection--but no data. For hours. I've read the forums and it's not a connectivity issue or an issue with the device being tracked.

In my case, Zabbix can't even reliably collect data from itself!


I've spent about 4 days on this and still can't get it to reliably collect data.

Any idea how to get this working or where to look? I've seen numerous complaints from other people over the years with similar issues, but no resolution or clues on how to resolve. I've tuned and tuned and am not finding anything to fix. My old install NEVER had problems like this.

server.conf:

StartPollers=100
StartPollersUnreachable=50
StartPingers=50
StartTrappers=50
StartDiscoverers=10
StartHTTPPollers=10
#CacheSize=128M
#HistoryCacheSize=64M
#HistoryIndexCacheSize=32M
#TrendCacheSize=32M
#ValueCacheSize=256M
StartJavaPollers=15
HousekeepingFrequency=4
MaxHousekeeperDelete= 5000
StartDBSyncers=10

The caches above are all way bigger now--1G - 2G. Server has 8 cores, 32G of RAM, only 9G in use and system load is about 3.07.


sean

[cyber]

you have just 20 hosts to monitor and you start up hundreds of different pollers/trappers? Pollers in 4 and 7 are much different.. There are types that did not exist in v4... 100 pollers? You probably need one (1) agent poller instead.. plus couple of pollers, if that server is really polling something
You should start with reducing the amount of processes and see if it does better.

10 db syncers?

StartDBSyncers


The number of pre-forked instances of history syncers.
Note: Be careful when changing this value, increasing it may do more harm than good. Roughly, the default value should be enough to handle up to 4000 NVPS.

Default: 4
Range: 1-100

[seanfulton]

I started with the default and tweaked when it went south. I can put it back to 4. But where did it come with 471K items shown on the dashboard page? I have more than 100 hosts that were configured for my old Zabbix, I did not configure them in the new version however in the logs I see them polling the Zabbix server. They are not set up.

[seanfulton]

#
# Mandatory: no
# Range: 0-1000
# Default:
# StartHistoryPollers=5
StartHistoryPollers=100


Yet I see:

Utilization of history syncer processes over 75%

More than 75% used in the history cache
And I have:
### Option: HistoryCacheSize
# Size of history cache, in bytes.
# Shared memory size for storing history data.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# HistoryCacheSize=16M
HistoryCacheSize=2G

### Option: HistoryIndexCacheSize
# Size of history index cache, in bytes.
# Shared memory size for indexing history cache.
#
# Mandatory: no
# Range: 128K-2G
# Default:
# HistoryIndexCacheSize=4M
HistoryIndexCacheSize=2G

Utilization of preprocessing manager processes over 75%

No idea what this impacts.

Also the front end is VERY slow pulling up my list of hosts now.

All data collection stopped at roughly 4 am today for no reason.

[tim.mooney]

But where did it come with 471K items shown on the dashboard page? I have more than 100 hosts that were configured for my old Zabbix, I did not configure them in the new version however in the logs I see them polling the Zabbix server. They are not set up.

I don't understand your install at all. You only have 24 hosts but your system's NVPS is 8009?

Do any of the templates you applied do auto-discovery / low-level discovery, and if so is it possible that the template "discovered" hundreds of thousands of items for one or more of your hosts?

If you look at your Hosts view, where it lists counts for items, triggers, etc. for each host, do some of them have a really large number of items?

[cyber]

yep.. that looks strange... If GUI really does not respond and server dies... you can set all hosts to disabled state directly in DB, so at least on startup it should not start polling anything and you can at least investigate the issue...

[jhboricua]

I have more than 100 hosts that were configured for my old Zabbix, I did not configure them in the new version however in the logs I see them polling the Zabbix server. They are not set up.

Are you re-using the IP or DNS name of the old Zabbix server for the new one? That would explain why the agents are trying to connect to the new server.

[seanfulton]

Yes, we are re-using the server name.