Zabbix 4.0.7 - History syncers slow when many events arrive

[[email protected]]

Environment:
Zabbix - 4.0.7
Hosts - 13035
Items - 3431108
Triggers - 387768
NVPS - 7610

I have noticed the history syncers get busy when lots of events arrive and last time this was 2700 in 10 minutes due to a large environment change being performed (including internal events ie item not supported).

There is many log entries with this entry in there - slow query: 11.164569 sec, "update ids set nextid=nextid+1 where table_name='event_suppress' and field_name='event_suppressid'".`

The behavior is that the history syncers are slow and the cache starts to build up until the above log entries disappear OR lots of hosts come out on their maintenance period (and we have a lot of hosts in maintenance after hours).

The issue is that if the history syncing gets far enough behind other nodata() triggers start to fire causing additional load and compounds the issue.

This never occured in Zabbix 3.4 and would like to know if anyone else has seen this situation

For those who wish to follow / watch the Zabbix issue: https://support.zabbix.com/browse/ZBX-16130?filter=-2

[[email protected]]

Addition:

In Zabbix 4.0 the some of the trigger evaluation was shifted from the timers to the history syncers, which is why the history syncers were never had this issue with syncing data.

We have a large environment that has thousands of servers that are placed into maintenance at night, in which we only see this issue at night.

Has anyone experienced this behavior?

Temporarily we have created triggers on our server and proxies that fire when syncing data is delayed and used trigger dependencies on the nodata() triggers to prevent them from firing until the history syncing has caught up.

I am sure there must be reasons for Zabbix to utilise the 'ids' table to determine the NextId, however I am wandering if these type of issues would still exist (or frontend update / create performance increased) if this was handled using a native database auto-increment field?

Cheers
James

[kloczek]

Environment:
Zabbix - 4.0.7
Hosts - 13035
Items - 3431108
Triggers - 387768
NVPS - 7610

I have noticed the history syncers get busy when lots of events arrive and last time this was 2700 in 10 minutes due to a large environment change being performed (including internal events ie item not supported).

Because you database backend is to slow.

[[email protected]]

Hi Kloczek,

Our database server is a 48 core,256GB ram on an EMC ExtremeIO SAN (SSD), experiencing 90% idle time and no disk io wait.

I believe the issue may be a result of shifting some of the trigger evaluation from the timers to the history syncers and now having to factor in an exclusive lock on the ids table while processing incoming events, effectively removing the parallelism of the history syncers as they all have to wait to get the exclusive lock; Hence the slow query being reported as the history syncers cant update the field because of the lock.

This never happened before as the history syncers never utilised the ids table as the history* and trends* tables dont have an id field.

I will reopen the issue https://support.zabbix.com/browse/ZBX-16130 and put a comment in to this effect.

Cheers
James