Zabbix server stop after around of 12h of working

[TheY]

Hello community,
Hope i will see solution for my issue.
We are facing issue with our zabbix server since one week.
Server deamon stop working after around of 10hours of working with log message below:

#####Log message part#####

39025:20190810:163620.713 server #31 started [trapper #4]
39029:20190810:163620.714 server #35 started [preprocessing manager #1]
39026:20190810:163620.715 server #32 started [trapper #5]
38975:20190810:163620.954 Got signal [signal:15(SIGTERM),sender_pid:39069,sender_uid:0,r eason:0]. Exiting ...
38975:20190810:163622.957 syncing history data...
38975:20190810:163622.957 syncing history data done
38975:20190810:163622.957 syncing trend data...
38975:20190810:163622.957 syncing trend data done
38975:20190810:163622.957 Zabbix Server stopped. Zabbix 3.4.15 (revision 86739).

####End log message part####

I try many actions to fix issue but i have same result.
My DebugLevel is 3

I set:many values without effective changes:
-VMwareCacheSize=512M
-HousekeepingFrequency=8
-MaxHousekeeperDelete=700000
-CacheSize=6G
-StartDBSyncers=4
-HistoryCacheSize=2G
-HistoryIndexCacheSize=2G
-TrendCacheSize=2G
-ValueCacheSize=4G

My environnement:
-Zabbix Server is 3.4.15 version on Red Hat 7.3
Around of 350 host with 50 templates and around of 7000 items.

Any one can help me to fix thix embarassing issue?

Regards.

[kloczek]

Look on top of the https://www.zabbix.com/documentation.../api/reference
Zabbix 3.4 is no longer supported.

[TheY]

Hello,
Thanks for reply, what do i need to do?
Upgrade or what?
Regards.

[zux]

That's first and second:

38975:20190810:163620.954 Got signal [signal:15(SIGTERM),sender_pid:39069,sender_uid:0,r eason:0]. Exiting ...

Someone killed it.

[andris]

Log file shows it was killed immediately after start, not after 12h. Who did it? sender_pid:39069.sender_uid:0 points to 'root' privileges. Maybe systemd logs show something?

[TheY]

Hello dear,
After restart it is effectivly stoped, but when i delete database, and restore former BD, it works for max 24hours.
Regards.

[andris]

Is something in database logs? Maybe Zabbix temporarily stops working because of DB problem and systemd thinks that Zabbix server hangs and restarts it?

[TheY]

Hello dear,
I will check mariadb log and share with you.
Thanks .

[TheY]

Hello,
MaraiDB don't send very helpful log message,
Bellow extraction of message log since 10/09/2019

#####Log start####


190810 18:48:52 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
190810 20:10:35 InnoDB: ERROR: the age of the last checkpoint is 9433818,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
190811 15:55:15 [Note] /usr/libexec/mysqld: Normal shutdown
190811 15:55:15 [Note] Event Scheduler: Purging the queue. 0 events
190811 15:55:16 InnoDB: Starting shutdown...
190811 15:55:16 InnoDB: Waiting for 200 pages to be flushed
190811 15:55:19 InnoDB: Shutdown completed; log sequence number 2250722568748
190811 15:55:19 [Note] /usr/libexec/mysqld: Shutdown complete

190811 15:55:19 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
190811 15:56:40 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190811 15:56:41 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 1559 ...
190811 15:56:41 InnoDB: The InnoDB memory heap is disabled
190811 15:56:41 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190811 15:56:41 InnoDB: Compressed tables use zlib 1.2.7
190811 15:56:41 InnoDB: Using Linux native AIO
190811 15:56:41 InnoDB: Initializing buffer pool, size = 128.0M
190811 15:56:41 InnoDB: Completed initialization of buffer pool
190811 15:56:41 InnoDB: highest supported file format is Barracuda.
190811 15:56:41 InnoDB: Waiting for the background threads to start
190811 15:56:42 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 2250722568748
190811 15:56:42 [Note] Plugin 'FEEDBACK' is disabled.
190811 15:56:42 [Note] Server socket created on IP: '0.0.0.0'.
190811 15:56:42 [Note] Event Scheduler: Loaded 0 events
190811 15:56:42 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
190812 12:29:20 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190812 12:29:21 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 1653 ...
190812 12:29:21 InnoDB: The InnoDB memory heap is disabled
190812 12:29:21 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190812 12:29:21 InnoDB: Compressed tables use zlib 1.2.7
190812 12:29:21 InnoDB: Using Linux native AIO
190812 12:29:21 InnoDB: Initializing buffer pool, size = 128.0M
190812 12:29:21 InnoDB: Completed initialization of buffer pool
190812 12:29:21 InnoDB: highest supported file format is Barracuda.
190812 12:29:22 InnoDB: Waiting for the background threads to start
190812 12:29:23 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 2258839004375
190812 12:29:23 [Note] Plugin 'FEEDBACK' is disabled.
190812 12:29:23 [Note] Server socket created on IP: '0.0.0.0'.
190812 12:29:23 [Note] Event Scheduler: Loaded 0 events
190812 12:29:23 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
190812 12:54:26 [Note] /usr/libexec/mysqld: Normal shutdown
190812 12:54:26 [Note] Event Scheduler: Purging the queue. 0 events
190812 12:54:26 InnoDB: Starting shutdown...
190812 12:54:27 InnoDB: Waiting for 6 pages to be flushed
190812 12:54:27 InnoDB: Shutdown completed; log sequence number 2258841323391
190812 12:54:27 [Note] /usr/libexec/mysqld: Shutdown complete

190812 12:54:28 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
190812 12:57:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190812 12:57:03 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 1588 ...
190812 12:57:04 InnoDB: The InnoDB memory heap is disabled
190812 12:57:04 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190812 12:57:04 InnoDB: Compressed tables use zlib 1.2.7
190812 12:57:04 InnoDB: Using Linux native AIO
190812 12:57:04 InnoDB: Initializing buffer pool, size = 128.0M
190812 12:57:04 InnoDB: Completed initialization of buffer pool
190812 12:57:04 InnoDB: highest supported file format is Barracuda.
190812 12:57:04 InnoDB: Waiting for the background threads to start
190812 12:57:05 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 2258841323391
190812 12:57:05 [Note] Plugin 'FEEDBACK' is disabled.
190812 12:57:05 [Note] Server socket created on IP: '0.0.0.0'.
190812 12:57:05 [Note] Event Scheduler: Loaded 0 events
190812 12:57:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
190812 13:09:24 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190812 13:09:24 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 1547 ...
190812 13:09:24 InnoDB: The InnoDB memory heap is disabled
190812 13:09:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190812 13:09:24 InnoDB: Compressed tables use zlib 1.2.7
190812 13:09:24 InnoDB: Using Linux native AIO
190812 13:09:25 InnoDB: Initializing buffer pool, size = 128.0M
190812 13:09:25 InnoDB: Completed initialization of buffer pool
190812 13:09:25 InnoDB: highest supported file format is Barracuda.
190812 13:09:25 InnoDB: Waiting for the background threads to start
190812 13:09:26 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 2258842304941
190812 13:09:26 [Note] Plugin 'FEEDBACK' is disabled.
190812 13:09:26 [Note] Server socket created on IP: '0.0.0.0'.
190812 13:09:26 [Note] Event Scheduler: Loaded 0 events
190812 13:09:26 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
190812 20:49:16 [Note] /usr/libexec/mysqld: Normal shutdown
190812 20:49:16 [Note] Event Scheduler: Purging the queue. 0 events
190812 20:49:16 InnoDB: Starting shutdown...
190812 20:49:21 InnoDB: Shutdown completed; log sequence number 2327495586669
190812 20:49:21 [Note] /usr/libexec/mysqld: Shutdown complete

190812 20:49:21 mysqld_safe mysqld from pid file /var/run/mariadb/mariadb.pid ended
190812 20:50:03 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
190812 20:50:03 [Note] /usr/libexec/mysqld (mysqld 5.5.60-MariaDB) starting as process 1597 ...
190812 20:50:04 InnoDB: The InnoDB memory heap is disabled
190812 20:50:04 InnoDB: Mutexes and rw_locks use GCC atomic builtins
190812 20:50:04 InnoDB: Compressed tables use zlib 1.2.7
190812 20:50:04 InnoDB: Using Linux native AIO
190812 20:50:04 InnoDB: Initializing buffer pool, size = 128.0M
190812 20:50:04 InnoDB: Completed initialization of buffer pool
190812 20:50:04 InnoDB: highest supported file format is Barracuda.
190812 20:50:04 InnoDB: Waiting for the background threads to start
190812 20:50:05 Percona XtraDB (http://www.percona.com) 5.5.59-MariaDB-38.11 started; log sequence number 2327495586669
190812 20:50:05 [Note] Plugin 'FEEDBACK' is disabled.
190812 20:50:05 [Note] Server socket created on IP: '0.0.0.0'.
190812 20:50:05 [Note] Event Scheduler: Loaded 0 events
190812 20:50:05 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.60-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server


#####Log End#####

Thanks

[TheY]

Hello,

Nobody to help me with this issue?

[zux]

grep zabbix /varlog/messages
or just look at that file. maybe OOM killer is stopping it.

[TheY]

Thanks,
This is result of the command:


####

Aug 14 11:16:07 rpm systemd: zabbix-server.service: Supervising process 29497 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:10 rpm systemd: zabbix-server.service: Supervising process 29604 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:13 rpm systemd: zabbix-server.service: Supervising process 29709 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:16 rpm systemd: zabbix-server.service: Supervising process 29817 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:18 rpm systemd: zabbix-server.service: Supervising process 29920 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:21 rpm systemd: zabbix-server.service: Supervising process 30035 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:24 rpm systemd: zabbix-server.service: Supervising process 30142 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:27 rpm systemd: zabbix-server.service: Supervising process 30249 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:29 rpm systemd: zabbix-server.service: Supervising process 30352 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:32 rpm systemd: zabbix-server.service: Supervising process 30457 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:35 rpm systemd: zabbix-server.service: Supervising process 30566 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:38 rpm systemd: zabbix-server.service: Supervising process 30669 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:41 rpm systemd: zabbix-server.service: Supervising process 30777 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:43 rpm systemd: zabbix-server.service: Supervising process 30879 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:46 rpm systemd: zabbix-server.service: Supervising process 30984 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:49 rpm systemd: zabbix-server.service: Supervising process 31091 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:52 rpm systemd: zabbix-server.service: Supervising process 31194 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:55 rpm systemd: zabbix-server.service: Supervising process 31303 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:16:57 rpm systemd: zabbix-server.service: Supervising process 31411 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:17:00 rpm systemd: zabbix-server.service: Supervising process 31517 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:17:03 rpm systemd: zabbix-server.service: Supervising process 31622 which is not our child. We'll most likely not notice when it exits.
Aug 14 11:21:54 rpm systemd: zabbix-server.service: Supervising process 32046 which is not our child. We'll most likely not notice when it exits.
Aug 14 16:34:05 rpm systemd: zabbix-server.service: Supervising process 1311 which is not our child. We'll most likely not notice when it exits.

####

[the.monitor]

There are a number of things that can cause Zabbix server to stop working.

Configuration Cache = if this runs out of space - zabbix server wont start (or will stop)

Check your server.conf and see if this can be tuned.

Using the Internal key "zabbix[rcache,buffer,pfree]" to see how much is remaining.

This is one suggestion that I don't see listed here so it may be worth checking.

Otherwise, please alter your DebugLevel in server.conf to 4 or 5 and review the more detailed logs after the next time it dies.

[lashialp]

After restart it is effectivly stoped, but when i delete database, and restore former BD, it works for max 24hours.