Is it large or not and what to do best?

[Gutsycat]

Hello.

My current instance is:
2 Xeon
64Gb RAM
15Tb SAS disks

One server with zabbix and mysql database 1.8Tb on it.

Also I have ~1k of hosts and 50-60k items

I'm stuck with problem.

When I start mysqlcheck -A or mariabackup --backup --incremental-basedir=... after some time zabbix server failed.

Is it time to plan another architecture?
Does 1.8Tb of mysql data needs to be clustered?

[Mike2K]

How many NVPS do you have ? and do you ever cleanup old data ? whats the history and trend settings of your checks ?

[Gutsycat]

How many NVPS do you have ? and do you ever cleanup old data ? whats the history and trend settings of your checks ?

I have about 300 nvps

[Mike2K]

That's not large...can you show us the zabbix performance graph ? Does the zabbix log show slow queries ?

[Gutsycat]

That's not large...can you show us the zabbix performance graph ?

Does the zabbix log show slow queries ?

Zabbix log no but mariadb does.

and do you ever cleanup old data ?

No we CAN NOT do such thing.

[DmitryL]

Add --single-transaction to your mysqldump command.
But even better, choose different backup tools.

[Mike2K]

No we CAN NOT do such thing.

Can you explain why you can't cleanup old data? Why would you need to keep everything?

[Gutsycat]

Can you explain why you can't cleanup old data? Why would you need to keep everything?

Yes, I can. Our business process expects to store data for 5 years, now only 2 years have passed for now.

[Jason]

Yes, I can. Our business process expects to store data for 5 years, now only 2 years have passed for now.

Then why not export the old data out from zabbix into elasticdb? That's built in to version 4.0 onwards.
Do you really need to keep all data or just trends? If just the trends then that's not too bad, but all of the data is going to be massive and I'd argue not necessary unless you really really need to know what the state was at every moment in time for 5 years...

[LenR]

There should be an error message at the time of the crash, without it, we're just guessing.

Did you partition your database? Replace housekeeper with partition management as recommended elsewhere.

Did you tune mysql? Things like large pages and innodb_buffer_pool.

[Gutsycat]

Maybe run replica DB? You can always do a backup of replica without worrying about production DB.

Yes, we finally did it. But I think that it is kind of workaround for our deployment.

There should be an error message at the time of the crash, without it, we're just guessing.

No any message... Zabbix just shows yellow balloon at the bottom and nothing in the log.

Did you partition your database? Replace housekeeper with partition management as recommended elsewhere.

Yes we are.

Did you tune mysql? Things like large pages and innodb_buffer_pool.

Sure.

Could the problem being with custom external check program which starts 1000 times and reads many of config files?

[bbrendon]

Gutsycat, Dmitryb already gave you the answer. You should listen instead of chasing your tail.