Sharded zabbix

[Yello]

Hi,
I'm mulling over trying to build zabbix with a sharded backend. To do this I'm thinking of using the spider engine.

Would anyone have any thoughts/suggestions on the best sharding scheme for zabbix as I've had no exposure to sharding up to now?


Regards,
David

[Yello]

Hi,
Would anyone have any suggestions on tables suitable for sharding? History and trends are obvious ones but are there others?

Regards,
David

[Colttt]

i would like to help you, but i dont understand you, sorry

[Yello]

i would like to help you, but i dont understand you, sorry

Oh sorry. What I'm alluding to is taking large, slow tables and spreading them across multiple remote database instances. A sharding engine (in this case spider) would normalize the view of the remote tables & data so that zabbix doesn't complain.

I see this as a way of partitioning data without hitting the mysql limitation of partitioning tables with foreign keys. What I'm trying to understand is what tables to be most amenable to this kind of treatment. I'll then have a look at setting this up as a test and seeing how it works.


Regards,
David

[Yello]

Some info can be found here:

http://www.percona.com/files/presentations/percona-live/london-2011/PLUK2011-mysql-partitioning-the-spider-solution.pdf

...and elsewhere.


Regards,
David

[Colttt]

ahh ok, now i know what you mean

Partitioning is with mysql not easy because foreign-keys.. you can use postgresql instead.

i cant you help, but do you have performance-problems? maybe you can use SSD or GlusterFS (is "similar" to spider).

the tables are you can use are only the small-tables, bigtables=are often in use

[Yello]

the tables are you can use are only the small-tables, bigtables=are often in use

I don't have performance problems. I'm trying to plan for scalability knowing that performance can be an issue down the line.

It's the bigger, high traffic,tables that inhibit scaling. They're what are most amenable to sharding. I'm trying to understand which of the zabbix 2.0 (or 1.8) tables are good candidates to be treated this way.


Regards,
David

[rsvancara]

Sharding

Sharding a database is a good strategy, if:

1. Your application can handle it natively
2. You have some kind of middleware that can handle it on behalf of the application.

I think you might see better performance using something like pgpool and postgresql where PGPool would distribute your queries amongst multiple database backends. Anyone have experience with this?

[Yello]

I think you might see better performance using something like pgpool and postgresql where PGPool would distribute your queries amongst multiple database backends. Anyone have experience with this?

As I understand it, this is partly what the spider engine does. A table, for example history is "replaced" by a spider engine history table which proxies reads/writes to a range of mysql backends holding the real table & data according to its configured sharding strategy.

So when zabbix thinks it is writing to or reading from history is actually having those requests sharded. Should be transparent. I say "should be" because I want to test it but am looking to understand the sweet spot of tables that might be amenable to this approach.


Regards,
David