Disk IO statistics broken in Linux

[Netbulae]

Hi,

We are trying to monitor the disk IO stats in Centos 7 with zabbix 2.4, but I cannot get it to work.

When trying the build in vfs.dev.read and vfs.dev.write, only the write returns any values

zabbix_get -s hostname -k vfs.dev.write[sda,ops,avg1]
0.000000

zabbix_get -s hostname -k vfs.dev.write[sda,ops,avg1]
2.933333

Also the built in discovery prototype has "vfs.dev.read[{#FSNAME},ops,avg1]" but $FSNAME doesn't work because it's a device (https://support.zabbix.com/browse/ZBXNEXT-1302).

And the "Type of information" is set to "Numeric (unsigned)" while it should be "Numeric (float)" for sps/ops.

But even when I modify the item, only the write statistics work, not the read.

The workaround posted here:

http://www.muck.net/19/getting-hard-disk-performance-stats-from-zabbix

Also doesn't give the proper results.

Any tips on how to get these statistics??

[kloczek]

Any tips on how to get these statistics??

Yep. Stop using avg1 type of metrics and start start reading IO counters to store "delta per second" vales.
If you will be reading one time per 1min IO counters and storing these values as "delta per second" you will have exactly the same vales. Problem is that wit reading avg1 metrics you will be not able change sampling rate without making historic data useless or recalculating them.

Really .. try first use first Linux templates from zabbix instead preparing your own.

[Netbulae]

Yep. Stop using avg1 type of metrics and start start reading IO counters to store "delta per second" vales.
If you will be reading one time per 1min IO counters and storing these values as "delta per second" you will have exactly the same vales. Problem is that wit reading avg1 metrics you will be not able change sampling rate without making historic data useless or recalculating them.

After a lot of googling, I found this post:

http://www.denniskanbier.nl/blog/monitoring/monitoring-disk-io-using-zabbix/

That describes the same..
.

Really .. try first use first Linux templates from zabbix instead preparing your own.

Well that's what I tried.... But it doesn't work.

I now thought maybe my template is outdated as I upgraded a couple of times.

So I downloaded the latest known on:

http://www.zabbix.org/wiki/Zabbix_Templates/Official_Templates

But this templete doesn't contain a vfs.dev.read/vfs.dev.write or discovery rule at all....

[tchjts1]

About 3/4 of the way down this page explains the usage for vfs.dev.read and write on the various OS's for Zabbix 2.4.x:

https://www.zabbix.com/documentation/2.4/manual/config/items/itemtypes/zabbix_agent

[Netbulae]

About 3/4 of the way down this page explains the usage for vfs.dev.read and write on the various OS's for Zabbix 2.4.x:
https://www.zabbix.com/documentation...s/zabbix_agent

Yes I know, but that info appears incorrect:

mode - possible values:
avg1 (one-minute average, default), avg5 (five-minute average), avg15 (15-minute average).
Note: The third parameter is supported only if the type is in: sps, ops, bps.

The poster above tell's me to use Delta's instead of avg1:

Yep. Stop using avg1 type of metrics and start start reading IO counters to store "delta per second" vales.
If you will be reading one time per 1min IO counters and storing these values as "delta per second" you will have exactly the same vales. Problem is that wit reading avg1 metrics you will be not able change sampling rate without making historic data useless or recalculating them.

The built in discovery rules I had for vfs.dev.read are not in the current template or maybe have been added a couple of years ago. So I'll built my own based on this info.

I think it would be nice if zabbix would support this out of the box....

[kloczek]

The built in discovery rules I had for vfs.dev.read are not in the current template or maybe have been added a couple of years ago. So I'll built my own based on this info.

I think it would be nice if zabbix would support this out of the box....

Seems you completely missed my point.

[Netbulae]

Seems you completely missed my point.

It appears so.... You talk about "Template OS Linux" when you say the default linux template with zabbix?

My co-worker added some disk io experimentation items and discovery rules to our "Template OS Linux" so that got me confused.

But there are as far as I see no disk IO items in the default zabbix templates, if there are can you point me to them? I could be wrong.

I built some items with values as "delta per second" now but had to change ops into operations as it doesn't make sense to store ops as "delta per second"

[kloczek]

It appears so.... You talk about "Template OS Linux" when you say the default linux template with zabbix?

No. You don't understand why you should be not using avg1/avg5/avg15 metrics when you have access to raw counters.
There is no items using avg1/avg5/avg15 keys (for IOs) in default template and seems you don't understand why it is.

[Netbulae]

No. You don't understand why you should be not using avg1/avg5/avg15 metrics when you have access to raw counters.
There is no items using avg1/avg5/avg15 keys (for IOs) in default template and seems you don't understand why it is.

Ahh, sorry. Now I understand.

I got rid of avg1/avg5/avg15 after your first comment so I DON'T use it anymore.

But it's a hassle that the items are not in the default template.

So I added some ITEMS for vfs.dev.read/write with the DELTA as you suggested.

Also had to do sector to MB/s conversion and will have to write DISCOVERY rules...

That's what I meant with it would have been nice to include it with the default template.....

As it's open source, I'll submit my template modifications back to the community so it will be in the default

[kloczek]

That's what I meant with it would have been nice to include it with the default template.....

IMO zabbix after initial installation should be without templates at all.
And/or templates with value mappings and regexps should be delivered as separated sql dump.
Another solution is fixing xml import/export mappings and regexps and delivery initial set of templates as xml.

Problem with initial set of templates delivered in form of sql dump is hat each hostid/itemid/etc is hardcoded and may overlap with definition already existing in zabbix setup so on release new set of templates importing them as sql dump is asking for troubles. This is causing that relaying on dist templates is very limited.