Another "I have gaps in my graphs" thread

[dvb]

First of all, I am running 1.8.4 on CentOS 5.6 x86_64 with Apache 2.2.3, Mysql 5.0.77 and PHP 5.1.6. It is mostly a "stock" config.

As others have noticed, I am seeing gaps in graphs that do not match up with any absence of data. So, I did some testing and thought I would share my findings. My apologies for not filing a formal bug report. I will do that when I have some time to review bug reporting protocol.

In the attached images, you can see a graph with gaps, a graph without gaps (same graph - I will explain below), some raw values for that period, and the item config feeding the part of the graph that's showing up with gaps.

When I configured the graph to a fixed Y axis maximum of 100000000 (100Mbps, even though the link is 1Gbps) rather than 'Calculated', the gaps went away.

When I configured a fixed maximum of 20000000 on the graph, I could perfectly reproduce the gap since the collected values from 9:29 a.m. onward are upwards of 20000000.

I then tried various values above 20000000. 21000000, 22000000, etc. I still had gaps until I got to 25000000. At 25000000, the gap was gone.

So, in non-programmer terms, the problem would appear to have something to do with what Zabbix *thinks* it is dealing with in terms of a maximum Y axis value when using 'Calculated' for the max.

Obviously, I could set the maximum to wire speed (in this case, 1Gbps), but this would result in a poor graph for a link that is so lightly utilized at the current time. I could also set some other maximum, such as 100Mbps, but that's a lot of work if you have a lot of graphs.

I hope this helps track down the problem. I will make an effort to get this info posted as a bug when I can.

[jansonz]

I have similar problem.
I'm running Zabbix 1.8.4
For me it's not some specific graph which has this problem, for me it happens on all the graphs at the same time (See attach. example.)

Has anybody experienced the same?

During that time I can see a lot of these messages in the debug log:

574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()
1574:20110616:090227.013 In zbx_hashset_search()
1574:20110616:090227.013 End of zbx_hashset_search()

Maybe this points to something as well:

1576:20110616:090228.531 In DCvacuum_text()
1576:20110616:090228.532 End of DCvacuum_text()
1576:20110616:090228.532 History text buffer is full. Sleeping for 1 second.
1592:20110616:090228.532 In DCvacuum_text()
1592:20110616:090228.532 End of DCvacuum_text()
1592:20110616:090228.532 History text buffer is full. Sleeping for 1 second.
1589:20110616:090228.532 In DCvacuum_text()
1589:20110616:090228.532 End of DCvacuum_text()
1589:20110616:090228.532 History text buffer is full. Sleeping for 1 second.
1596:20110616:090228.532 In DCvacuum_text()
1596:20110616:090228.532 End of DCvacuum_text()
1596:20110616:090228.532 History text buffer is full. Sleeping for 1 second.
1611:20110616:090228.581 End of process_escalations()


During that time I can see that the Zabbix item queue is growing quite high without any particular reason.

Sometimes it goes away by itself and sometimes I have to restart the zabbix_server process.

Some details about my Zabbix server:

Number of hosts: 186
Number of items: 6909
Number of triggers: 2543
Required server performance, new values per second: 111.65

Server HW:

4x Intel(R) Xeon(R) CPU E5520 @ 2.27GHz quad core
RAM: 16 GB
scsi0 : LSI SAS based MegaRAID driver
Vendor: SEAGATE Model: ST9300603SS Rev: FS62

As you can see the machine is quite powerful.

Does anybody have any ideas?

Thanks,
Janis

[jansonz]

Could it be this?

https://support.zabbix.com/browse/ZBX-3603

[dvb]

Different issue, I think. All of my history is intact. My issue has more to do with the graph simply not being drawn if the data exceeds a particular value. The data is all there, but where the data crosses that boundary (which seems to vary by graph), those parts of the graph end up blank, not just cut off at the top, as one might expect. All I have to do to get the gaps to go away is set a MAX Y axis value that is higher than the max value collected for that period. So for now, I'm working around the issue by setting an arbitrary maximum. For example, on a lightly loaded gigabit link, I might set the max to 100 megs so that the gaps are not present but the colored regions are not lost to scale.

[jansonz]

Could be.

I managed to get the gap problem fixed by disabling housekeeping. Since then the queue has been steady and Zabbix works great.

Not sure if this is the best fix, but at least it's working.

[Jason]

Theres a lot of posts on disabling housekeeping on larger installs. If you do disable that then you'll need to do the maintenance manually on the databast to stop it growing indefinitely. There are a few posts around the place on which tables you need to prune.

I run this maintenance on a weekly basis along with an optimisation of the tables (done in the middle of the night).

Jason