since update to 2.2.9 zabbix need more ressources and get more false positives alarms

[Colttt]

Hello,
last Thunday(02.04.2015) i upgraded from 2.2.6 to 2.2.9..
At first all works fine but then i get often Alarams, for example no SNMP Values for more than 5minutes or high ICMP response time (icmppingsec).. so i look at some server stats and see that zabbix need more ressources and i have a higher load..


I Hope someone can help me..

[jan.garaj]

Please post your graphs again, but with higher resolution (e.g. use some 3rd party image hosting). I can't read your metrics.

[Colttt]

Hello jan.garaj

here an pic in better resolution, thanks for help!

http://fs2.directupload.net/images/150413/72atknnx.png

[jan.garaj]

Usage is pretty low. Only jumps have been increased significantly.

Investigate:
- context switches
http://unix.stackexchange.com/questi...-process-makes
- interruptions
cat /proc/interrupts

[Colttt]

thanks for this hint, but also are the response time are higher (avg) since i have upgraded to zabbix 2.2.9

[Colttt]

any idea???

[Colttt]

Soo.. here are some investigations..

cat /proc/interrupts >> interrupts_0; sleep 60; cat /proc/interrupts >> interrupts_60
diff interrupts_0 interrupts_60 
10c10
<  15:      86442      69109   IO-APIC-edge      ata_piix
---
>  15:      86472      69138   IO-APIC-edge      ata_piix
44,45c44,45
<  72:   31920493       3633   PCI-MSI-edge      eth0-rxtx-0
<  73:         54   27389735   PCI-MSI-edge      eth0-rxtx-1
---
>  72:   31932179       3633   PCI-MSI-edge      eth0-rxtx-0
>  73:         54   27400534   PCI-MSI-edge      eth0-rxtx-1
47c47
<  75:    1616372    1035625   PCI-MSI-edge      vmw_pvscsi
---
>  75:    1616910    1035716   PCI-MSI-edge      vmw_pvscsi
51c51
< LOC:   13098078   13972546   Local timer interrupts
---
> LOC:   13102804   13978381   Local timer interrupts
54c54
< IWI:    1801915    1919114   IRQ work interrupts
---
> IWI:    1802476    1919966   IRQ work interrupts
56,58c56,58
< RES:  294744456  294727503   Rescheduling interrupts
< CAL:     527973     869043   Function call interrupts
< TLB:     179710     180415   TLB shootdowns
---
> RES:  294872443  294855571   Rescheduling interrupts
> CAL:     528010     869298   Function call interrupts
> TLB:     179796     180509   TLB shootdowns
62c62
< MCP:        534        534   Machine check polls
---
> MCP:        535        535   Machine check polls

soo the eth0-rxtx-* are high (IMHO) around 11000


soo but when i do this:

pidstat -wIt 10 1 >> pidstat_0; sleep 60; pidstat -wIt 10 1 >> pidstat_60
diff pidstat_0 pidstat_60
3,291c3,283
< 12:02:54 PM   UID      TGID       TID   cswch/s nvcswch/s  Command
[...]
< 12:03:04 PM   109      1586         -   2779.84      2.50  zabbix_server
< 12:03:04 PM   109         -      1586   2779.84      2.50  |__zabbix_server
< 12:03:04 PM   109      1587         -   3167.86      1.20  zabbix_server
< 12:03:04 PM   109         -      1587   3167.86      1.20  |__zabbix_server
< 12:03:04 PM   109      1588         -   2794.41      2.40  zabbix_server
< 12:03:04 PM   109         -      1588   2794.41      2.40  |__zabbix_server
< 12:03:04 PM   109      1589         -   3168.06      1.10  zabbix_server
< 12:03:04 PM   109         -      1589   3168.06      1.10  |__zabbix_server
< 12:03:04 PM   109      1590         -   3167.47      1.00  zabbix_server
< 12:03:04 PM   109         -      1590   3167.47      1.00  |__zabbix_server
< 12:03:04 PM   109      1591         -   3168.06      0.80  zabbix_server
< 12:03:04 PM   109         -      1591   3168.06      0.80  |__zabbix_server
< 12:03:04 PM   109      1592         -   2922.85      1.50  zabbix_server
< 12:03:04 PM   109         -      1592   2922.85      1.50  |__zabbix_server
< 12:03:04 PM   109      1593         -   3168.06      1.50  zabbix_server
< 12:03:04 PM   109         -      1593   3168.06      1.50  |__zabbix_server
< 12:03:04 PM   109      1594         -      3.09      0.80  zabbix_server
< 12:03:04 PM   109         -      1594      3.09      0.80  |__zabbix_server
< 12:03:04 PM   109      1595         -   3167.76      1.80  zabbix_server
< 12:03:04 PM   109         -      1595   3167.76      1.80  |__zabbix_server
< 12:03:04 PM   109      1596         -   3167.86      1.30  zabbix_server
< 12:03:04 PM   109         -      1596   3167.86      1.30  |__zabbix_server
< 12:03:04 PM   109      1597         -   3138.42      1.50  zabbix_server
< 12:03:04 PM   109         -      1597   3138.42      1.50  |__zabbix_server
[...]
---
> 12:04:04 PM   UID      TGID       TID   cswch/s nvcswch/s  Command
[...]
> 12:04:14 PM   109      1586         -   5320.92      1.49  zabbix_server
> 12:04:14 PM   109         -      1586   5320.92      1.49  |__zabbix_server
> 12:04:14 PM   109      1587         -   5453.39      2.19  zabbix_server
> 12:04:14 PM   109         -      1587   5453.39      2.19  |__zabbix_server
> 12:04:14 PM   109      1588         -   5467.03      3.19  zabbix_server
> 12:04:14 PM   109         -      1588   5467.03      3.19  |__zabbix_server
> 12:04:14 PM   109      1589         -   5455.08      1.89  zabbix_server
> 12:04:14 PM   109         -      1589   5455.08      1.89  |__zabbix_server
> 12:04:14 PM   109      1590         -   1686.75      0.30  zabbix_server
> 12:04:14 PM   109         -      1590   1686.75      0.30  |__zabbix_server
> 12:04:14 PM   109      1591         -   5467.23      1.79  zabbix_server
> 12:04:14 PM   109         -      1591   5467.23      1.79  |__zabbix_server
> 12:04:14 PM   109      1592         -   5080.98      3.19  zabbix_server
> 12:04:14 PM   109         -      1592   5080.98      3.19  |__zabbix_server
> 12:04:14 PM   109      1593         -   5449.00      1.59  zabbix_server
> 12:04:14 PM   109         -      1593   5449.00      1.59  |__zabbix_server
> 12:04:14 PM   109      1594         -   5467.33      1.00  zabbix_server
> 12:04:14 PM   109         -      1594   5467.33      1.00  |__zabbix_server
> 12:04:14 PM   109      1595         -   3567.53      1.69  zabbix_server
> 12:04:14 PM   109         -      1595   3567.53      1.69  |__zabbix_server
> 12:04:14 PM   109      1596         -   5321.12      1.79  zabbix_server
> 12:04:14 PM   109         -      1596   5321.12      1.79  |__zabbix_server
> 12:04:14 PM   109      1597         -   5249.90      1.99  zabbix_server
> 12:04:14 PM   109         -      1597   5249.90      1.99  |__zabbix_server
[...]
294,313c286,305
[...]
520,581c490,565
[...]
< Average:      109      1586         -   2779.84      2.50  zabbix_server
< Average:      109         -      1586   2779.84      2.50  |__zabbix_server
< Average:      109      1587         -   3167.86      1.20  zabbix_server
< Average:      109         -      1587   3167.86      1.20  |__zabbix_server
< Average:      109      1588         -   2794.41      2.40  zabbix_server
< Average:      109         -      1588   2794.41      2.40  |__zabbix_server
< Average:      109      1589         -   3168.06      1.10  zabbix_server
< Average:      109         -      1589   3168.06      1.10  |__zabbix_server
< Average:      109      1590         -   3167.47      1.00  zabbix_server
< Average:      109         -      1590   3167.47      1.00  |__zabbix_server
< Average:      109      1591         -   3168.06      0.80  zabbix_server
< Average:      109         -      1591   3168.06      0.80  |__zabbix_server
< Average:      109      1592         -   2922.85      1.50  zabbix_server
< Average:      109         -      1592   2922.85      1.50  |__zabbix_server
< Average:      109      1593         -   3168.06      1.50  zabbix_server
< Average:      109         -      1593   3168.06      1.50  |__zabbix_server
< Average:      109      1594         -      3.09      0.80  zabbix_server
< Average:      109         -      1594      3.09      0.80  |__zabbix_server
< Average:      109      1595         -   3167.76      1.80  zabbix_server
< Average:      109         -      1595   3167.76      1.80  |__zabbix_server
< Average:      109      1596         -   3167.86      1.30  zabbix_server
< Average:      109         -      1596   3167.86      1.30  |__zabbix_server
< Average:      109      1597         -   3138.42      1.50  zabbix_server
< Average:      109         -      1597   3138.42      1.50  |__zabbix_server
[...]
---
> Average:      109      1586         -   5320.92      1.49  zabbix_server
> Average:      109         -      1586   5320.92      1.49  |__zabbix_server
> Average:      109      1587         -   5453.39      2.19  zabbix_server
> Average:      109         -      1587   5453.39      2.19  |__zabbix_server
> Average:      109      1588         -   5467.03      3.19  zabbix_server
> Average:      109         -      1588   5467.03      3.19  |__zabbix_server
> Average:      109      1589         -   5455.08      1.89  zabbix_server
> Average:      109         -      1589   5455.08      1.89  |__zabbix_server
> Average:      109      1590         -   1686.75      0.30  zabbix_server
> Average:      109         -      1590   1686.75      0.30  |__zabbix_server
> Average:      109      1591         -   5467.23      1.79  zabbix_server
> Average:      109         -      1591   5467.23      1.79  |__zabbix_server
> Average:      109      1592         -   5080.98      3.19  zabbix_server
> Average:      109         -      1592   5080.98      3.19  |__zabbix_server
> Average:      109      1593         -   5449.00      1.59  zabbix_server
> Average:      109         -      1593   5449.00      1.59  |__zabbix_server
> Average:      109      1594         -   5467.33      1.00  zabbix_server
> Average:      109         -      1594   5467.33      1.00  |__zabbix_server
> Average:      109      1595         -   3567.53      1.69  zabbix_server
> Average:      109         -      1595   3567.53      1.69  |__zabbix_server
> Average:      109      1596         -   5321.12      1.79  zabbix_server
> Average:      109         -      1596   5321.12      1.79  |__zabbix_server
> Average:      109      1597         -   5249.90      1.99  zabbix_server
> Average:      109         -      1597   5249.90      1.99  |__zabbix_server
[...]

i think that is a Problem with zabbix_server..