Zabbix and phpvirtualbox aborts once in a while

[Joey]

Hi,

I have zabbix 2.2.1 set up on phpvirtualbox 4.3.6 with a mysql database Ver 14.14 Distrib 5.5.37.

Zabbix Dashboard :
# of hosts 93
# of items 3437
# of triggers 97
# of users 5
Required server performance 13.19

Phpvirtualbox zabbix macine :
Os Ubuntu
Base memory 8192MB
Processors 8
Boot Order CD/DVD-ROM, Hard Disk
Accelerator VT-x/AMD-V, Nested Paging

I seems to be up and running ok, monitoring most host with snmp polling, receiving traps from a couple of hosts, maps working with network links, email notifications via postfix relay to gmail, sms phone alerts,

I have an issue with zabbix aborting once in awhile, I was hoping that someone else has had the same issue and can shed some light on what is happening or point me in the right direction for diagnoses.

[tchjts1]

What do you mean "aborting"? You need to be more specific than that.

Have a look in zabbix_server.log and post the relevant errors here.

Also, to see if any of your Zabbix internal processes are overloaded, have a look at this post, at the last paragraph and the graphs that follow it. Then take a similar look at your setup.

Link: https://www.zabbix.com/forum/showthread.php?t=41219

[Joey]

Hi,
this is the only thing that I can see in zabbix log that is a bit suspicious .
zabbix log 1st part
1480:20140811:104721.981 End of DCconfig_get_poller_nextcheck():1407779242
1572:20140811:104721.981 In zbx_vc_add_value() itemid:26118 value_type:3 timestamp:1407779238.362730446
1480:20140811:104721.981 poller #15 [got 0 values in 0.000452 sec, idle 1 sec]
1572:20140811:104721.981 End of zbx_vc_add_value():FAIL
1572:20140811:104721.981 In zbx_vc_add_value() itemid:30918 value_type:3 timestamp:1407779238.384423985
1572:20140811:104721.981 End of zbx_vc_add_value():FAIL
1572:20140811:104721.981 In zbx_vc_add_value() itemid:26418 value_type:3 timestamp:1407779238.384707261
1572:20140811:104721.981 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:27738 value_type:3 timestamp:1407779238.385992620
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:28338 value_type:3 timestamp:1407779238.396177155
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:31068 value_type:3 timestamp:1407779238.399205752
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:26268 value_type:3 timestamp:1407779238.399190387
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:27828 value_type:3 timestamp:1407779238.400865739
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:28008 value_type:3 timestamp:1407779238.410102388
1572:20140811:104721.982 End of zbx_vc_add_value():SUCCEED
1572:20140811:104721.982 In zbx_vc_add_value() itemid:31098 value_type:3 timestamp:1407779238.410785995
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:28038 value_type:3 timestamp:1407779238.415246059
1572:20140811:104721.982 End of zbx_vc_add_value():FAIL
1572:20140811:104721.982 In zbx_vc_add_value() itemid:27798 value_type:3 timestamp:1407779238.420384980
1572:20140811:104721.982 End of zbx_vc_add_value():SUCCEED

zabbix log 2nd part (same log just further down in log)
1570:20140811:104723.093 query [txnlev:1] [insert into history (itemid,clock,ns,value) values (23302,1407779242,561065429,0.000000);
insert into history_uint (itemid,clock,ns,value) values (23662,1407779242,578909597,28256),(26272,14077792 42,606004381,2),(26542,1407779242,616034427,0),(26 422,1407779$
insert into history_text (id,itemid,clock,ns,value) values (2838692,28042,1407779242,615451392,'1:11');
]
1570:20140811:104723.094 In zbx_vc_add_value() itemid:23302 value_type:0 timestamp:1407779242.561065429
1570:20140811:104723.094 End of zbx_vc_add_value():FAIL
1570:20140811:104723.094 In zbx_vc_add_value() itemid:23662 value_type:3 timestamp:1407779242.578909597
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:26272 value_type:3 timestamp:1407779242.606004381
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:28042 value_type:4 timestamp:1407779242.615451392
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:26542 value_type:3 timestamp:1407779242.616034427
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:26422 value_type:3 timestamp:1407779242.622018148
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:26122 value_type:3 timestamp:1407779242.626462847
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 In zbx_vc_add_value() itemid:31642 value_type:3 timestamp:1407779242.682708543
1570:20140811:104723.095 End of zbx_vc_add_value():FAIL
1570:20140811:104723.095 End of DCmass_add_history()
1570:20140811:104723.095 In DCmass_update_triggers()
1570:20140811:104723.095 End of DCmass_update_triggers()


graph images from zabbix server

Server performance



Traffic on eth0


CPU Utilization


CPU Load

[tchjts1]

The link I gave in my previous response tells you how to look at the graphs for your Zabbix internal processes. Please re-read the last paragraph of that post and look at what those graphs are showing you.

As for your Zabbix logs, I would be more interested as to what they indicate on 8/3 around 5:30am when the data stopped populating the graphs. zabbix_agentd.log on your host(s) may also show information as to what is going on at that time.

Here are 2 of the graphs you should be looking at:

[Joey]

Internal processes busy


Gathering processes busy


zabbix agentd log
1043:20140808:165800.034 Starting Zabbix Agent [zabbix]. Zabbix 2.2.1 (revision 40808).
1043:20140808:165800.034 using configuration file: /usr/local/etc/zabbix_agentd.conf
1044:20140808:165800.036 agent #0 started [collector]
1045:20140808:165800.037 agent #1 started[listener #1]
1048:20140808:165800.037 agent #4 started [active checks #1]
1047:20140808:165800.040 agent #3 started[listener #3]
1046:20140808:165800.041 agent #2 started[listener #2]
1048:20140808:165800.041 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)
1048:20140808:165900.126 active check configuration update from [127.0.0.1:10051] is working again
1048:20140808:165900.126 no active checks on server [127.0.0.1:10051]: host [zabbix] not found
1048:20140808:170100.182 no active checks on server [127.0.0.1:10051]: host [zabbix] not found
1048:20140808:170300.288 no active checks on server [127.0.0.1:10051]: host [zabbix] not found
1048:20140808:170500.412 no active checks on server [127.0.0.1:10051]: host [zabbix] not found
1048:20140808:170700.598 no active checks on server [127.0.0.1:10051]: host [zabbix] not found
1048:20140808:170900.739 no active checks on server [127.0.0.1:10051]: host [zabbix] not found

then it continues on the rest of the log with "no active checks on server"

I will have to dig up the archive logs for the server log to try and find the aug 03 entries

[tchjts1]

Internal processes busy

zabbix agentd log
1043:20140808:165800.034 Starting Zabbix Agent [zabbix]. Zabbix 2.2.1 (revision 40808).
1043:20140808:165800.034 using configuration file: /usr/local/etc/zabbix_agentd.conf

1048:20140808:165800.041 active check configuration update from [127.0.0.1:10051] started to fail (cannot connect to [[127.0.0.1]:10051]: [111] Connection refused)
1048:20140808:165900.126 active check configuration update from [127.0.0.1:10051] is working again
1048:20140808:165900.126 no active checks on server [127.0.0.1:10051]: host [zabbix] not found

Your graphs look OK. I don't see where any of the processes are overloaded.

I do see two things I would check as indicated by your log. "Connection Refused" generally indicates that in your zabbix_agentd.conf file you don't have the correct value in Server= and/or ServerActive=, or the agent process simply is not running.

Both of those bold values above should be the IP of your Zabbix server.

Second thing is "host [zabbix] not found". That would be for anything that is an item type of Zabbix agent (Active) aka Active agent. For active items to work, there has to be an exact match of what you have in your zabbix_agentd.conf file for Hostname= with what you have for the host name in Zabbix frontend. This includes being case sensitive.

If you make any changes to your zabbix_agentd.conf file, you have to restart the Zabbix agent process.

[Joey]

Hi,
Sorry have bee busy for a bit, but have looked at the zabbix_agentd.conf file and it shows that both Server and ServerActive equal 127.0.0.1 the agent is only there to monitor zabbix-server itself.

### Option: Server
# List of comma delimited IP addresses (or hostnames) of Zabbix servers.
# Incoming connections will be accepted only from the hosts listed here.
# If IPv6 support is enabled then '127.0.0.1', '::127.0.0.1', '::ffff:127.0.0.1' are treated equally.
#
# Mandatory: no
# Default:
# Server=

Server=127.0.0.1

AND

### Option: ServerActive
# List of comma delimited IPort (or hostnameort) pairs of Zabbix servers for active checks.
# If port is not specified, default port is used.
# IPv6 addresses must be enclosed in square brackets if port for that host is specified.
# If port is not specified, square brackets for IPv6 addresses are optional.
# If this parameter is not specified, active checks are disabled.
# Example: ServerActive=127.0.0.1:20051,zabbix.domain,[::1]:30051,::1,[12fc::1]
#
# Mandatory: no
# Default:
# ServerActive=

ServerActive=127.0.0.1

AND Hostname

### Option: Hostname
# Unique, case sensitive hostname.
# Required for active checks and must match hostname as configured on the server.
# Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

Hostname=zabbix

I am a little unsure is, this the hostname in the server host file ?

[tchjts1]

Hostname=zabbix

This is the agent telling Zabbix server what host is giving it data. And this will be problematic if you are using any Active agent items. If this is the agent on your Zabbix server, you are telling Zabbix that the name it should populate data to in the frontend is "zabbix". Do you have a host in the frontend with the name "zabbix"?

From the graph screenshots, it looks like in the frontend you have the host named as "Zabbix server". That is not a match.

In your zabbix_agentd.conf file, change this:
Hostname=zabbix

to this (Case sensitive) :

Hostname=Zabbix server

And restart your agent process. That may not fix your current issue, but it is going to prevent issues going forward if you use Active check items.

[Joey]

Ok, have changed the host name to reflect the host created in zabbix (Zabbix server)

### Option: Hostname
# Unique, case sensitive hostname.
# Required for active checks and must match hostname as configured on the$
# Value is acquired from HostnameItem if undefined.
#
# Mandatory: no
# Default:
# Hostname=

Hostname=Zabbix server

Thank you I thought it was the host name of the server.

Do you think this is could be the reason it had aborted, or maybe it is something deeper, I did change the mysql database to innodb a couple of months back?

[tchjts1]

Is that the only host that stops reporting data, or is it all monitored hosts?

Is it your agent that is crashing, or is it your Zabbix server process that is crashing?

[Joey]

Hi tchjts1,

It is my zabbix server that shuts down, usually when i notice I check the phpvirualbox and it shows aborted with a do not enter symbol where is usually says running or powered off

[tchjts1]

Ok, so... being powered off isn't really a Zabbix related issue, right?

[Joey]

I think it might have to do with, when I had switched over to innodb on mysql database because i recently tried to do a mysqldump and it also caused it to show the abort status in phpvirtualbox,

[tchjts1]

Ok. Maybe someone familiar with that setup can be of help to you. It is not an environment I am personally familiar with. Sorry i can't help you more.

[Joey]

ok thanks for your replies.