Running 3.0.2rc on ubuntu 15.40.
Every once in a while apache2 hangs. "top" shows a few processes as follows:
These are running 100% in CPU. I tried a strace against them, they attach and show nothing. lsof shows nothing particularly informative, I see no php files open. the mod_status (server-status) will not run when it gets into this state, in fact I can't get anything to run.
The most interesting piece is netstat, it shows a port 80 connection to the same host for each one of these -- my terminal. Where I've got various zabbix screens up, all of which are hung at this point, but nothing in common about them that I can see.
I think this may be happening when VPN drops (I am connecting to this environment and the zabbix server over VPN). Maybe. At least it always shows up when I reconnect, but I have no clear way to tell when it actually started.
I am running debug mode in apache2 and nothing unusual shows up in the logs, I've scoured them and nothing that looks like a warning or error or resource shortage or anything really.
Restarting apache2 will fix the problem immediately, there's no need to restart zabbix.
Interestingly the API continues to run -- I've got emails that use the API as well as pulling graphs, and these continue to come out. It is interactive access that seems to hang.
I closed the browser sessions and netstat on my (windows) desktop shows fin_wait_2, and interestingly (remember VPN was down, in fact for hours) the netstat on the zabbix server went into CLOSE_WAIT, so they were still thinking a session was up even though VPN was down.
I've had this now happen 4-5 times. This is the first time I tracked it down far enough to know it related to my browser sessions, at least somewhat.
Is this kind of hang familiar to anyone?
Any suggestions for further debugging?
It is NOT reproducible on demand, I've terminated VPN with browsers open dozens of times, most time there is no impact. In fact, I'm not 100% sure that was the issue here, just suspicion. The server generally runs flawlessly -- only very rarely does it hang like this. So I cannot observe as it goes into the hang (for example), I need some way to look up a cause after it occurs. Again - apache debug mode is on already. Any other thoughts?
Every once in a while apache2 hangs. "top" shows a few processes as follows:
Code:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25657 www-data 30 10 373604 49236 28100 R 23.9 0.6 2:17.62 /usr/sbin/apache2 -k start 28626 www-data 30 10 384660 58732 27216 R 21.3 0.7 2:24.47 /usr/sbin/apache2 -k start 58364 www-data 30 10 384924 60128 27676 R 15.7 0.8 2:18.47 /usr/sbin/apache2 -k start 58446 www-data 30 10 382164 56840 27144 R 14.1 0.7 2:27.15 /usr/sbin/apache2 -k start 25656 www-data 30 10 394900 69524 27092 R 13.1 0.9 2:30.13 /usr/sbin/apache2 -k start 25662 www-data 30 10 379524 54796 27840 R 12.8 0.7 2:57.66 /usr/sbin/apache2 -k start
The most interesting piece is netstat, it shows a port 80 connection to the same host for each one of these -- my terminal. Where I've got various zabbix screens up, all of which are hung at this point, but nothing in common about them that I can see.
I think this may be happening when VPN drops (I am connecting to this environment and the zabbix server over VPN). Maybe. At least it always shows up when I reconnect, but I have no clear way to tell when it actually started.
I am running debug mode in apache2 and nothing unusual shows up in the logs, I've scoured them and nothing that looks like a warning or error or resource shortage or anything really.
Restarting apache2 will fix the problem immediately, there's no need to restart zabbix.
Interestingly the API continues to run -- I've got emails that use the API as well as pulling graphs, and these continue to come out. It is interactive access that seems to hang.
I closed the browser sessions and netstat on my (windows) desktop shows fin_wait_2, and interestingly (remember VPN was down, in fact for hours) the netstat on the zabbix server went into CLOSE_WAIT, so they were still thinking a session was up even though VPN was down.
I've had this now happen 4-5 times. This is the first time I tracked it down far enough to know it related to my browser sessions, at least somewhat.
Is this kind of hang familiar to anyone?
Any suggestions for further debugging?
It is NOT reproducible on demand, I've terminated VPN with browsers open dozens of times, most time there is no impact. In fact, I'm not 100% sure that was the issue here, just suspicion. The server generally runs flawlessly -- only very rarely does it hang like this. So I cannot observe as it goes into the hang (for example), I need some way to look up a cause after it occurs. Again - apache debug mode is on already. Any other thoughts?