Zabbix Pre 1.5 Build 4666: Can't recreate ZABBIX semaph

[gazzerh]

Try restarting your server and trying it again.

[surftimer]

Unfortunately after a reboot still the same Error message

[backup.localdomain ~]# su - zabbix -c "/etc/init.d/zabbix start"
/usr/sbin/zabbix_server [2984]: ERROR: Can't recreate shared memory for database cache. [too many attempts]
/etc/init.d/zabbix start: zabbix_server could not be started

[nelsonab]

My head hurts....

Gotta love it how this one's stuck around for a while... ;-)

I finally got the server and agent to work on x86_64 Centos 5. I too was afflicted with *MANY* semaphore and unable to create shared memory errors. What I found was that I had to very carefully remove the created shared spaces and semaphores.

When zabbix isn't running check the IPC information using "ipcs -a". To delete an entry use either the shmid, semid, or msquid ID's, not the Keys. To remove the key use ipcrm, check the man page for usage.

It seems that the Zabbix server and agent do not always properly shut down their semaphore spaces as they exist after Zabbix has been shut down. Because of this I've been dealing with many of the error messages seen earlier in this thread.

An FYI on future issues try and include an strace output. It helped me track down this issue earlier, and it will help others give better assistance.

As an example with the server run the following as root (substitute as appropriate):
su - zabbix -c "strace /path/to/zabbix/binary/zabbix_server"
This will ensure you get the strace fo just the zabbix server and that it will be run as the zabbix user. Otherwise if you strace on the whole line be prepared to strip out a lot of unrelated system calls. :-)

Hope this helps others.

Alexi, your thoughts? Is it possible zabbix is not always shutting itself down correctly? I show some stale Shared Memory Segments and Semaphore Arrays from the zabbix agent after a restart.

[BusteR81]

server from user acount ??

Also, make sure you do not start the zabbix_server or zabbix_agentd from a root account.

pardon me, but for the 1.4.x version, we usually start zabbix server using root account and agentd from user:zabbix account.

please correct me if i'm wrong...Thanks

[sebelk]

Please, we should understand that semaphores issue is a *current* problem in 1.5.1. I am using on 2 pretty different machines with Centos 5.1 and the problem is there. Rebooting system is a acceptable workaround on development releases but I hope this bug be fixed on 1.6 when one wish use it for production.

[disgruntleddutch]

Running Centos 5.1
Linux zabbix 2.6.18-53.el5xen #1 SMP Mon Nov 12 02:46:57 EST 2007 x86_64 x86_64 x86_64 GNU/Linux

[root@zabbix tmp]# /opt/zabbix/sbin/zabbix_server -V
ZABBIX Server (daemon) v1.5.1 (11 April 2008)
Compilation time: Apr 29 2008 20:48:56

When doing /opt/zabbix/sbin/zabbix_server as root I get the semaphore problem

When doing su - zabbix -c "/etc/init.d/zabbix_server start" I get the Can't recreate shared memory for database cache. [too many attempts] error.


UPDATE:
Downloaded and compiled Zabbix 1.5.2. Problem still exists however I now get the semaphore problem in both instances (the database error is gone).

Attached are more straces.
Attached is a zip file with the two zabbix straces.

[tighep]

I am running RHEL5, and I just compiled 1.5.2. I was able to start the server with the root account, and did not see the semaphore issue with this latest release.

[cbidwell]

How did you get yours to run on RHEL5? I'm running RHEL5 as well and still get the semaphore errors. Do you have SELINUX enabled or disabled? Any other settings which might cause these semaphore errors?

[tighep]

I have SELinux disabled, and I configured the server with configure --enable-server --with-mysql. There really isn't anything special with this test box, is was just updated to current via RHN before the 1.5.2 install, but I don't think has anything to do with it.

[Alexei]

The problem will be fixed prior to 1.6. It is just a matter of time really.

[jquast]

solaris smf method workaround for stale shared memory segment bug

I have this problem on SunOS 5.10 also. I have scripted stale shared memory segments to be destroyed on start in the stop/start method script:

#!/bin/ksh
BIN="/usr/local/sbin/zabbix_server"
CONFIG_FILE="/jstor/etc/zabbix/server.conf"
PIDFILE="/var/logs/zabbix/server.pid"
UID=zabbix
GID=zabbix
LOGPATH="/jstor/logs/zabbix"
export ORACLE_HOME=/usr/local/oracle/current

if [ ! -f ${CONFIG_FILE} ]; then
        echo "${CONFIG_FILE} not found!"
        exit 1
fi

if [ ! -x ${BIN} ]; then
        echo -n "${BIN} not installed! "
        exit 1
fi

if [ ! -d ${LOGPATH} ]; then
        mkdir -p ${LOGPATH}
        chmod ${UID}:${GID} ${LOGPATH}
fi

PID=`cat ${PIDFILE} 2>/dev/null`
case "$1" in
    start)
        if [ ! -z ${PID} ]; then
                if ps -p ${PID} 2>/dev/null 1>&2; then
                        echo "`basename $0` already running, calling stop method"
                        $0 stop
                else
                        echo "removing state pid ${PID} from ${PIDFILE}"
                        rm -f ${PIDFILE}
                fi
        fi
        # [url]http://www.zabbix.com/forum/showthread.php?t=11116[/url]
        echo "Searching for stale shared memory segments..."
        ipcs -A | grep zabbix | while read x dig key whatever; do
                echo "removing: $key ($whatever)"
                ipcrm -S $key
        done
        echo "Starting `basename $0`"
        exec ${BIN} -c ${CONFIG_FILE}
        ;;
    stop)
        if [ ! -z ${PID} ]; then
                if ps -p ${PID} 2>/dev/null 1>&2; then
                        echo "Shutting down `basename $0`"
                        kill $PID
                        sleep 5
                        kill -9 $PID 2>/dev/null
                else
                        echo "`basename $0` already stopped"
                fi
        else
                echo "No PID in ${PIDFILE}"
        fi
        rm -f ${PIDFILE}
        ;;
    restart)
        $0 stop
        $0 start
        ;;
    *)
        echo "Usage: $0 {start|stop|restart}"
        exit 1
        ;;
esac

the service log will look like:

[ Oct 30 15:48:12 Executing start method ("/jstor/svc-method/zabbix_server start") ]
Searching for stale shared memory segments...
removing: 0x6b01b2ef (--rw-rw-rw-   zabbix   zabbix   zabbix   zabbix      0    8388608  1306  1306 15:38:59 15:48:12 15
:38:59        0          system)
ipcrm: semget(0x6b01b2ef): not found
removing: 0x6301b2ef (--rw-rw-rw-   zabbix   zabbix   zabbix   zabbix      0   27978312  1306  1306 15:38:59 15:48:12 15
:38:59        0          system)
ipcrm: semget(0x6301b2ef): not found
removing: 0x7a01bad0 (--rw-rw-rw-   zabbix   zabbix   zabbix   zabbix      6    1216860  1279  1279 15:38:57 no-entry 15
:38:57        0          system)
removing: 0x7a01b2ef (--ra-ra-ra-   zabbix   zabbix   zabbix   zabbix     5 15:48:09 15:38:57          system)
removing: 0x7a01bad0 (--ra-ra-ra-   zabbix   zabbix   zabbix   zabbix     5 15:38:57 15:38:57          system)
ipcrm: semget(0x7a01bad0): not found
Starting zabbix_server
[ Oct 30 15:48:12 Method "start" exited with status 0 ]
[ Oct 30 16:06:05 Stopping because service restarting. ]

this is desperately needed on x86 SunOS 10.5 with oracle, because as you can see here, zabbix_server 1.7.1 process is disappearing every 15 minutes https://support.zabbix.com/browse/ZBX-1160