Zabbix не запускается после обновления (с 3.2.3 до 3.2.4)

[lpetr2]

zabbix-server живет на Centos 7. Его база на отдельно сервере на Centos 6, MAriaDB.

Обновлял так yum update zabbix* .

Обновления нашел и вроде все прошло успешно. Однако при запуске zabbix-server не запускается. Помогите разобраться в чем дело. БД работает, ошибок не видно.

Вот что есть:

[root@zabbix-server ~]# service zabbix-server start
Redirecting to /bin/systemctl start zabbix-server.service
Job for zabbix-server.service failed because a timeout was exceeded. See "systemctl status zabbix-server.service" and "journalctl -xe" for details.
[root@zabbix-server ~]# systemctl status zabbix-server.service
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: timeout) since Чт 2017-03-02 13:29:09 MSK; 7s ago
Process: 12434 ExecStop=/bin/kill -SIGTERM $MAINPID (code=killed, signal=TERM)
Main PID: 12064 (code=killed, signal=KILL)
CGroup: /system.slice/zabbix-server.service

мар 02 13:29:09 zabbix-server systemd[1]: Failed to start Zabbix Server.
мар 02 13:29:09 zabbix-server systemd[1]: Unit zabbix-server.service entered failed state.
мар 02 13:29:09 zabbix-server systemd[1]: zabbix-server.service failed.
[root@zabbix-server ~]# journalctl -xe
-- Результат: failed.
мар 02 13:29:09 zabbix-server systemd[1]: Unit zabbix-server.service entered failed state.
мар 02 13:29:09 zabbix-server systemd[1]: zabbix-server.service failed.
мар 02 13:29:20 zabbix-server systemd[1]: zabbix-server.service holdoff time over, scheduling restart.
мар 02 13:29:20 zabbix-server systemd[1]: Starting Zabbix Server...
-- Subject: Начинается запуск юнита zabbix-server.service
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman.../systemd-devel
--
-- Начат процесс запуска юнита zabbix-server.service.
мар 02 13:29:20 zabbix-server systemd[1]: zabbix-server.service start operation timed out. Terminating.
мар 02 13:29:20 zabbix-server systemd[1]: zabbix-server.service stop-final-sigterm timed out. Killing.
мар 02 13:29:20 zabbix-server systemd[1]: zabbix-server.service still around after final SIGKILL. Entering failed mode.
мар 02 13:29:20 zabbix-server systemd[1]: Failed to start Zabbix Server.
-- Subject: Ошибка юнита zabbix-server.service
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman.../systemd-devel
--
-- Произошел сбой юнита zabbix-server.service.
--
-- Результат: failed.
мар 02 13:29:20 zabbix-server systemd[1]: Unit zabbix-server.service entered failed state.
мар 02 13:29:20 zabbix-server systemd[1]: zabbix-server.service failed.

В логах заббикс сервера

Starting Zabbix Server. Zabbix 3.2.4 (revision 65975).
12064:20170302:113558.156 ****** Enabled features ******
12064:20170302:113558.156 SNMP monitoring: YES
12064:20170302:113558.156 IPMI monitoring: YES
12064:20170302:113558.156 Web monitoring: YES
12064:20170302:113558.157 VMware monitoring: YES
12064:20170302:113558.157 SMTP authentication: YES
12064:20170302:113558.157 Jabber notifications: YES
12064:20170302:113558.157 Ez Texting notifications: YES
12064:20170302:113558.157 ODBC: YES
12064:20170302:113558.157 SSH2 support: YES
12064:20170302:113558.157 IPv6 support: YES
12064:20170302:113558.157 TLS support: YES
12064:20170302:113558.157 ******************************
12064:20170302:113558.157 using configuration file: /etc/zabbix/zabbix_server.conf
12064:20170302:113558.479 current database version (mandatory/optional): 03020000/03020000
12064:20170302:113558.479 required mandatory version: 03020000
12064:20170302:113559.268 server #0 started [main process]
12069:20170302:113559.274 server #3 started [poller #1]
12070:20170302:113559.274 server #4 started [poller #2]
12068:20170302:113559.275 server #2 started [db watchdog #1]
12072:20170302:113559.275 server #6 started [poller #4]
12071:20170302:113559.278 server #5 started [poller #3]
12073:20170302:113559.280 server #7 started [poller #5]
12067:20170302:113559.282 server #1 started [configuration syncer #1]
12074:20170302:113559.282 server #8 started [poller #6]
12078:20170302:113559.285 server #12 started [poller #10]
12075:20170302:113559.287 server #9 started [poller #7]
12077:20170302:113559.291 server #11 started [poller #9]
12076:20170302:113559.292 server #10 started [poller #8]
12080:20170302:113559.296 server #14 started [poller #12]
12079:20170302:113559.300 server #13 started [poller #11]
12082:20170302:113559.305 server #16 started [poller #14]
12083:20170302:113559.306 server #17 started [poller #15]
12081:20170302:113559.307 server #15 started [poller #13]
12086:20170302:113559.322 server #20 started [poller #18]
12085:20170302:113559.323 server #19 started [poller #17]
12084:20170302:113559.330 server #18 started [poller #16]
12088:20170302:113559.342 server #22 started [poller #20]
12087:20170302:113559.350 server #21 started [poller #19]
12090:20170302:113559.364 server #24 started [poller #22]
12089:20170302:113559.376 server #23 started [poller #21]
12091:20170302:113559.387 server #25 started [poller #23]
12094:20170302:113559.388 server #28 started [poller #26]
12093:20170302:113559.399 server #27 started [poller #25]
12092:20170302:113559.400 server #26 started [poller #24]
12096:20170302:113559.434 server #30 started [poller #28]
12095:20170302:113559.436 server #29 started [poller #27]
12097:20170302:113559.450 server #31 started [poller #29]
12098:20170302:113559.456 server #32 started [poller #30]
12099:20170302:113559.458 server #33 started [poller #31]
12100:20170302:113559.487 server #34 started [poller #32]
12101:20170302:113559.500 server #35 started [poller #33]
12102:20170302:113559.537 server #36 started [poller #34]
12103:20170302:113559.539 server #37 started [poller #35]
12104:20170302:113559.540 server #38 started [poller #36]
12107:20170302:113559.573 server #41 started [poller #39]
12106:20170302:113559.576 server #40 started [poller #38]
12105:20170302:113559.580 server #39 started [poller #37]
12111:20170302:113559.615 server #43 started [poller #41]
12112:20170302:113559.624 server #44 started [poller #42]
12110:20170302:113559.638 server #42 started [poller #40]
12115:20170302:113559.666 server #47 started [poller #45]
12114:20170302:113559.666 server #46 started [poller #44]
12113:20170302:113559.674 server #45 started [poller #43]
12116:20170302:113559.675 server #48 started [poller #46]
12123:20170302:113559.714 server #50 started [poller #48]
12122:20170302:113559.735 server #49 started [poller #47]
12125:20170302:113559.745 server #52 started [poller #50]
12124:20170302:113559.747 server #51 started [poller #49]
12127:20170302:113559.786 server #54 started [poller #52]
12126:20170302:113559.787 server #53 started [poller #51]
12128:20170302:113559.840 server #55 started [poller #53]
12130:20170302:113559.847 server #57 started [poller #55]
12129:20170302:113559.849 server #56 started [poller #54]
12131:20170302:113559.851 server #58 started [poller #56]
12132:20170302:113559.866 server #59 started [poller #57]
12134:20170302:113559.882 server #60 started [poller #58]
12135:20170302:113559.887 server #61 started [poller #59]
12136:20170302:113559.895 server #62 started [poller #60]
12137:20170302:113559.920 server #63 started [poller #61]
12140:20170302:113559.941 server #66 started [poller #64]
12138:20170302:113559.946 server #64 started [poller #62]
12139:20170302:113559.950 server #65 started [poller #63]
12142:20170302:113559.979 server #68 started [poller #66]
12141:20170302:113559.988 server #67 started [poller #65]
12145:20170302:113600.014 server #70 started [poller #68]
12144:20170302:113600.017 server #69 started [poller #67]
12146:20170302:113600.019 server #71 started [poller #69]
12147:20170302:113600.044 server #72 started [poller #70]
12148:20170302:113600.047 server #73 started [poller #71]
12150:20170302:113600.084 server #75 started [poller #73]
12149:20170302:113600.112 server #74 started [poller #72]
12154:20170302:113600.120 server #77 started [poller #75]
12153:20170302:113600.128 server #76 started [poller #74]
12156:20170302:113600.154 server #78 started [poller #76]
12166:20170302:113600.190 server #80 started [poller #78]
12165:20170302:113600.201 server #79 started [poller #77]
12169:20170302:113600.220 server #83 started [poller #81]
12167:20170302:113600.221 server #81 started [poller #79]
12168:20170302:113600.223 server #82 started [poller #80]
12172:20170302:113600.245 server #84 started [poller #82]
12173:20170302:113600.256 server #85 started [poller #83]
12177:20170302:113600.286 server #87 started [poller #85]
12176:20170302:113600.286 server #86 started [poller #84]
12178:20170302:113600.298 server #88 started [poller #86]
12183:20170302:113600.332 server #90 started [poller #88]
12182:20170302:113600.359 server #89 started [poller #87]
12185:20170302:113600.366 server #92 started [poller #90]
12184:20170302:113600.368 server #91 started [poller #89]
12188:20170302:113600.374 server #95 started [poller #93]
12187:20170302:113600.388 server #94 started [poller #92]
12186:20170302:113600.390 server #93 started [poller #91]
12190:20170302:113600.422 server #97 started [poller #95]
12189:20170302:113600.436 server #96 started [poller #94]
12193:20170302:113600.452 server #100 started [poller #98]
12192:20170302:113600.453 server #99 started [poller #97]
12191:20170302:113600.454 server #98 started [poller #96]
12196:20170302:113600.492 server #103 started [poller #101]
12195:20170302:113600.494 server #102 started [poller #100]
12194:20170302:113600.502 server #101 started [poller #99]
12198:20170302:113600.529 server #105 started [poller #103]
12197:20170302:113600.531 server #104 started [poller #102]
12200:20170302:113600.544 server #107 started [poller #105]
12199:20170302:113600.552 server #106 started [poller #104]
12202:20170302:113600.580 server #109 started [poller #107]
12201:20170302:113600.582 server #108 started [poller #106]
12203:20170302:113600.612 server #110 started [poller #108]
12204:20170302:113600.614 server #111 started [poller #109]
12205:20170302:113600.615 server #112 started [poller #110]
12206:20170302:113600.619 server #113 started [poller #111]
12208:20170302:113600.663 server #115 started [poller #113]
12207:20170302:113600.669 server #114 started [poller #112]
12209:20170302:113600.687 server #116 started [poller #114]
12210:20170302:113600.694 server #117 started [poller #115]
12212:20170302:113600.717 server #119 started [poller #117]
12211:20170302:113600.721 server #118 started [poller #116]
12214:20170302:113600.746 server #121 started [poller #119]
12213:20170302:113600.750 server #120 started [poller #118]
12216:20170302:113600.788 server #122 started [poller #120]
12217:20170302:113600.794 server #123 started [poller #121]
12219:20170302:113600.818 server #125 started [poller #123]
12218:20170302:113600.820 server #124 started [poller #122]
12221:20170302:113600.848 server #126 started [poller #124]
12222:20170302:113600.855 server #127 started [poller #125]
12223:20170302:113600.874 server #128 started [poller #126]
12224:20170302:113600.876 server #129 started [poller #127]
12226:20170302:113600.898 server #130 started [poller #128]
12227:20170302:113600.900 server #131 started [poller #129]
12228:20170302:113600.926 server #132 started [poller #130]
12230:20170302:113600.928 server #134 started [poller #132]
12229:20170302:113600.931 server #133 started [poller #131]
12232:20170302:113600.974 server #136 started [poller #134]
12231:20170302:113600.980 server #135 started [poller #133]
12233:20170302:113600.989 server #137 started [poller #135]
12234:20170302:113601.001 server #138 started [poller #136]
12235:20170302:113601.029 server #139 started [poller #137]
12236:20170302:113601.032 server #140 started [poller #138]
12209:20170302:113601.049 Zabbix agent item "hdd.n6[sdb]" on host "Zabbix server" failed: first network error, wait for 15 seconds
12239:20170302:113601.051 server #143 started [poller #141]
12238:20170302:113601.052 server #142 started [poller #140]
12237:20170302:113601.055 server #141 started [poller #139]
12241:20170302:113601.101 server #145 started [poller #143]
12242:20170302:113601.106 server #146 started [poller #144]
12240:20170302:113601.111 server #144 started [poller #142]
12244:20170302:113601.140 server #148 started [poller #146]
12243:20170302:113601.151 server #147 started [poller #145]
12245:20170302:113601.160 server #149 started [poller #147]
12246:20170302:113601.189 server #150 started [poller #148]
12248:20170302:113601.194 server #152 started [poller #150]
12247:20170302:113601.196 server #151 started [poller #149]
12249:20170302:113601.204 server #153 started [unreachable poller #1]
12250:20170302:113601.219 server #154 started [unreachable poller #2]
12251:20170302:113601.221 server #155 started [unreachable poller #3]
12253:20170302:113601.258 server #157 started [unreachable poller #5]
12252:20170302:113601.285 server #156 started [unreachable poller #4]
12254:20170302:113601.289 server #158 started [unreachable poller #6]
12255:20170302:113601.290 server #159 started [unreachable poller #7]
12256:20170302:113601.296 server #160 started [unreachable poller #8]
12258:20170302:113601.327 server #162 started [unreachable poller #10]
12257:20170302:113601.340 server #161 started [unreachable poller #9]
12260:20170302:113601.355 server #164 started [unreachable poller #12]
12259:20170302:113601.357 server #163 started [unreachable poller #11]
12261:20170302:113601.386 server #165 started [unreachable poller #13]
12262:20170302:113601.408 server #166 started [unreachable poller #14]
12263:20170302:113601.414 server #167 started [unreachable poller #15]
12265:20170302:113601.434 server #169 started [unreachable poller #17]
12264:20170302:113601.436 server #168 started [unreachable poller #16]
12266:20170302:113601.461 server #170 started [unreachable poller #18]
12267:20170302:113601.463 server #171 started [unreachable poller #19]
12268:20170302:113601.484 server #172 started [unreachable poller #20]
12269:20170302:113601.496 server #173 started [unreachable poller #21]
12271:20170302:113601.508 server #174 started [unreachable poller #22]
12272:20170302:113601.518 server #175 started [unreachable poller #23]
12274:20170302:113601.534 server #177 started [unreachable poller #25]
12273:20170302:113601.542 server #176 started [unreachable poller #24]
12276:20170302:113601.561 server #179 started [unreachable poller #27]
12275:20170302:113601.563 server #178 started [unreachable poller #26]
12277:20170302:113601.568 server #180 started [unreachable poller #28]
12280:20170302:113601.607 server #182 started [unreachable poller #30]
12279:20170302:113601.618 server #181 started [unreachable poller #29]
12281:20170302:113601.628 server #183 started [trapper #1]
12282:20170302:113601.630 server #184 started [trapper #2]
12283:20170302:113601.637 server #185 started [icmp pinger #1]
12287:20170302:113601.669 server #187 started [icmp pinger #3]
12286:20170302:113601.692 server #186 started [icmp pinger #2]
12296:20170302:113601.704 server #189 started [icmp pinger #5]
12295:20170302:113601.706 server #188 started [icmp pinger #4]
12297:20170302:113601.707 server #190 started [icmp pinger #6]
12300:20170302:113601.727 server #191 started [icmp pinger #7]
12302:20170302:113601.737 server #192 started [icmp pinger #8]
12303:20170302:113601.738 server #193 started [icmp pinger #9]
12305:20170302:113601.755 server #195 started [alerter #1]
12304:20170302:113601.756 server #194 started [icmp pinger #10]
12306:20170302:113601.767 server #196 started [housekeeper #1]
12307:20170302:113601.782 server #197 started [timer #1]
12308:20170302:113601.784 server #198 started [http poller #1]
12310:20170302:113601.785 server #200 started [discoverer #2]
12309:20170302:113601.786 server #199 started [discoverer #1]
12312:20170302:113601.798 server #202 started [discoverer #4]
12311:20170302:113601.798 server #201 started [discoverer #3]
12314:20170302:113601.802 server #204 started [discoverer #6]
12313:20170302:113601.804 server #203 started [discoverer #5]
12315:20170302:113601.807 server #205 started [discoverer #7]
12316:20170302:113601.818 server #206 started [discoverer #8]
12317:20170302:113601.827 server #207 started [discoverer #9]
12319:20170302:113601.828 server #209 started [discoverer #11]
12318:20170302:113601.831 server #208 started [discoverer #10]
12320:20170302:113601.836 server #210 started [discoverer #12]
12322:20170302:113601.850 server #212 started [discoverer #14]
12321:20170302:113601.858 server #211 started [discoverer #13]
12324:20170302:113601.866 server #214 started [history syncer #1]
12323:20170302:113601.868 server #213 started [discoverer #15]
12326:20170302:113601.884 server #216 started [history syncer #3]
12325:20170302:113601.884 server #215 started [history syncer #2]
12327:20170302:113601.902 server #217 started [history syncer #4]
12328:20170302:113601.905 server #218 started [history syncer #5]
12330:20170302:113601.922 server #220 started [history syncer #7]
12329:20170302:113601.930 server #219 started [history syncer #6]
12332:20170302:113601.939 server #222 started [escalator #1]
12331:20170302:113601.942 server #221 started [history syncer #8]
12333:20170302:113601.945 server #223 started [proxy poller #1]
12335:20170302:113601.966 server #225 started [vmware collector #1]
12334:20170302:113601.968 server #224 started [self-monitoring #1]
12336:20170302:113601.973 server #226 started [task manager #1]
12255:20170302:113616.637 resuming Zabbix agent checks on host "Zabbix server": connection restored
12328:20170302:113623.931 slow query: 15.342438 sec, "select clock,ns,value from history_uint where itemid=24020 and clock<=1488267366 and clock>1487662566 order by clock desc limit 2851"
12064:20170302:113625.094 Got signal [signal:15(SIGTERM),sender_pid:12434,sender_uid:0,r eason:0]. Exiting ...
12439:20170302:113625.375 Starting Zabbix Server. Zabbix 3.2.4 (revision 65975).
12439:20170302:113625.376 Got signal [signal:15(SIGTERM),sender_pid:1,sender_uid:0,reaso n:0]. Exiting ...

[lpetr2]

Логи у zabbix-server больше не пишутся как не странно.

С виду все хорошо но не запускается сервер и не могу понять почему.

[lpetr2]

zabbix-agent при этом запускается без проблем

[Vadim Puzynia]

Аналогичная проблема, пока копаюсь, но результата нет.

[lpetr2]

Аналогичная проблема, пока копаюсь, но результата нет.

Тоже после обновления?

Если найдешь причину, то озвуч обязательно.

[Kos]

Ну, из приведённых Вами логов видно не так много:

• по логу Zabbix-сервера видно, что он был успешно запущен, через полминуты штатно остановлен (сигналом 15), и тут же снова была предпринята попытка его запустить (не дожидаясь окончания всех дочерних процессов и появления в логе записи "Zabbix Server stopped.");
• по логу демона systemd видно, что он пытался перезапустить сервер Zabbix, через 11 секунд решил, что тот не запустился успешно и попытался его перезапустить ещё раз;
• оба этих лога трудно сопоставлять, т.к. они за разное время.

Собственно, и всё. Какие при этом процессы остались висеть - неизвестно.

У меня есть подозрение, что дело не столько в сервере Zabbix, сколько в systemd. Я бы попробовал для начала изолировать проблему, запуская Zabbix-сервер вручную. Для этого нужно:

• остановить соответствующий сервис для демона systemd: "/bin/systemctl stop zabbix-server.service";
• можно даже его на время вообще отключить: "/bin/systemctl disable zabbix-server.service";
• убедиться, что нет других висящих процессов zabbix_server (если есть - убить);
• попытаться запустить его "руками": "/путь/к/вашему/серверу/zabbix_server -c /etc/zabbix/zabbix_server.conf".

Если не запускается - расследовать, почему (что пишет в stdout/stderr, в лог-файл). Если запускается - то разбираться уже с systemd (разглядывать его юнит-файл, проверять проставленные там пути, тайм-ауты и т.п.).

[asand3r]

Так у вас sigrerm присылается процессу заббикса, вот он и завершается.

[Viewer]

Я тоже на Centos 7 сейчас обновился с 3.2.3 до 3.2.4 и тоже сервр не стартанул.
В логи ничего нового не писал, включил дебаг=5 , перезапустил , тоже в логах пусто.
Откатился назад до 3.2.3

[lpetr2]

Я тоже на centos 7 сейчас обновился с 3.2.3 до 3.2.4 и тоже сервр не стартанул.
В логи ничего нового не писал, включил дебаг=5 , перезапустил , тоже в логах пусто.
Откатился назад до 3.2.3

Откат нормально прошел?

И как проводил откат?

[Viewer]

Откат нормально прошел?

И как проводил откат?

Сохранил конфиги из /etc/zabbix (на самом деле необязательно - они переименовываются)
yum remove zabbix-server-mysql
yum install zabbix-server-mysql-3.2.3-1.el7
Вернул конфиги, запустил.

Файлы в директориях externalscripts и alertscripts при удалении пакета остались на месте.

[lpetr2]

Тоже откатил на 3.2.3. Не понятно почему на новой версии не хочет запускаться.

Не стал копаться в systemd , так как врят ли с ним что-то. Ведь на ранней версии все прекрасно запускается. Никаких умолчательных параметров в самой службе systemd после установки Centos 7 не проводил.

Всегда проводил обновления zabbix(а) аналогичным способом. Аж с версии 3.0. Проблем никогда не было. А тут после обновы на тебе.

Может разработчики что-то смогут объяснить?
Есть вообще подтверждения об успешном обновлении на Centos 7. Если да то как обновлялись?

[kzaripov]

Сохранил конфиги из /etc/zabbix (на самом деле необязательно - они переименовываются)
yum remove zabbix-server-mysql
yum install zabbix-server-mysql-3.2.3-1.el7
Вернул конфиги, запустил.

Файлы в директориях externalscripts и alertscripts при удалении пакета остались на месте.

хватило
yum downgrade zabbix-*
все откатило

[lpetr2]

Смотрю 2 часа назад обновился репозиторий. Появился ещё один пакет с новым индексом. Никто не пробовал повторно обновиться?

[ ] zabbix-server-pgsql-3.2.4-1.el7.x86_64.rpm 2017-03-02 07:38 1.8M
[ ] zabbix-server-pgsql-3.2.4-2.el7.x86_64.rpm 2017-03-02 17:26 1.8M

[Vadim Puzynia]

Да, теперь все работает, можно ставить (я только что обкатал), но про бекапы забывать не стоит.

[lpetr2]

Тоже обновился. Проблем не возникло. Разработчики видимо исправили все.