No data coming from servers connected to Zabbix Proxy Server. No availability.

[ZabTheo]

I have a dozen servers in a DMZ which connect to a Zabbix Proxy server.
After many months, I realised those servers in the DMZ have no availability status for the Zabbix agent showing on the Zabbix server.
I have done some troubleshooting with my network team and there doesn't appear to be any restrictions on the network side.
Traffic is able to flow from the DMZ to the internal zabbix server on ports 10050 and 10051.

I noticed two things. Firstly, my internal zabbix server IP address was not listed in the "StatsAllowedIP" option on the conf file on the proxy server. I have added that now.
Secondly, I noticed on the Proxy server there were nothing in the zabbix logs, so I tried to restart the zabbix-proxy.service but it failed to come back up, so I restarted the proxy server.
Since restarting the proxy server, I now have log file entries.

Here is the log (I have just noticed "[Z3005] query failed: [0] database disk image is malformed", maybe this is the issue.

1243:20240322:171502.781 Got signal [signal:15(SIGTERM),sender_pid:43936,sender_uid:0,r eason:0]. Exiting ...
1224:20240322:173205.198 Starting Zabbix Proxy (active) [xx-DMZZABBIX-01]. Zabbix 5.0.38 (revision 0934551b8aa).
1224:20240322:173205.199 **** Enabled features ****
1224:20240322:173205.199 SNMP monitoring: YES
1224:20240322:173205.199 IPMI monitoring: YES
1224:20240322:173205.199 Web monitoring: YES
1224:20240322:173205.199 VMware monitoring: YES
1224:20240322:173205.199 ODBC: YES
1224:20240322:173205.199 SSH support: YES
1224:20240322:173205.199 IPv6 support: YES
1224:20240322:173205.199 TLS support: YES
1224:20240322:173205.199 **************************
1224:20240322:173205.199 using configuration file: /etc/zabbix/zabbix_proxy.conf
1224:20240322:173205.242 current database version (mandatory/optional): 05000000/05000005
1224:20240322:173205.242 required mandatory version: 05000000
1224:20240322:173205.243 proxy #0 started [main process]
1240:20240322:173205.244 proxy #1 started [configuration syncer #1]
2282:20240322:173206.852 proxy #3 started [trapper #2]
2281:20240322:173206.856 proxy #2 started [trapper #1]
2301:20240322:173206.858 proxy #21 started [task manager #1]
2285:20240322:173206.859 proxy #6 started [trapper #5]
2306:20240322:173206.862 proxy #26 started [poller #5]
2286:20240322:173206.864 proxy #7 started [preprocessing manager #1]
2308:20240322:173206.865 proxy #28 started [icmp pinger #1]
2300:20240322:173206.865 proxy #20 started [self-monitoring #1]
2292:20240322:173206.865 proxy #12 started [data sender #1]
2294:20240322:173206.867 proxy #14 started [http poller #1]
2296:20240322:173206.868 proxy #16 started [history syncer #1]
2304:20240322:173206.868 proxy #24 started [poller #3]
2293:20240322:173206.868 proxy #13 started [housekeeper #1]
2295:20240322:173206.868 proxy #15 started [discoverer #1]
2284:20240322:173206.870 proxy #5 started [trapper #4]
2288:20240322:173206.872 proxy #8 started [preprocessing worker #1]
2289:20240322:173206.872 proxy #9 started [preprocessing worker #2]
2297:20240322:173206.872 proxy #17 started [history syncer #2]
2298:20240322:173206.873 proxy #18 started [history syncer #3]
2299:20240322:173206.873 proxy #19 started [history syncer #4]
2290:20240322:173206.874 proxy #10 started [preprocessing worker #3]
2302:20240322:173206.874 proxy #22 started [poller #1]
2305:20240322:173206.877 proxy #25 started [poller #4]
2291:20240322:173206.879 proxy #11 started [heartbeat sender #1]
2303:20240322:173206.881 proxy #23 started [poller #2]
2283:20240322:173206.884 proxy #4 started [trapper #3]
2307:20240322:173206.886 proxy #27 started [unreachable poller #1]
2292:20240322:173206.905 [Z3005] query failed: [0] database disk image is malformed [select id,itemid,clock,ns,timestamp,source,severity,value ,logeventid,state,lastlogsize,mtime,flags from proxy_history where id>1404556323 order by id limit 1000]
1240:20240322:173206.916 received configuration data from server at "xx-zabbix.REDACTED.net", datalen 423383
2292:20240322:173206.958 [Z3005] query failed: [0] database disk image is malformed [select id,itemid,clock,ns,timestamp,source,severity,value ,logeventid,state,lastlogsize,mtime,flags from proxy_history where id>1404556323 order by id limit 1000]
2292:20240322:173206.958 database is down: retrying in 10 seconds
2296:20240322:173208.106 [Z3005] query failed: [0] database disk image is malformed [insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109687,1711128727,940754545,'99.808598',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (129907,1711128727,941262925,'5.504759',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147427,1711128727,944049230,'99.992609',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (149227,1711128727,945672819,'0.000000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (149287,1711128727,947196181,'99.997330',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131347,1711128727,948748082,'36958208.000000',0,1 711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131887,1711128727,948884929,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165967,1711128727,951863294,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131227,1711128727,951961976,'1058619392.000000',0 ,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109207,1711128727,955809604,'0.000000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (150427,1711128727,958184327,'73.019820',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (122047,1711128727,960085463,'2',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (84487,1711128727,963678321,'1',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (84667,1711128727,969536430,'5.135114',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109267,1711128727,974342795,'0.050000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (166027,1711128727,975119265,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165007,1711128727,980067408,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (110047,1711128727,986379878,'1063256064',0,171112 8728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164947,1711128727,991076683,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (132307,1711128727,991242122,'0.000000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165307,1711128727,995937420,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (84727,1711128727,996065421,'',2,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (146827,1711128727,997899895,'0.000000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (128887,1711128727,998281951,'0.050000',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147067,1711128728,906053,'99.787855',0,1711128728 );
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147127,1711128728,2540467,'1063256064',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147187,1711128728,4965820,'0.000000',0,1711128728 );
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164827,1711128728,5106617,'2',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147428,1711128728,9847847,'99.998474',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109688,1711128728,10116044,'99.946976',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (149228,1711128728,13138300,'0.068776',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (149288,1711128728,14878133,'99.999814',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109568,1711128728,15250133,'4215795712',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131348,1711128728,16349170,'12301752.000000',0,17 11128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131888,1711128728,17407091,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109148,1711128728,17603928,'4290768896',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131228,1711128728,19020005,'8.205019',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (131252,1711128728,19020005,'91.794981000000007',0 ,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165968,1711128728,20157442,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (150428,1711128728,22732247,'96.819527',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (122048,1711128728,23696983,'',2,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (122648,1711128728,27087459,'1',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (166028,1711128728,28890995,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109208,1711128728,29805827,'0.000000',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165008,1711128728,31842982,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164888,1711128728,44450588,'2',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (84668,1711128728,49150917,'3.187022',0,1711128728 );
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109268,1711128728,51989336,'0.000000',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164948,1711128728,56439296,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (109988,1711128728,63366630,'99.796950',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164828,1711128728,90380365,'[{"AdapterTypeId":0,"Description":"vmxnet3 Ethernet Adapter","Name":"vmxnet3 Ethernet Adapter","NetConnectionID":"Ethernet0","NetConnect ionStatus":2,"Speed":"10000000000"}]',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164924,1711128728,90380365,'10000000000',0,171112 8728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164926,1711128728,90380365,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164925,1711128728,90380365,'2',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (164831,1711128728,90380365,'{"data":[{"{#IFNAME}":"vmxnet3 Ethernet Adapter","{#IFDESCR}":"vmxnet3 Ethernet Adapter","{#IFALIAS}":"Ethernet0"}]}',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (70327,1711128728,93692942,'96.006516',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (70328,1711128728,93929439,'67.712260',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (110048,1711128728,95333955,'4284481536',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (165308,1711128728,95445520,'0',0,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (132308,1711128728,96251318,'0.000000',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (128888,1711128728,97258538,'0.000000',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (146828,1711128728,97395164,'0.000000',0,171112872 8);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (84728,1711128728,98074520,'',2,1711128728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (129908,1711128728,101161346,'3.327632',0,17111287 28);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147128,1711128728,101324714,'1063256064',0,171112 8728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147068,1711128728,101387199,'99.444580',0,1711128 728);
insert into proxy_history (itemid,clock,ns,value,flags,write_clock) values (147188,1711128728,102883155,'0.002778',0,17111287 28);
]
2293:20240322:180207.059 executing housekeeper
1240:20240322:183207.606 received configuration data from server at "xx-zabbix.REDACTED.net", datalen 423383

Still no latest data on Zabbix for these servers, and no zabbix agent availability.

Many thanks for your time!

[um3n]

Hey,
the database is correctly installed? Maybe just setup the database on the proxy again.
If that is already done, maybe you can check the database for the zabbix proxy.

Kind regards,

[ZabTheo]

I think so. We have two Zabbix proxy servers, 1 at each datacentre. They are installed and configured exactly the same apart of hostname and IP addresses. They connect through the same stretched DMZ. The second proxy server at the other datacentre works fine.

The proxy server is question has worked at some point because the servers have a couple of data entries under "latest data" but they haven't changed values in a very long time.

Are there any Zabbix DB troubleshooting checks I can perform? Thanks

[um3n]

Because it isn't working, you could try to drop the database and do a fresh install of it.
You could also check the mariadb logs (is it mariadb?) and see, if there is anything that seems off.

[ZabTheo]

My colleague fixed the DB as it was corrupt. Its now working. Thanks for your suggestion.