vmware-fqdn stopped creating hosts
Template: vmware fqdn, was working, but now doesn't add/remove hosts to zabbix.
I'm running Zabbix 6.4. vCenter 6.7.
This has been running for several years and working perfectly, this is not a new installation. But sometime in the last few months, Zabbix stopped updating VM's. New VM's are not being created or deleted.
All of the other monitoring appears to be working.
First thing I tried was deleting VMware FQDN, unlink and clear.
I ran housekeeping. I waited an hour. Then I added it back. This made no difference.
I have updated / rebooted the zabbix server.
I checked that the SOAP server is listening at https://vcenter.mydomain.local/sdk/vimService.wsdl, and it is.
I verified the service account has not changed.
The 2 collectors have started.
In zabbix_server.conf I have:
Then I checked the zabbix_server.log.
# zabbix_server -R log_level_increase="vmware collector"
I see that the collector has found a host called "Xanadu" which should be added to Zabbix, but it never adds it. Also, other hosts that I deleted continue to be displayed in Zabbix.
I don't know exactly when this broke, but looking at the current list of hosts in Zabbix I estimate it was a few months ago.
Everything else is working perfectly. This USED TO work. I have 39 other hosts that were created fine, until it broke.
I have checked everything I can think of, including reinstalling vmware fqdn. Can anyone give advice on what to do next?
I'm running Zabbix 6.4. vCenter 6.7.
This has been running for several years and working perfectly, this is not a new installation. But sometime in the last few months, Zabbix stopped updating VM's. New VM's are not being created or deleted.
All of the other monitoring appears to be working.
First thing I tried was deleting VMware FQDN, unlink and clear.
I ran housekeeping. I waited an hour. Then I added it back. This made no difference.
I have updated / rebooted the zabbix server.
I checked that the SOAP server is listening at https://vcenter.mydomain.local/sdk/vimService.wsdl, and it is.
I verified the service account has not changed.
The 2 collectors have started.
Code:
299981:20240206:152314.701 Starting Zabbix Server. Zabbix 6.4.11 (revision fb6439a26e3). 299981:20240206:152314.701 ****** Enabled features ****** 299981:20240206:152314.701 SNMP monitoring: YES 299981:20240206:152314.702 IPMI monitoring: YES 299981:20240206:152314.702 Web monitoring: YES 299981:20240206:152314.702 VMware monitoring: YES 299981:20240206:152314.702 SMTP authentication: YES 299981:20240206:152314.702 ODBC: YES 299981:20240206:152314.702 SSH support: YES 299981:20240206:152314.702 IPv6 support: YES 299981:20240206:152314.702 TLS support: YES 299981:20240206:152314.702 ****************************** 300049:20240206:152315.903 server #24 started [vmware collector #1] 300050:20240206:152315.905 server #25 started [vmware collector #2]
Code:
StartVMwareCollectors=2 VMwareFrequency=60 VMwarePerfFrequency=60 VMwareCacheSize=32M VMwareTimeout=120
Then I checked the zabbix_server.log.
# zabbix_server -R log_level_increase="vmware collector"
I see that the collector has found a host called "Xanadu" which should be added to Zabbix, but it never adds it. Also, other hosts that I deleted continue to be displayed in Zabbix.
Code:
300050:20240206:152417.426 End of vmware_service_get_vm_data():SUCCEED 300050:20240206:152417.434 In vmware_service_get_vm_folder() folder id:'group-v22' 300050:20240206:152417.434 End of vmware_service_get_vm_folder(): vm folder:vm 300050:20240206:152417.435 In vmware_vm_get_nic_devices() 300050:20240206:152417.435 End of vmware_vm_get_nic_devices() found:1 300050:20240206:152417.435 In vmware_vm_get_disk_devices() 300050:20240206:152417.437 End of vmware_vm_get_disk_devices() found:2 300050:20240206:152417.437 In vmware_vm_get_file_systems() 300050:20240206:152417.438 End of vmware_vm_get_file_systems() found:3 300050:20240206:152417.438 In vmware_vm_get_custom_attrs() 300050:20240206:152417.438 End of vmware_vm_get_custom_attrs() attributes:0 300050:20240206:152417.438 In vmware_service_get_alarms_data(), func_parent:'vmware_service_create_vm' 300050:20240206:152417.439 End of vmware_service_get_alarms_data() func_parent:'vmware_service_create_vm' found:0 total:7 300050:20240206:152417.439 End of vmware_service_create_vm():SUCCEED 300050:20240206:152417.439 In vmware_service_create_vm() vmid:'vm-1457' 300050:20240206:152417.439 In vmware_service_get_vm_data() vmid:'vm-1457' 300050:20240206:152417.446 vmware_service_get_vm_data() SOAP response: <?xml version="1.0" encoding="UTF-8"?> <soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <soapenv:Body> <RetrievePropertiesExResponse xmlns="urn:vim25"><returnval><objects><obj type="VirtualMachine">vm-1457</obj><propSet><name>availableField</name><val xsi:type="ArrayOfCustomFieldDef"></val></propSet><propSet><name>config.hardware</name><val xsi:type="VirtualHardware"><numCPU>8</numCPU><numCoresPerSocket>4</numCoresPerSocket><memoryMB>14336</memoryMB><virtualICH7MPresent>false</virtualICH7MPresent><virtualSMCPresent>false</virtualSMCPresent><device xsi:type="VirtualIDEController"><key>200</key><deviceInfo><label>IDE 0</label><summary>IDE 0</summary></deviceInfo><busNumber>0</busNumber></device><device xsi:type="VirtualIDEController"><key>201</key><deviceInfo><label>IDE 1</label><summary>IDE 1</summary></deviceInfo><busNumber>1</busNumber></device><device xsi:type="VirtualPS2Controller"><key>300</key><deviceInfo><label>PS2 controller 0</label><summary>PS2 controller 0</summary></deviceInfo><busNumber>0</busNumber><device>600</device><device>700</device></device><device xsi:type="VirtualPCIController"><key>100</key><deviceInfo><label>PCI controller 0</label><summary>PCI controller 0</summary></deviceInfo><busNumber>0</busNumber><device>500</device><device>12000</device><device>14000</device><device>1000</device><device>15000</device><device>4000</device><device>4001</device></device><device xsi:type="VirtualSIOController"><key>400</key><deviceInfo><label>SIO controller 0</label><summary>SIO controller 0</summary></deviceInfo><busNumber>0</busNumber><device>8000</device></device><device xsi:type="VirtualKeyboard"><key>600</key><deviceInfo><label>Keyboard </label><summary>Keyboard</summary></deviceInfo><controllerKey>300</controllerKey><unitNumber>0</unitNumber></device><device xsi:type="VirtualPointingDevice"><key>700</key><deviceInfo><label>Pointing device</label><summary>Pointing device; Device</summary></deviceInfo><backing xsi:type="VirtualPointingDeviceDeviceBackingInfo"> <deviceName></deviceName><useAutoDetect>false</useAutoDetect><hostPointingDevice>autodetect</hostPointingDevice></backing><controllerKey>300</controllerKey><unitNumber>1</unitNumber></device><device xsi:type="VirtualMachineVideoCard"><key>500</key><deviceInfo><label>Video card </label><summary>Video card</summary></deviceInfo><controllerKey>100</controllerKey><unitNumber>0</unitNumber><videoRamSizeInKB>8192</videoRamSizeInKB><numDisplays>1</numDisplays><useAutoDetect>false</useAutoDetect><enable3DSupport>false</enable3DSupport><use3dRenderer>automatic</use3dRenderer><graphicsMemorySizeInKB>262144</graphicsMemorySizeInKB></device><device xsi:type="VirtualMachineVMCIDevice"><key>12000</key><deviceInfo><label>VMCI device</label><summary>Device on the virtual machine PCI bus that provides support for the virtual machine communication interface</summary></deviceInfo><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>32</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>17</unitNumber><id>-1505665539</id><allowUnrestrictedCommunication>false</allowUnrestrictedCommunication><filterEnable>true</filterEnable></device><device xsi:type="VirtualUSBXHCIController"><key>14000</key><deviceInfo><label>USB xHCI controller </label><summary>USB xHCI controller</summary></deviceInfo><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>224</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>23</unitNumber><busNumber>0</busNumber><autoConnectDevices>false</autoConnectDevices></device><device xsi:type="VirtualLsiLogicSASController"><key>1000</key><deviceInfo><label>SCSI controller 0</label><summary>LSI Logic SAS</summary></deviceInfo><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>160</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>3</unitNumber><busNumber>0</busNumber><device>2000</device><hotAddRemove>true</hotAddRemove><sharedBus>noSharing</sharedBus><scsiCtlrUnitNumber>7</scsiCtlrUnitNumber></device><device xsi:type="VirtualAHCIController"><key>15000</key><deviceInfo><label>SATA controller 0</label><summary>AHCI</summary></deviceInfo><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>33</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>24</unitNumber><busNumber>0</busNumber><device>16000</device></device><device xsi:type="VirtualCdrom"><key>16000</key><deviceInfo><label>CD/DVD drive 1</label><summary>ISO [PowerVault] contentlib-f76a856f-dbbc-4519-9c84-eaa4c8af8707/774a1388-54cf-4e5e-88da-883ba0456958/gparted-live-1.3.1-1-amd64_4f1ca917-9e43-4593-b2ee-730ce92caf40.iso</summary></deviceInfo><backing xsi:type="VirtualCdromIsoBackingInfo"><fileName>[PowerVault] contentlib-f76a856f-dbbc-4519-9c84-eaa4c8af8707/774a1388-54cf-4e5e-88da-883ba0456958/gparted-live-1.3.1-1-amd64_4f1ca917-9e43-4593-b2ee-730ce92caf40.iso</fileName><datastore type="Datastore">datastore-1445</datastore></backing><connectable><startConnected>false</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><controllerKey>15000</controllerKey><unitNumber>0</unitNumber></device><device xsi:type="VirtualDisk"><key>2000</key><deviceInfo><label>Hard disk 1</label><summary>268,435,456 KB</summary></deviceInfo><backing xsi:type="VirtualDiskFlatVer2BackingInfo"><fileNam e>[dozer] Xanadu/Xanadu.vmdk</fileName><datastore type="Datastore">datastore-1444</datastore><backingObjectId></backingObjectId><diskMode>persistent</diskMode><split>false</split><writeThrough>false</writeThrough><thinProvisioned>true</thinProvisioned><uuid>6000C29e-cf12-b6a5-0e78-985f1d2d93d9</uuid><contentId>af10a9b9235cf1b8db6d2645ed97d6ae</contentId><digestEnabled>false</digestEnabled><sharing>sharingNone</sharing></backing><controllerKey>1000</controllerKey><unitNumber>0</unitNumber><capacityInKB>268435456</capacityInKB><capacityInBytes>274877906944</capacityInBytes><shares><shares>1000</shares><level>normal</level></shares><storageIOAllocation><limit>-1</limit><shares><shares>1000</shares><level>normal</level></shares><reservation>0</reservation></storageIOAllocation><diskObjectId>24-2000</diskObjectId></device><device xsi:type="VirtualFloppy"><key>8000</key><deviceInfo><label>Floppy drive 1</label><summary>Remote</summary></deviceInfo><backing xsi:type="VirtualFloppyRemoteDeviceBackingInfo"><d eviceName></deviceName><useAutoDetect>false</useAutoDetect></backing><connectable><startConnected>false</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><controllerKey>400</controllerKey><unitNumber>0</unitNumber></device><device xsi:type="VirtualVmxnet3"><key>4000</key><deviceInfo><label>Network adapter 1</label><summary>DMZ LAG</summary></deviceInfo><backing xsi:type="VirtualEthernetCardNetworkBackingInfo">< deviceName>DMZ LAG</deviceName><useAutoDetect>false</useAutoDetect><network type="Network">network-1448</network></backing><connectable><startConnected>true</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>192</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>7</unitNumber><addressType>assigned</addressType><macAddress>00:50:56:86:ea:89</macAddress><wakeOnLanEnabled>true</wakeOnLanEnabled><resourceAllocation><reservation> 0</reservation><share><shares>50</shares><level>normal</level></share><limit>-1</limit></resourceAllocation><uptCompatibilityEnabled>true</uptCompatibilityEnabled></device><device xsi:type="VirtualVmxnet3"><key>4001</key><deviceInfo><label>Network adapter 2</label><summary>LAN</summary></deviceInfo><backing xsi:type="VirtualEthernetCardNetworkBackingInfo">< deviceName>LAN</deviceName><useAutoDetect>false</useAutoDetect><network type="Network">network-1447</network></backing><connectable><startConnected>true</startConnected><allowGuestControl>true</allowGuestControl><connected>false</connected><status>untried</status></connectable><slotInfo xsi:type="VirtualDevicePciBusSlotInfo"><pciSlotNum ber>256</pciSlotNumber></slotInfo><controllerKey>100</controllerKey><unitNumber>8</unitNumber><addressType>assigned</addressType><macAddress>00:50:56:91:7e:1d</macAddress><wakeOnLanEnabled>true</wakeOnLanEnabled><resourceAllocation><reservation> 0</reservation><share><shares>50</shares><level>normal</level></share><limit>-1</limit></resourceAllocation><uptCompatibilityEnabled>true</uptCompatibilityEnabled></device></val></propSet><propSet><name>config.instanceUuid</name><val xsi:type="xsd:string">50386cf0-9652-7c02-ec43-a6b62c9e638d</val></propSet><propSet><name>config.uuid</name><val xsi:type="xsd:string">423822e5-5238-7616-21eb-ec6fa6415dfd</val></propSet><propSet><name>customValue</name><val xsi:type="ArrayOfCustomFieldValue"></val></propSet><propSet><name>datastore</name><val xsi:type="ArrayOfManagedObjectReference"><ManagedO bjectReference type="Datastore" xsi:type="ManagedObjectReference">datastore-1444</ManagedObjectReference><ManagedObjectReference type="Datastore" xsi:type="ManagedObjectReference">datastore-1445</ManagedObjectReference></val></propSet><propSet><name>guest.disk</name><val xsi:type="ArrayOfGuestDiskInfo"></val></propSet><propSet><name>guest.guestState</name><val xsi:type="xsd:string">notRunning</val></propSet><propSet><name>guest.toolsRunningStatus</name><val xsi:type="xsd:string">guestToolsNotRunning</val></propSet><propSet><name>guest.toolsVersion</name><val xsi:type="xsd:string">12294</val></propSet><propSet><name>layoutEx</name><val xsi:type="VirtualMachineFileLayoutEx"><file><key>0 </key><name>[dozer] Xanadu/Xanadu.vmx</name><type>config</type><size>4466</size><uniqueSize>4466</uniqueSize><accessible>true</accessible></file><file><key>1</key><name>[dozer] Xanadu/Xanadu.vmxf</name><type>extendedConfig</type><size>3837</size><uniqueSize>3837</uniqueSize><accessible>true</accessible></file><file><key>2</key><name>[dozer] Xanadu/Xanadu.nvram</name><type>nvram</type><size>8684</size><uniqueSize>8684</uniqueSize><accessible>true</accessible></file><file><key>3</key><name>[dozer] Xanadu/Xanadu.vmsd</name><type>snapshotList</type><size>0</size><uniqueSize>0</uniqueSize><accessible>true</accessible></file><file><key>4</key><name>[dozer] Xanadu/Xanadu.vmdk</name><type>diskDescriptor</type><size>0</size><uniqueSize>0</uniqueSize><accessible>true</accessible></file><file><key>5</key><name>[dozer] Xanadu/Xanadu-flat.vmdk</name><type>diskExtent</type><size>27087863808</size><uniqueSize>27087863808</uniqueSize><accessible>true</accessible></file><file><key>6</key><name>[dozer] Xanadu/vmware-420.log</name><type>log</type><size>312168</size><uniqueSize>312168</uniqueSize><accessible>true</accessible></file><file><key>7</key><name>[dozer] Xanadu/vmware-421.log</name><type>log</type><size>312174</size><uniqueSize>312174</uniqueSize><accessible>true</accessible></file><file><key>8</key><name>[dozer] Xanadu/vmware-422.log</name><type>log</type><size>312827</size><uniqueSize>312827</uniqueSize><accessible>true</accessible></file><file><key>9</key><name>[dozer] Xanadu/vmware-423.log</name><type>log</type><size>307668</size><uniqueSize>307668</uniqueSize><accessible>true</accessible></file><file><key>10</key><name>[dozer] Xanadu/vmware-418.log</name><type>log</type><size>311099</size><uniqueSize>311099</uniqueSize><accessible>true</accessible></file><file><key>11</key><name>[dozer] Xanadu/vmware-419.log</name><type>log</type><size>314724</size><uniqueSize>314724</uniqueSize><accessible>true</accessible></file><file><key>12</key><name>[dozer] Xanadu/vmware.log</name><type>log</type><size>314115</size><uniqueSize>314115</uniqueSize><accessible>true</accessible></file><file><key>13</key><name>[dozer] Xanadu/Xanadu-aux.xml</name><type>snapshotManifestList</type><size>0</size><uniqueSize>0</uniqueSize><accessible>true</accessible></file><disk><key>2000</key><chain><fileKey>4</fileKey><fileKey>5</fileKey></chain></disk><timestamp>2024-02-06T19:49:37.718186Z</timestamp></val></propSet><propSet><name>parent</name><val type="Folder" xsi:type="ManagedObjectReference">group-v22</val></propSet><propSet><name>resourcePool</name><val type="ResourcePool" xsi:type="ManagedObjectReference">resgroup-1442</val></propSet><propSet><name>summary.config.memorySizeMB </name><val xsi:type="xsd:int">14336</val></propSet><propSet><name>summary.config.name</name><val xsi:type="xsd:string">Xanadu</val></propSet><propSet><name>summary.config.numCpu</name><val xsi:type="xsd:int">8</val></propSet><propSet><name>summary.quickStats.balloone dMemory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.compress edMemory</name><val xsi:type="xsd:long">0</val></propSet><propSet><name>summary.quickStats.guestMem oryUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.hostMemo ryUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.overallC puUsage</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.privateM emory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.sharedMe mory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.swappedM emory</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.quickStats.uptimeSe conds</name><val xsi:type="xsd:int">0</val></propSet><propSet><name>summary.runtime.consolidati onNeeded</name><val xsi:type="xsd:boolean">false</val></propSet><propSet><name>summary.runtime.powerState</name><val xsi:type="VirtualMachinePowerState">poweredOff</val></propSet><propSet><name>summary.storage.committed</name><val xsi:type="xsd:long">27090065570</val></propSet><propSet><name>summary.storage.uncommitted </name><val xsi:type="xsd:long">263055872631</val></propSet><propSet><name>summary.storage.unshared</name><val xsi:type="xsd:long">27087863808</val></propSet><propSet><name>triggeredAlarmState</name><val xsi:type="ArrayOfAlarmState"></val></propSet></objects><objects><obj type="Folder">group-v22</obj><propSet><name>name</name><val xsi:type="xsd:string">vm</val></propSet><propSet><name>parent</name><val type="Datacenter" xsi:type="ManagedObjectReference">datacenter-21</val></propSet></objects></returnval></RetrievePropertiesExResponse> </soapenv:Body> </soapenv:Envelope>
I don't know exactly when this broke, but looking at the current list of hosts in Zabbix I estimate it was a few months ago.
Everything else is working perfectly. This USED TO work. I have 39 other hosts that were created fine, until it broke.
I have checked everything I can think of, including reinstalling vmware fqdn. Can anyone give advice on what to do next?
Comment