Disappering hosts

[Calypso]

Since a couple of weeks I notice that some of my hosts are disappearing from Zabbix.

The most strange part of it is that it seem to be the same hosts over and over again; for instance, in a list of 40 network switches, 3 of them are missing. When I add them again, they appear, they seem to gather data, and after a couple of hours they're gone again.

I've checked the database (when added I wrote down the hostid's), and they are really gone from the database. Also all items, except for the pingsec, are deleted. For some strange reason that simplecheck is still present in the items table.

Setup:
Zabbix 1.6.6 with 2 proxies (also 1.6.6) on MySQL, about 750 monitored hosts with 53000 items.

I also noticed that it doesn't seem to be proxy related, since I once added a host without the proxy, and it was still gone after some time. When I rename another host to the name and IP of the "missing" host, and then I clone that renamed one to the original name, the previously missing host disappears again (i.e. the hostid that is the oldest of the 2).

Didn't notice the behaviour before going to 1.6.6 - I don't like to downgrade, but if everything else fails I'm going to try that in a week or so.

[richlv]

do you have distributed monitoring with nodes configured ?
after the host disappears, pingsec item is still in the db, and it refers to a nonexistent host id ?
only single frontend is running against the same db ? only single server/proxy is running against the same db ? is discovery being used ?

[Calypso]

do you have distributed monitoring with nodes configured ?

No; I only use proxies

after the host disappears, pingsec item is still in the db, and it refers to a nonexistent host id ?

True

only single frontend is running against the same db ?

Yes

only single server/proxy is running against the same db ?

Yes, there is only one server running against the database; the proxies have their own database (local) and communicate through the server.

is discovery being used ?

Yes, but only for discovery, not for removal

[richlv]

after the host disappears, pingsec item is still in the db, and it refers to a nonexistent host id ?

True

well, this is quite weird. whatever is removing those hosts for you somehow manages to leave orphaned items...
is this removal visible in administration -> audit ?

is discovery being used ?

Yes, but only for discovery, not for removal

yeah, i was just thinking about what could write to the db periodically. might the disappearing hosts match any discovery conditions ?
can you try disabling discovery, adding those hosts and waiting for whatever time it usually takes for the hosts to disappear.
if the hosts stay visible, enable discovery and see whether that nukes the hosts. if not... it's something else

[Calypso]

With all discovery tasks disabled the hosts are still present. I'll now gradually start discovery tasks to see which one causes the problem. Or not ofcourse