Trying to import a modified template. Running into database issues [Zabbix 6.4.15]

[mdella]

So we have a template in our production environment that we have been using under v6.4 for quite some time. We have a staging system where we have been making modifications to the template prior to moving to production. We have used this method for quite some time but this last time we got a huge suprise in trying to import the updated template. Specifically we are suddenly getting thousands of lines of the following:

Error in query [INSERT INTO housekeeper (housekeeperid,tablename,field,value) VALUES ('62212448','events','itemid','233152'),('62212449','history','itemid','233152'),('62212450','history_log','itemid','233152'),('62212451','history_str','itemid','233152'),('62212452','history_text','itemid','233152'),('62212453','history_uint','itemid','233152'),('62212454','trends','itemid','233152'),('62212455','trends_uint','itemid','233152'),('62212456','events','itemid','233153'),('62212457','history','itemid','233153'),('62212458','history_log','itemid','233153'),('62212459','history_str','itemid','233153'),('62212460','history_text','itemid','233153'),('62212461','history_uint','itemid','233153'),...

If we turn off housekeeping in the administrative panel, we get thousands of lines of different errors.

So, how do we debug this or try and figure out how to import our new template. We have months of development in the modified template so its non-trivial to go backwards and try to figure out a merge process or incremental update. It can be done, but then this starts to point us to the point that templates are in fact not always backwards compatable and that the combination of changes to make them non-compatable are... mystic...

Note: Most of our pre-production deployment systems are non-HA. The two production systems we run are both HA. All environments use the MariaDB 10.6 Galera Cluster system with an HAProxy process running on each Zabbix server to reach the DB heads.

Marcos Della
Data Center Architect
Nutanix, Inc.

(blah, posted in the wrong place and now I can't move or delete this)

[Markku]

I have to say that I have no idea where that error comes from. However, in the error sample there are itemids mentioned. Maybe you can check what those itemids are and if that leads to something?

templates are in fact not always backwards compatable

Templates are definitely not backwards compatible, in regards to Zabbix version. The version is saved in the beginning of the exported template. So you cannot run a preprod system with newer Zabbix version and expect to export+import anything from there. But maybe you meant some other compatibility?

Markku

[mdella]

I have to say that I have no idea where that error comes from. However, in the error sample there are itemids mentioned. Maybe you can check what those itemids are and if that leads to something?

Templates are definitely not backwards compatible, in regards to Zabbix version. The version is saved in the beginning of the exported template. So you cannot run a preprod system with newer Zabbix version and expect to export+import anything from there. But maybe you meant some other compatibility?

Markku

So there are literally more than 45 thousand entries in that one database error line. So spot checking them just leads to item after item of different hosts that use the template. The staging and production systems are almost always on the same version. They are always on the same major and minor versions (6.4.13 vs 6.4.15 for instance). And the templates are all tagged with the same version numbers. We have been doing this for quite some time and do have a lot of experience with this method of development/review/roll to production. The problem for us is that this is the first time we've ever encountered not being able to import a template like this.

Additional information: When "turning off" the housekeeping function on the production during the import attempt, we bypassed all of these errors just to run into another set. The general feeling here is that there was something added to the new template that "shifted" somehow how all the pointers are dereferenced. At the moment, we're trying to see if there is any way to go line by line and incrementally feed the changes into our production system knowing that during this type of change we will be mostly blind for the duration of the manual conversion (unfortunately we're not talking one or two changes, but 3 months worth of development work).

Just shows us that we need to figure out a more incremental method of template design and release. (BTW, this is in preparation for us releasing the template to the public for our product).

Marcos

[Markku]

Does your housekeeper error (above) say something more about the error? What about the database engine, does it show any errors?

If I understand the purpose of the housekeeper table correctly, it will contain the items that should be deleted in the upcoming housekeeper runs. So, what happens (as I see it) is, when you imported the template, Zabbix figured out that there are lots and lots of items that should now be deleted. But it encountered some error when adding those to-be-deleted items' information in the housekeeper table.

Does that hint you anything? First, why would Zabbix think that lots of things should be deleted when importing the template? Second, what could be causing the database insert problem?

Markku

[cyber]

I cannot even understand, what has housekeeper to do with template import.. :/ ? There is nothing related to housekeeper in templates (only if you count item history/trend keeping periods)....

[mdella]

So this was an existing template that we ended up heavily modifying to move from single snmp queries to the newer (6.4) walk entries. So lots of things in the template got changed. The issue we face is that this template is used on hundreds of devices and its non-trivial for us to try and delete the old (and lose all the data to date) just to use the new template. But since it was only modified, we thought it could be reimported (we were doing the changes and testing on a staging system and wanted to bring it back to production).

Maybe that had something to do with it all. Either way, we still haven't found a way without losing all prior data to bring the template back in. The problem we face is that it has the same UUID, etc so we can't just bring it in in parallel and move things from one to another. Waiting for our PO to go thru so that we can get official help on this as this has now been a show stopper for us for several weeks now.

[Markku]

After this additional information, sounds like a perfect case for Zabbix (or Zabbix partner) professional service. I'd assume they would also be able to formulate appropriate feature request as well, if the template import algorithm has something to improve.

Markku