Snapshot of Zabbix in test environment without sending alerts

[digitaladdictions]

I am looking for how to safely bring up a copy of our Zabbix server and DB in a test environment to test major version upgrades or DB engine updates without the potential of accidentally sending alerts.

I am worried for example if an agent is configured with active checks that when we bring up a new instance pointed at a copy of the database that those active checks won't be received (since the agents are pointed at the real production server) and that the test server will send notifications because from its perspective the agent never checked-in.

I am aware of the ability to enable maintenance mode with no data collected which would seem to be what I would want in this case but this line from the documentation concerns me:

"Zabbix server must be running during maintenance. Timer processes are responsible for switching host status to/from maintenance at 0 seconds of every minute. A proxy will always collect data regardless of the maintenance type (including “no data” maintenance). The data is later ignored by the server if 'no data collection' is set."

Does this mean the Zabbix server would reach out and reconfigure the agent? I obviously do not want that since then the agent would not communicate an actual problem to the production server.

The way I am envisioning this is that I would:

1. Bring up snapshots of the EC2 and RDS instances that Zabbix runs on
2. Connect to the EC2 instance and stop the zabbix-server process
3. Update the web UI component to point at the new test DB.
4. Configure maintenance mode with no data collection
5. Update the Zabbix server to point at the new RDS instance after maintenance mode is already enabled
6. Start the Zabbix server

But like I said the fact that the documentation says the Zabbix server must be running concerns me.

[tim.mooney]

I am looking for how to safely bring up a copy of our Zabbix server and DB in a test environment to test major version upgrades or DB engine updates without the potential of accidentally sending alerts.

Assuming that your alert conditions are written so that alerts aren't sent a problem is detected with something in maintenance, then putting everything in maintenance would be one way to accomplish this.

For a test environment where I definitely did not want any alerts to escape, I would use a combination of firewall rules to block outgoing connections to things like SMTP, your paging gateway, etc. -- whatever connections Zabbix has to make to hand off a message for an alert, having your test environment use a host-based firewall rule to drop those packets.

In addition, in some cases you may be using commands that are installed on your Zabbix server especially to help with alerting, for example a notifier script that handles the details of connecting to a Slack channel or that connects to your paging service. For extra safety, you can replace those types of commands with scripts that accept their input but do nothing within, so that your alerting notifier helpers just throw the messages away, rather than send them on.

[cyber]

Server is not doing any reconfig of agents, ever. Host status is set to maintenance in DB...Agent will continue as usual, just its data will be ignored.