Zabbix Webhook Error "Sending failed: Microsoft.Azure.Cosmos.Table.StorageException"

[ZabTheo]

Has been working great for a couple weeks. This has just popped up. Is it a Zabbix or my Teams causing the issue?


Sending failed: Microsoft.Azure.Cosmos.Table.StorageException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

[ZabTheo]

Any suggestions whether this is a known Zabbix issue, or related to our Teams instance? Thanks!

[Di0li]

Hi,

I believe that the issue has occurred with us within the same timeframe. I have the exact same error and I also don't know why, as everything was functioning perfectly until now. Perhaps it's due to a change by Microsoft?

[ZabTheo]

Hi,

I believe that the issue has occurred with us within the same timeframe. I have the exact same error and I also don't know why, as everything was functioning perfectly until now. Perhaps it's due to a change by Microsoft?

It seems to be an Azure error. I think the Zabbix Webhook is hosted in Azure and theres an issue with it. Hopefully someone from Zabbix can jump in and shed some light on it.

[cyber]

I think the Zabbix Webhook is hosted in Azure and theres an issue with it.

Zabbix webhook is a script in your server.. it is not hosted in Azure...

[ZabTheo]

Zabbix webhook is a script in your server.. it is not hosted in Azure...

I will investigate further, but we have multiple Zabbix instances in different data centers and they all went down at the same time.

[cyber]

The reason can be on MS side, as @Di0li suggested. Some API version changes etc.. As error message contains words like "authenticate" and "Authorization", you probably should investigate that area, how your user is authenticated, how headers etc are set..
And in the end it might need adjustments also in official packages.. If MS has decided to change something..

[ZabTheo]

The reason can be on MS side, as @Di0li suggested. Some API version changes etc.. As error message contains words like "authenticate" and "Authorization", you probably should investigate that area, how your user is authenticated, how headers etc are set..
And in the end it might need adjustments also in official packages.. If MS has decided to change something..

I'm not sure what to check to be honest. I reinstalled the Zabbx Webhook app in Teams, and updated the URL. Still nothing
I then installed the MS 'Incoming Webhook' app, and replaced the URL with that one. Still nothing.

I understand what you are saying about Authentication text in the error, but with webhooks, we aren't authing any user. It's just a URL.

We have switched back to email alerts (which feed in to the same Teams channels) and thats working ok for the time being.

I am not sure what else I can check to be honest.

[Di0li]

ZabTheo I can fully relate to what you wrote.I have taken the same steps, but the error still persists. It's quite strange, to be honest.

[crewer]

Hi all, I can confirm the same situation. It has stopped working around 2023-05-15.
Have somebody more information? It is quite crucial for me.
Thank you in advance!

[ZabTheo]

I would hope Zabbix would provide an update or some sort of status on this situation.
I don't think there been any announcement from them on it.

[Evgenii Kravchenko]

Hi all,
I have the same situation, but I found a workaround: use the "Incoming Webhook" connector instead of "Zabbix Webhook".
After that, notifications work again.

[Di0li]

Evgenii Kravchenko thanks for sharing this workaround! I did not think about it!

[ZabTheo]

Hi all,
I have the same situation, but I found a workaround: use the "Incoming Webhook" connector instead of "Zabbix Webhook".
After that, notifications work again.

I tried this, and the notifications still didn't come.
I will test it again. Thanks for the heads up.

[ZabTheo]

Just had an alert come through via Zabbix Webhook!

EDIT: It's been dead since that one webhook alert came through. Same error as before.