Discussion thread for official integration with PagerDuty

[AlexL]

This thread is designed to provide grounds for discussion of the official Zabbix integration with PagerDuty.
The integration is available here: https://git.zabbix.com/projects/ZBX/...edia/pagerduty
Zabbix is always looking for ways to improve our services and to make our users happier.
We pride ourselves on doing our best each and every day, but we know that there is always something more to learn.
We would like to hear back from you to know what have you liked and what would you improve in integration.

[jashley]

I am an intermediate user of zabbix and have been using it since 1.8. Now running 4.4.5 and I am trying to get PagerDuty running the way I think it should run with Zabbix. The question I want to start with is: How many integrations can be set up from Zabbix to PagerDuty? I only seem to be able to set up one.

Also, the message sent to PagerDuty does not seem to have all of the macro information available in the Media Type.

[cloudops]

Make another webhook Media Type with the token for your different PD service.

You control which info you want to send to PD in your Action Operations.

[jashley]

Thanks for the reply..I have since found out that the documentation from PagerDuty has not kept up with what Zabbix has done. I do now have many integrations in PD. That being said... I have not been successful in seeing all of the macros available in Zabbix being sent to PD. When you look at the Media Type macros and what I can see PD is receiving I only see 5 or 6 of them. Do you know what I should modify so PD can get all of the Macros available?

[bilal.habib]

is it possible to include the Hostname in the summary of the PagerDuty alert? I tried

summary: params.eventname + ' : ' + params.hostname

but it did not change anything

[ddelgado]

Hello,
I am currently using Zabbix 3.4 and planning on updating to 5.0. Will the existing Pagerduty integration continue working in-place, or will I have to upgrade to the Pagerduty Webhook integration?

[rajiv_641]

i am trying to import media_pagerduty.yaml file in zabbix 5.0.9. the moment I try to import i get import failed error stating yaml format not supported

[bilal.habib]

I honestly have no clue. However, you could send a massage to the site's support and they could solve this doubt!

I actually got this working in the end, I just didn't upload the template with the overwrite option when i was testing

[kholliday]

Any official plans to enhance the JavaScript in the PagerDuty integration to retry a couple of times if http 200 or 202 result code is not met? I often run into http 429 rate limits, and I think that just causes Alerts to get 'dropped' by this integration.
I'd also be interested in any other comments about alternative workarounds or intended usage/configuration.

Edit: Found the Options tab which includes Concurrent sessions, Attempts, and Attempt interval fields. So I gather if response is not 202, it throws an error, which causes the alerter logic to retry it according to Options.

[kholliday]

Also to mention: I found that the older method of driving alerts to different PagerDuty ruleset or service endpoint keys without cloning the PagerDuty media type is better.

Instead of hardcoding token in media type field list, use {ALERT.SENDTO} to populate that field. Then just create a different User named to align with each desired endpoint, put endpoint key in the Send to field. An action is created for each desired destination, aligned with its corresponding token-containing User. Each action can be independently configured to forward different criteria of alerts to its target endpoint. These allow overlap if you want to fan out a given alert to multiple endpoints.

Hoping/expecting that key can be further obscured by storing it in a Zabbix Secrets in future.

[kkarney]

We are a Zabbix partner and have it installed at several customers as part of our MSP offering. We are looking to integrate an alerting app such as PagerDuty with Zabbix.

In my research, it seems like the nearest competitor to PagerDuty is iLert. Wondering if anyone has an opinion on which is the better choice?

I suppose that most folks on this thread are PagerDuty users, so that may be a silly question. But I figured I'd toss it out there in case anyone has done a compare and contrast... Thanks!

[wmckay85]

I posted elsewhere but I am having a problem with the integration. It worked back in May but I'm not sure why it's broken now: PagerDuty Webhook test works but fails to show up in pagerduty - ZABBIX Forums

[kwbarron]

I have just migrated and updated our Zabbix server from 5.0.11 (on CentOS 7) to 6.4.9 (on RHEL 8). It was previously running in Docker, but now is running on a VM for ease of admin.

We had the PD integration working on v5, but cannot get a successful test on v6. The error I get is this:

Details Media type test failed.

• PagerDuty notification failed : Error: cannot get URL: Couldn't connect to server.

I am making an assumption that URL it is referring to is our Zabbix URL (based on the yaml file I imported), and I have tried multiple iterations of the URL, even including https://IP/zabbix.

I am trying to get this up and running in production in the next 2 weeks, and I believe this is my final hurdle, so any assistance is greatly appreciated.

[kwbarron]

FYI for anyone that runs into this in the future. The fix is to add HTTPProxy in the Media Type. Zabbix apparently ignores the system wide proxy settings.

[lbm]

I have just migrated and updated our Zabbix server from 5.0.11 (on CentOS 7) to 6.4.9 (on RHEL 8). It was previously running in Docker, but now is running on a VM for ease of admin.

We had the PD integration working on v5, but cannot get a successful test on v6. The error I get is this:

Details Media type test failed.

• PagerDuty notification failed : Error: cannot get URL: Couldn't connect to server.

I am making an assumption that URL it is referring to is our Zabbix URL (based on the yaml file I imported), and I have tried multiple iterations of the URL, even including https://IP/zabbix.

I am trying to get this up and running in production in the next 2 weeks, and I believe this is my final hurdle, so any assistance is greatly appreciated.

Thanks a lot. This fixed the issue I had where zabbix was behind an HTTP/HTTPS proxy.

[kwbarron]

Thanks a lot. This fixed the issue I had where zabbix was behind an HTTP/HTTPS proxy.

Awesome. I'm glad I was able to save someone else the trial and error pain I went through.

[kquainta]

Greetings! I am struggling with a Zabbix 6.4 <> Pagerduty integration and am looking for suggestions. I have been following the "PagerDuty + Zabbix" instructions found HERE.

PagerDuty Config

• Created a Pagerduty service named "Zabbix 6.4"
• Used the "Zabbix Webhook (for 5.0 and 5.2)" integration (I have also tried the "Zabbix" and generic "Events API V2" integrations with no success.)
• Copied the Integration Key

Zabbix 6.4 Config

• Setup the Global Macro {$ZABBIX.URL} pointed to my Zabbix server web frontend.
• Created the PagerDuty Media type
  • Name: PagerDuty
  • Type: Webhook
  • Imported the media_pagerduty.yaml for Zabbix v6.4 found HERE
    • Confirmed that the code refers to this API endpoint:
      'https://events.pagerduty.com/v2/enqueue'
  • Added the PagerDuty integration key to the Token parameter
  • Performed a Media test successfull
  • Created a User named "Pagerduty User" and kept all defaults
  • Associated the Pagerduty media to the user
• Created a Trigger Action
  • Action:
  • Operation: Send message to users: Pagerduty User (Pagerduty User) via PagerDuty
    • Immediately
  • NOTE: I am not sure how to manually test the Trigger Action. This very well could be the issue. I need to research more on Zabbix Tigger Action functions.

Other Troubleshooting Steps

• DNS is resolving:
  telosuser@server1:~$ dig a +short events.pagerduty.com
  44.233.86.211
  44.231.93.240
  35.167.69.145
• The API endpoint is accessible and responding:
  telosuser@server1:~$ curl -kv https://events.pagerduty.com/v2/enqueue
  * Trying 44.233.86.211:443...
  * Connected to events.pagerduty.com (44.233.86.211) port 443 (#0)
  * ALPN, offering h2
  * ALPN, offering http/1.1
  * TLSv1.0 (OUT), TLS header, Certificate Status (22):
  * TLSv1.3 (OUT), TLS handshake, Client hello (1):
  * TLSv1.2 (IN), TLS header, Certificate Status (22):
  * TLSv1.3 (IN), TLS handshake, Server hello (2):
  * TLSv1.2 (IN), TLS header, Certificate Status (22):
  * TLSv1.2 (IN), TLS handshake, Certificate (11):
  * TLSv1.2 (IN), TLS header, Certificate Status (22):
  * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  * TLSv1.2 (IN), TLS header, Certificate Status (22):
  * TLSv1.2 (IN), TLS handshake, Server finished (14):
  * TLSv1.2 (OUT), TLS header, Certificate Status (22):
  * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  * TLSv1.2 (OUT), TLS header, Finished (20):
  * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
  * TLSv1.2 (OUT), TLS header, Certificate Status (22):
  * TLSv1.2 (OUT), TLS handshake, Finished (20):
  * TLSv1.2 (IN), TLS header, Finished (20):
  * TLSv1.2 (IN), TLS header, Certificate Status (22):
  * TLSv1.2 (IN), TLS handshake, Finished (20):
  * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  * ALPN, server accepted to use http/1.1
  * Server certificate:
  * subject: C=US; ST=California; L=San Francisco; O=PagerDuty, Inc.; CN=*.pagerduty.com
  * start date: Feb 12 00:00:00 2024 GMT
  * expire date: Mar 14 23:59:59 2025 GMT
  * issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
  * SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  * TLSv1.2 (OUT), TLS header, Supplemental data (23):
  > GET /v2/enqueue HTTP/1.1
  > Host: events.pagerduty.com
  > User-Agent: curl/7.81.0
  > Accept: */*
  >
  * TLSv1.2 (IN), TLS header, Supplemental data (23):
  * Mark bundle as not supporting multiuse
  < HTTP/1.1 405 Method Not Allowed
  < Server: nginx
  < Date: Thu, 04 Apr 2024 20:37:23 GMT
  < Content-Length: 18
  < Connection: keep-alive
  < cache-control: max-age=0, private, must-revalidate
  < x-request-id: F8MuKoqDEyBIVg4Apv-E
  <
  * Connection #0 to host events.pagerduty.com left intact

Conclusion
I have been testing by enabling and disabling a known service. The alert event appears consistently in Zabbix but I never receive anything in PagerDuty. I appreciate any suggestions anyone has as I have tried this solution multiple times with many variations without any success. I have another integration directly between Greylog and PagerDuty which works great so I know the PagerDuty size is working properly.