Delayed Alerting
Summary: SimonG is seeking advice on setting up an alerting system for network devices. They have configured an escalation chain with alerts and a distribution group receiving alerts after 10 minutes. However, the alerts repeat every 10 minutes for a single device failure, and adjusting the alert rate to one per hour only issues a single alert even if multiple devices fail. Support suggested altering the ping module's collection frequency, but SimonG is concerned about missing important alerts in between checks. They are considering options to reduce noise without setting one alert rule per device and are open to suggestions or feedback.
Hi,
Not sure if this is doable or not but I have set up an alert rule to track a collection of Network Devices and set an escalation chain with a single blank first entry and a distribution group as the second escalation point with as delay of 10 minutes.
The problem I am having is alerts are being sent every 10 minutes for a device being down, I altered the alert rate to 1 every hour but then it just generated a single device down alert so if multiple were down it would only tell me about 1.
Support advised to alter the ping module to collect status every hour for example but that's then a big gap in-between where something could be down without knowledge.
Without setting up a single alert rule per device - not sure how to achieve the above, I thought about automated sdt if an alert is generated to stop the noise but doesn't look possible.
Anyone ever had a requirement for anything like this or just purely unique (odd way) I am trying to do something?
Any advise or shut downs happy to hear :-)
Thanks,
Simon
Larsen Orchard
·8 months agoIf you want the alerts to stop after the destination is alerted once, then you simply add a blank entry as the last stage. So in your case (with 10 minute escalation chain interval), the first stage would be blank, then the second stage would be the actual alerts, then the third would be blank again.
If you want repeated alerts every hour, then this is a bit more tricky. I would probably just add 5 blank stages in between stages with destinations, then repeat it for as long as you want. This obviously isn't very efficient because if you wanted to do this for 24 hours, you'd have to manually create 144 stages before it reverted to alerting every 10 minutes again haha. If anyone else knows a better way, I'd be curious to hear:)