Handling Alert Storms?
Summary: Dalton Woolsey shares issues with BGP Alerts in a full-mesh network topology, mentioning plans to discuss the problem with a Customer Success Manager. Justin Lanoue responds, suggesting an alternative method using Edwin AI and CLI commands to enrich descriptions, automate grouping alerts, and bundle them into a service alert. However, Justin Lanoue notes that this approach feels like a workaround.
Good morning,
Just wanted to put this out there to see how everyone else is handling these types of situations.
Scenario: Switch goes offline.
Alerts that generate:
BGP Alerts x5
OSPF Neighbor Alerts x5
Interface alerts x10
Uptime alert when it comes back online
Maybe a syslog event
Problem:
With that being said, how does your team know that all these alerts were related and should be bundled into the Switch going offline alert as the parent ticket?
Maybe it's a portal issue on our end but like OSPF Neighbors when clicking on them doing even topology map to the others... Same with BGP.
You can't dependent alert map instance alerts, only resources which is useless here.
I could maybe configure cluster alerts when more than 1 or 2 alerts of the same type generate but it can't be specifically grouped to pick and choose which instances.
Services don't really cut it as they don't suppress the individual alerts so at this point is my best bet to just document what alerts when the switch goes down and apply that to the offline alert of the switch or maybe look into better topology mapping so it's clearer?
Dalton Woolsey
·2 weeks agoHi Justin,
We are experiencing the same issue with BGP Alerts in our full-mesh network topology where each site has a BGP router.
I'll bring this up with our CSM today and revert back with an update if a solution is found.
Thanks,
Dalton
Justin Lanoue
OP2 weeks agoTheir solution will be Edwin AI unfortunately.
What I've been trying to do is have the descriptions be "enriched" via CLI commands which have our "Circuit" number on them and then I use automation to group them into services... I then hope to have automation bundle all the individual alerts into the service alert via the "members" property on the service when it fires which tells me all the associated instances.
It seems like it may work but its a very hacky solution.
Dalton Woolsey
·2 weeks agoThanks for the update and I'd have to agree... that doesn't sound very clean and you'd think numerous LM customers have this issue. Purchasing Edwin AI certainly isn't a viable solution to such a simple problem.
Justin Lanoue
OP7 months agoTypo above...
OSPF Neighbors do not topology map to others, same with BGP.
Example:
Our best bet is to search that IP in the search bar and then it usually finds the device.