Alerting across multiple sites with different business hours
Hi team,
Got a dilemma I'm hoping someone can help with.
We have a customer with 200+ sites across multiple time zones.
Although some of these sites in the same TZ have the same opening hours, there are more that do not.
In the past we have used the earliest and latest times for each TZ as the times for the escalation chain.
But, this has meant our engineers are getting alert callouts when a lot of the sites are closed and we cannot perform any troubleshooting at site.
We need to stop this without making the job too onerous on our admin team.
Our idea is to have the customer manage some properties on each site device that specifies the business hours for each day.
Then we want alerting and escalation to be able to reference those properties and ONLY alert and escalate when within the specified business hours.
Anyone know how we can do this?
Thanks.
manish arora
·2 years agoHi Mike, I had same issue and I signed up for Pagerduty (Free account) and connected LM with pagerduty account and created the escalation policies in the pagerduty to forward the alerts to different teams (in different timezones) and also configured a backup where in if no one is availble or acknowedging that alert in pagerduty app then route it to someone else.
Mike Moniz
·2 years agoTo throw it out there, perhaps some 3rd party services may work better for implementing that logic. Perhaps something like OpsGenie or PagerDuty? Not endorsing either but they sound like they was designed with notification scheduling in mind.