Forum Discussion

dancox's avatar
dancox
Icon for Neophyte rankNeophyte
2 months ago

Escalation Chains for bank holidays vs standard weeks

Hi LogicMonitor community,

We are fairly new to the product, just looking to see if anyone had any advice for handling Escalation Chains when it comes to Bank Holiday weekends.

We currently have escalation chains created for our teams to handle automated call-outs for our critical devices/checks during standard weekends and outside of core hours. However with Bank Holiday weekends requiring the need to operate the same process on Friday/Monday's depending on the specific weekend, we'd need to ensure those additional days does are covered too. 

The obvious solution we've come up with is to create a Bank Holiday On-call escalation chain for each of the teams where we can be flexible and amend the days we'd want to be called out on, then swap out that escalation chain to any alert rules which required it. This would work, although considering we've got multiple teams configured with on-call escalation chains within LogicMonitor as well as many alert rules using them, this is a manual process we'd need to keep on top of before/after every bank holiday, which leaves some room for human error of course.

Has anyone implemented something similar and have any advice they would be willing to share? I can't imagine we would be the first customers to come across this scenario.

Thanks in advance!

escalation chain

8 Replies

Sort By
    • dancox's avatar
      dancox
      Icon for Neophyte rankNeophyte
      2 months ago

      Hi Suzanne, that would be really helpful, thank you :).

      I have raised a support ticket within our portal too, but any support or advice would be appreciated!

    • dancox's avatar
      dancox
      Icon for Neophyte rankNeophyte
      30 days ago

      Good morning Suzanne (here in the UK anyway), hope you are well!

      I received a response from the Support team within our portal which basically just talked us through building time-based escalation chains, which we already know how to and it didn't address the question I asked.

      After a quick Google search, I can also see that others have posed the exact same question as me, the most historic I could see being 11 years ago! Neither of which have been addressed by LM:

      Add ability to treat public holidays (arbitrary dates) as ''weekends'' | LogicMonitor - 327

      Bank holidays and alerting | LogicMonitor - 6371

      Would you be able to point this post in the right direction of someone so I can at least get acknowledgement there is no better method of managing the bank holiday periods than the method we are planning to above (below to summarise):

      - Build separate Bank Holiday escalation chains for each of our teams, amending the times of the Escalations as per the bank holiday period

      - Manually swap over our stand Escalation chains to the bank holiday chains for 92 alert rules prior to the bank holiday period

      - Manually swap back to standard escalation chains for 92 alert rules post bank holiday period

      I think this method would work, however it's very time consuming as I'm sure you can appreciate. Any advice you and the team have to improve this, or the dream scenario of having a roadmap item for this in the future would be welcome by more in the community than just us I'd imagine :)

      Many thanks,

      Dan

       

  • Mike_Moniz's avatar
    Mike_Moniz
    Icon for Professor rankProfessor
    29 days ago

    I've only worked at MSPs that are 24/7 and have ticketing systems, so I haven't run into this situation myself. Until/If LM provides the functionality I would suggest looking it either using a 3rd party service (Opsgenie? Pagerduty?) that can provide that you are looking for or use the LM API to modify the rules in a script to make switching quick/automated.

    For the LM API, looks like there is a /settings/alerts/rules/ endpoint that can modify or create rules. You can use that to modify the 92 rules for you without having to manually change each one. If I had that many rules, I likely would manage it completely in a script, Infrastructure-as-code style, so you don't need to keep both places up-to-date.

    • dancox's avatar
      dancox
      Icon for Neophyte rankNeophyte
      27 days ago

      Hi Mike,

      Thanks for your reply :) this is something we've already been exploring actually! We're hoping to be able to achieve this within the next couple of weeks before this bank holiday, if not then in time for the May bank holidays. 

      However whether we get a fix in place or not, it would be still very useful functionality to have within the application itself. Even if there was a method of swapping escalation chains within alert rules in bulk somehow (just brainstorming) this would save us a lot of time!

      Thanks,

      Dan

  • Cole_McDonald's avatar
    Cole_McDonald
    Icon for Professor rankProfessor
    26 days ago

    In the past, OpsGenie has been leveraged to deal with on-call changes successfully for cases like this.

    • dancox's avatar
      dancox
      Icon for Neophyte rankNeophyte
      22 days ago

      Thanks Cole :) we're currently working on a automated script using the API which should do the trick to help us switch escalation chains within alert rules and then reverse them back for standard after, which we can schedule to run on the specific times. Although as mentioned above it would be some useful functionality to have built in to the software for future

      • Cole_McDonald's avatar
        Cole_McDonald
        Icon for Professor rankProfessor
        21 days ago

        That's a fun solution.  I always enjoy finding the edges of a piece of software and figuring out how to push that edge a bit.