Forum Discussion

RickRod's avatar
RickRod
Icon for Neophyte rankNeophyte
9 months ago

Bypass or update logic to clear alert (LogicMonitor to ServiceNow Integration)

Hello.

I recently executed a use case where the following steps occurred:

  • Alert triggers in LogicMonitor and creates an incident in ServiceNow
  • The assigned team who works the incident, assigns it to the appropriate team/team member
  • The team/team member remediates the alert, adds their comments to the incident, and resolves the incident
  • Once LogicMonitor sees the alert has been remediated, it makes the http rest call to resolve the incident

What's happening currently is although the user resolves the incident, LogicMonitor will still proceed with resolving the issue based on the alert status "Cleared".  When that happens, the predefined values from the payload are replacing the information provided by the user.  I understand if I delete that alert status under HTTP Delivery, this would ultimately resolve my issue.  Another alternative would be to remove the key-value pair from the payload that updates the Resolution Notes section of the incident.

Is there a way for LogicMonitor to recognize the incident currently has a resolve status and not proceed to update the incident without me having to remove the Cleared status?

Thanks.

  • Anonymous's avatar
    Anonymous
    9 months ago
    Mike_Moniz wrote:

    I don't consider an issue to be resolved unless the alert has also cleared

    Truer words never posted.

    Mike_Moniz wrote:

    Instead I would suggest that clearing in LM just adds a note to the ticket that it has cleared but the engineer would review it to confirm it's really resolved.

    Agreed 100%.

  • Hmm, atleast for me, I don't consider an issue to be resolved unless the alert has also cleared. In my experience it leads to orphan alerts where the engineer thought they fixed the issue but didn't, or they disagree with the alert level. Then when something gets worse management find out that we had an alert for a month that was missed. If the alert threshold needs tuning, then that should be part of the ticket too.

    I also wouldn't let LM resolve an alert ticket itself. There can be alerts that "flap" where it causes an alert but then clears in a few minutes. This may lead to getting frequently alerts that open then auto-close tickets and no one notices a device has been unstable for days.

    Instead I would suggest that clearing in LM just adds a note to the ticket that it has cleared but the engineer would review it to confirm it's really resolved. I haven't really touched the SN integration much but I assume you can modify the biz rules or javascript code to check ticket status. LM itself is very simple in that it just sends the state change without any logic, you likely need to do that in SN itself, which has more flexibility from my understanding.

    • Anonymous's avatar
      Anonymous
      Mike_Moniz wrote:

      I don't consider an issue to be resolved unless the alert has also cleared

      Truer words never posted.

      Mike_Moniz wrote:

      Instead I would suggest that clearing in LM just adds a note to the ticket that it has cleared but the engineer would review it to confirm it's really resolved.

      Agreed 100%.

  • Anonymous's avatar
    Anonymous

    I have this exact problem with PagerDuty. Following...

    • RickRod's avatar
      RickRod
      Icon for Neophyte rankNeophyte

      I just recently learned about PagerDuty.  One of our LogicMonitor SMEs was explaining to me the other day some of the challenges PagerDuty has to offer 😁.

      • Anonymous's avatar
        Anonymous

        It's actually the only problem I've had with PD. LM has 4 actions it can take on alerts and PD has 3 alert states. No matter which way I do it, my alerts will never match the state in LM. If LM had branching logic in the integration actions (as in, i could use if statements to determine which portions of the JSON got sent with certain values), it would solve the problem completely.