Bypass or update logic to clear alert (LogicMonitor to ServiceNow Integration)
Hello.
I recently executed a use case where the following steps occurred:
- Alert triggers in LogicMonitor and creates an incident in ServiceNow
- The assigned team who works the incident, assigns it to the appropriate team/team member
- The team/team member remediates the alert, adds their comments to the incident, and resolves the incident
- Once LogicMonitor sees the alert has been remediated, it makes the http rest call to resolve the incident
What's happening currently is although the user resolves the incident, LogicMonitor will still proceed with resolving the issue based on the alert status "Cleared". When that happens, the predefined values from the payload are replacing the information provided by the user. I understand if I delete that alert status under HTTP Delivery, this would ultimately resolve my issue. Another alternative would be to remove the key-value pair from the payload that updates the Resolution Notes section of the incident.
Is there a way for LogicMonitor to recognize the incident currently has a resolve status and not proceed to update the incident without me having to remove the Cleared status?
Thanks.
RickRod
OP2 years agoThanks for your feedback Mike_Moniz
Mike Moniz
·2 years agoHmm, atleast for me, I don't consider an issue to be resolved unless the alert has also cleared. In my experience it leads to orphan alerts where the engineer thought they fixed the issue but didn't, or they disagree with the alert level. Then when something gets worse management find out that we had an alert for a month that was missed. If the alert threshold needs tuning, then that should be part of the ticket too.
I also wouldn't let LM resolve an alert ticket itself. There can be alerts that "flap" where it causes an alert but then clears in a few minutes. This may lead to getting frequently alerts that open then auto-close tickets and no one notices a device has been unstable for days.
Instead I would suggest that clearing in LM just adds a note to the ticket that it has cleared but the engineer would review it to confirm it's really resolved. I haven't really touched the SN integration much but I assume you can modify the biz rules or javascript code to check ticket status. LM itself is very simple in that it just sends the state change without any logic, you likely need to do that in SN itself, which has more flexibility from my understanding.
LM User
·2 years agoTruer words never posted.
Agreed 100%.
LM User
·2 years agoI have this exact problem with PagerDuty. Following...
RickRod
OP2 years agoI just recently learned about PagerDuty. One of our LogicMonitor SMEs was explaining to me the other day some of the challenges PagerDuty has to offer 😁.
LM User
·2 years agoIt's actually the only problem I've had with PD. LM has 4 actions it can take on alerts and PD has 3 alert states. No matter which way I do it, my alerts will never match the state in LM. If LM had branching logic in the integration actions (as in, i could use if statements to determine which portions of the JSON got sent with certain values), it would solve the problem completely.