API Method to CLEAR an Alert (toggle off=on Alert Enable)

That’s an interesting one. I had something like that setup for my Device by Collector Auto Grouping DS which would create a group structure with a group for each collector with its assigned devices in its group. There was an option called cascadeSDT. At the time, group SDT wasn’t working right, so i built it so that whenever the collector went into SDT, the group would get alerting disabled.

The problem was that long standing alerts, or alerts that had already been acked would get closed and reopened. If you’re already disabling/enabling to clear alerts (it’s already part of your workflow) then go for it. I’d rather have the ability to selectively clear an alert to see if it comes back. Sometimes someone will close a ticket even though the alert is still open. Being able to just clear the alert manually and let it come back would be good because it would open a new ticket stating that the problem is still happening.

This is maybe similar to me wanting all staff to be able to toggle alerting when SDT is set. This will clear the alert in the ticketing system.

Basically: if you want the alert cleared, you have to either fix the problem or remove the threshold so LM doesn’t think it’s a problem anymore. I do believe alerts clearing on object deletion is a resolved issue.

I remember the Clearing being an issue when we first onboarded, and I believe it was fixed in a series of myriad portal release a few years back. I have not seen the issues reoccur yet, but my scope has also changed such that it’s always possible the issue is not 100% handled. At the very least, it has not been such a significant problem as to warrant my attention.

Yes, we should lobby for instance/resource/website/... deletions to result in a set of instanceDeleted/resourceDeleted etc. Alert status notifications.

Heads up that sometimes LM will not send a clear message. It’s MUCH better than it use to be, but there might be situations where it might still happen. I think it might still happen if you delete a device/instance with active alerts, but I haven’t checked in a while. Suggest having a way to bypass any Close blocking if you do run into that, perhaps by privileged users.

Confucius says: "Only when Service Operations and LogicMonitor both agree, is an Incident truly over."

The simplest approach is to assign the current latest alerting state into a separate ticket field and not permit the ITSM workflow to close the ticket if that field is not set to "clear".

Our NOC solely uses ServiceNow Event Management, and we have no need for ‘toggling’ alerts due to how we’ve implemented our workflows.

If you are using outgoing webhooks to notify your ticketing system, you can configure your outbound integrations in a way that will ensure alerts re-clock -- this will ensure that currently active alarms always renotify. LogicMonitor Escalation Chains can be set to ‘escalate after XX minutes,’ even if there is only one stage -- it will simply renotify the exact same integration. This is how we use it, and we have never had to ‘temporarily turn off alerting’ to handle problematic/long-running alerts. 

I would strongly urge one against ‘toggling’ alerts as they remain active in order to clear them, because you may run into the very real problem whereby you leave those instances disabled, which is going to eventually bite you in the rear almost guaranteed. There are a lot of procedural things that I think you would be better suited towards working on (e.g. improving alerting hygiene/education and training/setting healthy boundaries) than facilitating bad practices. I say that as someone that has been on a similar journey, so I appreciate it’s commonly easier said than done :)

I suggest instead setup the ticketing system to prevent the ticket from being able to be closed, or to reopen if alert was found to be still active. You can use the API to query for active alerts when attempting to close an alert ticket. It can get a bit complicated if you merge alert tickets, but doable.

Otherwise you would need to disable then shortly after enable alerting, which adds delays and can cause alert duration in LM to be wrong and reports perhaps thrown off.

Wow - thanks for all the great ideas. I will review to see if they work.

Some clarification:

The issue that we are trying to solve is that our operational team works from ticketing system. They close the ticket sometimes without solving the alert. The alert remains active, and since they don’t work from logicmonitor, the alert will never re-trigger. This applies to all datasources, all devices.

So we need an API call from the ticketing system to clear the alert when the incident is closed, so it can re-trigger on next poll and open a new incident/ticket.