PSA: Changes in alert severity don't work logically with ticket closing. Be careful... | Product Discussions

Kel L · 2025-03-05T16:01:44.639000Z

This is a PSA for everyone that integrates LM with a ticketing system.We had a server fill up it's drive and crash last week because of how LM handles alerts that change severity level, so I wanted to summarize it in case anyone needs to make changes to your system. We use Zendesk and have things setup per the instructions on the LM Website: https://www.logicmonitor.com/support/alerts/integrations/create-update-close-tickets-zendesk-response-alertsActive opens a ticket, Escalated updates the ticket, Cleared closes the ticket. Pretty standard setup and 99.9% of the time, it works just fine. However, here's the scenario where it didn't.We had a server who's disk space was going up and down, but more up than down. That server hit a Warning level for Free Space and created us a ticket. If went up and down a bunch and eventually went over the Error threshold and created us an Error ticket. After that, it dipped back into the Warning threshold which issues a Clear to the Error ticket and closed it.The Free Space then went back to Error. LM used the Escalate option to Update the ticket. However, because it had previously closed the ticket, it was now updating a closed ticket and no one ever saw it. It continued to rise and eventually got to the Critical threshold which created us a new ticket. That one dipped back into the Error range as well which Cleared the ticket. It then went back into Critical which led to another instance of a closed ticket getting updates that no one ever saw. Eventually the server filled up and crashed.If LM would have issued the Active action, instead of the Escalate action, each time it went from a lower severity to a higher one, this wouldn't have been an issue. However, because LM considers the Alert as active until it completely clears, it decided to update the Error ticket even though it had already Cleared it out.To work around this, we are updating our Escalated step to hard code the Status:Open setting in there so any time LM decides to update a ticket, it will force it to Open Status. I don't think we should have to do this because LM shouldn't be updating tickets it had previously closed, but it is what it is.So, if you have your ticket flow setup like we do, just be aware of this possibility.

Discussions

PSA: Changes in alert severity don't work logically with ticket closing. Be careful... | Product Discussions | LogicMonitor Community