Orphaned tickets when a alert changes severity...
Hi,
I came in today and had a bunch of leftover tickets from the weekend. They were all for Low Disk Space since we had patching on Saturday. I know all the servers are working fine so I wasn't sure why I had these alerts. After some research, I think LM has a slight issue in how it interacts with Integrations. I'm specifically using Zendesk but I'm not sure if that matters.
Here's what happened.
- Error alert came in for a server with low disk space.
- LM used our ZD integration and fired the Active line which creates a ticket in ZD for us.
- 6 minutes later, the disk space dropped below the next threshold and became a Critical.
- LM used our ZD integration and fired the Active line which creates a ticket in ZD for us. Now we had two tickets for the same alert. One for the Error and one for the Critical.
- Later on, the alert cleared.
- LM used our ZD integration and fired the Clear line which closes the ticket in ZD for us. However, it only fired this line once, for the Critical alert, which closed the second ticket that had been created. This left the Error ticket still open and now orphaned.
The subject line of the tickets has the Severity in them, so it's right that it created a second ticket when the alert went Critical. However, it should have either then, or when it closed the alert, fired two Clear integrations to close out both the tickets that had been created.
Has anyone else ever noticed anything like this? I'm not sure if we have something broken or if this has always been like this or what.
Thanks.
Mike Moniz
·2 years agoRereading your steps. If #4 really did kick off an Active call instead of Escalated, that sounds like a bug or it cleared between alerts. I would confirm on the Integration Log tab.
(I meant Escalated instead of UPDATED, in previous message)
Kel L
OP2 years agoKel L
OP2 years agoO = Open
S = Solved/Closed
LM User
·2 years agoWhat Mike is suggesting is that it shouldn't have opened two tickets, but rather the escalation should have updated the existing ticket. But that sounds like that's the way you want it. If that's the case, is it because you modified the escalation payload to make the integration open a new ticket?
Mike Moniz
·2 years agoThe Log is showing that it sent an Active call when it should have been an Escalated call. The 2nd Active call would caused a new ticket and updated the externalticketid. What does your integration settings look like? You have different URLS enabled and escalated/de-escalated option enabled?
Kel L
OP1 year agoActually, I prefer that it opens a second ticket. If I change the Alert Rules so that Errors and Criticals both use the same rule, then it will update the existing ticket. However, because we put data in the ticket subject line like Error, 9 Gigs free, etc, If the existing ticket would get updated when it changed to a Critical, we wouldn't see that because the subject line wouldn't get updated. So I'd think the issue was still an Error when in reality it was now much worse.
Based on that, I prefer that it does create a second ticket so it says Critical in the subject. I just then want it to automatically close the Error ticket since it was replaced by the Critical ticket.
After waiting a few weeks for Support, the confirmed this is just how it works and nothing can be done. As usual, I was referred to the "Feature Request" link so I filled it out so my idea can go into the same black hole all the other ideas went into. ;)
Mike Moniz
·1 year agoIf I recall correctly, LM will send an updated subject/message when the level increases (but not decrease) which should help with that issue if that is only reason for multiple tickets. Although I'm still not sure why LM is not sending you Update calls when the level increases. Did support explain that issue?
Mike Moniz
·2 years agoI'm not familiar with Zendesk integrations. Did you set it up per https://www.logicmonitor.com/support/alerts/integrations/create-update-close-tickets-zendesk-response-alerts? If so, make sure you have ##externalticketid## within the URL for the UPDATE type as documented there so LM will update the existing ticket and not create new one.
You want to make sure that LM is tracking the ticket number via the ##externalticketid## token. You can't just use the LMA# for example as that will change when it jumps between alert levels.
Kel L
OP2 years agoYep. We have it setup exactly like that. The problem seems to be on LMs side in that it's sending a second Create but never sending the second Close.