Rate Limit Throttle Message Disable

If you're getting throttling of alert routing, disabling the message that is telling you that you are having this problem is akin to sticking your head in the sand. When throttling occurs, there is no guarantee that your ticketing system is getting all alerts you intend it to get. As such, there is no mechanism I'm aware of that will let you disable that mechanism.

You can address why you are getting that many alerts through alert tuning and by setting up multiple alert chains, although the latter opens you up to having to manage multiple chains pointing to the same escalation. 

What ticketing system do you use? You might consider switching to a webhook method of creating tickets. That method is unaffected by email rate limits, afaik. 

We are using Cherwell (it is new for us and so far not great). It isn't a very smart system from what I've heard in what current capabilities we can set for it to do. So I cannot just set on the Cherwell side, don't open a ticket from Throttle alert. 

This came to our attention when my teammates made some big changes to a device and forgot to put the device in SDT. We got about 33 tickets and some of the tickets were for the notice that it was hitting the Throttle. (I think that rate limit was 5 alerts and 10min, which I think as a bit too limiting.) I mainly don't want the Throttle alert to send to the email that opens a ticket. Is it possible to specify that I want all Throttle alerts to do something specify, like go to a different email?

Just now, grantae said:

Is it possible to specify that I want all Throttle alerts to do something specify, like go to a different email?

No, as far as I'm aware, this isn't possible. The throttle limits are per escalation chain, not per alert/alert rule.

It looks like Cherwell has a REST API, which means if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM. Just depends on Cherwell's capabilities.

Wow that article is pretty new!

I'm not super familiar with REST APIs, would you mind elaborating a bit more? When you say, "...if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM", who is "they"? Is this something I need to look into in LM or our Cherwell developer/admin or both? 

Sure, so LM can do various things with alerts. We can send emails (what you're currently doing). We can call you on your phone. We can send you an SMS text. Additionally, we can do something often referred to as a web hook. This means that LM will "execute" an HTTP request for each alert. This execution is similar to you putting an address in your browser and clicking go. The request is made, in this case, from your browser and the results come back to your browser. This is how all web browsing works. 

In the case of an alert integration within LM, LM can make an HTTP request when an alert comes through. All we really need to know is what URL to request from Cherwell that results in a ticket being created. Normally that's done with a special kind of request called a POST. So, all we need to know from Cherwell is what URL we need to post to in order to create a ticket. Usually, they'll also include some documentation that details how we can pass over the information from the alert (device name, datapoint, current value, etc.). 

Usually, sending alerts to a ticketing system via this mechanism has a few advantages, the primary being lower latency than email. It also continues to work even if the sending email server or Cherwell's receiving email server have issues.

More on our capabilities around this here: https://www.logicmonitor.com/support/alerts/integrations/custom-http-delivery

Thank you for the information, I'll look into that some more.

Is there a way to group alerts together? For example, I get 10-14 alerts when one of my branch sites goes down. On the device itself I get Ping, Global Counters, HTTPS, Global Performance, and Host Status. I also get the BGP Peering Detail from each device my branch peers with as well as on the branch itself (like 8 alerts just for the BGP). 

Usually the Host Status alert will cause the device to be marked as "dead" preventing/obscuring other alerts. I'd suggest looking into the timing of those alerts and the thresholds you have setup. 

Currently the Host Status alert is set for the default 300 seconds (6 min). Maybe I need to shorten that time, or increase the time before the other alerts kick on? (Leaning toward lowering the Host Status time but not sure to what yet.) I know currently a ping ticket is opened before a Host Status ticket right now, so I was thinking about tweaking this anyway. To confirm, if the Host Status alert is active it will stop other alerts for that device from popping up? (This doesn't prevent the BGP on both sides but maybe I can turn off the BGP alerts on the neighboring device side and just use the BGP alerts for the branch.)

That's probably a good start. However, there's some magic that goes on behind the scenes to determine a device status of "dead". Host Status definitely plays into it, but there are other things that play into it.

@Michael Rodrigues, care to chime in on this?

You can't change the Host Dead time, it's hardcoded backend logic. You can change the Host Status alert but I don't recommend it, as it doesn't change the back end logic for host death and suppression.

The only magic is the heartbeat interval on the HostStatus DS. It doesn't get reset by every collection type, for example SCRIPT modules will not reset it. Ping will. The backend looks at this value to determine host death and suppression.