Rate Limit Throttle Message Disable
"If the number of alerts delivered to the chain’s initial stage exceeds the rate limit, then a throttle message is sent to the individuals assigned to that stage. The message states that the number of alerts has exceeded the throttling level. From this point forward, alerts will be escalated to subsequent stages in accordance with your chain’s configuration. Throttle messages, however, will not be escalated and will continue to be sent to the first stage."
https://www.logicmonitor.com/academy/alert-rate-limiting
We have it set up that an email is sent to our ticketing system to open a ticket for Critical alerts. Turns out that the throttle message opens a ticket too. Is there a way to disable the throttle message?
Anne Grant
Posted 5 years ago·Last reply 5 years ago
17 comments
Anne Grant
OP5 years agoOh ok, that makes more sense. It "cancels" the alerts instead of delaying them.
LM User
·5 years agoWhen throttling is happening, the alerts aren't just queued up; they are not sent. So, if you got 48 normal tickets and 61 throttle tickets, the 48 tickets happened when you were not throttled and many many more happened while you were throttled.
Anne Grant
OP5 years agoWait... how did I get 61 of 109 being Throttle alerts if my settings are Throttle for 10min and 5 alerts? There shouldn't be a way to get more Throttle alerts than "real" tickets.
Anne Grant
OP5 years agoHmmm... I guess that is true. For our branches we use Palo Altos as the first reachable device at a branch site, however are major circuits in the core I believe those are between Cisco devices. It might be good to make branch down tickets for the sites with Host Status (and use use alert intervals for things like ping and over utilization for these sites) but maybe a different metric for core/distro devices. Hmmm....
I'll play with ideas on how to organize/configure the alerts better so I don't have a crazy flood of alerts that end up needing throttling. (Got like 109 tickets from yesterday, I think from a security scan. 61 of the tickets were the Throttle alerts!)
LM User
·5 years agoThat's really a subjective question. It really depends on how you define "down". I personally like to measure "site down" using IPSLA, but there are many ways to skin that cat.
Anne Grant
OP5 years agoI was actually just reading that warning in LM. It sounds like I should leave the Host Status at 6min, which means I should focus on editing the other alerts to wait at least 6min before triggering a ticket.
It is good to note that changing the idleinterval in Host Status doesn't sound like it will actually mark the device dead sooner since other backend logic is at work. I will be sure to leave it alone. I will look into editing the other alerts with this in mind.
Is Host Status the best metric to base down tickets off of; and ping with alert intervals that wait about 4 intervals (ping is set to 2min by default) so about 8min past before declaring a degrade and making a degrade ticket? Is there a better metric to use for down and degraded site tickets?
LM User
·5 years agoAh, that's it. Thanks @Michael Rodrigues. So, the host gets marked as dead when the heartbeat contains an epic that is more than 6 minutes old. When it's marked dead, that's when other alerts from the device are suppressed.
From the idleInterval datapoint description:
Does that get you a path moving forward?
LM User
·5 years agoThat's probably a good start. However, there's some magic that goes on behind the scenes to determine a device status of "dead". Host Status definitely plays into it, but there are other things that play into it.
@Michael Rodrigues, care to chime in on this?
Anne Grant
OP5 years agoCurrently the Host Status alert is set for the default 300 seconds (6 min). Maybe I need to shorten that time, or increase the time before the other alerts kick on? (Leaning toward lowering the Host Status time but not sure to what yet.) I know currently a ping ticket is opened before a Host Status ticket right now, so I was thinking about tweaking this anyway. To confirm, if the Host Status alert is active it will stop other alerts for that device from popping up? (This doesn't prevent the BGP on both sides but maybe I can turn off the BGP alerts on the neighboring device side and just use the BGP alerts for the branch.)
LM User
·5 years agoUsually the Host Status alert will cause the device to be marked as "dead" preventing/obscuring other alerts. I'd suggest looking into the timing of those alerts and the thresholds you have setup.
Anne Grant
OP5 years agoThank you for the information, I'll look into that some more.
Is there a way to group alerts together? For example, I get 10-14 alerts when one of my branch sites goes down. On the device itself I get Ping, Global Counters, HTTPS, Global Performance, and Host Status. I also get the BGP Peering Detail from each device my branch peers with as well as on the branch itself (like 8 alerts just for the BGP).
LM User
·5 years agoSure, so LM can do various things with alerts. We can send emails (what you're currently doing). We can call you on your phone. We can send you an SMS text. Additionally, we can do something often referred to as a web hook. This means that LM will "execute" an HTTP request for each alert. This execution is similar to you putting an address in your browser and clicking go. The request is made, in this case, from your browser and the results come back to your browser. This is how all web browsing works.
In the case of an alert integration within LM, LM can make an HTTP request when an alert comes through. All we really need to know is what URL to request from Cherwell that results in a ticket being created. Normally that's done with a special kind of request called a POST. So, all we need to know from Cherwell is what URL we need to post to in order to create a ticket. Usually, they'll also include some documentation that details how we can pass over the information from the alert (device name, datapoint, current value, etc.).
Usually, sending alerts to a ticketing system via this mechanism has a few advantages, the primary being lower latency than email. It also continues to work even if the sending email server or Cherwell's receiving email server have issues.
More on our capabilities around this here: https://www.logicmonitor.com/support/alerts/integrations/custom-http-delivery
Anne Grant
OP5 years agoWow that article is pretty new!
I'm not super familiar with REST APIs, would you mind elaborating a bit more? When you say, "...if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM", who is "they"? Is this something I need to look into in LM or our Cherwell developer/admin or both?
LM User
·5 years agoNo, as far as I'm aware, this isn't possible. The throttle limits are per escalation chain, not per alert/alert rule.
It looks like Cherwell has a REST API, which means if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM. Just depends on Cherwell's capabilities.
Anne Grant
OP5 years agoWe are using Cherwell (it is new for us and so far not great). It isn't a very smart system from what I've heard in what current capabilities we can set for it to do. So I cannot just set on the Cherwell side, don't open a ticket from Throttle alert.
This came to our attention when my teammates made some big changes to a device and forgot to put the device in SDT. We got about 33 tickets and some of the tickets were for the notice that it was hitting the Throttle. (I think that rate limit was 5 alerts and 10min, which I think as a bit too limiting.) I mainly don't want the Throttle alert to send to the email that opens a ticket. Is it possible to specify that I want all Throttle alerts to do something specify, like go to a different email?
LM User
·5 years agoIf you're getting throttling of alert routing, disabling the message that is telling you that you are having this problem is akin to sticking your head in the sand. When throttling occurs, there is no guarantee that your ticketing system is getting all alerts you intend it to get. As such, there is no mechanism I'm aware of that will let you disable that mechanism.
You can address why you are getting that many alerts through alert tuning and by setting up multiple alert chains, although the latter opens you up to having to manage multiple chains pointing to the same escalation.
What ticketing system do you use? You might consider switching to a webhook method of creating tickets. That method is unaffected by email rate limits, afaik.