"If the number of alerts delivered to the chain’s initial stage exceeds the rate limit, then a throttle message is sent to the individuals assigned to that stage. The message states that the number of alerts has exceeded the throttling level. From this point forward, alerts will be escalated to subsequent stages in accordance with your chain’s configuration. Throttle messages, however, will not be escalated and will continue to be sent to the first stage." https://www.logicmonitor.com/academy/alert-rate-limiting We have it set up that an email is sent to our ticketing system to open a ticket for Critical alerts. Turns out that the throttle message opens a ticket too. Is there a way to disable the throttle message?

If you're getting throttling of alert routing, disabling the message that is telling you that you are having this problem is akin to sticking your head in the sand. When throttling occurs, there is no guarantee that your ticketing system is getting all alerts you intend it to get. As such, there is no mechanism I'm aware of that will let you disable that mechanism. You can address why you are getting that many alerts through alert tuning and by setting up multiple alert chains, although the latter opens you up to having to manage multiple chains pointing to the same escalation. What ticketing system do you use? You might consider switching to a webhook method of creating tickets. That method is unaffected by email rate limits, afaik.

We are using Cherwell (it is new for us and so far not great). It isn't a very smart system from what I've heard in what current capabilities we can set for it to do. So I cannot just set on the Cherwell side, don't open a ticket from Throttle alert. This came to our attention when my teammates made some big changes to a device and forgot to put the device in SDT. We got about 33 tickets and some of the tickets were for the notice that it was hitting the Throttle. (I think that rate limit was 5 alerts and 10min, which I think as a bit too limiting.) I mainly don't want the Throttle alert to send to the email that opens a ticket. Is it possible to specify that I want all Throttle alerts to do something specify, like go to a different email?

Just now, grantae said: Is it possible to specify that I want all Throttle alerts to do something specify, like go to a different email? No, as far as I'm aware, this isn't possible. The throttle limits are per escalation chain, not per alert/alert rule. It looks like Cherwell has a REST API, which means if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM. Just depends on Cherwell's capabilities.

Wow that article is pretty new! I'm not super familiar with REST APIs, would you mind elaborating a bit more? When you say, "...if they built in the ability to create tickets into their API, you should be able to do it pretty easily with LM", who is "they"? Is this something I need to look into in LM or our Cherwell developer/admin or both?

Sure, so LM can do various things with alerts. We can send emails (what you're currently doing). We can call you on your phone. We can send you an SMS text. Additionally, we can do something often referred to as a web hook. This means that LM will "execute" an HTTP request for each alert. This execution is similar to you putting an address in your browser and clicking go. The request is made, in this case, from your browser and the results come back to your browser. This is how all web browsing works. In the case of an alert integration within LM, LM can make an HTTP request when an alert comes through. All we really need to know is what URL to request from Cherwell that results in a ticket being created. Normally that's done with a special kind of request called a POST. So, all we need to know from Cherwell is what URL we need to post to in order to create a ticket. Usually, they'll also include some documentation that details how we can pass over the information from the alert (device name, datapoint, current value, etc.). Usually, sending alerts to a ticketing system via this mechanism has a few advantages, the primary being lower latency than email. It also continues to work even if the sending email server or Cherwell's receiving email server have issues. More on our capabilities around this here: https://www.logicmonitor.com/support/alerts/integrations/custom-http-delivery

Rate Limit Throttle Message Disable

Anonymous
4 years ago
Ah, that's it. Thanks @Michael Rodrigues. So, the host gets marked as dead when the heartbeat contains an epic that is more than 6 minutes old. When it's marked dead, that's when other alerts from the device are suppressed.

From the idleInterval datapoint description:

Quote

The interval in seconds we do not get data from the host. NOTE: there is server side logic that declares a host DOWN after 6 minutes, suppressing other alerts. We do not recommend you change this alert.

Does that get you a path moving forward?
grantae
4 years ago
I was actually just reading that warning in LM. It sounds like I should leave the Host Status at 6min, which means I should focus on editing the other alerts to wait at least 6min before triggering a ticket.

20 minutes ago, Michael Rodrigues said:

You can't change the Host Dead time, it's hardcoded backend logic. You can change the Host Status alert but I don't recommend it, as it doesn't change the back end logic for host death and suppression.

The only magic is the heartbeat interval on the HostStatus DS. It doesn't get reset by every collection type, for example SCRIPT modules will not reset it. Ping will. The backend looks at this value to determine host death and suppression.

It is good to note that changing the idleinterval in Host Status doesn't sound like it will actually mark the device dead sooner since other backend logic is at work. I will be sure to leave it alone. I will look into editing the other alerts with this in mind.

Is Host Status the best metric to base down tickets off of; and ping with alert intervals that wait about 4 intervals (ping is set to 2min by default) so about 8min past before declaring a degrade and making a degrade ticket? Is there a better metric to use for down and degraded site tickets?
Anonymous
4 years ago

16 minutes ago, grantae said:

Is Host Status the best metric to base down tickets off of; and ping with alert intervals that wait about 4 intervals (ping is set to 2min by default) so about 8min past before declaring a degrade and making a degrade ticket? Is there a better metric to use for down and degraded site tickets?

That's really a subjective question. It really depends on how you define "down". I personally like to measure "site down" using IPSLA, but there are many ways to skin that cat.
grantae
4 years ago
Wait... how did I get 61 of 109 being Throttle alerts if my settings are Throttle for 10min and 5 alerts? There shouldn't be a way to get more Throttle alerts than "real" tickets.
Anonymous
4 years ago
When throttling is happening, the alerts aren't just queued up; they are not sent. So, if you got 48 normal tickets and 61 throttle tickets, the 48 tickets happened when you were not throttled and many many more happened while you were throttled.
grantae
4 years ago
Oh ok, that makes more sense. It "cancels" the alerts instead of delaying them.
grantae
4 years ago
Hmmm... I guess that is true. For our branches we use Palo Altos as the first reachable device at a branch site, however are major circuits in the core I believe those are between Cisco devices. It might be good to make branch down tickets for the sites with Host Status (and use use alert intervals for things like ping and over utilization for these sites) but maybe a different metric for core/distro devices. Hmmm....

I'll play with ideas on how to organize/configure the alerts better so I don't have a crazy flood of alerts that end up needing throttling. (Got like 109 tickets from yesterday, I think from a security scan. 61 of the tickets were the Throttle alerts!)

Forum Discussion

Rate Limit Throttle Message Disable

Related Content

UIv4 Resources Page - Disable polling at group-level

How do I clean up old warning messages?

Custom Property - disable SNMP polling.

Anybody else disabling Meraki instances by default?

New user - looking for information on parsing fields from Syslog message field

Recent Discussions

UIv4 instances when there are none?

Customer Support a Dumpster Fire

LM Servicenow Integration

LogicMonitor Datamart (Free / Open Source)

Groovy4, FastStringService, V.206, etc