Best Practices For Practitioners: Dynamic Thresholds
The modern IT infrastructure consists of a complex ecosystem of interconnected systems spanning cloud and on-premises environments, generating unprecedented volumes of monitoring data. Static thresholds may not be intuitive enough to capture the nuanced performance characteristics of these dynamic environments due to the amount of incoming data, leading to alert fatigue, missed critical events, and inefficient resource allocation. Dynamic thresholds represent an evolutionary step in monitoring technology, leveraging advanced algorithms to create intelligent, adaptive monitoring strategies that distinguish between normal performance variations and genuine anomalies. Key Principles Dynamic thresholds transform monitoring by introducing adaptive mechanisms that intelligently interpret performance data. By analyzing historical performance data, these adaptive mechanisms move beyond rigid, predefined alert triggers, instead creating context-aware monitoring that understands the unique behavioral patterns of each monitored resource. This approach simultaneously addresses two critical challenges in modern IT operations: reducing unnecessary alert noise while ensuring that significant performance deviations are immediately identified and communicated. Recommended Implementation Strategies When to Use Dynamic Thresholds Recommended for: Metrics with varying performance patterns across instances Complex environments with diverse resource utilization Metrics where static thresholds are difficult to establish Not Recommended for: Status datapoints (up/down) Discrete value metrics (e.g., HTTP error codes) Metrics with consistently defined good/bad ranges Configuration Levels Global Level Best when most instances have different performance patterns Ideal for metrics like: CPU utilization Number of connections/requests Network latency Resource Group Level Useful for applying consistent dynamic thresholds across similar resources Cascades settings to all group instances Instance Level Perfect for experimenting or handling outlier instances Recommended when you want to: Reduce noise for specific instances Test dynamic thresholds on a limited subset of infrastructure Technical Considerations Minimum Training Data 5 hours required for initial configuration Up to 15 days of historical data used for refinement Detects daily and weekly trends Alert Configuration Configure to both trigger and suppress alerts Adjust advanced settings like: Percentage of anomalous values Band factor sensitivity Deviation direction (upper/lower/both) Pro Tip: Combining Static and Dynamic Thresholds Static and dynamic thresholds are not mutually exclusive—they can be powerful allies in your monitoring strategy. By implementing both: Use dynamic thresholds to reduce noise and catch subtle performance variations Maintain static thresholds for critical, well-defined alert conditions Create a multi-layered alerting approach that provides both granular insights and critical fail-safes Example: Dynamic thresholds for warning/error levels to adapt to performance variations Static thresholds for critical alerts to ensure immediate notification of severe issues Recommended Configuration Strategy Enable dynamic thresholds for warning/error severity levels Maintain static thresholds for critical alerts Use the "Value" comparison method when possible Best Practices Checklist ✅ Analyze existing alert trends before implementation ✅ Start with a small, representative subset of infrastructure ✅ Monitor and adjust threshold sensitivity ✅ Combine with static thresholds for comprehensive coverage ✅ Regularly review and refine dynamic threshold configurations Monitoring and Validation Utilize Alert Thresholds Report to track configuration Use Anomaly filter to review dynamic threshold-triggered alerts Compare alert volumes before and after implementation Conclusion Dynamic thresholds represent a paradigm shift in performance monitoring, bridging the gap between traditional alerting mechanisms and the complex, fluid nature of modern IT infrastructures. By leveraging machine learning and statistical analysis, these advanced monitoring techniques provide IT operations teams with a more nuanced, intelligent, and efficient approach to detecting and responding to performance anomalies. As IT environments continue to grow in complexity and scale, dynamic thresholds will become an essential tool for maintaining system reliability, optimizing resource utilization, and enabling proactive operational management. The true power of dynamic thresholds lies not just in their technological sophistication but in their ability to transform how organizations approach system monitoring—shifting from a culture of constant reaction to one of strategic, data-driven performance management. Additional Resources Enabling Dynamic Thresholds
Band Factors and Dynamic Thresholds? Documentation?
I've not had to look at Dynamic Thresholds at all, for anything I've been asked to do. But now I've been asked to look into it for someone, and I have no idea what "Band Factors" are and I cant find anything in the first page of hits on searching LogicMonitor Band Factors. Is there documentation on this somewhere, or is a Band Factor something a normal human is expected to know? :) I dont get any good search results just looking for Band Factors without mentioning LM. Can someone point me to the documentation for band factors? I have a user who was interested in an alert if a value was more than 20% out of the usual values for a time of day, but I have no idea how that translates to a band factor. I know the issue is ignorance on my part, I'm hoping someone can point me to something for little self-remediation in that arena. :) Thanks!Do Static and Dynamic thresholds work together?
Hi, I don't understand Dynamic thresholds and the weird UI to set them up. The whole thing about band factors and things make no sense to me. Here's my specific scenario and maybe someone can tell me the best way to handle this. We have a server that spikes it's CPU up pretty high every weekday morning. Generally starts around 5AM and ends around 9-10AM. Some days (Fridays) it seems to go longer. We don't want to get any alerts when it does this. Here's the graph for December (even though it looks like the spikes only go to 60, they actually all go to 99 or 100 when zoomed in more): We tried setting a daily, recurring SDT, but that still shows the errors, it just doesn't notify us about them. We want LM to consider the morning Spikes as "normal" and to ignore them. We setup a Dynamic Threshold to see if that would help. Here's a screenshot of what that looks like: As you can see by the arrow, this doesn't seem to have "Learned" what normal is. It seems like it just waits for the CPU to spike and then adjusts the "Expected Range" to compensate. If it was actually Learning, it should have expected the spike, since it happens every weekday, and adjusted BEFORE it happened. Right? Also, we have the standard Static Thresholds also enabled so we alert at 90/95/98 for this server. We get alerts for it all the time and aren't sure how to properly set this up. If we use the Dynamic alerts, should we turn off the Static ones since one doesn't seem to override the other? Should the Dynamic expected range know that the morning spike is going to happen or is that not how dynamic thresholds work? We rarely use them because we just don't get how to use them properly even after reading all the KBs and such. Any ideas, opinions, etc would be great.
Dynamic Alerting/Threshold Setup
Hi, I've just started experimenting with Dynamic alerting, probably something I should have done ages ago! I've read the documentation and been through the training module. I'm experimenting with the SQL Server Global Performance LogicModule which seems a good candidate for this, we get a fair bit of alert noise from this in our environment. I've gone into the Global definition for this module and turned on dynamic alerting for any of the datapoints that have a static threshold alert set. Seems to be working nicely. I've set the Dynamic threshold to suppress notifications, but what is the best way to remove the static threshold all together. I'm not too keen on just deleting them from LogicModule. I suppose one approach could be to duplicate the LogicModule, and modify deleting the static thresholds that one so that my original one is still intact in case I want to go back to it for refer to what the static values were set too. Then disable the original LogicModule. I don't really want the static alert threshold showing up in my alert list at all. Another thing that would be nice (Let me know if it's already there) would be if I was able to see at a glance if an alert had been triggered by a breach in dynamic or static threshold?Dynamic vs Static Thresholds?
Hi, I've been using LM for years and have never really understood Dynamic Thresholds and when to and when to not use them. How do they work in conjunction with Static thresholds or should you only use one or the other. Here's the specific issue I'm working on right now. I have a server that spikes the CPU all the time during the day. Here's the current graph for this device: What we want LM to do is "learn" that from 5AM to 5PM (or whatever it is), that the CPU is normally high during this range and to not alert us. But in between those those times, and on weekends, it's not normal for this to happen. We have a Dynamic threshold setup, but because the CPU spikes from 0 to 100 very quickly, LM doesn't seem to like this. If we change the Band numbers to 3 or 4, then it just blankets everything from 1-100 in the band and we'd never get any alerts on anything. As you can see here, The band doesn't start to "grow" until after the CPU spikes for the first time, then it goes up and continues to go up after the CPU has come back down. Then the next time it spikes, the band grows again. As you can see by the Purple arror, the band was coming down, but then the CPU spiked again, but the band didn't grow in time so we got an alert. Maybe this example is not a good place to try to use Dynamic thresholds. Would we be better off to just set a Static threshold to alert with the normal numbers, but limit it from 17:00 - 5:00 so it only alerts overnight?Does a change in a Dynamic Threshold reset the training data?
We rarely use dynamic alerts but someone wanted to test for it. We set something up but accidentally left the polling interval at 5 (50 minutes). It happened to be set that way for a few weeks before we got around to testing an actual event that exceeded the band we set. But it DID work, it just took almost an hour to get the alert. Well today when we realized this, I tweaked the dynamic threshold, mainly just changing it to 1 interval (2 minutes) because we want the shortest possible response. But the engineer also dropped the bandwidth on the port to a super low value at basically the same time. Its been an hour or more, no alerts. But literally all i changed was the qualification to make it happen sooner. So the TL;DR is ..... does a change to a threshold make the training of the datapoint start over? I see references to how long it takes to train, but nothing on whether changes reset the training, or anything that would explain this.
