Campfire Chats Week 3: Alerts Recap
Overview
Week three of Camp LogicMonitor brought practitioners together for a deeper look at alerting in LogicMonitor, with a strong focus on how to make alerts more actionable and reduce unnecessary noise. The sessions covered alert thresholds, routing and escalation chains, dynamic thresholds, Scheduled Downtime (SDT), dependent alerts, integrations, and reporting. Much of the conversation was driven directly by customer questions and real-world alerting scenarios, from after-hours escalation to ServiceNow integrations and managing alerts during maintenance windows.
The biggest takeaway was that effective alerting is not simply about generating more notifications. It is about tuning thresholds appropriately, routing the right alerts to the right teams, suppressing expected or dependent noise, and giving responders enough context to act quickly.
Key Highlights
Alert routing is priority driven. Alert rules are evaluated in priority order and processing stops when an alert matches a rule, making rule order an important part of alert-routing design.
Escalation chains can support more sophisticated notification workflows. Teams can configure multiple stages, recipients, integrations, time-based behavior, and rate limiting to control who receives alerts and when.
Dynamic thresholds can help account for normal fluctuations. Instead of relying only on fixed values, dynamic thresholds learn from recent datapoint behavior and can be used to detect anomalies or suppress unnecessary alerts.
SDT remains an important tool for planned work. Monitoring and data collection continue during scheduled downtime, while notifications and escalation are suppressed to prevent expected maintenance from creating unnecessary noise.
Dependency awareness can dramatically reduce alert storms. Dependent Alert Mapping can identify an originating resource and suppress notifications for downstream dependent alerts while still keeping those alerts visible for context.
Q&A
Q: How can I send alerts to one group all the time, but include additional people outside normal business hours?
A: Use time-based escalation chains to change notification behavior based on the time of day. This allows your standard team to remain part of the notification path while additional on-call or after-hours recipients are included during designated periods.
Alert-rule priority is also important. Once an alert matches an alert rule, LogicMonitor stops evaluating additional rules and routes the alert through the escalation chain associated with that match. Design the alert rule and escalation chain together rather than expecting several matching alert rules to execute simultaneously.
Q: If multiple alert rules could match the same alert, do they all run?
A: No. Alert rules are evaluated in priority order, beginning with the lowest priority number. Once an alert matches a rule, processing stops and the alert is routed through that rule's escalation chain.
For example, if a broad rule suppressing Warning alerts is evaluated before a more specific disk Warning rule, the broad rule could capture the alert first. More specific routing rules should therefore be positioned appropriately relative to broader catch-all rules. This exact scenario came up during the session when a customer asked how a priority-one default Warning rule would interact with a more specific disk Warning rule.
Q: Can escalation chains send SMS notifications?
A: Alert delivery methods are configured through the recipients and contact methods associated with an escalation chain. The available delivery options depend on the contact information and notification methods configured in the portal. For environments that use external incident-management or paging platforms, integrations can also be incorporated into the alert-routing workflow.
Q: What does the rate limit on an escalation chain do?
A: Rate limiting helps protect teams from being overwhelmed during an alert storm. You define both a time period and the maximum number of alert notifications that can be delivered during that period. This can be especially useful when a larger infrastructure event suddenly generates many related alerts.
Q: How do I know whether a dynamic threshold is working?
A: Dynamic thresholds use recent historical datapoint behavior to establish an expected range and identify anomalous behavior. They are particularly useful for metrics where a single static threshold does not accurately represent normal conditions across every resource.
You can review and manage dynamic thresholds through Alert Tuning, including the threshold applied to the datapoint and whether alert generation is enabled.
Q: Can I configure different thresholds for a specific group of resources, such as printers?
A: Yes. Thresholds can be applied at different levels of the resource hierarchy, including resource groups. If a set of resources shares the same monitoring requirements, grouping those resources can make it easier to apply an appropriate threshold configuration without changing the global threshold for every resource using that DataSource. A printer-specific alerting scenario was one of the examples raised during the session.
Q: Can multiple threshold time periods overlap?
A: Thresholds can be configured for different time frames, but the configuration needs to account for how the applicable thresholds are evaluated at that level. If multiple threshold schedules are being used, their time frames should be designed intentionally rather than treated as independent duplicate alert paths.
Q: Why can I still see alerts for a resource that is in SDT?
A: Scheduled Downtime suppresses notifications and escalation, not monitoring itself. LogicMonitor continues collecting data and evaluating alert conditions during SDT, so alerts can still appear in the portal. This preserves visibility into what occurred during the maintenance window without notifying teams about conditions they already expect.
Q: Can recurring SDT have an overall start and end date?
A: Recurring SDT is built around a repeating schedule, such as daily, weekly, or monthly maintenance windows. One-time SDT provides a specific start and end date, while recurring configurations define when the downtime should repeat. Customers also discussed use cases such as aligning recurring downtime with Patch Tuesday.
Q: What are some best practices for reducing alert noise, and can I turn off alerts I do not want?
A: Start by determining whether each alert represents something that requires action. From there, tune static thresholds for your environment, use dynamic thresholds where normal behavior varies over time, adjust alert trigger and clear intervals, disable alert generation for datapoints that do not need operational action, use SDT for planned maintenance, and use dependency mapping to reduce downstream noise.
The goal is not simply fewer alerts. It is a higher signal-to-noise ratio so responders can trust that the alerts reaching them require attention.
Q: What changed with Dependent Alerts, and how can they help reduce noise?
A: Dependent Alert Mapping uses topology relationships to identify which resource is likely the originating cause of an incident and which alerts are downstream effects. When dependent notifications are suppressed, downstream alerts can remain visible for context without each one generating a separate routed notification.
This helps teams focus on the likely originating problem instead of responding independently to every symptom.
Q: If SNMP or WMI connectivity fails, can all monitors that depend on it roll up under that one alert?
A: This came up as a product-focused discussion during the session. The customer use case was to have WMI- or SNMP-dependent monitors stop generating individual alerts when the underlying collection mechanism is unavailable and instead roll those conditions into the primary WMI or SNMP failure.
That specific behavior was identified as a valuable product takeaway rather than a fully supported workflow demonstrated during the session.
Q: How can I see which modules or resources have thresholds configured?
A: The Alert Thresholds Report can be used to review threshold configuration across resources. This can help teams audit an environment and understand where alert thresholds or overrides have been applied. Customers also discussed some current limitations in how threshold levels are represented in that report.
Q: Why might I not have access to the Alert Thresholds Report?
A: Report visibility can be permission dependent. If the report or reporting options are not visible to your account, work with your LogicMonitor administrator to review your role and reporting permissions. This was confirmed directly in the session chat for a customer who could not access the report.
Q: Can integrations like ServiceNow, Ansible, or Freshservice be part of the same escalation workflow?
A: Integrations can be incorporated into alert-routing and escalation workflows, allowing alerts to trigger external systems rather than relying only on email notifications. The session also raised the question of centralizing multiple integrations within a single escalation chain rather than maintaining separate chains for each external platform.
Q: Where do API-based notifications such as Slack, PagerDuty, or ServiceNow fit into escalation chains?
A: These platforms are handled through integrations configured in LogicMonitor rather than basic email recipients. Once the integration is available in the portal, it can become part of the alert-routing workflow and escalation design.
Q: What is the recommended approach for integrating LogicMonitor with ServiceNow?
A: LogicMonitor can integrate with ServiceNow, so alerts feed incident-management workflows rather than relying solely on email notifications. The implementation should be designed around when incidents should be created, which alerts warrant ticket creation, and how updates or alert clears should be handled.
Customer Call-outs
The sessions generated a great mix of technical questions, shared experiences, and practitioner-to-practitioner discussion.
⭐️ “That’s an amazing [NOC] dashboard!”
⭐️ “Thanks Richard and Sky - you rock.”
Customers also shared their own experiences and ideas throughout the conversation, including using Alert Threshold reports, incorporating external data into NOC dashboards, and identifying opportunities to improve dependency-based alert suppression.
Keep the Conversation Going at Elevate
Want to take these conversations beyond the virtual Campfire?
Elevate Community Conference brings LogicMonitor customers, practitioners, and experts together for deeper technical learning, peer connection, and opportunities to exchange ideas with the people using LogicMonitor every day.
If you've enjoyed learning alongside other campers, Elevate is your chance to continue those conversations in person, hear how other customers are approaching their environments, and connect directly with the LogicMonitor community.
Register today using this promo code for a FREE ticket! Promo code: ECC26Comp06
Additional Resources
Want to dive deeper into this session? Watch the recording directly from the event page.
