Taming the Alert Chaos - Managing LogicMonitor Escalation Chains and Alert Rules with Terraform
The Challenge: When Growth Becomes Chaos
Idea 11 has been operating as an MSP for well over 10 years, with LogicMonitor being a cornerstone of our operations for nearly eight of those years. We've learnt some hard lessons about alert management as our business has evolved into a complex ecosystem serving diverse client needs.
Unlike traditional MSPs that deploy standardised toolsets across all clients, we operate across a spectrum of engagement models. Some clients use us as an extension of their existing IT teams, others rely on us as their complete IT department, and many leverage our monitoring infrastructure while maintaining their own operations (all or partial). This diversity creates unique challenges in alert management - different clients require different escalation paths, notification methods, and response procedures.
What started as a handful of clients with simple alert configurations has grown into hundreds of rules spanning multiple service levels and communication channels. This growth brought unexpected challenges that extend beyond typical MSP scenarios, making our solution valuable for any organisation managing
LogicMonitor at scale.
The most significant challenges we encountered included:
- Volume Overload: Hundreds of alert rules scattered across different clients and service levels
- Documentation Drought: Incomplete or non-existent documentation around individual configurations
Priority Conflicts: Multiple technicians creating rules with identical priority numbers, causing unpredictable alert routing
The breaking point came when we realised that troubleshooting a single alert issue required manually reviewing dozens of rules, trying to piece together the intended escalation flow. It was clear we needed a better approach.
While this solution emerged from MSP requirements, the foundational principles of managing LogicMonitor configurations as Infrastructure as Code can benefit any organisation. Whether you're an internal IT team, a traditional MSP, or an enterprise with complex monitoring needs, incorporating LogicMonitor configuration into your existing CI/CD pipelines provides the same benefits: version control, consistency, validation, and collaborative change management.
The Solution: Infrastructure as Code for Alert Rule Management
Terraform provides the perfect solution for managing LogicMonitor configurations at scale. By treating our alert rules and escalation chains as code, we gain:
- Version Control: Track changes and roll back problematic configurations
- Documentation: Self-documenting infrastructure with clear intent
- Consistency: Standardised patterns across all clients
- Collaboration: Team members conduct peer reviews before changes go live
Building the Foundation: Escalation Chains
Let's start with creating a robust escalation chain that includes both email and phone notifications:
resource "logicmonitor_escalation_chain" "client_critical_alerts" {
name = "Client Critical - ACME Corp"
description = "Critical alert escalation for ACME Corp"
enable_throttling = true
throttling_alerts = 10
throttling_period = 15
destinations = [{
type = "timebased"
period = [{
week_days = [1, 2, 3, 4, 5]
timezone = "Australia/Sydney"
start_minutes = 480
end_minutes = 1080
}]
stages = [
[{
type = "admin"
addr = "john.smith@acmecorp.com"
method = "email"
contact = ""
}, {
type = "admin"
addr = "john.smith@acmecorp.com"
method = "sms"
contact = "+61412345678"
}],
[{
type = "admin"
addr = "jane.doe@acmecorp.com"
method = "voice"
contact = "+61487654321"
}]
]
}]
}Creating Targeted Alert Rules
Now let's create alert rules that use our escalation chain for specific device groups. In this example, we have two existing device groups: Clients/Client_A and Clients/Client_B.
# Create an alert rule for any device in Client_A directory, specifically for CPUBusyPercent Alerts
resource "logicmonitor_alert_rule" "cpu_critical_client_a" {
name = "CPU Critical - Client A"
priority = 1000
datasource = "CPU-"
datapoint = "CPUBusyPercent"
instance = "*"
level_str = "Critical"
device_groups = ["Clients/Client_A"]
devices = ["*"]
escalating_chain_id = logicmonitor_escalation_chain.client_critical_alerts.id
escalation_interval = 15
}
# Create an alert rule for any device in Client_B directory, specifically for MemoryUsedPercent Alerts
resource "logicmonitor_alert_rule" "memory_critical_client_b" {
name = "Memory Critical - Client B"
priority = 1010
datasource = "Memory-"
datapoint = "MemoryUsedPercent"
instance = "*"
level_str = "Critical"
device_groups = ["Clients/Client_B"]
devices = ["*"]
escalating_chain_id = logicmonitor_escalation_chain.client_critical_alerts.id
escalation_interval = 15
}Validating Configurations Automatically
One of the most powerful aspects of managing LogicMonitor configurations as code is the ability to create automated validation tests. These tests catch configuration errors before they reach production, preventing common issues that plague manual alert management.
Our check_priorities.py script demonstrates this approach by solving a frequent problem: duplicate alert rule priorities. When multiple team members work on alert configurations, it's easy to accidentally reuse priority numbers, leading to unpredictable alert routing.
def check_alert_rule_priorities(file_path):
# Parse Terraform file for logicmonitor_alert_rule resources
rule_pattern = r'resource\s+"logicmonitor_alert_rule"\s+"([^"]+)"'
priority_pattern = r'priority\s*=\s*(\d+)'
priorities = defaultdict(list)
for match in re.finditer(rule_pattern, content):
rule_name = match.group(1)
# Extract and validate priority values
if priority in priorities:
print(f"ERROR: Duplicate priority {priority}")
return False
return TrueThis code snippet is taken from a much larger script, but it essentially checks each logicmonitor_alert_rule block and compares its priority numbers. If any priorities are identical, the following error will be produced:
This validation approach can be extended to check:
- Device group existence
- Escalation chain references
- Naming convention compliance
- Environment-specific priority ranges
By integrating these tests into your CI/CD pipeline, you ensure that only validated configurations reach production, transforming alert management from reactive troubleshooting to proactive quality assurance.
The Benefits We've Realised
Since implementing this Terraform-based approach, we've seen significant improvements:
- Clear Documentation: Every configuration is self-documenting with clear intent
- Rapid Deployment: New clients can be onboarded with tested, proven configurations
- Change Tracking: Git history shows exactly what changed and when
- Team Collaboration: Pull requests ensure peer review of all changes
Best Practices We've Adopted
- Modular Design: Create reusable modules for common patterns
- Naming Conventions: Consistent naming makes configurations easier to understand
- Priority Spacing: Leave gaps between priority numbers for future insertions
- Environment Separation: Use different priority ranges for different environments
- Documentation: Include descriptions explaining the business logic behind each rule
Next Steps
This foundation provides a solid starting point for managing LogicMonitor configurations as code. Consider extending this approach to:
- Datasource configurations
- Dashboard definitions
- Collector configurations
By treating your monitoring infrastructure as code, you'll transform alert management from a source of chaos into a well-orchestrated system that scales with your business.
Get started quickly with this GitHub Repository containing a ready-to-use foundational example that you can adapt to your environment.
