The Challenge: When Growth Becomes Chaos

Idea 11 has been operating as an MSP for well over 10 years, with LogicMonitor being a cornerstone of our operations for nearly eight of those years. We've learnt some hard lessons about alert management as our business has evolved into a complex ecosystem serving diverse client needs.

Unlike traditional MSPs that deploy standardised toolsets across all clients, we operate across a spectrum of engagement models. Some clients use us as an extension of their existing IT teams, others rely on us as their complete IT department, and many leverage our monitoring infrastructure while maintaining their own operations (all or partial). This diversity creates unique challenges in alert management - different clients require different escalation paths, notification methods, and response procedures.

What started as a handful of clients with simple alert configurations has grown into hundreds of rules spanning multiple service levels and communication channels. This growth brought unexpected challenges that extend beyond typical MSP scenarios, making our solution valuable for any organisation managing
LogicMonitor at scale.

The most significant challenges we encountered included:

Volume Overload: Hundreds of alert rules scattered across different clients and service levels
Documentation Drought: Incomplete or non-existent documentation around individual configurations
Priority Conflicts: Multiple technicians creating rules with identical priority numbers, causing unpredictable alert routing

The breaking point came when we realised that troubleshooting a single alert issue required manually reviewing dozens of rules, trying to piece together the intended escalation flow. It was clear we needed a better approach.

While this solution emerged from MSP requirements, the foundational principles of managing LogicMonitor configurations as Infrastructure as Code can benefit any organisation. Whether you're an internal IT team, a traditional MSP, or an enterprise with complex monitoring needs, incorporating LogicMonitor configuration into your existing CI/CD pipelines provides the same benefits: version control, consistency, validation, and collaborative change management.

The Solution: Infrastructure as Code for Alert Rule Management

Terraform provides the perfect solution for managing LogicMonitor configurations at scale. By treating our alert rules and escalation chains as code, we gain:

Version Control: Track changes and roll back problematic configurations
Documentation: Self-documenting infrastructure with clear intent
Consistency: Standardised patterns across all clients
Collaboration: Team members conduct peer reviews before changes go live

Building the Foundation: Escalation Chains

Let's start with creating a robust escalation chain that includes both email and phone notifications:

resource "logicmonitor_escalation_chain" "client_critical_alerts" {
  name        = "Client Critical - ACME Corp"
  description = "Critical alert escalation for ACME Corp"

  enable_throttling   = true
  throttling_alerts   = 10
  throttling_period   = 15

  destinations = [{
    type = "timebased"

    period = [{
      week_days     = [1, 2, 3, 4, 5]
      timezone      = "Australia/Sydney"
      start_minutes = 480
      end_minutes   = 1080
    }]

    stages = [
      [{
        type    = "admin"
        addr    = "john.smith@acmecorp.com"
        method  = "email"
        contact = ""
      }, {
        type    = "admin"
        addr    = "john.smith@acmecorp.com"
        method  = "sms"
        contact = "+61412345678"
      }],
      [{
        type    = "admin"
        addr    = "jane.doe@acmecorp.com"
        method  = "voice"
        contact = "+61487654321"
      }]
    ]
  }]
}

Creating Targeted Alert Rules

Now let's create alert rules that use our escalation chain for specific device groups. In this example, we have two existing device groups: Clients/Client_A and Clients/Client_B.

# Create an alert rule for any device in Client_A directory, specifically for CPUBusyPercent Alerts
resource "logicmonitor_alert_rule" "cpu_critical_client_a" {
  name                = "CPU Critical - Client A"
  priority            = 1000

  datasource = "CPU-"
  datapoint  = "CPUBusyPercent"
  instance   = "*"
  level_str  = "Critical"

  device_groups = ["Clients/Client_A"]
  devices       = ["*"]

  escalating_chain_id = logicmonitor_escalation_chain.client_critical_alerts.id
  escalation_interval = 15
}

# Create an alert rule for any device in Client_B directory, specifically for MemoryUsedPercent Alerts
resource "logicmonitor_alert_rule" "memory_critical_client_b" {
  name                = "Memory Critical - Client B"
  priority            = 1010

  datasource = "Memory-"
  datapoint  = "MemoryUsedPercent"
  instance   = "*"
  level_str  = "Critical"

  device_groups = ["Clients/Client_B"]
  devices       = ["*"]

  escalating_chain_id = logicmonitor_escalation_chain.client_critical_alerts.id
  escalation_interval = 15
}

Validating Configurations Automatically

One of the most powerful aspects of managing LogicMonitor configurations as code is the ability to create automated validation tests. These tests catch configuration errors before they reach production, preventing common issues that plague manual alert management.

Our check_priorities.py script demonstrates this approach by solving a frequent problem: duplicate alert rule priorities. When multiple team members work on alert configurations, it's easy to accidentally reuse priority numbers, leading to unpredictable alert routing.

def check_alert_rule_priorities(file_path):
    # Parse Terraform file for logicmonitor_alert_rule resources
    rule_pattern = r'resource\s+"logicmonitor_alert_rule"\s+"([^"]+)"'
    priority_pattern = r'priority\s*=\s*(\d+)'
    
    priorities = defaultdict(list)
    for match in re.finditer(rule_pattern, content):
        rule_name = match.group(1)
        # Extract and validate priority values
        if priority in priorities:
            print(f"ERROR: Duplicate priority {priority}")
            return False
    return True

This code snippet is taken from a much larger script, but it essentially checks each logicmonitor_alert_rule block and compares its priority numbers. If any priorities are identical, the following error will be produced:

Checking priorities in main.tf...
Found 2 alert rules
ERROR: Duplicate priority 1000 found in rules: cpu_critical_client_a, memory_critical_client_b

❌ Duplicate priorities found!

This validation approach can be extended to check:

Device group existence
Escalation chain references
Naming convention compliance
Environment-specific priority ranges

By integrating these tests into your CI/CD pipeline, you ensure that only validated configurations reach production, transforming alert management from reactive troubleshooting to proactive quality assurance.

The Benefits We've Realised

Since implementing this Terraform-based approach, we've seen significant improvements:

Clear Documentation: Every configuration is self-documenting with clear intent
Rapid Deployment: New clients can be onboarded with tested, proven configurations
Change Tracking: Git history shows exactly what changed and when
Team Collaboration: Pull requests ensure peer review of all changes

Best Practices We've Adopted

Modular Design: Create reusable modules for common patterns
Naming Conventions: Consistent naming makes configurations easier to understand
Priority Spacing: Leave gaps between priority numbers for future insertions
Environment Separation: Use different priority ranges for different environments
Documentation: Include descriptions explaining the business logic behind each rule

Next Steps

This foundation provides a solid starting point for managing LogicMonitor configurations as code. Consider extending this approach to:

Datasource configurations
Dashboard definitions
Collector configurations

By treating your monitoring infrastructure as code, you'll transform alert management from a source of chaos into a well-orchestrated system that scales with your business.

Get started quickly with this GitHub Repository containing a ready-to-use foundational example that you can adapt to your environment.