Collapse Clustered Instances at Group Level

Hi Mark - Thanks for taking the time to write this up.  We're currently knee (possibly neck) deep in designing how take what has historically been a device - instance centric view of the world and extend it to best accommodate applications, services, and clusters.  This is on our radar, and we'll keep your specific use case in mind.

I'm working through this specific issue right now as we move from SCOM to LogicMonitor.  Has anyone found a good workaround for this?  I'm getting a dozen alerts for a single event every time a volume in our Hyper-V Cluster goes over capacity threshold.

Here was my solution:

Add Custom Property to each device with the cluster ID (or an arbitrary GUID/unique naming convention instance).  We used HyperV.Cluster.GUID = <GUID generated by powershell with New-GUID>

Use that to build a dynamic group for each cluster

Use that group to allow for cluster alert for which ever sources you're trying to get visibility into, then tell it not to trigger those source alerts for the individual members of the group.

To help reduce the number of repeated emails generated by the escalation chains, add a blank entry to the end of the escalation chain you're using.  The escalation increments through the steps after the escalation time has passed, then repeats the last step once it gets to the end.  That can produce far more emailed alerts than are necessary.  Adding a blank last step causes the escalation to repeat that after making it through the chain.  You can add timing pauses into your escalation chain as well by adding blank steps between.  If the alert closes during that time, the rule ends and the following steps don't fire.

For instance, in Dynamics AX, the AOS servers can take a long time to come up.  Our team needs to know that it's gone down.  Our customer needs to know if it doesn't com up.  Rather than making a pair of sources that we can then create separate alerts for, We make the first step the contact to our team, then enough blanks to account for the time it normally takes for the service to restart.  After that, we add the customer's team, then a blank.

Our current ticketing system only accepts automated tickets via email.  These little tricks allow us to have tickets only generated once per alert, and allow us to make sure we're not panicking our customers for failovers / failures that are recovering as expected.

For Failover Clusters in Windows Server, here's a powershell propertySource that will add a category to members of clusters.  It's not the complete solution, but it'll allow you to better target eventSources to eliminate some of the strain on your Collectors:
 

Name:      Windows Failover Cluster Discovery
Group:     Windows Failover Cluster
appliesTo: isWIndows()

if ( test-path "\\##system.displayname##\C$\Windows\Cluster\CLUSDB" ) {
	"system.categories=ClusterMember"
}

It doesn't currently identify owners (although that can probably be done with an invoke-command { test-path "hklm:\0.Cluster" } as the owner of the cluster gets the extra Registry Cluster Hive copied during the failover. (https://blog.workinghardinit.work/2016/03/29/the-cluster-and-0-cluster-registry-hives/).

I have powershell that will identify Clusters by name and GUID and find the Nodes associated with them.  I've got to figure out how to get that to create dynamic groups and populate them using properties.  I'm most likely going to have to figure out how to make it be a timed script 1-2 times a day that then scans for cluster members and performs a bunch of queries using the failovercluster PS Module commandlets.  Then use the REST API to create the associated structures to get them grouped so we can add clustered alerts to them and turn off their individual alerts.

Then I can apply the same kind of logic to Hyper-Visor clustering.  I'm slowly chipping away at this to make it fit our business model's use case.

Can't edit the previous post... so here's a quick update: I lied.  I had thought HyperV clusters and SQL clusters used a different mechanism for their respective clustering.  They don't, they use the same mechanism.  This works for hyperV as well.  To make it work with vCenter, you'll just need to identify an identifying piece.  I am using the existence of the Cluster DB file.

Ok... Progress:

I'm about half way done with the rest of this script:

• Done:
  • Identify Cluster Member
  • Identify Parent Cluster
  • Find Group for the Cluster in LM using RestAPI
  • - found: Identify if the Device has already been tagged with a Failover.Cluster.GUID
  • - not found: (partially done) Add Group with appliedTo and customProperty of Failover.Cluster.ParentGUID set
• To Do (Some of this may end up being identified by our management as our own IP, so may not get posted):
  • Add Cluster Alerts to the group, disable individual member alerts
  • Add Cluster Specific EventSources and DataSources (This will be outside the scope of this main discovery PropertySource script)

Time for the weekend, I'll keep you all updated.

I've got the whole thing written, but troubleshooting scripting in the LM environment is tricky at best.  All of the pieces work individually, but seem to not do the things they are supposed to as soon as I put it in a PropertySource.  I'm trying the chrome plugin as well... I don't even seem to be able to get it to write a log... I have some really broken down troubleshooting to do to figure out how to troubleshoot the actual script.,, but it's promising. It:

A) checks to see that if the device is already tagged and tags if necessary

if it's not already tagged it:

B) identifies VMs that are part of a cluster

C} identifies the cluster they're part of

D) checks the group structure to determine if there is a group already made for that cluster

if it doesn't exist yet, it:

E) makes the dynamic group if it doesn't exist yet with the appropriate appliesTo to pull in the devices

After that, I just need to determine what cluster alerts to apply to it (and how to make that happen)

Alright... I'm really close.  What am I missing here?  It worked, then it didn't... now it applies the first output and stops doing the rest of them.  I know it's something small and dumb, but I've been staring at this for a while now:

Disclaimer:  Neither I nor Beyond Impact 2.0, LLC make any warrantee that this code will work properly in your environment.  Use cautiously.  Caveat Emptor.

# appliesTo: isWIndows()
$server             = "##system.hostname##"

# change the company name and groupParentID
# This is a manually created folder for holding the dynamic cluster groups.  Your ID may Vary.
$groupParentID      = "566"
$company            = "Your_Company_Name"

$URLRoot            = "https://$company.logicmonitor.com/santaba/rest"

# I added these at the top level of our heirarchy so everything I write has access to them.
# -- For the REST API calls --
$accessID           = "##LogicMonitor.accessId.key##"
$accessKey          = "##LogicMonitor.accessKey.key##"

function Send-Request {
    param (
        $cred,
        $accessid   = $null,
        $accesskey  = $null,
        $URL                ,
        $data       = $null,
        $version    = '2'  ,
        $httpVerb   = "GET"
    )

    if ( $accessId -eq $null) {
        $accessId   = $cred.UserName
        $accessKey  = $cred.GetNetworkCredential().Password
    }

    # Use TLS 1.2
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

    # Get current time in milliseconds
    $epoch          = [Math]::Round(
        ( New-TimeSpan `
            -start (Get-Date -Date "1/1/1970") `
            -end (Get-Date).ToUniversalTime()).TotalMilliseconds
        )

    # Concatenate Request Details
    $requestVars    = $httpVerb + $epoch + $data + $resourcePath

    # Construct Signature
    $hmac           = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key       = [Text.Encoding]::UTF8.GetBytes( $accessKey )
    $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) )
    $signatureHex   = [System.BitConverter]::ToString( $signatureBytes ) -replace '-'
    $signature      = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) )

    # Construct Headers
    $auth           = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
    $headers        = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add(     "Authorization", $auth              )
    $headers.Add(     "Content-Type" , 'application/json' )

    # uses version 2 of the API
    $headers.Add(     "X-version"    , $version           )

    # Make Request
    $response       = Invoke-RestMethod `
        -Uri           $URL      `
        -Method        $httpVerb `
        -Body          $data     `
        -Header        $headers

    $result         = $response

    Return $result
}

# *** Look for the Windows Failover Cluster DB ***
if ( test-path "\\$server\C`$\Windows\Cluster\CLUSDB" ) {
    
    "system.categories=ClusterMember"   
    
    $clusterInfo    = invoke-command `
        -ComputerName $server `
        -scriptBlock  {

            Import-Module failoverclusters
            $cluster = get-cluster

            "$($cluster.name):$($cluster.id)"
        }

    $clustername    = ($clusterInfo -split ':')[0]
    $clusterid      = ($clusterInfo -split ':')[1]

    $groupName      = "Failover Cluster - $clustername"

    # *** Read Groups ***

    # Construct URL
    $resourcePath   = "/device/groups"
    $url            = $URLRoot + $resourcePath

    # Make Request
    $response       = Send-Request  `
        -accessid     $accessID     `
        -accesskey    $accessKey    `
        -URL          $url         
        
    $group   = $response.items | ? name -eq $groupName
    
    if ( ($group | measure-object).count -gt 0 ) {

        # "*** Group Already exists.  Need Device properties? ***"

        try {
            $resource = "##Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }

    } else {
        # "*** create group & tag resource ***"

        # Construct URL
        $resourcePath = "/device/groups"
        $url          = $URLRoot + $resourcePath

        # Construct Data Body
        $data = `
@"
{
    `"name`"             : `"$groupName`"                                  ,
    `"parentId`"         : `"$groupParentID`"                              ,
    `"disableAlerting`"  : `"true`"                                        ,
    `"enableNetflow`"    : `"false`"                                       ,
    `"appliesTo`"        : `"Failover.Cluster.GUID == \`"$ClusterID\`"`"   ,
    `"customProperties`" : [{
        `"name`"         : `"Failover.Cluster.ParentGUID`"                 ,
        `"value`"        : `"$ClusterID`"
    }]
}
"@


        $response       = Send-Request  `
            -accesskey    $accessKey    `
            -accessid     $accessId     `
            -URL          $url          `
            -data         $data         `
            -httpVerb     "POST"

	# *** Verify GUID is stored in the device info ***
        try {
            $resource = "##Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }
    }
}

OK... I've got everything working but the folder creation... isolating and fixing.

Done!!! Here's the final code... same warnings as in my previous post:

$company            = "beyondimpact"
$URLRoot            = "https://$company.logicmonitor.com/santaba/rest"

$groupParentID      = "566"
$server             = "##system.displayname##"

$accessID           = "##LogicMonitor.accessId.key##"
$accessKey          = "##LogicMonitor.accessKey.key##"

function Send-Request {
    param (
        $cred,
        $accessid   = $null,
        $accesskey  = $null,
        $URL               ,
        $data       = $null,
        $version    = '2'  ,
        $httpVerb   = "GET"
    )

    if ( $accessId -eq $null) {
        $accessId   = $cred.UserName
        $accessKey  = $cred.GetNetworkCredential().Password
    }

    # Use TLS 1.2
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

    # Get current time in milliseconds
    $epoch          = [Math]::Round(
        ( New-TimeSpan `
            -start (Get-Date -Date "1/1/1970") `
            -end (Get-Date).ToUniversalTime()).TotalMilliseconds
        )

    # Concatenate Request Details
    $requestVars    = $httpVerb + $epoch + $data + $resourcePath

    # Construct Signature
    $hmac           = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key       = [Text.Encoding]::UTF8.GetBytes( $accessKey )
    $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) )
    $signatureHex   = [System.BitConverter]::ToString( $signatureBytes ) -replace '-'
    $signature      = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) )

    # Construct Headers
    $auth           = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
    $headers        = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add(     "Authorization", $auth              )
    $headers.Add(     "Content-Type" , 'application/json' )

    # uses version 2 of the API
    $headers.Add(     "X-version"    , $version           )

    # Make Request
    $response       = Invoke-RestMethod `
        -Uri           $URL      `
        -Method        $httpVerb `
        -Body          $data     `
        -Header        $headers

    $result         = $response

    Return $result
}

if ( test-path "\\$server\C`$\Windows\Cluster\CLUSDB" ) {
    
    "system.categories=ClusterMember"   
    
    $clusterInfo    = invoke-command `
        -ComputerName $server        `
        -scriptBlock  {
            Import-Module failoverclusters
            $cluster = get-cluster
            "$($cluster.name):$($cluster.id)"
        }

    $clustername    = ($clusterinfo -split ':')[0]
    $clusterid      = ($clusterinfo -split ':')[1]

    $groupName      = "Failover Cluster - $clustername"

    # Read Groups
    # Construct URL
    $resourcePath   = "/device/groups"
    $url            = $URLRoot + $resourcePath

    # Make Request
    $response       = Send-Request  `
        -accessid     $accessID     `
        -accesskey    $accessKey    `
        -URL          $url         
        
    $group   = $response.items | ? name -eq $groupName
    
    if ( ($group | measure-object).count -gt 0 ) {

        # "*** Group Already exists.  Need Device properties? ***"
        try {
            $resource = "##Auto.Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }

    } else {
        # "*** create group & tag resource ***"

        # Construct URL
        $resourcePath = "/device/groups"
        $url          = $URLRoot + $resourcePath

        # Construct Data Body
        $data = `
@"
{
    `"name`"             : `"$groupName`"                                     ,
    `"parentId`"         : `"$groupParentID`"                                 ,
    `"disableAlerting`"  : `"true`"                                           ,
    `"enableNetflow`"    : `"false`"                                          ,
    `"appliesTo`"        : `"Auto.Failover.Cluster.GUID == \`"$ClusterID\`"`" ,
    `"customProperties`" : [{
        `"name`"         : `"Auto.Failover.Cluster.ParentGUID`"               ,
        `"value`"        : `"$ClusterID`"
    }]
}
"@

        try {
            $response       = Send-Request  `
                -accesskey    $accessKey    `
                -accessid     $accessId     `
                -URL          $url          `
                -data         $data         `
                -httpVerb     "POST"
            
        } catch {
            $error[0] | out-file $logPath -append
        }

        # Add Properties
        "Failover.Cluster.GUID=$clusterid"
    }
}