Collapse Clustered Instances at Group Level

Locked

Discussion interactions are currently paused.

It has recently become clear to me that when you have multiple devices with the same instances (due to clustering or otherwise shared resources), there is no way to collapse these into a single alerting instance without a lot of manual and maintenance-intensive effort. The most recent example I have run into relates to vSphere with shared datastores (i.e., any implementation with vCenter, vMotion, etc.). In those cases, LM fails to distinguish the datastores presented at the cluster level from host-local datastores, and all are presented in parallel. With a vCenter and 8 hosts, this generates 9 alerts per clustered DS. I have been told by support I am out of luck on this for now, which is unfortunate, but I had an idea that could be used for this case along with other similar cases. Sadly, the name "Cluster Alerts" has already been taken, but I would call this idea that as well, just with a different methodology. So either an extension to Cluster Alerts or some new name TBD. The idea is to collapse specific instances within a group into a single instance at the group level. This way we can pick which instances are clustered (still manual, but manageable) and ensure only a single alert and view of those instances. Ideally, LM could be smart enough to identify the clustered elements and do this automatically, but this method would apply to any similar situation even if the API does not reveal the cluster binding details like ESX does. Please consider adding this enhancement soon!

Thanks,
Mark

mnagel

Posted 9 years ago·Last reply 7 years ago

11 comments

Cole McDonald

·7 years ago

Previous had an issue with the properties it was adding. If you're creating a new property from a proertySource script, it adds it as an "auto.*" property, which goes away as soon as the script stops processing. To add a new permanent custom property, you have to use the REST API, not just a "category.name=data" output from the script. Here's the final:


<span class="pln"># These first two lines will need to change to fit your environment.
# The groupParentID is the id of a group to house the dynamic groups that will be created...
# if that's you root level, use that ID.
# We're using a group named "Failover Clusters" in our heirarchy to house them.

#######
# Cole McDonald - Sr. Technical Analyst
# cole.mcdonald@beyondimpactllc.com
# Beyond Impact 2.0, llc
# No warranty provided for this code, use at your own risk
#######

$company            = "Your_Company_Name"
$groupParentID      = "566"

$URLRoot            = "https://$company.logicmonitor.com/santaba/rest"

$server             = "##system.displayname##"
$accessID           = "##LogicMonitor.accessId.key##"
$accessKey          = "##LogicMonitor.accessKey.key##"

function Send-Request {
    param (
        $cred,
        $accessid   = $null,
        $accesskey  = $null,
        $URL               ,
        $data       = $null,
        $version    = '2'  ,
        $httpVerb   = "GET"
    )

    if ( $accessId -eq $null) {
        $accessId   = $cred.UserName
        $accessKey  = $cred.GetNetworkCredential().Password
    }

    # Use TLS 1.2
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

    # Get current time in milliseconds
    $epoch          = [Math]::Round(
        ( New-TimeSpan `
            -start (Get-Date -Date "1/1/1970") `
            -end (Get-Date).ToUniversalTime()).TotalMilliseconds
        )

    # Concatenate Request Details
    $requestVars    = $httpVerb + $epoch + $data + $resourcePath

    # Construct Signature
    $hmac           = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key       = [Text.Encoding]::UTF8.GetBytes( $accessKey )
    $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) )
    $signatureHex   = [System.BitConverter]::ToString( $signatureBytes ) -replace '-'
    $signature      = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) )

    # Construct Headers
    $auth           = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
    $headers        = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add(     "Authorization", $auth              )
    $headers.Add(     "Content-Type" , 'application/json' )

    # uses version 2 of the API
    $headers.Add(     "X-version"    , $version           )

    # Make Request
    $response       = Invoke-RestMethod `
        -Uri           $URL             `
        -Method        $httpVerb        `
        -Body          $data            `
        -Header        $headers

    $result         = $response

    Return $result
}

if ( test-path "\\$server\C`$\Windows\Cluster\CLUSDB" ) {
    
    "system.categories=ClusterMember"   
    
    $clusterInfo    = invoke-command `
        -ComputerName $server        `
        -scriptBlock  {
            Import-Module failoverclusters
            $cluster = get-cluster
            "$($cluster.name):$($cluster.id)"
        }

    $clustername    = ($clusterinfo -split ':')[0]
    $clusterid      = ($clusterinfo -split ':')[1]

    $groupName      = "Failover Cluster - $clustername"

    # Read Groups
    # Construct URL
    $resourcePath   = "/device/groups"
    $url            = $URLRoot + $resourcePath

    # Make Request
    $response       = Send-Request  `
        -accessid     $accessID     `
        -accesskey    $accessKey    `
        -URL          $url         
        
    $group   = $response.items | ? name -eq $groupName
    
    if ( ($group | measure-object).count -gt 0 ) {

        # "*** Group Already exists.  Need Device properties? ***"
        try {
            $resource = "##Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            # Construct URL
            $resourcePath   = "/device/devices/##system.deviceid##/properties/"
            $url            = $URLRoot + $resourcePath

            # Construct Data Body
            $data = `
@"
    {
        `"type`"         : `"custom`"                ,
        `"name`"         : `"Failover.Cluster.GUID`" ,
        `"value`"        : `"$ClusterID`"
    }
"@
            $response       = Send-Request  `
                -accesskey    $accessKey    `
                -accessid     $accessId     `
                -URL          $url          `
                -data         $data         `
                -httpVerb     "POST"
        }
    } else {
        # "*** create group & tag resource ***"

        # Construct URL
        $resourcePath = "/device/groups"
        $url          = $URLRoot + $resourcePath

        # Construct Data Body
        $data = `
@"
{
    `"name`"             : `"$groupName`"                                ,
    `"parentId`"         : `"$groupParentID`"                            ,
    `"disableAlerting`"  : `"true`"                                      ,
    `"enableNetflow`"    : `"false`"                                     ,
    `"appliesTo`"        : `"Failover.Cluster.GUID == \`"$ClusterID\`"`" ,
    `"customProperties`" : [{
        `"name`"         : `"Failover.Cluster.ParentGUID`"               ,
        `"value`"        : `"$ClusterID`"
    }]
}
"@

        try {
            $response       = Send-Request  `
                -accesskey    $accessKey    `
                -accessid     $accessId     `
                -URL          $url          `
                -data         $data         `
                -httpVerb     "POST"
            

            # Add Properties
            # Construct URL
            $resourcePath   = "/device/devices/##system.deviceid##/properties/Failover.Cluster.GUID"
            $url            = $URLRoot + $resourcePath

            # Construct Data Body
            $data = `
@"
    {
            `"type`"         : `"custom`"                ,
            `"name`"         : `"Failover.Cluster.GUID`" ,
            `"value`"        : `"$ClusterID`"
    }
"@
            $response       = Send-Request  `
                -accesskey    $accessKey    `
                -accessid     $accessId     `
                -URL          $url          `
                -data         $data         `
                -httpVerb     "PUT"
        } catch {
            $error[0] | out-file $logPath -append
        }
    }
}</span>

Cole McDonald

·7 years ago

Done!!! Here's the final code... same warnings as in my previous post:


<span class="pln">$company            = "beyondimpact"
$URLRoot            = "https://$company.logicmonitor.com/santaba/rest"

$groupParentID      = "566"
$server             = "##system.displayname##"

$accessID           = "##LogicMonitor.accessId.key##"
$accessKey          = "##LogicMonitor.accessKey.key##"

function Send-Request {
    param (
        $cred,
        $accessid   = $null,
        $accesskey  = $null,
        $URL               ,
        $data       = $null,
        $version    = '2'  ,
        $httpVerb   = "GET"
    )

    if ( $accessId -eq $null) {
        $accessId   = $cred.UserName
        $accessKey  = $cred.GetNetworkCredential().Password
    }

    # Use TLS 1.2
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

    # Get current time in milliseconds
    $epoch          = [Math]::Round(
        ( New-TimeSpan `
            -start (Get-Date -Date "1/1/1970") `
            -end (Get-Date).ToUniversalTime()).TotalMilliseconds
        )

    # Concatenate Request Details
    $requestVars    = $httpVerb + $epoch + $data + $resourcePath

    # Construct Signature
    $hmac           = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key       = [Text.Encoding]::UTF8.GetBytes( $accessKey )
    $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) )
    $signatureHex   = [System.BitConverter]::ToString( $signatureBytes ) -replace '-'
    $signature      = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) )

    # Construct Headers
    $auth           = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
    $headers        = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add(     "Authorization", $auth              )
    $headers.Add(     "Content-Type" , 'application/json' )

    # uses version 2 of the API
    $headers.Add(     "X-version"    , $version           )

    # Make Request
    $response       = Invoke-RestMethod `
        -Uri           $URL      `
        -Method        $httpVerb `
        -Body          $data     `
        -Header        $headers

    $result         = $response

    Return $result
}

if ( test-path "\\$server\C`$\Windows\Cluster\CLUSDB" ) {
    
    "system.categories=ClusterMember"   
    
    $clusterInfo    = invoke-command `
        -ComputerName $server        `
        -scriptBlock  {
            Import-Module failoverclusters
            $cluster = get-cluster
            "$($cluster.name):$($cluster.id)"
        }

    $clustername    = ($clusterinfo -split ':')[0]
    $clusterid      = ($clusterinfo -split ':')[1]

    $groupName      = "Failover Cluster - $clustername"

    # Read Groups
    # Construct URL
    $resourcePath   = "/device/groups"
    $url            = $URLRoot + $resourcePath

    # Make Request
    $response       = Send-Request  `
        -accessid     $accessID     `
        -accesskey    $accessKey    `
        -URL          $url         
        
    $group   = $response.items | ? name -eq $groupName
    
    if ( ($group | measure-object).count -gt 0 ) {

        # "*** Group Already exists.  Need Device properties? ***"
        try {
            $resource = "##Auto.Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }

    } else {
        # "*** create group & tag resource ***"

        # Construct URL
        $resourcePath = "/device/groups"
        $url          = $URLRoot + $resourcePath

        # Construct Data Body
        $data = `
@"
{
    `"name`"             : `"$groupName`"                                     ,
    `"parentId`"         : `"$groupParentID`"                                 ,
    `"disableAlerting`"  : `"true`"                                           ,
    `"enableNetflow`"    : `"false`"                                          ,
    `"appliesTo`"        : `"Auto.Failover.Cluster.GUID == \`"$ClusterID\`"`" ,
    `"customProperties`" : [{
        `"name`"         : `"Auto.Failover.Cluster.ParentGUID`"               ,
        `"value`"        : `"$ClusterID`"
    }]
}
"@

        try {
            $response       = Send-Request  `
                -accesskey    $accessKey    `
                -accessid     $accessId     `
                -URL          $url          `
                -data         $data         `
                -httpVerb     "POST"
            
        } catch {
            $error[0] | out-file $logPath -append
        }

        # Add Properties
        "Failover.Cluster.GUID=$clusterid"
    }
}</span>

Cole McDonald

·7 years ago

OK... I've got everything working but the folder creation... isolating and fixing.

Cole McDonald

·7 years ago

Alright... I'm really close. What am I missing here? It worked, then it didn't... now it applies the first output and stops doing the rest of them. I know it's something small and dumb, but I've been staring at this for a while now:

Disclaimer: Neither I nor Beyond Impact 2.0, LLC make any warrantee that this code will work properly in your environment. Use cautiously. Caveat Emptor.


<span class="pln"># appliesTo: isWIndows()
$server             = "##system.hostname##"

# change the company name and groupParentID
# This is a manually created folder for holding the dynamic cluster groups.  Your ID may Vary.
$groupParentID      = "566"
$company            = "Your_Company_Name"

$URLRoot            = "https://$company.logicmonitor.com/santaba/rest"

# I added these at the top level of our heirarchy so everything I write has access to them.
# -- For the REST API calls --
$accessID           = "##LogicMonitor.accessId.key##"
$accessKey          = "##LogicMonitor.accessKey.key##"

function Send-Request {
    param (
        $cred,
        $accessid   = $null,
        $accesskey  = $null,
        $URL                ,
        $data       = $null,
        $version    = '2'  ,
        $httpVerb   = "GET"
    )

    if ( $accessId -eq $null) {
        $accessId   = $cred.UserName
        $accessKey  = $cred.GetNetworkCredential().Password
    }

    # Use TLS 1.2
    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

    # Get current time in milliseconds
    $epoch          = [Math]::Round(
        ( New-TimeSpan `
            -start (Get-Date -Date "1/1/1970") `
            -end (Get-Date).ToUniversalTime()).TotalMilliseconds
        )

    # Concatenate Request Details
    $requestVars    = $httpVerb + $epoch + $data + $resourcePath

    # Construct Signature
    $hmac           = New-Object System.Security.Cryptography.HMACSHA256
    $hmac.Key       = [Text.Encoding]::UTF8.GetBytes( $accessKey )
    $signatureBytes = $hmac.ComputeHash( [Text.Encoding]::UTF8.GetBytes( $requestVars ) )
    $signatureHex   = [System.BitConverter]::ToString( $signatureBytes ) -replace '-'
    $signature      = [System.Convert]::ToBase64String( [System.Text.Encoding]::UTF8.GetBytes( $signatureHex.ToLower() ) )

    # Construct Headers
    $auth           = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch
    $headers        = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
    $headers.Add(     "Authorization", $auth              )
    $headers.Add(     "Content-Type" , 'application/json' )

    # uses version 2 of the API
    $headers.Add(     "X-version"    , $version           )

    # Make Request
    $response       = Invoke-RestMethod `
        -Uri           $URL      `
        -Method        $httpVerb `
        -Body          $data     `
        -Header        $headers

    $result         = $response

    Return $result
}

# *** Look for the Windows Failover Cluster DB ***
if ( test-path "\\$server\C`$\Windows\Cluster\CLUSDB" ) {
    
    "system.categories=ClusterMember"   
    
    $clusterInfo    = invoke-command `
        -ComputerName $server `
        -scriptBlock  {

            Import-Module failoverclusters
            $cluster = get-cluster

            "$($cluster.name):$($cluster.id)"
        }

    $clustername    = ($clusterInfo -split ':')[0]
    $clusterid      = ($clusterInfo -split ':')[1]

    $groupName      = "Failover Cluster - $clustername"

    # *** Read Groups ***

    # Construct URL
    $resourcePath   = "/device/groups"
    $url            = $URLRoot + $resourcePath

    # Make Request
    $response       = Send-Request  `
        -accessid     $accessID     `
        -accesskey    $accessKey    `
        -URL          $url         
        
    $group   = $response.items | ? name -eq $groupName
    
    if ( ($group | measure-object).count -gt 0 ) {

        # "*** Group Already exists.  Need Device properties? ***"

        try {
            $resource = "##Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }

    } else {
        # "*** create group & tag resource ***"

        # Construct URL
        $resourcePath = "/device/groups"
        $url          = $URLRoot + $resourcePath

        # Construct Data Body
        $data = `
@"
{
    `"name`"             : `"$groupName`"                                  ,
    `"parentId`"         : `"$groupParentID`"                              ,
    `"disableAlerting`"  : `"true`"                                        ,
    `"enableNetflow`"    : `"false`"                                       ,
    `"appliesTo`"        : `"Failover.Cluster.GUID == \`"$ClusterID\`"`"   ,
    `"customProperties`" : [{
        `"name`"         : `"Failover.Cluster.ParentGUID`"                 ,
        `"value`"        : `"$ClusterID`"
    }]
}
"@


        $response       = Send-Request  `
            -accesskey    $accessKey    `
            -accessid     $accessId     `
            -URL          $url          `
            -data         $data         `
            -httpVerb     "POST"

	# *** Verify GUID is stored in the device info ***
        try {
            $resource = "##Failover.Cluster.GUID##"
        } catch {
            $resource = $null
        }

        if ( $resource -ne $clusterid ) {
            # Add Properties
            "Failover.Cluster.GUID=$clusterid"
        }
    }
}</span>

Cole McDonald

·7 years ago

I've got the whole thing written, but troubleshooting scripting in the LM environment is tricky at best. All of the pieces work individually, but seem to not do the things they are supposed to as soon as I put it in a PropertySource. I'm trying the chrome plugin as well... I don't even seem to be able to get it to write a log... I have some really broken down troubleshooting to do to figure out how to troubleshoot the actual script.,, but it's promising. It:

A) checks to see that if the device is already tagged and tags if necessary

if it's not already tagged it:

B) identifies VMs that are part of a cluster

C} identifies the cluster they're part of

D) checks the group structure to determine if there is a group already made for that cluster

if it doesn't exist yet, it:

E) makes the dynamic group if it doesn't exist yet with the appropriate appliesTo to pull in the devices

After that, I just need to determine what cluster alerts to apply to it (and how to make that happen)

Cole McDonald

·7 years ago

Ok... Progress:

I'm about half way done with the rest of this script:

Done:
- Identify Cluster Member
- Identify Parent Cluster
- Find Group for the Cluster in LM using RestAPI
- - found: Identify if the Device has already been tagged with a Failover.Cluster.GUID
- - not found: (partially done) Add Group with appliedTo and customProperty of Failover.Cluster.ParentGUID set
To Do (Some of this may end up being identified by our management as our own IP, so may not get posted):
- Add Cluster Alerts to the group, disable individual member alerts
- Add Cluster Specific EventSources and DataSources (This will be outside the scope of this main discovery PropertySource script)

Time for the weekend, I'll keep you all updated.

Cole McDonald

·7 years ago

Can't edit the previous post... so here's a quick update: I lied. I had thought HyperV clusters and SQL clusters used a different mechanism for their respective clustering. They don't, they use the same mechanism. This works for hyperV as well. To make it work with vCenter, you'll just need to identify an identifying piece. I am using the existence of the Cluster DB file.

Cole McDonald

·7 years ago

For Failover Clusters in Windows Server, here's a powershell propertySource that will add a category to members of clusters. It's not the complete solution, but it'll allow you to better target eventSources to eliminate some of the strain on your Collectors:


<span class="pln">Name:      Windows Failover Cluster Discovery
Group:     Windows Failover Cluster
appliesTo: isWIndows()

if ( test-path "\\##system.displayname##\C$\Windows\Cluster\CLUSDB" ) {
	"system.categories=ClusterMember"
}</span>

It doesn't currently identify owners (although that can probably be done with an invoke-command { test-path "hklm:\0.Cluster" } as the owner of the cluster gets the extra Registry Cluster Hive copied during the failover. (https://blog.workinghardinit.work/2016/03/29/the-cluster-and-0-cluster-registry-hives/).

I have powershell that will identify Clusters by name and GUID and find the Nodes associated with them. I've got to figure out how to get that to create dynamic groups and populate them using properties. I'm most likely going to have to figure out how to make it be a timed script 1-2 times a day that then scans for cluster members and performs a bunch of queries using the failovercluster PS Module commandlets. Then use the REST API to create the associated structures to get them grouped so we can add clustered alerts to them and turn off their individual alerts.

Then I can apply the same kind of logic to Hyper-Visor clustering. I'm slowly chipping away at this to make it fit our business model's use case.

Cole McDonald

·7 years ago

Here was my solution:

Add Custom Property to each device with the cluster ID (or an arbitrary GUID/unique naming convention instance). We used HyperV.Cluster.GUID = <GUID generated by powershell with New-GUID>

Use that to build a dynamic group for each cluster

Use that group to allow for cluster alert for which ever sources you're trying to get visibility into, then tell it not to trigger those source alerts for the individual members of the group.

To help reduce the number of repeated emails generated by the escalation chains, add a blank entry to the end of the escalation chain you're using. The escalation increments through the steps after the escalation time has passed, then repeats the last step once it gets to the end. That can produce far more emailed alerts than are necessary. Adding a blank last step causes the escalation to repeat that after making it through the chain. You can add timing pauses into your escalation chain as well by adding blank steps between. If the alert closes during that time, the rule ends and the following steps don't fire.

For instance, in Dynamics AX, the AOS servers can take a long time to come up. Our team needs to know that it's gone down. Our customer needs to know if it doesn't com up. Rather than making a pair of sources that we can then create separate alerts for, We make the first step the contact to our team, then enough blanks to account for the time it normally takes for the service to restart. After that, we add the customer's team, then a blank.

Our current ticketing system only accepts automated tickets via email. These little tricks allow us to have tickets only generated once per alert, and allow us to make sure we're not panicking our customers for failovers / failures that are recovering as expected.

Cole McDonald

·7 years ago

I'm working through this specific issue right now as we move from SCOM to LogicMonitor. Has anyone found a good workaround for this? I'm getting a dozen alerts for a single event every time a volume in our Hyper-V Cluster goes over capacity threshold.

Annie_Dunham

·9 years ago

Hi Mark - Thanks for taking the time to write this up. We're currently knee (possibly neck) deep in designing how take what has historically been a device - instance centric view of the world and extend it to best accommodate applications, services, and clusters. This is on our radar, and we'll keep your specific use case in mind.

1	SUYASH GAIKWAD
2	CH Chris Hasty
3	YA Yuliyan Angelov
4	KP Karsten Petersen
5	DW Darian Wong
6	DM Dhanikachalam Muthusamy