Example script for automated alert actions via External Alerting
Below is a PowerShell script that's a handy starting point if you want to trigger actions based on specific alert types. In a nutshell, it takes a number of parameters from each alertand has a section of if/elsestatements where you can specify what to do based on the alert.It leverages LogicMonitor'sExternal Alertingfeature so the script runs local to whatever Collector(s)you configure it on. I included a couple of example actions forpinging a device and forrestarting a service.It also includes some handy (optional) functions for logging as well as attaching a noteto thealert in LogicMonitor. NOTE: this script is provided as-is and you will need to customize it to suit your needs. Automated actions are something that must be approached with careful planning and caution!! LogicMonitor cannot be responsible for inadvertent consequences of using thisscript. If you want try it out, here's how to get started: Update the variables in the appropriate section near the top of the script with optional API credentialsand/or log settings. Also change any of the if/elseif statements (starting around line #95) to suit your needs. Save the script onto your Collector server.I named the file"alert_central.ps1" but feel free to call it something else. Make note of it’s full path (ex: “C:\scripts\alert_central.ps1”). NOTE: it’s notrecommended to place it under the Collector's agent/lib directory (typically "C:\Program Files (x86)\LogicMonitor\Agent\lib") since that location can be overwritten by collector upgrades. In your LogicMonitor portal go to Settings, then External Alerting. Click the Add button. Set the 'Groups' field as needed to limit the actions to alerts from any appropriategroup of resources. (Be sure the group's devices would be reachable from the Collector running the script) Choose the appropriate Collector in the Collectorfield. Set Delivery Mechanismto "Script" Enter the name you saved the scriptas (in step #2)in theScriptfield (ex. "alert_central.ps1"). Paste the following into the Script Command Linefield (NOTE: if you add other parameters here then be sure to also add them to the 'Param' line at the top of the script): "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##""##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" Example of the completed Add External Alerting dialog Click Save. This uses LogicMonitor's External Alerting featureso there are some things to be aware of: Since the script is called foreveryalert, the section of if/then statements at the bottom of the script is important for filtering what specific alerts you want to take action on. The Collector(s) oversee the running of thescript, so be conscience to any additional overhead the script actions may cause. It could take up to 60 seconds for the script to trigger from the time the alert comes in. This example is a PowerShell script so best suited for Windows-based collectors, but could certainly be re-written as a shell script for Linux-based collectors. Here's a screenshot of acleared alert where the script auto-restarted a Windows service and attached a note based on its actions. Example note the script added to the alert reflecting the automated action that was taken Below is the PowerShell script: # ---- # This PowerShell script can be used as a starting template for enabling # automated remediation for alerts coming from LogicMonitor. # In LogicMonitor, you can use the External Alerting feature to pass all alerts # (or for a specific group of resources) to this script. # ---- # To use this script: # 1. Update the variables in the appropriate section below with optional API and log settings. # 2. Drop this script onto your Collector server under the Collector's agent/lib directory. # 3. In your LogicMonitor portal go to Settings, then click External Alerting. # 4. Click the Add button. # 5. Set the 'Groups' field as needed to limit the actions to a specific group of resources. # 6. Choose the appropriate Collector in the 'Collector' field. # 7. Set 'Delivery Mechanism' to "Script" # 8. Enter "alert_central.ps1" in the 'Script' field. # 9. Paste the following into the 'Script Command Line' field: # "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##" "##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" # 10. Click Save. # The following line captures alert information passed from LogicMonitor (defined in step #9 above)... Param ($alertID = "", $alertStatus = "", $severity = "", $hostName = "", $sysName = "", $dsName = "", $instance = "", $datapoint = "", $metricValue = "", $alertURL = "", $dpDescription = "") ###--- SET THE FOLLOWING VARIABLES AS APPROPRIATE ---### # OPTIONAL: LogicMonitor API info for updating alert notes (the API user will need "Acknowledge" permissions)... $accessId = '' $accessKey = '' $company = '' # OPTIONAL: Set a filename in the following variable if you want specific alerts logged. (example: "C:\lm_alert_central.log")... $logFile = '' # OPTIONAL: Destination for syslog alerts... $syslogServer = '' ############################################################### ## HELPER FUNCTIONS (you likely won't need to change these) ## # Function for logging the alert to a local text file if one was specified in the $logFile variable above... Function LogWrite ($logstring = "") { if ($logFile -ne "") { $tmpDate = Get-Date -Format "dddd MM/dd/yyyy HH:mm:ss" # Using a mutex to handle file locking if multiple instances of this script trigger at once... $LogMutex = New-Object System.Threading.Mutex($false, "LogMutex") $LogMutex.WaitOne()|out-null "$tmpDate, $logstring" | out-file -FilePath $logFile -Append $LogMutex.ReleaseMutex()|out-null } } # Function for attaching a note to the alert... function AddNoteToAlert ($alertID = "", $note = "") { # Only execute this if the appropriate API information has been set above... if ($accessId -ne '' -and $accessKey -ne '' -and $company -ne '') { # Encode the note... $encodedNote = $note | ConvertTo-Json # API and URL request details... $httpVerb = 'POST' $resourcePath = '/alert/alerts/' + $alertID + '/note' $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath $data = '{"ackComment":' + $encodedNote + '}' # Get current time in milliseconds... $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) # Concatenate general request details... $requestVars_00 = $httpVerb + $epoch + $data + $resourcePath # Construct signature... $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars_00)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) # Construct headers... $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') # Make request to add note.. $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers # Change the following if you want to capture API errors somewhere... # LogWrite "API call response: $response" } } function SendTo-SysLog ($IP = "", $Facility = "local7", $Severity = "notice", $Content = "Your payload...", $SourceHostname = $env:computername, $Tag = "LogicMonitor", $Port = 514) { switch -regex ($Facility) { 'kern' {$Facility = 0 * 8 ; break } 'user' {$Facility = 1 * 8 ; break } 'mail' {$Facility = 2 * 8 ; break } 'system' {$Facility = 3 * 8 ; break } 'auth' {$Facility = 4 * 8 ; break } 'syslog' {$Facility = 5 * 8 ; break } 'lpr' {$Facility = 6 * 8 ; break } 'news' {$Facility = 7 * 8 ; break } 'uucp' {$Facility = 8 * 8 ; break } 'cron' {$Facility = 9 * 8 ; break } 'authpriv' {$Facility = 10 * 8 ; break } 'ftp' {$Facility = 11 * 8 ; break } 'ntp' {$Facility = 12 * 8 ; break } 'logaudit' {$Facility = 13 * 8 ; break } 'logalert' {$Facility = 14 * 8 ; break } 'clock' {$Facility = 15 * 8 ; break } 'local0' {$Facility = 16 * 8 ; break } 'local1' {$Facility = 17 * 8 ; break } 'local2' {$Facility = 18 * 8 ; break } 'local3' {$Facility = 19 * 8 ; break } 'local4' {$Facility = 20 * 8 ; break } 'local5' {$Facility = 21 * 8 ; break } 'local6' {$Facility = 22 * 8 ; break } 'local7' {$Facility = 23 * 8 ; break } default {$Facility = 23 * 8 } #Default is local7 } switch -regex ($Severity) { '^(ac|up)' {$Severity = 1 ; break } # LogicMonitor "active", "ack" or "update" '^em' {$Severity = 0 ; break } #Emergency '^a' {$Severity = 1 ; break } #Alert '^c' {$Severity = 2 ; break } #Critical '^er' {$Severity = 3 ; break } #Error '^w' {$Severity = 4 ; break } #Warning '^n' {$Severity = 5 ; break } #Notice '^i' {$Severity = 6 ; break } #Informational '^d' {$Severity = 7 ; break } #Debug default {$Severity = 5 } #Default is Notice } $pri = "<" + ($Facility + $Severity) + ">" # Note that the timestamp is local time on the originating computer, not UTC. if ($(get-date).day -lt 10) { $timestamp = $(get-date).tostring("MMM d HH:mm:ss") } else { $timestamp = $(get-date).tostring("MMM dd HH:mm:ss") } # Hostname does not have to be in lowercase, and it shouldn't have spaces anyway, but lowercase is more traditional. # The name should be the simple hostname, not a fully-qualified domain name, but the script doesn't enforce this. $header = $timestamp + " " + $sourcehostname.tolower().replace(" ","").trim() + " " #Cannot have non-alphanumerics in the TAG field or have it be longer than 32 characters. if ($tag -match '[^a-z0-9]') { $tag = $tag -replace '[^a-z0-9]','' } #Simply delete the non-alphanumerics if ($tag.length -gt 32) { $tag = $tag.substring(0,31) } #and truncate at 32 characters. $msg = $pri + $header + $tag + ": " + $content # Convert message to array of ASCII bytes. $bytearray = $([System.Text.Encoding]::ASCII).getbytes($msg) # RFC3164 Section 4.1: "The total length of the packet MUST be 1024 bytes or less." # "Packet" is not "PRI + HEADER + MSG", and IP header = 20, UDP header = 8, hence: if ($bytearray.count -gt 996) { $bytearray = $bytearray[0..995] } # Send the message... $UdpClient = New-Object System.Net.Sockets.UdpClient $UdpClient.Connect($IP,$Port) $UdpClient.Send($ByteArray, $ByteArray.length) | out-null } # Empty placeholder for capturing any note we might want to attach back to the alert... $alertNote = "" # Placeholder for whether we want to capture an alert in our log. Set to true if you want to log everything. $logThis = $false ############################################################### ## CUSTOMIZE THE FOLLOWING AS NEEDED TO HANDLE SPECIFIC ALERTS FROM LOGICMONITOR... # Actions to take if the alert is new or re-opened (note: status will be "active" or "clear")... if ($alertStatus -eq 'active') { # Perform actions based on the type of alert... # Ping alerts... if ($dsName -eq 'Ping' -and $datapoint -eq 'PingLossPercent') { # Insert action to take if a device becomes unpingable. In this example we'll do a verification ping & capture the output... $job = ping -n 4 $sysName # Restore line feeds to the output... $job = [string]::join("`n", $job) # Add ping results as a note on the alert... $alertNote = "Automation script output: $job" # Log the alert... $logThis = $true # Restart specific Windows services... } elseif ($dsName -eq 'WinService-' -and $datapoint -eq 'State') { # List of Windows Services to match against. Only if one of the following are alerting will we try to restart it... $serviceList = @("Print Spooler","Service 2") # Note: The PowerShell "-Contains" operator is exact in it's matching. Replace it with "-Match" for a loser match. if ($serviceList -Contains $instance) { # Get an object reference to the Windows service... $tmpService = Get-Service -DisplayName "$instance" -ComputerName $sysName # Only trigger if the service is still stopped... if ($tmpService.Status -eq "Stopped") { # Start the service... $tmpService | Set-Service -Status Running # Capture the current state of the service as a note on the alert... $alertNote = "Attempted to auto-restart the service. Its new status is " + $tmpService.Status + "." } # Log the alert... $logThis = $true } # Actions to take if a website stops responding... } elseif ($dsName -eq 'HTTPS-' -and $datapoint -eq 'CantConnect') { # Insert action here to take if there's a website error... # Example of sending a syslog message to an external server... $syslogMessage = "AlertID:$alertID,Host:$sysName,AlertStatus:$alertStatus,LogicModule:$dsName,Instance:$instance,Datapoint:$datapoint,Value:$metricValue,AlertDescription:$dpDescription" SendTo-SysLog $syslogServer "" $severity $syslogMessage $hostName "" "" # Attach a note to the LogicMonitor alert... $alertNote = "Sent syslog message to " + $syslogServer # Log the alert... $logThis = $true } } ############################################################### ## Final functions for backfilling notes and/or logging as needed ## (you likely won't need to change these) # Section that updates the LogicMonitor alert if 'alertNote' is not empty... if ($alertNote -ne "") { AddNoteToAlert $alertID $alertNote } if ($logThis) { # Log the alert (only triggers if a filename is given in the $logFile variable near the top of this script)... LogWrite "$alertID,$alertStatus,$severity,$hostName,$sysName,$dsName,$instance,$datapoint,$metricValue,$alertURL,$dpDescription" }1.3KViews21likes5CommentsPowershell: Expanding a variable inside single quotes to make an API call
Hi, I’m trying to use Powershell to make an API call and pass in the SDT start/end time as a variable. However, because the call has to be within single quotes, I can’t get it to expand the variable. I have $now set to the epoch time of now and $later set to the epoch time two hours later. I’m trying to make a call with something like: '{"sdtType":1,"type":"DeviceSDT","deviceId":13771,"startDateTime":$now,"endDateTime":$later}' Because the whole string is within a single quote, I can’t get PS to parse out the two variables and use their data. I tried escaping the quotes and also using -f formatting, but can’t get anything to work. Just wondering if anyone here knows how I can make this happen. I tried swapping all the single and double quotes, which lets the Powershell work, but then the API fails because it requires double quotes to be surround the field names and won't accept single quotes. Thanks.Solved300Views13likes4CommentsHow to create datasources from powershell script
Hello, I wrote a PS script that takes a look at all issued certs on my microsoft CA and outputs 4columns, The name of the cert, the effective date, the expiration date and the days remaining until cert expiration. Here is the script for reference: $templates = @('x.x.x.x.x.x.x.x.x.x.x.x') $certs = $null ForEach($template in $templates){ $certs += certutil -view -restrict "certificate template=$template,Disposition=20" -out "CommonName,NotBefore,NotAfter,CertificateTemplate" } $i = 0 $output = @( ForEach($line in $certs){ If($line -like "*Issued Common Name: *"){ $asdf = New-Object -TypeName psobject $asdf | Add-Member -membertype noteproperty -name 'Common Name' -value (($certs[$i] -replace "Issued Common Name: ","") -replace '"','').trim() $asdf | Add-Member -membertype NoteProperty -name 'Effective Date' -value (($certs[$i+1] -replace "Certificate Effective Date: ","") -replace '\d+\:\d+\s+\w+','').trim() $asdf | Add-Member -membertype NoteProperty -name 'Expiration Date' -value (($certs[$i+2] -replace "Certificate Expiration Date: ","") -replace '\d+\:\d+\s+\w+','').trim() $expirationDate = [datetime]::MinValue [datetime]::TryParse($asdf.'Expiration Date', [ref]$expirationDate) $daysRemaining = ($expirationDate - (Get-Date)).Days $asdf | Add-Member -MemberType NoteProperty -Name 'Days Remaining' -Value $daysRemaining $asdf } $i++ } ) $output How can I create a datasource within LM that will parse out each common name, tie it to its corresponding “days remaining” value and alert based on that? Is this possible?Solved234Views13likes6CommentsCylance Offline Mode
We are looking to try to utilize LM and monitor whether Cylance is running in offline mode on a Windows server. Our SOC was able to determine that if it switches to offline mode (which can happen without the NIC going down), it adds a registry entry: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Cylance\Desktop\VenueStatus Ideally we’d like to monitor for IF that registry item exists and when it does see it, it would send us an alert. I’m assuming this could be done through some PS scripting. But I’m not sure how to have that data interpreted into a usable data or event source in LM. Would appreciate any help you guys can offer here!121Views9likes2CommentsWindows Services Monitoring with quite a bit more Automation applied
So today we use LM's Microsoft Windows ServicesDataSource to monitor Windows Services. This DS uses Groovy Script and WMI calls under the hood to fetch the service metrics like state, start mode, status, etc... Everything works fine but one of the prerequisites is to go and manually populate the list of Windows services which then the DS parses out as a WILDVALUE variable in the script. You know, go to the device, click on Down Arrow (Manage Resource Options) --> Add Additional Monitoring --> and CHOOSE from the list of Windows Services. Rinse and Repeat and Save. Then the DS goes to work. Well, what if you have a list of over 100 Windows Services you need to add to let's say 20 Windows devices? That would take forever to populate that list manually... That's a problem number 1. Scratch that. This is not really a problem since one can run a PowerShell script (or Groovy Script) to perform this task using undocumented - but working very well - LM API calls. That problem is solved. Next - This list of over 100 Services needs to be *refreshed* every let's say 24 hours to remove nonexistent services and add new ones based on the Regex filter. That's a problem number 2. And again, one can do it programmatically running API calls but this is where I am trying to figure out how to do it. Run my script as a custom PropertySource? I am not really writing Resource Properties, I am updating instance list (Windows Services) within Additional Monitoring on bunch of Resources. Plus PropertySources are applied when ActiveDiscovery is run which is what, every 24 hours? Or should I write custom DataSource that would accomplish this refresh and specify 1 day collection period? Thanks.Solved103Views4likes2CommentsCreating a propertySource to populate a NOC widget in a dashboard... need ## in a string.
The NOC widget items have a field that requires me to have the string"##RESOURCEGROUP##" pushed through the JSON into the NOC Item…since I’m using a propertySource to run the script on a schedule (I have a larger VM with a collector with a longer script timeout just for doing deeper scripted work through the API or Full Domain sweep types of things that will take more time), The LM System is going to try to replace that at run time rather than returning the explicit string. Who knows the correct escape sequence for turning that into a string literal on its way into the RestApi Patch? Scripting questions through support is best effort, and I don’t usually come with easy questions.Solved98Views11likes5CommentsBatch PowerShell Script
Hello! I am running a batch PowerShell script in LogicMonitor. My intended purpose is to extract the end date SAML certificateof a Enterprise Application in my Azure environment. My Active discovery script runs perfectly. However when I test my collector script, it believes my ##WILDVALUE## is an empty string. When I poll for the data for my instance, it retrieves the correct value. However, No Data gets returned when I enable my instances. My WILDVALUE does not contain any of the invalid characters. My Datapoint is in the format of ##WILDVALUE##.key $EnterpriseAppSingle = "##WILDVALUE##" $EnterpriseApp = Get-AzAdServicePrincipal -ObjectID $EnterpriseAppSingle $CertExpires = ($EnterpriseApp.PasswordCredentials.EndDateTime[0] - $(Get-Date)).Days write-host “##WILDVALUE##.DaysLeft=$CertExpires" Any insight to this would be helpful!Solved68Views2likes2CommentsWhat should I do for SPSE is busy causing powershell request rejected?
Hi Guru, lately there is some instance triggering “NoData” Alert. All the alert from same server, andI found this error message from the Wrapper log, [SPSEEngine.execute:201] Execution PowerShell script exception, CONTEXT=script=datacollecting-IPAddress-DataSource-__ResourceName_FileName.stdout.ps1, errmsg= SPSE is busy, powershell request was rejected. After I restart the logic monitor service in collector server, it usually back to normal but will reoccurred after 1-2hours. Do I need to increase the timeout config or any other fix recommendation?Solved50Views7likes1CommentCylance Offline Mode Monitoring
We are looking to try to utilize LM and monitor whether Cylance is running in offline mode on a Windows server. Our SOC was able to determine that if it switches to offline mode (which can happen without the NIC going down), it adds a registry entry: Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Cylance\Desktop\VenueStatus Ideally we’d like to monitor for IF that registry item exists and when it does see it, it would send us an alert. I’m assuming this could be done through some PS scripting. But I’m not sure how to have that data interpreted into a usable data or event source in LM. Would appreciate any help you guys can offer here.46Views6likes2CommentsFinding the culprit for TCP_StatsCollector ConnectionsEstablished alert for Windows collectors
From the collector’s device page in the LM Portal or the collectors page, get to a debug console, then here’s your !POSH one-liner to get info about the destination device that is holding your ports captive. netstat -an| sls establish | foreach { ($_ -split "\s+")[3] } | group | sort count | select count, name -last 10 In the Netstat, a shows all, n shows IP addresses rather than solving the DNS for it. TheSelect-String (aliased as sls)passes only the “Established” connection entries from the netstat down the pipeline. The foreach{} splits each line ($_ is the current object being iterated by the foreach loop) on contiguous whitespace (I use this a lot!) and takes the third element (remote address:port) to passdown the pipeline It then passes Group-Object (aliased as group) which bundles identical strings and Sort-Object (aliased as sort)by the count property of the group object. The select displays grabs the calculated match count and the name properties to limit display and just shows the -last 10 of them (which are the biggest number of matched lines due to the sort previously applied. This should give you the target/s for troubleshooting further.45Views11likes5Comments