Any way to automate tasks via alerts?
Hi, I know LM doesn't support taking any action when there's an alert. However, I'm wondering if anyone has any neat ideas on how to accomplish something by way of some other automation program. Maybe Power Automate? Here's what I'm thinking. I have a module that monitors a service to see if it's running or not. If it's not, it generates an alert. This seems like the most basic thing to automate because all I'd want to do is run the start-service command to start it back up. So, I'm wondering if I can have the alert send an email to a certain address. That address could then watch for a certain email to arrive. It could then parse out the servername and service name and put that into a start-service -computername xxx -name yyy. Has anyone looked into doing anything like that and did you have any success? Thanks.93Views0likes9CommentsHow IT administrators can streamline operations using the LogicMonitor API
In this article, we’re going to review how LogicMonitor administrators can maximize efficiency and transform their IT operations using LogicMonitor’s REST API and PowerShell. We will cover the following use cases: Device onboarding/offboarding User management Retrieving data122Views3likes0CommentsExample script for automated alert actions via External Alerting
Below is a PowerShell script that's a handy starting point if you want to trigger actions based on specific alert types. In a nutshell, it takes a number of parameters from each alertand has a section of if/elsestatements where you can specify what to do based on the alert.It leverages LogicMonitor'sExternal Alertingfeature so the script runs local to whatever Collector(s)you configure it on. I included a couple of example actions forpinging a device and forrestarting a service.It also includes some handy (optional) functions for logging as well as attaching a noteto thealert in LogicMonitor. NOTE: this script is provided as-is and you will need to customize it to suit your needs. Automated actions are something that must be approached with careful planning and caution!! LogicMonitor cannot be responsible for inadvertent consequences of using thisscript. If you want try it out, here's how to get started: Update the variables in the appropriate section near the top of the script with optional API credentialsand/or log settings. Also change any of the if/elseif statements (starting around line #95) to suit your needs. Save the script onto your Collector server.I named the file"alert_central.ps1" but feel free to call it something else. Make note of it’s full path (ex: “C:\scripts\alert_central.ps1”). NOTE: it’s notrecommended to place it under the Collector's agent/lib directory (typically "C:\Program Files (x86)\LogicMonitor\Agent\lib") since that location can be overwritten by collector upgrades. In your LogicMonitor portal go to Settings, then External Alerting. Click the Add button. Set the 'Groups' field as needed to limit the actions to alerts from any appropriategroup of resources. (Be sure the group's devices would be reachable from the Collector running the script) Choose the appropriate Collector in the Collectorfield. Set Delivery Mechanismto "Script" Enter the name you saved the scriptas (in step #2)in theScriptfield (ex. "alert_central.ps1"). Paste the following into the Script Command Linefield (NOTE: if you add other parameters here then be sure to also add them to the 'Param' line at the top of the script): "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##""##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" Example of the completed Add External Alerting dialog Click Save. This uses LogicMonitor's External Alerting featureso there are some things to be aware of: Since the script is called foreveryalert, the section of if/then statements at the bottom of the script is important for filtering what specific alerts you want to take action on. The Collector(s) oversee the running of thescript, so be conscience to any additional overhead the script actions may cause. It could take up to 60 seconds for the script to trigger from the time the alert comes in. This example is a PowerShell script so best suited for Windows-based collectors, but could certainly be re-written as a shell script for Linux-based collectors. Here's a screenshot of acleared alert where the script auto-restarted a Windows service and attached a note based on its actions. Example note the script added to the alert reflecting the automated action that was taken Below is the PowerShell script: # ---- # This PowerShell script can be used as a starting template for enabling # automated remediation for alerts coming from LogicMonitor. # In LogicMonitor, you can use the External Alerting feature to pass all alerts # (or for a specific group of resources) to this script. # ---- # To use this script: # 1. Update the variables in the appropriate section below with optional API and log settings. # 2. Drop this script onto your Collector server under the Collector's agent/lib directory. # 3. In your LogicMonitor portal go to Settings, then click External Alerting. # 4. Click the Add button. # 5. Set the 'Groups' field as needed to limit the actions to a specific group of resources. # 6. Choose the appropriate Collector in the 'Collector' field. # 7. Set 'Delivery Mechanism' to "Script" # 8. Enter "alert_central.ps1" in the 'Script' field. # 9. Paste the following into the 'Script Command Line' field: # "##ALERTID##" "##ALERTSTATUS##" "##LEVEL##" "##HOSTNAME##" "##SYSTEM.SYSNAME##" "##DSNAME##" "##INSTANCE##" "##DATAPOINT##" "##VALUE##" "##ALERTDETAILURL##" "##DPDESCRIPTION##" # 10. Click Save. # The following line captures alert information passed from LogicMonitor (defined in step #9 above)... Param ($alertID = "", $alertStatus = "", $severity = "", $hostName = "", $sysName = "", $dsName = "", $instance = "", $datapoint = "", $metricValue = "", $alertURL = "", $dpDescription = "") ###--- SET THE FOLLOWING VARIABLES AS APPROPRIATE ---### # OPTIONAL: LogicMonitor API info for updating alert notes (the API user will need "Acknowledge" permissions)... $accessId = '' $accessKey = '' $company = '' # OPTIONAL: Set a filename in the following variable if you want specific alerts logged. (example: "C:\lm_alert_central.log")... $logFile = '' # OPTIONAL: Destination for syslog alerts... $syslogServer = '' ############################################################### ## HELPER FUNCTIONS (you likely won't need to change these) ## # Function for logging the alert to a local text file if one was specified in the $logFile variable above... Function LogWrite ($logstring = "") { if ($logFile -ne "") { $tmpDate = Get-Date -Format "dddd MM/dd/yyyy HH:mm:ss" # Using a mutex to handle file locking if multiple instances of this script trigger at once... $LogMutex = New-Object System.Threading.Mutex($false, "LogMutex") $LogMutex.WaitOne()|out-null "$tmpDate, $logstring" | out-file -FilePath $logFile -Append $LogMutex.ReleaseMutex()|out-null } } # Function for attaching a note to the alert... function AddNoteToAlert ($alertID = "", $note = "") { # Only execute this if the appropriate API information has been set above... if ($accessId -ne '' -and $accessKey -ne '' -and $company -ne '') { # Encode the note... $encodedNote = $note | ConvertTo-Json # API and URL request details... $httpVerb = 'POST' $resourcePath = '/alert/alerts/' + $alertID + '/note' $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath $data = '{"ackComment":' + $encodedNote + '}' # Get current time in milliseconds... $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) # Concatenate general request details... $requestVars_00 = $httpVerb + $epoch + $data + $resourcePath # Construct signature... $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars_00)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) # Construct headers... $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') # Make request to add note.. $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Body $data -Header $headers # Change the following if you want to capture API errors somewhere... # LogWrite "API call response: $response" } } function SendTo-SysLog ($IP = "", $Facility = "local7", $Severity = "notice", $Content = "Your payload...", $SourceHostname = $env:computername, $Tag = "LogicMonitor", $Port = 514) { switch -regex ($Facility) { 'kern' {$Facility = 0 * 8 ; break } 'user' {$Facility = 1 * 8 ; break } 'mail' {$Facility = 2 * 8 ; break } 'system' {$Facility = 3 * 8 ; break } 'auth' {$Facility = 4 * 8 ; break } 'syslog' {$Facility = 5 * 8 ; break } 'lpr' {$Facility = 6 * 8 ; break } 'news' {$Facility = 7 * 8 ; break } 'uucp' {$Facility = 8 * 8 ; break } 'cron' {$Facility = 9 * 8 ; break } 'authpriv' {$Facility = 10 * 8 ; break } 'ftp' {$Facility = 11 * 8 ; break } 'ntp' {$Facility = 12 * 8 ; break } 'logaudit' {$Facility = 13 * 8 ; break } 'logalert' {$Facility = 14 * 8 ; break } 'clock' {$Facility = 15 * 8 ; break } 'local0' {$Facility = 16 * 8 ; break } 'local1' {$Facility = 17 * 8 ; break } 'local2' {$Facility = 18 * 8 ; break } 'local3' {$Facility = 19 * 8 ; break } 'local4' {$Facility = 20 * 8 ; break } 'local5' {$Facility = 21 * 8 ; break } 'local6' {$Facility = 22 * 8 ; break } 'local7' {$Facility = 23 * 8 ; break } default {$Facility = 23 * 8 } #Default is local7 } switch -regex ($Severity) { '^(ac|up)' {$Severity = 1 ; break } # LogicMonitor "active", "ack" or "update" '^em' {$Severity = 0 ; break } #Emergency '^a' {$Severity = 1 ; break } #Alert '^c' {$Severity = 2 ; break } #Critical '^er' {$Severity = 3 ; break } #Error '^w' {$Severity = 4 ; break } #Warning '^n' {$Severity = 5 ; break } #Notice '^i' {$Severity = 6 ; break } #Informational '^d' {$Severity = 7 ; break } #Debug default {$Severity = 5 } #Default is Notice } $pri = "<" + ($Facility + $Severity) + ">" # Note that the timestamp is local time on the originating computer, not UTC. if ($(get-date).day -lt 10) { $timestamp = $(get-date).tostring("MMM d HH:mm:ss") } else { $timestamp = $(get-date).tostring("MMM dd HH:mm:ss") } # Hostname does not have to be in lowercase, and it shouldn't have spaces anyway, but lowercase is more traditional. # The name should be the simple hostname, not a fully-qualified domain name, but the script doesn't enforce this. $header = $timestamp + " " + $sourcehostname.tolower().replace(" ","").trim() + " " #Cannot have non-alphanumerics in the TAG field or have it be longer than 32 characters. if ($tag -match '[^a-z0-9]') { $tag = $tag -replace '[^a-z0-9]','' } #Simply delete the non-alphanumerics if ($tag.length -gt 32) { $tag = $tag.substring(0,31) } #and truncate at 32 characters. $msg = $pri + $header + $tag + ": " + $content # Convert message to array of ASCII bytes. $bytearray = $([System.Text.Encoding]::ASCII).getbytes($msg) # RFC3164 Section 4.1: "The total length of the packet MUST be 1024 bytes or less." # "Packet" is not "PRI + HEADER + MSG", and IP header = 20, UDP header = 8, hence: if ($bytearray.count -gt 996) { $bytearray = $bytearray[0..995] } # Send the message... $UdpClient = New-Object System.Net.Sockets.UdpClient $UdpClient.Connect($IP,$Port) $UdpClient.Send($ByteArray, $ByteArray.length) | out-null } # Empty placeholder for capturing any note we might want to attach back to the alert... $alertNote = "" # Placeholder for whether we want to capture an alert in our log. Set to true if you want to log everything. $logThis = $false ############################################################### ## CUSTOMIZE THE FOLLOWING AS NEEDED TO HANDLE SPECIFIC ALERTS FROM LOGICMONITOR... # Actions to take if the alert is new or re-opened (note: status will be "active" or "clear")... if ($alertStatus -eq 'active') { # Perform actions based on the type of alert... # Ping alerts... if ($dsName -eq 'Ping' -and $datapoint -eq 'PingLossPercent') { # Insert action to take if a device becomes unpingable. In this example we'll do a verification ping & capture the output... $job = ping -n 4 $sysName # Restore line feeds to the output... $job = [string]::join("`n", $job) # Add ping results as a note on the alert... $alertNote = "Automation script output: $job" # Log the alert... $logThis = $true # Restart specific Windows services... } elseif ($dsName -eq 'WinService-' -and $datapoint -eq 'State') { # List of Windows Services to match against. Only if one of the following are alerting will we try to restart it... $serviceList = @("Print Spooler","Service 2") # Note: The PowerShell "-Contains" operator is exact in it's matching. Replace it with "-Match" for a loser match. if ($serviceList -Contains $instance) { # Get an object reference to the Windows service... $tmpService = Get-Service -DisplayName "$instance" -ComputerName $sysName # Only trigger if the service is still stopped... if ($tmpService.Status -eq "Stopped") { # Start the service... $tmpService | Set-Service -Status Running # Capture the current state of the service as a note on the alert... $alertNote = "Attempted to auto-restart the service. Its new status is " + $tmpService.Status + "." } # Log the alert... $logThis = $true } # Actions to take if a website stops responding... } elseif ($dsName -eq 'HTTPS-' -and $datapoint -eq 'CantConnect') { # Insert action here to take if there's a website error... # Example of sending a syslog message to an external server... $syslogMessage = "AlertID:$alertID,Host:$sysName,AlertStatus:$alertStatus,LogicModule:$dsName,Instance:$instance,Datapoint:$datapoint,Value:$metricValue,AlertDescription:$dpDescription" SendTo-SysLog $syslogServer "" $severity $syslogMessage $hostName "" "" # Attach a note to the LogicMonitor alert... $alertNote = "Sent syslog message to " + $syslogServer # Log the alert... $logThis = $true } } ############################################################### ## Final functions for backfilling notes and/or logging as needed ## (you likely won't need to change these) # Section that updates the LogicMonitor alert if 'alertNote' is not empty... if ($alertNote -ne "") { AddNoteToAlert $alertID $alertNote } if ($logThis) { # Log the alert (only triggers if a filename is given in the $logFile variable near the top of this script)... LogWrite "$alertID,$alertStatus,$severity,$hostName,$sysName,$dsName,$instance,$datapoint,$metricValue,$alertURL,$dpDescription" }1.8KViews23likes5CommentsLogicMonitor, Groovy, Python and APIs Ohhh my
You just started your new job and they have LogicMonitor as their observability tool, that’s great! One huge step up from other organizations you’ve worked with. The last guy that worked on the platform did a pretty good job of keeping up with the platform but you see some improvements that would really help with the organization of the portal. You want to create multiple folder structures. The first group of nested folders will be based on the equipment location the second will be based on the equipment task. This will allow you to provide additional people access to the portal but limit their visibility into only the things that are in their location or their job function. Yup sounds great, your coworkers like it and more importantly your manager approves. Now how are you going to assign 700 pieces of equipment at 30 different sites to the correct folder with the correct permissions? Oh yeah you got a secret weapon, APIs! The goal of this and subsequent posts is to try to demystify APIs. What are they? How are they used? More importantly, how do they benefit you and of course your organization. This will be the first installment of community posts discussing APIs and how they are used to perform tasks within the LogicMonitor portal. If you are like I was and just dipping your toes into APIs and have started looking at videos on youtube or reading about the power of APIs you might be a bit overwhelmed. That’s perfectly fine, it’s totally understandable! APIs are a new tool that will help you perform repetitive tasks within LogicMonitor or any other application that provides APIs. Since LogicMonitor is a SaaS application you don’t have a database where you can perform queries or other actions. So queries and other tasks are replaced with the use of APIs. As you might have learned growing up, knowing what something is can make it less scary. So, what does API mean? API is an acronym for Application Programming Interface. See not that scary, yeah right. I can tell you that the vast majority of people that use APIs on a regular basis have no idea what the acronym stands for. What really is an API? An API, more specifically a REST API, is an instruction that you can use with an application to do something. The word DO is the key here, better said DO is the verb. The verbs that are used for API calls are as follows: GET allows you to get some information from the application POST allows you create something within the application DELETE allows you to delete something within the application PUT allows you to update something within the application PATCH allows you to update something within the application What’s that that I’m hearing through time and network space? Yup I’m hearing WHOA WHOA WHOA PUT allows you to update something within the application PATCH allows you to update something within the application Both have the same function? Why have two verbs that do the same thing? Well even though both perform a similar function each does it in a different manner. PUT replaces the entire item that you are updating with new data PATCH only replaces a portion of the item you are updating I will go into more detail for each verb in future installments of this series. If you are still with me. Really? I haven’t scared, bored or frustrated you with my ramblings yet? Ok so you are with me, cool. Why should you use up some of the storage in your brain to learn about APIs? APIs will simplify your life while using not only LogicMonitor but any application that provides you API access. Here are a couple more real world examples of how APIs can be leveraged to automate tasks within LogicMonitor A customer has several hundred stores across the United States. The customer wants the stores placed in nested folders Store folder - contains all devices at the store District folder - multiple stores make up a district State folder - All districts within a state Region folder - Multiple states make a region Trying to create this kind of structure manually would take days and be prone to errors. You can create the nested group structure by leveraging APIs with your favorite scripting language, I prefer Python, and a CSV file containing store information. While creating the group structure you can also add any necessary custom properties. Another real world example of leveraging APIs A customer has a very large user list within LogicMonitor, almost 800 users. These users can change roles and/or have their access suspended. Going through all 800 users to set roles, change user groups and update the account status can be daunting and honestly a brain melting task that you probably wouldn’t wish even on the guy that stole that prime parking spot from you at the mall during christmas shopping season. In comes super API and with a bit of Python coding you can perform this whole task automagically. PLUS, since this task was performed using a script you will be able to do this task repeatedly without breaking a sweat. In my next installment I will show how to start using APIs. Since it’s an unspoken rule that all guides need to provide links to relevant information, here are my hopefully useful links. LogicMonitor Swagger A listing of all actions that can be performed using the LogicMonitor API. Learning Python If you are new to coding I recommend Python since it is OS agnostic and has a massive user base and information. Learning Groovy Here you thought groovy was just a word from the 60s and 70s, Au contraire mon frère. Groovy is the primary scripting language used within LogicMonitor. We will go deeper into this Groovy language later on. LogicMonitor How can we do anything without including a link to the best observability platform around.154Views10likes1CommentPython SDK - JSON
Hi All, Apologies if this is a simple question, however I cannot find any documentation that can assist. My goal here is to have severity 3 alerts in a JSON file, however I am unable to convert this list ‘alerts’ to JSON and I cannot find any documentation in the SDK v3 that could address this. # create an instance of the API class api_instance = logicmonitor_sdk.LMApi(logicmonitor_sdk.ApiClient(configuration)) id = 138 # Integer | size = 1000 # Integer | (optional) offset = 0 # Integer | (optional) filter = f'severity:3,cleared:true,startEpoch>:{startEpoch},endEpoch<:{endEpoch}' # String | (optional) #Loop shiznit alerts = [] end_found = False while not end_found: current = api_instance.get_alert_list_by_device_group_id(id, size=size, offset=offset, filter=filter).items alerts += current offset += len(current) end_found = len(current) != size #Print and encode result to json. This allows you to use this variable later in an API Put. with open('./customer/json/cleared_sev3_alerts.json', 'w') as content_output_json: result_encoded = json.JSONEncoder(sort_keys=False, indent =4).encode(alerts) content_output_json.write(str(alerts)) content_output_json.close() print(len(alerts)) Whenever I try to encode this I get the following alert. TypeError: Object of type Alert is not JSON serializable Doing a jsonDump gives us the same output and I’ve scoured stackoverflow for a way to fix this, but no luck. Does anyone know how I can get around this? Thank you so muchSolved116Views3likes1CommentAlerts API - Size Limit
Hi Everyone, I am running into a size limit issue in my pursuit of creating a quarterly report for a customer. What I am trying to do is narrow down my filter to have any cleared alert that is a severity 4 that was closed during that quarter. My issue is two-fold. I am not sure of the syntax that would only show alerts that cleared during that quarter (I am trying to do the equivalent of ‘between’ in python IF statements)and the size limit of 1000 is limiting because I cannot get a count of cleared alerts. I couldn’t find anything regarding pagination that can be used. I have also attempted to do the F12 then network button trick, but there is nothing that stands out there or anything that I can identify. And I have tried looking at other questions and couldn’t find anything relating this. This is my query: ?size=1000&filter=severity:4,cleared:true,startEpoch>:{startEpoch}&filter=severity:4,cleared:true,endEpoch<:{endEpoch} Any guidance would be greatly appreciated.Solved361Views2likes2CommentsAutomation
As a CSM, I’m often told that they are tasked with automation - but when I dig deeper, often the company doesn’t know - they just want to take things off the plate of their teams. Have you been tasked with automation? What kind are you looking for? If you have put automation in place, please let me know what it does and how successful it is for you. Thank you36Views1like1CommentTaking automated actions on alerts
Hiya, I've been thinking about having LogicMonitor take corrective actions on some tasks. For example archiving files if a drive is getting close to full. I know that LogicMonitor does not have any event action abilities based on alerts, where you can run something when an alert occurs. I know that you can write scripts for DataSources, and you can do anything in a script so in theory you can have a DataSource do the cleanup during a check. I just feel there is little reason to do that as you are basically just using LogicMonitor as a glorified centralized Task Scheduler. - Thresholds are checked AFTER the script is run, so you can't use LogicMonitor's thresholds, SDTs or other functionality. The script would need to hard code threshold or have them placed in properties. - DataSource scripts have a limited time to run (default 2min I think), so it needs to complete fast or kick off a separate process. - Might be possible to implement Custom HTTP Integration, but that would require some endpoint to receive and act on it. I also think there is a limit on how many integrations you have and might be too limited to have one generic integration. I'm just talking out loud here and wondering if anyone else has thought about or implemented something like this. Thanks!25Views1like1CommentScript/API to automate phone call to IVR server
Hi all, I'm trying to find ideas, scripts, processes for a function our customer approached us with. In short, they want LM to place a call once per hour to the Interactive Voice Response application to ensure the IVR system answers the call, prompts are being displayed, and host database connection is established. My only is guess is to somehow create a Datasource that runs a script to interact with an API that somehow makes a phone call and then displays information as a result. I would like to know if this already part of LM or would need to be heavily developed in-house or is even remotely possible to implement. If you have had any success with this type of monitoring, I'd appreciate your insight. Thanks much8Views0likes4CommentsCustom reports
I have a requirement to automate reporting which is straightforward. However I also have a requirement to have the report include our company logo and use formatting of our choosing. The idea would be to have these reports automatically generated and delivered to customers without manual intervention. Hopefully this is a feature which I have yet to discover and if anyone has cracked it I would love to know the process.Solved10Views0likes2Comments