John_Lockie
Joined 4 years ago
User Profile
User Widgets
Contributions
Re: Monitoring Linux Processes (their status, resource usage, etc ) via SSH?
On 6/27/2022 at 10:27 AM, Austin Culbertson said: That being said, my personal belief is that process monitoring is wholly insufficient for monitoring the state of your hosts and processes (good for collecting metrics, though). I would encourage folks to investigate how they can introspect the health of their process/services via whatever means available when possible (Think: performing a DB query instead of monitoring for MySQL/Postgres; Running `service <name> status` commands instead of simply checking if the process is running, etc). This takes a little more work, but is ABSOLUTELY worth the time investment. Austin, could not agree more! We have a scheduler platform that relies on processes running on remote systems in order to be able to execute. They're not even registered as services (in Linux). And if the process fails the job fails. We would prefer to know if a job will fail before it runs, so monitoring the process is (so far) the best solution we can come up with. On another note: I really wish LogicMonitor would support webhooks....24Views0likes0Commentsdocusign health monitoring
Has anyone figured out a good way to monitor docusign? They post their status here: https://status.docusign.com/ but it's not a great way to scrap the page for new incidents. They have RSS feed, but LM does not support RSS (that I know of). And, AFAIK there is no cloud integration w/ docusign. Docusign is a crit service for us. We are moving toward synthetics so that might be a way, but in the meantime not sure if there's a simpler approach I am not thinking of....117Views16likes6CommentsRe: Using Collector to Run SQL Query....
This is great feedback....thank you both. Let me take this back to the team and see if we can use any of these ideas. Mike: to your comments in your 2nd and 3rd paragraphs: we initially assumed these errors would be in a $log file somwhere, and we could just monitor that. But as it turns out, this application only logs these exceptions in SQL (from what we have found so far). These are 1 time events, and the log itself is fairly inactive/only throws events when there are exceptions (failed login, failed ETL job, etc.). To that point, we had an automated process fail because SMTP was not working upstream. So the ability to send data blob out through SMTP did not work. And our secondary stages for this ETL process depend on an automated mailbox task, etc. So we did not know there was an issue until $business told us...which is not good. Ideally, when the app throws this exception in SQL logs we are alerted. We will have to manually check why an exception was thrown and then check further...but it's better than nothing. In meantime, I am working w/ devs to find out if there's another point in the process we can detect failures. But it's very tricky to do dead man switches in email....61Views1like0CommentsUsing Collector to Run SQL Query....
We have a custom datapoint we built that runs a basic "select top 1" command to get application exceptions (which the app stores in SQL) from a remote sql server. It appears that LM can only pull down integer data when doing this? In other words, this table we are looking in is for application "ExceptionLog" and has columns for "User", "Severity" and "Title" and "EventID". If any of the columns are anything other than an integer, it returns "No Data" in LM. There are particular service account we want to watch out for - so ideally our query can just "select top 1 User" every few minutes, then configure alert any time user == service account. Is this expected behavior, and has anyone found a workaround for this?258Views5likes9CommentsNotification for Failed Cloud Integration Collector....?
Example: with API based collection in Azure, I notice that the LM platform does not alert us when the API secret expires and collection fails. Is there a setting buried somewhere for this? It seems like our LM tenant should light up like a Christmas tree if collection fails due to bad auth....no? Also, side note: is there a datapoint for alerting when app registration client secrets are about to expire? That would be one good method.11Views0likes0CommentsRe: IssueWarningQuote / Exchange Online Mailbox Quota Alerts?
On 12/20/2021 at 11:00 AM, Mike Moniz said: I would ask support if they can get you a copy of one of the old DataSources as they do most of the work. I switched jobs so I no longer have a copy myself. Some generic suggestions: Don't suggest coding thresholds in DataSources directly, instead provide values that LM can set thresholds on. For example don't code QuoteWarning as 0=ok, 1=almost at warning, 2=over quota. Instead do something like RemainingBeforeQuotaWarning (size minus quota) or RemainingBeforeQuotaReadOnly that you can setup LM to what warning level you want. You then don't need to worry about quota or mailbox size changing. Connecting to MS will sometimes time out, so you might want to code multiple attempts to login in the code If the script takes too long (>2 min default), LM will kill the script which will leave the session option to MS. MS will only allow 3 at once, so you may want to keep it only open as long as needed and perhaps kill any existing session before starting. Also keep that in mind while testing. If you use PSRemoting and the script doesn't close the connection (or LM kills the script), it will leave some temp files behind which can very slowly fill up the collector's drive. Not sure if this applies for M365 connections. If you turn on Multi Instance but don't provide an autodiscovery script, you can add needed instances via Add Monitoring Instance in the Resources page. Rather then say hardcode service accounts in an autodiscover script. Good feedback. What do you mean about multi instance and no discovery script? We did not hard code service account in the autodiscover script (although technically the api call to get the creds is hard coded here). You piqued my curiosity with this comment... Regarding coding thresholds into the datasource, couldn't agree more. We ended up being able to grab all the data points that allow us flexibility w/ LM logic to be able to alert. For now we kept it simple by biulding alert threshold > 90 90 90 so that if the totalsize goes above 90 we trip a crit alert. Down the road we may be able to do something nice such as "totalsize > issuewarning" and then trigger it. So as we manage the issuewarning setting on a given mailbox, the LM logic is current...and no alert threshhold needs to be modified. This is the first time we have done powershell. We have a few groovy scripts on custom datasources that do things like check cloud storage for last file write date to accomplish a sort of "dead man's switch" on ETL stuff....and it has worked nicely. I am stoked to see we are able to do so much with powershell as well...!6Views0likes0CommentsRe: IssueWarningQuote / Exchange Online Mailbox Quota Alerts?
We solved this with a little engineering.... We have a windows collector - and installed the EOL module on this collector. We then created a custom datapoint with embedded powershell script. YMMV on authentication, but we built a read-only account for this, then stored in our keyvault, called via API and conver to secure string. The actual command we are using to collect the data we need is (I assume folks reading this can understand we use some variables for things like mailbox identity, and those are snipped for security reasons...): $info= Get-Mailbox -Identity $mailboxname | Select DisplayName, ProhibitSendQuota,IssueWarningQuota, @{Name="TotalItemSize"; Expression={( Get-MailboxStatistics -Identity $mailboxname | Select TotalItemSize).TotalItemSize}} $ProhibitSendQuota= $info.ProhibitSendQuota -replace '\ GB.*$' Write-Host "ProhibitSendQuota=$ProhibitSendQuota" $IssueWarningQuota= $info.IssueWarningQuota -replace '\ GB.*$' Write-Host "IssueWarningQuota=$IssueWarningQuota" $TotalItemSize=$info.TotalItemSize.Value -replace '\ GB.*$' Write-Host "TotalItemSize=$TotalItemSize" This gets us a clean table of data We also have 3 datapoints we can build alerts off of10Views0likes0CommentsRe: IssueWarningQuote / Exchange Online Mailbox Quota Alerts?
Yeah - haha....the new SaaS version is not as good as the old one. There are also fewer metrics around SharePoint sites. Some of this is Microsoft some of it is LM. But I agree w/ LM that these particular powershell commands are latent so it wasn't sustainable. I am wondering if we cannot use custom datasource and run itfrom a machine that has powershell (with EOL module). The script itself will take some thought, because there is no flag on accounts that have reached the "IssueWarningQuota". So we'd have to do some unique script that stores that field for each mailbox and matches it against current mailbox size, and alerts when current mailbox size > IssueWarningQuota....if we end up developing this I may post it here. There are specific service mailboxes we want to do this for, so possibly we only do it against those handful....seems feasible. We could even do a custom datasource for each service account and then we can simply grab mailbox size, and create custom alert when mailbox size is > what we are comfortable with. This is less dynamic, cause if an exchange admin changes IssueWarningQuota our datasource become out of sync, since we'd need to in theory update the alert as well.... We'll likely engineer something for this, and post back here.236Views0likes0Comments