What can i do when anomaly detection wont work ( something that is seen on a regular basis, and dynamic threshold also wont help where it is within range?
For example a drive on a server gets filled with data ( drive is normally cleared down on a daily basis ) but when someone decides to upload a larger than expected amount the drive hasn't been cleared or with other uploads throughout the day there isn't enough space.
You are happy if the drive is above 80% during the night because if it hasn't cleared it can be dealt with in the morning ( no need to get anyone out of bed ) but if there is a rapid spike ( more than 2.5% growth in used space in a 30min period ) then they need an alert to get out of bed and fix / make enough room for the data.
A possible solution is a datasource that will alert if the drive is over the 80% but only with that rapid growth.
DataSource calls the api for the last 30min worth of data and calculates the growth rate.
The below is the code for a C drive but the drive letter can be changed easily in the code below, same with the 2.5% and the 80% values, they could also be parameterised for different ranges on different devices.
<# Use TLS 1.2 #><br/>[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12<br/><br/><# account info #><br/>$accessId = '##apiaccessid.key##'<br/>$accessKey = '##apiaccesskey.key##'<br/>$company = '##company##'<br/>$deviceId = "##system.deviceId##"<br/><br/><# request details #><br/>$httpVerb = 'GET'<br/>$resourcePath = "/device/devices/$deviceId/devicedatasources"<br/>$queryParams = '?filter=dataSourceName:"WinVolumeUsage-"'<br/><br/><# Construct URL #><br/>$url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams<br/><br/><# Get current time in milliseconds #><br/>$epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)<br/><br/><# Concatenate Request Details #><br/>$requestVars = $httpVerb + $epoch + $data + $resourcePath<br/><br/><# Construct Signature #><br/>$hmac = New-Object System.Security.Cryptography.HMACSHA256<br/>$hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)<br/>$signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))<br/>$signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'<br/>$signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))<br/><br/><# Construct Headers #><br/>$auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch<br/>$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"<br/>$headers.Add("Authorization",$auth)<br/>$headers.Add("Content-Type",'application/json')<br/>$headers.Add("X-Version","3")<br/><br/><# Make Request #><br/>$response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers<br/><br/><# Get Device DataSource ID #><br/>$deviceDataSourceId = $response.items.id<br/><br/><# request details #><br/>$httpVerb = 'GET'<br/>$resourcePath = "/device/devices/$deviceId/devicedatasources/$deviceDataSourceId/data"<br/>$queryParams = ''<br/><br/><# Construct URL #><br/>$url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams<br/><br/><# Get current time in milliseconds #><br/>$epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds)<br/><br/><# Concatenate Request Details #><br/>$requestVars = $httpVerb + $epoch + $data + $resourcePath<br/><br/><# Construct Signature #><br/>$hmac = New-Object System.Security.Cryptography.HMACSHA256<br/>$hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey)<br/>$signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars))<br/>$signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-'<br/>$signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower()))<br/><br/><# Construct Headers #><br/>$auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch<br/>$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"<br/>$headers.Add("Authorization",$auth)<br/>$headers.Add("Content-Type",'application/json')<br/><br/><# Make Request #><br/>$response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers<br/><br/><# Print status and body of response #><br/>$status = $response.status<br/>$body = $response.data | ConvertTo-Json -Depth 5<br/><br/>function Select-Nth {<br/> param([int]$N) <br/><br/> $Input | Select-Object -First $N | Select-Object -Last 1<br/>}<br/><br/>$array1 = @($response.data.instances.'WinVolumeUsage-C:\'.values)<br/><br/>$first = $array1[0] | Select-Nth 3<br/>$last = $array1[19] |Select-Nth 3<br/><br/>$growth = $first - $last<br/><br/><br/>if (($growth -gt 2.5) -and ($first -ge 80)){<br/>return 1<br/>}else {<br/>return 2<br/>}
Hope this gives you some ideas to develop alerting further 😁
Cole McDonald
·3 years agoBackground: I did my Senior thesis for my Anthropology (Archaeology) degree on generating pre-historic soil topology based on minimal measured data.
One of the pieces that I was working on to supplement that effort was to try to recreate the long, mid, and smaller frequency height fluctuations for those ancient surfaces. This effort consisted of measuring average and standard deviations of the three. Using these 6 data points, I posit there should be the ability to get a weekly, daily and hourly variation values and the sums of those three at any point should fall within a known range. You’re already accessing that dataset according to your code for volume freespace.
I will be utilizing parts of your code to calculate growth and days until full (grabbing those historical values had eluded me for some reason).
Counter and Derive datapoint types can be very valuable for these types of metrics as well for very short term shifts (just previous value).
You can also set different values in the static thresholds for different times of day to help account for things like backups, etc.
Enough of my rambling… now I have to go write some new dataSources based on this discussion.