When an anomaly isn't an anomaly what could i do?
What can i do when anomaly detection wont work ( something that is seen on a regular basis, and dynamic threshold also wont help where it is within range? For example a drive on a server gets filled with data ( drive is normally cleared down on a daily basis ) but when someone decides to upload a larger than expected amount the drive hasn't been cleared or with other uploads throughout the day there isn't enough space. You are happy if the drive is above 80% during the night because if it hasn't cleared it can be dealt with in the morning ( no need to get anyone out of bed ) but if there is a rapid spike ( more than 2.5% growth in used space in a 30min period ) then they need an alert to get out of bed and fix / make enough room for the data. A possible solution is a datasourcethat will alert if the drive is over the 80% but only with that rapid growth. DataSource calls the api for the last 30min worth of data and calculates the growth rate. The below is the code for a C drive but the drive letter can be changed easily in the code below, same with the 2.5% and the 80% values, they could also be parameterised for different ranges on different devices. <# Use TLS 1.2 #> [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 <# account info #> $accessId = '##apiaccessid.key##' $accessKey = '##apiaccesskey.key##' $company = '##company##' $deviceId = "##system.deviceId##" <# request details #> $httpVerb = 'GET' $resourcePath = "/device/devices/$deviceId/devicedatasources" $queryParams = '?filter=dataSourceName:"WinVolumeUsage-"' <# Construct URL #> $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams <# Get current time in milliseconds #> $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) <# Concatenate Request Details #> $requestVars = $httpVerb + $epoch + $data + $resourcePath <# Construct Signature #> $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) <# Construct Headers #> $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') $headers.Add("X-Version","3") <# Make Request #> $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers <# Get Device DataSource ID #> $deviceDataSourceId = $response.items.id <# request details #> $httpVerb = 'GET' $resourcePath = "/device/devices/$deviceId/devicedatasources/$deviceDataSourceId/data" $queryParams = '' <# Construct URL #> $url = 'https://' + $company + '.logicmonitor.com/santaba/rest' + $resourcePath + $queryParams <# Get current time in milliseconds #> $epoch = [Math]::Round((New-TimeSpan -start (Get-Date -Date "1/1/1970") -end (Get-Date).ToUniversalTime()).TotalMilliseconds) <# Concatenate Request Details #> $requestVars = $httpVerb + $epoch + $data + $resourcePath <# Construct Signature #> $hmac = New-Object System.Security.Cryptography.HMACSHA256 $hmac.Key = [Text.Encoding]::UTF8.GetBytes($accessKey) $signatureBytes = $hmac.ComputeHash([Text.Encoding]::UTF8.GetBytes($requestVars)) $signatureHex = [System.BitConverter]::ToString($signatureBytes) -replace '-' $signature = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($signatureHex.ToLower())) <# Construct Headers #> $auth = 'LMv1 ' + $accessId + ':' + $signature + ':' + $epoch $headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]" $headers.Add("Authorization",$auth) $headers.Add("Content-Type",'application/json') <# Make Request #> $response = Invoke-RestMethod -Uri $url -Method $httpVerb -Header $headers <# Print status and body of response #> $status = $response.status $body = $response.data | ConvertTo-Json -Depth 5 function Select-Nth { param([int]$N) $Input | Select-Object -First $N | Select-Object -Last 1 } $array1 = @($response.data.instances.'WinVolumeUsage-C:\'.values) $first = $array1[0] | Select-Nth 3 $last = $array1[19] |Select-Nth 3 $growth = $first - $last if (($growth -gt 2.5) -and ($first -ge 80)){ return 1 }else { return 2 } Hope this gives you some ideas to develop alerting further😁