22 July 2020 - Dynamic Thresholds

18 hours ago, David Good said:

Also, what happens if you have, say, a dynamic threshold set for disk usage and the usage grows very slowly and steadily.  If I also have a static threshold set for when the disk is dangerously full will it still send notifications or will they be suppressed since the value isn't anomalous?

It's possible if the usage is slowly & steadily increasing that the band will adjust to that usage. You can avoid suppression by having static & dynamic thresholds at separate severities (eg static threshold for critical severity & dynamic for warn | error). Sometimes these metrics where you generally know that certain good and bad ranges always hold true (e.g. above 90% is always bad, below 90% is always good) are a better fit for static thresholds. Dynamic Thresholds are most useful when it is not possible or difficult to identify a range generally/globally across all instances (e.g. above 10MB/s may be bad for instance A, but for instance B it should be above 100MB/s).