User defined "host dead" status
Summary: The original post by LM User discusses the challenges faced with automatic alerts when a device is deemed 'dead' due to a down VPN tunnel, even though the devices remain connected to the Meraki switch. The author suggests improvements for handling device status alerts more efficiently, particularly through the adjustment of the idleInterval and server-side logic. These suggestions garnered support from multiple users, who shared similar issues and emphasized the need for configurable dead timers and better documentation. Patches expressed issues with host status monitoring due to the dead timer activating prematurely under current configurations. Furthermore, Mike Moniz offers a workaround with a code snippet using LiveHostSet to address the device status manually. Overall, participants, including some interested in official support, stressed the importance of user-defined alerts and the need for a more flexible system.
There are two ideas that I need help maturing before talking to LM about them. Both have to do with how LM uses server side logic to declare a device dead.
We need the ability to designate what metric declares a device as dead/undead and when.
What
We have several customers who have devices, usually UPS, at remote sites all connected to a Meraki switch. The collector is not at the remote site, but connects over a VPN tunnel, which may be torn down due to inactivity or could be flaky for any other reason. When the VPN tunnel goes down, the devices alert that they have gone down. We have added monitoring to the tunnel and also get alerted when it goes down. However, we'd like to prevent the host down alerts when the only problem is that the VPN tunnel is down. RCA (or recently renamed DAM?) would likely solve this, but defining that mapping manually or through a topologysource is not scalable (plus visibility into the RCA logic is never been good).
Luckily, Meraki has an API where we can query the status of devices connected to the switch. During a tunnel outage, this API data shows that the device is still connected to the switch and online. Since it's a UPS, that's sufficient. We've built the datasource required to monitor the devices via the Meraki API. However, since it's a scripted datasource, it doesn't reset the idleInterval. (Insert link here to a really good training or support doc explaining how idleInterval works.) Since none of the qualifying datasources are working on the UPS during the VPN outage, the idleInterval eventually climbs high enough to trigger a host down alert. When the host is declared down, other alerts, like the alerts from this new Meraki Client Device Status DS, are suppressed.
How can this be remedied?
So, we need the supported and documented ability to use the successful execution of a collection script to reset the idleInterval. I know this is possible today as I've seen it in several of LM's modules. However, I've never seen official documentation on how to do it. LM's probably worried someone will add it to all their scripts, which wouldn't be the right thing to do.
When
I know I'm not the only one. I need control over the server side logic that determines when the idleInterval declares a device dead. In the example above, we get a slew of host down alerts when the VPN tunnel goes down. However, usually within a few minutes, the VPN gets reestablished and the collector reestablishes connectivity to the device and the idleInterval resets, thus clearing the alerts. With a normal datapoint, I'd just lengthen the alert trigger interval for the idleInterval datapoint. This would mean that the device would have to be down for 15 minutes, 20 minutes, however long I want before generating the alert. What's great is that now we can do that on the group level, so I can target these devices specifically and not alert on them unless they've been down for a truly unacceptable amount of time (i.e. not just a VPN going down and coming right back up).
However, the idleInterval datapoint is an odd one. Two things happen. One happens when you surpass the threshold defined on the datapoint. I can't remember what the default is, but in my portal, that's > 300 or 5 minutes. At 6 minutes server side logic, which has been inspecting the idleInterval, decides that the device is down which has implications on suppressing other alerts on the device.
As far as I can tell, lengthening the alert trigger interval on the idleInterval datapoint has no effect if the window would exceed the 6 minutes that the server side logic uses to declare the device down.
What do we need?
We need the ability to set the amount of time that the server side logic uses to declare the device down. We need to be able to set that for some devices and not others. So we need to be able to set it globally, on the group level, and on the device level. Preferably this could be set in the alert trigger interval on the idleInterval datapoint since this mechanism already exists globally and on the group and device levels. Knowing that this could be a confusing way of defining it (since it's measured in poll cycles not minutes/seconds) so, it could alternatively be done as a special property on the group(s)/device(s).
I'm interested in hearing your thoughts, even if you are an LMer.
Stephen Cavernelis
·4 months agoTrying to wake up this thread again as I am sure there are many users with the same requirement!
In my case it is expected that some devices may be down for about 10 minutes a day - unfortunately when they down can't be determined. For some context we have network connected devices which are unplugged and physically cleaned before going into operation again. It's unrealistic for the cleaners to contact the team to put the devices in SDT everytime, and while we want to know the availability of the devices it's considered acceptable if it is offline for say 10 minutes.
Perhaps the easiest solution from LM perspective is to all LM admins to define the "resource down logic" on a per portal and per group basis - instead of having it hardcoded in the backend.
Right now we have our in-house middleware that processes alerts so we use that to suppress various alerts for x minutes before generating incidents (if the alerts still exist).
Stefan Wuensch
·4 months agoI just discovered this topic; I hope I'm not adding a comment that is inanely obvious.
I think the "perfect" solution would be a library / method like LiveHostSet (or an extension thereof) which would be able to perform the exact opposite. If I can determine from my own code that the Device should be called "Dead" then I would like to be able to do so!
For what it's worth I use the LiveHostSet in quite a few custom DataSources and it's a lifesaver!!
Mike Moniz
·1 year agoSo what is the thought on this part of the request others are making?
Michael Rodrigues
·1 year agoSeems like a reasonable first step into opening up the configurability of Host Down.
I'd love to remove the server-side logic altogether and just let users decide what constitutes "down" for a given host. We'd of course have some reasonable defaults.
Patrick Rouse
·1 year agoGreat ideas.Mike_Rodrigues let’s discuss what it would take to make this officially supported.
Michael Rodrigues
·1 year agoThis is something I've been thinking about for a while, but no related work has been prioritized.
Interestingly (or maybe just pedantically) we can't actually determine when an entire physical host is down; the best we can (and do) determine is that it is not reachable. We assume it's down after a server-side hard coded period of the idleInterval not resetting (as you've seen).
By the way, the idleInterval generally only resets for methods where we can assume that the host returning the data, is the current LM Resource of the current execution context. If you have a host "host.example.com" and you apply a script module to it that gets data from "otherdeviceapi.com", that doesn't confirm to LM that "host.example.com" is still returning data. This might be an overly rigid assumption. At least offering the ability to toggle that on a given module, would probably go a long way.
Back to host status, changing the alert does nothing but delay your alert (or send it before our servers declare it down, depending on which direction you go).
And yes, RCA is now DAM, Dependent Alert Mapping.
I would love to be able to do something like, take a SQL server, and configure "down" to be any of the following:
- SQL server is not responding
- SQL queries taking longer than N seconds to return
- Standard Host Down
We can also be much more certain about SQL being down vs inaccessible, than we can about the host itself being down vs inaccessible, because we can see that SQL stopped but Ping is still working (hypothetically).
I've kicked around a few ways of doing this in my head, but my favorite is just being able to "declare" an alert as "this indicates a host/service down condition".
pgordon
·1 year agoI also agree this sounds useful for certain use cases
Rob Sapsford
·1 year agoJust replying to express my support. You've covered my thoughts already.
Patches
·1 year agoI agree this would be a boon. I regularly process alerts which have time-to-ticket SLA's of 20 minutes. Because the dead timer kicks in after 6 minutes, our HostStatus/idleinterval alerts generate at 5 minutes and are then sent to an escalation chain with 15 minutes of empty stages.
This results in tickets generating on time, but also means our alert history is full of erroneous, 5-10 minute "outages".
Besides the clutter and general degredation of the alert views, this means we also have to perform all auditing of outage counts and frequency from our PSA. Not necessarily the end of the world, but certainly a degredation of LM's reporting functionality for us.
Separately, we have issues with devices monitored only via API, in particular the latest VMware SDWAN datasources.
To facilitate effective NetScan discovery, these resource have unique identifiers for hostnames which aren't pingable. So we don't have Pings or DNS available to update the idleinterval, and we rely entirely on the API feedback to determine whether an edge is up or down. HostStatus is non-functional and irrelevant for these resources.
Due to our scale and rate-limiting issues, the most frequently updated datasource only runs every 5 mins. Since this is the fastest collection, processing delays often result in resources being labeled as Dead when they aren't. This makes for a messy interface and causes concerns from clients who keep their eyes on the portal.
I'm actively working on a solution, but suffice to say that we can't generate additional API queries without risking rate limiting, and changing the resources hostnames to an IP either manually or automatically will create endless conflicts with the NetScan discovery script.
Being able to adjust the dead timer would resolve this outright.
Alternatively, as you mentioned, having an understanding of the scripting methods used by LM to update the idleinterval would be extremely helpful in cases like this. Then we could potentially implement datasources to keep certain resource types, like those edges, alive.
Mike Moniz
·1 year agoAs a workaround, and it's not documented (agree it should be) but here is code that will tell LM the device is live. Can be added to an existing groovy DataSource.
For others: just be careful with this. Since this is designed to bypass LM's built-in dead-device detection you want to determine that the device is really still up before calling this code. But may help you code around limits with the existing dead-device methods.
This is likely also unsupported by LM, although I got the code from some of their own DataSources.
Patches
·1 year agoMuch appreciated, Mike. This cleaned up my "dead" API-only devices. It'll be a solid stopgap until we get some form of official resolution.