Forum Discussion

Anonymous's avatar
Anonymous
24 days ago

User defined "host dead" status

There are two ideas that I need help maturing before talking to LM about them. Both have to do with how LM uses server side logic to declare a device dead.  We need the ability to designate what me...
Patches's avatar
Patches
Icon for Neophyte rankNeophyte

I agree this would be a boon. I regularly process alerts which have time-to-ticket SLA's of 20 minutes. Because the dead timer kicks in after 6 minutes, our HostStatus/idleinterval alerts generate at 5 minutes and are then sent to an escalation chain with 15 minutes of empty stages. 
This results in tickets generating on time, but also means our alert history is full of erroneous, 5-10 minute "outages".
Besides the clutter and general degredation of the alert views, this means we also have to perform all auditing of outage counts and frequency from our PSA. Not necessarily the end of the world, but certainly a degredation of LM's reporting functionality for us.


Separately, we have issues with devices monitored only via API, in particular the latest VMware SDWAN datasources. 
To facilitate effective NetScan discovery, these resource have unique identifiers for hostnames which aren't pingable. So we don't have Pings or DNS available to update the idleinterval, and we rely entirely on the API feedback to determine whether an edge is up or down. HostStatus is non-functional and irrelevant for these resources.

Due to our scale and rate-limiting issues, the most frequently updated datasource only runs every 5 mins. Since this is the fastest collection, processing delays often result in resources being labeled as Dead when they aren't. This makes for a messy interface and causes concerns from clients who keep their eyes on the portal. 
I'm actively working on a solution, but suffice to say that we can't generate additional API queries without risking rate limiting, and changing the resources hostnames to an IP either manually or automatically will create endless conflicts with the NetScan discovery script.

Being able to adjust the dead timer would resolve this outright. 

Alternatively, as you mentioned, having an understanding of the scripting methods used by LM to update the idleinterval would be extremely helpful in cases like this. Then we could potentially implement datasources to keep certain resource types, like those edges, alive.

Mike_Moniz's avatar
Mike_Moniz
Icon for Professor rankProfessor
2 days ago

As a workaround, and it's not documented (agree it should be) but here is code that will tell LM the device is live. Can be added to an existing groovy DataSource.

import com.santaba.agent.live.LiveHostSet

// Update the liveHostSet to tell the collector we are happy (updates Host Status)
hostId = hostProps.get("system.deviceId").toInteger()
liveHostSet =  LiveHostSet.getInstance()
liveHostSet.flag(hostId)

For others: just be careful with this. Since this is designed to bypass LM's built-in dead-device detection you want to determine that the device is really still up before calling this code. But may help you code around limits with the existing dead-device methods.

This is likely also unsupported by LM, although I got the code from some of their own DataSources.