Another crazy idea: Add ability to pass alert message from custom script
I have many custom data sources. I know I can only pass numeric results back to Lomo but I would love to be able to pass back to Lomo a text string that could be automatically added to a custom alert message. This would give me the ability to include more details in an alert message.
Gary_Dewrell
Posted 6 years ago·Last reply 4 years ago
21 comments
Mosh
·4 years agoIdeally, we would have the ability to configure a pre-alert script at data source level. The script would execute on the collector.
The script could be passed an "alert" object, which would have a "alert.message" property that the script can modify. The alert object could also have a "alert.cancel" property that could be set to true to cancel the alert, for example, if the pre-alert script (running on the collector) performed some logic that resolved the alert condition.
LM User
·4 years agoIt's possible that snippets may make this possible at some point. At the very least, there's a mechanism through which it could be done. Can't really speak to the roadmap and whether or not it would happen this way or some other way.
Gary_Dewrell
OP4 years agoJust curious if LM has considered this request and if it has made it into the dev roadmap?
mnagel
·6 years agoIf this F/R was done, we could use properties to communicate non-numeric results. So many benefits....
/topic/2988-need-method-to-set-properties-in-groovy-code/)
LM User
·6 years agoHaving them as tokens is different than storing them as properties. It's pretty easy to have active discovery scripts set properties for stdout and stderr, but as you mention, the info tab would get even more cluttered.
But if they were available as tokens, they could be used in alarm messages.
Mosh
·6 years agoWould be good to have STDOUT and STDERR as standard tokens. Though as they could contain a lot of content, would need a special view, opens in a new dialog to scroll through the content ideally in a console font, rather than the current method of viewing properties using the Info tab.
LM User
·6 years agoYeah, it's not great that we have to resort to it as often as we do. I'm working internally to get a new type of "collector" (i.e. the dropdown when building a datasource where you normally pick SNMP, WMI, BATCHSCRIPT, JDBC, etc.) called LMAPI. The idea would be to make it much easier to just call an LM API resource (without having to setup all the things you have to now) and parse through the data with a simple groovy script.
One thing I do with my LM API calls (well for all API calls that aren't associated with a particular device) is that I create a resource in the portal called "[portalname].logicmonitor.com". Among several advantages is the ability to monitor that resource with an auto balanced collector group, meaning that collectors can go offline without suddenly losing the data. Ownership of the keys is also addressed there.
Mike Moniz
·6 years agook, I gotcha.
I always worry when the answer becomes having LM call it's own API. It's a powerful option but I feel is a hacky workaround and becomes an excuse to not add functionality to the product directly. Need to know how many alerts a customer has? write a datasource that queries the API. Dashboard widget not provide the data you need? Write a datasource that directly modifies widgets...etc. Also as an MSP, I don't own the collector systems and concerned that the customer would be able to extract the API token (which must be write-enabled) and able to modify LM themselves (or their part of it).
But I'm all for having options and moving in the right direction though.
LM User
·6 years agoYes, it would be messy, but looking for minimum viable product to get moving in the right direction. If we had stdout/stderr as tokens, it wouldn't take much to build processing to loop over the lines, apply regex, push through a template, etc.
What i mean with the properties thing is to have the groovy script actually call the LM API as part of its execution. It would do an HTTP patch to the /device/devices/{deviceId}/devicedatasources/{hdsId}/instances or /device/devices/{deviceId}/properties/{name} endpoint (to add as instance or device property, respectively). The information pushed to the LM API in that HTTP patch would be the value of some variable in the groovy script.
For example:
Say you have a groovy based datasource: "IsItOkay", that gathers some data from the device. One of the pieces of data it gathers is the message you would want included in the alarm. Let's say the groovy script stores this message in a variable called `message`. After the part of the script that does the data collection finishes, you would have a section of the code that would do an HTTP Patch to the LM API updating/creating a property called `IsItOkay.detailedMessage` with the value of the `message` variable. The script would finish.
Once the script is finished, the data is compared to the threshold and since something is bad, it would open an alarm. The alarm body would reference all the tokens it references today with the addition of the `IsItOkay.detailedMessage` property.
Mike Moniz
·6 years agoFor this use it seems a bit messy to me and puts the work onto the integrated system to do the filtering, as stdout might be hundred of lines. Still sounds useful to have though as a workaround, although hopefully not as an excuse to build something more proper. :)/emoticons/smile@2x.png 2x" title=":)" width="20">
Not sure what you mean about the property thing. Right now the only way I know of to create auto properties is from Active Discovery scripts, which only run every 15 minutes at it's quickest, and only for multi-instance, or with PropertySources which only runs once a day. I think that would add too much delay and you may not know which message went with which alert (if it's flapping).
LM User
·6 years agoLM Employee here, from the Sales Engineering team. Hello!
What if, for scripted datasources, we had a token that contained the stdout and another token that contained the stderr of the most recent poll?
You could add to the groovy telling it to call the LM API and put that kind of data as a property on the device/instance. You'd be able to get the entire raw output into a token, and then into an alarm notification.
Agreed that real templating is needed. Personally I'd love to see Jinja2.
Gary_Dewrell
OP6 years agoWe still use Nagios also. This one feature is the only reason we do though. If LM every fixes this then we will be 100% LM.
mnagel
·6 years agoThere are gobs of reasons to be able to transform the raw tokens into meaningful rather than vanilla output. Most times, the output is just shy of "Something is wrong, good luck." We created a real template system for Nagios that provided very rich conditional information with callbacks to be able to collect additional data from impacted systems and this is one thing I really miss. LM has no apparent interest in doing anything like that, or at least has been silent in every F/R thread it has been mentioned within. My best option so far has been to send all tokens to our ticket system (which is a maintenance headache as the integration must list every token explicitly), but then I hit a wall with email integration bugs. Need to figure out how to put up a middle web API instead...
Mosh
·6 years agoAnother reason why I'd like to pass the alert message through a script first would be for SSL cert alerts. Rather than "SSL cert will expire in n days" I would prefer to have it say "SSL cert will expire on <date>". With a alert message script we could perform that calculation.
Mosh
·6 years agoJust the ability to pass the alert message through a scripted function that returns a string back the alert handling process would meet many people's requirements in this space. The alert message script could be part of the data source same as discovery and collection scripts. At datapoint level would be nice, but even at module level, it would be fine as we could if()...else or switch() case for data points.
mnagel
·6 years agoIn addition to enum management within modules, the right fix is to replace the simple token single-pass token substitution functionality with an actual templating engine. See https://docs.groovy-lang.org/docs/next/html/documentation/template-engines.html for a list -- LM needs to pick one and provide some sort of library function to enable management of template fragments.
Mike Moniz
·6 years agoI haven't thought of a way to do this either. Best I've come up with is for script-based DataSources is to output any text info in script output that DataPoints will ignore but will show up if you use Raw Data > Poll Now and scroll down to scriptOutput section. No where as as good as having the details in the alert message/ticket itself, but better than having to log into the problem system and work it out.
P.S. Calling LM"Lomo", ha! :)
mnagel
·6 years agoOn a test run with some enum properties defined, no joy. Maybe it can be done via an undocumented method, like braces or something, or more likely there is just no support for constructing tokens this way.
I did create the enum mappings as a property source (could also just be manually defined at the root, but using a PS is a lot faster for many items). This will pollute the property list pretty badly when used extensively unless there is an option to keep them hidden in the UI. Enum support really needs to be added to LogicModules directly.
mnagel
·6 years agoI just thought of a hack that might work. In the AD section, generate a bunch of auto properties that look like auto.enum.DSName.DPName.enumValue = enumName (substituting your DS name, DP name and enum value/name pairs, of course). Then in your alert message you may be able to reference ##AUTO.ENUM.DSNAME.DPNAME.##VALUE####. I have not tested whether nested token references work, but if they do, this could take away some of the pain. It might work as well with or without AD to put the definitions into a propertysource. Complete conjecture at this point, but I will probably whip something up to test as soon as I have some time....
mnagel
·6 years agoYou can reference properties (including automatic properties) as tokens, but there is no facility to send arbitrary text (e.g., enum to text lookup). Unfortunately, since the alert system is simple token substitution and has no programmatic capability, you cannot do any sort of conditional display or substitution. I have railed on this one virtually since we started with the product, but no joy, not even a hint of a plan to fix alerting. Our only option has been to use custom integrations with actual templating (like ERB, Template::Toolkit, etc.), but we have found various bugs (like custom email integrations do not pass ACK or SDT notices), which make it more annoying than it needs to be.