Another crazy idea: Add ability to pass alert message from custom script

1 hour ago, Stuart Weenig said:

What if, for scripted datasources, we had a token that contained the stdout and another token that contained the stderr of the most recent poll?

You could add to the groovy telling it to call the LM API and put that kind of data as a property on the device/instance. You'd be able to get the entire raw output into a token, and then into an alarm notification.

Agreed that real templating is needed. Personally I'd love to see Jinja2.

For this use it seems a bit messy to me and puts the work onto the integrated system to do the filtering, as stdout might be hundred of lines. Still sounds useful to have though as a workaround, although hopefully not as an excuse to build something more proper. :)/emoticons/smile@2x.png 2x" title=":)" width="20">

Not sure what you mean about the property thing. Right now the only way I know of to create auto properties is from Active Discovery scripts, which only run every 15 minutes at it's quickest, and only for multi-instance, or with PropertySources which only runs once a day. I think that would add too much delay and you may not know which message went with which alert (if it's flapping).

Yes, it would be messy, but looking for minimum viable product to get moving in the right direction. If we had stdout/stderr as tokens, it wouldn't take much to build processing to loop over the lines, apply regex, push through a template, etc.

What i mean with the properties thing is to have the groovy script actually call the LM API as part of its execution. It would do an HTTP patch to the /device/devices/{deviceId}/devicedatasources/{hdsId}/instances or /device/devices/{deviceId}/properties/{name} endpoint (to add as instance or device property, respectively). The information pushed to the LM API in that HTTP patch would be the value of some variable in the groovy script. 

For example:

Say you have a groovy based datasource: "IsItOkay", that gathers some data from the device. One of the pieces of data it gathers is the message you would want included in the alarm. Let's say the groovy script stores this message in a variable called `message`. After the part of the script that does the data collection finishes, you would have a section of the code that would do an HTTP Patch to the LM API updating/creating a property called `IsItOkay.detailedMessage` with the value of the `message` variable. The script would finish.

Once the script is finished, the data is compared to the threshold and since something is bad, it would open an alarm. The alarm body would reference all the tokens it references today with the addition of the `IsItOkay.detailedMessage` property. 

ok, I gotcha.

I always worry when the answer becomes having LM call it's own API. It's a powerful option but I feel is a hacky workaround and becomes an excuse to not add functionality to the product directly. Need to know how many alerts a customer has? write a datasource that queries the API. Dashboard widget not provide the data you need? Write a datasource that directly modifies widgets...etc. Also as an MSP, I don't own the collector systems and concerned that the customer would be able to extract the API token (which must be write-enabled) and able to modify LM themselves (or their part of it).

But I'm all for having options and moving in the right direction though.

Yeah, it's not great that we have to resort to it as often as we do. I'm working internally to get a new type of "collector" (i.e. the dropdown when building a datasource where you normally pick SNMP, WMI, BATCHSCRIPT, JDBC, etc.) called LMAPI. The idea would be to make it much easier to just call an LM API resource (without having to setup all the things you have to now) and parse through the data with a simple groovy script. 

One thing I do with my LM API calls (well for all API calls that aren't associated with a particular device) is that I create a resource in the portal called "[portalname].logicmonitor.com". Among several advantages is the ability to monitor that resource with an auto balanced collector group, meaning that collectors can go offline without suddenly losing the data. Ownership of the keys is also addressed there.

Would be good to have STDOUT and STDERR as standard tokens. Though as they could contain a lot of content, would need a special view, opens in a new dialog to scroll through the content ideally in a console font, rather than the current method of viewing properties using the Info tab.

Having them as tokens is different than storing them as properties. It's pretty easy to have active discovery scripts set properties for stdout and stderr, but as you mention, the info tab would get even more cluttered.

But if they were available as tokens, they could be used in alarm messages. 

If this F/R was done, we could use properties to communicate non-numeric results.  So many benefits....

)

Yes, that would make what I described above much easier. I mentioned it before, but I'm lobbying internally for there to be baked in methods for doing lm_api gets. That would pave the way for lm_api RW methods like lm_api.setProp("propname","propvalue"). Obviously, this only need to be added for use within the collection script as the discovery script already has this available for instance level properties and property sources has it available for device level properties.

That FR gets a +1 from me.

Just curious if LM has considered this request and if it has made it into the dev roadmap?

It's possible that snippets may make this possible at some point. At the very least, there's a mechanism through which it could be done. Can't really speak to the roadmap and whether or not it would happen this way or some other way.