How do I configure an alert, for a specific Instance name?

For the life of me I can't figure out to make the type of alert our org needs.

I have a datasource named "Event Log Errors v3"
This has a ton of different instances being generated and it's applied to any IsWindows() systems which we have over 10,000 of.
I need to set an alarm, for a specific instance name. If that instance name is seen with a value over X over X, then I want an alarm triggered, on any system.
I don't want to have to configure this a bazillion times (individual resource level), the alert should be uniform for all systems.
I can't set an alarm on the DataSource itself, as there is one datapoint right now that uses ##WildValue##.COUNT as its Key, so setting a threshold there would set the same threshold for all the hundreds of instances being created by this DataSource and blow up our help desk.
I can't set it at the Dynamic Group level right now because there I can pick the DataSource, but can't seem to specify a specific instance name from within the DataSource. It gives me the generic "count" entry.

What am I missing here? Any suggestions? I chatted with support but end up with a slew of doc page links that send me down rabbit holes or confuse me even more. This should be pretty simple one would think...

jonathanbarrow

Posted 2 years ago·Last reply 2 years ago

17 comments

jonathanbarrow

OP2 years ago

So in that code, the 1234 is the actual wildalias name you'd be looking for in your example. I wonder how that would work as I think the WildAlias in this scenario would be that long instance name we're using now, right?

This is the instance name we're trying to alarm off of.

"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin|Microsoft-Windows-TerminalServices-RemoteConnectionManager|2|1069"

LM User

·2 years ago

Yes, 1234 in the expression above would be the whole name. And the 5 would be the value of the .count datapoint you want to threshold on. ge() means greater than or equal to. This assumes the datapoint is called "count".

I think the problem with this is that the eq() function is looking for numbers, not a string. So it may always return 0. If that's the case, we may need to look into simplifying either the wildvalue (first part of discovery line before the ##) or the wildalias (second part of discovery line, the instance display name) down to a number.

In your name above is 1069 enough to uniquely identify that one instance? Or is it possible to have a 1069 with two different strings in the name before the 1069?

LM User

·2 years ago

For example, is it possible for these both to exist? Or will "1069" always be paired with "Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin|Microsoft-Windows-TerminalServices-RemoteConnectionManager|2"

"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin|Microsoft-Windows-TerminalServices-RemoteConnectionManager|2|1069"

"Something else entirely|2|1069"

jonathanbarrow

OP2 years ago

1069 will always be there, this logic builds the instance name for this code using the following naming scheme.

Log Name | Provider logging the error | Error level (2 = error, 1 = critical, 3 = warning) | Event ID

So in theory, yes another log or provider could output a 1069 as well.

jonathanbarrow

OP2 years ago

Testing this out...

Mike Moniz

·2 years ago

Is the count alert threshold you want for all of these instances, all going to be the same? Like you want to alert if 20 different event id counts go >= 2? Or could some of them be 5 counts, or 10 counts, etc?

Also if the count is just going to be 1, I would just look to use EventSources for alerting.

jonathanbarrow

OP2 years ago

No, they may vary, and 99% of the instances we have just for informational purposes but don't want alarming on them.

Right now this was built to feed data into other systems, but we just recently had an issue where management needs us to alert of this one particular item which is causing outages for a client which led them to asking how I can set an alarm on this particular item (instance).

I haven't used EventSource before, how would that differ?

Also, we don't currently own the Event Log features of LM as we haven't really had a need for it before, so any of those features, or native event log details I don't think I can get as we don't own that module. So to get them out of this major incident, they asked how I could accomplish this with the data we do have in the system now.

jonathanbarrow

OP2 years ago

Referencing my shared screenshot, I have the Alert Threshold box/data empty right now. I'd think I would need to set that, but in the code you shared which I changed the instance name on, it seems to be defined there. So how would that work with the "ge(count,1)),1,0" already in the groovy code?

jonathanbarrow

OP2 years ago

Nevermind, I get it. If the count of that instance is over 1, then i'd be outputting a 1, if not a 0. So I'd need to configure the alert threshold still and just have it trigger when it spots a 1 or higher. Sorry, little slow this am. :)

jonathanbarrow

OP2 years ago

Thank you so much, the #2 option sounds ideal but I'm not sure how to reference the ##WILDALIAS## token within the new datapoint I'm trying to create. Here is how the regular datapoint looks now, that's used for all the instances on the DS. I guess I'd need to create a new, one, but have it only report data for one specific instance name. Is the Key section where I would be using the alias you mentioned?

LM User

·2 years ago

You'd need to build a new complex datapoint, not a normal datapoint. Normal datapoints extract data from the output of the script/task. Complex datapoints evaluate normal datapoints and/or tokens. So you'd create a complex datapoint that looks something like this (YMMV):

if(and(eq(##WILDALIAS##,1234),ge(count,5)),1,0)

You might run into issues using the wildalias if it's not numeric, LM is tricky about which things the eq() function will evaluate.

Mike Moniz

·2 years ago

Do you have perhaps a specific example of instances (with made up data) for what you are looking to do?

For example if you want to set a count threshold of 5 for event id 1234, 2 count for event id 4321, and 10 for everything else. One option is to create special Datapoints that only deal with those event ids. like having Datapoints called "CountIfEventId1234" and "CountIfEventId4321" that always return 0 unless you have event ids 1234/4321. You an then set global thresholds for these special datapoints separate from the more generic Count. This wouldn't be all that great if you have a ton of exceptions though.

jonathanbarrow

OP2 years ago

You bet, here is the exact instance name we're trying to set an alarm for...

"Microsoft-Windows-TerminalServices-RemoteConnectionManager/Admin|Microsoft-Windows-TerminalServices-RemoteConnectionManager|2|1069"

jonathanbarrow

OP2 years ago

And here is a sample of what the discovery script outputs when creating the instances initially for each system.

When the collector script comes along, would be similar output but would be instance_name=somevalue in it's output.

LM User

·2 years ago

So the DS only has one datapoint and you'd like to set a threshold for when that datapoint goes over a certain value. However, you don't want to set it on every instance, just certain instances. Setting it per instance or per instance group isn't tenable because of the quantity of instances and instance groups. It's not reasonable to set a threshold on 10,000 instance groups (compounded by the number of different thresholds you want to set).

There are a couple ways to do this.

You could separate the datasource into multiple datasources, each one containing all the instances that would share a threshold. So if you have instances A, B, and C all in one DS now, but A's threshold is > 2, B's is >5, and C's is >100, you'd split into three datasources, each one filtering all but one specific instance. So you'd have DS_A, DS_B, & DS_C. You would use the same discovery for all three but create discovery filters to only end up with A's in DS_A, B's in DS_B, and C's in DS_C. This is the most easy to consume because the events are clearly split out into different places each with its own threshold. The disadvantage is that you might not have just A, B, and C, you might have A-ZZZ, meaning you'd need hundreds of datasources. This is the most straightforward option.
Another way would be to create a datapoint within your one datasource that evaluates the name of the instance along with the value of the count datapoint and returns a 1 or 0 depending on whether it's over threshold. You'd do this using the ##WILDALIAS## token. You'd still have one datasource, but there would be one datapoint per event name you want to alert on.
Another option altogether is to pipe the data into LM Logs and setup alert pipelines for the different events containing values higher than the acceptable value. If you need to be able to graph the .count datapoint, this isn't a good option. However, if the .count datapoint is counting the number of error events per name this might be a better option. LM Logs can now threshold looking for a log to happen > X times in a certain timeframe. So you would only get an alert if the event showed up in the logs more than 6 times within an hour for example. In this case, the logs coming in are the actual error logs rather than a summary saying that such-and-such log came in N times.

There might even be other ways depending on exactly what you are trying to collect.

jonathanbarrow

OP2 years ago

Thank you, I've toyed with that a bit but haven't been able to figure out how I can set a threshold specifics with that alert rule. I have the logic module picked, instance picked, and datapoint picked, but there is no setting for a value. Will it always trigger if there is any value, or can i set it if say it comes back over X, then it triggers?

Evelyn Caviedes

·2 years ago

Jonathan, You can create the dynamic group, and then go to settings-->Alert Rules and create rules on a specific resource or instance.

1	SUYASH GAIKWAD
2	RR Robin Ratcliffe
3	AH Alejandra Herrera
4	HS Hunter Sparks
5	LT Lily Trbovic
6	AG Asher Guinzy