New user - looking for information on parsing fields from Syslog message field
We’re just starting to get LogicMonitor setup on our network and, mainly to test the collector, I setup one of our HAProxy instances to forward it’s syslog to the collector and can confirm that the logs are being ingested.
However, because it’s bare syslog, all I can see is the bare message field. What I’m looking for is the functionality to pull particular fields out of the message and into fields.
For example, in the log line above, the fields are separated by a space. Among the fields included in the log line are items such as the Client IP, the FronteEnd and BackEnd which I was hoping to be able to extract into their own fields to help with reporting.
I’ve been looking through the documentation and have found the LogSource section but that doesn’t seem to be adding the field. Frankly, after a few hours of searching and experimenting,
I’m hoping someone could give me some guidance on how this can be done.
Thanks
Cole McDonald
·2 years ago@nageeb to better understand how LM is structured:
Our full alert pipeline:
Device > ‘Source appliesTo() matches the device > Active discovery adds an instance > datapoints are collected for that instance > crossing threshold triggers an alert which matches an alert rule and gets sent to an escalation chain which generates custom JSON > webhook integration> Teams channel for technician consumption.
The custom JSON adds dynamic buttons with links to the rest of our toolset (KB, RMM, Azure Portal, PW Manager, Asset Management, Etc.) These are populated from custom properties on the groups/devices.
It’s non-agented, so all code is running on the collector itself. If you need data from the device you’re targeting, the code running on the collector has to reach out remotely. I use Powershell in Windows, so things like:
get the data from the far end. If it’s more complex data using non-remote types of commands:
can get that script block to run on the remove machine and pass the results back to your script. There are caveats there such as making sure to use sessions that don’t load profiles if you’re using fslogix or upds that will need to mount a remote volume for each login (there can be thousands for that single account that may cause resource contention).
Topology and Services are my next area of exploration to see if I can recreate workflow based alerting for tiered enterprise application stacks. We host software, but don’t write it, so we have issues with the existing application monitoring options provided which require monitoring hooks be coded into the applications.
Groovy is a weird second cousin to Java and seems to run very efficiently.
The escalations don’t allow direct code execution (which would be amazing)… but the dashboarding is quite a bit more powerful than other platforms I’ve used in the past… and presenting data to “do-ers” in meaningful ways is where LM can really shine.
JJumpp
·2 years ago@nageeb try one of these. Im honestly not sure why some of them work at the moment, but the deviceId works for me pretty much every time.
and if your logs stop coming in, try that debug mode on the logging level for syslog. Anytime I’ve had logs stop (that wasnt from obvious user intervention) I was able to see the issue when i ran the debug command.
nageeb
OP2 years agoWow, I’m so grateful for the fantastic input!
As @Stuart Weenig mentioned, the idea of LogSources is still a bit confusing to me. The more I dive into the different concepts in LogicMonitor, the more confusing I find things to be and frankly, the more frustrated I am with the lackluster documentation. Having said that, the community posts are a goldmine!
@JJumpp’s explanation has cleared a few things up and I feel like I might be on the right track. However I’m still stuck at the Resource Mapping step. I have the servers setup and I can see them in the resources but I’m unclear about what question the Resource Map section is asking me. As far as I understand it, I’ve assigned the LogSource to a device in the “Applies To” section. I tried setting properties for the devices and using the LM Token type, I’ve tried regex but nothing seems to be working. To add to that, if the LogSource isn’t correctly configured, NO logs come into the log view, which is kind of annoying.
I’ve tried reading the docs on Resources and Topologies but there’s nothing in it that I can connect to what the LogSource Resource Map section is for.
I realize that LogicMonitor has a way of doing things which I’m still slowly wrapping my head around. I came from Datadog where the configuration sections listed the various steps of the pipeline in a way that seemed intuitive to me. I’m still learning the concepts and philosophy of LogicMonitor so I’m having to adjust my brain to the way things are done here :)
JJumpp
·2 years agoIdeally, if you are using the LogSource, you would not need to make any edits to your collector config, since all the settings, like resource mapping, are in the LogSource.
It allows you to account for multiple types of logs from multiple resources, so I do think the answer about the type being part of the filter is “Yes”
The LS that is type “syslog” would ignore a windows event from the same device, but if you needed to map them two different ways, maybe by IP for one and FQDN for the other, you could because thats what the LogSource is doing. It removes the need to change either your collector config or your resources.
Resource mapping in the LogSource doesn't care what is in the collector config, so its going to try to map off whatever property you tell it to.
LM User
·2 years agoYeah, but what if there are non-syslog logs associated with your haproxy devices? Do i need to exclude them in the AppliesTo somehow by referencing a log property like the _agent? Or is it enough that i’m specifying “LM Logs: Syslog” as the type? See where i’m going here? The module actually applies to logs, not to the device. Applying the logsource to the device is only part of the application and is relating on an object one degree away from the actual objects being processed. That’s what i mean by “the appliesto should apply to a log pipeline”. I want to make one LogSource that processes all my syslog across all devices, since all that syslog follows the same, well defined syntax. However, i also have other logs on devices that are not syslog and shouldn’t be processed by this logsource. Hence my question about the type also factoring into the filter. If it does, i can just put “isLinux() || isNetwork()” as my appliesto and it’ll parse all syslog across all devices.
So, the “key” is the name I choose for that field? That’s good news since i thought it was used to map to some attribute existent in the incoming log entry. Maybe that thing is called a key, but the UI should be updated to indicate that it’s going to be the name of the log entry attribute. Maybe call it “Desired Attribute Name”?
It looks like resource mapping is required. I’m not sure why this would be required since the resource mapping is already working based on the default settings already in the agent.conf for lmlogs. Those lines are currently like this:
The resource mapping lines are commented out, but IIRC, they are the default anyway. If so, I suppose I’d choose “LM Property(Token)” and use “device” as the key and “##system.hostname##” as the value? The documentation is really heavy on using a custom token. However, that probably represents 1% of the use cases out there. Most people are going to be sending from Linux and IOS. How do we map that (especially since it’s already mapping correctly without setting it)?
JJumpp
·2 years agoIm not a LogSource expert at the moment, but the HAProxy parsing was a situation that I recently tackled and am excited to share some knowledge around. So I’ll take a crack as some of these concerns too.
For me, I think of the LogSources almost like PropertySources for a log. They are there to pull the “metadata” of the logs, not to sort and alert on them like pipelines and alert conditions.
Its not a 1:1 translation by any means, and someone more versed in the LogSources might even cringe a little, but thats what works for me when thinking about them.
So, just like PropertySources, the Applies to is relevant to the device, not the group(pipeline). So if all of my haproxy devices (all delightfully named haproxy01, 02 etc) are eventually going to be sending logs with the same format to LM Logs, then my applies to might look like “system.displayname =~ “haproxy”
Over simplified, but you get where I’m going.
When I build a LogSource, Im trying to break it down into as many components as makes sense for me to work with later. And sometimes that means being overly obvious when Im configuring fields/tags or filters. If i want the Facility, then i make a field with the syslog attribute method, Set the Key as Facility and pick Facility out of the value dropdown. If i want that to be something i filter on, then i set the filter.
I do agree that there’s a lot of room for improvement in both the function and the documentation, so keep sending feedback. That’s how we make things change.
LM User
·2 years agoMaybe it’s just me, but I don’t understand log sources. For so long i thought they were going to be something else (datapoints from logs). It seems to me the applies to should apply to pipelines instead of resources.
Is “type” part of the filter? If i choose LM Syslog, will only logs for devices in the appliesto that are sent as syslogs to an LM collector be processed by this logsource?
The support documents spend an inordinate amount on resource mapping, which is already working for my syslogs coming in. Is there any documentation on how to setup the mapping of syslog attributes? What do i put for the key? Do i just leave it blank?
Seems like it would take someone at LM familiar with the workings of LogSources about 15 minutes to write up a LogSource template. Why are they waiting for customers to do it for them?
JJumpp
·2 years agoThis isn’t enough. There should be more decoding happening with syslog given it’s maturity. Having priority as its own field is part of the way there, but there should be decoding ability to split it into facility and severity and map them to their well known labels.
Why isn’t there a logsource out of the box to decode syslog and extract the relevant fields? Even if it’s just a template that I can copy and apply to certain sets of logs.
You arent wrong. There should be LogSources available OOB that already have some of the basics configured so you can just clone and carry on with matching your specific use case. Since LogSources are still in Open Beta, I cant speak to if/when those would be available.
In the meantime, we can still build our own. and for Syslog, the basic Application, Facility, Severity attributes can be parsed without using special regex.
All 3 are predefined parameters under the Fields/Tags section if you choose Syslog Attribute from the dropdown.
LM User
·2 years agoThis isn’t enough. There should be more decoding happening with syslog given it’s maturity. Having priority as its own field is part of the way there, but there should be decoding ability to split it into facility and severity and map them to their well known labels.
Why isn’t there a logsource out of the box to decode syslog and extract the relevant fields? Even if it’s just a template that I can copy and apply to certain sets of logs.
Cole McDonald
·2 years agoI haven’t dealt with these directly, but most of the other ‘Sources in LM will let you use Grep in your datapoints.
for those:
() surrounds the piece you actually want to grab
.* is one or more characters
^ is line start
$ is line end
\s+ is any one or more contiguous whitespace
[] surrounds ranges/sets
searching for IP in there would be something like this in regex:
.*([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})
.* is anything before the pattern, the () is the returned piece, \. is a literal dot character [0-9] is a single digit matching that range, {1,3} modifies the digit to be 1-3 contiguous matches
JJumpp
·2 years agoA LogSource can break out all of these fields using the Regex Dynamic Group method. You can create multiple fields in a single line/expression.
https://www.logicmonitor.com/support/syslog-logsource-configuration#h-configuration-options
Before anything, make sure your collector is the right version to use LogSources and chose the “Enable LM Logs” option for the resource if the logs are being sent to a different collector than the one collecting regular monitoring metrics.
Then build your Regex.
It would start something like
([a-zA-Z0-9_ ]*\d+:\d+:\d+) (\S+) (\S+) .+:?.+
and keep breaking down each section you need to parse into different groups. Some characters may need to be escaped to be accepted by the LogSource, but once you have it in, it works pretty well.
May take a few minutes to kick in, but you should start seeing them soon. Line up the keys that correspond to each group in your expression in a comma separated list, no spaces (log_time,server,socket, etc)
May sure none of your fields are named something that could conflict with the standard format, like “timestamp” or “source” since that may break your ingestion.
For troubleshooting, temporarily turn on your collector logging for syslog to debug (https://www.logicmonitor.com/support/collectors/collector-management/collector-logging) then go into your debug facility and try
!tail ..\logs\wrapper.log 1000
to see any errors that may be associated with your logs.
Make sure to turn off that debug level when you are done.
Hope this helps!