New user - looking for information on parsing fields from Syslog message field

This isn’t enough. There should be more decoding happening with syslog given it’s maturity. Having priority as its own field is part of the way there, but there should be decoding ability to split it into facility and severity and map them to their well known labels.

Why isn’t there a logsource out of the box to decode syslog and extract the relevant fields? Even if it’s just a template that I can copy and apply to certain sets of logs.

You arent wrong.  There should be LogSources available OOB that already have some of the basics configured so you can just clone and carry on with matching your specific use case.  Since LogSources are still in Open Beta, I cant speak to if/when those would be available.  

In the meantime, we can still build our own.  and for Syslog, the basic Application, Facility, Severity attributes can be parsed without using special regex.  
All 3 are predefined parameters under the Fields/Tags section if you choose Syslog Attribute from the dropdown.  

Maybe it’s just me, but I don’t understand log sources. For so long i thought they were going to be something else (datapoints from logs). It seems to me the applies to should apply to pipelines instead of resources. 

Is “type” part of the filter? If i choose LM Syslog, will only logs for devices in the appliesto that are sent as syslogs to an LM collector be processed by this logsource?

The support documents spend an inordinate amount on resource mapping, which is already working for my syslogs coming in. Is there any documentation on how to setup the mapping of syslog attributes? What do i put for the key? Do i just leave it blank?

Seems like it would take someone at LM familiar with the workings of LogSources about 15 minutes to write up a LogSource template. Why are they waiting for customers to do it for them?

@nageeb try one of these.  Im honestly not sure why some of them work at the moment, but the deviceId works for me pretty much every time. 
and if your logs stop coming in, try that debug mode on the logging level for syslog.  Anytime I’ve had logs stop (that wasnt from obvious user intervention) I was able to see the issue when i ran the debug command. 

1. system.deviceId -> ##system.deviceId##
2. IP -> ##system.ips##
3. Hostname -> ##system.hostname##
4. Hostname without DNS -> ##system.sysname##
   
    

@nageeb to better understand how LM is structured:

• Devices are the core piece of the data collection puzzle.
• (Groups are separately defined objects that are added as properties to devices)
• ‘Sources are just scripts on timers that collect datapoints.
• ‘Sources get applied to devices through discovery or manual application to create instances (device + ‘Source = instance).
• A value from an instance crossing a defined threshold raises an alert
• Alerts are processed through the Alert Rule stack from low to high
• Alert Rules send the alert to an Escalation Chain
• Escalations can send emails, text messages, or target Integrations (Anything predefined or using a webhook)

Our full alert pipeline:

Device > ‘Source appliesTo() matches the device > Active discovery adds an instance > datapoints are collected for that instance > crossing threshold triggers an alert which matches an alert rule and gets sent to an escalation chain which generates custom JSON > webhook integration> Teams channel for technician consumption.

The custom JSON adds dynamic buttons with links to the rest of our toolset (KB, RMM, Azure Portal, PW Manager, Asset Management, Etc.)  These are populated from custom properties on the groups/devices.

It’s non-agented, so all code is running on the collector itself.  If you need data from the device you’re targeting, the code running on the collector has to reach out remotely.  I use Powershell in Windows, so things like:

get-service spooler -computername ##system.displayname##

get the data from the far end.  If it’s more complex data using non-remote types of commands:

invoke-command -scriptblock {netstat -ano} -computername ##system.displayname##

can get that script block to run on the remove machine and pass the results back to your script.  There are caveats there such as making sure to use sessions that don’t load profiles if you’re using fslogix or upds that will need to mount a remote volume for each login (there can be thousands for that single account that may cause resource contention).

Topology and Services are my next area of exploration to see if I can recreate workflow based alerting for tiered enterprise application stacks.  We host software, but don’t write it, so we have issues with the existing application monitoring options provided which require monitoring hooks be coded into the applications.

Groovy is a weird second cousin to Java and seems to run very efficiently.

The escalations don’t allow direct code execution (which would be amazing)… but the dashboarding is quite a bit more powerful than other platforms I’ve used in the past… and presenting data to “do-ers” in meaningful ways is where LM can really shine.

Im not a LogSource expert at the moment, but the HAProxy parsing was a situation that I recently tackled and am excited to share some knowledge around. So I’ll take a crack as some of these concerns too.

For me, I think of the LogSources almost like PropertySources for a log.  They are there to pull the “metadata” of the logs, not to sort and alert on them like pipelines and alert conditions.  
Its not a 1:1 translation by any means, and someone more versed in the LogSources might even cringe a little, but thats what works for me when thinking about them.  

So, just like PropertySources, the Applies to is relevant to the device, not the group(pipeline).  So if all of my haproxy devices (all delightfully named haproxy01, 02 etc) are eventually going to be sending logs with the same format to LM Logs, then my applies to might look like “system.displayname =~ “haproxy”

Over simplified, but you get where I’m going.

When I build a LogSource, Im trying to break it down into as many components as makes sense for me to work with later.  And sometimes that means being overly obvious when Im configuring fields/tags or filters.  If i want the Facility, then i make a field with the syslog attribute method, Set the Key as Facility and pick Facility out of the value dropdown.  If i want that to be something i filter on, then i set the filter. 

I do agree that there’s a lot of room for improvement in both the function and the documentation, so keep sending feedback.  That’s how we make things change. 

Yeah, but what if there are non-syslog logs associated with your haproxy devices? Do i need to exclude them in the AppliesTo somehow by referencing a log property like the _agent? Or is it enough that i’m specifying “LM Logs: Syslog” as the type? See where i’m going here? The module actually applies to logs, not to the device. Applying the logsource to the device is only part of the application and is relating on an object one degree away from the actual objects being processed. That’s what i mean by “the appliesto should apply to a log pipeline”. I want to make one LogSource that processes all my syslog across all devices, since all that syslog follows the same, well defined syntax. However, i also have other logs on devices that are not syslog and shouldn’t be processed by this logsource. Hence my question about the type also factoring into the filter. If it does, i can just put “isLinux() || isNetwork()” as my appliesto and it’ll parse all syslog across all devices.

So, the “key” is the name I choose for that field? That’s good news since i thought it was used to map to some attribute existent in the incoming log entry. Maybe that thing is called a key, but the UI should be updated to indicate that it’s going to be the name of the log entry attribute. Maybe call it “Desired Attribute Name”?

It looks like resource mapping is required. I’m not sure why this would be required since the resource mapping is already working based on the default settings already in the agent.conf for lmlogs. Those lines are currently like this:

# Enable LM Logs
lmlogs.syslog.enabled=true

# Resource properties which we want to use for device mapping
# lmlogs.syslog.property.name=system.hostname

# Resource properties value format. Possible values are IP,FQDN,HOSTNAME
# lmlogs.syslog.hostname.format=IP

# lmlogs.syslog.UseHostNameFromMessages=false
lmlogs.syslog.useTimestampWhenCollectorReceivedLogs=true

lmlogs.syslog.UseSocketAddressIfhostNameisInvalid=true

lmlogs.thread.count.for.ingest.api.communication=20


# Configuration to enable the log collection for windows event logs

# Mention the poll interval for windows event log collection in minutes
# lmlogs.windowseventlogs.pollInterval=1

lmlogs.encrypt.data.saved.in.queue=false
lmlogs.salt.for.encryption=

lmlogs.logsource.syslog.unmapped.resource=

The resource mapping lines are commented out, but IIRC, they are the default anyway. If so, I suppose I’d choose “LM Property(Token)” and use “device” as the key and “##system.hostname##” as the value? The documentation is really heavy on using a custom token. However, that probably represents 1% of the use cases out there. Most people are going to be sending from Linux and IOS. How do we map that (especially since it’s already mapping correctly without setting it)?

Ideally, if you are using the LogSource, you would not need to make any edits to your collector config, since all the settings, like resource mapping, are in the LogSource. 
It allows you to account for multiple types of logs from multiple resources, so I do think the answer about the type being part of the filter is “Yes”
The LS that is type “syslog” would ignore a windows event from the same device, but if you needed to map them two different ways, maybe by IP for one and FQDN for the other, you could because thats what the LogSource is doing.  It removes the need to change either your collector config or your resources.
Resource mapping in the LogSource doesn't care what is in the collector config, so its going to try to map off whatever property you tell it to. 

I haven’t dealt with these directly, but most of the other ‘Sources in LM will let you use Grep in your datapoints.

for those:

() surrounds the piece you actually want to grab

.* is one or more characters

^ is line start

$ is line end

\s+ is any one or more contiguous whitespace

[] surrounds ranges/sets

searching for IP in there would be something like this in regex:

.*([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})

.* is anything before the pattern, the () is the returned piece, \. is a literal dot character [0-9] is a single digit matching that range, {1,3} modifies the digit to be 1-3 contiguous matches