TCP Syslog?

Hey @mnagel, I don't have any Cisco reference to this issue on other devices, but we've seen it reported on other devices.

But yes, essentially syslog4j does not fill the hostname field in as described in the RFC, it just fails to parse the message. We modified to do what's described in the RFC; we were not in compliance with it previously.

So, there's no hard requirement that you send us RFC compliant messages, but if syslog4j can't parse them, they won't work in LM until we patch it. When we told you "it's not working because it's not RFC compliant", we probably should have said "our implementation can't currently handle it, but we'll fix it".



 

I was provided that link previously, but that bug ID applies to a narrow set of devices (ASR5K) and releases.  Is there other documentation saying Cisco is not sending complete syslog packets?  The first and last device we tested with was a Catalyst 2960X stack and we wanted to filter storm control messages into an event.  What we found was that unless we used .*, the filter would not match and the results were garbled.  I assume this means there is an undocumented dependency on RFC5424.

An RFC 3164 syslog message format is:

   The full format of a syslog message seen on the wire has three
   discernable parts.  The first part is called the PRI, the second part
   is the HEADER, and the third part is the MSG.  The total length of
   the packet MUST be 1024 bytes or less.  There is no minimum length of
   the syslog message although sending a syslog packet with no contents
   is worthless and SHOULD NOT be transmitted.

There is a hostname field within the HEADER part:

   The HEADER contains two fields called the TIMESTAMP and the HOSTNAME.
   The TIMESTAMP will immediately follow the trailing ">" from the PRI
   part and single space characters MUST follow each of the TIMESTAMP
   and HOSTNAME fields.  HOSTNAME will contain the hostname, as it knows
   itself.  If it does not have a hostname, then it will contain its own
   IP address.  If a device has multiple IP addresses, it has usually
   been seen to use the IP address from which the message is
   transmitted.  An alternative to this behavior has also been seen.  In
   that case, a device may be configured to send all messages using a
   single source IP address regardless of the interface from which the
   message is sent.  This will provide a single consistent HOSTNAME for
   all messages sent from a device.

I captured UDP/514 traffic from one random Catalyst switch we saw some issues on and this is the result:

14:47:02.785771 IP (tos 0x0, ttl 254, id 3202, offset 0, flags [none], proto: UDP (17), length: 139) calswerp01.52088 > <<TARGET>>.syslog: SYSLOG, length: 111
        Facility local6 (22), Severity alert (1)
        Msg: 724147: Feb 26 14:47:01.775 PST: %PLATFORM_ENV-1-[|syslog]
14:49:49.366330 IP (tos 0x0, ttl 254, id 3203, offset 0, flags [none], proto: UDP (17), length: 154) calswerp01.52088 > <<TARGET>>.syslog: SYSLOG, length: 126
        Facility local6 (22), Severity alert (1)
        Msg: 724148: Feb 26 14:49:47.351 PST: %PLATFORM_ENV-1-[|syslog]
14:52:03.216378 IP (tos 0x0, ttl 254, id 3204, offset 0, flags [none], proto: UDP (17), length: 139) calswerp01.52088 > <<TARGET>>.syslog: SYSLOG, length: 111
        Facility local6 (22), Severity alert (1)
        Msg: 724149: Feb 26 14:52:02.208 PST: %PLATFORM_ENV-1-[|syslog]

So, it seems like the HEADER part is not there, but even this case is considered in the RFC:

4.3.2 Valid PRI but no TIMESTAMP or invalid TIMESTAMP

   If a relay does not find a valid TIMESTAMP in a received syslog
   packet, then it MUST add a TIMESTAMP and a space character
   immediately after the closing angle bracket of the PRI part.  It
   SHOULD additionally add a HOSTNAME and a space character after the
   TIMESTAMP.  These fields are described here and detailed in Section
   4.1.2.  The remainder of the received packet MUST be treated as the
   CONTENT field of the MSG and appended.  Since the relay would have no
   way to determine the originating process from the device that
   originated the message, the TAG value cannot be determined and will
   not be included.

   The TIMESTAMP will be the current local time of the relay.

   The HOSTNAME will be the name of the device, as it is known by the
   relay.  If the name cannot be determined, the IP address of the
   device will be used.

   If the relay adds a TIMESTAMP, or a TIMESTAMP and HOSTNAME, after the
   PRI part, then it MUST check that the total length of the packet is
   still 1024 bytes or less.  If the packet has been expanded beyond
   1024 bytes, then the relay MUST truncate the packet to be 1024 bytes.
   This may cause the loss of vital information from the end of the
   original packet.  It is for this reason that it is RECOMMENDED that
   the PRI and HEADER parts of originally generated syslog packets
   contain the values and fields documented in Section 4.1.

Basically, if the hostname is not provided, it is the connecting IP address, as all other syslog receivers assume.  Perhaps the issue here is that log4j is failing to fill in those fields as documented in the RFC?

Most network equipment generally emits RFC3164 format despite the newer/obsoleting RFC, and we have to live in the real world.  We experience no issues with Cisco format on any syslog server we use like syslogd, rsyslogd, syslog-ng, etc., and we've had no issues with SumoLogic. 

I would say at minimum, if there is a required RFC version, that should be documented.  More to the point, the real world where folks have to do actual useful monitoring must be accounted for.

Hey @mnagel, my understanding is that neither RFC allows for a message lacking the hostname, which is what we were seeing with lots of Cisco syslog. They seem to admit as much: https://quickview.cloudapps.cisco.com/quickview/bug/CSCvc35989

We use syslog4j under the hood to parse messages. It could not parse them out of the box, so we modified it to handle Cisco's syslog. RFC compliant or not, we can't reasonably not support syslog for one of the largest network vendors in the world.

The graylog project did essentially the same thing to handle Cisco and Fortinet syslog.


As for the field limitations, I need to do some more reading. I realize you have another solution, but I'd still like to improve what we have.

@Michael Rodrigues thanks for the update -- will try it out again once I have upgraded to the new version.

I was told this same thing by our CSM at the time, but the idea that Cisco syslog was somehow unsupported was very odd, and there is nothing in  https://www.logicmonitor.com/support/eventsources/types-of-events/syslog-monitoring/ that mentions any specific RFC requirement.  I am aware the newer https://tools.ietf.org/html/rfc5424 has definitions to allow for structured fields and this obsoletes the original https://tools.ietf.org/html/rfc3164, but certainly most devices out there like Cisco, HP, etc. still use 3164 so it was odd to hear "violates the RFC".  Even then, RFC 5424 allows for 3164-style unstructured messages (where there are zero structured elements).

The impact of this on our attempts to leverage syslog eventsources was that we could not create working filters on the message field, which is traditionally one of very few fields you can expect in syslog messages.  RFC3164 fields (some, not all) continue to be the only ones you can define in the eventsource filter, so I am still unsure what RFC is being violated by Cisco:

With our SumoLogic setup, we are able to create all sorts of fields from regex matches so we have stuff like cisco_subsys, cisco_severity, etc. from the %XXX-M-XXX: string within messages, for example.  I don't really see LM working for syslog in its current implementation, unfortunately.  My goal has been to integrate with SumoLogic instead via their API.
 

@mnagel we did fix an issue with Cisco sending syslog that wasn't RFC compliant in 27.600. I don't have any info about TCP syslog currently, but I can talk to the collector team.

Sadly, syslog doesn't really work, at least not for Cisco devices .  +1 for having it work AND do so with both UDP and TCP.  That said, until there is a basic counting/correlation function, it won't really help too much.  We ended up deploying SumoLogic to deal with network device logs.

+1

I agree - this would be a useful feature