Forum Discussion

sfrancis's avatar
sfrancis
8 years ago

ELK as a Service

One thing everybody is looking for is convergence, a single tool that does everything for observability. Monitoring, metrics, log analysis - LM does a good job on the first two, but I still need a separate tool to get useful metrics and trends out of my application logs. LM should look into adding ELK-as-a-Service to the LM feature stack (provide customers with an API endpoint they can feed logs to or something), and then customers could have service-level monitoring (URL response times, etc.), plus the traditional LM suite of monitors/metrics, plus LM Cloud, *plus* the most useful info of all: data mined from application logs. That's generally where the really good insights come from (and most of what's unique to each customer's business/offering).

 
ELK is well-known, open source, and fairly mature. Relatively easy to scale as well; should be easy for LM engineering to put together for a proof of concept anyway.
 
Meanwhile, I'm looking at things like Papertrail, Librato and Logz.io for my application logs - but I'd really like to have One Tool to Rule Them All.
logs
metrics
syslog
  • Jeff_Behl's avatar
    Jeff_Behl
    Former Employee
    8 years ago

    Hi sfrancis,

    Thanks for the comment.  We've discussed log monitoring but not tackled it for a number of reasons, one of which being that the Splunk's, SumoLogic's, etc., of the world are very robust tools that do what they do really well especially when it comes to the terabytes worth of data that is often thrown at them.  Most companies have a solution like this in place already, so integrating with them more tightly is our preferred path at the moment.

    Having said that, "push data" is something we've got on the docket for next year.  While that certainly will include technologies like StatsD, it's not to the exclusion of other formatted data (eg JSON payloads) where something like a ELK may be the right endpoint.  We already take in "events", so it's not too much of a stretch.  To be determined, and input welcome. 

    Jeff