Forum Discussion

hvittal's avatar
2 years ago

eBPF monitoring support for Linux hosts

I have seen this with some other enterprise solutions and it’s a popular choice when we don’t want ‘APM’ level metrics but we are interested in the HTTP metrics such as requests/sec, error/sec, etc.

Redhat has a decent article on this:

Monitoring eBPF-based metrics (redhat.com)

Some blogs focussed on HTTP metrics:
eBPF enhanced HTTP observability — L7 metrics and tracing | by Apache SkyWalking | Jan, 2023 | Medium

I am starting to see DevOps requests this sort of monitoring especially on third party software where instrumentation for APM is not possible or preferred by vendor. 

We’d recently had a requirement similar to this for an enterprise product and LM can only poll HTTP and record results and not be able to track incoming requests to the server.

Quite like to see this being supported - DataDog already supports this and I can see this being quite important going forward.

  • Anonymous's avatar
    Anonymous

    Something to think about: LM doesn’t (and will likely never) generate metrics. It only gathers metrics. eBPF looks like a way of generating metrics, so really the only thing to think about is how to gather those metrics into LM. The first thing that comes to mind is opentelemetry and the second is fluentd.

    Does eBPF support streaming metrics through the opentelemetry protocol? If so, the existing methods for ingesting OTel data could be used, although they are limited through licensing to APM customers because they’re the most common users of OTel. If this kind of metric can be streamed through the OTel protocol, LM might have to think about decoupling the collection method and the data and licensing on the kind of data received (infrastructure management like eBPF, or trace data like APM) instead of the protocol used to collect it.

    Does FluentD have a plugin for receiving eBPF telemetry data? If so, you could easily use the LM gem in FluentD to turn that data around and put it into LM Logs. From there, when LogSources are eventually released, their functionality should cover converting that data from logs into datapoints that can be graphed/alerted on.

    I cringe whenever someone says LM can’t do something, because usually if LM can’t do it, it’s because it can’t be done.

  • Something to think about: LM doesn’t (and will likely never) generate metrics. It only gathers metrics. eBPF looks like a way of generating metrics, so really the only thing to think about is how to gather those metrics into LM. The first thing that comes to mind is opentelemetry and the second is fluentd.

    Does eBPF support streaming metrics through the opentelemetry protocol? If so, the existing methods for ingesting OTel data could be used, although they are limited through licensing to APM customers because they’re the most common users of OTel. If this kind of metric can be streamed through the OTel protocol, LM might have to think about decoupling the collection method and the data and licensing on the kind of data received (infrastructure management like eBPF, or trace data like APM) instead of the protocol used to collect it.

    Does FluentD have a plugin for receiving eBPF telemetry data? If so, you could easily use the LM gem in FluentD to turn that data around and put it into LM Logs. From there, when LogSources are eventually released, their functionality should cover converting that data from logs into datapoints that can be graphed/alerted on.

    I cringe whenever someone says LM can’t do something, because usually if LM can’t do it, it’s because it can’t be done.


    By that logic, LM shouldn’t have support integration into anything - just provide a platform to run groovy functions and hope someone comes up with a shoddy solution to dump data in. 

    I expect an enterprise product - to keep on top of new technologies and support newer metric collection methods so you don’t have to create a custom solution for everything.

    My point being, LogicMonitor should support polling/prasing this data directly via their collectors… like other vendors are doing. If I didn't think LM can support this, I wouldn’t be raising it as a request idea… it clearly can, just a matter of adding official support into the product to do so… which I am sure LM as a vendor is in a better position to do than any individual contributor. 

  • Anonymous's avatar
    Anonymous

    I get it. But if you wait for LM to do the things you need, you might be waiting for a long time. 

    just provide a platform to run groovy functions

    That is LM in a nutshell. It was what LM built first, then set about building (better than shoddy) solutions to dump data in. It’s just a cloud based task dispenser to local collectors who execute the tasks and report back the output. However, they also employ a very good team of “Monitoring Engineers” who write the LogicModules OOTB.

  • I get it. But if you wait for LM to do the things you need, you might be waiting for a long time. 

    just provide a platform to run groovy functions

    That is LM in a nutshell. It was what LM built first, then set about building (better than shoddy) solutions to dump data in. It’s just a cloud based task dispenser to local collectors who execute the tasks and report back the output. However, they also employ a very good team of “Monitoring Engineers” who write the LogicModules OOTB.

    As new-ish customer of LM, I am yet to gauge what “waiting for a long time” means but having worked with other vendors, it’s not surprising… I am just hoping LMer gets bored someday and happens to scroll through this post and takes this up as a serious feature request :) Hopefully I can then retire any shoddy solutions I come up with! Thanks for the suggestions, always appreciated!

  • LMer gets bored someday and happens to scroll through this post and takes this up as a serious feature request 🙂 Hopefully I can then retire any shoddy solutions I come up with! 

    This doesn’t happen (at least not currently, feature requests here don’t yet feed into the system). You need to actually request it through the product. See sticky post: 

    https://community.logicmonitor.com/feature-requests-5/how-to-submit-a-feature-request-3268

    Thanks, yes - I have done that as well… I was hoping these are both related somehow but at least here if other customers can chime in support it’d be more visible.. than the ‘official’ method where I have no idea if anyone is even interested.

  • @Stuart Weenig Always appreciate your feedback, buddy!

    @hvittal I have added this to our sheet and will follow up with Product as well.