netflow data access via API | Feature Requests

mnagel · 2018-07-18T16:47:49Z

Among other reasons, it would be very desirable to have access to Netflow data via the API for at least these reasons: * detect missing netflow data due to misconfiguration or similar; this includes both persistent lack of data and data gaps * detect and alert on unusual traffic in a customized manner Missing data without awareness is probably the single biggest weakness in LM, and with Netflow there is literally nothing that can be done short of visual inspection of every device in the UI. Thanks, Mark

Henry Steinhauer

·5 years ago

Hi, I am a new user to Logic Monitor - and this thread caught my attention.

Yes - Knowing when Netflow suddenly stops being received - which implies stops being sent.
Yes - it would be very nice to know this sooner rather than later.

Did you get your API working with the new version?

This post is 2+ years old - I hope you have made process getting this monitored.

Mosh

·8 years ago

Thanks @Sarah Terry

Sarah Terry

·8 years ago

@Mosh @mnagel We will publish the ability to get netflow data with v2 of the API, which is coming soon. v2 is not quite ready to be published yet, so use it with caution until it's officially published (it is subject to change), but we should be ready in the next couple of releases - stay tuned!

Mosh

·8 years ago

Thanks @mnagel Tested from Postman and works for me too.

mnagel

OP8 years ago

2 hours ago, mnagel said:

OK -- under control for now. DISCLAIMER: everything is subject to change, as it was before when accessing undocumented REST resources.

The short answer is you must now include an 'X-version: 2' header in your request headers, or add 'v=2' as a query param. When you do this, it is supposed to work for any endpoint except one (can't recall which), but I updated my module to use v2 only as needed for specific endpoint patterns.

The other important finding is that the data structure returned for the v2 API is different. Instead of items being a subkey of data, it is (for now anyway) a top-level key. I also had been checking the status key (copy of the HTTP response code), but that was removed.

I am still working on updates to my scripts, but the netflow script is working again. I will post my revised LMAPI.pm to github when it is stable.

Updated WM::LMAPI.pm at https://github.com/willingminds/lmapi-scripts.git -- still working on more issues, but this handles most of my pain.

mnagel

OP8 years ago

2 hours ago, mnagel said:

I have been given a tentative fix, will post if I get it working...

OK -- under control for now. DISCLAIMER: everything is subject to change, as it was before when accessing undocumented REST resources.

The short answer is you must now include an 'X-version: 2' header in your request headers, or add 'v=2' as a query param. When you do this, it is supposed to work for any endpoint except one (can't recall which), but I updated my module to use v2 only as needed for specific endpoint patterns.

The other important finding is that the data structure returned for the v2 API is different. Instead of items being a subkey of data, it is (for now anyway) a top-level key. I also had been checking the status key (copy of the HTTP response code), but that was removed.

I am still working on updates to my scripts, but the netflow script is working again. I will post my revised LMAPI.pm to github when it is stable.

mnagel

OP8 years ago

I have been given a tentative fix, will post if I get it working...

Mosh

·8 years ago

Forgot to include the headers I was referring to:

Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Charset: utf-8, iso-8859-1;q=0.5

Mosh

·8 years ago

Hmm. I've tried setting the Accept* header values to try to appease the 406, but still not working for me either.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/406

mnagel

OP8 years ago

On 8/14/2018 at 4:15 PM, mnagel said:

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.). But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

So it turns out in v109 this and a bunch of other endpoints broke (e.g., netscans). It is not like we were not warned this might happen when using undocumented endpoints. From what I have heard, it seems that now that the v2 API is in development, all the endpoints we were using to obtain data otherwise impossible to access stopped working.

[mnagel@colby lmapi-scripts]$ ./get-top-talkers --company xxx
Problem with request:
Request: https://xxx.logicmonitor.com/santaba/rest/device/devices/7/flows?size=10&sort=-usage
Response Status: 406 Not Acceptable
at ./get-top-talkers line 53

Sigh.

mnagel

OP8 years ago

2 hours ago, Mosh said:

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used. If you wanted to narrow the time period to check for data, you should be able to play with these two params.

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.). But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

Mosh

·8 years ago

On 8/13/2018 at 9:52 PM, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed. I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice! I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this. Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both). Definitely important to include the size parameter, though. Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
if ($device->{enableNetflow} == JSON::true) {
for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
print Dumper($flow);
}
}
}

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used. If you wanted to narrow the time period to check for data, you should be able to play with these two params.

mnagel

OP8 years ago

On 7/30/2018 at 8:36 AM, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed. I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice! I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this. Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both). Definitely important to include the size parameter, though. Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
if ($device->{enableNetflow} == JSON::true) {
for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
print Dumper($flow);
}
}
}

Mosh

·8 years ago

(BTW, re SNMP, I ended up implementing a uptime via SNMP poll just to alert if there is no SNMP response.)

Mosh

·8 years ago

42 minutes ago, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed. I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

There's no documentation, however, I can see these REST endpoints are invoked by the Traffic view:

Top Talkers
/santaba/rest/device/devices/{deviceID}/topTalkersGraph?netflowFilter={}&end=1532968403&start=1532961173&time=2hour&_=1532968368909

Endpoints
/santaba/rest/device/devices/{deviceID}/endpoints?filter=type%3A"destination"&sort=-usage&time=2hour&start=1532961173&end=1532968403&netflowFilter={}&size=10&offset=0&_=1532968368905

Ports
/santaba/rest/device/devices/3942/ports?sort=-usage&time=2hour&start=1532961306&end=1532968536&netflowFilter={}&size=10&offset=0&_=1532968503441

Flows
/santaba/rest/device/devices/{deviceID}/flows?sort=-usage&time=2hour&start=1532957720&end=1532964950&netflowFilter={}&size=10&offset=0&_=1532964896784

mnagel

OP8 years ago

15 minutes ago, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed. I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Mosh

·8 years ago

@mnagel Do you mean the flow data as it appears in the Traffic view?

1	SUYASH GAIKWAD
2	PJ Pankaj Jain
3	KT Kyle Taylor
4	KV karthikeyan VG
5	ME Mir Ehtesham Ali
6	SN Sangeetha Nataraj