netflow data access via API

(BTW, re SNMP, I ended up implementing a uptime via SNMP poll just to alert if there is no SNMP response.)

2 hours ago, mnagel said:

I have been given a tentative fix, will post if I get it working...

OK -- under control for now.  DISCLAIMER: everything is subject to change, as it was before when accessing undocumented REST resources.

The short answer is you must now include an 'X-version: 2' header in your request headers, or add 'v=2' as a query param.  When you do this, it is supposed to work for any endpoint except one (can't recall which), but I updated my module to use v2 only as needed for specific endpoint patterns. 

The other important finding is that the data structure returned for the v2 API is different.  Instead of items being a subkey of data, it is (for now anyway) a top-level key.  I also had been checking the status key (copy of the HTTP response code), but that was removed. 

I am still working on updates to my scripts, but the netflow script is working again.  I will post my revised LMAPI.pm to github when it is stable.

@mnagel Do you mean the flow data as it appears in the Traffic view?

15 minutes ago, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

42 minutes ago, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

 

There's no documentation, however, I can see these REST endpoints are invoked by the Traffic view:

Top Talkers
/santaba/rest/device/devices/{deviceID}/topTalkersGraph?netflowFilter={}&end=1532968403&start=1532961173&time=2hour&_=1532968368909

Endpoints
/santaba/rest/device/devices/{deviceID}/endpoints?filter=type%3A"destination"&sort=-usage&time=2hour&start=1532961173&end=1532968403&netflowFilter={}&size=10&offset=0&_=1532968368905

Ports
/santaba/rest/device/devices/3942/ports?sort=-usage&time=2hour&start=1532961306&end=1532968536&netflowFilter={}&size=10&offset=0&_=1532968503441

Flows
/santaba/rest/device/devices/{deviceID}/flows?sort=-usage&time=2hour&start=1532957720&end=1532964950&netflowFilter={}&size=10&offset=0&_=1532964896784
 

On 7/30/2018 at 8:36 AM, Mosh said:

@mnagel Do you mean the flow data as it appears in the Traffic view?

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice!  I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this.  Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both).  Definitely important to include the size parameter, though.  Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
    if ($device->{enableNetflow} == JSON::true) {
        for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
            print Dumper($flow);
        }
    }
}

On 8/13/2018 at 9:52 PM, mnagel said:

@mosh Correct -- there is no access to this information via the API and no alerting capability, so if data is not populating for any reason, it is an unpleasant surprise when the data is needed.  I run into this all the time, but have been able to create alert rules for key datapoints that let me know when stuff is broken (e.g., I found recently SNMP breaks when you have intermediate firewalls since the code re-uses a single session and sometimes passes data too infrequently to keep the "connection" active).

Very nice!  I have an initial script now that pulls top flows for netflow-enabled devices, so now I must see what I will do with this.  Minimally, detect lack of data and either opsnotes additions for exceptional usage or email alerts (or both).  Definitely important to include the size parameter, though.  Core of current in-progress script:

my $devices = $lmapi->get_all(path => "/device/devices", fields => "id,displayName,enableNetflow");
for my $device (@$devices) {
    if ($device->{enableNetflow} == JSON::true) {
        for my $flow ($lmapi->get_all(path => "/device/devices/$device->{id}/flows", size => 10, sort => '-usage', time => '2hour')) {
            print Dumper($flow);
        }
    }
}

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used.  If you wanted to narrow the time period to check for data, you should be able to play with these two params.

2 hours ago, Mosh said:

 

I believe the two parameters "start" and "end" are the epochs to use for the "time" parameter. Could be that omitting these means current time is automatically used.  If you wanted to narrow the time period to check for data, you should be able to play with these two params.

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.).  But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

On 8/14/2018 at 4:15 PM, mnagel said:

Yep, I concluded they were autoset based on current time, which is typically what I want (verify if we have recent data, generate alerts based on usage, etc.).  But sure, if I want to do a sliding analysis over previous windows, those params will be needed.

So it turns out in v109 this and a bunch of other endpoints broke (e.g., netscans).  It is not like we were not warned this might happen when using undocumented endpoints.  From what I have heard, it seems that now that the v2 API is in development, all the endpoints we were using to obtain data otherwise impossible to access stopped working.

[mnagel@colby lmapi-scripts]$ ./get-top-talkers --company xxx
Problem with request:
    Request: https://xxx.logicmonitor.com/santaba/rest/device/devices/7/flows?size=10&sort=-usage
    Response Status: 406 Not Acceptable
 at ./get-top-talkers line 53

Sigh.

Hmm.  I've tried setting the Accept* header values to try to appease the 406, but still not working for me either.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/406