Forum Discussion

mnagel's avatar
mnagel
Icon for Professor rankProfessor
3 years ago

Netflow filtering

For Netflow display, there are numerous filters available, including interfaces.  We used this to show flows for each ISP connection at one location, but found the table to be empty most of the time for one ISP.  Support went to dev and came back with this:

I'm advised that during data aggregation on the collector-side, we only display flows which are contributing to 95% of the Netflow traffic. I'm told that once we have the set of flows which together contribute to 95% of traffic, LM will paint this data in the UI. The rest of the 5% traffic flow is ignored in the UI.

I disagree this should be an expected behavior since the implication is that flows are reduced overall before filtering and that makes filtering effectively useless.  Any such narrowing should be done based on the filters, then reduced for display in the UI.  However, I was told this is not a bug but a feature request so I am posting in hopes it will be addressed sometime soon.

  • Marc___'s avatar
    Marc___
    3 years ago

    Not speaking on behalf of anyone at LM, just a random guy who has worked with the backend of a lot on flow analysis tools, I'd be legit shocked if they changed this in product.  Dropping the lowest 5% of conversations is an optimization strategy that is pretty common among flow tools, and most of them just bury it deep in some setting somewhere and users rarely stumble across it so they just never notice.  I think the support person may have explained it wrong, unless LM implemented this in a really unusual way what usually happens is not that the bottom 5% just don't get displayed, the bottom 5% are usually not even getting written into the database at all.  In most cases dropping that last 5% of low volume flows records ends up reducing the actual number of flows you have to write to the DB and store by a huge amount, i've seen 100% collection increase the DB storage size by 80x compared to just keeping the largest 95% of conversations. There ends up being millions of little fragments of conversations that never amount to even 1kb of traffic, and the computational load to store and display them when they hardly even show up on a chart is usually not worth it for most people.  The only tools I've seen that don't implement that kind of dropping strategy are selling monster appliances marketed as security solutions where every single byte gets tracked and there is no such thing as a budget constraint.

  • mnagel's avatar
    mnagel
    Icon for Professor rankProfessor
    3 years ago

    Unfortunately, LM restricts data much worse than that. If it was 95% we’d probably be fine. Support update:

    The latest re-explanation I got was, again, that flows sent by Netflow device are aggregated and capped to either top 1000 flows, or flows that make up 95% of traffic, whichever is earlier.

    In this case, I will probably just need to stop exporting internal interface flows to deal with the restriction.