Ping Failing from collector to device and back?
If I do a ping from the command prompt of the Client to the Collector it works
The firewall is wide opened!!!
What did I miss?
Thanks,
Dom
Dominique
Posted 4 years ago·Last reply 2 years ago
40 comments
LM User
·2 years agoICMP is a terrible way for the Collector to determine “Host Dead” status. Better would be if any DataSource is able to collect data from it. If data can be collected, the host isn’t dead.
That actually is close to how it works now. ICMP does reset the idleInterval datapoint (or whatever the internal flag is), which is what determines host down status. However, it’s not the only thing. Any datasource that can be trusted to actually get a reply from a device should reset the idleInterval datapoint. This includes any SNMP datasources, website/http datasources, etc. It does not include scripted datasources. The thinking there is that a scripted datasource might be contacting a 3rd party system to collect data and not actually getting an actual response from the device itself. So, anything that is guaranteed to return data from the device itself should reset the idle interval counter.
The bigger feature request here is that customers need a way to modify/override the built in criteria for considering a device down. For some people, pingability is enough. For others, it needs to be pingable and responding to some other query. The customers need the ability to determine (on the device, group, and global levels) what constitutes a device being down. For example, i would need to be able to say that ping has to be up, but also x/y of these other datasources must also be returning data.
mnagel
·2 years agoICMP is a terrible way for the Collector to determine “Host Dead” status. Better would be if any DataSource is able to collect data from it. If data can be collected, the host isn’t dead.
ICMP itself seems to be fine now, actually. The problem that persists is SNMP when an intermediate stateful inspection engine (firewall) invalidates sessions. UDP is stateless, but SNMP uses a session ID most modern firewalls recognize. Once the session ID is broken, LM stops working since the developers chose to blindly use the same session ID indefinitely. My guess is with the new collector code they periodically refresh the session ID so it eventually recovers rather than trigger a new session after a failed poll or two. The right way is very often not the way these developers roll, sadly,
David Bond
·2 years agoICMP is a terrible way for the Collector to determine “Host Dead” status. Better would be if any DataSource is able to collect data from it. If data can be collected, the host isn’t dead.
mnagel
·2 years agoI am pleased to announce that LM has (after nearly 5 years of back-and-forth -- my first attempt to get this addressed was in June 2018) finally has fixed both the SNMP and ping issues impacted by intermediate firewall session invalidation -- update from support last week:
Hopefully this is actually the case, but if so it will be very nice to tell our clients this longtime bug has finally been quashed.
So I’ve had some time now on EA 34.300 with one of our “problem children” and I am saddened to report the SNMP issues have not been addressed, at least not sufficiently. What I have observed during a spate of recent ISP disruptions for monitoring of a remote site (via IPSec tunnel) is that LogicMonitor eventually seems to figure it out and will begin collecting data, but it takes roughly 2 hours. Having 2 hour gaps is better than indefinite gaps, but it is still unacceptable.
mnagel
·3 years agoI am pleased to announce that LM has (after nearly 5 years of back-and-forth -- my first attempt to get this addressed was in June 2018) finally has fixed both the SNMP and ping issues impacted by intermediate firewall session invalidation -- update from support last week:
Hopefully this is actually the case, but if so it will be very nice to tell our clients this longtime bug has finally been quashed.
LM User
·3 years agoAuto balanced collector group (ABCG) does not mean load balancing. It is overload redistribution.
Example: Two collectors with rebalance threshold of 10,000. One collector has 9,999, the other has 5. No rebalancing happens in this case.
When/if the first collector gets more than 10,000, the device with the highest number of instances will be reassigned to a different collector. If that brings the count back below 10,000, rebalancing stops.
Example: Two collectors with rebalance threshold of 10,000. One collector has 10,001, the other has 5. Largest device on the first collector has 200 instances. Rebalancing happens. The first collector ends up with 9,801, the other has 205.
ABGC != Load Balancing.
Joe Williams
·3 years agoAdjusting the threadpools isn’t something to be done lightly. We have a few standards around them in our deployments based on collector size, but each collector is its own thing. I am hesitant to say what we even do as it probably isn’t the right thing for others to do.
I suggest adding the collector into monitoring and watching the collector graphs. Look at the queue depth, tasks failing, etc. And see where it makes sense to adjust things.
We have also noticed, even if a collector is memory starved, most times, it is better served with giving it more vcpu if possible.
Dominique
OP3 years agoThis is the recommended architecture.
If your firewalls are blocking legitimate business traffic, they need to not do that.
Hello,
Yes we are trying to have the collectors and their client on the same segment
Yes we have multiple Firewalls all through the Network not on the client itself… I will recheck them as the issue does not seem to be wide spread even through a group of servers within a common application which typically are all on the same subnet and have the same firewalls.
Thanks,
Dom
Dominique
OP3 years agoInteresting, I haven't run into that yet myself. I typically have collectors located on the same network segment as the devices being monitored so not hitting firewalls, but some situations do go thru some. Is this specific to Windows or Linux Collectors?
Hello,
I would have to check with the Linux Team as I have only Windows Server 2016 Collectors on my side.
Thanks,
Dom
Dominique
OP3 years agoI suppose it is possible there are two different issues -- threadpool change requirements needed indicates internal resources are exhausted for checks. The original issue is still our constant problem -- static ID values cause sessions to become invalidated by firewalls when there is a disruption on the firewall path. A new session must be observed by the firewall to start letting traffic through again, which can only be done (currently) by restarting the collector since the code allocates those session ID values only one at startup. LM has been informed about this repeatedly, is aware of it and does nothing to fix it. The only thing I’ve seen is a doc note blaming particular firewalls, but it impacts pretty much any stateful inspection firewall. More and more folks use firewalls for internal segmentation and there are many cases where a remote collector is needed due to lack of resources to deploy a local collector.
Hello,
Yes I think the collector are overloaded as all of them for the datacenter implied here are over the Load Balancing settings which would mean there is no more balancing anymore !!!
Thanks,
Dom
Dominique
OP3 years ago@Joe Williams
what size were your collectors and what did you end up adjusting the threadpool too?
I’m new to LM and have been working on implementing it within my company the past few months. I’ve been facing this issue since December and have been going back and forth with LM Support. First, they had me do a fresh install, then they said there was a bug in v 33.x for the collectors and I needed to downgrade and the last step was to completely delete the collectors from the portal and add them again. At this point LM support claims they have done everything they can and have escalated to an internal ticket but I haven’t been able to receive any updates on the status of that ticket. I’m willing to make adjusts me to the threadpools/timeouts if needed to see if this resolved the issue for me.
Hello,
Windows Server 2016 Standard
CPU: Quad 2.40 Ghz Intel Xeon Silver 4214R
RAM: 16 GB
Disk: 75 GB
Collector Version: 32.003
Let me know what I should do to adjust the threadpool?
Thanks,
Dom
Dominique
OP3 years agoInteresting, I haven't run into that yet myself. I typically have collectors located on the same network segment as the devices being monitored so not hitting firewalls, but some situations do go thru some. Is this specific to Windows or Linux Collectors?
Hello,
It is a Windows Server 2019 Standard for the client VIPEIEEMP01, Windows Server 2016 Standard for the “preferred” collector VIPLGCMON02.
The Client has an external IP, so it is attached to the closest Datacenter for its collector.
Thanks,
Dom
Joe Williams
·3 years ago@Joe Williams
what size were your collectors and what did you end up adjusting the threadpool too?
For us it was a careful dance of watching the graphs and the debug commands. I can’t tell you what to adjust yours to as each collector is unique.
mnagel
·3 years agoI suppose it is possible there are two different issues -- threadpool change requirements needed indicates internal resources are exhausted for checks. The original issue is still our constant problem -- static ID values cause sessions to become invalidated by firewalls when there is a disruption on the firewall path. A new session must be observed by the firewall to start letting traffic through again, which can only be done (currently) by restarting the collector since the code allocates those session ID values only one at startup. LM has been informed about this repeatedly, is aware of it and does nothing to fix it. The only thing I’ve seen is a doc note blaming particular firewalls, but it impacts pretty much any stateful inspection firewall. More and more folks use firewalls for internal segmentation and there are many cases where a remote collector is needed due to lack of resources to deploy a local collector.
MaddyM
·3 years ago@Joe Williams
what size were your collectors and what did you end up adjusting the threadpool too?
I’m new to LM and have been working on implementing it within my company the past few months. I’ve been facing this issue since December and have been going back and forth with LM Support. First, they had me do a fresh install, then they said there was a bug in v 33.x for the collectors and I needed to downgrade and the last step was to completely delete the collectors from the portal and add them again. At this point LM support claims they have done everything they can and have escalated to an internal ticket but I haven’t been able to receive any updates on the status of that ticket. I’m willing to make adjusts me to the threadpools/timeouts if needed to see if this resolved the issue for me.
Joe Williams
·3 years ago@mnagel For us it did resolve it. We had consistent collectors and devices that this happened with. After adjusting the threadpools, those collectors and devices didn’t show the issue.
mnagel
·3 years agoWe have observered similar issues with Ping failing via the collector application but not from the OS it self. A restart would resolve the issue for awhile. What we found was adjusting the threadpool for Ping to fix our problem. It seems the collector would max out of threads for ping and then just start failing. After adjusting the threadpool count we didn’t encounter this issue again.
Are you certain that fixed the issue? Because that would require a collector restart and that is what fixes the problem since it forces generation of new ID values for ICMP and SNMP “sessions”.
This problem has been going on for years and LM seems to have no plan to fix it. We routinely lose hours of data due to intermediate firewall session invalidation and I’ve seen only a glimmer of interest from folks at LM. The collector code needs to be updated to generate new ICMP and SNMP “sessions” for each check, or at least do so periodically (e.g., every 5-10 minutes) so this stops happening.
Joe Williams
·3 years agoWe have observered similar issues with Ping failing via the collector application but not from the OS it self. A restart would resolve the issue for awhile. What we found was adjusting the threadpool for Ping to fix our problem. It seems the collector would max out of threads for ping and then just start failing. After adjusting the threadpool count we didn’t encounter this issue again.
Michael Dieter
·3 years agoThis behavior seems like it may be related to, or overlap with, an issue I first observed in January 2020 (I think) and was never able to resolve.
Randomly, subsets of our Juniper switches (and only switches, no other devices) would trip alerts indicating 100% ping loss. It would usually auto-resolve after 60-90 minutes and never left evidence behind--that I could find--why the condition started or cleared up.
During the time the alerts were in effect, I had other non-collector sources of pings to the same switches that were not disrupted and I could ping back to the collectors involved from the switch command line. SSH, SNMP, other communication between collectors and switches showed no problem.
Of note, none of the traffic between collectors and switches traversed a firewall.
The real kicker to me was that I had never seen this behavior until I upgraded collectors to 29.003. If I rolled collectors back to 28.x, the issue did not occur. As soon as I pushed forward again to 29.x it started happening again. I opened a case with support and I spent a lot of tedious time trying to figure out where traffic was getting dropped to no avail; after several months I was not able to convince them to move from their “its something in your environment” stance. As much as I wanted an answer I simply could not afford to devote the time needed to sustain an investigation.
Ultimately I applied system.category “NoPing” to switches and moved on.
Mike Moniz
·3 years agoI looks like I’m seeing the same issues now with some customers where ping stops working until the collector is restarted. So add me to the list of affected people.
AustinC
·3 years agoI've pushed our CSM a good deal on infrequent SNMP failures, and I was able to hop on a Zoom call with their Devs and walk them through the exact behavior we were seeing (I keyed on this by virtue of alerts for SNMP host down, despite 'Poll Now' clearly showing a response), and I recently received some tacit confirmation that this 'bug' was, indeed, acknowledged. I wasn't aware of the exact cause of the problem, but was able to clearly demonstrate to them that this presents as a bug, clearly not a 'feature' request.
Obviously, no indication of timeline for fix, but this seemed like something that finally landed -- Hopefully this will lead to a fundamental fix. I'll be sure to highlight this thread to our CSM as well, in case that might help at all.
mnagel
·3 years agoI would not hold my breath. I pushed my CSM at the time on this issue back in 2018 and they refused to consider it a bug, but a feature request. I also brought that up recently with my current CSM and I got crickets. I dutifully followed through, but since the feature request "system" is nearly worthless, nothing has been done. I cannot begin to enumerate the number of embarrassing conversations with clients that start like "Why is LogicMonitor alarming about SNMP being down or a host being down when we can get to those devices just fine?" The workaround is time-intensive (manual collector restarts) and the repeated data loss is unforgivable. I don't know what possessed the developers to generate a fixed SNMP session ID or ICMP ID once when the collector starts rather than at each new get/walk or ping. It is the ultimate in false optimization and causes unending problems for any but the smallest simplest network. LM should be ashamed of letting this continuing to happen.
Allan_Hota
·4 years agoAny updates on this? ?
Rakzskull
·4 years agoAny further updates here?
I'm also facing the same issue for Palo Alto FW.
Vitor_Santos
·4 years agoWe've already faced this internally with some customers too (where having 1 collector per subnet is not feasible - due to the size, licensing, etc...).
If I'm not mistaken the OS is irrelevant to this issue (since it's the way LM is working currently).
Michael Rodrigues
·4 years agoStill working on this, hoping to have a better update soon.
mnagel
·4 years agoThank you for jumping in! I did not have any idea where to acquire a Sumo suit :).
Michael Rodrigues
·4 years agoCollector PM here. Quick glance at tickets suggests this ICMP firewall issue realization never made it to the collector team. I do recall the similar issue with SNMP.
I'll bring this up with the collector team tomorrow night. While best not to ping through a FW, if all we have to do to fix it is randomize the ICMP ID, that seems like a reasonable ask. I'm finding old TS tickets that appear to be this same issue, whether it was recognized at the time or not.
Sarah Terry is still here, by the way, she is Senior Director of Product these days.
LM User
·4 years agoHow about this: you come to Elevate in June and we'll get some fluffy sumo wrestling suits on and duke it out? haha. We'll make one of the sales guys expense it. We'll collapse in the end and realize we're agreeing with each other.
mnagel
·4 years agoYou can say it is semantics, but ICMP is connectionless and generally firewalls need to do inspection to identify sessions. For ICMP that is the ICMP ID that together with the src and dst address allow firewalls to allow an echo reply response after seeing the outgoing echo. An echo reply that does not match will be dropped since an unsolicited ICMP packet should not just be sent to targets. Because LM has this bug where it uses the same ICMP ID for all ping checks, it trips firewalls that do inspection. If you argue folks should not use firewalls internally, that is battling windmills -- it is very common and getting more so to limit lateral attacks. That each of our tickets has generated zero understanding and a punt to open a feature request is just sad. I am not going to get into the general abilities of our successive CSMs, but if you look at my first referenced you will see the way these things tend to end up.
mnagel
·4 years agoI have been told "Other ticket numbers for the ping/SNMP issue are 286866 and the latest - 337366." Generally we are told to open a FR each time.
LM User
·4 years agoWe're arguing semantics now, so i'll bow out.
As far as FR numbers, if you only put it here in the community, it likely didn't get entered into the system, so product didn't even know about it. If you spoke to your CSM about it, they would have put it into the system and good CSMs keep track of those entries. If you did it through the feedback system in the product, it would have made it into the system, but your CSM might not have seen it. Granted, the FR system needs a major overhaul. It's one of the big focuses of our upcoming focus on community (including a new hosting platform).
mnagel
·4 years agoIt is no longer legitimate traffic after the "session" is invalid -- this is common firewall behavior that would be avoided by using a fresh ICMP ID for each ping check, like everyone else does.
FR number? When did that start being a thing? I thought you just create them in the forum and cross your fingers someone sees them. If feedback, then no, those are usually one-way -- I know tickets are generated internally because one of our CSMs actually shared them with me to help prioritize, but usually they are invisible with no followup (with one exception historically for API issues when Sarah Terry was there). The ticket ID for the last time I tried to get help on this is 107847 (last updated 7/23/2018). The SNMP info above came from a more recent interaction by someone else on my team -- not sure of the ticket ID on that one.
LM User
·4 years agoI get and agree that LM is causing the case, but if the firewall is eventually blocking the traffic, the firewall blocking the legitimate traffic (that looks illegitimate). Do you know the FR number?
mnagel
·4 years agoFolks are not going to place collectors in every subnet and due to increased security concerns, there will be more and more situations where this will be an issue.
As far as "blocking legitimate traffic" that is not what is happening here (OP specifically said the firewall was wide open). It is allowing the traffic, but firewalls track sessions and due to bad programming, LM triggers firewalls to block traffic in some cases. For example, we had a remote location (all WAN sites transit firewalls, a very common architecture) that had suffered a power outage. Pings began failing because LM reuses the same ICMP ID forever and the original session established previously was no longer valid.
As I mentioned, I escalated this to our CSM in 2018 and got back "I get it, but you need to open a feature request". Since then, someone in LM has at least figured out this needs support for SNMP -- this is what we were provided and it works fairly well (that seems to only target SNMPv3, but we tend to use that when possible so it is OK).
We get frequent annoyed tickets from clients who are told by LM that a host is not responding to ping, which is trivially proved wrong by them. Our only solution when it happens is to restart the collector. You know, rather than LM fixing broken code.
LM User
·4 years agoThis is the recommended architecture.
If your firewalls are blocking legitimate business traffic, they need to not do that.
mnagel
·4 years agoThe world is passing LM by there -- most organizations these days are moving to internal compartmentalization, which means firewalls of some sort. We have generally seen it for smaller remote sites that have no desire or facilities for a local collector. I don't think it matters what the collector platform is since the checks are all the same (Java/Groovy). I tried a lot with our CSM back in 2018 and while they agreed in principal, it was considered a "feature request" and well, we know where those generally end up. I was also treated to circular logic (determined to be a problem with Palo Alto firewalls only, which is completely untrue, and evidence was an LM support page saying that Palo Altos have that problem).
Mike Moniz
·4 years agoInteresting, I haven't run into that yet myself. I typically have collectors located on the same network segment as the devices being monitored so not hitting firewalls, but some situations do go thru some. Is this specific to Windows or Linux Collectors?
mnagel
·4 years agoThis is an old bug. I tried (and keep trying) to get it fixed for years, but no luck so far. The problem is that any firewall keeps a session table. ping is not session-based like TCP, but firewalls still keep track of ICMP ID and use timers to invalidate sessions. Same for UDP (SNMP). The LM collector code is "lazy" and does not generate new session-equivalents for successive checks, so eventually the traffic is dropped by the firewall because it is matched to an invalid session. You can workaround this by restarting the collector. For SNMP they have recently added some knobs in the collector config to help, but for ICMP it is still messed up.
Mike Moniz
·4 years agoTry running a ping from the collector server to "VIPEIEEMP01" and see if that both results to an IP address and if ping works. Was this working at some point then stopped working or never worked?