Azure Autobalanced Collectors - 5 minute windows of ping failures
Here's a weird one. We have a customer with three auto-balanced collectors in Azure. They see the following pattern for ping loss (0 is good):
- 12:34: 0
- 12:35: 0
- 12:36: 100
- 12:37: 100
- 12:38: 100
- 12:39: 100
- 12:40: 100
- 12:41: 0
- 12:42: 0
- 12:43: 0
So regularly (6 or 7 times a day for ALL Resources), there are 5 minute windows where the ping loss is 100%. PRTG (admittedly on a different Azure subnet) is showing no issues whatsoever.
So... what's to blame?
David Bond
Posted 11 months ago·Last reply 11 months ago
9 comments
Mike Moniz
·11 months agoI can't find the discussion but I recall there was some issue brought up before about the way that Java/LM does ping would not not increase some packet id # which would cause some routers/firewalls to block them eventually, and restarting the Collector would clear it. There are some situations where I've had to restart a collector to fix a ping issue. But it's always been that ping stops working completely until the collector restart and not just stop for a few minutes then fix itself.
Is there anything special about the routing between the collector and a device that stops pinging? Does it happen even if you are pining 127.0.0.1 (like pinging the collector itself)? Are the devices affected all on the same region/network/subnet? Is special routing table or 3rd party router/firewall between them? Perhaps worth running Wireshark or network watcher during the issue (if you can catch it live) and see what the traffic looks like. What does the LM wrapper/sbproxy log show if anything? etc.
Joe Williams
·11 months agoWow, if true, you just solved an issue I encountered like 6 years ago. We would consistently lose ping via the collector, but we could ping from a server 1 digit away on IPs, same subnet, same vswitch, etc.
Mike Moniz
·11 months agoFound some references in these forums if it helps:
Ping Failing from collector to device and back? | LogicMonitor - 10447
SQL Server Ping critical alert shows up and clears | LogicMonitor - 16101
Ping Datasource periodic failure after Collector upgrade: any ideas? | LogicMonitor - 9251
David Bond
OP11 months agoWe've moved PRTG to the same subnet and again, PRTG does not suffer from this issue. It seems to be isolated to LogicMonitor.
Patrick Anderer
·11 months agoWhen you look at the Collector device that is monitoring the devices where the PING loss is happening, I would recommend checking the Collector Data Collecting Tasks datasource to see if there is some sort of cyclical overload of those tasks happening. Specifically I would look at the graphs for Unavailable Thread Scheduling and Queue datapoints before, during and after the Ping loss time periods. It may indicate that the collector ABCG is not keeping up with Ping specifically, which is something you may be able to tune around.
Joe Williams
·11 months agoIs it just Ping? Or other datasources as well?
Have you identified if its a random collector or always the same collector?
Is there a pattern with the times?
David Bond
OP11 months agoJust ping, and no apparant pattern. I find it difficult to believe that LogicMonitor is at fault, but PRTG (also running on the same Azure subnet) is not affected. It's such as strange pattern. Why would separate sets of ten pings, one minute apart ALL be affected and ALWAYS for 5 minutes. If I didn't know better, I would suspect some weird ARP or routing issue in the Collector's Windows Server OS, but that doesn't seem right either.
Utterly baffled.