Ping Datasource periodic failure after Collector upgrade: any ideas?

I'm with you on the dynamic filtering wavelength, but its not being enforced by FW or IPS.....there is neither between collectors and the switches at L2 or L3.   Another possibility would be a failure of LACP load-balancing hashing between 2 devices using link aggregation between them, resulting in some frames disappearing into a black hole.  Juniper's default mechanism relies on L2 source & destination.  Where this idea runs into trouble is that L2 src & dst are static (unless they are not, but this requires more effort to run down and there are no indicators of concurrent issues that would be expected with fluctuating L2 src or dst); frames disappearing into a black hole would be doing so independent of tcp/udp/ip/icmp payload and it seems too unlikely that icmp is the only one that ever vanishes.  Still, this thread could be investigated further but of course at a high cost in terms of time & effort.  

Next thought was that CoS queues are starving out icmp at some point between collector and switch.  We rely on Juniper CoS default behavior, however the end to end data path does traverse multiple different switch and router families and the default treatment is not identical in each. So another interesting thread to pursue but one with an even worse time/effort cost profile.  On the other hand, possibly the biggest of many problems with this idea is that the end to end CoS treatment is static too (we haven't ripped and replaced HW, and the respective Juniper OS versions haven't changed)  so believing this is the cause requires me to accept that it is purely co-incidental that the issue never occurred in the 5+ years and trillions of pings that took place up to and including collector v28.005

Which brings me full-circle: a number of competing possibilities each of which is plausible but all of which come with a high price tag that I have to pay by myself.  My Venmo balance checkbook balance (don't laugh, I actually do still have a paper checkbook) is pretty good but I'm not blowing it on this.  This is not a rabbit hole, its a worm-hole; I'm going to have to tolerate that I have to go around it right now.