Forum Discussion

Sam_Gendler's avatar
7 years ago

On an EC2 instance in an ECS cluster, SNMP filesystem monitoring alerts on docker filesystems

SNMP based monitoring of an EC2 instance that is a member of an ECS cluster picks up filesystems at  /var/lib/docker/containers/<container_id> which go away once the docker container exits.  This causes logicmonitor to alert on the 'StorageNotAccessible' data point of the 'Filesystem Capacity' data source.  Does anyone have a good fix for this that doesn't just turn off monitoring of those filesystems?  And do I care, anyway?  They seem to be some kind of overlay filesystem that is just content on a normal filesystem that is also being monitored, so I can probably just exempt /var/lib/docker/containers/* from being monitored at all, though I'd love to have it do that automatically just for hosts that are AWS/EC2 instances.  I'll eventually figure out how to do that via the documentation, but it may be faster to ask here.

Incidentally, there seem to be 2 kinds of mounts with that path pattern, one shared memory segment and the other overlay.  It looks like logicmonitor ignores shm filesystems.  Perhaps it could ignore overlay by default, too?