Fix Limitation in clearing of stateful LM Logs alerts (flapping)
Summary: Matt Whitney, also known as Matt Whitney, addresses a limitation in stateful LM Logs alerts, where alerts cannot be cleared successfully if the clearing condition occurs too quickly after the alerting condition. This issue arises from a brief or intermittent condition that clears within seconds, causing orphaned alerts. LM Support confirmed this as a limitation. Matt suggests implementing a delay in processing the clearing condition to allow the alert to clear properly. Additionally, he proposes an option to delay the triggering of alerts for conditions that might clear within a short period, such as 30 seconds, to prevent issues with transient conditions.
Currently there is a limitation with stateful LM Logs alerts which prevents successful clearing of alerts if the "clearing condition" comes in too quickly after the "alerting condition".
This causes orphaned alerts for us if there is a brief/intermittent condition that clears within a few seconds.
We worked with LM Support and they confirmed this is a limitation. If a stateful alert triggers and the clearing condition is ingested while LM is still processing the original alert, then the alert will not clear. Because there is no alert object created yet when the clearing condition is ingested/processed.
One suggestion we had is to put a delay on the logic for processing the clearing of stateful alerts. A delay of a few seconds would be enough time so that LM can process the original alert before trying to clear it, even if the condition is flapping and the clear comes in less than a second after the trigger message.
Either way, LM needs to have some method in place so that flapping/transient conditions do not cause problems with stateful alerting.
Patrick Sites
·5 months agoThis has been placed in the backlog for discussions on approach. No commit at this time as more research will be needed.
Matt Whitney
OP5 months agoAnother suggestion to help with this is giving us the option of putting a delay on the triggering of stateful alerts. For example, if an alerting condition is detected, only trigger the alert if there is no clearing condition detected within 30 seconds.