Forum Discussion

Anmolk17's avatar
Anmolk17
Icon for Neophyte rankNeophyte
24 days ago

Clear alert on No data

Hi Folks,

I’m running into an issue with alerts for event-driven services (DynamoDB, AWS Glue, etc.) where data points exist only briefly and are otherwise “No data.” When an alert fires because a threshold is breached, it never clears because our monitoring tool (LM) has no way to mark alerts as cleared when the state becomes “No data.”

One workaround is to create a lower-severity alert for “No data,” but that would leave a warning-level alert present almost all the time, which isn’t ideal.

Has anyone faced this before? If you solved it, could you share how you handled alert clearing or suppression for no-data conditions?

Best Regards

AK

2 Replies

  • Mike_Moniz's avatar
    Mike_Moniz
    Icon for Professor rankProfessor
    24 days ago

    If this is specific to Cloud monitoring, they should be accounting for this. They don't alert on No Data but instead use Complex Datapoints with statements like "if(un(ReadsPerSecond),0..." so that if the value is No Data (aka unknown) it saves it as a 0. DataSource AWS_DynamoDB is an example of this. If you're not seeing that, you may need to upgrade your modules.

    In a general case, when you have a real event-based item to check, where there is no real end time, EventSources may be better option to use. It will cause an alert when the issue is detected and automatically clear after x minutes.

  • Justin_Lanoue's avatar
    Justin_Lanoue
    Icon for Advisor rankAdvisor
    18 days ago

    This also happens with the SSL_Certificates module.

    We are alerting on DaysRemaining_NotSelfSigned but if someone were to renew the certificate randomly as a self-signed certificate, then it would never clear because the datapoint goes to DaysRemaining_SelfSigned and DaysRemaining_NotSelfSigned becomes "no data" and never clears.

    Theres also a bug where it will randomly flap to thinking its self-signed and then flap back to not self-signed which gives the same issue/result.