ACK alerts follow the same escalation chain as the original alerts, this is awful for an after hours team. We would like ACK alerts to be sent only via email and not follow the same path as the critical outage alert. Otherwise, it results in everyone being woken up to be told that "everything is fine".
I think this is handled/set in the Alert rule dialog (see screenlshot)
OK, further thought and discussion means that we will separate the Suppress Alert Clear from Suppress Alert action notifications (or whatever we call it in the UI, for Acks and SDTs.) There is a valid case (single LM responder to a chain) where they want alert clears, but not to be told of their own SDTs. This should be released before end of March.
Thanks for the detailed reply Steve. That all sounds very logical to me and the team here would be quite happy with that. When do you expect to have that short term change deployed?
So, firstly, apologies that this has taken so long to get to...What I describe above (allowing filters per rule, for Ack, clear, SDT, etc) is still planned, but due to various dependencies on things changing in the UI, still a ways out. What I would like to do in the short term is to simply make the Suppress Alert Clear option also suppress the Ack and SDT messages. So if this is selected (and probably renamed something like Suppress alert status updates) an alert will still escalate (through multiple stages); an increase in severity will still be sent; but messages regarding Acks, SDTs and clears will not. To my mind these are logically similar. If you do not want a notification of an alert ending (so you no longer have to deal with it), you probably also do not want to know that someone put it in SDT, or Acked, so you also do not have to deal with it. So this does not address people that use the first stage for ticketing systems, but should cover most other use cases. Feedback?
Acknowledgement messages are not dependent on alert clear setting. We'll treat this the same as Alert Cleared messages. The defaults will preserve existing behavior.
I believe those of us that have inserted our own processes in the alert flow using a webhook would rather this remain the same. Acks and Resolves do not cause anyone to be disturbed because we have complete control over the alert flow. I use a webhook to maintain a standalone dashboard that relies on receiving Acks and Resolves to maintain status of incidents on the display.
I wonder if an ACK depends on the Enable Alert Cleared Notifications under Settings. Does unsetting it stop the ACKs from sending Notifications?
