Signs your alerting needs tuning
- A channel receives alerts that nobody investigates.
- The same alert opens and closes several times an hour.
- One incident produces many notifications from many hosts or tag combinations.
- On-call receives alerts that cannot be acted on until business hours.
- Someone has disabled notifications for the alert channel.
The three settings
Every anomaly detection trigger has three settings that control when alerts open and close. Tuning almost always means adjusting one of them.Set thresholds from metric history
Before choosing a threshold, graph the metric over at least two weeks and look at its normal range, including its daily and weekly peaks. A threshold set below a routine Monday morning peak will notify you every Monday morning. A useful starting point is the value the metric reaches during a busy but healthy period, plus enough margin that normal variation does not cross it. If you cannot describe what you would do when the threshold is crossed, the threshold is probably not the right one.Use warm-up to filter out spikes
Warm-up is how long AppSignal waits before opening an alert. During that wait, the metric must stay above or below the threshold, depending on the trigger. With a warm-up of0, AppSignal can open an alert as soon as the metric crosses the threshold.
Use warm-up to avoid alerts for short metric spikes, such as one slow deploy, one garbage collection pause, or one restarting host. Start with zero to two minutes for most production alerts. Use a longer warm-up only when the metric often spikes and recovers quickly.
Set warm-up based on how much the metric changes minute to minute. A metric that changes quickly often needs a warm-up. A metric that changes gradually often needs little or no warm-up.
Useful warm-ups are usually short. Beyond about 10 minutes you are mostly delaying the notification rather than improving it. If you need 30 or 60 minutes to reduce notifications, review the threshold or scope instead.
Use cooldown when alerts open and close repeatedly
Cooldown is the number of minutes the metric must stay recovered before the alert closes and a new alert can open. For “more than” triggers, this means the metric stays below the threshold. For “less than” triggers, this means the metric stays above the threshold. Without cooldown, a metric that repeatedly crosses its threshold produces a stream of open and close notifications for one incident. A short cooldown turns that stream into a single alert. Set it slightly longer than the period where the metric usually moves above and below the threshold. Cooldown is a targeted fix rather than a default. Most triggers never need one, because most metrics do not sit exactly on their threshold. Leave it at0 until you see a trigger open and close repeatedly for what was clearly one incident, then set it from how long those gaps actually were.
Scope: one trigger can open many alerts
This is the most common cause of many notifications from one incident. A trigger does not open one alert. It opens one alert per matching series. For host triggers, that means one alert per host. The hostname field accepts a wildcard, and its default value of* matches every host reporting to the app. A fleet of 30 hosts crossing a CPU threshold at the same time produces 30 alerts.
Narrow the scope so the notification says exactly what is affected:
- Set a specific hostname, or a prefix such as
web-*, instead of leaving the default*. - Add tags so the trigger only watches the series you care about, such as
mountpoint=/for disk usage orregion=eufor a custom metric. - Scope error rate and response time triggers to a single namespace, so
web,background, andadmincan have different thresholds and different notifiers.
Choose a metric that reflects user impact
Changing trigger settings cannot fix a trigger that watches the wrong metric. Prefer rates over counts. Error count crosses a fixed threshold whenever traffic grows, so it notifies you on your busiest days regardless of whether anything is wrong. Error rate stays stable as traffic changes. Use counts only for metrics where any occurrence matters, such as errors in a payment flow. Prefer percentiles over means. A mean response time hides a slow experience for part of your traffic. If 5% of requests take 8 seconds, the mean may barely move. Alert on p90 or p95 instead. Prefer user-visible symptoms over causes. Alert on errors, latency, traffic, or failed business actions, and use dashboards to find the cause. High CPU usage is not itself a problem if response times are fine. This keeps the number of alerts proportional to the number of real incidents. Set missing data to zero deliberately. A trigger assumes a data point arrives every minute. For metrics that only report when something happens, such as a counter, enable treat missing datapoints as 0 so the alert closes when activity stops. Leave it off when a gap in data means “no information” rather than “zero”.Route by severity, not by habit
Sending everything to one channel causes alert fatigue. Each trigger can use its own notifier, so use that to separate urgent alerts from alerts that can wait.
Give staging its own channel or leave its triggers without a notifier. Staging alerts arriving in a production channel are one of the fastest ways to teach a team to ignore that channel.
Severity is not the only useful split. Once you have more than a handful of triggers, separate chat channels per domain, such as one for application errors and one for infrastructure, keep each channel small enough that the people who watch it recognize when something is unusual. A single channel receiving everything makes important notifications easier to miss.
Route incidents and triggers differently
Routing has two layers, and they behave differently. For anomaly detection triggers, choose the notifiers on the trigger itself. Removing all notifiers leaves the trigger active and still records alerts, but sends no notifications. For error and performance incidents, set notification defaults per namespace rather than configuring each incident. Anadmin namespace set to Never Notify removes a whole category of low-value notifications at once. Namespace defaults are also where you route web, background, and admin incidents to different destinations.
Notifiers are configured once for the organization and can be shared across apps, so a notifier used by many apps needs care. Check where a notifier is used before editing it.
Make the notification explain itself
Alert fatigue is not only about how many notifications arrive. It is also about how much work each one takes to understand. An alert that says a number crossed a threshold forces the responder to investigate what that threshold means. An alert that explains itself can be acted on immediately, or correctly ignored. Three fields provide this context, and all three are worth filling in on every trigger. Name. Say what is wrong, not which metric moved. A trigger list where many entries are called Custom metric cannot be reviewed or triaged. Description. Treat it as a one-line runbook. State what the condition means and what the responder should do about it. These are the two questions a responder has to answer, and the description is the best place to answer them:- “This host is swapping, so every request it serves is slower. Check what is using the memory and notify the operations channel.”
- “Less than 100 MB of free memory. Check what is using it and add capacity before the kernel starts killing processes.”
- “This service has a known memory leak. Memory grows until the kernel kills the process, so restart it and only escalate if it returns within the hour.”
What you cannot tune, and what should not alert
Two things about trigger alerts cannot be tuned, and one category of metric should not alert in the first place. Frequency options apply to incidents and log triggers, not to anomaly detection trigger alerts. Error and performance notifications use incident notification options, such as Every Occurrence, First in Deploy, First After Close, and Never Notify, set per namespace in notification defaults. Log triggers have similar threshold options. Anomaly detection trigger alerts follow the alert state machine instead. Only email repeats while an anomaly alert stays open. Email notifiers have a Reminder interval of 15 minutes, 30 minutes, or 1 hour. Slack, PagerDuty, Opsgenie, and webhooks have no equivalent. What to do instead. To stop a trigger notifying anyone, remove its notifiers, or archive it. If an open alert must keep sending reminders until someone acts, pair email with an on-call tool that has its own escalation policy. If a long-running alert is filling a chat channel, the cause is repeated opening and closing rather than reminders, so the fix is a longer cooldown. For other cases, decide whether the condition should send a notification at all.
If a metric is interesting but not actionable, put it on a dashboard instead of sending notifications for it.
Review triggers regularly
Alert configuration becomes outdated. Triggers outlive the incidents that motivated them, thresholds set for last year’s traffic no longer fit, and hosts named in a trigger get replaced. Review your triggers roughly every quarter and ask, for each one:- Has it opened an alert in the last 90 days? If not, open it and check the chart preview. No data, or a hostname field warning that the host was not found, means it is watching something that no longer exists. Renamed or replaced hosts are the usual cause, and the trigger does not notify you when it matches nothing.
- Did anyone act on it? An alert nobody acts on should be retuned, rerouted, or archived.
- Did it open many alerts for one incident? Narrow its scope.
- Is its notifier still the right one? Sending non-urgent problems to on-call is a main source of alert fatigue.
- Does it explain itself? A trigger with no name and no description cannot be triaged, because nobody can tell what it does without reconstructing it.
- Is it a duplicate? Two triggers with the same metric, threshold, and notifier send two notifications for one problem.
- Does anyone still own it? Triggers created by people who have left are the most likely to be outdated and the least likely to be reviewed.
Review triggers with no notifier before removing them. Some are deliberately configured without notifications but are still useful on a dashboard or in the alert history. Others had their notifiers removed during an incident and were never restored, which means a metric you believe is covered is not.
Recap: tuning one trigger that sends too many notifications
This section introduces nothing new. It puts the previous sections in order against one example: a trigger on host CPU usage above 80%, with no warm-up, no cooldown, and the default hostname of*, sending several on-call notifications a night when there is no user-visible issue.
- Check the metric. CPU usage is a cause, not a user-visible symptom, and response time and error rate are both healthy during the CPU spikes.
- Narrow the scope. The spikes come from background workers, so the hostname becomes
web-*. - Set the threshold from history. Two weeks of data show web hosts routinely reaching 85% at evening peak, so the threshold moves to 95%.
- Add a warm-up. Two minutes, because a short CPU spike is normal and should not notify on-call.
- Add a cooldown. Set a short cooldown, so a host near the threshold produces one alert rather than six.
- Reroute it. Sustained CPU saturation is a capacity problem, not an outage, so it goes to team chat and the on-call notifier is removed.
- Add the alert that was missing. The team needed an alert for user impact, so they add a p95 response time trigger on the
webnamespace, routed to on-call. Alerting in production covers finding the rest.
Next steps
- Choose what to monitor with alert recipes.
- Check setup order and coverage with alerting in production.
- Review how warm-up and cooldown work in detail.
- Configure notification settings for error and performance incidents.
- Set up check-ins for scheduled work instead of using long warm-ups.