> ## Documentation Index
> Fetch the complete documentation index at: https://docs.appsignal.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tune your alerts

> Reduce alert fatigue by tuning thresholds, warm-up, cooldown, and scope so every notification is worth reading.

An alert is only useful if someone acts on it. When a team receives more notifications than it can act on, it starts ignoring all of them, including the one that mattered. That is alert fatigue, and it is usually a configuration problem rather than a monitoring problem.

This page covers how to tune an alert that already exists. If you are deciding what to alert on in the first place, start with [alert recipes](/alerting/recipes). If your problem is the whole set rather than one trigger, [reduce alerting noise](/alerting/reduce-noise) works through the changes that remove the most notifications first.

## Signs your alerting needs tuning

* A channel receives alerts that nobody investigates.
* The same alert opens and closes several times an hour.
* One incident produces many notifications from many hosts or tag combinations.
* On-call receives alerts that cannot be acted on until business hours.
* Someone has disabled notifications for the alert channel.

Each of these maps to a specific setting described in the following sections.

## Threshold, warm-up, and cooldown

Every [anomaly detection trigger](/anomaly-detection) has three settings that control when alerts open and close. Tuning almost always means adjusting one of them.

| Setting   | Question it answers                                        | Symptom when it is wrong                   |
| --------- | ---------------------------------------------------------- | ------------------------------------------ |
| Threshold | How bad does it have to be?                                | Alerts open during normal operation        |
| Warm-up   | How long does it have to stay bad?                         | Alerts open on single-minute spikes        |
| Cooldown  | How long does it have to be better before we call it over? | The same alert opens and closes repeatedly |

### Set thresholds from metric history

Before choosing a threshold, graph the metric over at least two weeks and look at its normal range, including its daily and weekly peaks. A threshold set below a routine Monday morning peak will notify you every Monday morning.

A useful starting point is the value the metric reaches during a busy but healthy period, plus enough margin that normal variation does not cross it. If you cannot describe what you would do when the threshold is crossed, the threshold is probably not the right one.

### Use warm-up to filter out spikes

Warm-up is how long AppSignal waits before opening an alert. During that wait, the metric must stay above or below the threshold, depending on the trigger. With a warm-up of `0`, AppSignal can open an alert as soon as the metric crosses the threshold.

Use warm-up to avoid alerts for short metric spikes, such as one slow deploy, one garbage collection pause, or one restarting host. Start with zero to two minutes for most production alerts. Use a longer warm-up only when the metric often spikes and recovers quickly.

Set warm-up based on how much the metric changes minute to minute. A metric that changes quickly often needs a warm-up. A metric that changes gradually often needs little or no warm-up.

| Signal                          | Starting warm-up  | Reasoning                                                                                   |
| ------------------------------- | ----------------- | ------------------------------------------------------------------------------------------- |
| Error rate                      | 1–2 minutes       | Changes quickly. Filters out deploy-time spikes                                             |
| Response time percentiles       | 1–3 minutes       | Latency can change minute to minute                                                         |
| Throughput, in either direction | 1–2 minutes       | Short increases and decreases are common and usually recover                                |
| Queue depth or backlog          | 1–2 minutes       | Can grow quickly as work arrives                                                            |
| CPU and load average            | 1–3 minutes       | Short increases are normal; sustained high usage is not                                     |
| Memory and swap usage           | 0–2 minutes       | These move smoothly, so crossing the threshold is already meaningful                        |
| Disk usage                      | 10–60 minutes     | Usually changes slowly, so a longer warm-up can reduce noise without hiding a fast incident |
| Uptime monitors                 | At least 1 minute | Ignores brief network problems between regions                                              |

Useful warm-ups are usually short. Beyond about 10 minutes you are mostly delaying the notification rather than improving it. If you need 30 or 60 minutes to reduce notifications, review the threshold or scope instead.

<Warning>
  Do not use a long warm-up to check whether a scheduled process is running. A trigger only evaluates data that arrives, so an absent job may never open an alert. AppSignal [recommends against](/anomaly-detection#warm-up) using anomaly detection this way. Use [check-ins](/check-ins) for cron jobs, workers, and heartbeats.
</Warning>

### Use cooldown when alerts open and close repeatedly

Cooldown is the number of minutes the metric must stay recovered before the alert closes and a new alert can open. For "more than" triggers, this means the metric stays below the threshold. For "less than" triggers, this means the metric stays above the threshold.

Without cooldown, a metric that repeatedly crosses its threshold produces a stream of open and close notifications for one incident.

A short cooldown turns that stream into a single alert. Set it slightly longer than the period where the metric usually moves above and below the threshold.

Cooldown is a targeted fix rather than a default. Most triggers never need one, because most metrics do not sit exactly on their threshold. Leave it at `0` until you see a trigger open and close repeatedly for what was clearly one incident, then set it from how long those gaps actually were.

## Scope: one trigger can open many alerts

This is the most common cause of many notifications from one incident.

A trigger does not open one alert. It opens one alert per matching series. For host triggers, that means one alert per host. The hostname field accepts a wildcard, and its default value of `*` matches every host reporting to the app. A fleet of 30 hosts crossing a CPU threshold at the same time produces 30 alerts.

Narrow the scope so the notification says exactly what is affected:

* Set a specific hostname, or a prefix such as `web-*`, instead of leaving the default `*`.
* Add [tags](/anomaly-detection#tags) so the trigger only watches the series you care about, such as `mountpoint=/` for disk usage or `region=eu` for a custom metric.
* Scope error rate and response time triggers to a single [namespace](/application/namespaces), so `web`, `background`, and `admin` can have different thresholds and different notifiers.

When you add tags, AppSignal warns you if the tag combination has not been seen in the last hour. That warning usually means the trigger will never match data. Use one of the tag-key combinations listed in the trigger form, and set each value to an exact value or a wildcard.

Uptime monitors behave the same way. Each of the four regions is tracked separately, so one monitor can open four alerts for a single outage, and one alert when a single region cannot reach an endpoint that is otherwise healthy.

## Choose a metric that reflects user impact

Changing trigger settings cannot fix a trigger that watches the wrong metric.

**Prefer rates over counts.** Error count crosses a fixed threshold whenever traffic grows, so it notifies you on your busiest days regardless of whether anything is wrong. Error rate stays stable as traffic changes. Use counts only for metrics where any occurrence matters, such as errors in a payment flow.

**Prefer percentiles over means.** A mean response time hides a slow experience for part of your traffic. If 5% of requests take 8 seconds, the mean may barely move. Alert on p90 or p95 instead.

**Prefer user-visible symptoms over causes.** Alert on errors, latency, traffic, or failed business actions, and use dashboards to find the cause. High CPU usage is not itself a problem if response times are fine. This keeps the number of alerts proportional to the number of real incidents.

**Set missing data to zero deliberately.** A trigger assumes a data point arrives every minute. For metrics that only report when something happens, such as a counter, enable **treat missing datapoints as 0** so the alert closes when activity stops. Leave it off when a gap in data means "no information" rather than "zero".

## Route by severity, not by habit

Sending everything to one channel causes alert fatigue. Each trigger can use its own notifier, so use that to separate urgent alerts from alerts that can wait.

| Urgency                         | Example                                                        | Destination                                                                                                             |
| ------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Someone must act now            | Error rate above threshold in `web`, health check failing      | On-call tool such as [PagerDuty](/application/integrations/pagerduty) or [Opsgenie](/application/integrations/opsgenie) |
| Someone should look today       | Response time degradation, queue backlog growing               | Team chat such as [Slack](/application/integrations/slack)                                                              |
| Someone should notice this week | Disk usage approaching full capacity, SSL certificate expiring | Email or a low-traffic chat channel                                                                                     |
| No notification needed          | Known third-party errors already being tracked                 | No notifier for trigger alerts, or **Never Notify** for incident and log notifications                                  |

Give staging its own channel or leave its triggers without a notifier. Staging alerts arriving in a production channel are one of the fastest ways to teach a team to ignore that channel.

Severity is not the only useful split. Once you have more than a handful of triggers, separate chat channels per domain, such as one for application errors and one for infrastructure, keep each channel small enough that the people who watch it recognize when something is unusual. A single channel receiving everything makes important notifications easier to miss.

### Route incidents and triggers differently

Routing has two layers, and they behave differently.

For anomaly detection triggers, choose the notifiers on the trigger itself. Removing all notifiers leaves the trigger active and still records alerts, but sends no notifications.

For error and performance incidents, set [notification defaults per namespace](/application/notification-settings#organization-and-app-namespace-defaults) rather than configuring each incident. An `admin` namespace set to [**Never Notify**](/application/notification-settings#never-notify) removes a whole category of low-value notifications at once. Namespace defaults are also where you route `web`, `background`, and `admin` incidents to different destinations.

Notifiers are configured once for the organization and can be shared across apps, so a notifier used by many apps needs care. Check where a notifier is used before editing it.

## Make the notification explain itself

Alert fatigue is not only about how many notifications arrive. It is also about how much work each one takes to understand. An alert that says a number crossed a threshold forces the responder to investigate what that threshold means. An alert that explains itself can be acted on immediately, or correctly ignored.

Three fields provide this context, and all three are worth filling in on every trigger.

**Name.** Say what is wrong, not which metric moved. A trigger list where many entries are called **Custom metric** cannot be reviewed or triaged.

**Description.** Treat it as a one-line runbook. State what the condition means and what the responder should do about it. These are the two questions a responder has to answer, and the description is the best place to answer them:

* "This host is swapping, so every request it serves is slower. Check what is using the memory and notify the operations channel."
* "Less than 100 MB of free memory. Check what is using it and add capacity before the kernel starts killing processes."
* "This service has a known memory leak. Memory grows until the kernel kills the process, so restart it and only escalate if it returns within the hour."

The third example is worth noticing. Documenting a known cause on the trigger stops the same investigation being repeated by whoever is on call this time, and it tells the responder when not to act.

**Dashboard.** Each trigger can link a dashboard, and the link is included in the notification. Link to the dashboard that shows the metric in context alongside related metrics. This gives the responder a starting point for investigation.

## What you cannot tune, and what should not alert

Two things about trigger alerts cannot be tuned, and one category of metric should not alert in the first place.

**Frequency options apply to incidents and log triggers, not to anomaly detection trigger alerts.** Error and performance notifications use [incident notification options](/application/notification-settings#notification-options), such as **Every Occurrence**, **First in Deploy**, **First After Close**, and **Never Notify**, set per namespace in [notification defaults](/application/notification-settings#organization-and-app-namespace-defaults). Log triggers have similar threshold options. Anomaly detection trigger alerts follow the alert state machine instead.

**Only email repeats while an anomaly alert stays open.** Email notifiers have a **Reminder interval** of 15 minutes, 30 minutes, or 1 hour. Slack, PagerDuty, Opsgenie, and webhooks have no equivalent.

**What to do instead.** To stop a trigger notifying anyone, remove its notifiers, or archive it. If an open alert must keep sending reminders until someone acts, pair email with an on-call tool that has its own escalation policy. If a long-running alert is filling a chat channel, the cause is repeated opening and closing rather than reminders, so the fix is a longer [cooldown](#use-cooldown-when-alerts-open-and-close-repeatedly). For other cases, decide whether the condition should send a notification at all.

* Alert on an error rate above 5% in `web` for three minutes, not on every individual exception, however rare.
* Alert on p95 response time for checkout doubling, not on a mean that moved from 180 ms to 200 ms.
* Alert on throughput at zero when it is normally 400 req/min, not on traffic halving overnight, as it does every night.
* Alert on a disk at 85% and gaining a point an hour, not on CPU touching 100% for one minute during a deploy.
* Alert on a nightly invoicing job that never reported, not on the same job starting two minutes late.
* Alert on a payment provider timing out 50 times in an hour, not on one request that timed out and succeeded on retry.

If a metric is interesting but not actionable, put it on a [dashboard](/metrics/dashboards) instead of sending notifications for it.

## Review triggers regularly

Alert configuration becomes outdated. Triggers outlive the incidents that motivated them, thresholds set for last year's traffic no longer fit, and hosts named in a trigger get replaced.

Review your triggers roughly every quarter and ask, for each one:

1. **Has it opened an alert in the last 90 days?** If not, open it and check the chart preview. No data, or a hostname field warning that the host was not found, means it is watching something that no longer exists. Renamed or replaced hosts are the usual cause, and the trigger does not notify you when it matches nothing.
2. **Did anyone act on it?** An alert nobody acts on should be retuned, rerouted, or archived.
3. **Did it open many alerts for one incident?** Narrow its scope.
4. **Is its notifier still the right one?** Sending non-urgent problems to on-call is a main source of alert fatigue.
5. **Does it [explain itself](#make-the-notification-explain-itself)?** A trigger with no name and no description cannot be triaged, because nobody can tell what it does without reconstructing it.
6. **Is it a duplicate?** Two triggers with the same metric, threshold, and notifier send two notifications for one problem.
7. **Does anyone still own it?** Triggers created by people who have left are the most likely to be outdated and the least likely to be reviewed.

<Note>
  Review triggers with no notifier before removing them. Some are deliberately configured without notifications but are still useful on a dashboard or in the alert history. Others had their notifiers removed during an incident and were never restored, which means a metric you believe is covered is not.
</Note>

You can list every configured trigger with the [AppSignal CLI](/cli/triggers) or through the [MCP server](/mcp-server) to review them without opening each one in the interface.

## Recap: tuning one trigger that sends too many notifications

This section introduces nothing new. It puts the previous sections in order against one example: a trigger on host CPU usage above 80%, with no warm-up, no cooldown, and the default hostname of `*`, sending several on-call notifications a night when there is no user-visible issue.

1. [Check the metric.](#choose-a-metric-that-reflects-user-impact) CPU usage is a cause, not a user-visible symptom, and response time and error rate are both healthy during the CPU spikes.
2. [Narrow the scope.](#scope-one-trigger-can-open-many-alerts) The spikes come from background workers, so the hostname becomes `web-*`.
3. [Set the threshold from history.](#set-thresholds-from-metric-history) Two weeks of data show web hosts routinely reaching 85% at evening peak, so the threshold moves to 95%.
4. [Add a warm-up.](#use-warm-up-to-filter-out-spikes) Two minutes, because a short CPU spike is normal and should not notify on-call.
5. [Add a cooldown.](#use-cooldown-when-alerts-open-and-close-repeatedly) Set a short cooldown, so a host near the threshold produces one alert rather than six.
6. [Reroute it.](#route-by-severity-not-by-habit) Sustained CPU saturation is a capacity problem, not an outage, so it goes to team chat and the on-call notifier is removed.
7. **Add the alert that was missing.** The team needed an alert for user impact, so they add a p95 response time trigger on the `web` namespace, routed to on-call. [Alerting in production](/alerting/production#check-your-coverage) covers finding the rest.

The result is fewer notifications, and the ones that remain describe something a person can act on.

## Next steps

* Choose what to monitor with [alert recipes](/alerting/recipes).
* Check setup order and coverage with [alerting in production](/alerting/production).
* Review how [warm-up and cooldown](/anomaly-detection#warm-up-and-cooldown) work in detail.
* Configure [notification settings](/application/notification-settings) for error and performance incidents.
* Set up [check-ins](/check-ins) for scheduled work instead of using long warm-ups.