Reduce alerting noise

A new application can send too many alerts. Thresholds are set before anyone knows the normal metric range, triggers watch every host and every tag value, and every notification goes to one channel. Within a few weeks the team starts ignoring that channel, and shortly after that somebody disables notifications for it. This page is the tuning pass that fixes it. The steps are ordered by how much noise each one removes, so working from the top down gets most of the benefit early. For the mechanics of each setting, see tune your alerts. The two steps that usually remove the most notifications are using exact tag filters instead of broad wildcards and keeping tag values to a small fixed set. Both prevent one trigger from opening alerts for values you did not mean to monitor.

Decide what a notification costs before you route it

Most notification noise comes from alerts being sent to a channel that implies more urgency than they have. Choose the destination based on how quickly someone needs to act.

Required response time	Destination
Customers lose money or access tonight	On-call, such as PagerDuty or Opsgenie
It gets worse, but nothing is lost yet	Team chat, such as Slack
Tomorrow is genuinely fine	Email, or a low-traffic chat channel
Nothing, because it is already known	No notifier at all

Each trigger selects its own notifiers. A trigger with no notifier still records its alerts, so you can monitor a metric without sending notifications. Give staging its own channel, or leave its triggers without a notifier. Staging alerts arriving in a production channel are the fastest way to teach a team to ignore that channel.

Scope every trigger to the series you actually mean

A trigger does not open one alert. It opens one alert per matching series. A host trigger left at its default hostname of * covers every host reporting to the app, so 30 hosts crossing a threshold together produce 30 notifications for one problem. Narrow the scope until the notification says exactly what is affected:

Replace * with a specific hostname, or a prefix such as web-*. Tag values accept wildcards, so hostname=frontend* scopes a trigger to one class of machine.
Scope error rate and response time triggers to a single namespace, so web, background, and admin can carry different thresholds and different destinations.
Add a tag for the specific resource you care about, such as one mountpoint rather than every mounted filesystem.

Use exact tag filters or send a dedicated alert metric

This limitation causes many noisy-alerting support requests, and it is worth understanding before you design a metric. Suppose you send a counter named api_request, tagged with status, so you can break it down by 200, 404, and 500. If you only care about one exact value, such as status=500, use that exact value in the trigger. The problem starts when you want a group of values: all 5xx responses, anything except 200, or every failed response. A tag filter cannot express exclusions or ranges, and * matches every value for that tag. A broad wildcard trigger can open alerts for successful statuses alongside failures, and most of what arrives describes an application working correctly. The solution is to make the condition you want to alert on its own series, at the point you send it. Keep the tagged metric when it is useful for charting, and send a second, narrower one for alerting:

status_code = response.code.to_s

# Useful for charts broken down by status.
Appsignal.increment_counter("api_request", 1, :status => status_code)

# Useful for alerting because it only increments for 5xx responses.
Appsignal.increment_counter("api_request_failed", 1) if status_code.to_i >= 500

status = response.status

# Useful for charts broken down by status.
Appsignal.increment_counter("api_request", 1, %{status: status})

if status >= 500 do
  # Useful for alerting because it only increments for 5xx responses.
  Appsignal.increment_counter("api_request_failed", 1)
end

Then configure the trigger to use the narrower metric:

Shell

appsignal-cli triggers create --app "MyApp" --environment production \
  --name "Server errors from the payment API" \
  --metric-name api_request_failed --kind Advanced --field counter \
  --comparison-operator ">" --condition-value 10 \
  --warmup-duration 3 --cooldown-duration 5 \
  --notifier-ids <NOTIFIER_ID>

The alert now opens only for the condition that should send a notification. Decide this while you are writing the custom metric, because it is easier than redesigning the metric after alerts start opening.

Keep tag values to a small, fixed set

Never use a value that is unique per event as a tag value. Request identifiers, order numbers, customer identifiers, and timestamps all look like useful context and all cause the same two problems. Every distinct value is a separate series. A trigger over that metric opens one alert per value, which can create an unbounded number of alerts. High-cardinality tags also make metric charts and alerting hard to manage. A tag value should have a handful of variations that you could list from memory. For example, tag by a bounded dimension such as region, not by a unique identifier:

# A tag with a small set of values is safe to graph and alert on.
Appsignal.increment_counter("order_processed", 1, :region => order.region)

# A tag with a small set of values is safe to graph and alert on.
Appsignal.increment_counter("order_processed", 1, %{region: order.region})

The order identifier still matters when you are investigating, so put it in a log line, where high cardinality is expected. Use metric tags for dimensions you would want to group a chart by. The custom metrics reference has more on choosing tags, including keeping the same combination everywhere you report a metric.

Alert on rates, not counts

An error count crosses a fixed threshold whenever traffic grows, so it notifies you on your busiest day regardless of whether anything is wrong. Error rate stays meaningful at 4 a.m. and at peak. Switching from a count to a rate is often the single edit that turns a nightly alert into a useful one. Keep counts only where any occurrence matters and volume is naturally low, such as failures in a payment flow.

Match warm-up to how much the metric changes

Warm-up is how long AppSignal waits before opening an alert. During that wait, the metric must stay above or below the threshold, depending on the trigger. This filters out single-minute spikes, such as one slow deploy, one garbage collection pause, or one restarting host. Set warm-up based on how much the metric changes minute to minute, not how serious the metric is. Start with zero to two minutes for most production alerts. Use a longer warm-up only when a metric often spikes and recovers quickly. Memory and swap move gradually, so they often need little or no warm-up. If you need 30 or 60 minutes to reduce notifications, review the threshold or scope instead.

Use cooldown when a metric stays near its threshold

Cooldown keeps one incident from producing a stream of open and close notifications while a metric repeatedly crosses the threshold. Most triggers never need one. The exception is slow-moving resource metrics, which may need a longer cooldown: a disk close to 80% may cross that line several times before someone frees space, so without a long cooldown it can alert again every time the value moves above and below the threshold. Thirty to 60 minutes turns several notifications into one alert that stays open until it is fixed.

Do not raise the cooldown on an uptime monitor

Raising the cooldown is the standard fix for a trigger that opens and closes repeatedly, but it is the wrong fix here. An uptime monitor failing means the endpoint did not respond. Suppressing the second notification does not make the first one less accurate. Reach for warm-up instead. AppSignal checks your endpoint every minute from four regions and tracks each region separately, so one outage can open up to four alerts, and a network problem between a single region and your endpoint opens one while your application is healthy. A warm-up of at least one minute filters out that case without hiding a real outage.

Make every notification explain itself

Alert fatigue is not only about how many notifications arrive. It is also about how much work each one takes to understand. Three fields provide that context: Name. Say what is wrong, not which metric moved. A custom metric trigger starts out named Custom metric, and a list of triggers all called that cannot be triaged. Description. Treat it as the runbook. Short is fine, but the field supports several paragraphs, commands, and links. The most useful descriptions include the information a responder would otherwise have to reconstruct: the number that matters (“the limit from the load balancer is 320”), the remedies in order, a diagnostic step, how long the fix takes, and what the cause was last time. Dashboard. Link the dashboard that shows the metric alongside related metrics, so the notification gives the responder a starting point for investigation.

Review your triggers on a schedule

Alert configuration becomes outdated. Triggers outlive the incidents that motivated them, thresholds set for last year’s traffic no longer fit, and hosts named in a trigger get replaced. Roughly quarterly, ask of each trigger: has it opened in the last 90 days, did anyone act on it, did it open many alerts for one incident, and is its destination still right? An alert nobody acts on should be retuned, rerouted, or archived. Reviewing the whole set is faster from the terminal than by opening each trigger:

Shell

# Every trigger, as JSON you can diff, grep, or hand to an agent
appsignal-cli --output json triggers list --app "MyApp" --environment production

# Narrow to one metric or one kind when you are hunting duplicates
appsignal-cli triggers list --app "MyApp" --environment production --metric-name response_time

Retuning is a single command, but updates need the full trigger definition. Start from the current values in triggers list, change the fields you want to retune, and include optional values you want to keep, such as notifier IDs, tags, and the dashboard link:

Shell

appsignal-cli triggers update --id <TRIGGER_ID> --app "MyApp" --environment production \
  --name "Slow web requests" \
  --metric-name response_time --kind Advanced --field p95 \
  --comparison-operator ">" --condition-value 750 \
  --warmup-duration 5 --cooldown-duration 5 \
  --notifier-ids <NOTIFIER_ID>

Changing trigger settings can affect active alert state, so avoid retuning a trigger while you are using an open alert for incident response. Archiving a trigger also closes its alerts and incidents, which is useful for a trigger nobody acts on:

Shell

appsignal-cli triggers archive --id <TRIGGER_ID> --app "MyApp" --environment production

The MCP server exposes the same read and write access to triggers, so you can ask an agent to list every trigger that has not opened in 90 days and propose what to archive. See the AppSignal CLI reference for all flags.

How AppSignal alerts on AppSignal

We run AppSignal on AppSignal, and our operations team has been tuning that setup for years. The shape of it is a reasonable target to aim at:

What we watch	Condition	Warm-up	Cooldown	Destination
Disk usage on `/`	Above 80%	10 minutes	60 minutes	On-call
Free memory	Below 100 MB	0	0	On-call
Swap in use	Above 50%	0	0	On-call
Items waiting in a queue, tagged by queue	More than 10,000	2 minutes	0	On-call
Processing lag, at p95	Above 2,000 ms	0	0	On-call
Connections to one class of host	More than 800	2 minutes	0	Team chat only
Throughput in a background namespace	30,000 req/min or more	2 minutes	0	On-call
Replication lag	Above 60 seconds	2 minutes	0	Email and on-call

Five patterns in that table are useful for production alerts:

Warm-ups are short. Most are zero to two minutes. The only long one is on disk usage, and it is paired with an hour of cooldown rather than used instead of it.
Cooldown is used once, deliberately. Only disk usage usually remains above the threshold until someone takes action, so only disk usage needs it.
Not every alert notifies on-call. The connection ceiling goes to team chat, because it predicts a capacity problem rather than describing an outage.
Email is added, not substituted. Replication lag reaches an on-call tool and email together, because email is the only channel that keeps reminding while an alert stays open.
Every trigger carries a description. The most useful ones name the relevant limit, give two remedies in order, and record what the cause turned out to be last time.

The triggers that are hardest to review are the ones left named Custom metric. Naming is a small task, but it has a large effect on review and triage.

Next steps

Work through the settings in detail with tune your alerts.
Check nothing important is unmonitored with alerting in production.
Find a starting configuration for a specific alert in alert recipes.
Shape a metric for alerting when you send it, with custom metrics.

Features

Application health

Logs & metrics

Infrastructure

Application customization

Decide what a notification costs before you route it

Scope every trigger to the series you actually mean

Use exact tag filters or send a dedicated alert metric

Keep tag values to a small, fixed set

Alert on rates, not counts

Match warm-up to how much the metric changes

Use cooldown when a metric stays near its threshold

Do not raise the cooldown on an uptime monitor

Make every notification explain itself

Review your triggers on a schedule

How AppSignal alerts on AppSignal

Next steps

​Decide what a notification costs before you route it

​Scope every trigger to the series you actually mean

​Use exact tag filters or send a dedicated alert metric

​Keep tag values to a small, fixed set

​Alert on rates, not counts

​Match warm-up to how much the metric changes

​Use cooldown when a metric stays near its threshold

​Do not raise the cooldown on an uptime monitor

​Make every notification explain itself

​Review your triggers on a schedule

​How AppSignal alerts on AppSignal

​Next steps

Decide what a notification costs before you route it

Scope every trigger to the series you actually mean

Use exact tag filters or send a dedicated alert metric

Keep tag values to a small, fixed set

Alert on rates, not counts

Match warm-up to how much the metric changes

Use cooldown when a metric stays near its threshold

Do not raise the cooldown on an uptime monitor

Make every notification explain itself

Review your triggers on a schedule

How AppSignal alerts on AppSignal

Next steps