> ## Documentation Index > Fetch the complete documentation index at: https://docs.appsignal.com/llms.txt > Use this file to discover all available pages before exploring further. # Alerting in production > Set up alerting for a production environment in layers, check your coverage for gaps, and decide how fast you need to know about each kind of failure. This page is about getting production alerting right the first time: what to set up, in what order, how to confirm it works, and how to check that nothing important is unmonitored. It assumes you know what you want to alert on. If you do not yet, start with [alert recipes](/alerting/recipes). If you already have alerts and they are too noisy, go to [tune your alerts](/alerting/tuning). ## Before you create a trigger Three setup steps make every alert that follows more useful, and all three are easier to do first than to retrofit. **Add your notifiers.** Triggers reference notifiers, so creating them first means you can route each alert as you build it rather than going back over everything later. Set up at least two destinations: somewhere urgent, such as an on-call tool, and somewhere non-urgent, such as a team chat channel. Notifiers are configured once for the organization and shared across apps. **Send [deploy markers](/application/markers/deploy-markers).** Most production incidents start with a release. A marker turns "when did this start?" into a question you can answer in seconds, and it is what makes the [**First in Deploy** notification setting](/application/notification-settings#first-in-deploy) work. **Split your app into [namespaces](/application/namespaces).** Namespaces are the main lever for alerting at different severities without configuring each alert individually. If `web`, `background`, and `admin` traffic all land in one namespace, every threshold you set has to suit all three at once, which usually means it suits none of them. ## Add alerts in layers Adding every alert at once produces too many notifications on day one and teaches the team to ignore them. Add a layer, live with it for a week, then add the next. 1. **Reachability.** An [uptime monitor](/uptime-monitoring/setup) on a health-check endpoint. This is the only alert that still works when your application cannot report anything at all. 2. **User-visible symptoms.** Error rate and response time on your user-facing namespace. These alerts correspond to errors or slow response times experienced by users. 3. **Scheduled and background work.** [Check-ins](/check-ins) for cron jobs and worker heartbeats, and an absence alert for any queue that should never be empty of activity. 4. **Saturation.** Host disk, memory, and swap. These are causes rather than symptoms, so route them to chat rather than on-call. 5. **Business metrics.** [Custom metrics](/metrics/custom) for the application-specific activity only you can measure, such as completed payments or signups per minute. Layers one and two cover most real incidents. Layer five detects application-specific failures that other alerts cannot detect. ## Keep staging and test noise out of production Production alerting only stays credible if non-production noise never reaches it. * Give each environment its own app in AppSignal, so thresholds and notifiers are set per environment rather than shared. * Give staging its own chat channel, or leave staging triggers with no notifier at all. A trigger with no notifier still records alerts you can review later, without notifying anyone. * Use different thresholds. Staging traffic is lower and changes more than production traffic, so a threshold tuned on production data may never open an alert or may open alerts too often. * Keep personal test channels out of production apps. A notifier can be shared by several apps, and trigger alerts are routed by the notifiers selected on each trigger. ## Check your coverage Too many notifications are the visible failure. The hidden failure is an outage nobody was alerted to. Work through the failure modes rather than the features, and check that each one has an alert: | If this happens | What detects it | | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | The application is completely unreachable | [Uptime monitor](/uptime-monitoring/setup) | | The application responds but returns errors, so actions such as adding an item to a cart fail | Error rate trigger on the user-facing namespace | | The application responds slowly | p90 or p95 performance trigger | | Requests stop arriving at all, because something upstream broke before they reached the app | Throughput floor, or a trigger on throughput equal to `0` | | A background worker dies, so confirmation emails or invoices stop being sent | Absence alert on a job counter, or a heartbeat [check-in](/check-ins) | | A scheduled job does not run, so today's data is missing | Cron [check-in](/check-ins) | | A queue backs up, so work is accepted but takes hours to be processed | Queue depth or queue time trigger | | A host runs out of disk, memory, or swap, so the application stops abruptly | Host metric triggers | | A dependency fails intermittently but retries succeed, such as one payment provider or delivery partner | [Log trigger](/logging/triggers), or a metric extracted from those log lines | | A business process stops without errors appearing in AppSignal, such as orders, payments, or bookings no longer going through | Custom metric equal to `0`, or below a minimum threshold | | A TLS certificate is close to expiration, so browsers warn that the site is not secure | SSL expiration monitoring, available as an [AppSignal Labs](/labs) feature | | A metric stops being reported entirely | A trigger with **treat missing datapoints as 0** enabled | The last row is the one most often missing. Every other alert on this list needs data to arrive before it can open. Two questions are useful for reviewing the whole set. First, for each row you have covered, does the alert reach someone who can act at the time of day it is most likely to open? Second, of your last few production incidents, how many were first reported by a person rather than by an alert? That second number is a useful measure of coverage. ## Decide how fast you need to know Detection is not instant, and it is worth deciding deliberately how much delay each alert can carry. Three factors add up to the time between a problem starting and a notification arriving: * **Aggregation.** Metrics are evaluated once per minute, so there is always up to a minute before the offending value is even considered. * **[Processing](/anomaly-detection#data-processing).** AppSignal waits for the data for a minute to arrive from all of your servers before evaluating it. * **Warm-up.** The number of minutes AppSignal waits before opening the alert. Warm-up delays the alert. If a trigger has a 2-minute warm-up, the metric must stay above or below the threshold for 2 minutes before AppSignal opens the alert. These are reasonable targets for how quickly you need to know about each kind of failure: * **Checkout or payments broken, so customers cannot buy, book, or transfer money.** As fast as possible, so little or no warm-up. * **Site or app unreachable.** 1–2 minutes. * **Elevated error rate, so customers are hitting errors.** 1–2 minutes. * **Response time degradation, so requests take longer than usual.** 1–3 minutes. * **Queue backing up, so work is still being processed but arrives late.** 1–3 minutes. * **Disk usage approaching full capacity.** 10–60 minutes. If an alert is noisy but also needs to be fast, do not solve it with a longer warm-up. Narrow the scope or move the threshold instead, both of which reduce false alarms without costing detection time. Anomaly detection cannot tell you that something did not happen within a window shorter than the data it receives. Do not use a long warm-up as a substitute for [check-ins](/check-ins) when monitoring scheduled work. ## Verify an alert before you rely on it A trigger that matches nothing can look like a healthy system. Before you count an alert as coverage, confirm all of the following: 1. **The chart preview shows data.** An empty preview on the trigger form means the trigger currently matches nothing. 2. **The hostname resolves.** If the hostname field warns that the host was not found, the trigger is probably watching a machine that has been renamed or replaced. 3. **The tag combination is recognized.** AppSignal warns when a tag combination has not been seen in the last hour. Use one of the tag-key combinations listed in the trigger form, and set each value to an exact value or a wildcard. 4. **A notifier is attached.** A trigger with no notifier records alerts without sending notifications. 5. **The notifier works.** Send a test notification, and confirm it arrives in the channel you expect rather than one that looks similar. 6. **Absence alerts have missing data handled.** For any trigger whose threshold is zero or a minimum value, confirm **treat missing datapoints as 0** is enabled, or it will not open in exactly the case you built it for. ## Maintain alerting after setup Alerting is not finished when it is set up. * **Write the runbook into the trigger.** A name that says what is wrong, a description that says what to do, and a linked dashboard. Covered in [make the notification explain itself](/alerting/tuning#make-the-notification-explain-itself). * **Avoid retuning during an incident.** Changing trigger settings can affect the alert you are currently tracking, including closing or replacing active alert state. * **Review quarterly.** Alerting decays as hosts are replaced, services are renamed, and traffic grows. The [review checklist](/alerting/tuning#review-triggers-regularly) covers what to look for. * **Add an alert after each incident, and remove one too.** Post-incident review is the best time to notice both what was missing and what opened without requiring action. ## Next steps * Pick configurations from [alert recipes](/alerting/recipes). * Reduce volume with [tune your alerts](/alerting/tuning). * Set [notification defaults per namespace](/application/notification-settings#organization-and-app-namespace-defaults). * Set up [check-ins](/check-ins) for scheduled work.