Before you create a trigger
Three setup steps make every alert that follows more useful, and all three are easier to do first than to retrofit. Add your notifiers. Triggers reference notifiers, so creating them first means you can route each alert as you build it rather than going back over everything later. Set up at least two destinations: somewhere urgent, such as an on-call tool, and somewhere non-urgent, such as a team chat channel. Notifiers are configured once for the organization and shared across apps. Send deploy markers. Most production incidents start with a release. A marker turns “when did this start?” into a question you can answer in seconds, and it is what makes the First in Deploy notification setting work. Split your app into namespaces. Namespaces are the main lever for alerting at different severities without configuring each alert individually. Ifweb, background, and admin traffic all land in one namespace, every threshold you set has to suit all three at once, which usually means it suits none of them.
Set up in layers
Adding every alert at once produces too many notifications on day one and teaches the team to ignore them. Add a layer, live with it for a week, then add the next.- Reachability. An uptime monitor on a health-check endpoint. This is the only alert that still works when your application cannot report anything at all.
- User-visible symptoms. Error rate and response time on your user-facing namespace. These alerts correspond to errors or slow response times experienced by users.
- Scheduled and background work. Check-ins for cron jobs and worker heartbeats, and an absence alert for any queue that should never be empty of activity.
- Saturation. Host disk, memory, and swap. These are causes rather than symptoms, so route them to chat rather than on-call.
- Business metrics. Custom metrics for the application-specific activity only you can measure, such as completed payments or signups per minute.
Keep production separate
Production alerting only stays credible if non-production noise never reaches it.- Give each environment its own app in AppSignal, so thresholds and notifiers are set per environment rather than shared.
- Give staging its own chat channel, or leave staging triggers with no notifier at all. A trigger with no notifier still records alerts you can review later, without notifying anyone.
- Use different thresholds. Staging traffic is lower and changes more than production traffic, so a threshold tuned on production data may never open an alert or may open alerts too often.
- Keep personal test channels out of production apps. A notifier can be shared by several apps, and trigger alerts are routed by the notifiers selected on each trigger.
Check your coverage
Too many notifications are the visible failure. The hidden failure is an outage nobody was alerted to. Work through the failure modes rather than the features, and check that each one has an alert:
The last row is the one most often missing. Every other alert on this list needs data to arrive before it can open.
Two questions are useful for reviewing the whole set. First, for each row you have covered, does the alert reach someone who can act at the time of day it is most likely to open? Second, of your last few production incidents, how many were first reported by a person rather than by an alert? That second number is a useful measure of coverage.
Decide how fast you need to know
Detection is not instant, and it is worth deciding deliberately how much delay each alert can carry. Three factors add up to the time between a problem starting and a notification arriving:- Aggregation. Metrics are evaluated once per minute, so there is always up to a minute before the offending value is even considered.
- Processing. AppSignal waits for the data for a minute to arrive from all of your servers before evaluating it.
- Warm-up. The number of minutes AppSignal waits before opening the alert.
If an alert is noisy but also needs to be fast, do not solve it with a longer warm-up. Narrow the scope or move the threshold instead, both of which reduce false alarms without costing detection time.
Verify an alert before you rely on it
A trigger that matches nothing can look like a healthy system. Before you count an alert as coverage, confirm all of the following:- The chart preview shows data. An empty preview on the trigger form means the trigger currently matches nothing.
- The hostname resolves. If the hostname field warns that the host was not found, the trigger is probably watching a machine that has been renamed or replaced.
- The tag combination is recognized. AppSignal warns when a tag combination has not been seen in the last hour. Use one of the tag-key combinations listed in the trigger form, and set each value to an exact value or a wildcard.
- A notifier is attached. A trigger with no notifier records alerts without sending notifications.
- The notifier works. Send a test notification, and confirm it arrives in the channel you expect rather than one that looks similar.
- Absence alerts have missing data handled. For any trigger whose threshold is zero or a minimum value, confirm treat missing datapoints as 0 is enabled, or it will not open in exactly the case you built it for.
Operate it
Alerting is not finished when it is set up.- Write the runbook into the trigger. A name that says what is wrong, a description that says what to do, and a linked dashboard. Covered in make the notification explain itself.
- Avoid retuning during an incident. Changing trigger settings can affect the alert you are currently tracking, including closing or replacing active alert state.
- Review quarterly. Alerting decays as hosts are replaced, services are renamed, and traffic grows. The review checklist covers what to look for.
- Add an alert after each incident, and remove one too. Post-incident review is the best time to notice both what was missing and what opened without requiring action.
Next steps
- Pick configurations from alert recipes.
- Reduce volume with tune your alerts.
- Set notification defaults per namespace.
- Set up check-ins for scheduled work.