Notification settings
Whenever AppSignal detects a new error or performance measurement, we open an incident. You can find the incidents in the Errors and Performance sections.
AppSignal can send out notifications whenever a new incident, or new instances of an already existing incident, is detected. Notifications can happen by email or by one of our many notifiers.
Notification options
Whenever a new incident gets created it will inherit the app's notification defaults. When an Error or Performance incident has been created, its notification settings can be changed on the incident page itself.
If an incident was closed, it will be reopened when a new notification is sent out.
There are five notification options available:
Every Occurrence
A notification is sent every time an incident is triggered. This option has a cool down of 5 minutes. This means that if an incident is triggered 20 times a minute, for 20 minutes, you'll receive a notification on minute 0, 5, 10, 15 and 20.
First in Deploy
A notification is sent on the first occurrence of an incident after a new deploy marker is received. You'll need to set up deploy markers for AppSignal to detect a deploy and know when an incident occurred for the first time in a deploy.
First After Close
A notification is sent at the first occurrence of an incident after the incident was previously closed. Incidents can opened and closed in the sidebar in the Incident detail page. This option is is a good option for incidents that aren't triggered by a code bug, but out an outside source (e.g. 3rd party has a connection issue) and can't be closed by deploying a new version of the app.
Never Notify
No notification is ever sent out when the incident is triggered. Mostly used for errors that will continue to happen and you'd like to keep track of, but aren't getting fixed soon. If you don't want to receive the incident at all, ignoring the action or ignoring the error may help.
Every Nth per Hour/Day
No notification is sent every nth time per hour, or day, depending on your preferences. There are many scenarios where this may be beneficial:
- Billing Error: Imagine a billing error occurring now and then, outside of your control. It happens a maximum of 5 times a day, and you want to know when this suddenly increases. Setting it to "every 10th time a day" will send you an alert on every 10th, 20th, 30th occurrence, etc.
- API Error: You might have an API that many people use. When experimenting with an API, making mistakes that trigger errors is common. You might only want to be alerted when the number of errors reaches a threshold. You could, for instance, set it to notify you every 1000th time an hour, which might indicate that the API is failing for different reasons.
- During Incident: Imagine something is severely broken. Hundreds of errors an hour are happening, but you're on top of it. It might not be a good idea to ignore all alerts completely, but getting each one of them will drown you in notifications. This is the perfect moment to manage the number of notifications you receive by temporarily setting your errors to Nth/hour.
Above is a visualization of when you would receive alerts. In this scenario, notifications are set to "Every 100th per hour".
Error incident notification settings
Error incidents are identified by the error (class)name, such as ActiveRecord::RecordNotFound and if the error happened inside of an action, the action name. (e.g. StandardError in BlogpostsController#show).
Performance incident notification settings
Performance incidents are identified by their "action" name, such as BlogpostsController#show.
Once we detect a new action we create an incident for it. At this point you can override the default notification settings and change when you'd like to be notified and what threshold the duration of the performance incident has to reach before a notification is sent.
E.g. for actions where response times can be slow, because they interact with 3rd parties, you can set the duration threshold to 10 seconds, while for the homepage you can set a threshold of 200 milliseconds.
The threshold filter is used in combination with the notification settings. E.g. if you have First in deploy and 200ms as the notification settings and threshold, an incident occurrence has to satisfy both in order for a notification to be sent out.
Organization and app namespace defaults
Notification settings on an incident can only be changed after it has occurred at least once. This can be an issue if you never want to be notified, or want to set a different performance incident threshold than the default 200 milliseconds.
It's possible to configure notification defaults per app namespace or even for the entire organization. This will make it easier to apply the desired notification settings to all new incidents and all existing incidents, for which the notification settings haven't been customized.
App notification defaults
Each application has its own namespace defaults for notification settings. A new incident in the specified namespace will inherit the default settings set on this page. For more information, see the notification defaults inheritance section.
It's recommended to split up the application in namespaces that group potential incidents by severity. This way you can configure notification settings for a group of incidents and there's no need to configure each incident separately. For example, for an app's admin panel you could create a namespace called admin with a default notification setting of "Never notify" as errors that occur in this namespace have less priority.
Organization notification defaults
It's also possible to set up notification defaults on your organization. These defaults will be used when AppSignal detects and creates a new application. When it creates the new app it will apply the organization's notification defaults as the app's app notification defaults. Changes to the organization notification defaults do not apply to already existing apps. For more information, see the notification defaults inheritance section.
These organization level notification defaults can be set up in the organization's admin panel. It's possible to configure defaults for each namespace detected in any of the current applications in your organization.
Notification defaults inheritance
The notification settings bubble up from the incident settings to the app's namespace settings. If the notification settings for an incident aren't changed the app's namespace defaults are used. This means you can also change all the incident notification settings for incidents that haven't customized their notification settings by changing the app defaults.
- When a new app is detected by AppSignal, the organization's namespace defaults are applied.
- When organization namespace notification defaults are changed, they only apply to new apps.
- When an app's namespace notification defaults are changed, they apply to all new incidents and existing incidents without customized notification settings.
- When a new incident is detected by AppSignal, the app's namespace defaults are used.
- When an incident's notification settings are customized, the incident's customized notification settings are used from then on.