Sidekiq

Sidekiq is a simple and efficient background processor for Ruby. It’s also the processor we use to process jobs in AppSignal. When AppSignal detects Sidekiq metrics, it will create an Intelligence dashboard, allowing you to monitor core metrics visually.

Sidekiq monitoring for Ruby on Rails apps

The AppSignal Ruby gem automatically inserts a listener into the Sidekiq server middleware stack if the Sidekiq module is present if you use Rails. No further action is required. The Sidekiq integration is compatible with Active Job.

Sidekiq monitoring for Ruby apps

Adding this config should only be necessary when Sidekiq doesn’t automatically load AppSignal through the app’s framework, like Rails. Add this snippet to your Sidekiq config with the right environment and name:

require 'appsignal'

Sidekiq.configure_server do |config|
  config.on(:startup) do
    # Start AppSignal
    Appsignal.start
  end

  config.on(:shutdown) do
    # Stop the agent
    Appsignal.stop('Sidekiq shutdown')
  end
end

Performance monitoring

Once AppSignal begins reporting Sidekiq metrics, AppSignal will give you performance insights via:

Event timeline
Error tracking
Incident grouping
Sample breakdown

Event timeline

perform_job.sidekiq events will be shown in the event timeline on the performance incident detail page:

Error tracking

Track exceptions that occur when running Sidekiq jobs. When Sidekiq encounters a problem before or after a job has been processed, such as parsing JSON from Redis, it will raise an error. This error is reported under the SidekiqInternal action on the background namespace, as the job context is unknown when the error occurs.

Report errors on job discard

Set the sidekiq_report_errors config option to discard to only report errors when a job is discarded. When a job is discarded, all job retries have been exhausted, and the job is no longer retried. Read the Sidekiq documentation to learn more about Sidekiq exception handling and failed job retries.

When using Sidekiq with Active Job’s retry system, configure it to report errors only on discard, so that it won’t report errors multiple times.After the Active Job retries are exhausted, the Sidekiq retry system will take over. Disable the Sidekiq retry system if this is unwanted behavior.

Incident grouping

AppSignal detects job names from the Sidekiq worker class name, suffixed with the perform method name, resulting in something like MyWorker#perform.

Sample performance

Sample breakdowns allow you to review Sidekiq’s performance quickly and spot performance problems without having to dive deep into the details.

Enqueue instrumentation

Enqueuing a Sidekiq job records an enqueue.sidekiq event, titled after the job being enqueued. AppSignal records enqueue events on the active transaction’s event timeline, for example when you enqueue a job from within a web request or from within another job. It only records them when a transaction is active, so enqueuing a job outside of a transaction records nothing. To stop recording enqueue events across all background job integrations, set the enable_job_enqueue_instrumentation config option to false. This does not affect the instrumentation of the jobs themselves.

Intelligence dashboard

When AppSignal receives Sidekiq metrics, it will create a Sidekiq Intelligence dashboard, available from the dashboard section of the AppSignal app. The Sidekiq Intelligence dashboard will have the following graphs:

Graph	Metrics	Tags
Connection count	`sidekiq_connection_count`	`hostname`
Duration per worker	`transaction_duration`	`action` `namespace`
Job status per queue	`sidekiq_queue_job_count`	`status` `queue`
Overall job status	`sidekiq_job_count`	`status` `hostname`
Queue latency	`sidekiq_queue_latency`	`hostname` `queue`
Queue length	`sidekiq_queue_length`	`hostname` `queue`
Redis memory usage	`sidekiq_memory_usage` `sidekiq_memory_usage_rss`	`hostname`
Throughput per worker	`transaction_duration`	`action` `namespace`
Worker/processes count	`sidekiq_worker_count` `sidekiq_process_count`	`hostname`

Tags give you a contextual breakdown of Sidekiq performance information. AppSignal reports the following tags for Sidekiq jobs:

Name	Description
`action`	The name of the action the metric was reported from (e.g., `YourWorker#perform`).
`hostname`	The name of the host the metric was reported from.
`namespace`	The name of the namespace the metric was reported from.
`queue`	Named queue in which jobs are processed, e.g. `default` or `mailer`.
`status`	Status of each job, either `processed` or `failed`. Jobs that are `failed` are also counted as `processed`.

Each tag will be represented with a colored line on the graph:

Connection count graph

The Connection count graph shows the count of Sidekiq connections per host. You can use the Connection count graph to monitor Sidekiq connections per host, spot connection trends and bottlenecks, and optimize resources.

Duration per worker graph

The Duration per worker graph shows the amount of time that it took for jobs to execute, grouped by namespace and action. You can use the Duration per worker graph to monitor the performance of workers per action and namespace, giving you a helicopter view of Sidekiq performance and allowing you to quickly identify and investigate spikes in duration time.

Job status per queue graph

The Job status per queue graph shows the number of jobs that were executed, grouped by their resulting status and by the queue in which they were enqueued. You can use the Job status per queue graph to monitor job error counts and performance based on queue, identify bottlenecks, and optimize your background jobs for scalability.

Overall job status graph

The overall job status graph shows the number of expected Sidekiq jobs per status and namespace. You can use the Overall job status graph to monitor failed jobs and track job distribution across namespaces.

Queue latency graph

The Queue latency graph shows the latency the queue experienced at the time of measurement. Sidekiq calculates this by subtracting the time the last job was enqueued from the time of measurement. This value is reported as milliseconds. You can use the Queue latency graph to spot delays in processing jobs, detect queue congestion, and understand how quickly background jobs start.

Queue length graph

The Queue length graph shows the length of Sidekiq queues per queue and namespace. You can use the Queue length graph to monitor queue performance and spot and solve Sidekiq bottlenecks.

Redis memory usage graph

The Redis memory usage graph shows:

The total memory allocated by Redis (sidekiq_memory_usage)
The resident set size, the physical memory used by Redis as seen by the operating system (sidekiq_memory_usage_rss)

You can use the Redis memory usage graph to monitor Sidekiq’s memory usage, improve resource management, and optimize memory-intensive jobs.

Throughput per worker graph

The Throughput per worker graph shows the number of jobs that were executed, grouped by hostname. You can use this graph to monitor worker performance, grouped by the class that defines the job.

Worker/processes count graph

The Worker/processes count graph shows the number of Sidekiq workers and executed processes. You can use the Worker/processes count graph to manage workload distribution, optimize resource allocation, and identify performance problems.

Hostname configuration

AppSignal attempts to detect the hostname of the Redis instance your Sidekiq instance uses to store its queues. If the detection is not accurate, it’s possible to customize the hostname configuration by overriding the default Sidekiq probe. First, you’ll need to override the default Sidekiq probe by registering a new probe with the same name (:sidekiq). This probe will need a configuration hash, including the :hostname key, with the new hostname value. By specifying the :hostname config option in the Sidekiq minutely probe, the metrics will be tagged with the given hostname value. The :hostname config option value is not used to establish a Redis or Sidekiq connection. For example:

# config/initializers/appsignal.rb or a file that's loaded on boot

# Ruby gem 2.11.0 and newer
Appsignal::Minutely.probes.register(
  :sidekiq, # Use the same key as the default Sidekiq probe to override it
  Appsignal::Probes::SidekiqProbe.new(:hostname => "my_sidekiq_hostname")
)

# Ruby gem 2.10.x and older
Appsignal::Minutely.probes.register(
  :sidekiq, # Use the same key as the default Sidekiq probe to override it
  Appsignal::Hooks::SidekiqProbe.new(:hostname => "my_sidekiq_hostname")
)

In version 2.11.0 of the Ruby gem, the SidekiqProbe constant was moved to its own module. Upon calling the constant, a warning will be printed and logged. Update to the new constant name Appsignal::Probes::SidekiqProbe to remove the warning.

SDKs & telemetry

Language SDKs

Telemetry tools

Sidekiq monitoring for Ruby on Rails apps

Sidekiq monitoring for Ruby apps