Sidekiq

Sidekiq is a simple and efficient background processor for Ruby. It's also the processor we use to process jobs in AppSignal.

When AppSignal detects Sidekiq metrics, it will create an Automated Dashboard, allowing you to monitor core metrics visually.

Sidekiq monitoring for Ruby on Rails apps

The AppSignal Ruby gem automatically inserts a listener into the Sidekiq server middleware stack if the Sidekiq module is present if you use Rails. No further action is required.

The Sidekiq integration is compatible with Active Job.

Sidekiq monitoring for Ruby apps

Add this snippet to your Sidekiq config with the right environment and name:

require 'appsignal'
 
Sidekiq.on(:startup) do
  # Start AppSignal
  Appsignal.start
end
 
Sidekiq.on(:shutdown) do
  # Stop the agent
  Appsignal.stop('Sidekiq shutdown')
end

Performance monitoring

Once AppSignal begins reporting Sidekiq metrics, AppSignal will give you performance insights via:

• Event timeline
• Error tracking
• Incident grouping
• Sample breakdown

Event timeline

perform_job.sidekiq events will be shown in the event timeline on the performance incident detail page:

Error tracking

Track exceptions that occur when running Sidekiq jobs.

When Sidekiq encounters a problem before or after a job has been processed, such as parsing JSON from Redis, it will raise an error. This error is reported under the SidekiqInternal action on the background namespace, as the job context is unknown when the error occurs.

Report errors on job discard

Set the sidekiq_report_errors config option to discard to only report errors when a job is discarded. When a job is discarded, all job retries have been exhausted, and the job is no longer retried. Read the Sidekiq documentation to learn more about Sidekiq exception handling and failed job retries.

Incident grouping

Job names are automatically detected based on the Sidekiq worker class name and suffixed with the perform method name, resulting in something like: MyWorker#perform.

Sample performance

Sample breakdowns allow you to review Sidekiq's performance quickly and spot performance problems without having to dive deep into the details.

Automated dashboard

When AppSignal receives Sidekiq metrics, it will create a Sidekiq automated dashboard, available from the dashboard section of the AppSignal app.

The Sidekiq automated dashboard will have the following graphs:

Tags give you a contextual breakdown of Sidekiq performance information. AppSignal reports the following tags for Sidekiq jobs:

Each tag will be represented with a colored line on the graph:

Connection count graph

The Connection count graph shows the count of Sidekiq connections per host.

You can use the Connection count graph to monitor Sidekiq connections per host, spot connection trends and bottlenecks, and optimize resources.

Duration per worker graph

The Duration per worker graph shows the amount of time that it took for jobs to execute, grouped by namespace and action.

You can use the Duration per worker graph to monitor the performance of workers per action and namespace, giving you a helicopter view of Sidekiq performance and allowing you to quickly identify and investigate spikes in duration time.

Job status per queue graph

The job status per queue with a priority graph shows the number of jobs that were executed, grouped by their resulting status, by the queue in which they were queued.

You can use the Job status per queue graph to monitor job error counts and performance based on queue, identify bottlenecks, and optimize your background jobs for scalability.

Overall job status graph

The overall job status graph shows the number of expected Sidekiq jobs per status and namespace.

You can use the Overall job status graph to monitor failed jobs and track job distribution across namespaces.

Queue latency graph

The Queue latency graph shows the latency the queue experienced at the time of measurement. Sidekiq calculates this by subtracting the time the last job was enqueued from the time of measurement. This value is reported as milliseconds.

You can use the Queue latency graph to spot delays in processing jobs, detect queue congestion, and understand how quickly background jobs start.

Queue length graph

The Queue length graph shows the length of Sidekiq queues per queue and namespace.

You can use the Queue length graph to monitor queue performance and spot and solve Sidekiq bottlenecks.

Redis memory usage graph

The Redis memory usage graph shows:

• The Virtual Memory Size memory usage of Redis (sidekiq_memory_usage_rss)
• The Resident Set Size memory usage of Redis (sidekiq_memory_usage_rss)

You can use the Redis memory usage graph to monitor Sidekiq's memory usage, improve resource management, and optimize memory-intensive jobs.

Throughput per worker graph

The Throughput per worker graph shows the number of jobs that were executed, grouped by hostname.

You can use this graph to monitor worker performance, grouped by the class that defines the job.

Worker/processes count graph

The Worker/processes count graph shows the number of Sidekiq workers and executed processes.

You can use the Worker/processes count graph to manage workload distribution, optimize resource allocation, and identify performance problems.

Hostname configuration

AppSignal attempts to detect the hostname of the Redis instance your Sidekiq instance uses to store its queues. If the detection is not accurate, it's possible to customize the hostname configuration by overriding the default Sidekiq probe.

First, you'll need to override the default Sidekiq probe by registering a new probe with the same name (:sidekiq). This probe will need a configuration hash, including the :hostname key, with the new hostname value. By specifying the :hostname config option in the Sidekiq minutely probe, the metrics will be tagged with the given hostname value. The :hostname config option value is not used to establish a Redis or Sidekiq connection.

For example:

# config/initializers/appsignal.rb or a file that's loaded on boot
 
# Ruby gem 2.11.0 and newer
Appsignal::Minutely.probes.register(
  :sidekiq, # Use the same key as the default Sidekiq probe to override it
  Appsignal::Probes::SidekiqProbe.new(:hostname => "my_sidekiq_hostname")
)
 
# Ruby gem 2.10.x and older
Appsignal::Minutely.probes.register(
  :sidekiq, # Use the same key as the default Sidekiq probe to override it
  Appsignal::Hooks::SidekiqProbe.new(:hostname => "my_sidekiq_hostname")
)

In version 2.11.0 of the Ruby gem, the SidekiqProbe constant was moved to its own module. Upon calling the constant, a warning will be printed and logged. Update to the new constant name Appsignal::Probes::SidekiqProbe to remove the warning.