What this covers
- Grouping jobs into a batch and registering a success callback.
- Why the callback lives in its own class rather than in a block.
- Passing context to a callback that runs on a different machine.
Requirements
Batches are a Sidekiq Pro feature. The open source
sidekiq-batch gem implements the same API, which is useful for trying the pattern out. For production, Sidekiq Pro is the better-supported option.Write the jobs in plain Sidekiq
Batches are a Sidekiq feature, not an Active Job one. Jobs that take part in a batch should use Sidekiq directly, so that Active Job’s own wrapping does not sit between the batch and its jobs. Converting a job is a two-line change: includeSidekiq::Job rather than inheriting from ApplicationJob, and enqueue with perform_async rather than perform_later.
Sidekiq::Job is the current name. Older code and older screencasts use Sidekiq::Worker, which still works as an alias.Create the batch
Build the batch where the work is triggered, then enqueue the jobs insidebatch.jobs:
on(:complete) is the other callback you can register. It fires when every job has finished, whether or not they all succeeded. Use :success when the next step depends on the work having succeeded.
Write the callback as a class
The callback is a class rather than a block, and the reason is worth understanding rather than working around. By the time a batch completes, the process that created it is gone. The callback runs as its own Sidekiq job, potentially on a different machine, minutes later. No request, no local variable, and no model instance is still in scope. A block would close over state that no longer exists. So Sidekiq instantiates your callback class fresh and hands it the batch status and the options hash you registered:Chain batches together
A callback can create another batch. That is how you build a workflow of more than two stages. Create the servers. In the callback, configure them as a new batch. In that callback, mark the cluster active.Monitor batches with AppSignal
AppSignal instruments Sidekiq through its Sidekiq integration, and reports jobs in thebackground namespace.
Batches change what is worth watching:
- The callback is the job that matters. A batch is only useful if the callback runs. Let error tracking tell you when it raises. A callback that fails silently leaves a workflow half-finished, with the earlier jobs all reporting success. It runs as a Sidekiq job, so AppSignal instruments it automatically. Add instrumentation events to see where its time goes.
- One failure stops the batch. With an
on(:success)callback, a single failed job means the callback never runs. The failure will be in your error incidents; the absent callback will not be anywhere unless you look for it. - Long workflows deserve a check-in. Multi-stage batches that must complete on a schedule are a good fit for check-ins, which report the run that did not happen rather than the run that failed.
- Tag jobs with the batch. Adding the batch or workflow ID as a tag lets you pull every job from one run together when you are working out which stage stalled. AppSignal stores each job’s arguments as parameters.
Transcript
Transcribed from the video and lightly edited: the automatic captions misheard a number of product and API names, and those have been corrected.Read the transcript
Read the transcript
0:02 What’s up guys? This episode we’re gonna be talking about building advanced background workers with Sidekiq batches. This is really cool because it allows you to take a set of background jobs that are running and then batch them together. So when they’re finished, you can call a callback and actually go from serial execution to parallel back to serial. So normally when you’re using background jobs, you’re going from rails which is running in serial to parallel and then you’re done.0:33 But this is important for complex workflows because a lot of times, like when you’re building something like Hatchbox, one of the things that you need to keep in mind is you’re doing complex things is, for example, when you’re doing stuff in the background like creating and configuring servers, sometimes those need to wait for a block of work to be done. So for example, with a cluster of servers, you might have a load balancer that delegates to a bunch of web servers. And this one, we wanna add a web server to it. So we would have a load balancer and two web servers and this one’s also functioning as our cron and background workers and database server as well. But you can imagine that any setup like this is going to have similar problems.1:18 So the problem with this is when you go and do the work on the backend. If you submit this, we’re gonna know that you added a web server, so we need to create that new server in DigitalOcean. And so we can go and fire off that job to do that. And we’re gonna know that we need to update the load balancer and reconfigure it with the IP address of this new web server. However, if we’re gonna add, say, maybe a bunch of these, we couldn’t do this very easily.1:45 And the reason for that is because we’re going to create these two web servers on DigitalOcean, they may take a random amount of time to create. And we actually can’t configure the load balancer immediately. And we couldn’t just say, okay, configure all the machines, create the new machines because this load balancer has to wait until the new web servers are created and we have IP addresses for them. We can’t add their IP addresses if we don’t have them and into the load balancer. And so that’s our problem.2:16 We need to go and create all these servers in a batch and wait until all of them are finished and then we can go and configure all the machines. That way we have all of the required information to go do the configuration. So that’s where Sidekiq batches came into play. Now we’re gonna talk about Sidekiq batches. This is a pro feature on Sidekiq.2:35 So it will cost you money, but it’s definitely worth it and comes with some awesome other features that would be required pretty much for any production application using, you know, advanced background workers like this. So you also have the ability to use a open source version of Sidekiq batches, which adds the same API. I’m gonna show you that one just so that you guys can clone this repo and try it out as well, but you probably will want to use Sidekiq pro. So batches are really cool. You basically just write a little configuration like this and you create a new batch, set a description if you want, but most importantly, you can create a callback.3:16 Now, callbacks are probably a little funny looking for you, but the reason for this being in another class where your callbacks are defined is because this is all running in background workers that could run across different machines and you’re not gonna have access to wherever this was originally run. So by the time that it’s successful and calls the callback, you’re not gonna have that original, you know, Rails request or model instance anymore. That might be on a whole separate machine. And so this is going to require you to create a new class instance to go inside of that. And then you basically just use this jobs block to create all your new workers and that’s about it.3:59 And your jobs will run and that batch and then we’ll instantiate this callback class, whatever you define it as. It’ll pass in any extra options that you might want to add so you can go ahead and have that. But that’s kind of it. So let’s take a look at an example. We’re gonna try and use this concept as an example just so you can wrap your head around it.4:20 So what I’ve got here is a brand new Rails application and I’ve added a few gems to it using the Jumpstart template. But namely we’re gonna add Sidekiq batch. So you can either use Sidekiq Pro or Sidekiq batch. This one’s gonna allow you guys to clone this repo and check it out. So we’re gonna use that.4:38 Then we just need to create a couple jobs or, you know, a place that will create the batch at least. I’ve done that in a job itself. So I have a job that creates a batch and runs those. But you could do this inside of your models, your controllers, your plain old Ruby objects, anywhere you want to create the batch. All that code is going to obviously be run in Sidekiq though.4:58 So all of their worker stuff is going to run there. So let’s create a job really quick. Let’s say edit app jobs, create server job.rb. So our create server job is gonna have the perform method. And we need to include Sidekiq worker here.5:19 And the reason we want to use this instead of from Active Job is because with Sidekiq batches, you want to make sure that you stick to Sidekiq itself. You don’t want to have any Active Job stuff interfering with that and this will just work much, much nicer this way if you’re only interacting with Sidekiq. So you may need to convert some of your jobs over to pure Sidekiq, but it’s really easy. The only thing you really need to change is this line here, don’t inherit from Active Job and say perform async instead of perform later with Active Job. So that’s about it.5:55 And in here you want to do some work. So you want to do your work in parallel. That’s about it. You don’t need to do anything too fancy here. And wherever you want to trigger these workers is where you want to create the job stuff or the batch stuff.6:13 So let’s create another job called create cluster job. Class create cluster job, perform. And in here’s where we want to create our batch. So we’ll say batch equals Sidekiq batch.new, batch.description equals creating cluster. And then we’ll also say batch.on success.6:36 We want to call some sort of callback. So you can give it a class name here. You can also, at least with the Sidekiq batch, give it a string here and it will constantize that as well. So if we just had a class called created here, class created, we could do that. We would need to namespace it of course.6:58 Create cluster job created. That way we have access to that really easily. We can also do this just as a class. And then you can optionally pass in hash of options. So if you needed the cluster ID to be passed in, you could do that as well.7:18 So you could do something like this where you passed in the cluster ID. We don’t have a cluster model. So I’m just going to do 999 there. So we have some example for you to see. And then we’ll say batch.jobs do.7:28 And in here, we need to queue up our other jobs. So for example, we’ll just keep this easy. We’ll say five dot times. We will have this. We’ll take the number there.7:41 Create server job. Perform async will pass in I there. And that will be that. So this will make sure that those get added. And as you go and do this, it will queue up the jobs and start running them.8:02 And then inside your callback class here, you just need to say on success. And optionally, you can also do a def on complete, if you want. There is a nuance or difference between the two. I’ve always used on success. So that’s what we’re going to use here.8:20 Both of them take a status and options. So you can print out the status and the options as well. Options should be just this hash exactly as you pass it in. And then status will be the Sidekiq status. So let’s put a separator here and put created cluster.8:40 Period. So we should see that whenever these server jobs are finished. So this is the work that you’re going to want to do in parallel. So let’s just sleep for like a random amount of time between one and 10 seconds. And then we’ll print out creating server dot dot dot.8:59 And we can give it the ID here that we passed in. And then after it’s done, created. Just so we can see in our logs, the exact server lines that are matching each other. So we can count them and make sure that all of that is correct. ‘Cause we should only see this print out one time.9:26 We should see this stuff print out. However many times we call it. So five times in this case, 10 times if you change it to 10, whatever. And so we now have this job basically triggering a bunch of other ones. And this code is going to be run whenever it’s finished, just a single time.9:47 And it will be also be running inside kick. So that’s another thing to keep in mind. That’s why this class is instantiated because it’s running on whatever server that will run on inside a side kick. So it won’t have access to any of the instance variables or other local stuff. ‘Cause this will be a brand new instance of this created class when it runs.10:07 That’s why you need to have the ability to pass in these options to access say your cluster idea when this is finished to figure out which cluster you need to mark as active, for example. So with that said, we can go to the Rails console and try this out. Let’s start up side kick. So we have that running and we’ll say, create cluster job.new.perform. And if everything went correctly, we should see it queued up five jobs.10:34 That’s the idea of those jobs. Side kick and here we can see them running. And so we’ve got servers zero through four. So five of them and you can see that they’re running and whatever order side kick process them first. You can see three was finished before some of the other ones and all of that has completed successfully.10:56 So we see it started five jobs and it completed five jobs. And then at the very end, it started a new side kick batch callback worker, which is what the gem implemented for us triggers that as a job and then this is run just that single time. So this is cool because now we have the ability to go from a single thing running to multiple and then back to a single thing. And this is really awesome because it allows us to do this, distributed over our side kick cluster as well. This is very common to do with threading where you have just a single machine, spin up a bunch of threads, do some parallel work and then join and wait until those threads are finished and then do stuff in serial again.11:43 But with side kick, we can do this distributed across our side kick worker servers. So that is really cool and allows us to do a lot more this way in a lot nicer fashion because this can now be scaled up across multiple machines, which is fantastic. So side kick pro is where you can get this or use the gem. Definitely would recommend side kick pro. It’s gonna have better sport.12:08 It’s gonna probably implement all of this more efficiently or whatever. And it’s definitely going to get improved as time goes on. So this feature is definitely kind of a requirement as you get into building more advanced background jobs. You’re probably gonna need this for certain situations because you have a workflow that you need your code to run. And this is gonna help you build those workflows.12:29 For example, your created callback could actually start up another batch and trigger another one. So this is pretty interesting. You can have this stuff start chaining batches together. And that’s what I do in Hatchbox. So it’s pretty neat to see how all of this works and what complex stuff you can do.12:49 So this is very cool. And this is of course a complex feature that’s required for many applications. And so I hope you guys enjoyed this episode. If you want to see more advanced background workers or side kick stuff, let me know in the comments below. Now we’ll talk to you guys in the next episode.13:05 Peace.