It’s great to see Sidekiq still going strong after all these years! The iteration feature is really interesting, and I can think of many situations where it would have come in very handy but I had to reach for a more complicated solution. Don’t think I’ve seen that in similar libraries.
Every big Rails team I have worked on ends up building their own janky version of it (I’ve built a couple), very nice to have Sidekiq provide an implementation.
It was an absolute delight to discover that modern CSS, with its support for variables, can be used without a build process.
I went on my journey to that realization over the last 18 months. I haven’t phased out SCSS everywhere yet, but once native CSS nesting crosses 95% support, I’ll default to vanilla CSS, and be skeptical of deviating in any new projects.
I was curious how this looked in terms of code… I’m a little surprised at how concise/spartan it is. There doesn’t seem to be a css reset even? Is that not needed anymore?
There doesn’t seem to be a css reset even? Is that not needed anymore?
Modern browsers differ much less in terms of default styles, so not really, unless you want absolutely pixel perfect equivalent design between browsers.
The main purpose was to DRY up the CSS. It makes theming possible but not really something that makes sense for Sidekiq and my users so I haven’t spent any time or thought on it.
That makes sense. I suppose I expected to find a complete bootstrap replacement - rather than “just the laces we thread”. Obviously the simpler solution is better when that’s all you need.
I hope this isn’t too off topic but it’s on queues for rails.
Did anyone switch from Sidekiw to SolidQueue and can share the experience?
I am curious because while sidekiq is nice I do think that with the limited scope that such systems have having to buy the for-pay features is something that might not be feasible for small or non profit projects when they simply are relying heavily on queues because of the application.
In general running your queue on the same database as your app is gonna be a bad time, performance wise. Now, solid queue can go on a different database on the same machine (SQLite) but usually the database is the first thing that needs to be scaled and competing with it for resources is also not a great idea. If you run it, I would run it on a different machine and different than your main database. At that point you have to pay for another box/service for your queue datastore anyway.
In general running your queue on the same database as your app is gonna be a bad time, performance wise.
Could you go into that a more?
Sounds a bit like something breaking down quickly. We are talking about relatively simple tables, with relatively simple queries. Stuff where I’d expect that more complex applications would potentially dozens of requests (or alternatively somewhat complex joins and conditions, queries), so it feels like a query that just locks the next thing shouldn’t really carry weight overall.
usually the database is the first thing that needs to be scaled
That also really depends on the application. Both on doing silly things in application code and doing silly things regarding database query. Doing a DB server with let’s say 1 TB NVMe and 128 GiB and an identical secondary for around ~150 USD can get you really far, even if you do a bit more complex stuff (eg. on-demand GIS with measuring, manipulating and relating polygons) even if you don’t invest a lot of effort into making everything super efficient. And having a couple of hundred thousand entries for your queue doesn’t sound like it should be that big of a deal. And if it is then just use a separate DB for your queue?
So in other words: Yes, you are technically competing for resources, but we are talking about a pretty easy task for any DB?
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
It’s not about “complex” or “easy” or not, it’s about load and locking. Congestion.
And if it is then just use a separate DB for your queue?
I’m not sure why there’s a question mark. That was a recommendation in my comment. The next logical step though was…if you’re going to use a different data store anyway, why not just use redis/keyval which has persistent queue data structures.
One weird thing though: it’s surprisingly hard to migrate off of a queue backend (or can be). It’s easier than migrating from MySQL to PostgreSQL, but perhaps harder than people realize.
We use delayedjob for a legacy but important app, and without realizing it we’re relying on database transaction semantics for some critical behavior. The effort and cost to identify and replicate all that behavior that would allow us to move to a different queue backend isn’t justifiable, so we are kinda just…stuck with it. At least for now, on that one app.
Even migrating from a queue in the same DB to a different DB could yield slightly different behavior. Not saying it can’t be done or that it will be prohibitively difficult or expensive, but rather: make a plan for how you will possibly scale your database and queue systems in the future now.
For me: I choose Sidekiq and keyval/redis. In addition to what I’ve already mentioned, I know I can get support if required and I know it won’t be community abandonware like resque or webpacker. I also have a personal relationship with Mike and hang with him at confs and he’s generally accessible online and active in the communities. Mentioning both for disclosure and to contrast with Dave, who isn’t accessible unless you’re in an inner circle.
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
Are you able to provide more insight there?
In my experience say you take a DB setup like to above for ~150 USD (or 76 without a replica - in other words we will only use the replica for fail over, no reads from it) and with lets say 10k active users a day, let’s say that gives you an average of 1k DB queries/second on average with let’s say 10k DB queries/second in more peak time. In my experience for relatively basic CRUD apps with little optimization and somewhat idiomatic Rails code throwing out JSON for web and mobile apps to ingest will be barely noticeable in terms of DB load. Even if you don’t optimize it. Even if you turn logging up for every statement for monitoring. Even if you use something like ZFS so you can make point in time snapshots. Yet your Rails application will start to struggle in peak times, eating up CPU.
But I have to say while I work on a Rails app right now that is way bigger than the above I am certainly not an expert when it comes to Rails or even Ruby.
Here’s an article that shows load average on basically the smallest crud app you can think of, and it’s also open source. In this case the article walks through finding and eliminating a bug/problem, but it shows the correlation between usage and load to help build up intuition.
You get something related to the resource, to authenticate the resource (if you use any kind of ACL mechanism like cancancan), might be two queries if it’s for example that being based on relationships to other users
You get the thing that the endpoint is about
It might be something more involved where you end up doing additional queries
That’s already at least 3-4 queries. And that’s the most basic one request can be if you separate authentication and authorization into their own things.
Now think about the fact that oftentimes a page load does more than one ajax request. So you have a multitude of them. Eg. you might have whatever the page is about, then in addition checking something like notification, some inbox. You might have a general endpoint for the current user profile, settings, etc. You might in some situations do non user triggered updates, for example to mark stuff as read or similar. In addition if it’s something social you might have something to give suggestions. etc. So the first page load have multiple of these 3-4 queries. Sometimes you also want to do some more generic calls. For example a geoip call that might still be authenticated or simply load some news that might still be authenticated or use it based on the user, so again you have to do the authentication part. In addition sometimes stuff ends up in the queue, to come back to topic. So that ID there has to be queried again. Sometimes you interact with the outside world, so you query, the communicate, then query again. Sometimes stuff is callbacks. Sometimes requests fail and are retried (again, mobile app, so network quality varies, it’s an outdoorsy app). Then you have some caching queries and since it’s Rails there is the classic touch: true association.
If you then go further like having activities, you might do stuff where user uploads something, then uploads pictures individually so that it’s not one big request, but if you are a mobile app multiple so it’s easier in terms of retrying when only one changes (think wifi-cell switch).
Now if your site is more complex you multiply those 3 with even more. Very different of course if you just use Rails directly for rendering. But if you have endpoints for a lot of features that stuff of course adds up, when they are queried independently, so the client can compose it.
So overall you end up with a bigger amount of initial queries (because many AJAX calls) on first page load and then reduce it. Same for when the app is opened. Couple of initial queries, that often mostly do the authentication part and then their main thing.
Just checked. The initial page load for a logged in user is 12 AJAX requests. All with at least Authentication -> ACL -> main request(s). Could be optimized for sure.
But then again, as mentioned the 75 USD/month primary DB is pretty bored, as it should it. So the need to improve there hasn’t yet arisen. The additional Rails instances are also mostly therefor failover. Compared to so many other things the infrastructure cost is pretty much irrelevant. Pretty much every few minutes of unnecessary meetings covers the DB costs for months.
All of that is without any requests not initiated by users directly. Think of accepting callbacks (payment, etc.), reporting, monitoring, altering, etc. related queries regarding the current state of the DB. Or status endpoints that trigger requests that are pulled in regular intervals 24/7. Not all of these go through Rails of course. A lot of them go either directly to the DB or through something else. And it feels like the queue would be nothing more than one of these small side things.
It’s great to see Sidekiq still going strong after all these years! The iteration feature is really interesting, and I can think of many situations where it would have come in very handy but I had to reach for a more complicated solution. Don’t think I’ve seen that in similar libraries.
It’s directly adapted from https://github.com/Shopify/job-iteration
Every big Rails team I have worked on ends up building their own janky version of it (I’ve built a couple), very nice to have Sidekiq provide an implementation.
Congrats on the release!
I went on my journey to that realization over the last 18 months. I haven’t phased out SCSS everywhere yet, but once native CSS nesting crosses 95% support, I’ll default to vanilla CSS, and be skeptical of deviating in any new projects.
I was curious how this looked in terms of code… I’m a little surprised at how concise/spartan it is. There doesn’t seem to be a css reset even? Is that not needed anymore?
Stylesheet:
https://github.com/sidekiq/sidekiq/blob/main/web/assets/stylesheets/style.css
Layout (html page template):
https://github.com/sidekiq/sidekiq/blob/main/web/views/layout.erb
A view:
https://github.com/sidekiq/sidekiq/blob/main/web/views/scheduled.erb
Looks quite reasonable - but also looks like they’re not really using their semantic variables that much (yet)? Like theme/custom color support?
Modern browsers differ much less in terms of default styles, so not really, unless you want absolutely pixel perfect equivalent design between browsers.
The main purpose was to DRY up the CSS. It makes theming possible but not really something that makes sense for Sidekiq and my users so I haven’t spent any time or thought on it.
That makes sense. I suppose I expected to find a complete bootstrap replacement - rather than “just the laces we thread”. Obviously the simpler solution is better when that’s all you need.
Great job!
I hope this isn’t too off topic but it’s on queues for rails.
Did anyone switch from Sidekiw to SolidQueue and can share the experience?
I am curious because while sidekiq is nice I do think that with the limited scope that such systems have having to buy the for-pay features is something that might not be feasible for small or non profit projects when they simply are relying heavily on queues because of the application.
In general running your queue on the same database as your app is gonna be a bad time, performance wise. Now, solid queue can go on a different database on the same machine (SQLite) but usually the database is the first thing that needs to be scaled and competing with it for resources is also not a great idea. If you run it, I would run it on a different machine and different than your main database. At that point you have to pay for another box/service for your queue datastore anyway.
My preference has been to stick with Sidekiq.
Could you go into that a more?
Sounds a bit like something breaking down quickly. We are talking about relatively simple tables, with relatively simple queries. Stuff where I’d expect that more complex applications would potentially dozens of requests (or alternatively somewhat complex joins and conditions, queries), so it feels like a query that just locks the next thing shouldn’t really carry weight overall.
That also really depends on the application. Both on doing silly things in application code and doing silly things regarding database query. Doing a DB server with let’s say 1 TB NVMe and 128 GiB and an identical secondary for around ~150 USD can get you really far, even if you do a bit more complex stuff (eg. on-demand GIS with measuring, manipulating and relating polygons) even if you don’t invest a lot of effort into making everything super efficient. And having a couple of hundred thousand entries for your queue doesn’t sound like it should be that big of a deal. And if it is then just use a separate DB for your queue?
So in other words: Yes, you are technically competing for resources, but we are talking about a pretty easy task for any DB?
Yeah it shouldn’t be a problem, but the database needs to be operated properly like having the right indexes and using SKIP LOCKED for queue ops.
https://web.archive.org/web/20160416164956/https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/
(Sadly EnterpriseDB have fucked the 2nd Quadrant blog.)
Yes, but that’s Sidekiq’s or SolidQueue’s job.
Ages ago built my own queues, both in Redis and PG.
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
It’s not about “complex” or “easy” or not, it’s about load and locking. Congestion.
I’m not sure why there’s a question mark. That was a recommendation in my comment. The next logical step though was…if you’re going to use a different data store anyway, why not just use redis/keyval which has persistent queue data structures.
One weird thing though: it’s surprisingly hard to migrate off of a queue backend (or can be). It’s easier than migrating from MySQL to PostgreSQL, but perhaps harder than people realize.
We use delayedjob for a legacy but important app, and without realizing it we’re relying on database transaction semantics for some critical behavior. The effort and cost to identify and replicate all that behavior that would allow us to move to a different queue backend isn’t justifiable, so we are kinda just…stuck with it. At least for now, on that one app.
Even migrating from a queue in the same DB to a different DB could yield slightly different behavior. Not saying it can’t be done or that it will be prohibitively difficult or expensive, but rather: make a plan for how you will possibly scale your database and queue systems in the future now.
For me: I choose Sidekiq and keyval/redis. In addition to what I’ve already mentioned, I know I can get support if required and I know it won’t be community abandonware like resque or webpacker. I also have a personal relationship with Mike and hang with him at confs and he’s generally accessible online and active in the communities. Mentioning both for disclosure and to contrast with Dave, who isn’t accessible unless you’re in an inner circle.
Are you able to provide more insight there?
In my experience say you take a DB setup like to above for ~150 USD (or 76 without a replica - in other words we will only use the replica for fail over, no reads from it) and with lets say 10k active users a day, let’s say that gives you an average of 1k DB queries/second on average with let’s say 10k DB queries/second in more peak time. In my experience for relatively basic CRUD apps with little optimization and somewhat idiomatic Rails code throwing out JSON for web and mobile apps to ingest will be barely noticeable in terms of DB load. Even if you don’t optimize it. Even if you turn logging up for every statement for monitoring. Even if you use something like ZFS so you can make point in time snapshots. Yet your Rails application will start to struggle in peak times, eating up CPU.
But I have to say while I work on a Rails app right now that is way bigger than the above I am certainly not an expert when it comes to Rails or even Ruby.
Here’s an article that shows load average on basically the smallest crud app you can think of, and it’s also open source. In this case the article walks through finding and eliminating a bug/problem, but it shows the correlation between usage and load to help build up intuition.
https://schneems.com/2017/07/18/how-i-reduced-my-db-server-load-by-80/
I don’t really understand the relevance. As mentioned the database is mostly bored even during peak times. That’s the whole point I’m making.
That’s 6 queries per minute per user on average, which seems like A LOT. How many queries per page load is that?
Let’s see
That’s already at least 3-4 queries. And that’s the most basic one request can be if you separate authentication and authorization into their own things.
Now think about the fact that oftentimes a page load does more than one ajax request. So you have a multitude of them. Eg. you might have whatever the page is about, then in addition checking something like notification, some inbox. You might have a general endpoint for the current user profile, settings, etc. You might in some situations do non user triggered updates, for example to mark stuff as read or similar. In addition if it’s something social you might have something to give suggestions. etc. So the first page load have multiple of these 3-4 queries. Sometimes you also want to do some more generic calls. For example a geoip call that might still be authenticated or simply load some news that might still be authenticated or use it based on the user, so again you have to do the authentication part. In addition sometimes stuff ends up in the queue, to come back to topic. So that ID there has to be queried again. Sometimes you interact with the outside world, so you query, the communicate, then query again. Sometimes stuff is callbacks. Sometimes requests fail and are retried (again, mobile app, so network quality varies, it’s an outdoorsy app). Then you have some caching queries and since it’s Rails there is the classic
touch: trueassociation.If you then go further like having activities, you might do stuff where user uploads something, then uploads pictures individually so that it’s not one big request, but if you are a mobile app multiple so it’s easier in terms of retrying when only one changes (think wifi-cell switch).
Now if your site is more complex you multiply those 3 with even more. Very different of course if you just use Rails directly for rendering. But if you have endpoints for a lot of features that stuff of course adds up, when they are queried independently, so the client can compose it.
So overall you end up with a bigger amount of initial queries (because many AJAX calls) on first page load and then reduce it. Same for when the app is opened. Couple of initial queries, that often mostly do the authentication part and then their main thing.
Just checked. The initial page load for a logged in user is 12 AJAX requests. All with at least Authentication -> ACL -> main request(s). Could be optimized for sure.
But then again, as mentioned the 75 USD/month primary DB is pretty bored, as it should it. So the need to improve there hasn’t yet arisen. The additional Rails instances are also mostly therefor failover. Compared to so many other things the infrastructure cost is pretty much irrelevant. Pretty much every few minutes of unnecessary meetings covers the DB costs for months.
All of that is without any requests not initiated by users directly. Think of accepting callbacks (payment, etc.), reporting, monitoring, altering, etc. related queries regarding the current state of the DB. Or status endpoints that trigger requests that are pulled in regular intervals 24/7. Not all of these go through Rails of course. A lot of them go either directly to the DB or through something else. And it feels like the queue would be nothing more than one of these small side things.
I hope that clarifies things a bit. :)