It’s great to see Sidekiq still going strong after all these years! The iteration feature is really interesting, and I can think of many situations where it would have come in very handy but I had to reach for a more complicated solution. Don’t think I’ve seen that in similar libraries.
Every big Rails team I have worked on ends up building their own janky version of it (I’ve built a couple), very nice to have Sidekiq provide an implementation.
It was an absolute delight to discover that modern CSS, with its support for variables, can be used without a build process.
I went on my journey to that realization over the last 18 months. I haven’t phased out SCSS everywhere yet, but once native CSS nesting crosses 95% support, I’ll default to vanilla CSS, and be skeptical of deviating in any new projects.
I was curious how this looked in terms of code… I’m a little surprised at how concise/spartan it is. There doesn’t seem to be a css reset even? Is that not needed anymore?
There doesn’t seem to be a css reset even? Is that not needed anymore?
Modern browsers differ much less in terms of default styles, so not really, unless you want absolutely pixel perfect equivalent design between browsers.
The main purpose was to DRY up the CSS. It makes theming possible but not really something that makes sense for Sidekiq and my users so I haven’t spent any time or thought on it.
I hope this isn’t too off topic but it’s on queues for rails.
Did anyone switch from Sidekiw to SolidQueue and can share the experience?
I am curious because while sidekiq is nice I do think that with the limited scope that such systems have having to buy the for-pay features is something that might not be feasible for small or non profit projects when they simply are relying heavily on queues because of the application.
In general running your queue on the same database as your app is gonna be a bad time, performance wise. Now, solid queue can go on a different database on the same machine (SQLite) but usually the database is the first thing that needs to be scaled and competing with it for resources is also not a great idea. If you run it, I would run it on a different machine and different than your main database. At that point you have to pay for another box/service for your queue datastore anyway.
In general running your queue on the same database as your app is gonna be a bad time, performance wise.
Could you go into that a more?
Sounds a bit like something breaking down quickly. We are talking about relatively simple tables, with relatively simple queries. Stuff where I’d expect that more complex applications would potentially dozens of requests (or alternatively somewhat complex joins and conditions, queries), so it feels like a query that just locks the next thing shouldn’t really carry weight overall.
usually the database is the first thing that needs to be scaled
That also really depends on the application. Both on doing silly things in application code and doing silly things regarding database query. Doing a DB server with let’s say 1 TB NVMe and 128 GiB and an identical secondary for around ~150 USD can get you really far, even if you do a bit more complex stuff (eg. on-demand GIS with measuring, manipulating and relating polygons) even if you don’t invest a lot of effort into making everything super efficient. And having a couple of hundred thousand entries for your queue doesn’t sound like it should be that big of a deal. And if it is then just use a separate DB for your queue?
So in other words: Yes, you are technically competing for resources, but we are talking about a pretty easy task for any DB?
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
It’s not about “complex” or “easy” or not, it’s about load and locking. Congestion.
And if it is then just use a separate DB for your queue?
I’m not sure why there’s a question mark. That was a recommendation in my comment. The next logical step though was…if you’re going to use a different data store anyway, why not just use redis/keyval which has persistent queue data structures.
One weird thing though: it’s surprisingly hard to migrate off of a queue backend (or can be). It’s easier than migrating from MySQL to PostgreSQL, but perhaps harder than people realize.
We use delayedjob for a legacy but important app, and without realizing it we’re relying on database transaction semantics for some critical behavior. The effort and cost to identify and replicate all that behavior that would allow us to move to a different queue backend isn’t justifiable, so we are kinda just…stuck with it. At least for now, on that one app.
Even migrating from a queue in the same DB to a different DB could yield slightly different behavior. Not saying it can’t be done or that it will be prohibitively difficult or expensive, but rather: make a plan for how you will possibly scale your database and queue systems in the future now.
For me: I choose Sidekiq and keyval/redis. In addition to what I’ve already mentioned, I know I can get support if required and I know it won’t be community abandonware like resque or webpacker. I also have a personal relationship with Mike and hang with him at confs and he’s generally accessible online and active in the communities. Mentioning both for disclosure and to contrast with Dave, who isn’t accessible unless you’re in an inner circle.
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
Are you able to provide more insight there?
In my experience say you take a DB setup like to above for ~150 USD (or 76 without a replica - in other words we will only use the replica for fail over, no reads from it) and with lets say 10k active users a day, let’s say that gives you an average of 1k DB queries/second on average with let’s say 10k DB queries/second in more peak time. In my experience for relatively basic CRUD apps with little optimization and somewhat idiomatic Rails code throwing out JSON for web and mobile apps to ingest will be barely noticeable in terms of DB load. Even if you don’t optimize it. Even if you turn logging up for every statement for monitoring. Even if you use something like ZFS so you can make point in time snapshots. Yet your Rails application will start to struggle in peak times, eating up CPU.
But I have to say while I work on a Rails app right now that is way bigger than the above I am certainly not an expert when it comes to Rails or even Ruby.
It’s great to see Sidekiq still going strong after all these years! The iteration feature is really interesting, and I can think of many situations where it would have come in very handy but I had to reach for a more complicated solution. Don’t think I’ve seen that in similar libraries.
It’s directly adapted from https://github.com/Shopify/job-iteration
Every big Rails team I have worked on ends up building their own janky version of it (I’ve built a couple), very nice to have Sidekiq provide an implementation.
Congrats on the release!
I went on my journey to that realization over the last 18 months. I haven’t phased out SCSS everywhere yet, but once native CSS nesting crosses 95% support, I’ll default to vanilla CSS, and be skeptical of deviating in any new projects.
I was curious how this looked in terms of code… I’m a little surprised at how concise/spartan it is. There doesn’t seem to be a css reset even? Is that not needed anymore?
Stylesheet:
https://github.com/sidekiq/sidekiq/blob/main/web/assets/stylesheets/style.css
Layout (html page template):
https://github.com/sidekiq/sidekiq/blob/main/web/views/layout.erb
A view:
https://github.com/sidekiq/sidekiq/blob/main/web/views/scheduled.erb
Looks quite reasonable - but also looks like they’re not really using their semantic variables that much (yet)? Like theme/custom color support?
Modern browsers differ much less in terms of default styles, so not really, unless you want absolutely pixel perfect equivalent design between browsers.
The main purpose was to DRY up the CSS. It makes theming possible but not really something that makes sense for Sidekiq and my users so I haven’t spent any time or thought on it.
Great job!
I hope this isn’t too off topic but it’s on queues for rails.
Did anyone switch from Sidekiw to SolidQueue and can share the experience?
I am curious because while sidekiq is nice I do think that with the limited scope that such systems have having to buy the for-pay features is something that might not be feasible for small or non profit projects when they simply are relying heavily on queues because of the application.
In general running your queue on the same database as your app is gonna be a bad time, performance wise. Now, solid queue can go on a different database on the same machine (SQLite) but usually the database is the first thing that needs to be scaled and competing with it for resources is also not a great idea. If you run it, I would run it on a different machine and different than your main database. At that point you have to pay for another box/service for your queue datastore anyway.
My preference has been to stick with Sidekiq.
Could you go into that a more?
Sounds a bit like something breaking down quickly. We are talking about relatively simple tables, with relatively simple queries. Stuff where I’d expect that more complex applications would potentially dozens of requests (or alternatively somewhat complex joins and conditions, queries), so it feels like a query that just locks the next thing shouldn’t really carry weight overall.
That also really depends on the application. Both on doing silly things in application code and doing silly things regarding database query. Doing a DB server with let’s say 1 TB NVMe and 128 GiB and an identical secondary for around ~150 USD can get you really far, even if you do a bit more complex stuff (eg. on-demand GIS with measuring, manipulating and relating polygons) even if you don’t invest a lot of effort into making everything super efficient. And having a couple of hundred thousand entries for your queue doesn’t sound like it should be that big of a deal. And if it is then just use a separate DB for your queue?
So in other words: Yes, you are technically competing for resources, but we are talking about a pretty easy task for any DB?
Yeah it shouldn’t be a problem, but the database needs to be operated properly like having the right indexes and using SKIP LOCKED for queue ops.
https://web.archive.org/web/20160416164956/https://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5/
(Sadly EnterpriseDB have fucked the 2nd Quadrant blog.)
Yes, but that’s Sidekiq’s or SolidQueue’s job.
Ages ago built my own queues, both in Redis and PG.
I work for Heroku where I get performance tickets and I’ve seen hundreds, if not thousands of Rails apps in a performance context. I also am on the Nate Berkopec performance slack. And database load is pretty much the primary bottleneck for most rails apps.
It’s not about “complex” or “easy” or not, it’s about load and locking. Congestion.
I’m not sure why there’s a question mark. That was a recommendation in my comment. The next logical step though was…if you’re going to use a different data store anyway, why not just use redis/keyval which has persistent queue data structures.
One weird thing though: it’s surprisingly hard to migrate off of a queue backend (or can be). It’s easier than migrating from MySQL to PostgreSQL, but perhaps harder than people realize.
We use delayedjob for a legacy but important app, and without realizing it we’re relying on database transaction semantics for some critical behavior. The effort and cost to identify and replicate all that behavior that would allow us to move to a different queue backend isn’t justifiable, so we are kinda just…stuck with it. At least for now, on that one app.
Even migrating from a queue in the same DB to a different DB could yield slightly different behavior. Not saying it can’t be done or that it will be prohibitively difficult or expensive, but rather: make a plan for how you will possibly scale your database and queue systems in the future now.
For me: I choose Sidekiq and keyval/redis. In addition to what I’ve already mentioned, I know I can get support if required and I know it won’t be community abandonware like resque or webpacker. I also have a personal relationship with Mike and hang with him at confs and he’s generally accessible online and active in the communities. Mentioning both for disclosure and to contrast with Dave, who isn’t accessible unless you’re in an inner circle.
Are you able to provide more insight there?
In my experience say you take a DB setup like to above for ~150 USD (or 76 without a replica - in other words we will only use the replica for fail over, no reads from it) and with lets say 10k active users a day, let’s say that gives you an average of 1k DB queries/second on average with let’s say 10k DB queries/second in more peak time. In my experience for relatively basic CRUD apps with little optimization and somewhat idiomatic Rails code throwing out JSON for web and mobile apps to ingest will be barely noticeable in terms of DB load. Even if you don’t optimize it. Even if you turn logging up for every statement for monitoring. Even if you use something like ZFS so you can make point in time snapshots. Yet your Rails application will start to struggle in peak times, eating up CPU.
But I have to say while I work on a Rails app right now that is way bigger than the above I am certainly not an expert when it comes to Rails or even Ruby.
That’s 6 queries per minute per user on average, which seems like A LOT. How many queries per page load is that?