Hi folks,
I’ve migrated the site to a new server setup. We’re now on a 4 vCPU / 8GB RAM / 160GB disk VPS at Digital Ocean with their managed MySQL database. (Which our replying_comments view overwhelms, so that’s off until we figure out how to tune it or redesign the feature.)
Our sysadmin @alynpost is resigning as moderator and sysadmin to focus on other projects. Prgmr will no longer be donating hosting. For security’s sake, I’ve reset all tokens and you’ll have to log in again - sorry for the hassle.
- Peter
Thank you @alynpost for providing the hosting and sysadmin help ‘til now.
Is there any way to support Lobsters’ hosting? Now that the server is not donated, I wouldn’t mind chipping in a little bit every month to subsidize operating costs.
+1 to this, I’d donate!
+1 I would too.
Ditto. I can provide Sysadmin expertise and/or moderator time as well, if desired.
Ever hear the internet rule of thumb: “Anyone who asks to be a moderator should never be made one.” ?
Ditto re: sysadmin / SRE work.
Ditto, can volunteer with SRE work
+1
Is there any risk that Lobste.rs could go offline in the future due to running costs?
No. The new hosting bill is $75/month, which I don’t mind at all.
Isn’t that very overpriced? 40€/month at hetzner gets you a dedicated machine with a Ryzen 5 3600, 64GB of RAM and 512GB of SSD on RAID1 (no affiliation or anything, it’s just the provider I know).
Hetzner also just uses electricity from sustainable sources, while with digital ocean it depends on the location
Hetzner is the goat! I use them for my VPS and it’s the best deal I’ve seen yet for cloud services. The fact that they’re environmentally friendly as well makes it that much better!
Does Hetzner have managed MySQL? Seems like it’s a big hassle removed there.
You can rent a managed server with Hetzner and they have a panel to install and mange MySQL on it, but I don’t think it’s comparable to DigitalOcean’s managed offerings.
Would be really interesting to hear what they’re doing with “managed”. Because based on the prices I’d say prgrmr.com is also not cheap compared to the hardware you get.
Does hosting lobster requires lots of CPU or RAM?
It’s Rails. So both :)
#rust
Would you consider accepting donations for hosting?
I appreciate the offers but prefer not to, no. Still looking for someone to print-on-demand stickers, though.
I’ll buy $75 worth of stickers every month to show my appreciation.
Minor dissenting opinion:
I support a lot of people on Patreon and expect nothing in return. Chipping in $5/month to Lobste.rs because I like the community and the stuff that gets shared here isn’t a tall order, and won’t come with any entitlement. (A lot of the people I support are artists and content creators that are usually in high demand from the rest of the community.)
I can’t speak for the rest of the community, but I don’t think I’m particularly saintly in this regard. :P
If the expenses grow, please don’t rule this option out entirely.
It seems to me that the expectation comes from the design of sites which ask for monthly donations. Thinking out loud here, but a donations system which really was just a donations system, something more similar to ko-fi and didn’t have names attached, might help highlight the fact that by donating one is helping out rather than a new account tier?
I personally also donate on Patreon and expect nothing.
Thank you! That is a great attitude.
I have one concern though. What happens when lobste.rs keeps growing and the bill increases? What is your maximum you would spend on the site? Wouldn‘t it be better to care about that rather earlier than later?
By design, Lobsters grows pretty slowly. I’m thinking of design decisions like invites vs open signups, and a narrow focus rather than a subreddit for everything. Growth is not a goal like it would be in a startup, and I’d pause invites if we saw some kind of huge spike.
Right off we should have plenty of spare capacity. I aimed to overprovision this new server and we’ll see if I eyeballed that correctly as we reach peak traffic during the US work week. If the hosting bill goes to about 10x current I’ll start reconsidering donations. But that may never happen! Hosting costs slowly decline as power gets cheaper, data centers get built, and fiber gets laid. Lobsters is cheap to run because it’s a CRUD SQL app pushing around text a few kilobytes at a time and our size increases slowly. I hope not to jinx it, but it seems likely that our hosting bill is flat or declines over the next decade.
Not print-on-demand afaik, but Sticker Mule has been great to work with in the past for me.
Redbubble do print on demand for stickers, iirc.
I’m definitely in the market for some stickers if you find a service or have any left over from the first batch!
I had a mild panic when my first TOTP code didn’t work. Second try was the charm, though.
Today I learned that Lobsters has TOTP
I think that the TOTP codes are on a subtly different clock now. They seem to expire a little bit early.
uh nice to know this exists :D
Ah that explains why I was logged out. If you ever decide you’re accepting donations I’d be happy to give some. Out of curiosity, did anything bad happen between alynpost and or was this a planned action?
It’s mentioned in the post details, emphasis mine:
Cheers @alynpost o7
Not sure if it’s just me, but since this hosting update, the favicon in my browser tab is just a generic icon now, not the Lobsters one.
For me, it presents the previous icon as usual (both in Chrome, where I’m logged on, and on another browser I use for work.
It’s being served correctly at the moment. Browsers tend to cache favicons way more aggressively than they ought to, especially when they’re temporarily in a broken state. :/
Any ideas when message reply notifications will come back? Without that, you can’t have a dialogue without polling every story that you’ve commented on and that’s likely to make the quality of discussions suffer a lot.
I’ve exhausted DigitalOcean’s support for their managed database service, so it’ll happen as soon as I can get a couple hours to stand up a database on a vps. Your threads page is maybe a more convenient place to look for replies in the meantime.
Any update on this? Since the migration, I’ve seen a noticeable drop in the depth of discussions. Even when I dig into the ‘Your Threads’ page and reply, it’s pretty rare for the person that I reply to will do the same. The ‘Replies’ link is still there, so people may not realise that it doesn’t actually work anymore and it requires a lot more effort to find replies.
I would be nice if you can share the issues of DO’s managed database, so we can avoid or keep an eye on it when using managed database in side projects. Is it just being slow in general, or limiting QPS to lobsters.
I’ve been talking about it in the chat room since the move. Our problem is a combination of the issue explained in the post (MySQL doesn’t push the
whereclause down into the view, so does a full table scan of all 300kcommentsinstead of using an index to look at ~4 rows) and general slowness, which may be due to particularities of server config, other mariadb/mysql differences, or simply under-provisioning. I traded messages with DO support for a while and was escalated past tier 1 support, but it was not a productive conversation.In typical busy times Lobsters only does 6-7 QPS. DO considered a narrow fixed-width table with a million rows to be large, so maybe they also consider that a high rate, but QPS didn’t explicitly come up. We did immediately bump our head on a max connection limit of 60 when we cut over and had to reduce puma workers (if they can’t get a db connection they throw 500s at the client).
So, about that Postgres support… :P
It’s tempting. The recursive CTEs could provide nice features for threads. Off the top of my head there’s some maintenance scripts to update (database backup script, any kind of regular maintenance like if vacuum is still a thing), then making very sure we’re not trashing data with a charset mismatch or other subtle difference (how?), and tidying a couple mysqlisms in the codebase (probably done if I get #861 merged). So it’s not a lot more work than standing up mariadb in the first place.
@law @skelly @benj @pronoiac - you put your hand up in this thread for sysadmin tasks, do any of you have experience along these lines, time to help update ansible + answer questions as I stand up a box, and a block of a few hours (outside US central working hours) to do the cutover in the next week or so?
I don’t mind looking at the PostgreSQL parts specifically (i.e. test/fix the Lobsters codebase) if that’s the way you decide to go, or at MySQL optimisations if we decide to stick with that. I have nothing super-pressing going on at the moment, so I should have enough time, it’s mostly just a matter of wanting to be reasonably sure it’s actually going to be used (the previous PostgreSQL PRs have lingered for quite a while).
Yes. The vast majority of my coding expertise is in Python, but the rest of what you describe is right up my alley. I can send you more examples of my experience/bona-fides, what’s a good email for you (or would you prefer a DM?)
I’ll be starting a new job next week (the comment was before I got the job :-) so I think after a week or so I’ll know how my schedule looks like. I’m very familiar with general Linux server config/maintenance from a Linux cert I took a few years ago. If by February you still need help I should be a lot more available
It looks like the Lobsters Twitter bot has stopped posting stories. Might it be related to this?
Lobsters doesn’t need much servers as such, the registration is limited because of invite system, You can only post links, no images, so, Bandwidth usage is too less, Even a light server would be more than enough for this.
I am wondering if a simple $5 DO instance with sqlite3 could handle all traffic, including search support from FST3/4 extensions.