I really enjoy the few posts recently about migrating away from the cloud to e.g. colocating or self-hosting. Kind of makes me feel like returning to simpler solutions (though colocating arguably is more involved).
I think these posts were inevitable given how cloud vendors haven’t changed their prices for the last 10 years even though the hardware got cheaper and more efficient in every way. With them reaping larger and larger margins from improving hardware underneath, it makes more and more sense not to pay that premium.
This is an interesting thought. It’s certainly more involved in terms of “get a screwdriver and hump your 20 kg server into your car and drive it to Nebraska” work, but you probably spend a lot less time staring at the AWS dashboard agonizing over what the hell some particular security policy means.
As someone who knows how to host something but is not deeply versed in AWS: colocating a server would be a breeze to me (I would know, I’m doing it) meanwhile opening the AWS console to figure out how to setup an ISAM role or not fuck up my S3 bucket permissions or figure out how to meter billing or figure out which of the 70 kinds of instances would be the least cost inefficient all sounds way more involved to me. It also sounds like hell.
I’m in exactly the opposite position - I have years of experience with AWS (and yes, it is hell) but would not be that confident setting up a colocated server. Part of my wants to do it as a sideproject, but I don’t think I can justify the cost (and, frankly, it’s a lot more bandwidth and compute than I need).
It’s actually easy to try, I believe. How much bandwidth and compute do you need? What kinds of side projects do you have now? I think just renting a 4€ VPS from hetzner gets you that first experience.
Or find a used NUC or rack server on eBay and hook it up to your home router. It’s not the same experience as colocation obviously, but I would imagine there is a lot of overlap (I have been doing this for years, but have never colocated). See also r/homelab.
If it’s just a side project, that’s probably going to be fine.
Yeah, my current setup is a very basic version of this i.e. a rpi4 plugged into my router. Current flat is a bit space constrained, but setting up a better homelab space is on my wishlist for when we size up!
Similar enough, but I think for that purpose, people already have their workstation at home. I still believe putting the thing “live” on the internet brings the value, if only to make you ask LLMs how to setup DNS and a simple basic firewall and perhaps some automatic security updates for when you forget about your server for two months.
If you don’t do it regularly, you’ll forget about it, but if you do it irregularly enough, you start to appreciate some of these hosted, cloud solutions.
You often hear this story in tech news: some project got increasingly popular, so they had to migrate from their simple, one-computer setup to some kind of autoscaling cloud hosting solution to meet demands
Does someone have an example of this? Given that EC2 et all are the cost effective solution for so few groups (most are either too small, to big, and have workloads too persistent to benefit) I wonder what someone who actually makes the move in that direction and benefits looks like
I once had a client that strongly believed that they have that use case and because it seemed like an obvious case, my guess was that they were in fact reaping the benefit like no other.
Then I did the math and not even in that “perfect example” (strong traffic spikes only on certain days/hour in a week) it did not make sense. Even though their setup was built that way from the beginning.
So I helped changing it and the result was “Wow, it’s so much faster”, while saving lots of costs and complexity.
They switched to dedicated servers (and some vservers). The scale up resources are now just always there and are used for failover, etc.
So in short: I yet have to see a company where the benefits reach beyond a theoretical assumptions.
I’m curious how it went? I worked at a major corp that went from in prem to cloud also, and it was such a disaster that only pride red sunk cost prevent them from reverting
I can’t help but feel that Zig is throwing out the baby with the bathwater with this move. AWS’s bandwidth costs are egregious, and peaking at ~$700/mo of costs for a static site definitely warrants some kind of change, but I just feel like moving to a single dedicated server instance is not the best play
There are some obvious downsides to just running everything on a single dedicated instance, like getting rid of geographic distribution. It also takes work to make sure the kernel, HTTP server, etc. are all kept up-to-date, plus dealing with backups (hopefully!) and the likely maintenance to add more storage in the future. Plus of course the post claims that 99% uptime is perfectly fine for Zig’s use case, but that works out to ~3 days of downtime per year. Not only can that break CI/CD pipelines (caching is good, but expecting that every CI/CD pipeline uses a cache and that others provide free mirrors of Zig is just externalizing the cost), but downtime would also harm user onboarding and prevent folks from accessing the online docs!
Still though, I am fully sympathetic to saving $700 per month for a static site. At a minimum, I feel like Zig is big enough where it’s pretty hard to justify having only one instance instead of like 2-3 for a high-availability setup. But I think a CDN is still the right play. I think Zig’s website would probably cost $5/mo or less on Cloudflare (free bandwidth, but paying for R2 for storage), or on Bunny.net it’d work out to $100/mo or so for 10TB/mo of bandwidth. I also tried working out pricing with Fastly but they don’t publish pricing without an account. Any one of them would get back to several 9’s of uptime– and for a cost comparable to the Hetzner setup if you factor in maintenance costs
(But also take my feedback with a grain of salt, I’m not currently a Zig user so I don’t actually have any stake in the game!)
Not only can that break CI/CD pipelines (caching is good, but expecting that every CI/CD pipeline uses a cache and that others provide free mirrors of Zig is just externalizing the cost)
If CI/CD processes are so important that they can’t handle 99% availability of a dependency’s website, then yes.. they should be caching that dependency. Ensuring CI/CD pipelines are always able to execute is the burden of the owner of those pipelines, not the dependency’s owner.
Perhaps I’m too generous in my reading, but I didn’t interpret the bit about mirroring as expecting people to step up with mirroring as much as casually mentioning if a person is interested in doing it they’d welcome it.
I’ll definitely grant you that unnecessary rebuilding is wasteful, but for GitHub Actions, redownloading is your only option since each runner is ephemeral (well, more specifically, when you use GitHub’s hosted runners). GitHub does have an action for caching, but it saves and restores the cache using a REST API behind the scenes…
So then as far as redownloading already-built binaries being wasteful, well… I don’t really have any intuition about that. I guess there are two extremes: on one end, you have one massive provider handling storage and footing the bandwidth costs; and on the other, the provider acts as a source of truth, then that trickles down to mirrors, which then trickles down to effectively a per-user cache. I would’ve thought the latter would be more wasteful when considering the total combined energy use (since each user would need to handle their own redundant storage), but actually I’m not really sure which would be less wasteful overall?
The people who download code on every CI run from third party sites [1] and expect it to work all the time are like the people who used to hotlink JavaScript code directly from https://json.org
on their real live websites that they expect to work and be secure
And then Doug Crockford got sick of that and added an infinite loop at the top – saying that you’re supposed to edit it out, and host it yourself !!!
Funny thing is that I just tried to visit this page, and it doesn’t have the code anymore, just the infinite loop lol
Since JSON is built into the browser (and everywhere else, now including https://www.oilshell.org/), I guess the old code is obsolete
[1] to be fair I do download SOME code from third party sites (not all code), but I also don’t expect it to work all the time. Right now I bake most stuff into a Docker image, so it depends on 1 service and not N services, but I want to get off of that too
Ensuring CI/CD pipelines are always able to execute is the burden of the owner of those pipelines, not the dependency’s owner.
Definitely! If your CI pipeline is mission-critical, it’s on you to make sure it doesn’t fail, which means mirroring the things you can’t do without
But I’m also sympathetic to CI pipelines that don’t need to be 100% reliable, where the occasional failure is fine. Sure, Zig’s stance is passable. But I know that if I was picking between, say, Zig and Rust, I’m pretty sure that all the Rust infra I’d use has >99% uptime, and so I could more easily get away with using Rust without setting up caching and it’d be pretty unlikely to cause problems
Perhaps I’m too generous in my reading, but I didn’t interpret the bit about mirroring as expecting people to step up with mirroring as much as casually mentioning if a person is interested in doing it they’d welcome it.
I think I took away a more pessimistic reading after clicking through to this forum post. Here’s a relevant snippet:
“Zig is not fetched from ziglang.org, but rather from a third-party mirror. This decision was made to avoid frequent downloads from the official Zig website, which have a tendency to use up a lot of server resources, the funding for which the ZSF would prefer to utilize elsewhere.”
So, if I’m understanding it right, a Zig CI integration, maintained by a Zig core team member, exclusively uses third-party mirrors instead of fetching from Zig’s official website to help save on server costs. It’s hard for me to interpret this as any other way than the Zig Software Foundation expecting the community to host multiple mirrors, and to collectively foot the bill!
While we are thankful to those who graciously mirror Zig, we are not expecting it.
We offer to fully saturate our 1 GB/s network connection with all the juicy zig downloads the world desires. If that becomes a problem, the world needs to sort it out amongst themselves. We’re happy to cooperate, of course, such as by providing torrent files or something along those lines.
I would manage a single dedicated instance, performing the potentially tens of minutes of maintenance needed each month, for free – and I do, for my mail server, matrix server, http server, etc.
Not sure if it’s still the case for their “development platform” (i.e. R2, D1, Workers, etc.), as they have clear prices for those services that make sense to me.
Is it really fair to compare Zig’s infra costs vs Rust’s infra costs? Rust has crates.io and docs.rs, which I imagine handle an enormous traffic. Zig doesn’t have anything like this - not really sure why, a different approach with different trade offs. But it doesn’t make the comparison of “we provide a completely different service which happens to be much cheaper” fair.
I’m often asked why doesn’t zig have a central package repository, and decentralizing the costs is one of the biggest reasons, so I wanted to take this opportunity to highlight that difference.
The original article did make me wonder if crates.io was a mistake. I like what Go did, at least before they added a Google-operated proxy to the mix: packages pulled from Git repositories, with the possibility of self-hosting. Decentralized (more or less), as the Internet should be.
As for docs.rs, I think we could drop that now, if we normalized developers downloading crates and viewing locally generated docs. IDEs could make this easy.
I am sure they are doing it but add Cloudflare to it and if you ever need to a secondary machine instead of a beefier one and you still way cheaper and yet way more flexible.
What exactly was it that made the website deployments so much faster? The two CI job lists look dramatically different from each other in form, not just in the time taken.
Oh, okay, you’re talking about this post, which clarifies that the website has also been re-engineered under the hood. The OP didn’t make that clear, IMO… it seemed to imply that moving to a self-hosted server magically cut the deploy time by a factor of 40!
tl;dr if you want to mirror static content, shared hosting is the absolute cheapest and least fuss way to do it
I think it still has a bad reputation from ~20 years ago and oversubscribing, but hardware got fast, kernels got fast, and static content is easier than dynamic
I think it still has a bad reputation from ~20 years ago and oversubscribing, but hardware got fast, kernels got fast, and static content is easier than dynamic
The Cloud is not all that different at the end of the day :^)
I really enjoy the few posts recently about migrating away from the cloud to e.g. colocating or self-hosting. Kind of makes me feel like returning to simpler solutions (though colocating arguably is more involved).
I enjoy the insight that a website just isn’t important enough to justify the vastly greater cost and complexity of a higher-availability solution.
Are any of my personal websites or services all that critical? I don’t think so.
I think these posts were inevitable given how cloud vendors haven’t changed their prices for the last 10 years even though the hardware got cheaper and more efficient in every way. With them reaping larger and larger margins from improving hardware underneath, it makes more and more sense not to pay that premium.
This is an interesting thought. It’s certainly more involved in terms of “get a screwdriver and hump your 20 kg server into your car and drive it to Nebraska” work, but you probably spend a lot less time staring at the AWS dashboard agonizing over what the hell some particular security policy means.
As someone who knows how to host something but is not deeply versed in AWS: colocating a server would be a breeze to me (I would know, I’m doing it) meanwhile opening the AWS console to figure out how to setup an ISAM role or not fuck up my S3 bucket permissions or figure out how to meter billing or figure out which of the 70 kinds of instances would be the least cost inefficient all sounds way more involved to me. It also sounds like hell.
I’m in exactly the opposite position - I have years of experience with AWS (and yes, it is hell) but would not be that confident setting up a colocated server. Part of my wants to do it as a sideproject, but I don’t think I can justify the cost (and, frankly, it’s a lot more bandwidth and compute than I need).
It’s actually easy to try, I believe. How much bandwidth and compute do you need? What kinds of side projects do you have now? I think just renting a 4€ VPS from hetzner gets you that first experience.
Or find a used NUC or rack server on eBay and hook it up to your home router. It’s not the same experience as colocation obviously, but I would imagine there is a lot of overlap (I have been doing this for years, but have never colocated). See also r/homelab.
If it’s just a side project, that’s probably going to be fine.
Yeah, my current setup is a very basic version of this i.e. a rpi4 plugged into my router. Current flat is a bit space constrained, but setting up a better homelab space is on my wishlist for when we size up!
Similar enough, but I think for that purpose, people already have their workstation at home. I still believe putting the thing “live” on the internet brings the value, if only to make you ask LLMs how to setup DNS and a simple basic firewall and perhaps some automatic security updates for when you forget about your server for two months.
If you don’t do it regularly, you’ll forget about it, but if you do it irregularly enough, you start to appreciate some of these hosted, cloud solutions.
There’s even a technical term for migration from Cloud back to On-prem: “Repatriation”.
Does someone have an example of this? Given that EC2 et all are the cost effective solution for so few groups (most are either too small, to big, and have workloads too persistent to benefit) I wonder what someone who actually makes the move in that direction and benefits looks like
I once had a client that strongly believed that they have that use case and because it seemed like an obvious case, my guess was that they were in fact reaping the benefit like no other.
Then I did the math and not even in that “perfect example” (strong traffic spikes only on certain days/hour in a week) it did not make sense. Even though their setup was built that way from the beginning.
So I helped changing it and the result was “Wow, it’s so much faster”, while saving lots of costs and complexity.
They switched to dedicated servers (and some vservers). The scale up resources are now just always there and are used for failover, etc.
So in short: I yet have to see a company where the benefits reach beyond a theoretical assumptions.
I worked at a major streaming company that went from a handful of boxes to a cloud solution, around 2017.
Although maybe that’s not the kind of “project” you are interested in.
I’m curious how it went? I worked at a major corp that went from in prem to cloud also, and it was such a disaster that only pride red sunk cost prevent them from reverting
It went fairly smoothly, and as far as I can tell, they are still on the same cloud.
For me as a developer, it was hardly noticeable, so I think the ops team did a good job.
I can’t help but feel that Zig is throwing out the baby with the bathwater with this move. AWS’s bandwidth costs are egregious, and peaking at ~$700/mo of costs for a static site definitely warrants some kind of change, but I just feel like moving to a single dedicated server instance is not the best play
There are some obvious downsides to just running everything on a single dedicated instance, like getting rid of geographic distribution. It also takes work to make sure the kernel, HTTP server, etc. are all kept up-to-date, plus dealing with backups (hopefully!) and the likely maintenance to add more storage in the future. Plus of course the post claims that 99% uptime is perfectly fine for Zig’s use case, but that works out to ~3 days of downtime per year. Not only can that break CI/CD pipelines (caching is good, but expecting that every CI/CD pipeline uses a cache and that others provide free mirrors of Zig is just externalizing the cost), but downtime would also harm user onboarding and prevent folks from accessing the online docs!
Still though, I am fully sympathetic to saving $700 per month for a static site. At a minimum, I feel like Zig is big enough where it’s pretty hard to justify having only one instance instead of like 2-3 for a high-availability setup. But I think a CDN is still the right play. I think Zig’s website would probably cost $5/mo or less on Cloudflare (free bandwidth, but paying for R2 for storage), or on Bunny.net it’d work out to $100/mo or so for 10TB/mo of bandwidth. I also tried working out pricing with Fastly but they don’t publish pricing without an account. Any one of them would get back to several 9’s of uptime– and for a cost comparable to the Hetzner setup if you factor in maintenance costs
(But also take my feedback with a grain of salt, I’m not currently a Zig user so I don’t actually have any stake in the game!)
If CI/CD processes are so important that they can’t handle 99% availability of a dependency’s website, then yes.. they should be caching that dependency. Ensuring CI/CD pipelines are always able to execute is the burden of the owner of those pipelines, not the dependency’s owner.
Perhaps I’m too generous in my reading, but I didn’t interpret the bit about mirroring as expecting people to step up with mirroring as much as casually mentioning if a person is interested in doing it they’d welcome it.
Absolutely. It’s bonkers re-downloading and often rebuilding dependencies from scratch on every CI run, an apalling waste of time and energy.
I’ll definitely grant you that unnecessary rebuilding is wasteful, but for GitHub Actions, redownloading is your only option since each runner is ephemeral (well, more specifically, when you use GitHub’s hosted runners). GitHub does have an action for caching, but it saves and restores the cache using a REST API behind the scenes…
So then as far as redownloading already-built binaries being wasteful, well… I don’t really have any intuition about that. I guess there are two extremes: on one end, you have one massive provider handling storage and footing the bandwidth costs; and on the other, the provider acts as a source of truth, then that trickles down to mirrors, which then trickles down to effectively a per-user cache. I would’ve thought the latter would be more wasteful when considering the total combined energy use (since each user would need to handle their own redundant storage), but actually I’m not really sure which would be less wasteful overall?
You shouldn’t be redownloading it from the upstream sources. You should have your own caches that you pull upstream to.
The people who download code on every CI run from third party sites [1] and expect it to work all the time are like the people who used to hotlink JavaScript code directly from https://json.org
on their real live websites that they expect to work and be secure
And then Doug Crockford got sick of that and added an infinite loop at the top – saying that you’re supposed to edit it out, and host it yourself !!!
Funny thing is that I just tried to visit this page, and it doesn’t have the code anymore, just the infinite loop lol
http://json.org/json.js
http://json.org/json.as
for(;;){}Since JSON is built into the browser (and everywhere else, now including https://www.oilshell.org/), I guess the old code is obsolete
[1] to be fair I do download SOME code from third party sites (not all code), but I also don’t expect it to work all the time. Right now I bake most stuff into a Docker image, so it depends on 1 service and not N services, but I want to get off of that too
Definitely! If your CI pipeline is mission-critical, it’s on you to make sure it doesn’t fail, which means mirroring the things you can’t do without
But I’m also sympathetic to CI pipelines that don’t need to be 100% reliable, where the occasional failure is fine. Sure, Zig’s stance is passable. But I know that if I was picking between, say, Zig and Rust, I’m pretty sure that all the Rust infra I’d use has >99% uptime, and so I could more easily get away with using Rust without setting up caching and it’d be pretty unlikely to cause problems
I think I took away a more pessimistic reading after clicking through to this forum post. Here’s a relevant snippet:
So, if I’m understanding it right, a Zig CI integration, maintained by a Zig core team member, exclusively uses third-party mirrors instead of fetching from Zig’s official website to help save on server costs. It’s hard for me to interpret this as any other way than the Zig Software Foundation expecting the community to host multiple mirrors, and to collectively foot the bill!
To be crystal clear:
I would manage a single dedicated instance, performing the potentially tens of minutes of maintenance needed each month, for free – and I do, for my mail server, matrix server, http server, etc.
You have to pay me to work on AWS, though.
Cloudflare’s list price is secretly a promotional price. Beyond a certain secret threshold, you’ll be forced into an enterprise agreement.
Not sure if it’s still the case for their “development platform” (i.e. R2, D1, Workers, etc.), as they have clear prices for those services that make sense to me.
Is it really fair to compare Zig’s infra costs vs Rust’s infra costs? Rust has crates.io and docs.rs, which I imagine handle an enormous traffic. Zig doesn’t have anything like this - not really sure why, a different approach with different trade offs. But it doesn’t make the comparison of “we provide a completely different service which happens to be much cheaper” fair.
I’m often asked why doesn’t zig have a central package repository, and decentralizing the costs is one of the biggest reasons, so I wanted to take this opportunity to highlight that difference.
The original article did make me wonder if crates.io was a mistake. I like what Go did, at least before they added a Google-operated proxy to the mix: packages pulled from Git repositories, with the possibility of self-hosting. Decentralized (more or less), as the Internet should be.
As for docs.rs, I think we could drop that now, if we normalized developers downloading crates and viewing locally generated docs. IDEs could make this easy.
With go you can use any proxy while still using the original import urls. Makes it quite easy to host your own.
I am sure they are doing it but add Cloudflare to it and if you ever need to a secondary machine instead of a beefier one and you still way cheaper and yet way more flexible.
What exactly was it that made the website deployments so much faster? The two CI job lists look dramatically different from each other in form, not just in the time taken.
the subsequent news entry might shed some light on that mystery :^)
Oh, okay, you’re talking about this post, which clarifies that the website has also been re-engineered under the hood. The OP didn’t make that clear, IMO… it seemed to imply that moving to a self-hosted server magically cut the deploy time by a factor of 40!
It actually did, Zine is fast, but so is Hugo. The whole cloud runner circus thing is insanely slow, we just got used to it being this way.
Oh this comment about shared hosting is actually in reply to this story, I clicked through from the other one:
https://lobste.rs/s/v0jpcf/zig_website_has_been_re_engineered#c_gnmfoq
tl;dr if you want to mirror static content, shared hosting is the absolute cheapest and least fuss way to do it
I think it still has a bad reputation from ~20 years ago and oversubscribing, but hardware got fast, kernels got fast, and static content is easier than dynamic
The Cloud is not all that different at the end of the day :^)