There were a variety of companies starting around 2010 doing hyperconverged infrastructure – off-the-shelf clusters with networking, storage, and VM hypervisors all built in. The truth is none of them were very successful. They all ran into scaling issues with at least one of the parts (usually storage) and, like an all-in-one PC, you couldn’t easily substitute out the parts that weren’t working for parts from other vendors. A few still exist like Nutanix but have underwhelming adoption rates. The Oxide folks are very smart, and I’m sure they will do a great job, but I’m not quite sure how they’ll approach this problem.
And our original pitch deck has an appendix slide on each and every one of them! ;) For many of these companies, I spoke to the founders or other key early folks to understand what had gone right – and what had gone wrong. This is not a deep thought, but the number one problem was: by the time we figured out that we actually needed to not use commodity junk, we ran out of money. (Some of them never made this realization – and just ran out of money.) We internalized this lesson by being very explicit about what we were doing – and then raising enough money to do it. (The first capital in was a $12M early close on a $20M seed round.) It would have been way, way easier to raise less (e.g. $3M) – but we knew that we would be another tombstone if we did.
So we took a longer, steeper, and MUCH more technical route that any of those companies. The downside is that the MVP is huge and took us a while to build; the upside is that we have had the right foundation to actually attack the problems that you outline properly!
We’re not posting the whole deck (sorry), but if you have a question about a particular company (and our appendix slide on that company isn’t too mean), I’ll post the slide for the company…
We spoke with [elided] as well as many Nutanix customers
The early success of Nutanix was in VDI – but as it became successful, customers tried to use it in other capacities for which it is less apt
Nutanix is, at essence, a storage product – but customers are looking to use it in an on-prem cloud capacity, leading to disappointment
Through customer conversations, Nutanix at epicenter of painful outages, due to the interface between software and hardware
From our experience, we know that their move away from an integrated HW appliance and to a SW-only solution will exacerbate this!
Looking back on that from 2023, I think the last point is the one that bears emphasis: prior to Oxide, we had extensive experience trying to ship infrastructure software on third-party (“commodity”) hardware and firmware, and the results just have problems that become increasingly insurmountable. We knew this coming into Oxide, but at Oxide we re-learned this: we had to develop our software before we had hardware in hand – which has necessitated us having a bunch of that commodity hardware on which to develop. And the pain in getting that infrastructure stood up and managed – even at very small scale! – was a good reminder why we have taken an integrated approach.
From the outset at Oxide, and as I outlined in my 2020 Stanford talk, we have had three core beliefs as a company:
Cloud computing is the future of all computing infrastructure.
…
Of these beliefs, the first is not at all controversial:
yeah sorry I think that belief is actually incredibly harmful to society, so to call it “not at all controversial” reads as at best a bold claim, and at worst a willful silencing of dissenting opinions.
The idea that all computing should come down to a client-server paradigm where the server is owned by one party and the client is another party that pays rent per-usage is inherently aligned with rentier economics. For Oxide to say “cloud computing is the future of all computing infrastructure” is wild to me, because their criticism of existing cloud providers is making an argument against rentier capitalism (why rent a cloud computer when you can buy a cloud computer). It’s not a challenge to the fundamental structures of rentier capitalism, it’s the assertion that everyone who can program the computer should be a landlord.
If your argument is “rentier capitalism is effective and if we do it we will make money”, ok, sure, go for it. The faux moralizing that they do is at best a form of self-delusion, and at worst a hypocritical and intentional sleight of hand. The idea that they’re doing something brave and new and benevolent is naive.
Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared
pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that
can be rapidly provisioned and released with minimal management effort or service provider interaction
This is definition that we at Oxide have been operating under – and we emphatically agree with you that the idea that cloud computing must be rented is wrong. We are trying to provide the world another option: a cloud that can be purchased and owned rather than rented.
I was definitely yay-ing at the idea of not renting your cloud computer. But given the level of compute power we have today and the likely cost of a rack of such compute power (I could only find “contact sales”) it seems like the market for the purchasable device is primarily or only people that will then rent it out to others. Or large corps, but those already can do their own data center.
And then you get the economics of cloud computing, which to me mostly (exclusively?) make sense because the compute power in a single device is now so great that for most server-style applications, you don’t actually need a whole computer. Which means that you can defray the high fixed cost of operating such a computer by multiplexing it among many different users.
That model, which to me seems fundamental to the cloud, doesn’t work if load is segmented into rack-sized units.
And how operates these racks? Where do they live?
So I definitely like, but don’t quite see how it is supposed to work.
There is a level of medium-sized corps that are not big enough to build their own DCs nor design and build their own rack-scale computers, but big enough to own entire racks in rented datacenter space. These are currently commodity racks with commodity servers in them.
There are more companies at that intermediate scale than you might think. Buying a ~$1,000,000 Oxide rack (pure speculation on the price here) is multiple orders of magnitude cheaper than building an entire DC.
But even purchased and owned, it won’t change me having shit internet with 1-10% packet loss depending on the weather and not being able to even connect to the cloud I own… All my local stuff is resilient to this by nature.
It also won’t help with any application that needs low latency such as anything involving music, e.g. < 8 ms roundtrip
I think you might be too expansive in your reading of cloud computing. My reading of that is more “computing where we treat one physical device as N virtual devices, and we can move those devices around as needed”.
Like in basically all enterprise needs you’re gonna be in some model where you have data that is manipulated by N end clients (all those end clients can be internal to your company!), and cloud computing feels like the “sane” way to manage that (compared to “we have a bunch of physical devices and one of them is the important one with a sticker on it that says that if this gets turned off all of our POSes will fall offline”)
Cloud computing is describing a mechanism to manage a lot of compute, not a business model.
I think that’s conflating distributed computing with cloud computing.
in basically all enterprise needs
yes, enterprise needs. I think when your primary customer is enterprises, that opining about how big and bad AWS is is … well, a bit hypocritical. They make data center hardware that costs hundreds of thousands of dollars and the vast majority of their customers will be people who build SaaS. That’s about as distant from the original goals of the FOSS movement as can be.
You can be a medium-sized school district and be “enterprise-sized”. Imagine a school district that is self hosting ~everything. They would have a utility for this! Loads of places have on-prem needs that are just not about SaaS at all, and are merely “we have an internal IT team that has institutional knowledge of running on-prem stuff for our own needs, and this is cheaper than renting out stuff, and easier to work with than other on-prem solutions”. The world of needing many computers goes beyond selling SaaS.
“Distributed Computing”, meanwhile, does not cover with certainty stuff like being able to move VMs across physical machines (some random supercomputer running a big Hadoop workload might be distributed, but it’s not “cloud computing” IMO). The operational abilities matter here, not the CPU/machine count.
Most schools don’t want to do the software development that would require, so they end up renting SaaS products. Also, you’d need to have pretty heavy consumption for the math to pencil out. $1m/5 year depreciation → $17K/monthly. Are you really going to be spending that much on cloud? I mean, it definitely happens, but it’s a lot of money for some custom forum software and whatnot.
$17k monthly is something, right, but school districts have loads of users. Chicago Public Schools (obviously huge) is 350k students. So even at $1/student/month, services get really expensive really quick.
I am legitimately confused at how many people in this thread do not believe that there are places with server racks. Universities run hybrid setups, loads of offices still have internal IT teams (even if it’s mostly just operations)…. remember, we’re not necessarily talking about a huge team of software developers. We’re talking about an operations team that might cobble together off-the-shelf software to run to serve users. There are reasons that phrases like “Active Directory” exist!
Oh, I believe there is definitely a market for it. I’m just not sure about a school district because of the software problem. Maybe a university would use it just for research though.
I wonder if a medium-sized school district, to use your example, should centralize their server-side computing to the point of buying full racks though. Why not have separate servers at each school, plus another one at the central office? Oxide, on the other hand, is only selling racks.
It’s hard for me to imagine that they shouldn’t centralize their server side compute. More locations adds more costs in pretty much every way. Harder to physically maintain/install (because of travel time if nothing else), harder to physically secure, more real-estate, etc.
The only thing you really save on is bandwidth (what unit is that anyways? bits*meters?)… I doubt that that is often a limiting constraint.
I suppose there is a redundancy argument in having multiple sites if your software is designed to take advantage of the redundancy, not sure if that’s realistic for the software a school district uses though.
yeah sorry I think that belief is actually incredibly harmful to society
This is a stretch. A belief is just a belief, especially a belief in how web applications hosting is trending. Whether or not they believe in it has no bearing on what will happen. The reality is that pandora’s box has already been opened. The ease with which you can dispatch resources using cloud platforms is so appealing that it renders short-term cost and other considerations irrelevant. This is the market at play. Nobody is forcing developers and entrepreneurs to use cloud platforms, they do it of their own accord. Oxide is just stating the obvious. Don’t shoot the messenger.
As far as your beliefs about how computing should be structured, Oxide is doing something that actually mediates the trend of all web hosting being consolidated into the hands of a few operators. I’m not sure why you would attack them. Ideologically you have much bigger fish to fry.
That said, I think Oxide will fail. By fail, I mean their best outcome will be an acquisition. The founders will at least do decently well. This is just my perspective on their reality, this isn’t what I want to happen.
From my reading, they were saying the compute infrastructure is typically datacentre based, and they would like to make it so one party doesn’t need to rent the datacentre from another party precisely for the “rentier capitalism” problems (one party can own, operate and use their own data centre).
Keep in mind the modern buzzwords - companies that build a data centre typically call it their own “private cloud” these days. E.g. company XYZ builds “XYZCloud” or some-such for internal usage. A part of this is providing public-cloud-like APIs and abstractions for the internal developers to leverage, hence why oxide builds both hardware+software in order to go after (and expand!) this market.
yes I think operating any form of SaaS is a form of rentier capitalism and that this line of products is going to derive the vast majority of its revenue from SaaS providers. Congrats, you don’t rent your servers, woohoo. If you operate SaaS, you’re still building software on a model where the user doesn’t run the computation on their computer. That’s fundamentally the same power dynamic, not a different one. The fact that the SaaS provider doesn’t rent from an IaaS provider but instead owns their own hardware does not change the relationship between the SaaS provider and the end-user, the relationship between the SaaS provider and society, or the power dynamic of the system as a whole.
Starting a company that sells machines to SaaS providers and describing yourself “a champion of open source” strikes me as ideologically inconsistent. When you are enabling a model of computation and end-user interaction that is anti user freedom, I think that makes you as far from “a champion of open source” as can be.
It really feels like you’re saying that it’s alright to build computers with open source software, as long as you don’t sell them to people who are going to use them to provide services for money? Can we sell them to people who lease cars? What about a public transport authority which charges people per bus ride?
I do get your point. I am definitely a proponent of end users having hardware and software for local general-purpose computing, local-first software, etc.
I’d still argue that, say, having my company’s infrastructure to run compute on rather than relying on my laptop can be a boon (and has been a boon). This possibly depends on your job/industry, but e.g. processing a petabyte of data just isn’t going to happen on a personal computer. Or I might want to host a persistent service for others in the business, have a place to backup data, put a database, etc. Or host our own git, CI/CD, test environments, etc, rather than renting these from github and AWS. Or self host our email and website. Etc.
Yes you can use Oxide servers to host SaaS for your customers… or you can just, you know, use them.
I agree with you about rentier capitalism, power dynamics, and harm.
However, I think your framing about open source is a bit incomplete, and ends up undermining your point. “Open source” has always been the defanged corporate alternative to Free Software, so it’s entirely consistent with rentier capitalism.
Conflating the two means your argument rests on proving they’re not really open-source, which they demonstrably are. The cleaner angle, I think, is to take on Open Source itself as insufficient at best to ensuring software freedom–at worst, diametrically opposed. (But to be completely fair to Oxide, I don’t see them claiming that’s what they’re trying to do. They are a VC-funded startup, after all.)
Starting a company that sells machines to SaaS providers and describing yourself “a champion of open source” strikes me as ideologically inconsistent.
First off, the nod to open source is just reflexive marketing speech at this point. Read some magazines from the 80s and be struck by the use of “proprietary” as a positive term in many of them for an example.
Second, nothing about Free Software/Open Source is against making money. Stallman explicitly cited providing support and other services as the way forward for companies providing Free Software. Open Source filed off the parts that were unpalatable for companies and pitched the model explicitly as a cheaper, better way to build software for profit.
(Stallman also didn’t own the computer and printer, whose driver was the central example for why he got mad at closed software).
I do realize that in parts of the nebulous “FLOSS meme-sphere”, owning one’s own hardware and relying on decentralized protocols to spread out the means of computation is the ideal. But there’s actually very little within the concrete manifestations of FLOSS (the licenses and how they are followed) that naturally leads to such a situation. In fact, the economic realities of building and running software actively mitigates against it.
Although it is news from a business, hence business news, there’s a lot of technical commentary inside too. IMO it’s not filled with buzzwords and gives enough for us to discuss their offering.
What it gives us, beyond a couple tidbits about fans and simpler electrical design, is mostly just a business pitch dressed up to feel attractive to developers.
There are interesting articles linked off of it, yes–but I think precedent here is typically to submit those articles directly (see also why we don’t really do newsletter submissions…better to submit the stories directly).
I have tremendous respect for the Oxide folks and what they’ve accomplished; make no mistake, though, that this is marketing and just a press release.
To me, this is on the bubble. There’s almost enough information about the hardware in their rack to count as a Lobster’s story, but I think it’s maybe just a hair shy. But maybe someone else will have a different opinion.
The golden rule: does reading it make people better programmers? I think some of the discussion here about how to think about hardware and software in tandem feels like this is on-topic (But I’m biased, cuz I listen to their podcast, am bought into the idea, and so know more context than what someone just reading this might get).
The entire software stack seems to be open source. That alone makes it relevant here…no? I personally have a ton of questions just skimming through a couple of repos there… I’d say this is wonderful stuff!
Oxide starts around $500k and only goes up from there I believe. But at this level every sale is an enterprise sale with the high touch sales process to go with it. If you buy a rack from Amazon for on-prem AWS the starting point is: have you paid for enterprise support?
I didn’t have the patience to go through their wall of small text. I always thought “the cloud” meant renting someone else’s computers and letting them take care of the scaling, backups and failovers. Is this just a … on prem server farm?
I view it as a modern version of the mainframe. One big box arrives at your doorstep. You plug it in and everything is already there and you can use it.
I find the mainframe comparison useful. The things that keep it from being a true mainframe are no longer as important to many – it’s not so important it doesn’t act as a single large computer, or bake more reliability features deep into the hardware (vs. say handling high database availability mainly by failing over). 64-core/1TB RAM/32TB SSD nodes are large enough for a lot already, and most apps are already written as a distributed system where app processes talk to DBs, caches, queues, and storage services–like, that’s the norm even when scale doesn’t require it. (Also, this product has bits like distributed block storage that bring a bit of that “big reliable hardware” illusion back.)
What is like a mainframe, from a business perspective, is:
you’re buying a thing as a capital expenditure
one purchase is supposed to cover CPU, RAM, storage, and interconnect for your whole workload, and all the overprovisioning you need for HA/continuing after failures, non-prod servers, etc.
one vendor is taking responsibility for integration and reliability of the overall system (I hope; at the prices I expect they charge, it doesn’t seem like they can shrug at reliability issues).
With the technical aspects of a true mainframe much less in demand today, and the mainframe-like aspects it does have being rare today, it does feel like a modern mainframe in some relevant sense.
I wonder if the potential market–essentially, folks who want to own the resources of at least 16 of these compute sleds and have some real dissatisfaction with using a cloud vendor–will bite at this, or prefer racking Supermicro servers and generic switches or using some other flavor of non-big-3 hosting. The buzz itself might actually be what lets this take off enough to get a durable foothold when similar-seeming things have not. Of course, it also needs to work well in practice, something not really proven yet, and it has to be priced so the business can work!
I also wonder if Oxide or others are going to offer cloud hosting based on this. With what they have today, it’d be very different from what the big providers offer (just as “cloud” offerings from older-school hosting providers are), but I could see selling it on things like the latency/bandwidth advantages of a group of nodes and their storage all being connected to the same top-of-rack switches; you might be able to gin up good benchmarks based on that.
from what I read, it seems like it’s a low-hassle server farm, complete with software and firmware updates. It also seems to be configurable via APIs, similar to the cloud. I imagine you could terraform VMs and resources on it.
Unlike the cloud, you don’t pay a perpetual “tax”, it’s just a one-time purchase. Reading a few articles lately (e.g. DHH) about going on-prem from the cloud, the big argument against going back to on-prem is the hassle, and this seems to address that fairly well.
FWIW, we have an EBS-like block storage system, but we do not yet have an S3-like object store (the economics of bulk object storage on all-flash systems are not fantastic anyway). I think it’s pretty reasonable to think of it as in the EC2 space at this early moment! We’re excited about layering more on top in the future.
(Edit: I guess I don’t really think of IAM or VPC or Route 53 as being separate from EC2, but it you do, we have stuff in those areas also!)
I don’t have an Oxide Rack brochure sitting in front of me to type out all the features. You might want to do some investigation to evaluate whether it’s just “random VPS provider in a box” or not.
The real point that is being made in this thread is that it’s not a rack that does nothing. It’s a rack that, within hours of being delivered off the truck, can be setup and used to provision production workloads. Their goal is to reduce that time as much as possible.
That is a great feature set. It is however besides the point calling it “AWS in a box”, which what I am trying to say. AWS has hundreds of services, from very low level to very high level. There is no way that you get anything close to that from Oxide. You get the hardware you could build that on, yes. Will it be highly integrated hardware/software and probably quite good? I guess so. I trust their taste. Is this “AWS in a box”? No it is not. That’s all.
That’s probably not an unreasonable way to think about it. One idly wonders if they’ll have a marketplace for service implementations that run on Oxide machines that can cover more of the use cases than “compute + storage + IAM auth”.
Cloud means a lot of different things. That is why nearly every analysis of deployment choices related to the cloud comes with a hefty pile of caveats. The cloud is all of:
Locating your compute in a data center.
A software layer provided by the data center to orchestrate the hardware.
Relying on the data center to pre-acquire hardware.
Associated software services that scaffold atop the orchestration layer to serve a COTS/SaaS role.
The Oxide rack handles the second and it allows you to handle the first and third. Those three together make up the 2011-ish version of AWS. The fourth is a more recent maturation of the cloud offerings. Given Oxide’s young age, they aren’t their yet, but I suspect they could grow to serve a similar role.
Usually the cost of these things is a rental fee. However that is a sales strategy by the cloud provider. The advantage to them is that rental fees present a perpetual option to increase prices and to increase the services used by a customer. Cloud providers could choose to sell their customers a portion of their data center, along with a futures contract for power and network delivery. As growth slows, I suspect we will see relationships like this, along with the associated financialization and arbitrage that goes along with those types of relationships.
Those three together make up the 2011-ish version of AWS. […] The fourth is a more recent maturation of the cloud offerings
In 2011 AWS already had (next to EC2 and S3) EMR, RDS, ELB, SNS, IAM, Route 53, Elastic Beanstalk, SES which are all in the 4th category. I would not call that a “recent maturation”.
How is this any different than just buying preconfigured server rack(s) from any manufacturer that offers them (practically all)? Many will even install private cloud software from the factory (OpenStack, various propriety ones). Even if your prefferred manufacturer won’t, all major manufacturers will PXE boot out of the box/crate, which makes it easy to install/run whatever you want.
I think Oxide would point out that you can do that, and then you’ll discover that there is a bizarre bug that causes rare 100ms stalls, and the server vendor will point at the SSD vendor will point at the OS vendor will point at the motherboard vendor will point at…
Oxide instead owns and is responsible for the entire stack.
(Bryan Cantrill and several Oxide folks are ex-Joyent, so they have a lot of stories on the “joy” of running a cloud on off-the-shelf hardware. [Update: you may be interested in this comment.])
There were a variety of companies starting around 2010 doing hyperconverged infrastructure – off-the-shelf clusters with networking, storage, and VM hypervisors all built in. The truth is none of them were very successful. They all ran into scaling issues with at least one of the parts (usually storage) and, like an all-in-one PC, you couldn’t easily substitute out the parts that weren’t working for parts from other vendors. A few still exist like Nutanix but have underwhelming adoption rates. The Oxide folks are very smart, and I’m sure they will do a great job, but I’m not quite sure how they’ll approach this problem.
And our original pitch deck has an appendix slide on each and every one of them! ;) For many of these companies, I spoke to the founders or other key early folks to understand what had gone right – and what had gone wrong. This is not a deep thought, but the number one problem was: by the time we figured out that we actually needed to not use commodity junk, we ran out of money. (Some of them never made this realization – and just ran out of money.) We internalized this lesson by being very explicit about what we were doing – and then raising enough money to do it. (The first capital in was a $12M early close on a $20M seed round.) It would have been way, way easier to raise less (e.g. $3M) – but we knew that we would be another tombstone if we did.
So we took a longer, steeper, and MUCH more technical route that any of those companies. The downside is that the MVP is huge and took us a while to build; the upside is that we have had the right foundation to actually attack the problems that you outline properly!
Is there a way to read those particular slides? 👉👈
We’re not posting the whole deck (sorry), but if you have a question about a particular company (and our appendix slide on that company isn’t too mean), I’ll post the slide for the company…
Nutanix is the other stack that I think most of us here would be interested in.
Our Nutanix slide (ca. October 2019), FWIW:
Looking back on that from 2023, I think the last point is the one that bears emphasis: prior to Oxide, we had extensive experience trying to ship infrastructure software on third-party (“commodity”) hardware and firmware, and the results just have problems that become increasingly insurmountable. We knew this coming into Oxide, but at Oxide we re-learned this: we had to develop our software before we had hardware in hand – which has necessitated us having a bunch of that commodity hardware on which to develop. And the pain in getting that infrastructure stood up and managed – even at very small scale! – was a good reminder why we have taken an integrated approach.
yeah sorry I think that belief is actually incredibly harmful to society, so to call it “not at all controversial” reads as at best a bold claim, and at worst a willful silencing of dissenting opinions.
The idea that all computing should come down to a client-server paradigm where the server is owned by one party and the client is another party that pays rent per-usage is inherently aligned with rentier economics. For Oxide to say “cloud computing is the future of all computing infrastructure” is wild to me, because their criticism of existing cloud providers is making an argument against rentier capitalism (why rent a cloud computer when you can buy a cloud computer). It’s not a challenge to the fundamental structures of rentier capitalism, it’s the assertion that everyone who can program the computer should be a landlord.
If your argument is “rentier capitalism is effective and if we do it we will make money”, ok, sure, go for it. The faux moralizing that they do is at best a form of self-delusion, and at worst a hypocritical and intentional sleight of hand. The idea that they’re doing something brave and new and benevolent is naive.
So, for whatever it’s worth NIST actually has a definition of cloud computing:
This is definition that we at Oxide have been operating under – and we emphatically agree with you that the idea that cloud computing must be rented is wrong. We are trying to provide the world another option: a cloud that can be purchased and owned rather than rented.
I was definitely yay-ing at the idea of not renting your cloud computer. But given the level of compute power we have today and the likely cost of a rack of such compute power (I could only find “contact sales”) it seems like the market for the purchasable device is primarily or only people that will then rent it out to others. Or large corps, but those already can do their own data center.
And then you get the economics of cloud computing, which to me mostly (exclusively?) make sense because the compute power in a single device is now so great that for most server-style applications, you don’t actually need a whole computer. Which means that you can defray the high fixed cost of operating such a computer by multiplexing it among many different users.
That model, which to me seems fundamental to the cloud, doesn’t work if load is segmented into rack-sized units.
And how operates these racks? Where do they live?
So I definitely like, but don’t quite see how it is supposed to work.
There is a level of medium-sized corps that are not big enough to build their own DCs nor design and build their own rack-scale computers, but big enough to own entire racks in rented datacenter space. These are currently commodity racks with commodity servers in them.
There are more companies at that intermediate scale than you might think. Buying a ~$1,000,000 Oxide rack (pure speculation on the price here) is multiple orders of magnitude cheaper than building an entire DC.
But even purchased and owned, it won’t change me having shit internet with 1-10% packet loss depending on the weather and not being able to even connect to the cloud I own… All my local stuff is resilient to this by nature.
It also won’t help with any application that needs low latency such as anything involving music, e.g. < 8 ms roundtrip
I think you might be too expansive in your reading of cloud computing. My reading of that is more “computing where we treat one physical device as N virtual devices, and we can move those devices around as needed”.
Like in basically all enterprise needs you’re gonna be in some model where you have data that is manipulated by N end clients (all those end clients can be internal to your company!), and cloud computing feels like the “sane” way to manage that (compared to “we have a bunch of physical devices and one of them is the important one with a sticker on it that says that if this gets turned off all of our POSes will fall offline”)
Cloud computing is describing a mechanism to manage a lot of compute, not a business model.
I think that’s conflating distributed computing with cloud computing.
yes, enterprise needs. I think when your primary customer is enterprises, that opining about how big and bad AWS is is … well, a bit hypocritical. They make data center hardware that costs hundreds of thousands of dollars and the vast majority of their customers will be people who build SaaS. That’s about as distant from the original goals of the FOSS movement as can be.
You can be a medium-sized school district and be “enterprise-sized”. Imagine a school district that is self hosting ~everything. They would have a utility for this! Loads of places have on-prem needs that are just not about SaaS at all, and are merely “we have an internal IT team that has institutional knowledge of running on-prem stuff for our own needs, and this is cheaper than renting out stuff, and easier to work with than other on-prem solutions”. The world of needing many computers goes beyond selling SaaS.
“Distributed Computing”, meanwhile, does not cover with certainty stuff like being able to move VMs across physical machines (some random supercomputer running a big Hadoop workload might be distributed, but it’s not “cloud computing” IMO). The operational abilities matter here, not the CPU/machine count.
Most schools don’t want to do the software development that would require, so they end up renting SaaS products. Also, you’d need to have pretty heavy consumption for the math to pencil out. $1m/5 year depreciation → $17K/monthly. Are you really going to be spending that much on cloud? I mean, it definitely happens, but it’s a lot of money for some custom forum software and whatnot.
$17k monthly is something, right, but school districts have loads of users. Chicago Public Schools (obviously huge) is 350k students. So even at $1/student/month, services get really expensive really quick.
I am legitimately confused at how many people in this thread do not believe that there are places with server racks. Universities run hybrid setups, loads of offices still have internal IT teams (even if it’s mostly just operations)…. remember, we’re not necessarily talking about a huge team of software developers. We’re talking about an operations team that might cobble together off-the-shelf software to run to serve users. There are reasons that phrases like “Active Directory” exist!
Oh, I believe there is definitely a market for it. I’m just not sure about a school district because of the software problem. Maybe a university would use it just for research though.
I wonder if a medium-sized school district, to use your example, should centralize their server-side computing to the point of buying full racks though. Why not have separate servers at each school, plus another one at the central office? Oxide, on the other hand, is only selling racks.
It’s hard for me to imagine that they shouldn’t centralize their server side compute. More locations adds more costs in pretty much every way. Harder to physically maintain/install (because of travel time if nothing else), harder to physically secure, more real-estate, etc.
The only thing you really save on is bandwidth (what unit is that anyways? bits*meters?)… I doubt that that is often a limiting constraint.
I suppose there is a redundancy argument in having multiple sites if your software is designed to take advantage of the redundancy, not sure if that’s realistic for the software a school district uses though.
This is a stretch. A belief is just a belief, especially a belief in how web applications hosting is trending. Whether or not they believe in it has no bearing on what will happen. The reality is that pandora’s box has already been opened. The ease with which you can dispatch resources using cloud platforms is so appealing that it renders short-term cost and other considerations irrelevant. This is the market at play. Nobody is forcing developers and entrepreneurs to use cloud platforms, they do it of their own accord. Oxide is just stating the obvious. Don’t shoot the messenger.
As far as your beliefs about how computing should be structured, Oxide is doing something that actually mediates the trend of all web hosting being consolidated into the hands of a few operators. I’m not sure why you would attack them. Ideologically you have much bigger fish to fry.
That said, I think Oxide will fail. By fail, I mean their best outcome will be an acquisition. The founders will at least do decently well. This is just my perspective on their reality, this isn’t what I want to happen.
From my reading, they were saying the compute infrastructure is typically datacentre based, and they would like to make it so one party doesn’t need to rent the datacentre from another party precisely for the “rentier capitalism” problems (one party can own, operate and use their own data centre).
Keep in mind the modern buzzwords - companies that build a data centre typically call it their own “private cloud” these days. E.g. company XYZ builds “XYZCloud” or some-such for internal usage. A part of this is providing public-cloud-like APIs and abstractions for the internal developers to leverage, hence why oxide builds both hardware+software in order to go after (and expand!) this market.
yes I think operating any form of SaaS is a form of rentier capitalism and that this line of products is going to derive the vast majority of its revenue from SaaS providers. Congrats, you don’t rent your servers, woohoo. If you operate SaaS, you’re still building software on a model where the user doesn’t run the computation on their computer. That’s fundamentally the same power dynamic, not a different one. The fact that the SaaS provider doesn’t rent from an IaaS provider but instead owns their own hardware does not change the relationship between the SaaS provider and the end-user, the relationship between the SaaS provider and society, or the power dynamic of the system as a whole.
Starting a company that sells machines to SaaS providers and describing yourself “a champion of open source” strikes me as ideologically inconsistent. When you are enabling a model of computation and end-user interaction that is anti user freedom, I think that makes you as far from “a champion of open source” as can be.
It really feels like you’re saying that it’s alright to build computers with open source software, as long as you don’t sell them to people who are going to use them to provide services for money? Can we sell them to people who lease cars? What about a public transport authority which charges people per bus ride?
I do get your point. I am definitely a proponent of end users having hardware and software for local general-purpose computing, local-first software, etc.
I’d still argue that, say, having my company’s infrastructure to run compute on rather than relying on my laptop can be a boon (and has been a boon). This possibly depends on your job/industry, but e.g. processing a petabyte of data just isn’t going to happen on a personal computer. Or I might want to host a persistent service for others in the business, have a place to backup data, put a database, etc. Or host our own git, CI/CD, test environments, etc, rather than renting these from github and AWS. Or self host our email and website. Etc.
Yes you can use Oxide servers to host SaaS for your customers… or you can just, you know, use them.
I agree with you about rentier capitalism, power dynamics, and harm.
However, I think your framing about open source is a bit incomplete, and ends up undermining your point. “Open source” has always been the defanged corporate alternative to Free Software, so it’s entirely consistent with rentier capitalism.
Conflating the two means your argument rests on proving they’re not really open-source, which they demonstrably are. The cleaner angle, I think, is to take on Open Source itself as insufficient at best to ensuring software freedom–at worst, diametrically opposed. (But to be completely fair to Oxide, I don’t see them claiming that’s what they’re trying to do. They are a VC-funded startup, after all.)
First off, the nod to open source is just reflexive marketing speech at this point. Read some magazines from the 80s and be struck by the use of “proprietary” as a positive term in many of them for an example.
Second, nothing about Free Software/Open Source is against making money. Stallman explicitly cited providing support and other services as the way forward for companies providing Free Software. Open Source filed off the parts that were unpalatable for companies and pitched the model explicitly as a cheaper, better way to build software for profit.
(Stallman also didn’t own the computer and printer, whose driver was the central example for why he got mad at closed software).
I do realize that in parts of the nebulous “FLOSS meme-sphere”, owning one’s own hardware and relying on decentralized protocols to spread out the means of computation is the ideal. But there’s actually very little within the concrete manifestations of FLOSS (the licenses and how they are followed) that naturally leads to such a situation. In fact, the economic realities of building and running software actively mitigates against it.
This is ulitmately business news. While they have a lot of geek cred, I don’t see how this post is particularly relevant here.
Although it is news from a business, hence business news, there’s a lot of technical commentary inside too. IMO it’s not filled with buzzwords and gives enough for us to discuss their offering.
What it gives us, beyond a couple tidbits about fans and simpler electrical design, is mostly just a business pitch dressed up to feel attractive to developers.
There are interesting articles linked off of it, yes–but I think precedent here is typically to submit those articles directly (see also why we don’t really do newsletter submissions…better to submit the stories directly).
I have tremendous respect for the Oxide folks and what they’ve accomplished; make no mistake, though, that this is marketing and just a press release.
To me, this is on the bubble. There’s almost enough information about the hardware in their rack to count as a Lobster’s story, but I think it’s maybe just a hair shy. But maybe someone else will have a different opinion.
The golden rule: does reading it make people better programmers? I think some of the discussion here about how to think about hardware and software in tandem feels like this is on-topic (But I’m biased, cuz I listen to their podcast, am bought into the idea, and so know more context than what someone just reading this might get).
The entire software stack seems to be open source. That alone makes it relevant here…no? I personally have a ton of questions just skimming through a couple of repos there… I’d say this is wonderful stuff!
Just the awareness that there exist a product that is like “on-prem turn-key cloud” is a informative, IMO.
What would a rack like this typically sell for? Oxide only seems to have a “contact sales” page.
Oxide starts around $500k and only goes up from there I believe. But at this level every sale is an enterprise sale with the high touch sales process to go with it. If you buy a rack from Amazon for on-prem AWS the starting point is: have you paid for enterprise support?
I believe fully decked out non-GPU rack from HPE or Cisco would also be at around such numbers.
oooh so I’m sticking with my old supermicro 846 duck-taped shut for a while longer then
RiiR for the cloud/data-center.
What’s RiiR?
“Rewrite it in Rust”
Finally, a computer the owner has full and proper control over, with sensible open firmware.
As long as you have a million dollars to buy an entire rack :(
No option for third-party sleds?
That would be counter to their whole value prop. The entire system is meant to be tightly integrated and fully turn-key.
I mean the software and hardware specs are open source AFAIK so you could make your own sleds in theory.
I didn’t have the patience to go through their wall of small text. I always thought “the cloud” meant renting someone else’s computers and letting them take care of the scaling, backups and failovers. Is this just a … on prem server farm?
I view it as a modern version of the mainframe. One big box arrives at your doorstep. You plug it in and everything is already there and you can use it.
Thinking about this some more it is maybe more like Sun infamous DC in a container: https://en.wikipedia.org/wiki/Sun_Modular_Datacenter
Makes sense, since they’re ex-Sun people.
I find the mainframe comparison useful. The things that keep it from being a true mainframe are no longer as important to many – it’s not so important it doesn’t act as a single large computer, or bake more reliability features deep into the hardware (vs. say handling high database availability mainly by failing over). 64-core/1TB RAM/32TB SSD nodes are large enough for a lot already, and most apps are already written as a distributed system where app processes talk to DBs, caches, queues, and storage services–like, that’s the norm even when scale doesn’t require it. (Also, this product has bits like distributed block storage that bring a bit of that “big reliable hardware” illusion back.)
What is like a mainframe, from a business perspective, is:
With the technical aspects of a true mainframe much less in demand today, and the mainframe-like aspects it does have being rare today, it does feel like a modern mainframe in some relevant sense.
I wonder if the potential market–essentially, folks who want to own the resources of at least 16 of these compute sleds and have some real dissatisfaction with using a cloud vendor–will bite at this, or prefer racking Supermicro servers and generic switches or using some other flavor of non-big-3 hosting. The buzz itself might actually be what lets this take off enough to get a durable foothold when similar-seeming things have not. Of course, it also needs to work well in practice, something not really proven yet, and it has to be priced so the business can work!
I also wonder if Oxide or others are going to offer cloud hosting based on this. With what they have today, it’d be very different from what the big providers offer (just as “cloud” offerings from older-school hosting providers are), but I could see selling it on things like the latency/bandwidth advantages of a group of nodes and their storage all being connected to the same top-of-rack switches; you might be able to gin up good benchmarks based on that.
from what I read, it seems like it’s a low-hassle server farm, complete with software and firmware updates. It also seems to be configurable via APIs, similar to the cloud. I imagine you could terraform VMs and resources on it.
Unlike the cloud, you don’t pay a perpetual “tax”, it’s just a one-time purchase. Reading a few articles lately (e.g. DHH) about going on-prem from the cloud, the big argument against going back to on-prem is the hassle, and this seems to address that fairly well.
I think it’s like an “AWS in a box”, in that it’s a big pile of hardware, but with custom software that is programmable via an API.
I want one, of course, because I am a crank.
maybe EC2 in a box, but not AWS in a box.
I believe they have an S3 like blob storage, IAM like capabilities and more. It’s better than “just ec2.”
FWIW, we have an EBS-like block storage system, but we do not yet have an S3-like object store (the economics of bulk object storage on all-flash systems are not fantastic anyway). I think it’s pretty reasonable to think of it as in the EC2 space at this early moment! We’re excited about layering more on top in the future.
(Edit: I guess I don’t really think of IAM or VPC or Route 53 as being separate from EC2, but it you do, we have stuff in those areas also!)
Oh! Makes sense! Thanks for clarifying as someone with first hand understanding!
Fair enough. That is still light years away from “AWS in a box”. Maybe it is “your random VPS provider in a box”.
I don’t have an Oxide Rack brochure sitting in front of me to type out all the features. You might want to do some investigation to evaluate whether it’s just “random VPS provider in a box” or not.
The real point that is being made in this thread is that it’s not a rack that does nothing. It’s a rack that, within hours of being delivered off the truck, can be setup and used to provision production workloads. Their goal is to reduce that time as much as possible.
That is a great feature set. It is however besides the point calling it “AWS in a box”, which what I am trying to say. AWS has hundreds of services, from very low level to very high level. There is no way that you get anything close to that from Oxide. You get the hardware you could build that on, yes. Will it be highly integrated hardware/software and probably quite good? I guess so. I trust their taste. Is this “AWS in a box”? No it is not. That’s all.
That’s probably not an unreasonable way to think about it. One idly wonders if they’ll have a marketplace for service implementations that run on Oxide machines that can cover more of the use cases than “compute + storage + IAM auth”.
EC2+, maybe
Cloud means a lot of different things. That is why nearly every analysis of deployment choices related to the cloud comes with a hefty pile of caveats. The cloud is all of:
The Oxide rack handles the second and it allows you to handle the first and third. Those three together make up the 2011-ish version of AWS. The fourth is a more recent maturation of the cloud offerings. Given Oxide’s young age, they aren’t their yet, but I suspect they could grow to serve a similar role.
Usually the cost of these things is a rental fee. However that is a sales strategy by the cloud provider. The advantage to them is that rental fees present a perpetual option to increase prices and to increase the services used by a customer. Cloud providers could choose to sell their customers a portion of their data center, along with a futures contract for power and network delivery. As growth slows, I suspect we will see relationships like this, along with the associated financialization and arbitrage that goes along with those types of relationships.
In 2011 AWS already had (next to EC2 and S3) EMR, RDS, ELB, SNS, IAM, Route 53, Elastic Beanstalk, SES which are all in the 4th category. I would not call that a “recent maturation”.
source: https://en.wikipedia.org/wiki/Timeline_of_Amazon_Web_Services
How is this any different than just buying preconfigured server rack(s) from any manufacturer that offers them (practically all)? Many will even install private cloud software from the factory (OpenStack, various propriety ones). Even if your prefferred manufacturer won’t, all major manufacturers will PXE boot out of the box/crate, which makes it easy to install/run whatever you want.
I think Oxide would point out that you can do that, and then you’ll discover that there is a bizarre bug that causes rare 100ms stalls, and the server vendor will point at the SSD vendor will point at the OS vendor will point at the motherboard vendor will point at…
Oxide instead owns and is responsible for the entire stack.
(Bryan Cantrill and several Oxide folks are ex-Joyent, so they have a lot of stories on the “joy” of running a cloud on off-the-shelf hardware. [Update: you may be interested in this comment.])