Decentralization is not going to happen, at least not as long as the decentralization is being pushed by the kinds of people who comment in favor of decentralization on tech forums.
Decentralization means that each producer of a base image chooses their own way of distributing it. So the base image for your “compile” container might be someone who insists on IPFS as the one true way to decentralize, while the base image for your “deploy and run” container is someone who insists on BitTorrent, and the base image for your other service is someone who insists on a self-hosted instance of some registry software, and…
Well, who has time to look up and keep track of all that stuff and wire up all the unique pipelines for all the different options?
So people are going to centralize, again, likely on GitHub. At best they’ll start having to specify the registry/namespace and it’ll be a mix of most things on GitHub and a few in other registries like Quay or an Amazon-run public registry.
This is the same reason why git, despite being a DVCS, rapidly centralized anyway. Having to keep track of dozens of different self-hosted “decentralized” instances and maintain whatever account/credentials/mailing list subscriptions are needed to interact with them is an absolute pain. Only needing to have a single GitHub account (or at most a GitHub account and a GitLab account, though I don’t recall the last time I needed to use my GitLab account) is such a vast user-experience improvement that people happily take on the risks of centralization.
Maybe one day someone will come along and build a nice, simple, unified user interface that covers up the complexity of a truly decentralized system, and then we’ll all switch to it. But none of the people who care about decentralization on tech forums today seem to have the ability to build one and most don’t seem to care about the lack of one. So “true” decentralized services will mostly go on being a thing that only a small circle of people on tech forums use.
(even Mastodon, which has gained a lot of use lately, suffers from the “pick a server” onboarding issue, and many of the people coming from Twitter have just picked the biggest/best-known Mastodon server and called it a day, thus effectively re-centralizing even though the underlying protocol and software are decentralized)
Decentralization means that each producer of a base image chooses their own way of distributing it.
I imagine that image producers could also agree on a one distribution mechanism that doesn’t have to rely on a handful of centralized services. It doesn’t have to be a mix of incompatible file transfer protocols either, that would be really impractical.
This is the same reason why git, despite being a DVCS, rapidly centralized anyway.
The main reason was (is) probably convenience yes, but I think that Git has a different story: I may be wrong, but I don’t think that Docker was about decentralizing anything, ever. I would rather compare Git and GitHub’s relation to that of SMTP and Gmail.
Maybe one day someone will come along and build a nice, simple, unified user interface that covers up the complexity of a truly decentralized system, and then we’ll all switch to it.
Maybe, that would be convenient. I may be a bit tired, or read too much from your response, but I feel that you’re annoyed when someone points out that more centralization isn’t the best solution to centralization issues.
I fear that, because Docker sold us the idea that there’s only their own Dockerfile format, that builds on images that must be hosted on Docker Hub, we didn’t think about alternatives – well, until they added “and now you must pay to keep using things that we offered for free”. Let’s not discard all discussions on the topic of decentralization too quickly, as we could improve on Docker, and we need more ideas.
I imagine that image producers could also agree on a one distribution mechanism that doesn’t have to rely on a handful of centralized services. It doesn’t have to be a mix of incompatible file transfer protocols either, that would be really impractical.
In the various threads about Docker, people are already proposing all sorts of incompatible transfer protocols and hosting mechanisms, and displaying no interest in cooperating with each other on developing a unified standard.
The best I think we can hope for is that we get a duopoly of popular container registries, so that tooling has to accommodate the idea that there is no single “default” (as happens currently with Docker treating their own registry as the default). But I think it’s more likely that network effects and cohesiveness of user experience will push most people onto a single place, likely GitHub.
The main reason was (is) probably convenience yes, but I think that Git has a different story: I may be wrong, but I don’t think that Docker was about decentralizing anything, ever.
My point was to say “look, this thing that was explicitly designed to be decentralized, and is in a category of tools that literally have the word ‘decentralized’ in the name, still ended up centralized, that does not give high confidence that trying to decentralize Docker registries, which were not even designed with decentralization in mind, will succeed in a meaningful way”.
Let’s not discard all discussions on the topic of decentralization too quickly, as we could improve on Docker, and we need more ideas.
I will just be brutally honest here: the success rate of truly “decentralized” systems/services in the real world is incredibly low. Partly this is because they are primarily of interest to tech people who are willing to put up with a tool that is metaphorically covered in poisoned razor blades if it suits some theoretical ideal they have in mind, and as a result the user experience of decentralized systems/services tends to be absolutely abysmal. Partly this is because social factors end up concentrating and centralizing usage anyway, and this is a problem that is hard/impossible to solve at the technical level, but most people who claim to want decentralized systems are only operating on the technical design.
So I do discard discussions on decentralization, and quickly. Perhaps this time someone really will come up with a decentralized solution that gets mass adoption and manages to stay de facto decentralized even after the mass adoption occurs. But to me that statement is like “I should buy a lottery ticket, because perhaps this time I will win”.
Well, what can I say. These are some strong opinions, probably soured from various experiences. I’m not trying to convince you in particular, but do hope that we can aim higher than a duopoly. Thanks for the chat. :)
This is the same reason why git, despite being a DVCS, rapidly centralized anyway. Having to keep track of dozens of different self-hosted “decentralized” instances and maintain whatever account/credentials/mailing list subscriptions are needed to interact with them is an absolute pain
GitHub has benefits from centralisation because, as a publisher of an open-source project, I want it to be easy for people to file bugs and contribute code. Most people who might do this have a GitHub account and so anything either hosted on GitHub or hosted somewhere with sign-in with GitHub makes this easy.
I’m not sure that this applies to container images. People contributing to the code that goes into a container or raising issues against that code will still go to GitHub (or similar) not DockerHub. People interact with DockerHub largely via {docker,buildah} pull, or FROM lines in Dockerfiles. These things just take a registry name and path. As a consumer of your image, it makes absolutely no difference to me what the bit before the first / in your image name is. If it’s docker.io, quay.io, azurecr.io, or whatever, the tooling works in precisely the same way.
The only place where it makes a difference is in private registries, where ideally I want to be able to log in with some credentials that I have (and, even more ideally, I want it to be easy to grab the image in CI).
I see container registries as having more in common with web sites than code forges. There are some incentives towards centralisation (lower costs, outsourced management), but they’re not driven by network effects. I might host my web site using GitHub Pages so that GitHub gets to pay for bandwidth, but visitors find it via search engines and don’t care where it’s hosted, especially if I use my own domain for it.
Indeed. You need a place that has offers large amounts of storage and bandwidth for low cost, if not free entirely. And bandwidth has had a ridiculous premium for a very long time, which makes it very hard to find such a service.
You could find VPS providers at a reasonable rate for both of these, but now you’re managing a server on top of that. I’m not opposed to that effort, but that is not a common sentiment. 😅
A shared redirector should need less space/bandwidth than actual independent hosting, but backend-hopping would become easier… And the projects themselves (who don’t want to annoy the backends too much) don’t even need to acknowledge it aloud.
the thing is, even if you pay for them, they never seem to be satisfied. it’s also why the whole “if you don’t pay then you are the product” saying is basically false… (selling customer data is even more profitable when customers pay them to sell it)
Some people are just bad actors. But when you find a good provider, they’re much more likely to want to keep your data for you if they’re profiting from their work.
How could we know about Docker though? It’s not like the service is free, they have always had a revenue stream, and it’s the industry standard. You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases? Do you choose a different provider just because the free tier happens to be good enough and you don’t want to use something you’re not paying for?
It’s not like the service is free, they have always had a revenue stream,
Not enough of one. Docker’s lack of sustainable revenue has been discussed for years. Their business plan from the outside looked like ‘give away great tools, get bought by someone who needs them’.
and it’s the industry standard.
No, OCI is the industry standard, Docker is a company that sells the most polished tools for dealing with the industry standard.
You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases?
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
I’m pretty sure other OCI implementations are significantly less used than Docker? It certainly was a few years ago when a lot of the decisions to use Docker were made.
“Industry standard” doesn’t mean a formal standard managed by a standards body.
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
When you depend on a free offering from a company with an obviously unsustainable business model, you assume it will go away at some point and have a plan B. If they move it to a paid offering that is cheaper than your plan B, then you may choose to use that instead, but you should always assume it will disappear. Possibly with no notice if the company goes bust.
But how do you price that in when the industry-standard service is free for your use-case and they’re not giving any indication of what your use-case might cost in the future? How do you price in a completely unknown potential future price which may or may not materialise?
The price depends on what strategy you use to manage the risk
You pay to avoid the risk. You host your own docker registry, or you don’t use docker at all. The price is whatever you have to pay.
You don’t pay anything upfront and assume the risk. If it happen, it’s the end of the road, and so be it. The price is what you lose if it happens. If you have a small project you do for fun, this may be the easiest solution. If the project is your livelihood, this is a dangerous gamble. Maybe look at option 1 or 3 instead.
You pay for a contingency plan to reduce the impact of the risk. Maybe you keep a docker registry hosted somewhere, ready to take over when your main registry fail. Since you don’t pay the bandwidth, it remain pretty low cost until the risk happens. Or you always keep enough cash flow to switch to a paid plan if the free option disappear. The price is what you pay to keep the contingency available, plus the price to use it if it happens.
The solution is probably a service that makes hosting free but charges for downloads. For commercial products, increasing downloads increases revenue and so it’s completely fine to increase the costs to the publisher as the number of downloads goes up, to cover the traffic. For community-developed projects, most users pay nothing.
GitHub is probably in a good position to keep this kind of thing free because they want to monetise the things that pull containers. In particular, Actions and CodeSpaces are both very large consumers of containers and benefit from making it easy for people to expose a large number of container images to their customers.
Today’s it’s Docker, yesterday it was Google code, CentOS, Heroku or MongoDB, tomorrow it is gonna be Gmail, Npm, GitHub, Discord, or even a paid service like AWS. Service provided by company come and go. Free service, especially, come and go very quickly, as they are prone to be cut first when shareholder ask for more profit.
Usually when that happen, people seems to want me to feel offended and angry at whoever pulled the service. But just can’t muster the feeling anymore. It’s just nature following it’s course you know? Loss is part of life, it make me sad, but there is no point in being angry about it.
Open source have little money, and often depend on public charity so to speak, so I understand that they have little choice but to use those services. I see no good solution. All I can do is to accept that all of it, free service and open source project that depend on it, are perishable. And to act accordingly, by having a copy of the data I need on storage I control, or by making whatever open source library easy to replace. Then whatever happen, I move on with my life.
The problem isn’t the services going away, it’s the namespaces going away. Services like GitHub, Docker, and so on make it easy to have your project’s canonical address live in a domain that they own. Someone like the FSF could do a lot of good by registering a few domains for F/OSS projects and managing redirects so that you could still host your work with these services but they’d be redirected elsewhere. For example, if containers.myproject.isfreesoftware.org was a cname for docker.io then I could trivially switch from using docker.io to azurecr.io or quay.io without my downstream users noticing.
A big part of the problem is that Docker Inc framed things on their site such that specific images were implied to be “canonical” for a project despite frequently having no association whatsoever with the people who actually run the project in question. They’ve been misleading people for years and as far as I can tell no one really cared or noticed? I never could understand why people ever trusted Docker.
If true, that’s a violation of even most permissive F/OSS licenses (the name of the project must not be used to endorse…), so there’s a lot of potential legal liability that someone looking at buying Docker Inc. would need to factor in.
I think a lawyer must have stepped in since the last time I looked at it, because the situation is at least a little better than it used to be: https://registry.hub.docker.com/_/clojure/
In the center section it now says “Maintained by: the Docker Community” which was not there previously. However, the page title still says “clojure - Official image” which is incorrect, and as you’ve pointed out, probably also illegal as per this section of the license for the project in question:
You may not implicitly or explicitly assert or imply any
connection with, sponsorship or endorsement by the Original
Author, Licensor and/or Attribution Parties, as appropriate, of
You or Your use of the Work, without the separate, express prior
written permission of the Original Author, Licensor and/or
Attribution Parties.
Hosting has always been the Achilles heel of open source, the thing that keeps the cost of reproduction for software from being exactly zero. I guess this is what BitTorrent was meant to solve by allowing users to contribute back hosting responsibilities, but it still has worse UX than just downloading something from a single host and doesn’t work for live services.
I don’t have a solution for this, but suggesting GitHub as an alternative to now-hostile Docker seems to move the issue from one silo to the next.
Decentralization is not going to happen, at least not as long as the decentralization is being pushed by the kinds of people who comment in favor of decentralization on tech forums.
Decentralization means that each producer of a base image chooses their own way of distributing it. So the base image for your “compile” container might be someone who insists on IPFS as the one true way to decentralize, while the base image for your “deploy and run” container is someone who insists on BitTorrent, and the base image for your other service is someone who insists on a self-hosted instance of some registry software, and…
Well, who has time to look up and keep track of all that stuff and wire up all the unique pipelines for all the different options?
So people are going to centralize, again, likely on GitHub. At best they’ll start having to specify the registry/namespace and it’ll be a mix of most things on GitHub and a few in other registries like Quay or an Amazon-run public registry.
This is the same reason why git, despite being a DVCS, rapidly centralized anyway. Having to keep track of dozens of different self-hosted “decentralized” instances and maintain whatever account/credentials/mailing list subscriptions are needed to interact with them is an absolute pain. Only needing to have a single GitHub account (or at most a GitHub account and a GitLab account, though I don’t recall the last time I needed to use my GitLab account) is such a vast user-experience improvement that people happily take on the risks of centralization.
Maybe one day someone will come along and build a nice, simple, unified user interface that covers up the complexity of a truly decentralized system, and then we’ll all switch to it. But none of the people who care about decentralization on tech forums today seem to have the ability to build one and most don’t seem to care about the lack of one. So “true” decentralized services will mostly go on being a thing that only a small circle of people on tech forums use.
(even Mastodon, which has gained a lot of use lately, suffers from the “pick a server” onboarding issue, and many of the people coming from Twitter have just picked the biggest/best-known Mastodon server and called it a day, thus effectively re-centralizing even though the underlying protocol and software are decentralized)
I imagine that image producers could also agree on a one distribution mechanism that doesn’t have to rely on a handful of centralized services. It doesn’t have to be a mix of incompatible file transfer protocols either, that would be really impractical.
The main reason was (is) probably convenience yes, but I think that Git has a different story: I may be wrong, but I don’t think that Docker was about decentralizing anything, ever. I would rather compare Git and GitHub’s relation to that of SMTP and Gmail.
Maybe, that would be convenient. I may be a bit tired, or read too much from your response, but I feel that you’re annoyed when someone points out that more centralization isn’t the best solution to centralization issues.
I fear that, because Docker sold us the idea that there’s only their own Dockerfile format, that builds on images that must be hosted on Docker Hub, we didn’t think about alternatives – well, until they added “and now you must pay to keep using things that we offered for free”. Let’s not discard all discussions on the topic of decentralization too quickly, as we could improve on Docker, and we need more ideas.
In the various threads about Docker, people are already proposing all sorts of incompatible transfer protocols and hosting mechanisms, and displaying no interest in cooperating with each other on developing a unified standard.
The best I think we can hope for is that we get a duopoly of popular container registries, so that tooling has to accommodate the idea that there is no single “default” (as happens currently with Docker treating their own registry as the default). But I think it’s more likely that network effects and cohesiveness of user experience will push most people onto a single place, likely GitHub.
My point was to say “look, this thing that was explicitly designed to be decentralized, and is in a category of tools that literally have the word ‘decentralized’ in the name, still ended up centralized, that does not give high confidence that trying to decentralize Docker registries, which were not even designed with decentralization in mind, will succeed in a meaningful way”.
I will just be brutally honest here: the success rate of truly “decentralized” systems/services in the real world is incredibly low. Partly this is because they are primarily of interest to tech people who are willing to put up with a tool that is metaphorically covered in poisoned razor blades if it suits some theoretical ideal they have in mind, and as a result the user experience of decentralized systems/services tends to be absolutely abysmal. Partly this is because social factors end up concentrating and centralizing usage anyway, and this is a problem that is hard/impossible to solve at the technical level, but most people who claim to want decentralized systems are only operating on the technical design.
So I do discard discussions on decentralization, and quickly. Perhaps this time someone really will come up with a decentralized solution that gets mass adoption and manages to stay de facto decentralized even after the mass adoption occurs. But to me that statement is like “I should buy a lottery ticket, because perhaps this time I will win”.
Well, what can I say. These are some strong opinions, probably soured from various experiences. I’m not trying to convince you in particular, but do hope that we can aim higher than a duopoly. Thanks for the chat. :)
GitHub has benefits from centralisation because, as a publisher of an open-source project, I want it to be easy for people to file bugs and contribute code. Most people who might do this have a GitHub account and so anything either hosted on GitHub or hosted somewhere with sign-in with GitHub makes this easy.
I’m not sure that this applies to container images. People contributing to the code that goes into a container or raising issues against that code will still go to GitHub (or similar) not DockerHub. People interact with DockerHub largely via
{docker,buildah} pull, orFROMlines inDockerfiles. These things just take a registry name and path. As a consumer of your image, it makes absolutely no difference to me what the bit before the first/in your image name is. If it’s docker.io, quay.io, azurecr.io, or whatever, the tooling works in precisely the same way.The only place where it makes a difference is in private registries, where ideally I want to be able to log in with some credentials that I have (and, even more ideally, I want it to be easy to grab the image in CI).
I see container registries as having more in common with web sites than code forges. There are some incentives towards centralisation (lower costs, outsourced management), but they’re not driven by network effects. I might host my web site using GitHub Pages so that GitHub gets to pay for bandwidth, but visitors find it via search engines and don’t care where it’s hosted, especially if I use my own domain for it.
Indeed. You need a place that has offers large amounts of storage and bandwidth for low cost, if not free entirely. And bandwidth has had a ridiculous premium for a very long time, which makes it very hard to find such a service.
You could find VPS providers at a reasonable rate for both of these, but now you’re managing a server on top of that. I’m not opposed to that effort, but that is not a common sentiment. 😅
Time for a project that combines Docker with Bittorrent.
A shared redirector should need less space/bandwidth than actual independent hosting, but backend-hopping would become easier… And the projects themselves (who don’t want to annoy the backends too much) don’t even need to acknowledge it aloud.
I wonder why nobody mentions quay.io. They offer similar services, and it’s owned by RedHat.
I was thinking the same thing. I’m hoping this pushes more folks over to quay.io.
And RedHat has proven repeatedly that they are an incredibly good citizen of the community. I’m a fan.
Maybe the workaround is to pay for the services you need, so you won’t have to keep moving from one investor-bilking scam to the next?
the thing is, even if you pay for them, they never seem to be satisfied. it’s also why the whole “if you don’t pay then you are the product” saying is basically false… (selling customer data is even more profitable when customers pay them to sell it)
Some people are just bad actors. But when you find a good provider, they’re much more likely to want to keep your data for you if they’re profiting from their work.
How could we know about Docker though? It’s not like the service is free, they have always had a revenue stream, and it’s the industry standard. You see that, look at the pricing model, and … what do you do when you see that their free tier happens to be enough for your use cases? Do you choose a different provider just because the free tier happens to be good enough and you don’t want to use something you’re not paying for?
Not enough of one. Docker’s lack of sustainable revenue has been discussed for years. Their business plan from the outside looked like ‘give away great tools, get bought by someone who needs them’.
No, OCI is the industry standard, Docker is a company that sells the most polished tools for dealing with the industry standard.
You look at their free tier and see that something free is going to have to go away if they aren’t bought by GitHub or similar (and then it may still go away, just be bundled with something you might already be paying for) and avoid making plans around it always being there for free.
I’m pretty sure other OCI implementations are significantly less used than Docker? It certainly was a few years ago when a lot of the decisions to use Docker were made.
“Industry standard” doesn’t mean a formal standard managed by a standards body.
Okay, and how do you do that when there’s no indication at all about what a potential future price might be? Do you put your finger in the air and guess? What if it ends up being significantly more expensive than you guessed?
When you depend on a free offering from a company with an obviously unsustainable business model, you assume it will go away at some point and have a plan B. If they move it to a paid offering that is cheaper than your plan B, then you may choose to use that instead, but you should always assume it will disappear. Possibly with no notice if the company goes bust.
It may well be; but, if your usecase has the hidden requirement of dependability/continuity of service, you need to price that in.
There ain’t no such thing as a free lunch, and never has been.
But how do you price that in when the industry-standard service is free for your use-case and they’re not giving any indication of what your use-case might cost in the future? How do you price in a completely unknown potential future price which may or may not materialise?
The price depends on what strategy you use to manage the risk
You can’t know, and nobody sensible expects you to. That’s what happens when you outsource something, whether you pay for it or not.
Paid services disappear constantly too! Company collapses, pivots, acquihires, etc.
https://ourincrediblejourney.tumblr.com/
Until they decide that they need more.
Gitlab going from $4 -> $29 per user per month for my same usage in 2 years for example.
The solution is probably a service that makes hosting free but charges for downloads. For commercial products, increasing downloads increases revenue and so it’s completely fine to increase the costs to the publisher as the number of downloads goes up, to cover the traffic. For community-developed projects, most users pay nothing.
GitHub is probably in a good position to keep this kind of thing free because they want to monetise the things that pull containers. In particular, Actions and CodeSpaces are both very large consumers of containers and benefit from making it easy for people to expose a large number of container images to their customers.
A little misunderstanding. Docker has apologised: https://www.docker.com/blog/we-apologize-we-did-a-terrible-job-announcing-the-end-of-docker-free-teams/
I’d like to see a “requestor pays” container registry. The fee could cover the registry costs and you could add a markup to fund/donate the developer.
Today’s it’s Docker, yesterday it was Google code, CentOS, Heroku or MongoDB, tomorrow it is gonna be Gmail, Npm, GitHub, Discord, or even a paid service like AWS. Service provided by company come and go. Free service, especially, come and go very quickly, as they are prone to be cut first when shareholder ask for more profit.
Usually when that happen, people seems to want me to feel offended and angry at whoever pulled the service. But just can’t muster the feeling anymore. It’s just nature following it’s course you know? Loss is part of life, it make me sad, but there is no point in being angry about it.
Open source have little money, and often depend on public charity so to speak, so I understand that they have little choice but to use those services. I see no good solution. All I can do is to accept that all of it, free service and open source project that depend on it, are perishable. And to act accordingly, by having a copy of the data I need on storage I control, or by making whatever open source library easy to replace. Then whatever happen, I move on with my life.
The problem isn’t the services going away, it’s the namespaces going away. Services like GitHub, Docker, and so on make it easy to have your project’s canonical address live in a domain that they own. Someone like the FSF could do a lot of good by registering a few domains for F/OSS projects and managing redirects so that you could still host your work with these services but they’d be redirected elsewhere. For example, if
containers.myproject.isfreesoftware.orgwas a cname fordocker.iothen I could trivially switch from usingdocker.iotoazurecr.ioorquay.iowithout my downstream users noticing.A big part of the problem is that Docker Inc framed things on their site such that specific images were implied to be “canonical” for a project despite frequently having no association whatsoever with the people who actually run the project in question. They’ve been misleading people for years and as far as I can tell no one really cared or noticed? I never could understand why people ever trusted Docker.
If true, that’s a violation of even most permissive F/OSS licenses (the name of the project must not be used to endorse…), so there’s a lot of potential legal liability that someone looking at buying Docker Inc. would need to factor in.
I think a lawyer must have stepped in since the last time I looked at it, because the situation is at least a little better than it used to be: https://registry.hub.docker.com/_/clojure/
In the center section it now says “Maintained by: the Docker Community” which was not there previously. However, the page title still says “clojure - Official image” which is incorrect, and as you’ve pointed out, probably also illegal as per this section of the license for the project in question:
Hosting has always been the Achilles heel of open source, the thing that keeps the cost of reproduction for software from being exactly zero. I guess this is what BitTorrent was meant to solve by allowing users to contribute back hosting responsibilities, but it still has worse UX than just downloading something from a single host and doesn’t work for live services.