I have lots of repositories on GitHub that I share publicly (because why not), but which I don’t consider worth listing on a website where people go to find serious programming libraries.
I think there’s a misunderstanding here not only about pkg.go.dev, but about sites like RubyGems.
The first package managers to achieve mainstream adoption among open-source programmers were run by Linux distribution such as Debian or Red Hat. These package managers introduced two ideas: (1) each package has a unique name assigned by the package maintainer[0], and (2) the set of available packages is curated. A package is only added to the Debian package list when someone wants it and is willing to commit to maintaining it, which usually implies a certain level of quality.
Sites like RubyGems, NPM, Hackage, and PyPI came about later. They imitate the structure of Linux distribution package managers by assigning each package a name and a (possibly third-party) maintainer, but they dropped the curation. Anybody can add a package at any time, there’s not even a requirement that the code builds successfully.
[0] I run apt install gcc to install the software hosted at https://gcc.gnu.org/. The assignment of short name gcc to that particular package is arbitrary – it might have been gnu-gcc instead if there was already a George’s Compiler Collection in popular use. The lack of namespacing is was regrettable even in the ‘90s, and IMO it’s completely unreasonable that modern package registries like crates.io still use a flat namespace.
To put this another way, I think that there is (and should be) a significant distinction between sharing something on GitHub (or the equivalent) and sharing something on RubyGems or pkg.go.dev. In effect, the people I talked with in Go’s slack didn’t think that this distinction was real or useful. Maybe they’re right. I wonder what other people think.
The Go developers, for better or worse, do not spend much effort looking at the state of the world when they decide how to solve any given problem. For packages, they decided that without curation assigning a distinct name and maintainer to each package is silly. Go’s package semantics are therefore “web-native”, in that third-party Go packages are located rather than identified and there’s not even a hint of curation. Anybody with a website can host a Go package that stands on even ground with official packages hosted on golang.org.
In a world where packages have no names, only locations, the purpose of pkg.go.dev is like that of archive.org. It is a mirror of what exists (or existed). There’s no need for curation because there’s no name conflicts, and Google isn’t likely to run out of storage space any time soon, so why not just slurp down every copy of every Go package that’s ever been installed?
The lack of namespacing is was regrettable even in the ‘90s, and IMO it’s completely unreasonable that modern package registries like crates.io still use a flat namespace.
I think the reason to use URLs as a namespace is that there is already a single, global, unique namespace that people have put a lot of security work into. You’re not going to be able to make something more secure than DNS + TLS. At best, you can remove some of the footguns of that system, but you’re probably still going to be relying on DNS + TLS yourself for users to connect to your package site anyway. Remove unnecessary layers: just use URLs as your namespace directly instead of connecting to pypi.org and then looking up uniquename.
I know there’s no trivial fix here, but, related to “Why Share What You Don’t Want to Share?”, it’s notable how unclear the expectations are around the majority of public code out there. Your language’s standard library and lots of big or long-standing projects have an org behind them, some momentum, and a policy, but most projects depend on some code that has none of those.
Sometimes this plays out as people who just released some code being approached on a public issue tracker as if they were paid maintainers. For me, sometimes publishing code was held up by it not feeling like a proper product, and I’ve heard the same for others. Now and then we end up with dependencies actually proving untrustworthy, and picking packages can involve lore or frustration (“use Y instead of X; X looks great but you’ll find nasty bugs two days into using it”).
Platforms like GitHub, pkg.go.dev, or crates.io do some things for you–you get some indicators of popularity, recency and level of activity, folks can use badges to suggest e.g. they at least have some kind of tests, and once a dependency passes the smell test you can dive in for more thorough checks. But it all feels kind of unnecessarily hazy, both for those trying to make educated guesses about what to depend on and those putting code out there.
If I had a magic wand to wave, maybe there would be more explicit and systematic (but not binding) statements of project intent and status out there, e.g. ‘this is just for fun’ or ‘it’s a personal project, but I intend to keep it stable’ or ‘we at jfkldajskl corp. depend on this in production and maintain it’. Trusted folks doing curation (less “awesome X”, more orgs you recognize who use packages in anger blessing them), maybe linked to some resources for projects/maintainers, could help too.
The point of this isn’t to rail about the economics, but it also isn’t lost on me this is the second-order effect of how a lot of open source turns people’s hobby work into value for well-funded companies. Maybe part of a durable solution would be for those that want to rely on a project kick in enough to help make it reliable–improved channels/systems for that to happen and the companies involved deciding to use those channels.
What do people think happens when a package’s documentation appears on pkg.go.dev? Are people using that site for package discovery? I commit go packages to git(hub|lab) in various states of usability, and pkg.go.dev renders the documentation for it. I’ve never considered them scraping my packages and putting up a page with docs to be any kind of “publishing”. What about godocs.org? Do you “publish” there separately? Several other commenter have attempted to describe the difference between pkg.go.dev and pypi/npm/etc but those are almost entirely different things in my head. pkg.go.dev is like doxygen, but fully automatic for every go package, and hosted for you by Google. Another example that was pointed out to me is docs.rs, which seems to automatically document Rust crates.
Sorry. That means I wasn’t clear enough. To be clearer, I mistakenly assumed that pkg.go.dev was like pypi, CPAN, RubyGems, or LuaRocks. I was definitely wrong, but that’s what I thought. I also didn’t realize that pkg.go.dev was merely “like doxygen, but fully automatic for every go package.” To be honest, I’m not sure that’s a universal view (though some people have said to me that they visit pkg.go.dev mostly for documentation.) For example, @peterbourgon, in another comment here, says, “[pkg.go.dev is] a catalog of all the code that exists and can be consumed. It’s a lazy read-thru cache.” That’s not quite the same thing as automatic doxygen for all Go packages. On pkg.go.dev’s about page, they say only, “Welcome to pkg.go.dev, your source for information about Go packages and modules.” That’s probably not meant as a definitive statement of what pkg.go.dev is for. Nevertheless, it doesn’t say or suggest exactly what you or Peter say. (My point being, I’m not sure that pkg.go.dev has a single clear purpose. That’s fine, but many people seem to think otherwise. Maybe some of them are right, but I’m not sure.)
I’m not sure how much all of this matters, but I wrote the post to describe my own misunderstanding and to help prevent others from having the same confusion. I also wanted to find out what other people thought about pkg.go.dev. Thanks for your response.
I think while Peter is technically correct there, since there is a search functionality so you can technically browse it. It shouldnt be considered a curated catalog. It’s just a catalog of all go software they can find anywhere on the internet. I searched for a project of mine that is primarily in Java but has a go client, and I found not only the go clients, but also internal CI components of the main project that happen to be written in go but would never be useful by anyone outside outside the main java project.
edit: also to be clear, I don’t think your post is confusing, I’m just confused how one could arrive at the idea that they should be treated like pypi/npm/etc, since it has fairly different functionality. I suppose if no one told you that go doesnt have one of those, you could reasonably assume it worked like every other language out there.
Here are two things that I learned after I listed a Go module on pkg.go.dev. Neither was obvious to me; hopefully, this will be useful to other people too.
The pkg.go.dev site isn’t a registry that you explicitly publish your module into, or anything like that. It’s a catalog of all the code that exists and can be consumed. It’s a lazy read-thru cache. The existence of a package in that site doesn’t imply anything about it’s production-readiness or etc.
All that said, it is easy to avoid this problem (once you know to avoid it). On projects that you may share publicly in a git repository but that you don’t consider serious enough for pkg.go.dev, don’t use a proper, full import path
It is not a problem that needs to be avoided. A project that is included in pkg.go.dev doesn’t imply anything beyond existence. There is no reason to make your code unusable (as is suggested here) in order to keep it out of that index.
To put this another way, I think that there is (and should be) a significant distinction between sharing something on GitHub (or the equivalent) and sharing something on RubyGems or pkg.go.dev.
RubyGems is a registry that code authors publish to explicitly. pkg.go.dev has totally different semantics.
The pkg.go.dev site isn’t a registry that you explicitly publish your module into, or anything like that. It’s a catalog of all the code that exists and can be consumed. It’s a lazy read-thru cache.
Right. I didn’t realize that. That’s my main point at the top of the post: I was very confused about what pkg.go.dev is and how it works. For what it’s worth, I tried hard not to blame anyone but me for that misunderstanding. I’m not upset with anyone. I don’t think pkg.go.dev is doing anything evil or wrong. But I was very confused about how it works and what it is.
The existence of a package in that site doesn’t imply anything about it’s production-readiness or etc.
I also didn’t realize that many people who write Go feel this way (though not all, based on my conversations with people about this on- and offline). Here’s a (random) quote from someone on the Gopher slack, “for me, seeing something on pkg.go.dev seems to suggest some level of support or guarantee for a module” (their emphasis). This other person may also be wrong, but people definitely don’t all have the same understanding of pkg.go.dev. I wrote the post partly to work out my own understanding of the site and partly to hear what others thought. Thanks for telling me what you think.
It is not a problem that needs to be avoided. A project that is included in pkg.go.dev doesn’t imply anything beyond existence. There is no reason to make your code unusable (as is suggested here) in order to keep it out of that index.
I can only say that I disagree. Here’s one example. When I read The Go Programming Language, I took notes in markdown and wrote code for lots of the exercises. I kept all that in a git repository, and I put the repository on GItHub so that I could easily work on it both at home and at work. I made the repository public because (a) why not and (b) others had done the same and their notes and code helped me. That repository ended up on pkg.go.dev, and I wish it hadn’t. In the future, I would use the trick I mention in the post to avoid this.
I understand everything you’re saying about no implication of production-readiness and so on, but I still think it’s ridiculous for my notes on GOPL to end up on pkg.go.dev. It’s doubly ridiculous since the repo didn’t have a license that pkg.go.dev recognizes. As a result, it’s listed there permanently and the code is not visible. (Which, I guess I’m grateful for since lots of it is embarrassing code.) I could file an issue to remove the listing, but that also feels silly. Why give busy work to someone else?
but people definitely don’t all have the same understanding of pkg.go.dev.
A real problem! Which I hope this post, and the comments to it, are able to correct, at least in some small way.
I still think it’s ridiculous for my notes on GOPL to end up on pkg.go.dev. It’s doubly ridiculous since the repo didn’t have a license that pkg.go.dev recognizes. As a result, it’s listed there permanently and the code is not visible. (Which, I guess I’m grateful for since lots of it is embarrassing code.) I could file an issue to remove the listing, but that also feels silly. Why give busy work to someone else?
Anything and everything on github.com, or any other source of truth for source code, can “end up” on pkg.go.dev. What goes there isn’t really a decision that anyone makes, and isn’t something you should feel like you need to manage. The site is more infrastructure than product. It’s a special-purpose archive.org.
For what it’s worth, I agree it’s weird, and surprising. The whole philosophy behind Go modules is surprising. But the ship has sailed.
This seems weird. I was planning on building a few packages and hosting them on github to get some feedback before I get it “officially” published but looks like I need to put some hack in so that it doesn’t reflect as a relase package.
Any way to see the slack discussion? Interested in what arguments did the maintainers have.
Is a hack necessary? Can`t you just version it something like v0.0.1-alpha or put a line in the README?
If people ignore that and import your toy code into their production system anyway, it really is their own fault. Whether they found your code on Github or on pkg.go.dev is irrelevant.
I think there’s a misunderstanding here not only about pkg.go.dev, but about sites like RubyGems.
The first package managers to achieve mainstream adoption among open-source programmers were run by Linux distribution such as Debian or Red Hat. These package managers introduced two ideas: (1) each package has a unique name assigned by the package maintainer[0], and (2) the set of available packages is curated. A package is only added to the Debian package list when someone wants it and is willing to commit to maintaining it, which usually implies a certain level of quality.
Sites like RubyGems, NPM, Hackage, and PyPI came about later. They imitate the structure of Linux distribution package managers by assigning each package a name and a (possibly third-party) maintainer, but they dropped the curation. Anybody can add a package at any time, there’s not even a requirement that the code builds successfully.
I think most people would hesitate before they added a dependency on https://cs.nowhere.edu/~jdoe/algo103/utils/right-pad, but they don’t realize
npm install right-padmeans the same thing.[0] I run
apt install gccto install the software hosted at https://gcc.gnu.org/. The assignment of short namegccto that particular package is arbitrary – it might have beengnu-gccinstead if there was already a George’s Compiler Collection in popular use. The lack of namespacing is was regrettable even in the ‘90s, and IMO it’s completely unreasonable that modern package registries like crates.io still use a flat namespace.The Go developers, for better or worse, do not spend much effort looking at the state of the world when they decide how to solve any given problem. For packages, they decided that without curation assigning a distinct name and maintainer to each package is silly. Go’s package semantics are therefore “web-native”, in that third-party Go packages are located rather than identified and there’s not even a hint of curation. Anybody with a website can host a Go package that stands on even ground with official packages hosted on
golang.org.In a world where packages have no names, only locations, the purpose of pkg.go.dev is like that of archive.org. It is a mirror of what exists (or existed). There’s no need for curation because there’s no name conflicts, and Google isn’t likely to run out of storage space any time soon, so why not just slurp down every copy of every Go package that’s ever been installed?
I think the reason to use URLs as a namespace is that there is already a single, global, unique namespace that people have put a lot of security work into. You’re not going to be able to make something more secure than DNS + TLS. At best, you can remove some of the footguns of that system, but you’re probably still going to be relying on DNS + TLS yourself for users to connect to your package site anyway. Remove unnecessary layers: just use URLs as your namespace directly instead of connecting to pypi.org and then looking up uniquename.
I know there’s no trivial fix here, but, related to “Why Share What You Don’t Want to Share?”, it’s notable how unclear the expectations are around the majority of public code out there. Your language’s standard library and lots of big or long-standing projects have an org behind them, some momentum, and a policy, but most projects depend on some code that has none of those.
Sometimes this plays out as people who just released some code being approached on a public issue tracker as if they were paid maintainers. For me, sometimes publishing code was held up by it not feeling like a proper product, and I’ve heard the same for others. Now and then we end up with dependencies actually proving untrustworthy, and picking packages can involve lore or frustration (“use Y instead of X; X looks great but you’ll find nasty bugs two days into using it”).
Platforms like GitHub, pkg.go.dev, or crates.io do some things for you–you get some indicators of popularity, recency and level of activity, folks can use badges to suggest e.g. they at least have some kind of tests, and once a dependency passes the smell test you can dive in for more thorough checks. But it all feels kind of unnecessarily hazy, both for those trying to make educated guesses about what to depend on and those putting code out there.
If I had a magic wand to wave, maybe there would be more explicit and systematic (but not binding) statements of project intent and status out there, e.g. ‘this is just for fun’ or ‘it’s a personal project, but I intend to keep it stable’ or ‘we at jfkldajskl corp. depend on this in production and maintain it’. Trusted folks doing curation (less “awesome X”, more orgs you recognize who use packages in anger blessing them), maybe linked to some resources for projects/maintainers, could help too.
The point of this isn’t to rail about the economics, but it also isn’t lost on me this is the second-order effect of how a lot of open source turns people’s hobby work into value for well-funded companies. Maybe part of a durable solution would be for those that want to rely on a project kick in enough to help make it reliable–improved channels/systems for that to happen and the companies involved deciding to use those channels.
This entire post is very confusing to me.
What do people think happens when a package’s documentation appears on pkg.go.dev? Are people using that site for package discovery? I commit go packages to git(hub|lab) in various states of usability, and pkg.go.dev renders the documentation for it. I’ve never considered them scraping my packages and putting up a page with docs to be any kind of “publishing”. What about godocs.org? Do you “publish” there separately? Several other commenter have attempted to describe the difference between pkg.go.dev and pypi/npm/etc but those are almost entirely different things in my head. pkg.go.dev is like doxygen, but fully automatic for every go package, and hosted for you by Google. Another example that was pointed out to me is docs.rs, which seems to automatically document Rust crates.
Sorry. That means I wasn’t clear enough. To be clearer, I mistakenly assumed that pkg.go.dev was like pypi, CPAN, RubyGems, or LuaRocks. I was definitely wrong, but that’s what I thought. I also didn’t realize that pkg.go.dev was merely “like doxygen, but fully automatic for every go package.” To be honest, I’m not sure that’s a universal view (though some people have said to me that they visit pkg.go.dev mostly for documentation.) For example, @peterbourgon, in another comment here, says, “[pkg.go.dev is] a catalog of all the code that exists and can be consumed. It’s a lazy read-thru cache.” That’s not quite the same thing as automatic doxygen for all Go packages. On pkg.go.dev’s about page, they say only, “Welcome to pkg.go.dev, your source for information about Go packages and modules.” That’s probably not meant as a definitive statement of what pkg.go.dev is for. Nevertheless, it doesn’t say or suggest exactly what you or Peter say. (My point being, I’m not sure that pkg.go.dev has a single clear purpose. That’s fine, but many people seem to think otherwise. Maybe some of them are right, but I’m not sure.)
I’m not sure how much all of this matters, but I wrote the post to describe my own misunderstanding and to help prevent others from having the same confusion. I also wanted to find out what other people thought about pkg.go.dev. Thanks for your response.
I think while Peter is technically correct there, since there is a search functionality so you can technically browse it. It shouldnt be considered a curated catalog. It’s just a catalog of all go software they can find anywhere on the internet. I searched for a project of mine that is primarily in Java but has a go client, and I found not only the go clients, but also internal CI components of the main project that happen to be written in go but would never be useful by anyone outside outside the main java project.
edit: also to be clear, I don’t think your post is confusing, I’m just confused how one could arrive at the idea that they should be treated like pypi/npm/etc, since it has fairly different functionality. I suppose if no one told you that go doesnt have one of those, you could reasonably assume it worked like every other language out there.
The pkg.go.dev site isn’t a registry that you explicitly publish your module into, or anything like that. It’s a catalog of all the code that exists and can be consumed. It’s a lazy read-thru cache. The existence of a package in that site doesn’t imply anything about it’s production-readiness or etc.
It is not a problem that needs to be avoided. A project that is included in pkg.go.dev doesn’t imply anything beyond existence. There is no reason to make your code unusable (as is suggested here) in order to keep it out of that index.
RubyGems is a registry that code authors publish to explicitly. pkg.go.dev has totally different semantics.
Right. I didn’t realize that. That’s my main point at the top of the post: I was very confused about what pkg.go.dev is and how it works. For what it’s worth, I tried hard not to blame anyone but me for that misunderstanding. I’m not upset with anyone. I don’t think pkg.go.dev is doing anything evil or wrong. But I was very confused about how it works and what it is.
I also didn’t realize that many people who write Go feel this way (though not all, based on my conversations with people about this on- and offline). Here’s a (random) quote from someone on the Gopher slack, “for me, seeing something on pkg.go.dev seems to suggest some level of support or guarantee for a module” (their emphasis). This other person may also be wrong, but people definitely don’t all have the same understanding of pkg.go.dev. I wrote the post partly to work out my own understanding of the site and partly to hear what others thought. Thanks for telling me what you think.
I can only say that I disagree. Here’s one example. When I read The Go Programming Language, I took notes in markdown and wrote code for lots of the exercises. I kept all that in a git repository, and I put the repository on GItHub so that I could easily work on it both at home and at work. I made the repository public because (a) why not and (b) others had done the same and their notes and code helped me. That repository ended up on pkg.go.dev, and I wish it hadn’t. In the future, I would use the trick I mention in the post to avoid this.
I understand everything you’re saying about no implication of production-readiness and so on, but I still think it’s ridiculous for my notes on GOPL to end up on pkg.go.dev. It’s doubly ridiculous since the repo didn’t have a license that pkg.go.dev recognizes. As a result, it’s listed there permanently and the code is not visible. (Which, I guess I’m grateful for since lots of it is embarrassing code.) I could file an issue to remove the listing, but that also feels silly. Why give busy work to someone else?
A real problem! Which I hope this post, and the comments to it, are able to correct, at least in some small way.
Anything and everything on github.com, or any other source of truth for source code, can “end up” on pkg.go.dev. What goes there isn’t really a decision that anyone makes, and isn’t something you should feel like you need to manage. The site is more infrastructure than product. It’s a special-purpose archive.org.
For what it’s worth, I agree it’s weird, and surprising. The whole philosophy behind Go modules is surprising. But the ship has sailed.
This seems weird. I was planning on building a few packages and hosting them on github to get some feedback before I get it “officially” published but looks like I need to put some hack in so that it doesn’t reflect as a relase package.
Any way to see the slack discussion? Interested in what arguments did the maintainers have.
Is a hack necessary? Can`t you just version it something like v0.0.1-alpha or put a line in the README?
If people ignore that and import your toy code into their production system anyway, it really is their own fault. Whether they found your code on Github or on pkg.go.dev is irrelevant.
I’ve been releasing Go packages on Github for years, believe me, you don’t have to worry about getting too much unwanted uptake. :-)
If you log into the main Gophers slack (https://join.slack.com/t/gophers/shared_invite/zt-1t9iutngg-Qxwm38dbU4v8UYsaeKrrTQ) and join the
pkgsitechannel, you can view the whole conversation. (It’s not a very busy channel, and the conversation was just a few days ago. Just scroll up a bit.)