Personally I trust Russ Cox’s judgement… though I could see how people who worked on ‘dep’ would be furious. Go has a reputation for taking community direction with a grain of salt. The go team certainly is not afraid to do unpopular things in the goal of simplicity.
vgo is certainly different than dep, and in some ways it’s simpler, but in other ways it pushes a lot more complexity on the user. I think on balance it’s got to be a wash, at least for now.
How they managed to not solve many hard problems of a language, it’s tooling or production workflow, but also solve a set to get a huge amount of developer mindshare is something I think we should get historians to look into.
I used Go professionally for ~2+ years, and so much of it was frustrating to me, but large swaths of our team found it largely pleasant.
I’d guess there is a factor depending on what you want from a language. Sure, it doesn’t have generics and it’s versioning system leaves a lot to be wished for. But personally, if I have to write anything with networking and concurrency, usualy my first choice is Go, because of it’s very nice standard library and a certain sense of being thought-thorugh when it comes to concurrency/parallelism - at least so it appears to be when comparing it to other imperative Java, C or Python. Another popular point is how the language, as compared to C-ish languages doesn’t give you too much freedom when it comes to formatting – there isn’t a constant drive to use as few characters as possible (something I’m very prone to doing), or any debates like tabs vs. spaces, where to place the opening braces, etc. There’s really something reliving about this to me, that makes the language, as you put it, “pleasant” to use (even if you might not agree with it)
And regarding the standard library, one thing I always find interesting is how far you can get by just using what’s already packaged in Go itself. Now I haven’t really worked on anything with more that 1500+ LOC (which really isn’t much for Go), and most of the external packages I used were for the sake of convince. Maybe this totally changes when you work in big teams or on big projects, but it is something I could understand people liking. Especially considered that the Go team has this Go 1.x compatibility promise, so that you don’t have to worry that much about versioning when it comes to the standard lib packages.
I guess the worst mistake one can make is wanting to treat it like Haskell or Python, forcing a different padigram onto it. Just like one might miss macros when one changes from C to Java, or currying when one switches from Haskell to Python, but learns to accept these things, and think differently, so I belive, one should approach Go, using it’s strengths, which it has, instead of lamenting it’s weaknesses (which undoubtedly exist too).
I think their driving philosophy is that if you’re uncertain of something, always make the simpler choice. You sometimes go to wrong paths following this, but I’d say that in general this is a winning strategy. Complexity can always be bolted on later, but removing it is much more difficult.
The whole IT industry would be a happier place if it followed this, but seems to me that we usually do the exact opposite.
I think their driving philosophy is that if you’re uncertain of something, always make the simpler choice.
Nah - versioning & dependency management is not some new thing they couldn’t possibly understand until they waited 8 years. Same with generics.
Where generics I can understand a complexity argument for sure, versioning and dependency management are complexities everyone needed to deal with either way.
If you understand the complexity argument for generics, then I think you could accept it for dependency management too. For example, Python, Ruby and JavaScript have a chaotic history in terms of the solution they adopted for dependency management, and even nowadays, the ecosystem it not fully stabilized. For example, in the JavaScript community, Facebook released yarn in October 2016, because the existing tooling was not adequate, and more and more developers are adopting it since then. I would not say that dependency management is a fully solved problem.
I would not say that dependency management is a fully solved problem.
Yes it is, the answer is pinning all dependencies, including transitive dependencies. All this other stuff is just heuristics that end up failing later on and people end up pinning anyways.
I agree about pinning. By the way, this is what vgo does. But what about the resolution algorithm used to add/upgrade/downgrade dependencies? Pinning doesn’t help with this. This is what makes Minimal Version Selection, the strategy adopted by vgo, original and interesting.
I’m not sure I understand what the selection algorithm is doing then. From my experience: you change the pin, run your tests, if it passes, you’re good, if not, you fix code or decide not to change the version. What is MVS doing for this process?
When you upgrade a dependency that has transitive dependencies, then changing the pin of the upgraded dependency is not enough. Quite often, you also have to update the pin of the transitive dependencies, which can have an impact on the whole program. When your project is large, it can be difficult to do manually. The Minimal Version Selection algorithm offers a new solution to this problem. The algorithm selects the oldest allowed version, which eliminates the redundancy of having two different files (manifest and lock) that both specify which modules versions to use.
Unless it wasn’t clear in my original comment, when I say pin dependencies I am referring to pinning all dependencies, including transitive dependencies. So is MVS applied during build or is it a curation tool to help discover the correct pin?
I’m not sure I understand your question. MVS is an algorithm that selects a version for each dependency in a project, according to a given set of constraints. The vgo tool runs the MVS algorithm before a build, when a dependency has been added/upgraded/downgraded/removed. If you have the time, I suggest you read Russ Cox article because it’s difficult to summarize in a comment ;-)
I am saying that with pinned dependencies, no algorithm is needed during build time, as there is nothing to compute for every dependency version is known apriori.
I had a similar experience with Elm. In my case, it seemed like some people weren’t in the habit of questioning the language or thinking critically about their experience. For example, debugging in Elm is very limited. Some people I worked with came to like the language less for this reason. Others simply discounted their need for better debugging. I guess this made the reality easier to accept. It seemed easiest for people whose identities were tied to the language, who identified as elm programmers or elm community members. Denying personal needs was an act of loyalty.
How they managed to not solve many hard problems of a language, it’s tooling or production workflow, but also solve a set to get a huge amount of developer mindshare is something I think we should get historians to look into.
Maybe a dumb questions, but in semver what is the point of the third digit? A change is either backwards compatible, or it is not. To me that means only the first two digits do anything useful? What am I missing?
It seems like the openbsd libc is versioned as major.minor for the same reason.
I still don’t understand what the purpose of the PATCH version is? If minor versions are backwards compatible, what is the point of adding a third version number?
They want a difference between new functionality (that doesn’t break anything) and a bug fix.
I.e. if it was only X.Y, then when you add a new function, but don’t break anything.. do you change Y or do you change X? If you change X, then you are saying I broke stuff, so clearly changing X for a new feature is a bad idea. So you change Y, but if you look at just the Y change, you don’t know if it was a bug-fix, or if it was some new function/feature they added. You have to go read the changelog/release notes, etc. to find out.
with the 3 levels, you know if a new feature was added or if it was only a bug fix.
Clearly just X.Y is enough. But the semver people clearly wanted that differentiation, they wanted to be able to , by looking only at the version #, know if there was a new feature added or not.
Imagine you have authored a library, and have released two versions of it, 1.2.0 and 1.3.0. You find out there’s a security vulnerability. What do you do?
You could release 1.4.0 to fix it. But, maybe you haven’t finished what you planned to be in 1.4.0 yet. Maybe that’s acceptable, maybe not.
Some users using 1.2.0 may want the security fix, but also do not want to upgrade to 1.3.0 yet for various reasons. Maybe they only upgrade so often. Maybe they have another library that requires 1.2.0 explicitly, through poor constraints or for some other reason.
In this scenario, releasing a 1.2.1 and a 1.3.1, containing the fixes for each release, is an option.
It sort of makes sense but if minor versions were truly backwards compatible I can’t see a reason why you would ever want to hold back. Minor and patch seem to me to be the concept just one has a higher risk level.
Without the patch version it makes it much harder to plan future versions and the features included in those versions. For example, if I define a milestone saying that 1.4.0 will have new feature X, but I have to put a bug fix release out for 1.3.0, it makes more sense that the bug fix is 1.3.1 rather than 1.4.0 so I can continue to refer to the planned version as 1.4.0 and don’t have to change everything which refers to that version.
I remember seeing a talk by Rich Hickey where he criticized the use of semantic versioning as fundamentally flawed. I don’t remember his exact arguments, but have sem ver proponents grappled effectively with them? Should the Go team be wary of adopting sem ver? Have they considered alternatives?
I didn’t watch the talk yet, but my understanding of his argument was “never break backwards compatibility.” This is basically the same as new major versions, but instead requiring you to give a new name for a new major version. I don’t inherently disagree, but it doesn’t really seem like some grand deathblow to the idea of semver to me.
IME, semver itself is fundamentally flawed because humans are the deciders of the new version number and we are bad at it. I don’t know how many times I’ve gotten into a discussion with someone where they didn’t want to increase the major because they thought high major’s looked bad. Maybe at some point it can be automated, but I’ve had plenty of minor version updates that were not backwards compatible, same for patch versions. Or, what’s happened to me in Rust multiple times, is the minor version of a package incremented but the new feature depends on a newer version of the compiler, so it is backwards breaking in terms of compiling. I like the idea of a versioning scheme that lets you tell the chronology of versions but I’ve found semver to work right up until it doesn’t and it’s always a pain. I advocate pinning all deps in a project.
It’s impossible for computers to automate. For one, semver doesn’t define what “breaking” means. For two, the only way that a computer could fully understand if something is breaking or not would be to encode all behavior in the type system. Most languages aren’t equipped to do that.
Elm has tools to do at least a minimal kind of check here. Rust has one too, though not widely as used.
. I advocate pinning all deps in a project.
That’s what lockfiles give you, without the downsides of doing it manually.
Having one file to specify dependencies and one file to lock them is quite flexible. You can choose to commit the lock file or not. The proposed approach removes this flexibility and the choice. Similar results could be obtained with dep by using a different solver (sticking to minimal versions) to generate the lock file.
In Go, the dependencies are already specified in the source code, so we don’t need a file for this. The go.mod file is used to specify dependency versions. The go.mod file can act as as a lock file because of the Minimal Version Selection algorithm.
Personally I trust Russ Cox’s judgement… though I could see how people who worked on ‘dep’ would be furious. Go has a reputation for taking community direction with a grain of salt. The go team certainly is not afraid to do unpopular things in the goal of simplicity.
I am reminded of this comment from Russ, which I think explains rather a lot: https://news.ycombinator.com/item?id=4535977
This is the direction I’ve been rooting for.
vgo is certainly different than dep, and in some ways it’s simpler, but in other ways it pushes a lot more complexity on the user. I think on balance it’s got to be a wash, at least for now.
What are the complexities pushed onto the users?
The Go project is absolutely fascinating to me.
How they managed to not solve many hard problems of a language, it’s tooling or production workflow, but also solve a set to get a huge amount of developer mindshare is something I think we should get historians to look into.
I used Go professionally for ~2+ years, and so much of it was frustrating to me, but large swaths of our team found it largely pleasant.
I’d guess there is a factor depending on what you want from a language. Sure, it doesn’t have generics and it’s versioning system leaves a lot to be wished for. But personally, if I have to write anything with networking and concurrency, usualy my first choice is Go, because of it’s very nice standard library and a certain sense of being thought-thorugh when it comes to concurrency/parallelism - at least so it appears to be when comparing it to other imperative Java, C or Python. Another popular point is how the language, as compared to C-ish languages doesn’t give you too much freedom when it comes to formatting – there isn’t a constant drive to use as few characters as possible (something I’m very prone to doing), or any debates like tabs vs. spaces, where to place the opening braces, etc. There’s really something reliving about this to me, that makes the language, as you put it, “pleasant” to use (even if you might not agree with it)
And regarding the standard library, one thing I always find interesting is how far you can get by just using what’s already packaged in Go itself. Now I haven’t really worked on anything with more that 1500+ LOC (which really isn’t much for Go), and most of the external packages I used were for the sake of convince. Maybe this totally changes when you work in big teams or on big projects, but it is something I could understand people liking. Especially considered that the Go team has this Go 1.x compatibility promise, so that you don’t have to worry that much about versioning when it comes to the standard lib packages.
I guess the worst mistake one can make is wanting to treat it like Haskell or Python, forcing a different padigram onto it. Just like one might miss macros when one changes from C to Java, or currying when one switches from Haskell to Python, but learns to accept these things, and think differently, so I belive, one should approach Go, using it’s strengths, which it has, instead of lamenting it’s weaknesses (which undoubtedly exist too).
I think their driving philosophy is that if you’re uncertain of something, always make the simpler choice. You sometimes go to wrong paths following this, but I’d say that in general this is a winning strategy. Complexity can always be bolted on later, but removing it is much more difficult.
The whole IT industry would be a happier place if it followed this, but seems to me that we usually do the exact opposite.
Nah - versioning & dependency management is not some new thing they couldn’t possibly understand until they waited 8 years. Same with generics.
Where generics I can understand a complexity argument for sure, versioning and dependency management are complexities everyone needed to deal with either way.
If you understand the complexity argument for generics, then I think you could accept it for dependency management too. For example, Python, Ruby and JavaScript have a chaotic history in terms of the solution they adopted for dependency management, and even nowadays, the ecosystem it not fully stabilized. For example, in the JavaScript community, Facebook released yarn in October 2016, because the existing tooling was not adequate, and more and more developers are adopting it since then. I would not say that dependency management is a fully solved problem.
Yes it is, the answer is pinning all dependencies, including transitive dependencies. All this other stuff is just heuristics that end up failing later on and people end up pinning anyways.
I agree about pinning. By the way, this is what vgo does. But what about the resolution algorithm used to add/upgrade/downgrade dependencies? Pinning doesn’t help with this. This is what makes Minimal Version Selection, the strategy adopted by vgo, original and interesting.
I’m not sure I understand what the selection algorithm is doing then. From my experience: you change the pin, run your tests, if it passes, you’re good, if not, you fix code or decide not to change the version. What is MVS doing for this process?
When you upgrade a dependency that has transitive dependencies, then changing the pin of the upgraded dependency is not enough. Quite often, you also have to update the pin of the transitive dependencies, which can have an impact on the whole program. When your project is large, it can be difficult to do manually. The Minimal Version Selection algorithm offers a new solution to this problem. The algorithm selects the oldest allowed version, which eliminates the redundancy of having two different files (manifest and lock) that both specify which modules versions to use.
Unless it wasn’t clear in my original comment, when I say pin dependencies I am referring to pinning all dependencies, including transitive dependencies. So is MVS applied during build or is it a curation tool to help discover the correct pin?
I’m not sure I understand your question. MVS is an algorithm that selects a version for each dependency in a project, according to a given set of constraints. The
vgotool runs the MVS algorithm before a build, when a dependency has been added/upgraded/downgraded/removed. If you have the time, I suggest you read Russ Cox article because it’s difficult to summarize in a comment ;-)I am saying that with pinned dependencies, no algorithm is needed during build time, as there is nothing to compute for every dependency version is known apriori.
I agree with this.
I had a similar experience with Elm. In my case, it seemed like some people weren’t in the habit of questioning the language or thinking critically about their experience. For example, debugging in Elm is very limited. Some people I worked with came to like the language less for this reason. Others simply discounted their need for better debugging. I guess this made the reality easier to accept. It seemed easiest for people whose identities were tied to the language, who identified as elm programmers or elm community members. Denying personal needs was an act of loyalty.
I think you’ll find they already have!
Maybe a dumb questions, but in semver what is the point of the third digit? A change is either backwards compatible, or it is not. To me that means only the first two digits do anything useful? What am I missing?
It seems like the openbsd libc is versioned as major.minor for the same reason.
Minor version is backwards compatible. Patch level is both forwards and backwards compatible.
Thanks! I somehow didn’t know this for years until I wrote a blog post airing my ignorance.
PATCH version when you make backwards-compatible bug fixes See: https://semver.org
I still don’t understand what the purpose of the PATCH version is? If minor versions are backwards compatible, what is the point of adding a third version number?
They want a difference between new functionality (that doesn’t break anything) and a bug fix.
I.e. if it was only X.Y, then when you add a new function, but don’t break anything.. do you change Y or do you change X? If you change X, then you are saying I broke stuff, so clearly changing X for a new feature is a bad idea. So you change Y, but if you look at just the Y change, you don’t know if it was a bug-fix, or if it was some new function/feature they added. You have to go read the changelog/release notes, etc. to find out.
with the 3 levels, you know if a new feature was added or if it was only a bug fix.
Clearly just X.Y is enough. But the semver people clearly wanted that differentiation, they wanted to be able to , by looking only at the version #, know if there was a new feature added or not.
To show that there was any change at all.
Imagine you don’t use sha1’s or git, this would show that there was a new release.
But why can’t you just increment the minor version in that case? a bug fix is also backwards compatible.
Imagine you have authored a library, and have released two versions of it,
1.2.0and1.3.0. You find out there’s a security vulnerability. What do you do?You could release
1.4.0to fix it. But, maybe you haven’t finished what you planned to be in 1.4.0 yet. Maybe that’s acceptable, maybe not.Some users using
1.2.0may want the security fix, but also do not want to upgrade to1.3.0yet for various reasons. Maybe they only upgrade so often. Maybe they have another library that requires1.2.0explicitly, through poor constraints or for some other reason.In this scenario, releasing a
1.2.1and a1.3.1, containing the fixes for each release, is an option.It sort of makes sense but if minor versions were truly backwards compatible I can’t see a reason why you would ever want to hold back. Minor and patch seem to me to be the concept just one has a higher risk level.
Perhaps a better definition is library minor version changes may expose functionality to end users you did not intend as an application author.
I think it’s exactly a risk management decision. More change means more risk, even if it was intended to be benign.
Without the patch version it makes it much harder to plan future versions and the features included in those versions. For example, if I define a milestone saying that
1.4.0will have new feature X, but I have to put a bug fix release out for1.3.0, it makes more sense that the bug fix is1.3.1rather than1.4.0so I can continue to refer to the planned version as1.4.0and don’t have to change everything which refers to that version.I remember seeing a talk by Rich Hickey where he criticized the use of semantic versioning as fundamentally flawed. I don’t remember his exact arguments, but have sem ver proponents grappled effectively with them? Should the Go team be wary of adopting sem ver? Have they considered alternatives?
I didn’t watch the talk yet, but my understanding of his argument was “never break backwards compatibility.” This is basically the same as new major versions, but instead requiring you to give a new name for a new major version. I don’t inherently disagree, but it doesn’t really seem like some grand deathblow to the idea of semver to me.
IME, semver itself is fundamentally flawed because humans are the deciders of the new version number and we are bad at it. I don’t know how many times I’ve gotten into a discussion with someone where they didn’t want to increase the major because they thought high major’s looked bad. Maybe at some point it can be automated, but I’ve had plenty of minor version updates that were not backwards compatible, same for patch versions. Or, what’s happened to me in Rust multiple times, is the minor version of a package incremented but the new feature depends on a newer version of the compiler, so it is backwards breaking in terms of compiling. I like the idea of a versioning scheme that lets you tell the chronology of versions but I’ve found semver to work right up until it doesn’t and it’s always a pain. I advocate pinning all deps in a project.
It’s impossible for computers to automate. For one, semver doesn’t define what “breaking” means. For two, the only way that a computer could fully understand if something is breaking or not would be to encode all behavior in the type system. Most languages aren’t equipped to do that.
Elm has tools to do at least a minimal kind of check here. Rust has one too, though not widely as used.
That’s what lockfiles give you, without the downsides of doing it manually.
Having one file to specify dependencies and one file to lock them is quite flexible. You can choose to commit the lock file or not. The proposed approach removes this flexibility and the choice. Similar results could be obtained with dep by using a different solver (sticking to minimal versions) to generate the lock file.
In Go, the dependencies are already specified in the source code, so we don’t need a file for this. The go.mod file is used to specify dependency versions. The go.mod file can act as as a lock file because of the Minimal Version Selection algorithm.