1.

Man, there’s so many people that don’t read anything.

I currently fill the role of IT and it’s absolutely bonkers the stuff people get stuck on. It’s so basic, and the screen is telling you what’s wrong, but people aren’t even parsing it.

1.

The irony here is that the article claims that people actually do spend a significant amount of their time reading compiler errors.

1.

The bonus irony is that it’s discussing a paper that, guess what, no one will read! Because it links to a paywalled version of it instead of the author’s copy available freely on their web site: https://people.engr.ncsu.edu/ermurph3/papers/icse17.pdf

1. 10

Minimal Version Selection itself is fine, in my experience neither better nor worse than what I guess is the de facto standard of… Maximal {Patch, Major} Version Selection? It biases for churn reduction during upgrades, maybe, which at least for me doesn’t have much net impact on anything.

But a lot of the ancillary behaviors and decisions that come with MVS as expressed in Go modules are enormously frustrating — I would go so far as to say fundamentally incompatible with the vast majority of dependency management workflows used by real human beings. As an example, in modules, it is effectively not possible to constrain the minor or patch versions of a dependency. You express module v1.2.3 but if another dependency in your dep graph wants module v1.4.4 you’ll get v1.4.4. There is no way to express v1.2.x or v1.2.3+. You can abuse the require directive to lock to a single specific version, but that constraint isn’t transitive. And module authors can retract specific versions, but that power is not granted to consumers.

edit: for more of my thoughts on this exciting topic see Semantic Import Versioning is unsound

1. 5

I would go so far as to say fundamentally incompatible with the vast majority of dependency management workflows used by real human beings.

I hope that this post with the experience reports of many real human beings claiming that it works well for them will help you reconsider this opinion. Perhaps it’s not as much of a vast majority as you think it is.

1. 3

None of the experience reports really speak to the points I raise. Maybe that’s your point.

2. 2

The position of the Go team is (was?) that packages with the same import path must be backwards compatible. I guess their point is that 1.4.4 should be compatible with 1.2.3, but that’s not how the rest of the software world has worked in the past decade. It’s a big if, but if all the go programmers agree, it works.

1. 6

That’s not (only?) the position of the Go team, it’s a requirement of semantic versioning. People fuck it up all the time but that’s the definition. One way to look at the problem with modules is that they assume nobody will fuck it up. Modules assumes all software authors treat major version API compatibility as a sacrosanct and inviolable property, and therefore provides no affordances to help software consumers deal the fuckups that inevitably occur. If one of your dependencies broke an API in a patch version bump, well, use a better different dependency, obviously!

Ivory Tower stuff, more or less. And in many ways Go is absolutely an Ivory Tower language! It tells you the right way to do things and if you don’t agree then too bad. But the difference is that if you don’t like the “I know better than you” decisions that Go the language made, you can pick another language. But if you don’t like the “I know better than you” decisions that modules made, you can’t pick another package management tool. Modules has a monopoly on the space. That means it doesn’t have the right to make the same kind of normative assumptions and assertions that the language itself makes. It has to meet users where they are. But the authors don’t understand this.

1. 3

Semantic versioning is, unfortunately, incompatible with graceful deprecation. Consider the following example:

• A is the version you’re using.
• B introduces a new API and deprecates an old one
• C removes the deprecated API.

In SemVer, these thing would be numbered something like 1.0, 1.1, 2.0. The jump from 1.1 to 2.0 is a breaking change because the old API went away at that point. If you paid attention to deprecation warnings when you upgraded to 1.1 and fixed them then the 1.1 -> 2.0 transition is not a breaking change though and SemVer has no way of expressing this with a single exported version and this leads to complex constraints on the import version (in this three-version case, the requirement is >=1.1, <3.0). A lot of these things would be simpler if APIs, not packages, were versioned and the package advertised the range of the API that it advertised. Then you’d see:

• A is the 1.0 API
• B supports the 1.0 API and the 2.0 API
• C supports the 2.0 API

As a consumer, I’d just specify that I need the 1.0 API until I’ve migrated my code and then that I need the 2.0 API.

1. 1

One way to look at the problem with modules is that they assume nobody will fuck it up.

So does everyone else in the world who builds with ^version and no lock file, except for them, the breakage happened when the author of their dependency published a new version rather than when they themselves performed a conscious action to alter dependencies in some way.

1. 3

Yeah but in most package management systems you can pin specific versions.

2. 1

[…] and therefore provides no affordances to help software consumers […]

If one of your dependencies broke an API in a patch version bump, well, use a better different dependency, obviously!

You can use an exclude directive to remove the breaking version from consideration. If upstream delays fixing the breaking change, you can fork the last good version and use a replace directive to substitute your own patched module in place of the old module.

It’s hard to imagine this hypothetical happening, however (and please correct me if I’m wrong). MVS selects the maximum of minimum required versions. If some module called “broken” patches and the new version breaks your own software, there is no way for that update to propagate to your software unless both 1) a different dependency of your module called “dep” decides to start requiring the new version of “broken” and 2) you update your dependencies to require the updated version of “dep”. (1) Implies that “dep” truly requires the patch (that broke your code), and (2) implies that you truly require the new features in module “dep”. By transitivity… there is no conceivable way to fix the problem without patching it yourself and replacing.

There’s actually a whole paragraph on this topic of “high fidelity” in Russ Cox’s original blog post about the version selection algorithm.

1. 2

You can use an exclude directive to remove the breaking version from consideration.

I meant “broke an API” in the “API compatibility” sense, not in the “wrote a bug” sense. That kind of broken carries forward.

2. 1

So your article states: “It’s a contract with its consumers, understood by default to be supported and maintained indefinitely.”

I don’t think this follows from anything you have written or anything I have read about SIV. The way SIV works sounds to me like if you want to deprecate features from your library you should provide a reasonable deprecation policy which includes a time period for which you will provide bug-fixes for the old major version and a time period for which you will backport security-fixes for the old major version at which point you stop supporting that version since you’ve done the best you could to get old users moved to a new version. This to me seems like a lot of major software (not written in the past 5 years) basically works.

“At Google, package consumers expect their dependencies to be automatically updated with e.g. security fixes, or updated to work with new e.g. infrastructural requirements, without their active intervention.”

I expect this to happen on my linux desktop too. I don’t see a difference in expectations there.

“Stability is so highly valued, in fact, that package authors are expected to proactively audit and PR their consumers’ code to prevent any observable change in behavior.”

I think if you feel like writing a library/module/dependency then this is the kind of mindset you are obliged to take. Anything short of this kind of approach to writing a library/module/dependency is irresponsible and makes you unqualified to write libraries, modules or dependencies. This, to me, seems to have been the mindset for a long time in software until languages came along with language package managers and version pinning in the last few years. And I don’t think that this has been a positive change for anyone involved.

“As I understand it, even issues caused by a dependency upgrade are considered the fault of the dependency, for inadequate risk analysis, rather than the fault of the consumer, for inadequate testing before upgrading to a new version.”

And I agree with this wholeheartedly, in fact this is the mindset used by users of linux distributions and distribution maintainers.

“Modules’ extraordinary bias toward consumer stability may be ideal for the software ecosystem within Google, but it’s inapproriate for software ecosystems in general.”

I think it’s not inappropriate, it’s totally appropriate. I just think that modern software ecosystems have gotten lazy because it’s easier than doing it right (which is what google seems to be advocating for a return to).

I should say, I don’t disagree with the point you make that intrinsically linking the major version to the package name is a good idea. Go should definitely NOT do that for the reasons you outlined. It would also be an easy indicator for me when picking a project to use in my codebase: Is the codebase on major version 156? Yes? Then I probably don’t want to touch it because the developers are not taking the responsibility of maintaining a dependency very seriously.

People who want to play in the sandpit of version pinning and ridiculously high major version numbers because they think software development is an area where no thought or effort should be put into backwards compatibility should be welcome to use whatever language they want to without being artificially limited.

Now, conversely, I would say, there seems like an obvious solution to this problem too. If you want to use semver while keeping to the golang rules, why not just encode the real semver version within the go version: “0.015600310001”. Sure, it’s not exactly so human readable, but it seems to encode the right information and you just need to pretty print it.

“Additionally, policies that bias for consumer stability rely on a set of structural assumptions that may exist in a closed system like Google, but simply don’t exist in an open software ecosystem in general.”

I will take things back to the world of linux distributions where these policies actually do seem to exist.

“A bias towards consumers necessarily implies some kind of bias against authors.”

Yes, and this is a good thing. Being an author of a dependency is a very big responsibility and a lot of modern build systems and language package managers fail to make that very clear

“But API compatibility isn’t and can’t be precisely defined, nor can it even be discovered, in the P=NP sense.”

This is true, but in reality there’s a pretty big gulf between best effort approaches to API compatibility (see: linux kernel) and zero effort approaches to API compatibility (see: a lot of modern projects in modern languages).

“Consequently, SIV’s model of versioning is precisely backwards.”

Actually it would be semver’s fault not SIV’s surely.

“Finally, this bias simply doesn’t reflect the reality of software development in the large. Package authors increment major versions as necessary, consumers update their version pins accordingly, and everyone has an intuitive understanding of the implications, their risk, and how to manage that risk. The notion that substantial version upgrades should be trivial or even automated by tooling is unheard of.”

Maybe today this is the case, but I am pretty sure this is only a recent development. Google isn’t asking you to do something new, google is asking you to do something old.

“Modules and SIV represent a normative argument: that, at least to some degree, we’re all doing it wrong, and that we should change our behavior.”

You’re all doing it wrong and you should change your behavior.

“The only explicit benefit to users is that they can have different versions of the “same” module in their compilation unit.”

You can achieve this without SIV, SIV to me actually seems like just a neat hack to avoid having to achieve this without SIV.

In any case, I think I’ve made my point mostly and at this point I would be repeating myself.

I wonder what you think.

1. 1

People who want to play in the sandpit of version pinning and ridiculously high major version numbers because they think software development…

…is a means and not an end are the norm, not the exception. And the fact that people work this way is absolutely not because they’re lazy, it’s because it’s the rational choice given their conditions, the things they’re (correctly!) trying to optimize for, and the (correct!) risk analysis they’ve done on all the variables at play.

I appreciate your stance but it reflects an Ivory Tower approach to software development workflows (forgive the term) which is generally both infeasible and actually incorrect in the world of market-driven organizations. That’s the context I speak from and the unambiguous position I’ve come to after close to 20 years’ experience in the space, working myself in a wide spectrum of companies and consulting in exactly these topics for ~100 orgs at this point.

Google has to work this way because their codebase is so pathological they have no other choice. Many small orgs, or orgs decoupled from typical market dynamics, can work this way because they have the wiggle room, so to speak. They are the exceptions.

1. 1

the things they’re (correctly!) trying to optimize for, and the (correct!) risk analysis they’ve done on all the variables at play

Disagree.

At least don’t call the majority of software developers “engineers” if you’re going to go this way.

The fact that this is considered an engineering discipline with such low standards is really an insult to actual engineering disciplines. I can totally see how certain things don’t need to be that rigorous, but really, seriously, what is happening is not par for the course.

The fact that everyone including end users has become used to the pathological complacency of modern software development is really seriously ridiculous and not an excuse to continue down this path. I would go so far to say that it’s basically unethical to keep pretending like nothing matters more than making something which only just barely works within some not even that optimal constraints for the least amount of money. It’s a race to the bottom, and it won’t end well. It’s certainly not sustainable.

I appreciate your stance but it reflects an Ivory Tower approach to software development workflows (forgive the term) which is generally both infeasible and actually incorrect in the world of market-driven organizations. That’s the context I speak from and the unambiguous position I’ve come to after close to 20 years’ experience in the space, working myself in a wide spectrum of companies and consulting in exactly these topics for ~100 orgs at this point.

It’s incorrect in the world of market-driver organizations only because there’s a massive gap between the technical ability of the consumers of these technologies and the producers, so much so that it’s infeasible to expect a consumer of these technologies to be able to see them for the trash they are. But I think that this is not “correct” it’s just “exploitative”. Exploitative of the lack of technical skill and understanding of the average consumer of these technologies.

I don’t think the “correct” response is “do it because everyone else is”. It certainly seems unethical to me.

That being said, you are talking about this from a business point of view not an open source point of view. At least until open source got hijacked by big companies, it used to be about small scale development by dedicated developers interested in making things good for the sake of being good and not for the sake of a bottom line. This is why for the most part my linux system can “just update” and continue working, because dedicated volunteers ensure it works.

Certainly I don’t expect companies to care about this kind of thing. But if you’re talking about solely the core open source world, “it’s infeasible in this market” isn’t really an argument.

Google has to work this way because their codebase is so pathological they have no other choice. Many small orgs, or orgs decoupled from typical market dynamics, can work this way because they have the wiggle room, so to speak. They are the exceptions.

I honestly don’t like or care about google or know how they work internally. I also don’t like go’s absolutely inane idea that it’s sensible to “depend” on a github hosted module and download it during a build. There’s lots of things wrong with google and go but I think that this versioning approach has been a breath of fresh air which suggests that maybe just maybe things may be changing for the better. I would never have imagined google (one of the worst companies for racing to the bottom) to be the company to propose this idea but it’s not a bad idea.

1. 2

The fact that everyone including end users has become used to the pathological complacency of modern software development is…

…reality. I appreciate your stance, but it’s unrealistic.

1. 26

This post talks about the downsides but does not acknowledge the underlying problem that is being corrected, namely that “go get” today does two completely different things: (1) add or update dependencies of a program, and (2) install binaries.

Nearly all the time, you only mean to do one of these things, not both. If you get in the habit of using “go get” to update your dependencies, you might run a command like “go get path/…” and be surprised when it installs “path/internal/ls” to your $HOME/bin directory. We have had people claim that’s a security issue, and whether that’s true or not, it’s certainly a usability issue. In this example, you only want the dependency change and are getting the install as an unwanted side-effect. After the transition, the rule will be simple: “go get only changes your dependency versions”. On the other hand, we have commands in READMEs that say things like “go get -u rsc.io/2fa”. Today that means to fetch the latest version of rsc.io/2fa, then upgrade all its dependencies, recursively, producing a build configuration that may never have been tested, and then build the binary and install it. In this example, you only want the install and are getting the “update dependencies” as an unwanted side effect. If instead you use “go install rsc.io/2fa@latest” (which works today in Go 1.16), you get the latest tagged version of rsc.io/2fa, using exactly the dependency versions it declares and was tested with, which is what you really want when you are trying to install a binary. We didn’t just make a gratuitous change to the way the command is spelled: we fixed the semantics too. The double meaning of “go get” was introduced in GOPATH mode many years ago. It is my fault, and all I can say is that it seemed like a good idea at the time, especially in the essentially unversioned context of GOPATH-based development. But it’s a mistake, and it is a big enough mistake to correct for the next million users of Go. The post claims this is an abrupt deprecation schedule, but the change was announced in the Go 1.16 release notes and will take effect in Go 1.18. Once Go 1.17 is released, the two supported versions of Go will be Go 1.16 and Go 1.17, meaning all supported Go versions will know what “go install path@version” means, making it time to start nudging people over to using it with prints from the go command. This two-cycle deprecation schedule, where we make sure that we don’t encourage transition work until all supported versions of Go (the last two) support the new behavior, is our standard procedure for this. I suspect that Go 1.18 will keep the warning print, so you’d actually get two releases of warning prints as well. We try very hard not to make breaking changes in tool behavior. This one is necessary, and we’ve done everything we can to make sure it is a smooth transition. The new behavior should be obvious - the tool tells you what is going on and what you need to do differently - and easy to learn. 1. 2 I know it’s too late to change anything now, but in retrospect, would it have been better to leave go get in the old GOPATH world and introduce the module version of go get under a different name? 1. 1 A quick note: “go get” does three things today, not two. The third is that it clones the version control repository for some Go package or module (into$GOPATH/src). In the process of doing this, it reaches through vanity import paths and works out the correct VCS system and clone path to use. As far as I know, in the non-GOPATH world there is currently no way to do this yourself, either for programs or for other packages. It would be nice at the very least to have a command that resolved through vanity import paths to tell you the underlying thing to clone, and with what.

(Cloning this way also locks you to the current location of the repository. One advantage of vanity import paths is that people can change them, and have, and you will automatically pick up that change.)

The other thing that is not possible in the non-GOPATH world is to install the very latest VCS version of a program (at least in any convenient way). As noted, the @latest syntax is the latest tagged version, not the latest VCS version. This means that if you want to have someone test your very latest development version, you need to either tag it or tell them to clone the repo and build it themselves; you can’t give them a simple, generic ‘go install …@…’ command to do it. (You can give them a VCS version, but there are various problems with this.)

1. 1

The other thing that is not possible in the non-GOPATH world is to install the very latest VCS version of a program (at least in any convenient way).

You can use “go install …@branchname” to get the latest commit for whatever branch.

1. 1

Oh oops, that’s my mistake. I missed that in the Version queries section of the specification.

(If reaching for the specification strikes people as odd, I can only plead that I couldn’t readily find documentation of this in the “go” help and “go help modules” did point me to the specification.)

1. 9

Does it mean that the promise to not break API doesn’t extend to CLI? https://golang.org/doc/go1compat#expectations

Keeping things stable is what makes the success of Go.

1. 7

Indeed, the final section of that page you linked says that tools are exempt. I can’t think of any previous examples that broke the tooling as much as this will, though.

1. 6

It seems short-sighted.

go get is invoked by a ton of other programs. For example, VSCode plugins love to run it to install missing developer tools. It’s effectively an API, but that’s going through the CLI instead of accessing a Go API directly.

1. 8

There’s been loads of subtle breakage and changes in the Go tooling ever since modules were introduced. It rather sucks, but it’s better than the alternative of keeping it full of all sorts of legacy behaviour where commands do something different depending on the environment and/or which directory you’re in, which is super-confusing.

It’s easy to say it’s “short-sighted”, but all options suck – they just suck in different ways.

2. 0

I’ve been listening to the gotime podcast. Go get is a feature to people new to programming – not a bug.

1. 6

I wish this included some explanation of why these are useful or explanations of why they are. Like, why use zerolog over logrus/zap/whatever? Why use gorilla/mux over gin or go-chi or the standard library? Why have that overly-complex unique code thing to determine where a logged error is thrown instead of making sure your errors have stack traces?

1. 4

I’m not the author of the post, but I can try to answer some questions:

Like, why use zerolog over logrus/zap/whatever?

zerolog and zap is what the author of logrus admit he would write if he wrote logrus/v2. zerolog claims to be a “better” version of zap (there are claims of performance improvement on other people’s computers)

Why use gorilla/mux over gin or go-chi or the standard library?

gorilla/mux is much lightweight as opposed to gin which has renderers, etc… Also, there are claims of very high performance on other people’s computers. With gorlla/mux you can plug your own json de-/serialization library which is faster than gin’s.

Why have that overly-complex unique code thing to determine where a logged error is thrown instead of making sure your errors have stack traces?

¯\(ツ)/¯ No idea… The unique code approach feels very hacky and brittle to me.

1. 3

I would even suggest “httprouter” as a more lightweight alternative to gorilla/mux.

1. 2

Most “higher level” Go routers are probably based on it anyways.

2. 1

We use unique logging codes for our logs, and it makes it easier to locate a particular statement in either the logs, or the source code. Now, we generate the unique code by hand, but it’s not a horrendous issue for us (we have a separate document that records each logging statement).

3. 1

I’d be curious to know the best practices for annotating errors with stack traces. Any good library suggestions?

1. 3

github.com/pkg/errors has some nice helpers, as described in its documentation. The issue is that it requires changing your code to using errors.Errorf() instead of fmt.Errorf().

1. 3

Also, errors.Wrapf is very useful when handling errors from packages which don’t use errors.

1. 2

You can wrap errors using fmt.Errorf() and the %w verb, and then later deal with them using errors.Is() or errors.As() and friends.

1. 1

That doesn’t give you stack traces though, correct?

2. 2

YES. This is probably my biggest issue with supporting production Go code. Errors are still too hard to locate.

1. 1

I personally use this to manage all that. If you use the errors.Wrap() function to wrap errors, it’ll add stack traces. And if you want to make a new error, the errors.New() function will also add a stack track where it was called. Then when you’re logging the error make sure it’s being logged in a way that will not just print the message. Most loggers should have a way to do that (I know zerolog and logrus do).

1. 1

Shameless plug for my errors library: https://pkg.go.dev/github.com/zeebo/errs/v2

• captures stack traces, but only once even if wrapped multiple times
• plays nicely with standard library errors package with errors.Is and errors.As
• has a “tag” feature to let you associate and query tags with errors
• helper to keep track of groups of errors
1. 3

I am confused how the presented scheme is anything close to tracing. The first step is

The plaintext that is to be traced is submitted along with RF, NF and context.

But NF is a 256bit random nonce that no one other than the sender and recipient have access to. You may be able to guess a plaintext, but there’s no way you can guess that.

Additionally, it seems to me that if you have access to an oracle that can say if a given ciphertext is equal to some plaintext, you have broken ciphertext indistinguishability, a property that is very important to confidentiality (“Indistinguishability is an important property for maintaining the confidentiality of encrypted communications.”)

1. 1

There would be a step where the reveal of this nonce would be compelled, similarly to how message franking implements such a step in its current form. The idea is that you can just substitute the rationale for this step from “abuse reporting” to “message tracing”.

1. 2

How is compelling the reveal of the nonce any different from compelling the reveal of the plaintext? They’re stored next to each other and the only parties that have the nonce are the same parties that have the plaintext. The difference between “abuse reporting” and “message tracing” is which party is performing the action, and that makes all the difference.

1. 2

As far as I understand, the nonce serves to validate the initial HMAC, which serves as a pre-commitment to the authenticity of the message within its original context.

1. 8

This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

So we rewrote gRPC and migrated our live network. DRPC is a drop-in replacement that handles everything we needed from gRPC

and

It’s worth pointing out that DRPC is not the same protocol as gRPC, and DRPC clients cannot speak to gRPC servers and vice versa.

How it is a drop in replacement if both client and server needs to be changed

1. 15

This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

The wire format used is defined here.

Logically, a sequence of Packets are sent back and forth over the wire. Each Packet has a enumerated kind, a message id (to order messages within a stream), a stream id (to identify which stream), and a payload. To bound the payload size, Packets are split into Frames which are marshaled and sent.

A marshaled Frames has a single header byte, the varint encoded stream and message ids, a varint encoded length, and that many bytes of payload. The header byte contains the kind, if it is the last frame for a Packet, and a control bit reserved for future use (the implementation currently ignores any frame with that bit set).

Because there’s no multiplexing at this layer, the reader can assume the Frames come in contiguously with non-decreasing ids, limiting the amount of memory and buffer space required to a single Packet. The Frame writing code is as simple as appending some varints, and the reading code is about 20 lines, neither of which have any code paths that can panic.

How it is a drop in replacement if both client and server needs to be changed

There’s a drpcmigrate package that has some helpers to let you serve both DRPC and gRPC clients on the same port by having the DRPC clients send a header that does not collide with anything. You first migrate the servers to do both, and can then migrate the clients.

The drop in replacement part refers to the generated code being source code API compatible in the sense that you can register any gRPC server implementation with the DRPC libraries with no changes required. The semantics of the streams and stuff are designed to match.

Sorry if this was unclear. There’s multiple dimensions of compatibility and it’s hard to clearly talk about them.

1. 8

(not the one who asked). Thank you for the comment. Do you plan to document the protocol somewhere (other than code)? It seems to me a RPC protocol needs a healthy ecosystem of implementations in many languages to be viable long term :-)

(edit: I mean document it cleanly, with request/reply behavior, and talk about the various transports that are supported.)

1. 6

You are absolutely right that having good documentation is essential, and I’ve created a github issue to start to keep track of the most important bits I can think of off the top of my head. To be honest, I’ve always struggled a bit with documentation, but I’m trying to work on it. In fact, the current CI system will fail if any exported symbols do not have a doc string. :) Thanks for bringing this up.

2. 2

Thanks for the explanation! Are there any plans for writing Python client/server?

1. 4

I do hope to get more languages supported. I created an issue to add Python support. I don’t currently deal with any Python code, so help will probably be needed to make sure it “fits” correctly with existing codebases.

1. 1

I have started looking into the drpcwire code, I will wait for the documentation. Meanwhile, is there any place, like IRC, Discord where drpc/storj developers hang out? The Python issue is tagged help-wanted, so, can I be of any help?

1. 3

The worst part of gRPC is its crappy protobuf notation. This tool doesn’t address anything about that.

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

1. 24

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

Sockpuppeting on Lobsters is heavily frowned upon.

1. 13

promote … with a throwaway account.

That. Don’t use throwaway accounts and shill projects.

1. 4

Do you mean the service definitions being .proto files? If so, DRPC has a very modular design, and all of the functionality is implemented by referring to only the interfaces defined in the main module. Because of that, you can effectively manually write out the code generated glue, or generate it in some other way. Here’s an example of that: https://play.golang.org/p/JQcS2A9S8QX

1. 1

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

Which tool were you trying to promote?

1. 6

He got banned again so we may never know

1. 13

Who thinks it’s a good idea to tell everybody they’re a spammer?

1. 2

Growth hackers.

2. 4

1. 4

You can check the moderation log for timestamp 2021-04-17 12:40 -0500

1. 1

click through to their username, which includes a ban reason.

1. 8

I haven’t read rsc’s critique of the article yet, but while reading the tables I’m thinking, why are they comparing different CockroachDB versions? Surely, if the point was to find bloat over time due to changes in Go, one would want to look at the same program across Go versions?

1. 25

I took the time to do my best to compile an older and a newer version of the code with many different Go versions and here’s what I got:

        v1.0          v20.2.0
1.8     58,099,688    n/a
1.9     57,897,616    314,191,032
1.10    57,722,520    313,669,616
1.11    48,961,712    233,170,304
1.12    52,440,168    236,192,600
1.13    50,844,048    214,373,144
1.14    50,527,320    212,699,656
1.15    47,910,360    201,391,416
1.16    47,317,624    205,018,136


I couldn’t compile v20.2.0 on go1.8 because of the large use of type aliases that would take a very long time to remove. I already had to make some other smaller changes or backports of standard library code to have it compile, and type aliases crossed the line for me.

The overall binary size has decreased by 20% to 33% which means that all of this “dark code” is just their inability to understand their binary, and all of the growth is due to their code base.

I noticed in the process of getting these numbers that an awful lot of C/C++ compiling and javascript webpacking was going on. It turns out, there’s about 7M of javascript embedded in the Go binary, as well as these archives linked in:

librocksdb.a       69M
libcryptopp.a      28M
libprotobuf.a      22M
libgssapi_krb5.a   3.9M
ibroachccl.a       2.3M
libgeos.so         2.3M
libproj.a          2.2M
libsnappy.a        168K


I have no idea how much they make up of the resulting binary, but I’d be surprised if it was negligible.

1. 8

Thank you!

2. 2

Most likely the current version doesn’t build with old Go. Probably there exists an older version that builds across all Go versions, but then the numbers wouldn’t be as “impressive” :-P

1. 2

Yes this is the usual reason. When I benchmarked how Rust compilation speed changed over the years, it was tricky to find a library that compiled down to Rust 1.0. I ended up using an old version of library to parse URL.

1. 53

1. A minor issue, but the Go program has a data race. The i loop variable is captured in the spawned goroutines while it is mutated by the outer loop. Fixing it doesn’t change the results of the program. It’s also immediately caught when built with the -race flag. Rust being data race free by construction is a super power.

2. Changing the amount of spawned goroutines/threads to 50,000 has a noticeable difference between the two on my machine. The Go program still completes in a timely fashion with ~5x the rss and no negative impacts on the rest of my system:

real 11.25s
user 41.49s
sys  0.70s


but the Rust version immediately causes my system to come to a crawl. UIs stop responding, audio starts stuttering, and eventually it crashes:

thread 'main' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', src/libcore/result.rs:1189:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
Command exited with non-zero status 101
real 16.39s
user 23.76s
sys  98.50s


So while it may not be significantly lighter than threads as far as this one measure is concerned, there is definitely some resource that threads are taking up that goroutines are not. I checked on some production monitoring systems to see how many active goroutines they had, and they were sitting comfortably in the 20-30k area. I wonder if they were using threads how they would fare.

1. 8

Rust being data race free by construction is a super power.

Well, it also necessarily prohibits entire classes of useful and productive patterns… it was a design decision, with pros and cons, but certainly not strictly better.

1. 4

Strong plus +1. I wish, for application development, it was possible to remove lifetimes, borrow-checker and manual memory management (which are not that useful for this domain) and to keep only fearless concurrency (which is useful even (more so?) in high-level languages). Alas, it seems that Rust’s thread-safety banana is tightly attached to the rest of the jungle.

1. 7

FWIW. Ponylang has exactly this:

• no lifetimes, data is either known statically via simple escape analysis or traced at runtime
• no borrow checker, although it does have a stricter reference capability system
• no manual memory management: uses a deterministically invoked (excluding shared, send data) Mark-and-no-sweep GC
• has the equivalent Send and Sync traits in its reference capability system which provide the same static guarantees

As with everything though it has its own trade offs; Whether that be in its ref-cap system, lack of explicit control in the system, etc.

2. 4

To prevent that Rust error, I believe you need to change vm.max_map_count: https://github.com/rust-lang/rust/issues/78497#issuecomment-730055721

1. 4

That seems to me like the kind of thing Rust should do to some extent on its own!

2. 3

I’d be interested to see how many threads ended up scheduled with the Go example. My guess is not many al all: IIRC the compiler recognizes those sleeps as a safe place to yield control from the goroutines. So I suspect you end up packing them densely on to threads.

1. 25

I think threads are better than a lot of people think, but in this case the measurement strikes me as a bit naive. Go very likely consumes more memory than strictly needed because of the GC overhead. The actual memory occupied by goroutines could be less than half the memory used in total, but here we only see goroutine overhead + GC overhead without a way of separating them.

1. 8

What overhead specifically? Stack maps for GC?

The benchmark doesn’t seem to have many heap objects, and I thought Go used a conservative/imprecise GC anyway so objects don’t have GC space overhead (mark bits, etc.). Although I think the GC changed many times and I haven’t followed all the changes.

1. 7

The GC has been fully precise since Go 1.4.

1. 6

OK that’s what I suspected, but as far as I can tell it’s a moot point because the benchmark doesn’t have enough GC’d objects for it to matter. Happy to be corrected.

2. 5

I think when people speak of GC overhead they’re not talking about the per-object overhead of tracking data. Rather, the total amount of memory space one needs to set aside to cover both memory actually in use, and memory no longer in use but not yet collected: garbage. This can often be quite a lot larger than the working set, especially if there is a lot of churn.

1. 3

Where does that exist in the benchmark?

I see that each goroutine calculates a hash and calls time.Sleep(). I don’t see any garbage (or at least not enough to know what the grandparent comment is talking about)

1. 1

The space can be reserved for future collections anyway. It’s well known that GCs require more space than strictly needed for alive objects.

1. 8

This was an interesting article, it breaks down the issues with net.IP well, and describes the path to the current solution well.

But.

This isn’t a difficult problem. Don’t waste a ton of space, don’t allocate everywhere, make it possible to make the type a key in the language’s standard map implementation. In C++, this would’ve been easy. In Rust, this would’ve been easy. In C, this would’ve been easy (assuming you’re using some kind of halfway decent map abstraction). It doesn’t speak well of Go’s aspiration to be a systems programming language that doing this easy task in Go requires a bunch of ugly hacks and a separate package to make a string deduplicator which uses uintptrs to fool the garbage collector and relies on finalizers to clean up. I can’t help but think that this would’ve been a very straightforward problem to solve in Rust with traits or C++ with operator overloading or even Java with its Comparable generic interface.

That’s not to say that the resulting netaddr.IP type is bad, it seems like basically the best possible implementation in Go. But there are clearly some severe limitations in the Go language to make it necessary.

1. 11

Almost all of the complexity that happened here is related to the ipv6 zone string combined with fitting the value in 24 bytes. Given that a pointer is 8 bytes and an ipv6 address is 16 bytes, you must use only a single pointer for the zone. Then, having amortized zero allocations with no space leaks for the zone portion, some form of interning with automatic cleanup is required.

If this is as easy as you claim in C/C++/Rust/whatever real systems language you want, can you provide a code snippet implementing it? I’d be happy to audit to see if it does meet the same (or better!) constraints.

1. 6

Here’s a C++ version: https://godbolt.org/z/E3WGPb - see the bottom for a usage example.

Now, C++ is a terrible language in many ways. It makes everything look super complicated, and there’s a lot of seemingly unnecessary code there, but almost all of that stems from having to make my own RAII type, which includes writing the default constructor, the move constructor, the copy constructor, the destructor, the move operator= and the copy operator=. That complexity is just par for the course in C++.

One advantage of the netaddr.IP type is that it doesn’t allocate for every zone, just for every new zone, thanks to the “intern” system. My code will allocate space for the zone for every IPv6 address with a zone. One could definitely implement a “zone cache” system for my IPZone type though, maybe using a shared_ptr instead of a raw pointer for refcounting. One would have to look at usage patterns to see whether the extra complexity and potential memory/CPU overhead would be worth it or if zones are so infrequently used that it doesn’t matter. At least you have the choice in C++ though (and it wouldn’t rely on finalizers and fooling the GC).

1. 7

They also had the choice to just make a copy of every string when parsing and avoid all of the “ugly hacks”. Additionally, a shared_ptr is 16 bytes, so you’d have to figure out some other way to pack that in to the IPAddress without allocations. So far, I don’t think you’ve created an equivalent type without any “ugly hacks”. Would you like to try again?

1. 6

I don’t think they had the choice to just copy the zone strings? My reading of the article was that the intern system was 100% a result of the constraint that A) IP addresses with no zone should be no bigger than 24 bytes and B) it should be possible to use IP addresses as keys. I didn’t see concern over the memory usage of an IP address’s zone string. Whether that’s important or not depends on whether zones are used frequently or almost never.

It’s obviously hard to write a type when the requirements are hypothetical and there’s no data. But here’s a version with a zone string cache: https://godbolt.org/z/P9MWvf. Here, the zone is a uint64_t on the IP address, where 0 represents an IPv4 address, 1 represents an IPv6 address with no zone, and any other number refers to some refcounted zone kept in that IPZoneCache class. This is the “zone mapping table” solution mentioned in the article, but it works properly because the IPAddress class’s destructor decrements the reference count.

1. 7

I don’t think they had the choice to just copy the zone strings? My reading of the article was that the intern system was 100% a result of the constraint that A) IP addresses with no zone should be no bigger than 24 bytes and B) it should be possible to use IP addresses as keys.

Indeed, interning is required by the 24 byte limit. That Interning avoids copies seems to be a secondary benefit meeting the “allocation free” goal. It was a mistake to imply that copying would allow a 24 byte representation and that interning was only to reduce allocations.

That said, your first solution gets away with avoiding interning because it uses C style (null terminated) strings so the reference only takes up a single pointer. Somehow, I don’t think that people would be happier if Go allowed or used C style strings, though, and some might consider using them an “ugly hack”.

I didn’t see concern over the memory usage of an IP address’s zone string. Whether that’s important or not depends on whether zones are used frequently or almost never.

One of the design criteria in the article was “allocation free”.

It’s obviously hard to write a type when the requirements are hypothetical and there’s no data. But here’s a version with a zone string cache: https://godbolt.org/z/P9MWvf.

Great! From what I can tell, this does indeed solve the problem. I appreciate you taking the time to write these samples up.

I have a couple of points to make about your C++ version and some hypothetical C or Rust versions as compared to the Go version, though.

1. It took your C++ code approximately 60 lines to create the ref-counted cache for interning. Similarly, stripping comments and reducing the intern package they wrote to a similar feature set also brings it to around 60 lines. Since it’s not more code, I assume the objection is to the kind of code that is written? If so, I can see that the C++ code you provided looks very much like straightforward C++ code whereas the Go intern package is very much not. That said, the authors of the intern package often work on the Go runtime where these sorts of tricks are more common.

2. In a hypothetical C solution that mirrors your C++ solution, it would need a hash-map library (as you stated). Would you not consider it an ugly hack to have to write one of those every time? Would that push the bar for implementing it C from “easy” towards “difficult”? Why should the Go solution not be afforded the same courtesy under the (now valid) assumption that an intern library exists?

3. I’ll note that when other languages gain a library that increases the capabilities, even if that library does unsafe hacks, it’s often viewed as a positive sign that the language is powerful enough to express the concept. Why not in this case?

4. In a hypothetical Rust solution, the internal representation (I think. Please correct me if I’m wrong) can’t use the enum feature because the tag would push the size limits past 24 bytes. Assuming that’s true, would you consider it an ugly hack to hand-roll your own union type, perhaps using unsafe, to get the same data size layout?

5. All of these languages would trivially solve the problem easily and idiomatically if the size was allowed to be 32 bytes and allocations were allowed (this is take 2 in the blog post). Similarly, I think they all have to overcome significant and non-obvious challenges to hit 24 bytes with no allocations as they did.

Anyway, I want to thank you for engaging and writing some code to demonstrate the type in C++. That’s effort you don’t usually get on the internet. This conversation has caused me to update my beliefs to agree more with adding interning or weak references to the language/standard library. Hopefully my arguments have been as useful to you.

2. 4

I agree—if Go is a systems language. But I don’t think it ever was supposed to be. Or if it was, it’s (in my opinion) really bad at it. Definitely worse than even something like C#, for exactly the reasons you’re highlighting.

I think Go was more originally designed to be a much faster language than Python (or perhaps Java), specifically for Google’s needs, and thus designed to compete with those for high-performance servers. And it’s fine at that. And I’ve thought about solving this kind of issue in those languages, too, using things like array in Python for example.

So I agree Go isn’t a good systems language, but I think that was a bit of retcon. It’s a compiled high-level language that could replace Python usage at Google. It’s not competing with Rust, C, Zig, etc.

1. 3

Ok, I can buy that. IIRC, it was originally promoted as a systems language, but it seems like they’ve gone away from that branding as well. There’s a lot of value to something like “a really fast, natively compiled Python”.

But even then, this article seems to demonstrate a pretty big limitation. Something as simple as using a custom IP address type as the key in a map, ignoring everything performance-related, seems extremely difficult. How would you write an IP address struct which stores an IPv4 address or an IPv6 address with an optional zone, which can be used a the key in a map, even ignoring memory usage and performance? Because that would be easy in Python too; just implement __hash__ and __eq__.

This is a problem which isn’t just related to Go’s positioning, be it a “systems language” or a “faster python”. Near the bottom we have C, where an IP address -> whatever map is about as difficult as any other kind of map. Slightly above, we have C++ and Rust, where the built-in types let you use your IP address class/struct as a key with no performance penalty, since you stamp out a purpose-built “IP address to whatever” map using templates. Above that again, we have Java and C#, which also makes it easy, though at a performance cost due to virtual calls (because genetics aren’t templates), though maybe the JIT optimises out the virtual call, who knows. Near the top, we have Python which makes it arguably even more straightforward than Java thanks to duck typing.

Basically, unless you put Go at the very bottom of the stack alongside C, this should be an easy task regardless of where you consider Go to fit in.

1. 3

IIRC, it was originally promoted as a systems language, but it seems like they’ve gone away from that branding as well.

I believe you’re correct about how Google promoted it. I just remember looking at it, thinking “this is absolutely not a systems language; it’s Limbo (https://en.wikipedia.org/wiki/Limbo_(programming_language), but honestly kind of worse, and without the interesting runtime,” and continuing to not use it. So I’m not sure the team itself actually thought they were doing a systems language.

But even then, this article seems to demonstrate a pretty big limitation. Something as simple as using a custom IP address type as the key in a map, ignoring everything performance-related, seems extremely difficult.

I completely agree, but that’s changing the discussion to whether Go is a good language, period. And since I mostly see that devolving into a flame war, I’m just going to just say that I think you have a lot of company, and also that clearly lots of people love the language despite any warts it has.

1. 2

I completely agree, but that’s changing the discussion to whether Go is a good language, period. And since I mostly see that devolving into a flame war, I’m just going to just say that I think you have a lot of company, and also that clearly lots of people love the language despite any warts it has.

My relationship with the language is… Complicated. I often enjoy it, I use it for work, and when I just want to write a small tool (such as when I wrote a process tree viewer) it’s generally my go-to “scripting” language these days. But I hate how the module system puts URLs to random git hosting websites in my source code, there’s a lot of things I dislike about the tooling, and the inability write a datastructure which acts like the built-in datastructures and the inability to write a type which works with the built-in datastructures are both super annoying issues which none of the other languages I use have. I’m hoping Go 2 will fix some of the bigger problems, and I’m always worried about which directions the corporate management at Google will take the language or its tooling/infrastructure.

But you’re right, this is tantamount to flamewar bait so I’ll stop now.

1. 12

It’s nice to bring some nuance to the discussion: some languages and ecosystems have it worse than others.

To add some more nuance, here’s a tradeoff about the “throw it in an executor” solution that I rarely see discussed. How many threads do you create?

Well, first, you can either have it be bounded or unbounded. Unbounded seems obviously problematic because the whole point of async code is to avoid the heaviness one thread per task, and you may end up hitting that worst case.

But bounded has a less obvious issue in that it effectively becomes a semaphore. Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex) and a thread pool of size 1. If you attempt to throw both on a thread pool, and A ends up scheduled and B doesn’t, you get a deadlock.

You don’t even need dependencies between tasks, either. If you have an async task that dispatches a sync task that dispatches an async task that dispatches a sync task, and your threadpool doesn’t have enough room, you can hit it again. Switching between the worlds still comes with edge cases.

This may seem rare and it probably is, especially for threadpools of any appreciable size, but I’ve hit it in production before (on Twisted Python). It was a relief when I stopped having to think about these issues entirely.

1. 3

Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex)

Isn’t this an antipattern for async in general? Typically you’d either a) make sure to release the mutex before yielding, or b) change he interaction to “B notifies A”, right?

1. 4

Changing the interaction to “B notifies A” doesn’t fix anything because presumably A waits until it is notified, taking up a threadpool slot, making it so that B can never notify A. Additionally, it’s not always obvious when one sync task depends on another, especially if you allow your sync tasks to block on the result of an async task. In my experience, that sort of thing happens when you have to bolt the two worlds together.

1. 2

It’s a general problem. It can happen whenever you have a threadpool, no matter whether it’s sync or async.

2. 3

But bounded has a less obvious issue in that it effectively becomes a semaphore. Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex) and a thread pool of size 1. If you attempt to throw both on a thread pool, and A ends up scheduled and B doesn’t, you get a deadlock.

Thread-based work scheduling can, imo, be just as complicated as async scheduling. The biggest difference is that async scheduling makes you pay the cost in code complexity (through function coloring, concurrency runtimes, etc) while thread-based scheduling makes you pay the cost in operational and architectural complexity (by deciding how many thread pools to have, which tasks should run on which pools, how large each pool should be, how long we should wait to retry to grab a thread from the pool, etc, etc). While shifting the complexity to operational and architectural complexity might seem to shift the work up to operators or some dedicate operationalizing phase, in practice the context lost by lifting decisions up to this level can make tradeoffs for pools and tasks non-obvious, making it harder to make good decisions. Also, as workloads change over time, new thread pools may need to be created and these new pools necessitate rebalancing of other pools, which requires a lot of churn. Async has none of these drawbacks (though to be clear, it has its own unique drawbacks.)

1. 8

I’ve never designed a system like this or worked on a system designed like this before. I’ve never had one task depend on the value of another task while both tasks were scheduled simultaneously.

Here’s perhaps a not-unreasonable scenario: imagine a cache with an API to retrieve some value for a key if it exists and otherwise compute, store, and return it. The cache exports an async API and the callback it runs to compute the value ends up dispatching a sync task to a threadpool (maybe it’s a database query using a sync library). We want the cache to be able to be accessed from multiple threads, so it is wrapped in a sync mutex.

Now imagine that an async task tries to use the cache that is backed by a threadpool of size 1. The task disaptches a thread which acquires the sync mutex, calls to get some value (waiting however on the returned future), and assuming it doesn’t exist, the cache blocks forever because it cannot dispatch the task to produce the value. The size of 1 isn’t special: this can happen with any bounded size thread pool under enough concurrent load.

One may object to the sync mutex, but you can have the same issue if the cache is recursive in the sense that producing a value may depend on the cache populating other values. I don’t think that’s very far fetched either. Alternatively, the cache may be a library used as a component of a sync object that is expected to be used concurrently and that is the part that contains the mutex.

In my experience, the problem is surprisingly easy to accidentally introduce when you have a code base that frequently mixes async and sync code dispatching to each other. Once I started really looking for it, I found many places where it could have happened in the (admittedly very wacky) code base.

1. 3

Fair enough that is a situation that can arise. Those situations I would probably reach for either adding an expiry to my threaded tasks or separating thread pools for DB or cache threads from general application threads. (Perhaps an R/W Lock would help over a regular mutex, but I realize that’s orthogonal to the problem at hand here and probably a pedagogical simplification.) The reality is that mixing sync and async code can be pretty fraught if you’re not careful.

2. 2

I have seen similar scenarios without a user visible mutex: you get deadlocks if a thread on a bounded thread pool waits for another task scheduled on the same thread pool

Of course, there are remedies, e.g. never schedule subtasks on the same thread pool. Timeouts help but still lead to abysmall behavior under load because your threads just idle around until the timeout triggers.

1. 1

Note that you can also run async Rust functions with zero (extra) threads, by polling it on your current thread. A threadpool is not a requirement.

1. 3

Isn’t that equivalent to either a threadpool of size 1 or going back to epoll style event loops? If it’s the former, you haven’t gained anything, and if it’s the latter, you’ve thrown out the benefits of the async keyword.

1. 3

Async has always been a syntax sugar for epoll-style event loops. Number of threads has nothing to do with it, e.g. tokio can switch between single and multi-threaded execution, but so can nginx.

Async gives you higher-level composability of futures, and the ease of writing imperative-like code to build state machines.

1. 29

One thing I don’t think I’m ever going to get is how much Go does in comments. In my mind, comments are a way to communicate something to other humans, or, sometimes, a way to communicate something to some external tool (such as a documentation generator). However, in Go, you configure whether the file should be built in comments, you write C code in comments with CGo, you write shell commands for code generation in comments, and, with 1.16, you embed data into your binary using comments.

I’m all for a kind of extensible meta language; a way to have commands in the code which the compiler proper will skip, but which official tooling might interpret. Something like a #pragma. But… in my mind, the official language toolchain shouldn’t extensively parse my comments.

On the positive side though, I think what embed does is really cool. More languages should have a way to embed arbitrary data into the binary (without hacks).

1. 17

If it helps, you can consider the string //go: to be equivalent to the string #pragma and not think of it as a comment. Reusing the comment syntax has the advantage that parsers/formatters don’t have to worry about parsing anything other than comments. Much like how a shebang (#!/usr/bin/env bash) happens to be a comment in a bunch of scripting languages.

1. 17

Yea, if it was only that, I wouldn’t have been too worried. However, you also have a case where a comment changes the semantics of the following import after the comment. The whole thing seems like a giant hack to me; like “changing the grammar is too much work, so we’ll just make the tooling parse your comments so we don’t have to change the compiler”. I’m not saying that’s how the process is; I’m saying that’s how it feels as an observer.

I would’ve probably done something like C, where lines starting with # are obviously going to be interpreted differently from other lines. The compiler could ignore all lines starting with #, so that the tooling could add new directives in a backwards-compatible way, and parsers/formatters could largely ignore them. It just would’ve been a structured thing instead of essentially parsing prose.

1. 10

The whole thing seems like a giant hack to me; like “changing the grammar is too much work, so we’ll just (…)

This is how I feel about imports as strings. I see no good reason why we need quotation marks around the import paths in Go, it should be obvious for a parser how to deal with a path there. In general the Go syntax is… inconsistent at best.

1. 3

I see no good reason why we need quotation marks around the import paths in Go

This is a guess, but it may be to make plumbing easier in Plan 9/Acme. Syntax like <something> and "something" is easier to match than just something.

1. 1

I think the comment you’re replying to is arguing that it’s not just "something" vs. something, but import "something" vs. import something.

1. 3

Yeah, I know, but multiple imports still need quotation marks to be plumbable:

import (
"path/to/package1"
"path/to/package2"
)


works, whereas

import (
path/to/package1
path/to/package2
)


wouldn’t.

1. 2

Not sure why that would be the case. the plumber gets the whole string either way, expanded out to the surrounding whitespace if there’s no selection.

1. 4

But the plumber has to know somehow that it is a Go import path. The quotation mark syntax in Go (and C) makes that clear.

For example, should test, without quotes, really be treated as a Go/C import path? Wouldn’t that be too broad a rule?

1. 1

The plumber also gets a bunch of context, like the place you plumbed from. It should use that.

2. 3

I thought the same at first, but it’s simpler for the lexer/tokenizer to not have to know what mode it’s in. It’ll (presumably) just output a STRING token, and it’s only the parser that has to know that it’s in an import directive. Go is partly about keeping parsing (and lexing) simple and fast.

3. 6

It’s not my most favourite syntax either, but using #pragma or //go: seems like a really minor issue. I’m not sure what the motivation was for choosing //go: over # or #pragma.

a case where a comment changes the semantics of the following import after the comment

I assume you mean package foo // import "bar" type comments? That one, in particular, was a mistake IMO. But also kind of obsoleted by modules, and not really related to //go: directives.

1. 1

I don’t know what package foo // import "bar" does. Could you elaborate on it?

I was talking mostly about CGo, where you have a giant comment with a mix of C code and #cgo pseudo-directives followed by a line with import "C", where the comment (minus the #cgo lines) is compiled by a C compiler and linked with the binary.

1. 3

It enforces that the package is imported as bar; often used for “vanity paths”; e.g. lobste.rs/mypkg instead of github.com/lobsters/mypkg. This prevents some problem where one dependency might use the lobste.rs path, and the other the github.com path. It’s a bit of a hack, and you can do the same with a go.mod file now.

cgo is kind of a tricky beast, yeah; but also fairly rare. And since you can write any C code in that comment, I’m also not sure how else to do it? Overall, it seems to work fairly well and in the grand scheme of things, it seems like a fairly minor issue to me.

2. 37

But it’s a great way to pretend that it’s a small language with few reserved words.

1. 7

Even with the comment syntax, Go has fewer reserved words than C.

1. 7

While I agree with your comment, it’s a non sequitur. Having fewer reserved words and being a smaller language are two different things. LISP has 0 reserved words, therefore it must be the smallest language, right?

1. 4

Go is still a small language.

2. 3

This is the long-standing rationale for not having comments in JSON, and I think it stands the test of time (and complaints of programmers).

1. 16

This was just paternalistic nonsense on Crockford’s part. While I don’t really understand the Go author’s decision to use comments for pragmas, I would never in my life give up all comments to put a stop to it. An absolute textbook example of cutting off one’s nose to spite one’s face.

1. 16

I think it makes perfect sense for JSON’s intended usage as a data interchange format with good interoperability between all sorts of different environments. Extensions have made this much harder than it needs to be on more than a few occasions in the past.

Now, if you want to use JSON for your config files then sure, it’s annoying. But that wasn’t the intended usage. If you’re driving a square peg through a round hole then you shouldn’t really complain about the peg shape but get a different peg instead.

1. 7

I still don’t buy it. If we’re considering the intended usage, one of the goals that gets thrown around is that it is “easy for humans to read and write” – just as long as they never need to convey something to another human being in the form of a comment!

There are so many other under-specified parts of JSON, like what to do with large integers, or what happens when a document contains duplicate keys. It is extremely frustrating that comments were intentionally and unnecessarily stripped out, while the hard problems were apparently just ignored.

1. 8

HTTP or SMTP are easy for humans to read and write, and don’t have comments either because in the intended usage space it’s not really needed. Like those data formats, JSON is primarily intended to send text from one program to the other, and the “easy for humans to read and write” is more as a debugging and testing aid than anything else.

There are so many other under-specified parts of JSON, like what to do with large integers, or what happens when a document contains duplicate keys. It is extremely frustrating that comments were intentionally and unnecessarily stripped out, while the hard problems were apparently just ignored.

Sure, it could be improved, maybe, but as you mention these are not easy things to define in a compatible way since different languages deal with these things in different ways. But I don’t think that’s a good reason to introduce more interoperability issues.

2. 2

The comment syntax is mainly used together with cgo. Including a header instead of inlining C makes it feel less hacky and allows for syntax highlighting of the C code for a larger range of editors: // #include "project.h".

1. 2

IIRC there was discussion of adding pragma syntax, but it was decided that it’s not worth it to add yet another syntax, since there already existed comment-based pragmas that would have to be kept because of the go 1 compat promise.

1. 6

(Reading the proposal) I am not convinced by type lists as proposed, they seem un-moduloar.

Summary: The bounds on type parameters are “constraints”, which are an extension of existing interface types (which describe a set of methods that the object should have). This makes it easy to declare that values of a parameter type must have a certain method. However, we may also want to use operators / multimethods on those values, and interface types are not expressive enough to specify this. The solution in the Generics proposal is type lists, basically you can define a constraint with a hardcoded list of allowed “underlying types” for your values (in addition to interface constraints).

For example to say “x < y should be supported on values of my type parameter”, the proposal suggests to define the following constraint:

// Ordered is a type constraint that matches any ordered type.
// An ordered type is one that supports the <, <=, >, and >= operators.
type Ordered interface {
type int, int8, int16, int32, int64,
uint, uint8, uint16, uint32, uint64, uintptr,
float32, float64,
string
}


I think this is un-modular. This means the standard library will contain a single, authoritative list of all types that can use the comparison operator in generic code. Maybe it’s okay for numeric comparison, but in general this approach sounds like it will add a lot of friction. You want to allow library authors to introduce concepts, and then library users to introduce their own user-defined types satisfying these concepts (as interface types allow for methods as opposed to operators); but this design forces library authors to decide in advance of a fixed list of admitted types.

If you compare with Haskell type classes or Rust traits or Scala implicits, in those languages it is easy to express that comparison operators for a type should be available. In Haskell for example Ord a is a constraint that basically requires that an operator (<) :: a -> a -> Bool is available, and usage of x < y under an explicit Ord a constraint will correctly infer that this is the operator we mean. In other words, a constraint on a type a come with operations that mention a, in more flexible ways than just “I want values at a to have this method”. Ord a is an example of constraint allowing to use a binary operators, but for example Default a is a constraint with that provides a default value of type a (default :: a), so here there is no value of a to call the operation on, a is the output of the overloaded operation default.

1. 0

You want to allow library authors to introduce concepts, and then library users to introduce their own user-defined types satisfying these concepts (as interface types allow for methods as opposed to operators); but this design forces library authors to decide in advance of a fixed list of admitted types.

I don’t think this is true - the example you give is special, because the standard library knows only those types can be comparable, since there’s no operator overloading.

Can you give an example of your issue without using that special case? My understanding is that library authors can define a type constraint that uses an interface, and downstream can then just implement that interface.

Eg.

interface Comparable {
CompareTo(other Comparable) int
}

func SomeAlgo[T Comparable](input []T) {
..
}

1. 2

As I said: it depends on how your concepts can be formulated. If they fit the mold of interfaces (they are about methods that you can call on the values of the parametrized type), then this works well. But when they don’t, it does not. Binary methods / multi-parameter functions like “compare” are not easily presented as interfaces.

In your example, nothing tells you that the two elements to be compared have to be of the same type, while you get this assurance for free with a signature such as (<) :: a -> a -> Bool (it says that the operator < takes two a and returns a Bool). In practice the CompareTo function for a user-defined type will do a type check and fail if the argument does not have the same type, but you can still easily mix things up, pass an argument of the wrong type to a CompareTo method, and get no warning that things are wrong. This is precisely the sort of imprecision that people are trying to solve by adding genericcs in the first place.

1. 1

As I said: it depends on how your concepts can be formulated. If they fit the mold of interfaces (they are about methods that you can call on the values of the parametrized type), then this works well. But when they don’t, it does not.

“Methods that you can call on values” is the only way to express concepts in Go. So I don’t think this is really a restriction. Or, at least, not a new restriction. I understand why some people would feel encumbered by this lack of expressivity — I do too, sometimes — but it’s an important part of what keeps Go so simple.

1. 2

This restriction was apparently problematic enough to the authors of the proposal that it’s their third version trying to devise mechanisms to avoid it (two previous versions used “contracts” presented under different forms). In my top comment, I am making the point that the mechanism they currently propose to work-around the restriction, namely “type lists”, is un-modular and may turn out to cause friction in practice, because it assumes that the person who defines a generic bound knows about all potential users of the bound (technically, this is often called a “closed-world assumptIon”).

(The previous reply made the point that the example is specific to a hardcoded operator of the standard library, so this is fine. I think that if the mechanism was only meant to apply in a couple hardcoded cases, it could be hidden underneath a few predeclared “primitive contracts” (such as comparable, compiler-defined without showing the definition) and not exposed to users. If it is exposed to users, it is probably because there are intended uses in user-libraries.)

1. 2

I believe type lists only exist to allow generic functions to specify which built-in language syntactic forms are allowed on the generic types. The reason why comparable is compiler defined is because it’s impossible to actually define with type lists. Equality (== and !=) is indeed an open set of underlying types (namely, some struct types). Every other built-in bit of syntax does follow the closed-world assumption: there is a finite list of underlying types that support <, for example. I cannot, at least, think of any reason to use type lists except to allow support for things like indexing, range, operators, etc.

I think the reason why they did not define a set of built in primitive contracts is because there’s something of an explosion of possibilities and required names. Here’s a table I made of the different ways a type can be used and what underlying types support them (that I make no claims about being fully correct or exhaustive, it was just a quick attempt):

operation              underlying types
--------------         ----------------
<, >, <=, >=           [u]int?, string, float?
==, !=                 [u]int?, string, float?, complex?, some structs
+                      [u]int?, string, float?, complex?
-, *, /                [u]int?, float?, complex?
|, ^, %, &, &^         [u]int?
<<, >>                 [u]int? but right hand side must be uint?
&&, ||, !              bool
x <- y                 chan T, chan<- T
<-x                    chan T, <-chan T
x[n]                   [n]T, *[n]T, []T, map[K]V, string
x[a:b]                 [n]T, *[n]T, []T, string
x[a:b:c]               [n]T, *[n]T, []T
range                  [n]T, *[n]T, []T, map[K]V, string, chan T, <-chan T
T{}                    [n]T, []T, map[K]V, structs

builtins               underlying types
---------------        ----------------
len(x)                 [n]T, *[n]T, []T, map[K]V, chan T, <-chan T, chan<- T, string
cap(x)                 [n]T, *[n]T, []T, chan T, <-chan T, chan<- T
make(T)                map[K]V, chan T, <-chan T, chan<- T
make(T, n)             map[K]V, chan T, <-chan T, chan<- T, []T
make(T, n, m)          []T
delete(m, k)           map[K]V

other special cases    why
-------------------    ---
append(t []T, ts ...T) but if T is byte, then ts can be a string
copy(d, s []T)         but if T is byte, then s can be a string
complex(a, b)          float? and outputs complex?
real(a), imag(a)       complex? and outputs float?


I hope that demonstrates some of the complexity of attempting to come up with a taxonomy of the allowed syntactic forms and why it might be reasonable to just leave that up to the users, especially when it is, indeed, a closed world except for equality. If I recall, the idea of defining builtin contracts was brought up many times during the design iteration, and it always failed when working out the specific details.

All that said, I find type lists to be particularly inelegant, and my least favorite aspect of the design. I hope something else is found to solve the problem: The desire to explicitly specify what syntax you are allowed to use on generic types so that you cannot silently break users of your generic function during a refactoring. For example, changing a usage of T{} to make(T) isn’t always valid for every T.

Also, as you noted, the example above that gave a non-type safe compare interface and function using it. The type safe version would be

type Comparable[T any] interface {
CompareTo(T) int
}

func SomeAlgo[T Comparable[T]](input []T) {
..
}


which requires methods of signature like

func (m MyType) CompareTo(n MyType) int { ... }


to satisfy the interface.

1. 1

1. 1

Your example defines an interface ComparableWith[T] of things that can be compared with T (I made a slight renaming to clarify my question). Is it possible in Go to express the interface of “being comparable to oneself”? I can always use [T ComparableWith[T]] to express the same bound, but it would be nice to be able to write

type Comparable interface {
CompareTo(Self) int
}

fun SomeAlgo[T Comparable](input []T) { ... }


I’m not aware of a way to express those “self types” in Go (and I can’t find them in the specification), but I’m not a Go expert so I thought I would ask.

1. 1

There is no “self type” currently in the language and I don’t believe anything like that exists in the current proposal, so I do not believe so.

1. 2

In this specific case, I believe the model only fails if you pick the constants appropriately: spawn more goroutines than buffered slots. But that’s basically the bug in the first place, so it leaves me with the feeling that it helps you find the bug if you know what the bug is already. That said, I can see how writing the model might help focus the author on the choice of constants and help them discover the bug.

I’d be interested to hear the authors thoughts on that. Does that feeling sound fair? Are there tools that would find choices for the constants that fail for you?

1. 6

In this case I modeled it to closely match the code, but normally you’d design the spec so that it “sweeps” through a space of possible initial values. It’d look something like

variables
Routines \in {1..x: x \in 1..MaxRoutines};
NumTokens \in 1..MaxTokens;
channels = [limitCh |-> 0, found |-> {}];
buffered = [limitCh |-> NumTokens];
initialized = [w \in Routines |-> FALSE];


Then it would try 1 routine and 1 token, 1 routine and 2 tokens, etc.

1. 1

Here I believe the NumRoutines constant exists to ensure the model is bounded– TLC will check every number of potential goroutines from 1 to whatever NumRoutines ends up being. In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

In my experience writing the model helps you better understand the problem and thus pick appropriate constants (as you mentioned). But even if it didn’t, it wouldn’t be unreasonable with this spec to plug in 10 for NumRoutines and see what happens.

With TLA+ models I find that I wind up with an emotion similar to writing untested code: if it 100% works the first time I get immediately suspicious and start poking at things until they break as a means to convince myself it was actually working.

1. 1

In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

I disagree. Found is unbuffered, and limitCh is buffered. I believe it will only find the bug if NumRoutines >= the buffer size of limitCh.

1. 1

Oh beans, in my pre-meeting rush I misread the spec. Which is definitely a separate problem!

So it looks like limitCh has a buffer the size of NumTokens and you’d want to ensure you pick NumRoutines > NumTokens during one of the runs, right? I’m not sure there’s a tool that checks your work there.

2. 1

Here I believe the NumRoutines constant exists to ensure the model is bounded– TLC will check every number of potential goroutines from 1 to whatever NumRoutines ends up being. In addition, there are actually only two buffered channels to begin with (“limitCh” and “found”). As long as NumRoutines >= 3, then, TLA will find this issue.

While that’s best practice, here I fixed the number of routines at NumRoutines for simplicity. It will find also find a bug with 2 routines and 1 token.

1. 21

This is hands down the best article I have ever read on NAT traversal. I’ve spent years of my life dealing with all of the fun issues that come up (for example, some UPnP implementations care about the Case-Sensitivity of the headers sent in the faux-HTTP request, and they don’t all agree!), and I still learned things reading it.

1. 10

Thank you! In return, I’ve just learned that some UPnP implementations care about header case. Thanks, I hate it!

But seriously, that’s great intel and TIL :). If you have references for what empirical behaviors you discovered, I’m interested!

1. 5

I don’t have any references, but I can say that I’ve also seen:

1. Routers may or may not like SOAPAction header value double quoted.
2. Routers may reject have multiple mappings to the same internal port, even with distinct destination IPs. For example, if you have a mapping for external port 8080 to internal address 192.168.1.100:7070, it will reject adding a mapping for external port 8181 to internal address 192.168.1.200:7070 because the internal port “collides”.

I think that’s all I can remember, and I don’t remember any data about how often these things occurred. Consumer routers are amazing creatures.

1. 5

Amazing creatures indeed. Thanks for the tips! So far, it seems that, thankfully, most routers these days that offer UPnP IGD also offer NAT-PMP or PCP. UPnP does grab a teeny bit more of the long tail, but you can sort of get away with ignoring UPnP in a lot of cases.

1. 5

Been waiting for this release, very nice! Hope binary sizes keep shrinking.

1. 2

Provided they don’t become absurd, why do you care about binary sizes?

1. 3

I’ve got some pretty slow internet - this morning I was downloading terraform 0.13 at ~70kb/s, haha. Granted, it’s a particularly large example.

Go hello world is ~10 MB. I understand there’s a lot of runtime packed in there (or something else? Go binaries compress particularly well, so there’s not a lot of entropy somewhere), but for some perspective I’ll beat the dead horse and say statically including musl libc weighs ~1 MB at most. So, it would be pretty cool to have like, 5 MB go hello world. I’d guess the go linker could be more smart, I never actually bothered to look at what’s taking up space.

In the end, it’s not really an issue for me personally. Definitely being idealistic here. I’d guess the engineering effort involved would just not be worth it, at least not to Google.

1. 1

10MB seemed outrageous to me, so I checked. Using

package main
import "fmt"
func main() { fmt.Println("Hello, world!") }


the binary output is 2MB for both 1.14 and 1.15. So I guess it’s pretty cool?

Also, since this release does shrink binary sizes, it shows that people are spending the engineering effort involved to shrink them.

1. 2

Ah sorry, I didn’t actually bother to verify my poor memory there. Thanks.

2. 1

Yes. I wish there was something like Go that could produce relly tiny ELF files.

1. 3

Does tinygo help?

1. 2

Have you looked at upx? Shrinks my Go binary by 50%.

-rwxr-xr-x   1 mikeperham  staff   5910544 Aug 12 12:32 faktory
-rwxr-xr-x   1 mikeperham  staff  11506756 Aug 12 12:32 faktory-big


https://upx.github.io

1. 3

It helps, but I’m thinking that an “hello world” ELF file making one system call to print the message and one to exit the program should take less than two kilobytes, not close to two megabytes (or more).

What is the Go compiler storing in a “hello world” executable to make it that large, and more importantly: why are unused bytes not automatically removed?

1. 3

Compared to C, there’s the addition of a garbage collector and a unicode database, at minimum.

1. 1

1. 2

Can you provide a list of all of the locations for all of the unicode databases on every one of these systems: https://github.com/golang/go/blob/master/src/go/build/syslist.go#L10

Additionally, provide the methods of interfacing with them, what their differences are, and what their commonalities are? Please be sure to do that for every version of Go for the last 10 years.

Too much work? I agree.

1. 1

Not one designed for fast in-memory access using, say, a paged array data structure. You really don’t want to parse and convert the UnicodeData.txt files provided by the unicode consortium into a reasonable query structure every time you start a binary that does text processing.

2. 1

You want Rust? Go always includes a managed runtime for GC, et al.

1. 1

No, I get that Go and Rust aims for different things, but I don’t understand why the runtime is included if it’s not being used.

1. 2

Because for any non-trivial program, the runtime will be included. Why care about optimizing binary size of a program that is useless?

1. 1

Embedded platforms often require small sizes. And I would love to use Go for writing a 4k demoscene demo, if it had been possible.

Regardless of the above, why include unused bytes in the first place?

1. 2

Because there’s engineering effort involved both in initial implementation and long term maintenance. Approximately zero programs would benefit from that engineering effort.

1. 1

I agree that there is effort required to implement it in the first place, but fewer bytes emitted usually results in a lower maintenance cost. That’s my intuition, at least.

All it takes is one dedicated developer that decides to send a pull request.

1. 2

I think you are misestimating multiple things.

1. The set of useful programs that do not use the runtime. You cannot allocate, use goroutines, use channels, use maps, use interfaces, or large portions of the reflect library, just to get started.
2. The effort involved to make the runtime removed from the small programs that don’t use it. The dead code elimination required is a non-trivial problem due to things like package initialization, generating tracebacks on crashes, etc.
3. The ongoing effort involved in maintaining that code. It would have to remain under test, work on all of the environments that support it, etc. That the compiler outputs less bytes in some specialized programs that can basically only add numbers does not imply that there would be less things to maintain going forward.

There is no calculus that makes this a useful optimization. The developers consistently have been putting in large efforts (like rewriting the linker, for example) to get even marginal (1-5%) reductions in binary size. Not every idea is a good one.

2. 1

should take less than two kilobytes

Compiled a simple “hello world” in both C and Go to static binaries and stripped them.

-rwxr-xr-x 1 rjp rjp 715912 Aug 14 08:42 hw-c
-rw-r--r-- 1 rjp rjp     97 Aug 14 08:41 hw.c
-rwxr-xr-x 1 rjp rjp 849976 Aug 14 08:42 hw-go
-rw-r--r-- 1 rjp rjp     52 Aug 14 08:41 hw.go


This is GCC 10.1.0, Go 1.14.6 on Linux 5.7.5-arch1-1, x86_64.

If I upx them, I get

-rwxr-xr-x 1 rjp rjp 283608 Aug 14 08:42 hw-c
-rwxr-xr-x 1 rjp rjp 346748 Aug 14 08:42 hw-go

1. 1

Yes, I dont get why those has to be that large either. Very few assembly instructions are needed.

1. 1

I believe

var _ I = (*A)(nil)


would save you an alloc all together.

1. 2

While that is theoretically correct, it turns out that

var _ I = new(A)


is compiled away, so there is no alloc to remove.

1. 1

But is it Turing complete?

I’ll see myself out.