1.

As a special but important case, note that 1/3, with no intervening spaces, is a single rational number, not the expression 1 divided by 3. This can affect precedence: 3/64 is 2 while 3 / 64 is 1/8 since the spacing turns the / into a division operator. Use parentheses or spaces to disambiguate: 3/(64) or 3 /64.

This seems… really bad. Lexer behavior is based on values? Why is 1/3 special but 1/5 isn’t? Is 1.0/3.0 covered? The language has first class rationals, why is this necessary?

1.

Why do you think it depends on values? It says both 1/3 and 3/6 have the property, so why would you believe 1/5 does not?

1.

It appears I misread, still seems kinda weird though, why have that? (Also markdown removing the asterisks in my post isn’t helping)

1.

Maybe that’s Pike’s background as physicist acting up :-). Having looked at similar formulae throughout my university years, reading 1/2 * m * v^2 as (1/2) * m * v^2 feels pretty natural to me, too – it’s very close to how you’d write it (\frac{1}{2} mv^2) if we weren’t stuck with 1970s terminals in 2022.

It’s also extraordinarily useful to have rational number support in calculators (but note that I don’t know if Ivy does these things specifically). For example it makes checking your results so much easier, because it preserves easily-checked proportions. There are various quantities that you expect to vary in a particular way – for example, to decrease linearly with, or with the square or cube of, the distance from a particular point. So if you know the value of that quantity at L=2 um, you expect a nice, round value at L = 4 um or L = 20 um. But if the length gets lost in the floating point rounding error, you don’t get those nice results, and depending on context, you often lose the ability to tell if two pages of thick integrals you really don’t feel like checking are right just by checking a few points in a table. There are also various “special” well-known values of otherwise complicated-looking functions – for example, you expect various special “nice” values for I and V at specific points on a transmission line because various ugly coefficients (sometimes irrational) cancel each other out. If you plug that in a purely numerical system, they get lost in floating-point errors and you don’t easily recognize them anymore.

FWIW, there’s a whole series of tricks that you learn as an engineering/physics/chemistry/whatever undergrad to use these things more efficiently, ranging from doing part of the computation on paper to get some of the uglier coefficients out to cleverly computing things in a slightly different order on your calculator to goad it into rounding values just right (RPN calculators can make that a little easier FWIW).

1.

1e5 is different from 1 e 5. For languages in the algol (c, pascal) syntax lineage, is it commonly illegal to have two terms in sequence. Not so in others, however. In apl, pertinently, 1e5 is a single number, while 1 e 5 invokes function e with arguments 1 and 5 (or, perhaps, passes 1 to the adverb e and invokes the derived function on 5; or, even, in dialects with stranding, might be a 3-element array comprising 1, e, and 5). In ml derivatives, ‘f 1e5’ invokes f with the single argument 1e5; ‘f 1 e 5’ invokes f with the three arguments 1, e, and 5 (modulo currying). In lisp, the situation is almost exactly the same as in TFA: ‘3/4’ reads as a ratio, but ‘3 / 4’ reads as the integer 3, the symbol /, and the integer 4.

That said, I do find the rule to be in somewhat poor taste (if not inconsistent in that respect with other choices made by the dialect), and prefer j’s notation, cf ‘3r4’.

markdown removing the asterisks

You may escape them using \.

1.

As a special but important case, note that 1/3, with no intervening spaces, is a single rational number, not the expression 1 divided by 3. This can affect precedence: 3/6*4 is 2 while 3 / 6*4 is 1/8 since the spacing turns the / into a division operator. Use parentheses or spaces to disambiguate: 3/(6*4) or 3 /6*4.

1. 2

The Go memory model has been revised to align Go with the memory model used by C, C++, Java, JavaScript, Rust, and Swift.

Does this improve Go’s C interoperability?

1. 5

No, I think it mostly just formalizes what the compiler has already been doing the whole time. The difficulty with C interoperability has always been due to the runtime: switching from goroutines with small stacks to a thread with a large enough stack for C code to safely run, and the garbage collector always needing precise information about what memory locations contain reachable pointers into the Go managed heap to track and potentially update.

https://research.swtch.com/mm is a great series of articles on the work that was done.

1. 8

C does not provide maps, and when I really need one I can implement it in less than 200 lines (won’t be generic, though). Were I to design a language like Hare or Zig, providing an actual (hash) map implementation would be way down my list of priorities. Even if it belongs in the standard library, my first order of business would be to make sure we can implement that kind of things.

In fact, Go made a mistake when it provided maps directly without providing general purpose generics. That alone hinted at a severe lack of orthogonality. If maps have to be part of the core language, that means users can’t write one themselves. Which means they probably can’t write many other useful data structures. As Go authors originally did, you could fail to see that if the most common ones (arrays, hash tables…) are already part of the core language.

The most important question is not whether your language has maps. It’s whether we can add maps that really matters. Because if we can’t, there’s almost certainly a much more serious root cause, such as the lack of generics.

1. 11

I think this is too a one-sided debate. Generics have benefits and drawbacks (to argument from authority, see https://nitter.net/graydon_pub/status/1036279571341967360).

Go’s original approach of providing just three fundamental generic data structures (vec, map, and chan) definitely was a worthwhile experiment in language design, and I have the feeling that it almost worked.

1. 10

At this point I’d argue that the benefits of even the simplest version of generics (not bounded, template-style or ML-functor-style, whatever) are so huge compared to the downsides, that it’s just poor design to create a new statically typed language without them. It’s almost like creating a language without function calls.

Go finally fixed that — which doesn’t fix all the other design issues like zero values or lack of sum types — but their initial set of baked-in generic structures was necessary to make the language not unbearable to use. If they hadn’t baked these in, who would use Go at all?

1. 3

other design issues like zero values

Could you share more here? I agree about Go generics, but its zero values are one thing I miss when using other imperative languages. They’re less helpful in functional languages, but I even miss zero values when using OCaml in an imperative style.

1. 5

Zero values are:

• not always something that makes sense (what’s the 0 value for a file descriptor? an invalid file descriptor, is what. For a mutex? same thing.) The criticism in a recent fasterthanlime article points this out well: Go makes up some weird rules about nil channels because it has to, instead of just… preventing channels from being nil ever.
• error prone: you add a field to a struct type, and suddenly you need to remember to update all the places you create this struct
• encouraging bad programming by not forcing definition to go with declaration. This is particularly true in OCaml, say: there are no 0 values, so you always have to initialize your variables. Good imperative languages might allow var x = undefined; (or something like that) but should still warn you if a path tries to read before writing to the field.
1. 3

nitpick: Go’s sync.Mutex has a perfectly valid and actually useful zero value: an unlocked mutex.

That said, I broadly agree with you; some types simply do not have a good default, and the best solution is not to fudge it and require explicit initialization.

@mndrix, note that there is a middle ground that gives you the best of both worlds: Haskell and Rust both have a Default type class/trait, that can be defined for types for which it does makes sense. Then you can just write

let foo = def
in ...


(or rust):

let foo = Default::default();


Note you can even write this in Go, it just applies to more types than it should:

func Zero[T any]() T {
var ret T
return ret
}

// Use:
foo := Zero[T]()


You could well define some mechanism for restricting this to certain types, rather than just any. Unfortunately, it’s hard for me to see how you could retrofit this.

1. 1

Thank you for the correction!

2. 3

not always something that makes sense (what’s the 0 value for a file descriptor? an invalid file descriptor, is what. For a mutex? same thing.)

Partially agreed on a mutex (though on at least some platforms, a 0 value for a pthread mutex is an uninitialised, unlocked, mutex and will be lazily initialised on the first lock operation). If you bias your fd numbers by one then a 0 value corresponds to -1, which is always invalid and is a useful placeholder, but your example highlights something very important: the not-present value may be defined externally.

I saw a vulnerability last year that was a direct result of zero initialisation, of a UID field. A zero value on *NIX means root. If you hit the code path that accidentally skipped initialising the field properly, then the untrusted thing would run as root. Similarly, on most *NIX systems (all that I know of, though POSIX doesn’t actually mandate this), fd 0 is stdin, which is (as you point out) a terrible default.

Any time you’re dealing with an externally defined interface, there’s a chance that either there is no placeholder value or there is a placeholder value and it isn’t 0.

1. 2

not always something that makes sense

Agreed. However, my experience is that zero values are sensible for roughly 90% of types, and Go’s designers made the right Huffman coding decision here.

The criticism in a recent fasterthanlime article points this out well: Go makes up some weird rules about nil channels because it has to, instead of just… preventing channels from being nil ever.

For anyone who comes along later, I think this is the relevant fasterthanlime article. Anyway, the behavior of nil and closed channels is well-grounded in the semantics of select with message passing, and quite powerful in practice. For me, this argument ends up favoring zero values, for channels at least.

you add a field to a struct type, and suddenly you need to remember to update all the places you create this struct

My experience has been that they’d all be foo: 0 anyway. Although in practice I rarely use struct literals outside of a constructor function in Go (same with records in OCaml) because I inevitably want to enforce invariants and centralize how my values are created. In both languages, I only have to change one place after adding a field.

by not forcing definition to go with declaration

The definition there but it’s implicit. I guess I don’t see much gained by having repetitive = 0 on each declaration, like I often encounter in C.

1. 1

what’s the 0 value for a file descriptor?

standard input

1. 2

I hope you’re not serious. I mean, sure, but it makes absolutely no sense whatsoever that leaving a variable uninitialized just means “use stdin” (if it’s still open).

1. 1

File descriptor 0 is standard input on unix systems. (Unless you close it and it gets reused, of course, leading to fun bugs when code expects it to be standard input.)

1. 1

As ludicrous as it would be, it would be a natural default value to have for careless language implementers, and before you know it users come to expect it. Even in C, static variables are all zero initialised and using one on read(2) would indeed read from standard input.

I’m sure we can point out various language quirks or weird idioms that started out that way.

2. 1

A zero-value file descriptor is invalid, sure, but a zero-value mutex is just an unlocked mutex. Why would that be invalid?

3. 1

There is no “zero” postal code, telephone number or user input.

2. 7

Thing is, Go not only is statically typed, it is garbage collected.

As such, it is quite natural for it to use heap allocation for (almost) everything, and compensate for this with a generational GC. Now things get a little more complicated if they want to support natively sized integers (OCaml uses 31/62-bit integers to have one bit to distinguish them from pointers, so the GC isn’t confused), but the crux of the issue is that when you do it this way, generics become dead simple: everything is a pointer, and that’s it. The size of objects is often just irrelevant. It may sometime be a problem when you want to copy mutable values (so one might want to have an implicit size field), but for mere access, since everything are pointers size does not affect the layout of your containing objects.

This is quite different from C++ and Rust, whose manual memory management and performance goals kinda force them to favour the stack and avoid pointers. So any kind of generic mechanism there will have to take into account the fact that every type might have a different size, forcing them to go to a specialization based template, which may be more complex to implement (especially if they want to be clever and specialise by size instead of by type).

What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages, and the resulting ignorance caused them to fool themselves into thinking generics were complicated. No they aren’t. It would have taken a couple additional weeks to implement them at most, and that would have saved time elsewhere (for instance they wouldn’t have needed to make maps a built in type, and pushed that out to the standard library).

I have personally implemented a small scripting language to pilot a test environment. Plus it had to handle all C numeric types, because the things it tests were low level. I went for static typing for better error reporting, local type inference to make things easier on the user, and added an x.f() syntax and a simple type based static dispatch over the first argument to get an OO feel. I quickly realised that some of the functions I needed required generics, so I added generics. It wasn’t perfect, but it took me like 1 week. I know that generics are simple.

The reason the debate there is so one sided is because Go should have had generics for the start. The benefits are enormous, the drawbacks very few. It’s not more complex for users who don’t use generics, generic data structures can still be used as if they were built in, it hardly complicates the implementation, and it improves orthogonality across the board.

“LoL no generics” was the correct way to react, really.

1. 8

It still seems to me that you are overconfident in this position. That’s fanboyism from my side, but Graydon certainly read TAPL, and if Graydon says “there’s a tradeoff between expressiveness and cognitive load” in the context of Go’s generics, it does seem likely that there’s some kind of tradeoff there. Which still might mean “LoL no generics” is the truth, but not via a one-sided debate.

Having covered meta issues, let me respond to specific points, which are all reasonable, but also are debatable :)

First, I don’t think the GC/no-GC line of argument holds for Go, at least in a simple form. Go deliberately distinguishes value types and pointer types (up to having a dedicated syntax for pointers), so “generics are easy ‘cause everything is a pointer” argument doesn’t work. You might have said that in Go everything should have been a pointer, but that’s a more complex argument (especially in the light of Java trying to move away from that).

Second, “It’s not more complex for users who don’t use generics,” – this I think is just in general an invalid line of argumentation. It holds in specific context: when you own the transitive closure of the code you are working with (handmade-style projects, or working on specific things at the base of the stack, like crypto libraries, alone or in a very small and tightly-knit teams). For “industrial” projects (and that’s the niche for Go) user’s simply don’t have the luxury of ignoring parts of the language. If you work on an average codebase with >10 programmers and >100k lines of code, the codebase will use everything which is accepted by the compiler without warnings.

Third, I personally am not aware of languages which solve generics problem in a low cognitive-load way. Survey:

C++ is obviously pretty bad in terms of complexity – instantiation-time errors, up-until-recently a separate “weird machine” for compile-time computations, etc.

Rust – it does solve inscrutable instantiation-time errors, but at the cost of far-more complex system, which is incomplete (waiting for GATod), doesn’t compose with other language features (async traits, const traits), and still includes “weird machine” for compile-time evaluation.

Zig. Zig is exciting – it fully and satisfactory solves the “weird machine” problem by using the same language for compile-time (including parametric polymorphism) and run-time computation. It’s also curious in that, as far as I understand, it also essentially “ignores TAPL” – there are no generics in the type system. It, however, hits “instantiation time errors” on the full speed. It seems to be that building tooling for such a language would be pretty hard (it would be anti-go in this sense). But yeah, Zig so far for me is one of the languages which might have solved generics.

Haskell – it’s a bit hard to discuss what even is the generics impl in Haskell, as it’s unclear which subset of pragmas are we talking about, but I think any subset generally leads to galactic-brain types.

OCaml – with questionable equality semantics, and modular implicit which are coming soon, I think it’s clear that generics are not solved yet. It also has a separate language for functors, which seems pretty cognitively complex.

Scala – complexity-wise, I think it’s a super-set of both Haskell and Java? I don’t have a working memory of Scala to suggest specific criticisms, but I tend to believe “Scala is complex” meme. Although maybe it’s much better in Scala 3?

Java – Java’s type system is broken in a trivial way (covariant arrays) and a (couple of?) interesting way (they extended type-inference when they added lambdas, and that inference allowed materialization of some un-denotable types which break the type system). I am also not sure that its “LoL covariant arrays”, as some more modern typed languages also make this decision, citing reduction of cognitive load. And variance indeed seems to be quite complex topic – Effective Java (I think?) spends quite some pages explaining “producer extends, consumer super”.

C# – I know very little about C#, but it probably can serve as a counter example for “generics in GC languages are simple”. Like Go, C# has value types, and, IIRC, it implements generics by just-in-time monomorphisation, which seems to be quite a machinery.

Now, what I think makes these systems complicated is not just parametric polymorphism, but bounded quantitication. The desire to express not only <T>, but <T: Ord>. Indeed, to quote TAPL,

This chapter introduces bounded quantification, which arises when polymorphism and subtyping are combined, substantially increasing both the expressive power of the system and its metatheoretic complexity.

I do think that there’s an under-explored design space of non-bounded generics, and very much agree with @c-cube . I am not quite convince that it would work and that Go should have been SML without functors and with channels, but that also doesn’t seem obviously worse than just three generic types! The main doubt for me is that having both interfaces and unbounded generics feels weird. But yeah, once I have spare time for implementing a reasonable complete language, unbounded generics is what I’d go for!

EDIT: forgot Swift, which absolutely tops my personal chart of reasons why adding generics to a language is not a simple matter: https://forums.swift.org/t/swift-type-checking-is-undecidable/39024.

1. 1

Ah. that’s one hypothesis down then. Thanks for the correction.

It holds in specific context: when you own the transitive closure of the code you are working with (handmade-style projects, or working on specific things at the base of the stack, like crypto libraries, alone or in a very small and tightly-knit teams). For “industrial” projects (and that’s the niche for Go) user’s simply don’t have the luxury of ignoring parts of the language. If you work on an average codebase with >10 programmers and >100k lines of code, the codebase will use everything which is accepted by the compiler without warnings.

OK, while I do have some experience with big projects, my best work by far was in smaller ones (including my crypto library, which by the way did not even need any generics to implement). What experience I do have with bigger projects however have shown me that most of the time (that is, as long as I don’t have to debug something), the only pieces of language I have to care about are those used for the API of whatever I’m using. And those tend to be much more reasonable than whatever was needed to implement them. Take the C++ STL for an extreme example: when was the last time you actually specified the allocator of a container? Personally I’ve never done it in over 10 years being paid to work with C++.

I personally am not aware of languages which solve generics problem in a low cognitive-load way

I have written one. Not public, but that language I’ve written for test environments? It had generics (unbounded, no subtyping), and I didn’t even tell my users. I personally needed them to write some of the functions of the standard library, but once that was done, I thought users would not really need them. (Yeah, it was easier to add full blown generics than having a couple ad-hoc generic primitives.)

Now, what I think makes these systems complicated is not just parametric polymorphism, but bounded quantitication. The desire to express not only <T>, but <T: Ord>.

In the code I write, which I reckon has been heavily influenced by an early exposure to OCaml (without the object part), I almost never use subtyping. Like, maybe 3 or 4 times in my entire career, two of which in a language that didn’t have closures (C++98 and C). If I design a language for myself, subtyping will be way down my list of priorities. Generics and closures will come first, and with closures I’ll have my poor man’s classes in the rare cases I actually need them. Even in C I was able to add virtual tables by hand that one time I had to have subtype polymorphism (It’s in my crypto library, it’s the only way I found to support several EdDSA hashes without resorting to a compilation flag).

I’ve heard of subtyping being successfully used elsewhere. Niklaus Witrth took tremendous advantage of it with its Oberon programming language and operating system. But I believe he didn’t have parametric polymorphism or closures either, and in this case I reckon subtyping & class based polymorphism are an adequate substitute.

Type classes (or traits) however are really enticing. I need more practice to have a definite opinion on them, though.

1. 2

Ah. that’s one hypothesis down then. Thanks for the correction.

To clarify, graydon, as far as I know, didn’t participate in Go’s design at all (he designed Rust), so this has no bearing on your assumption about designers of Go.

1. 2

Crap! Well, at least his argument has value.

2. 2

If generics are simple, can you read this issue thread and tell everyone else how to fix comparable to be consistent? TIA.

1. 2

Generics are simple under a critical condition: Design them from the ground up

Done after the fact in a system not designed for them, of course they’re going to be difficult. Then again, why waste a couple weeks of up front design when you can afford years of waiting and months of pain?

1. 3

I don’t agree that generics are simple, but I agree that designing them in from the start is vastly easier than trying to retrofit them to an existing language. There are a lot of choices in how generics interact with your type system (especially in the case of a structural type system, which Go has, and even more so in an algebraic type system). If you build generics in as part of your type system from the start then you can explore the space of things allowed by generics and the other features that you want. If you don’t, then you may find that the point that you picked in the space of other things that you want is not in the intersection of that and generics.

1. 0

I don’t agree that generics are simple

I kinda had to change my mind on that one. Generics can be very simple in some contexts (like my own little language), but I see now that Go wasn’t one of them.

If you build generics in as part of your type system from the start then you can explore the space of things allowed by generics and the other features that you want. If you don’t, then you may find that the point that you picked in the space of other things that you want is not in the intersection of that and generics.

One thing I take for granted since I went out of college in 2007, is that we want generics. If your language is even slightly general purpose, it will need generics. Even my little specialised language, I didn’t plan to add generics, but some functions in my standard library required it. So this idea of even attempting to design a language without generics feels like an obvious waste of time to me. Likewise for closures and sum types by the way.

There is one thing that can make me change my mind: systematic experimentation for my niche of choice. I design my language, and I write a real program with it, trying to use as few features as possible. For instance, my experience in writing a cryptographic library in C convinced me they don’t need generics at all. Surprisingly though, I did see a case for subtype polymorphism, in some occasions, but that happens rarely enough that an escape hatch like writing your vtable by hand is good enough. I believe Jonathan Blow is doing something similar for his gaming language, and Niklaus Wirth definitely did the same for Pascal (when he devised Modula and Oberon).

1. 2

One thing I take for granted since I went out of college in 2007, is that we want generics. If your language is even slightly general purpose, it will need generics

I completely agree here. I wrote a book about Go that was finished just after Go reached 1.0 and, even then, I thought it was completely obvious that once you’ve realised that you need generic maps you should realise that you will need other generic types. Having maps (and arrays and slices) as special-case generics felt like a very bad decision.

Mind you, I also thought that having a language that encouraged concurrency and made data races undefined behaviour, but didn’t provide anything in the type system to allow you to define immutable types or to limit aliasing was also a terrible idea. It turns out that most of the folks I’ve spoken to who use Go use it as a statically compiled Python replacement and don’t use the concurrency at all. There was a fantastic paper at ASPLOS a couple of years back that looked at concurrency bugs in Go and found that they are very common.

1. 1

I think Oberon and Go have a lot in common. Both were designed by an experienced language constructor late in his career, emphasising “simplicity” over everything else, leaving out even basic things such as enums, even though he had created more expressive languages earlier on.

1. 2

I tend to be more sympathetic to Wirth’s decisions, because he was working in a closed ecosystem he completely controlled and understood. I mean, they were like less than 5 people working on the entire OS + compiler + main applications, and in the case of the Lilith computer, and FPGA Oberon, even the hardware!

He could have criteria such as “does this optimisation makes the entire compiler bootstrap faster? Here it’s for speed, but I can see the idea that every single piece of complexity has to pay for itself. It was not enough for a feature to be beneficial, the benefits had to outweigh the costs. And he could readily see when they did, because he could fit the entire system in his head.

Go is in a different situation, where from the start it was intended for a rather large audience. Thus, the slightest benefit to the language, even if it comes at a significant up-front cost, is liable to pay huge dividends as it becomes popular. So while I can believe even generics may not be worth the trouble for a 10K LOC system (the size of the Oberon system), that’s a different story entirely when people collectively write hundreds of millions of lines of code.

1. 2

The best characterisation of Go and Oberon I can come up with is »stubborn«.

2. 2

While Go is garbage collected, it is still very much “value oriented” in that it gives control to the programmer for the layout of your memory, much like C++ and Rust. Just making everything a pointer and plugging your ears to the issues that brings isn’t solving the problem.

I’m glad that you added generics to your small scripting language for a test environment in 1 week. I don’t think that speaks much to the difficulty in adding it to a different language with a different type system and different goals. When they started the language, things like “very fast compile times” where very high priority, with the template system of C++ generics heavily inspiring that goal. It would be inane to start a project to avoid a problem in a language and then cause the exact same problem in it.

So, they didn’t want to implicitly box everything based on their experience with Java, and they didn’t want to template everything based on their experience with C++. Finding and implementing a middle ground is, in fact, difficult. Can you name any languages that avoid the problems described in https://research.swtch.com/generic?

The problem with “lol no generics” is that the word “generics” sweeps the whole elephant under the rug. There are a fractal of decisions to make when designing them with consequences in the type system, runtime, compiler implementation, programmer usability, and more. I can’t think of any languages that have the exact same generics system. Someone who prefers Rust generics can look at any other language and say “lol no traits”, and someone who prefers avoiding generics entirely (which, I promise, is a reasonable position to hold) may look around and say “lol 2 hour compile times”. None of those statements advance any conversation or are a useful way to react.

1. 1

I can’t think of any languages that have the exact same generics system. Someone who prefers Rust generics can look at any other language and say “lol no traits”,

Aren’t Rust traits analogous to Swift protocols?

1. -1

While Go is garbage collected, it is still very much “value oriented” in that it gives control to the programmer for the layout of your memory, much like C++ and Rust.

That kind of changes everything… Oops. (Really, I mean it.)

When they started the language, things like “very fast compile times” where very high priority, with the template system of C++ generics heavily inspiring that goal.

Several things conspire to make C++ compile times slow. The undecidable grammar, the complex templates, and the header files. Sure we have pre-compiled headers, but in general, those are just copy pasta that are being parsed and analysed over and over and over again. I’ve seen bloated header-only libraries add a full second of compilation time per .cpp file. And it was a logging library, so it was included everywhere. Properly isolating it solved the problem, and the overhead was reduce to one second for the whole project.

A simpler grammar is parsed basically instantly. Analysis may be slower depending on how advanced static checks are, but at least in a reasonable language it only has to happen once. Finally there’s code generation for the various instantiations, and that may take some time if you have many types to instantiate. But at least you don’t have to repeat the analysis, and the most efficient optimisations don’t take that much compilation time anyway.

1. 3

The undecidable grammar, the complex templates, and the header files. Sure we have pre-compiled headers, but in general, those are just copy pasta that are being parsed and analysed over and over and over again.

The parsing and analysis isn’t the whole problem. The C++ compilation model is an incremental evolution of Mary Allen Wilkes’ design from the ’70s, which was specifically designed to allow compiling complex problems on machines with around 2 KiB of memory. Each file is compiled separately and then pasted together in a completely separate link phase. In C++, inline and templated functions (including methods on templated classes) are emitted in every compilation unit that uses them. If 100 files use std::vector<int>::push_back then 100 instances of the compiler will create that template instantiation (including semantic analysis), generate IR for it, optimise it, and (if they still have calls to it left after inlining) spit out a copy of it in a COMDAT in the final binary. It will then be discarded at the end.

Sony has done some great work on a thing that they call a ‘compilation database’ to address this. In their model, when clang sees a request for std::vector<int>::push_back, it queries a central service to see if it’s already been generated. It can skip the AST generation and pull the IR straight from the service. Optimisers can then ignore this function except for inlining (and can provide partially optimised versions to the database). A single instance is emitted in the back end. This gives big compile time speedups, without redesigning the language.

It’s a shame that the Rust developers didn’t build on this model. Rust has a compilation model that’s more amenable to this kind of (potentially distributed) caching than C++.

2. 1

What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages,

I’ll just leave this here https://www.research.ed.ac.uk/en/publications/featherweight-go

1. 0

I meant back when Go first came out. That was over 12 years ago, in 2009. This paper is from 2020.

Nevertheless, the present thread significantly lowered my confidence in that claim. I am no longer certain Go designers failed to read Pierce’s work or equivalent, I now merely find it quite plausible.

2. 1

What’s clear to me is that Go’s designers didn’t read Pierce’s Type and Programming Languages . . .

Do you really think that Ken Thompson, Rob Pike, and Robert Greisemer were ignorant to this degree? That they made the design decisions they did based on a lack of theoretical knowledge?

1. 2

None of them is known for statically typed functional languages, and I know for a fact there is little cross talk between that community and the rest of the world. See Java, designed in 1995, twenty years after ML showed the world not only how to do generics, but how neat sum types are. Yet Java’s designers chose to have null instead, and generics came only years later. Now I kinda forgive them for not including generics at a time they likely believed class based polymorphism would replace parametric polymorphism (a.k.a. generics), but come on, adding null when we have a 20 year old better alternative out there?

So yeah, ignorance is not such an outlandish hypotheses, even with such people. (Edit: apparently one of them did read TAPL, so that should falsify the ignorance hypothesis after all.)

But that’s not my only hypothesis. Another possibility is contempt for their users. In their quest for simplicity, they may have thought the brains of Go programmers would be too weak to behold the frightening glory of generics. That instead they’d stay in the shelter of more familiar languages like Python or C. I’m not sure how right they may have been on that one to be honest. There are so many people that don’t see the obvious truth that programming is a form of applied maths (some of them explicitly fled maths), that I can understand they may panic at the first sight of an unspecified type. But come on, we don’t have to use generics just because they’re there. There’s no observable difference between a built in map and one that uses generics. Users can use the language now, and learn generics later. See how many people use C++’s STL without knowing the first thing about templates.

Yet another hypothesis is that they were in a real hurry, JavaScript style, and instead of admitting they were rushed, they rationalised the lack of generics like it was a conscious decision. Perhaps they were even told by management to not admit to any mistake or unfinished job.

1. 6

But that’s not my only hypothesis. Another possibility is contempt for their users. … Yet another hypothesis is that they were in a real hurry, JavaScript style, and instead of admitting they were rushed, they rationalised the lack of generics like it was a conscious decision. Perhaps they were even told by management to not admit to any mistake or unfinished job.

This is exhausting and frustrating. It’s my own fault for reading this far, but you should really aim for more charity when you interpret others.

1. 3

In the words of Rob Pike himself:

The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

I’d say that makes it quite clear.

1. 0

Ah, I didn’t remember that quote, thank you. That makes the contempt hypothesis much more plausible.

That being said, there’s a simple question of fact that is very difficult to ascertain: what the “average programmer” is capable of understanding and using, and at what cost? I personally have a strong intuition that generics don’t introduce unavoidable complexity significant enough to make people lives harder, but I’m hardly aware of any scientific evidence to that effect.

We need psychologists and sociologists to study is.

2. 1

I’ve ran out of charitable interpretations to be honest. Go designers made a mistake, plain and simple. And now that generics have been added, that mistake has mostly been fixed.

1. 1

I’m surprised you’re saying this after noticing that you didn’t know basic things about Go’s type system and implementation and learning those details “changes everything” (which, in my opinion, is commendable). Indeed, you’ve also apparently learned many facts about the authors and their history in this thread. Perhaps this is a good moment to be reflective about the arguments you’re presenting, with how much certainty you’re presenting them, and why.

1. 2

A couple things:

• I still think that omitting generics from a somewhat general purpose language past 2005 or so is a mistake. The benefits are just too large.
• I’ve seen comments about how the standard library itself had to jump through some hoops that wouldn’t be there if Go had generics from the start. So Go authors did have some warning.
• Go now has generics, even though adding them after the fact is much harder. There can be lots of reasons for this change, but one of them remains an admission of guilt: “oops we should have added generics, here you are now”.

So yeah, I still believe beyond reasonable doubt that omitting generics back then was a mistake.

3. 3

All of your “analyses” are rooted in a presumption of ignorance, or malice, or haughty superiority, or some other bad-faith foundation. Do you really thing that’s the truth of the matter?

There are so many people that don’t see the obvious truth that programming is a form of applied maths

Some programming is a form of applied math. Most programming, as measured by the quantity of code which exists and is maintained by human beings, is not. Most programming is the application of computational resources to business problems. It’s imperative, it’s least-common-denominator, and it’s boring.

1. 1

All of your “analyses” are rooted in a presumption of ignorance, or malice, or haughty superiority, or some other bad-faith foundation. Do you really thing that’s the truth of the matter?

The only way it’s false is if omitting generics was the right thing to do. I don’t believe that for a second. It was a mistake, plain and simple. And what could possibly cause mistakes, if not some form of incompetence or malice?

Most programming is the application of computational resources to business problems. It’s imperative, it’s least-common-denominator, and it’s boring.

It’s also maths. It’s also the absolutely precise usage of a formal notation that ends up being transformed into precise instructions for an (admittedly huge) finite state machine. Programs are still dependency graphs, whose density is very important for maintainability — even the boring ones.

It’s not the specific kind of maths you’ve learned in high school, but it remains just as precise. More precise in fact, given how unforgiving computers are.

1. 2

The only way it’s false is if omitting generics was the right thing to do. I don’t believe that for a second.

“The right thing to do” is a boolean outcome of some function. That function doesn’t have a a single objective definition, it’s variadic over context. Can you not conceive of a context in which omitting generics was the right thing to do?

1. 2

I see some:

1. Designing a language before Y2K. Past 2005, it is too easy to know about them to ignore them.
2. Addressing a specific niche for which generics don’t buy us much.
3. Generics are too difficult to implement.
4. Users would be too confused by generics.
5. Other features incompatible with generics are more important.

Go was designed too late for (1) to fly with me, and it is too general purpose for (2). I even recall seeing evidence that its standard library would have significantly benefited from generics. I believe Rust and C++ have disproved (3) despite Go using value types extensively. And there’s no way I believe (4), given my experience in OCaml and C++. And dammit, Go did add generics after the fact, which disavows (4), mostly disproves (3), and utterly destroys (5). (And even back then I would have a hard time believing (5), generics are too important in my opinion.)

So yeah, I can come up with various contexts where omitting generics is the right think to do. What I cannot do is find one that is plausible. If you can, I’m interested.

1. 1

[Go] is too general purpose for (2). I even recall seeing evidence that its standard library would have significantly benefited from generics.

You don’t need to speculate about this stuff, the rationale is well-defined and recorded in the historical record. Generics were omitted from the initial release because they didn’t provide value which outweighed the cost of implementation, factoring in overall language design goals, availability of implementors, etc. You can weight those inputs differently than the authors did, and that’s fine. But what you can’t do is claim they were ignorant of the relevant facts.

I believe Rust and C++ have disproved (3)

A language is designed as a whole system, and its features define a vector-space that’s unique to those features. Details about language L1 don’t prove or disprove anything about language L1. The complexity of a given feature F1 in language L1 is completely unrelated to any property of that feature in L2. So any subjective judgment of Rust has no impact on Go.

Go did add generics after the fact, which disavows (4), mostly disproves (3), and utterly destroys (5).

Do you just not consider the cost of implementation and impact on the unit whole as part of your analysis? Or do you weight these things so minimally as to render them practically irrelevant?

Generics did not materially impact the success of the goals which Go set out to solve initially. Those goals did not include any element of programming language theory, language features, etc., they were explicitly expressed at the level of business objectives.

1. 1

You don’t need to speculate about this stuff, the rationale is well-defined and recorded in the historical record.

What record I have read did not convince me. If you know of a convincing article or discussion thread, I’d like to read it. A video would work too.

I believe Rust and C++ have disproved (3)

A language is designed as a whole system […]

I picked Rust for a specific reason: manual memory management, which means value types everywhere, and the difficulties they imply for generics. That said, I reckon that Go had the additional difficulty of having suptyping. But here’s the thing: in a battle between generics and subtyping, if implementing both is too costly, I personally tend to sacrifice subtyping. In a world of closures, subtyping and suptype polymorphism simply are not needed.

Do you just not consider the cost of implementation and impact on the unit whole as part of your analysis?

I’m not sure what you mean there… I think pretty much everyone agrees that designing and implementing generics up front is much easier than doing so after the fact, in a system not designed for them. If the Go team/community were able to shoulder the much higher cost of after-the-fact generics, then they almost certainly could have shouldered the cost of up-front generics back then —even though the team was much smaller.

Generics did not materially impact the success of the goals which Go set out to solve initially.

Well if they just wanted to have a big user base, I agree. The Google brand and the reputation of its designers did most of that work. As for real goals, they’re the same as any language: help the target audience write better programs for cheaper in the target niche. And for this, I have serious doubts about the design of Go.

Now as @xigoi pointed out, Go authors targetted noobs. That meant making the language approachable by people who don’t know the relevant theory. That didn’t mean making the language itself dumb. Because users can’t understand your brilliant language doesn’t mean they won’t be able to use it. See every C++ tutorial ever, where you’re introduced to its features bit by bit. For instance when we learn I/O in C++ we don’t get taught about operator overloading (<< and >> magically work on streams, and we don’t need to know why just yet). Likewise we don’t learn template meta programming when we first encounter std::vector.

People can work with generics before understanding them. They won’t write generic code just yet, but they absolutely can take advantage of already written code. People can work with algebraic data types. They won’t write those types right away, but they can absolutely take advantage of the option type for the return value of functions that may fail.

A language can be brilliant and approachable. Yet Go explicitly chose to be dumb, as if it was the only way to be easy to work with. Here’s the thing though: stuff like the lack of generics and sum types tends to make Go harder to work with. Every time someone needed a generic data structure, they had to sacrifice type safety and resort to various conversion to and from the empty interface. Every time someone needs to report failures to the caller, they ended up returning multiple values, making things not only cumbersome, but also fairly easy to miss —with sum types at least the compiler warns you when you forget a case.

It’s all well and good to design a language for other people to use, but did they study the impact of their various decision on their target audience? If I’m writing a language for myself I can at least test it on myself, see what feels good, what errors I make, how fast I program… and most of my arguments will be qualitative. But if I’m writing for someone else, I can’t help but start out with preconceived notions about my users. Maybe even a caricature. At some point we need to test our assumptions.

Now I say that, such studies are bloody expensive, so I’m not sure what’s the solution there. When I made my little language, I relied on preconceived notions too. We had the requirements of course, but all I knew wast that my users weren’t programmers by trade. So I made something that tries its best to get out of their way (almost no explicit typing by default), and reports errors early (static typing rules). I guess I got lucky, because I was told later that they were happy with my language (and home grown languages have this reputation for being epically unusable).

1. 1

But here’s the thing: in a battle between generics and subtyping, if implementing both is too costly, I personally tend to sacrifice subtyping. In a world of closures, subtyping and suptype polymorphism simply are not needed.

Do you consider generics and subtyping and polymorphism and other programming language properties means to an end, or ends in themselves?

1. 0

Of course they’re a means to an end, why do you even ask?

In the programs I write, I need inheritance or subtyping maybe once a year. Rarely enough that using closures as a poor’s man classes is adequate. Heck, even writing the odd virtual table in C is enough in practice.

Generics however were much more useful to me, for two purposes: first, whenever I write a new data structure or container, it’s nice to have it work on arbitrary data types. For standard libraries it is critical: you’ll need what, arrays & slices, hash tables, maybe a few kind of trees (red/black, AVL…).

I don’t write those on a day to day basis, though. I’ve also grown susceptible to Mike Acton’s arguments that being generic often causes more problems than it solves, at least when speed matters. One gotta shape one’s program to one’s data, and that makes generic data structures much less useful.

My second purpose is less visible, but even more useful: the right kind of generics help me enforce separation of concerns to prevent bugs. That significantly speeds up my development. See, when a type is generic you can’t assume anything about values of that type. At best you can copy values around. Which is exactly what I’m looking for when I want to isolate myself from that type. That new data structure I’m devising just for a particular type? I’m still going to use generics if I can, because they make sure my data structure code cannot mess with the objects it contains. This drastically reduces the space of possible programs, which is nice when the correct programs are precisely in that space. You can think of it as defining bugs out of existence, like Ousterhout recommends in A Philosophy of Software Design.

2. 1

A language is designed as a whole system, and its features define a vector-space that’s unique to those features.

How do you add two languages, or multiply a language by an element of a field?

1. 7

Looking through the code, it appears to first hex encode and then truncate the resulting hash. That means no matter how much input you feed it, you don’t have a 256 bit input seed, you only have at best a 128 bit input seed. Now, ed25519 claims to target a 128 bit security level. I’m also not a cryptographer, but I’d guess that because the constructor expects 32 byte inputs or it panics, 256 bits of entropy is probably required to get the claimed security levels, and your keys may be significantly weaker.

Additionally, the tool doesn’t consume the full input, just the first line, which means that if you do ( echo foo; echo bar ) | anything2ed25519, you get the same key as ( echo foo ) | anything2ed25519. That seems like a dangerous footgun on an already dangerous tool.

Given the above problems, I don’t think the disclaimer of “the crypto is safe” on your README is warranted.

1. 1

Good spot. I’ll fix that and remove the safe crypto at once. :)

However, the limit is in golang’s implementation;

NewKeyFromSeed calculates a private key from a seed. It will panic if len(seed) is not SeedSize. This function is provided for interoperability with RFC 8032. RFC 8032’s private keys correspond to seeds in this package.

That limit doesn’t really make sense either since the next thing the code does is to sha512 sums it.

1. 1

That limit, and the sha512, are both described to be those specific values by the referenced RFC. I’d imagine they exist for a reason.

2. 1

That means no matter how much input you feed it, you don’t have a 256 bit input seed, you only have at best a 128 bit input seed.

Please read more carefully then. It’s SHA256 summed before the step you’re talking about meaning everything you input will be taken into the seed, but as a sum of a 256 bit hash. The first line bug was true, and it’s patched. (Thanks a lot for telling also)

It was written 3-4 or maybe more years ago so I had to refresh myself with the code and I realised it is hashed before truncate. See https://github.com/mikalv/anything2ed25519/blob/main/main.go#L52

1. 2

I’m not sure how to more clearly explain this, but I’ll try again. The output of a sha256 hash is 256 bits (32 bytes). By hex encoding it, you end up with 512 bits (64 bytes) of data representing the original 256 bits. By then passing the first 32 bytes of the hex encoded data to the seed generation, you are only actually passing the first 16 bytes (or 128 bits) of the hash.

In other words, hexEncode(data[:16]) == hexEncode(data)[:32] (https://go.dev/play/p/DApGUgSuDy6)

1. 2

Oh thanks. Perfectly now, I see my mistake and has pushed a fix for it where I don’t encode it before passing it. I should probably have reviewed it much more closely before publishing as I’m more aware of such and other common crypto mistakes today than back some years. My bad, sorry!

And again, thanks a lot for the constructive feedback :)

1. 1

No problem! Thanks for hearing it out and fixing the problems.

1. 1

Yea, sorry I read your first comment once again and I did get it on the first comment as well, but I had to have misread it or something the first time. I also updated the blog post itself warning about it’s being patched and a thanks to you for pointing it out :)

1. 18

This article incorrectly states that Zig has “colored” async functions. In reality, Zig async functions do not suffer from function coloring.

Yes, you can write virtually any software in Zig, but should you? My experience in maintaining high-level code in Rust and C99 says NO.

Maybe gain some experience with Zig in order to draw this conclusion about Zig?

1. 5

Not sure if he changed the text but the article mentions the async color problem such that it could be considered applying generally. But the article doesn’t link that to Zig explicitly or did I miss it?

It would be fair to mention how Zig solved it as he mentions it for Go.

1. 9

This response illustrates the number one reason I am not a fan of Zig: its proponents, like the proponents of Rust, are not entirely honest about it.

In reality, Zig async functions do not suffer from function coloring.

This is a lie. In fact, that article, while a great piece of persuasive writing, is also mostly a lie.

It tells the truth in one question in the FAQ:

Q: SO I DON’T EVEN HAVE TO THINK ABOUT NORMAL FUNCTIONS VS COROUTINES IN MY LIBRARY?

No, occasionally you will have to. As an example, if you’re allowing your users to pass to your library function pointers at runtime, you will need to make sure to use the right calling convention based on whether the function is async or not. You normally don’t have to think about it because the compiler is able to do the work for you at compile-time, but that can’t happen for runtime-known values.

In other words, Zig still suffers from the function coloring problem at runtime. If you do async in a static way, the compiler will be able to cheese the function coloring problem away. In essence, the compiler hides the function coloring problem from you when it can.

But when you do it at runtime and the compiler can’t cheese it, you still have the function coloring problem.

I think it is a good achievement to make the compiler able to hide it most of the time, but please be honest about it.

1. 17

Calling this dishonest and a lie is incredibly uncharitable interpretation of what is written. Even if you’re right that it’s techncially incorrect, at worst it’s a simplification made in good faith to be able to talk about the problem, no more of a lie than teaching newtoniam mechanics as laws of physics in middle school is a lie because of special relativity, or teaching special relativity in high school is a lie because of general relativity.

Also, I’m not familiar with zig, but from your description I think you’re wrong to claim that functions are colored. Your refutation of that argument is that function pointers are colored, but functions are a distinct entity from function pointers - and one used much more frequently in most programming languages that have both concepts. Potentially I’m misunderstanding something here though, there is definitely room for subtlety.

1. 11

Calling this a dishonest and a lie is incredibly uncharitable interpretation of what is written.

No, it’s not. The reason is because they know the truth, and yet, they still claim that Zig functions are not colored. It is dishonest to do so.

It would be completely honest to claim that “the Zig compiler can make it appear that Zig functions are not colored.” That is entirely honest, and I doubt it would lose them any fans.

But to claim that Zig functions not colored is a straight up lie.

Even if you’re right that it’s techncially incorrect,

I quoted Kristoff directly saying that Zig functions are colored. How could I be wrong?

at worst it’s a simplification made in good faith to be able to talk about the problem, no more of a lie than teaching newtoniam mechanics as laws of physics in middle school is a lie because of special relativity, or teaching special relativity in high school is a lie because of general relativity.

There are simplifications that work, and there are simplifications that don’t.

Case in point: relativity. Are you ever, in your life, going to encounter a situation where relativity matters? Unless you’re working with rockets or GPS, probably not.

But how likely is it that you’re going to run into a situation where Zig’s compiler fails to hide the coloring of functions? Quite likely.

Here’s why: while Kristoff did warn library authors about the function coloring at runtime, I doubt many of them pay attention because of the repetition of “Zig functions are not colored” that you hear all of the time from Andrew and the rest. It’s so prevalent that even non-contributors who don’t understand the truth jump into comments here on lobste.rs and on the orange site to defend Zig whenever someone writes a post about async.

So by repeating the lie so much, Zig programmers are taught implicitly to ignore the truthful warning in Kristoff’s post.

Thus, libraries get written. They are written ignoring the function coloring problem because the library authors have been implicitly told to do so. Some of those libraries take function pointers for good reasons. Those libraries are buggy.

Then those libraries get used. The library users do not pay attention to the function coloring problem because they have been implicitly told to do so.

And that’s how you get bugs.

It doesn’t even need to be libraries. In my bc, I use function pointers internally to select the correct operation. It’s in C, but if it had been in Zig, and I had used async, I would probably have been burned by it if I did not know that Zig functions are colored.

Also, I’m not familiar with zig, but from your description I think you’re wrong to claim that functions are not colored. Your refutation of that argument is that function pointers are colored, but functions are a distinct entity from function pointers - and one used much more frequently in most programming languages that have both concepts. Potentially I’m misunderstanding something here though, there is definitely room for subtlety.

You are absolutely misunderstanding.

How can function pointers be colored? They are merely pointers to functions. They are data. Data is not colored; code is colored. Thus, function pointers (data that just points to functions) can’t be colored but functions (containers for code) can be.

If data could be colored, you would not be able to print the value of the pointer without jumping through hoops, but I bet if you did the Zig equivalent of printf("%p\n", function_pointer); it will work just fine.

So if there is coloring in Zig, and Kristoff’s post does admit there is, then it has to be functions that are colored, not function pointers.

In Kristoff’s post, there is this comment in some of the example code:

// Note how the function definition doesn't require any static
// async marking. The compiler can deduce when a function is
// async based on its usage of await.


He says “when a function is async…” An async/non-async dichotomy means there is function coloring.

What the compiler does is automagically detect async functions (as Kristoff says) and inserts the correct code to call it according to its color. That doesn’t mean the color is gone; it means that the compiler is hiding it from you.

For a language whose designer eschews operator overloading because it hides function calls, it feels disingenuous to me to hide how functions are being called.

All of this means that Zig functions are still colored. It’s just that, at compile time, it can hide that from you. At runtime, however, it can’t.

And that is why Zig functions are colored.

1. 7

I have a hard time following all the animosity in your replies. Maybe I’m just not used to having fans on the internet :^)

In my article, and whenever discussing function coloring, I, and I guess most people, define “function coloring” the problem of having to mark functions as async and having to prepend their invocation with await, when you want to get their result. The famous article by Bob Nystrom, “What Color is Your Function?” also focuses entirely on the problem of syntactical interoperability between normal, non-async code and async, and how the second infects codebases by forcing every other function to be tagged async, which in turn forces awaits to be sprinkled around.

In my article I opened mentioning aio-libs, which is a very clear cut example of this problem: those people are forced to reinvent the wheel (ie reimplement existing packages) because the original codebases simply cannot be reasonably used in the context of an async application.

This is the problem that Zig solves. One library codebase that, with proper care, can run in both contexts and take advantage of parallelism when available. No async-std, no aio-libs, etc. This works because Zig changes the meaning and usage of async and await compared to all other programming languages (that use async/await).

You seem to be focused on the fact that by doing async you will introduce continuations in your program. Yes, you will. Nobody said you won’t. What you define as “cheesing” (lmao) is a practical tool that can save a lot of wasted effort. I guess you could say that levers and gears cheesed the need for more physical human labor, from that perspective.

Sure, syntax and the resulting computational model aren’t completely detached: if you do have continuations in your code, then you will need to think about how your application is going to behave. Duh, but the point is libraries. Go download OkRedis. Write an async application with it, then write a blocking applicaton with it. You will be able to do both, while importing the same exact declarations from my library, and while also enjoying speedups in the async version, if you allowed for concurrent operations to happen in your code.

But how likely is it that you’re going to run into a situation where Zig’s compiler fails to hide the coloring of functions? Quite likely.
Thus, libraries get written. They are written ignoring the function coloring problem because the library authors have been implicitly told to do so. Some of those libraries take function pointers for good reasons. Those libraries are buggy.

No. Aside from the fact that you normally just pass function identifiers around, instead of pointers, function pointers have a type and that type also tells you (and the compiler) what the right calling convention is. On top of that, library authors are most absolutely not asked to ignore asyncness. In OkRedis I have a few spots where I explicitly change the behavior of the Redis client based on whether we’re in async mode or not.

The point, to stress it one last time, is that you don’t need to have two different library codebases that require duplicated effort, and that in the single codebase needed, you’re going to only have to make a few changes to account for asyncness. In fact, in OkRedis I only have one place where I needed to account for that: in the Client struct. Every other piece of code in the entire library behaves correctly without needing any change. Pretty neat, if you ask me.

1. 2

I have a hard time following all the animosity in your replies. Maybe I’m just not used to having fans on the internet :^)

The “animosity” (I was more defending myself vigorously) comes from Andrew swearing at me and accusing me, which he might have had a reason.

In his post, he claimed I said he was maliciously lying, but I only said that he was lying. I separate unintentional lies from intentional lies, and I believe all of you are unintentionally lying. Because I realized he thought that, I made sure to tell him that and tell him what I would like to see.

In my article, and whenever discussing function coloring, I, and I guess most people, define “function coloring” the problem of having to mark functions as async and having to prepend their invocation with await, when you want to get their result. The famous article by Bob Nystrom, “What Color is Your Function?” also focuses entirely on the problem of syntactical interoperability between normal, non-async code and async,

In Bob Nystrom’s post, this is how he defined function coloring:

The way you call a function depends on its color.

That’s it.

Most people associate color with async and await because that’s how JavaScript, the language from his post, does it. But that’s not how he defined it.

After playing with Zig’s function pointers, I can say with confidence that his definition, “The way you call a function depends on its color,” does apply to Zig.

and how the second infects codebases by forcing every other function to be tagged async, which in turn forces awaits to be sprinkled around.

This is what Zig does better. It limits the blast radius of async/await. But it’s still there. See the examples from my latest reply to Andrew. I had to mark a call site with @asyncCall, including making a frame. But then, I couldn’t call the blue() function because it still wasn’t async. So if I were to make it work, I would have to make blue() async. And I could do that while still making the program crash half the time.

(Side note: I don’t know how to write out the type of async function. Changing blue() to async is not working with the [2]@TypeOf(blue) trick that I am using. It’s still giving me the same compile error.)

In my article I opened mentioning aio-libs, which is a very clear cut example of this problem: those people are forced to reinvent the wheel (ie reimplement existing packages) because the original codebases simply cannot be reasonably used in the context of an async application.

This is the problem that Zig solves. One library codebase that, with proper care, can run in both contexts and take advantage of parallelism when available. No async-std, no aio-libs, etc. This works because Zig changes the meaning and usage of async and await compared to all other programming languages (that use async/await).

This is not what you are telling people, however. You are telling them that Zig does not have function colors. Those two are orthogonal.

And I also doubt that Zig actually solves that problem. I do not know Zig, and it took me all of 30 minutes to 1) find a compiler bug and 2) find an example where you cannot run code in both contexts.

You seem to be focused on the fact that by doing async you will introduce continuations in your program. Yes, you will. Nobody said you won’t. What you define as “cheesing” (***) is a practical tool that can save a lot of wasted effort. I guess you could say that levers and gears cheesed the need for more physical human labor, from that perspective.

I have no idea what swear word you used there (I have a filter that literally turns swear words into three asterisks like you see there), but this is why I am not happy with Andrew. Now, I am not happy with you.

I used “cheesing” because while it is certainly a time saver, it’s still cheating. Yes, levers and gears cheese the application of force. That’s not a bad thing. Computers are supposed to be mental levers or “bicycles for the mind.” Cheesing is a good thing.

And yes, I am focused on introducing continuations into the program because there is a better way to introduce continuations and still get concurrency.

In fact, I am going to write a blog post about that better way. It’s called structured concurrency, and it introduces continuations by using closures to push data down the stack.

Sure, syntax and the resulting computational model aren’t completely detached: if you do have continuations in your code, then you will need to think about how your application is going to behave. Duh, but the point is libraries. Go download OkRedis. Write an async application with it, then write a blocking applicaton with it. You will be able to do both, while importing the same exact declarations from my library, and while also enjoying speedups in the async version, if you allowed for concurrent operations to happen in your code.

Where’s the catch? There’s always a catch. Please tell me the catch.

In fact, this whole thing is about me asking you, Andrew, and the others to be honest about what catches there are in Zig’s async story.

Likewise, I’m going to have to be honest about what catches there are to structured concurrency, and you can hold me to that when the blog post comes out.

No. Aside from the fact that you normally just pass function identifiers around, instead of pointers, function pointers have a type and that type also tells you (and the compiler) what the right calling convention is.

That is just an admission that functions are colored, if they have different types.

On top of that, library authors are most absolutely not asked to ignore asyncness. In OkRedis I have a few spots where I explicitly change the behavior of the Redis client based on whether we’re in async mode or not.

They are not explicitly asked. I said “implicitly” for a reason. “It’s not what programming languages do, it’s what they [and their communities] shepherd you to.” By telling everyone that Zig does not have function colors, you are training them to not think about it, even the library authors. As such, you then have to find those library authors, tell them to think about it, and explain why. It would save you and Andrew time if you just were upfront about what Zig does and does not do. And you would have, on average, better libraries.

The point, to stress it one last time, is that you don’t need to have two different library codebases that require duplicated effort, and that in the single codebase needed, you’re going to only have to make a few changes to account for asyncness. In fact, in OkRedis I only have one place where I needed to account for that: in the Client struct. Every other piece of code in the entire library behaves correctly without needing any change. Pretty neat, if you ask me.

That is neat. I agree. I just want Zig users to understand that, not be blissfully unaware of it.

1. 1

The “animosity” (I was more defending myself vigorously) comes from Andrew swearing at me and accusing me, which he might have had a reason.

You called me a liar in the first comment you wrote.

Where’s the catch? There’s always a catch. Please tell me the catch.

Since I’m such a liar, why don’t you write some code and show me, and everyone else, where the catch is.

1. 1

Since I’m such a liar, why don’t you write some code and show me, and everyone else, where the catch is.

Well, I don’t need to write code, but I can use your own words. You said that, “Every suspend needs to be matched by a corresponding resume” or there is undefined behavior. When asked if that could be a compiler warning, you said, “That’s unfortunately impossible, as far as I know.”

That’s the catch.

1. 2

Why would you even use suspend and resume in a normal application? Those are low level primitives. I didn’t use either in any part of my blog post, and in fact you won’t find them inside OkRedis either. Unless you’re writing an event loop and wiring it to epoll or io_uring, you only need async and await.

This is not a philosophical debate: talk is cheap, as they say, so show me the code. I showed you mine, it’s OkRedis.

1. 1

Why would you even use suspend and resume in a normal application? Those are low level primitives.

Then why are they the first primitives you introduce to new users in the Zig documentation? They should have been last, with a clear warning about their caveats, if you even have them in the main documentation at all.

This is not a philosophical debate: talk is cheap, as they say, so show me the code. I showed you mine, it’s OkRedis.

I’m not going to download OkRedis or write code with it. I only learned enough Zig to make my examples to Andrew compile, and I have begun to not like Zig at all. It’s confusing and a mess, in my opinion.

But if you think that the examples I gave Andrew are not good enough, I don’t know what to tell you. I guess we’ll see if they are good enough for the people that read my blog post on it.

But I do have another question: people around Zig have said that its async story does not require an event loop, but none have explained why. Can you explain why?

1. 3

Then why are they the first primitives you introduce to new users in the Zig documentation? They should have been last, with a clear warning about their caveats, if you even have them in the main documentation at all.

They’re the basic building block used to manipulate async frames (Zig’s continuations). First you complained that my blog post didn’t talk about how async frames work, and that I meant to deceive people by not talking about it, then you read the language reference and say it should not even mention the language features that implement async frames.

With your attitude in this entire discussion, you put yourself in a position where you have an incentive to not understand things, even well established computer science concepts such as continuations. If we talk at a high level, it’s a lie, if we get into the details, it’s confusing (and at this point we know what you mean to say: designed to be confusing). I can’t help you once you go there.

I’m looking forward to reading your blog post, although in all frankness you should consider doing some introspection before diving into it.

1. 1

They’re the basic building block used to manipulate async frames (Zig’s continuations). First you complained that my blog post didn’t talk about how async frames work, and that I meant to deceive people by not talking about it, then you read the language reference and say it should not even mention the language features that implement async frames.

That’s the language reference? I thought it was the getting started documentation. Those details are not good to put in documentation for getting started, but I agree that they are good for a language reference. I would still put them last, though.

With your attitude in this entire discussion, you put yourself in a position where you have an incentive to not understand things, even well established computer science concepts such as continuations.

That’s a little ad hominem. I can understand continuations and not understand how they are used in Zig because the language reference is confusing. And yes, it is confusing.

If we talk at a high level, it’s a lie, if we get into the details, it’s confusing

It turns out that the problem is in your documentation and in your blog post. You can talk about it at a high level as long as your language about it is accurate. You can talk about the low level details once the high level subtleties are clarified.

(and at this point we know what you mean to say: designed to be confusing). I can’t help you once you go there.

I do not believe Zig was designed to be confusing, but after using it, I can safely say that the language design was not well done to prevent such confusion.

As an example, and as far as I understand at the moment, the way Zig “gets around” the function colors problem is to reuse the async and await keywords slightly differently than other languages and uses suspend to actually make a function async. So in typical code, async and await do not have the function coloring problem. Which is great and all, but the subtleties of using them are usually lost on programmers coming from other languages.

When I first heard about Zig, by the way, I was excited about it. This was back in 2018, I think, during the part of its evolution where it had comptime but not much more complexity above C. I thought comptime was great (that opinion has changed, but that’s a different story), and that the language looked promising.

Fast forward to today: Zig is immensely more complex than it was back then, and I don’t see what that complexity has bought.

That’s not a problem in and of itself, but complexity does make things harder, which means the documentation should be clearer and more precise. And the marketing should be the same.

My beef with Zig boils down to those things not happening.

Well, okay, I do have another beef with Zig: it sets the wrong tone. Programming languages, once used, set the tone for the industry, and I think Zig sets the wrong tone. So does Rust for that matter. But I can talk about that more in my blog post.

I’m looking forward to reading your blog post, although in all frankness you should consider doing some introspection before diving into it.

I have done introspection. I’ve learned where the function coloring problem actually is in Zig, and I’ve adopted new language to not come off in the wrong way. And I’ll do that in my blog post.

2. 3

For me, the coloring problem describes both the static and runtime semantics. Does Zig handle the case where a function called with async enters some random syscall or grabs a mutex that blocks for a long time and isn’t explicitly handled by whatever the runtime system is or does that end up blocking the execution of other async tasks?

The reason why the runtime semantics matter to me when it comes to concurrency is because if you can block threads, then you implicitly always have a bounded semaphore (your threadpool) that you have to think about at all times or your theoretically correct concurrency algorithm can actually deadlock. That detail is unfortunately leaked.

1. 6

If you grab a standard library mutex in evented I/O mode then it interacts with the event loop, suspending the async function rather than e.g. a futex() syscall. The same code works in both contexts:

mutex.lock();
defer mutex.unlock();


There are no function colors here; it will do the correct thing in evented I/O mode and blocking I/O mode. The person who authored the zig package using a mutex does not have to be aware of the intent of the application code.

This is what the Zig language supports. Let me check the status of this feature in the standard library… looks like it’s implemented for file system reads/writes but it’s still todo for mutexes, sleep, and other kinds of I/O. This is all still quite experimental. If you’re looking for a reason to not use Zig, it’s that - not being stable yet. But you can’t say that Zig has the same async function coloring problem as other languages since it’s doing something radically different.

1. 4

Thanks for the explanation and standard library status information.

I think the ability to make a function async at call time rather than at definition time is the best idea in Go’s concurrency design, and so, bringing something like that to a language with a much smaller runtime and no garbage collector is exciting. I look forward to seeing how this, and all of the other interesting ideas in Zig, comes together.

(p.s. thanks so much for zig cc)

1. 15

Note that Rust does this automatically for you since Rust 1.18, released in 2017. By coincidence, the example case used, (u8, u16, u8), is exactly same in the Rust release note.

1. 2

Does anyone know if Go could do this? And was it ever considered?

1. 4

As far as I know, there is nothing in the spec that guarantees struct order layout, and the only way to really observe it is to use the unsafe package which is not covered by the Go 1 compatibility promise. So technically it could.

That said, changing it now would break too much code that uses structs with cgo/assembly/syscalls, so I doubt it will happen any time soon. If it ever does, I’d expect it will come with a similar annotation to Rust’s #[repr(C)], at the very least.

Here’s an issue where these things have been considered: https://github.com/golang/go/issues/36606

2. 1

That seems rather brutal for binary compatibility or when interacting with C. I assume it can be turned off on a struct by struct basis?

1. 12

Yes, #[repr(C)], it’s right there in the linked release notes :-)

1. 3

And all the details (and other representations) are documented: https://doc.rust-lang.org/reference/type-layout.html#representations

1. 4

Man, there’s so many people that don’t read anything.

I currently fill the role of IT and it’s absolutely bonkers the stuff people get stuck on. It’s so basic, and the screen is telling you what’s wrong, but people aren’t even parsing it.

1. 4

The irony here is that the article claims that people actually do spend a significant amount of their time reading compiler errors.

1. 3

The bonus irony is that it’s discussing a paper that, guess what, no one will read! Because it links to a paywalled version of it instead of the author’s copy available freely on their web site: https://people.engr.ncsu.edu/ermurph3/papers/icse17.pdf

1. 10

Minimal Version Selection itself is fine, in my experience neither better nor worse than what I guess is the de facto standard of… Maximal {Patch, Major} Version Selection? It biases for churn reduction during upgrades, maybe, which at least for me doesn’t have much net impact on anything.

But a lot of the ancillary behaviors and decisions that come with MVS as expressed in Go modules are enormously frustrating — I would go so far as to say fundamentally incompatible with the vast majority of dependency management workflows used by real human beings. As an example, in modules, it is effectively not possible to constrain the minor or patch versions of a dependency. You express module v1.2.3 but if another dependency in your dep graph wants module v1.4.4 you’ll get v1.4.4. There is no way to express v1.2.x or v1.2.3+. You can abuse the require directive to lock to a single specific version, but that constraint isn’t transitive. And module authors can retract specific versions, but that power is not granted to consumers.

edit: for more of my thoughts on this exciting topic see Semantic Import Versioning is unsound

1. 5

I would go so far as to say fundamentally incompatible with the vast majority of dependency management workflows used by real human beings.

I hope that this post with the experience reports of many real human beings claiming that it works well for them will help you reconsider this opinion. Perhaps it’s not as much of a vast majority as you think it is.

1. 3

None of the experience reports really speak to the points I raise. Maybe that’s your point.

2. 2

The position of the Go team is (was?) that packages with the same import path must be backwards compatible. I guess their point is that 1.4.4 should be compatible with 1.2.3, but that’s not how the rest of the software world has worked in the past decade. It’s a big if, but if all the go programmers agree, it works.

1. 6

That’s not (only?) the position of the Go team, it’s a requirement of semantic versioning. People fuck it up all the time but that’s the definition. One way to look at the problem with modules is that they assume nobody will fuck it up. Modules assumes all software authors treat major version API compatibility as a sacrosanct and inviolable property, and therefore provides no affordances to help software consumers deal the fuckups that inevitably occur. If one of your dependencies broke an API in a patch version bump, well, use a better different dependency, obviously!

Ivory Tower stuff, more or less. And in many ways Go is absolutely an Ivory Tower language! It tells you the right way to do things and if you don’t agree then too bad. But the difference is that if you don’t like the “I know better than you” decisions that Go the language made, you can pick another language. But if you don’t like the “I know better than you” decisions that modules made, you can’t pick another package management tool. Modules has a monopoly on the space. That means it doesn’t have the right to make the same kind of normative assumptions and assertions that the language itself makes. It has to meet users where they are. But the authors don’t understand this.

1. 3

Semantic versioning is, unfortunately, incompatible with graceful deprecation. Consider the following example:

• A is the version you’re using.
• B introduces a new API and deprecates an old one
• C removes the deprecated API.

In SemVer, these thing would be numbered something like 1.0, 1.1, 2.0. The jump from 1.1 to 2.0 is a breaking change because the old API went away at that point. If you paid attention to deprecation warnings when you upgraded to 1.1 and fixed them then the 1.1 -> 2.0 transition is not a breaking change though and SemVer has no way of expressing this with a single exported version and this leads to complex constraints on the import version (in this three-version case, the requirement is >=1.1, <3.0). A lot of these things would be simpler if APIs, not packages, were versioned and the package advertised the range of the API that it advertised. Then you’d see:

• A is the 1.0 API
• B supports the 1.0 API and the 2.0 API
• C supports the 2.0 API

As a consumer, I’d just specify that I need the 1.0 API until I’ve migrated my code and then that I need the 2.0 API.

1. 1

One way to look at the problem with modules is that they assume nobody will fuck it up.

So does everyone else in the world who builds with ^version and no lock file, except for them, the breakage happened when the author of their dependency published a new version rather than when they themselves performed a conscious action to alter dependencies in some way.

1. 3

Yeah but in most package management systems you can pin specific versions.

2. 1

[…] and therefore provides no affordances to help software consumers […]

If one of your dependencies broke an API in a patch version bump, well, use a better different dependency, obviously!

You can use an exclude directive to remove the breaking version from consideration. If upstream delays fixing the breaking change, you can fork the last good version and use a replace directive to substitute your own patched module in place of the old module.

It’s hard to imagine this hypothetical happening, however (and please correct me if I’m wrong). MVS selects the maximum of minimum required versions. If some module called “broken” patches and the new version breaks your own software, there is no way for that update to propagate to your software unless both 1) a different dependency of your module called “dep” decides to start requiring the new version of “broken” and 2) you update your dependencies to require the updated version of “dep”. (1) Implies that “dep” truly requires the patch (that broke your code), and (2) implies that you truly require the new features in module “dep”. By transitivity… there is no conceivable way to fix the problem without patching it yourself and replacing.

There’s actually a whole paragraph on this topic of “high fidelity” in Russ Cox’s original blog post about the version selection algorithm.

1. 2

You can use an exclude directive to remove the breaking version from consideration.

I meant “broke an API” in the “API compatibility” sense, not in the “wrote a bug” sense. That kind of broken carries forward.

2. 1

So your article states: “It’s a contract with its consumers, understood by default to be supported and maintained indefinitely.”

I don’t think this follows from anything you have written or anything I have read about SIV. The way SIV works sounds to me like if you want to deprecate features from your library you should provide a reasonable deprecation policy which includes a time period for which you will provide bug-fixes for the old major version and a time period for which you will backport security-fixes for the old major version at which point you stop supporting that version since you’ve done the best you could to get old users moved to a new version. This to me seems like a lot of major software (not written in the past 5 years) basically works.

“At Google, package consumers expect their dependencies to be automatically updated with e.g. security fixes, or updated to work with new e.g. infrastructural requirements, without their active intervention.”

I expect this to happen on my linux desktop too. I don’t see a difference in expectations there.

“Stability is so highly valued, in fact, that package authors are expected to proactively audit and PR their consumers’ code to prevent any observable change in behavior.”

I think if you feel like writing a library/module/dependency then this is the kind of mindset you are obliged to take. Anything short of this kind of approach to writing a library/module/dependency is irresponsible and makes you unqualified to write libraries, modules or dependencies. This, to me, seems to have been the mindset for a long time in software until languages came along with language package managers and version pinning in the last few years. And I don’t think that this has been a positive change for anyone involved.

“As I understand it, even issues caused by a dependency upgrade are considered the fault of the dependency, for inadequate risk analysis, rather than the fault of the consumer, for inadequate testing before upgrading to a new version.”

And I agree with this wholeheartedly, in fact this is the mindset used by users of linux distributions and distribution maintainers.

“Modules’ extraordinary bias toward consumer stability may be ideal for the software ecosystem within Google, but it’s inapproriate for software ecosystems in general.”

I think it’s not inappropriate, it’s totally appropriate. I just think that modern software ecosystems have gotten lazy because it’s easier than doing it right (which is what google seems to be advocating for a return to).

I should say, I don’t disagree with the point you make that intrinsically linking the major version to the package name is a good idea. Go should definitely NOT do that for the reasons you outlined. It would also be an easy indicator for me when picking a project to use in my codebase: Is the codebase on major version 156? Yes? Then I probably don’t want to touch it because the developers are not taking the responsibility of maintaining a dependency very seriously.

People who want to play in the sandpit of version pinning and ridiculously high major version numbers because they think software development is an area where no thought or effort should be put into backwards compatibility should be welcome to use whatever language they want to without being artificially limited.

Now, conversely, I would say, there seems like an obvious solution to this problem too. If you want to use semver while keeping to the golang rules, why not just encode the real semver version within the go version: “0.015600310001”. Sure, it’s not exactly so human readable, but it seems to encode the right information and you just need to pretty print it.

“Additionally, policies that bias for consumer stability rely on a set of structural assumptions that may exist in a closed system like Google, but simply don’t exist in an open software ecosystem in general.”

I will take things back to the world of linux distributions where these policies actually do seem to exist.

“A bias towards consumers necessarily implies some kind of bias against authors.”

Yes, and this is a good thing. Being an author of a dependency is a very big responsibility and a lot of modern build systems and language package managers fail to make that very clear

“But API compatibility isn’t and can’t be precisely defined, nor can it even be discovered, in the P=NP sense.”

This is true, but in reality there’s a pretty big gulf between best effort approaches to API compatibility (see: linux kernel) and zero effort approaches to API compatibility (see: a lot of modern projects in modern languages).

“Consequently, SIV’s model of versioning is precisely backwards.”

Actually it would be semver’s fault not SIV’s surely.

“Finally, this bias simply doesn’t reflect the reality of software development in the large. Package authors increment major versions as necessary, consumers update their version pins accordingly, and everyone has an intuitive understanding of the implications, their risk, and how to manage that risk. The notion that substantial version upgrades should be trivial or even automated by tooling is unheard of.”

Maybe today this is the case, but I am pretty sure this is only a recent development. Google isn’t asking you to do something new, google is asking you to do something old.

“Modules and SIV represent a normative argument: that, at least to some degree, we’re all doing it wrong, and that we should change our behavior.”

You’re all doing it wrong and you should change your behavior.

“The only explicit benefit to users is that they can have different versions of the “same” module in their compilation unit.”

You can achieve this without SIV, SIV to me actually seems like just a neat hack to avoid having to achieve this without SIV.

In any case, I think I’ve made my point mostly and at this point I would be repeating myself.

I wonder what you think.

1. 1

People who want to play in the sandpit of version pinning and ridiculously high major version numbers because they think software development…

…is a means and not an end are the norm, not the exception. And the fact that people work this way is absolutely not because they’re lazy, it’s because it’s the rational choice given their conditions, the things they’re (correctly!) trying to optimize for, and the (correct!) risk analysis they’ve done on all the variables at play.

I appreciate your stance but it reflects an Ivory Tower approach to software development workflows (forgive the term) which is generally both infeasible and actually incorrect in the world of market-driven organizations. That’s the context I speak from and the unambiguous position I’ve come to after close to 20 years’ experience in the space, working myself in a wide spectrum of companies and consulting in exactly these topics for ~100 orgs at this point.

Google has to work this way because their codebase is so pathological they have no other choice. Many small orgs, or orgs decoupled from typical market dynamics, can work this way because they have the wiggle room, so to speak. They are the exceptions.

1. 1

the things they’re (correctly!) trying to optimize for, and the (correct!) risk analysis they’ve done on all the variables at play

Disagree.

At least don’t call the majority of software developers “engineers” if you’re going to go this way.

The fact that this is considered an engineering discipline with such low standards is really an insult to actual engineering disciplines. I can totally see how certain things don’t need to be that rigorous, but really, seriously, what is happening is not par for the course.

The fact that everyone including end users has become used to the pathological complacency of modern software development is really seriously ridiculous and not an excuse to continue down this path. I would go so far to say that it’s basically unethical to keep pretending like nothing matters more than making something which only just barely works within some not even that optimal constraints for the least amount of money. It’s a race to the bottom, and it won’t end well. It’s certainly not sustainable.

I appreciate your stance but it reflects an Ivory Tower approach to software development workflows (forgive the term) which is generally both infeasible and actually incorrect in the world of market-driven organizations. That’s the context I speak from and the unambiguous position I’ve come to after close to 20 years’ experience in the space, working myself in a wide spectrum of companies and consulting in exactly these topics for ~100 orgs at this point.

It’s incorrect in the world of market-driver organizations only because there’s a massive gap between the technical ability of the consumers of these technologies and the producers, so much so that it’s infeasible to expect a consumer of these technologies to be able to see them for the trash they are. But I think that this is not “correct” it’s just “exploitative”. Exploitative of the lack of technical skill and understanding of the average consumer of these technologies.

I don’t think the “correct” response is “do it because everyone else is”. It certainly seems unethical to me.

That being said, you are talking about this from a business point of view not an open source point of view. At least until open source got hijacked by big companies, it used to be about small scale development by dedicated developers interested in making things good for the sake of being good and not for the sake of a bottom line. This is why for the most part my linux system can “just update” and continue working, because dedicated volunteers ensure it works.

Certainly I don’t expect companies to care about this kind of thing. But if you’re talking about solely the core open source world, “it’s infeasible in this market” isn’t really an argument.

Google has to work this way because their codebase is so pathological they have no other choice. Many small orgs, or orgs decoupled from typical market dynamics, can work this way because they have the wiggle room, so to speak. They are the exceptions.

I honestly don’t like or care about google or know how they work internally. I also don’t like go’s absolutely inane idea that it’s sensible to “depend” on a github hosted module and download it during a build. There’s lots of things wrong with google and go but I think that this versioning approach has been a breath of fresh air which suggests that maybe just maybe things may be changing for the better. I would never have imagined google (one of the worst companies for racing to the bottom) to be the company to propose this idea but it’s not a bad idea.

1. 2

The fact that everyone including end users has become used to the pathological complacency of modern software development is…

…reality. I appreciate your stance, but it’s unrealistic.

1. 26

This post talks about the downsides but does not acknowledge the underlying problem that is being corrected, namely that “go get” today does two completely different things: (1) add or update dependencies of a program, and (2) install binaries.

Nearly all the time, you only mean to do one of these things, not both. If you get in the habit of using “go get” to update your dependencies, you might run a command like “go get path/…” and be surprised when it installs “path/internal/ls” to your $HOME/bin directory. We have had people claim that’s a security issue, and whether that’s true or not, it’s certainly a usability issue. In this example, you only want the dependency change and are getting the install as an unwanted side-effect. After the transition, the rule will be simple: “go get only changes your dependency versions”. On the other hand, we have commands in READMEs that say things like “go get -u rsc.io/2fa”. Today that means to fetch the latest version of rsc.io/2fa, then upgrade all its dependencies, recursively, producing a build configuration that may never have been tested, and then build the binary and install it. In this example, you only want the install and are getting the “update dependencies” as an unwanted side effect. If instead you use “go install rsc.io/2fa@latest” (which works today in Go 1.16), you get the latest tagged version of rsc.io/2fa, using exactly the dependency versions it declares and was tested with, which is what you really want when you are trying to install a binary. We didn’t just make a gratuitous change to the way the command is spelled: we fixed the semantics too. The double meaning of “go get” was introduced in GOPATH mode many years ago. It is my fault, and all I can say is that it seemed like a good idea at the time, especially in the essentially unversioned context of GOPATH-based development. But it’s a mistake, and it is a big enough mistake to correct for the next million users of Go. The post claims this is an abrupt deprecation schedule, but the change was announced in the Go 1.16 release notes and will take effect in Go 1.18. Once Go 1.17 is released, the two supported versions of Go will be Go 1.16 and Go 1.17, meaning all supported Go versions will know what “go install path@version” means, making it time to start nudging people over to using it with prints from the go command. This two-cycle deprecation schedule, where we make sure that we don’t encourage transition work until all supported versions of Go (the last two) support the new behavior, is our standard procedure for this. I suspect that Go 1.18 will keep the warning print, so you’d actually get two releases of warning prints as well. We try very hard not to make breaking changes in tool behavior. This one is necessary, and we’ve done everything we can to make sure it is a smooth transition. The new behavior should be obvious - the tool tells you what is going on and what you need to do differently - and easy to learn. 1. 2 I know it’s too late to change anything now, but in retrospect, would it have been better to leave go get in the old GOPATH world and introduce the module version of go get under a different name? 1. 1 A quick note: “go get” does three things today, not two. The third is that it clones the version control repository for some Go package or module (into$GOPATH/src). In the process of doing this, it reaches through vanity import paths and works out the correct VCS system and clone path to use. As far as I know, in the non-GOPATH world there is currently no way to do this yourself, either for programs or for other packages. It would be nice at the very least to have a command that resolved through vanity import paths to tell you the underlying thing to clone, and with what.

(Cloning this way also locks you to the current location of the repository. One advantage of vanity import paths is that people can change them, and have, and you will automatically pick up that change.)

The other thing that is not possible in the non-GOPATH world is to install the very latest VCS version of a program (at least in any convenient way). As noted, the @latest syntax is the latest tagged version, not the latest VCS version. This means that if you want to have someone test your very latest development version, you need to either tag it or tell them to clone the repo and build it themselves; you can’t give them a simple, generic ‘go install …@…’ command to do it. (You can give them a VCS version, but there are various problems with this.)

1. 1

The other thing that is not possible in the non-GOPATH world is to install the very latest VCS version of a program (at least in any convenient way).

You can use “go install …@branchname” to get the latest commit for whatever branch.

1. 1

Oh oops, that’s my mistake. I missed that in the Version queries section of the specification.

(If reaching for the specification strikes people as odd, I can only plead that I couldn’t readily find documentation of this in the “go” help and “go help modules” did point me to the specification.)

1. 9

Does it mean that the promise to not break API doesn’t extend to CLI? https://golang.org/doc/go1compat#expectations

Keeping things stable is what makes the success of Go.

1. 7

Indeed, the final section of that page you linked says that tools are exempt. I can’t think of any previous examples that broke the tooling as much as this will, though.

1. 6

It seems short-sighted.

go get is invoked by a ton of other programs. For example, VSCode plugins love to run it to install missing developer tools. It’s effectively an API, but that’s going through the CLI instead of accessing a Go API directly.

1. 8

There’s been loads of subtle breakage and changes in the Go tooling ever since modules were introduced. It rather sucks, but it’s better than the alternative of keeping it full of all sorts of legacy behaviour where commands do something different depending on the environment and/or which directory you’re in, which is super-confusing.

It’s easy to say it’s “short-sighted”, but all options suck – they just suck in different ways.

2. 0

I’ve been listening to the gotime podcast. Go get is a feature to people new to programming – not a bug.

1. 6

I wish this included some explanation of why these are useful or explanations of why they are. Like, why use zerolog over logrus/zap/whatever? Why use gorilla/mux over gin or go-chi or the standard library? Why have that overly-complex unique code thing to determine where a logged error is thrown instead of making sure your errors have stack traces?

1. 4

I’m not the author of the post, but I can try to answer some questions:

Like, why use zerolog over logrus/zap/whatever?

zerolog and zap is what the author of logrus admit he would write if he wrote logrus/v2. zerolog claims to be a “better” version of zap (there are claims of performance improvement on other people’s computers)

Why use gorilla/mux over gin or go-chi or the standard library?

gorilla/mux is much lightweight as opposed to gin which has renderers, etc… Also, there are claims of very high performance on other people’s computers. With gorlla/mux you can plug your own json de-/serialization library which is faster than gin’s.

Why have that overly-complex unique code thing to determine where a logged error is thrown instead of making sure your errors have stack traces?

¯\(ツ)/¯ No idea… The unique code approach feels very hacky and brittle to me.

1. 3

I would even suggest “httprouter” as a more lightweight alternative to gorilla/mux.

1. 2

Most “higher level” Go routers are probably based on it anyways.

2. 1

We use unique logging codes for our logs, and it makes it easier to locate a particular statement in either the logs, or the source code. Now, we generate the unique code by hand, but it’s not a horrendous issue for us (we have a separate document that records each logging statement).

3. 1

I’d be curious to know the best practices for annotating errors with stack traces. Any good library suggestions?

1. 3

github.com/pkg/errors has some nice helpers, as described in its documentation. The issue is that it requires changing your code to using errors.Errorf() instead of fmt.Errorf().

1. 3

Also, errors.Wrapf is very useful when handling errors from packages which don’t use errors.

1. 2

You can wrap errors using fmt.Errorf() and the %w verb, and then later deal with them using errors.Is() or errors.As() and friends.

1. 1

That doesn’t give you stack traces though, correct?

2. 2

YES. This is probably my biggest issue with supporting production Go code. Errors are still too hard to locate.

1. 1

I personally use this to manage all that. If you use the errors.Wrap() function to wrap errors, it’ll add stack traces. And if you want to make a new error, the errors.New() function will also add a stack track where it was called. Then when you’re logging the error make sure it’s being logged in a way that will not just print the message. Most loggers should have a way to do that (I know zerolog and logrus do).

1. 1

Shameless plug for my errors library: https://pkg.go.dev/github.com/zeebo/errs/v2

• captures stack traces, but only once even if wrapped multiple times
• plays nicely with standard library errors package with errors.Is and errors.As
• has a “tag” feature to let you associate and query tags with errors
• helper to keep track of groups of errors
1. 3

I am confused how the presented scheme is anything close to tracing. The first step is

The plaintext that is to be traced is submitted along with RF, NF and context.

But NF is a 256bit random nonce that no one other than the sender and recipient have access to. You may be able to guess a plaintext, but there’s no way you can guess that.

Additionally, it seems to me that if you have access to an oracle that can say if a given ciphertext is equal to some plaintext, you have broken ciphertext indistinguishability, a property that is very important to confidentiality (“Indistinguishability is an important property for maintaining the confidentiality of encrypted communications.”)

1. 1

There would be a step where the reveal of this nonce would be compelled, similarly to how message franking implements such a step in its current form. The idea is that you can just substitute the rationale for this step from “abuse reporting” to “message tracing”.

1. 2

How is compelling the reveal of the nonce any different from compelling the reveal of the plaintext? They’re stored next to each other and the only parties that have the nonce are the same parties that have the plaintext. The difference between “abuse reporting” and “message tracing” is which party is performing the action, and that makes all the difference.

1. 2

As far as I understand, the nonce serves to validate the initial HMAC, which serves as a pre-commitment to the authenticity of the message within its original context.

1. 8

This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

So we rewrote gRPC and migrated our live network. DRPC is a drop-in replacement that handles everything we needed from gRPC

and

It’s worth pointing out that DRPC is not the same protocol as gRPC, and DRPC clients cannot speak to gRPC servers and vice versa.

How it is a drop in replacement if both client and server needs to be changed

1. 15

This is interesting and I am prolly going to use it in our services. Could you talk more about the underlying protocol?

The wire format used is defined here.

Logically, a sequence of Packets are sent back and forth over the wire. Each Packet has a enumerated kind, a message id (to order messages within a stream), a stream id (to identify which stream), and a payload. To bound the payload size, Packets are split into Frames which are marshaled and sent.

A marshaled Frames has a single header byte, the varint encoded stream and message ids, a varint encoded length, and that many bytes of payload. The header byte contains the kind, if it is the last frame for a Packet, and a control bit reserved for future use (the implementation currently ignores any frame with that bit set).

Because there’s no multiplexing at this layer, the reader can assume the Frames come in contiguously with non-decreasing ids, limiting the amount of memory and buffer space required to a single Packet. The Frame writing code is as simple as appending some varints, and the reading code is about 20 lines, neither of which have any code paths that can panic.

How it is a drop in replacement if both client and server needs to be changed

There’s a drpcmigrate package that has some helpers to let you serve both DRPC and gRPC clients on the same port by having the DRPC clients send a header that does not collide with anything. You first migrate the servers to do both, and can then migrate the clients.

The drop in replacement part refers to the generated code being source code API compatible in the sense that you can register any gRPC server implementation with the DRPC libraries with no changes required. The semantics of the streams and stuff are designed to match.

Sorry if this was unclear. There’s multiple dimensions of compatibility and it’s hard to clearly talk about them.

1. 8

(not the one who asked). Thank you for the comment. Do you plan to document the protocol somewhere (other than code)? It seems to me a RPC protocol needs a healthy ecosystem of implementations in many languages to be viable long term :-)

(edit: I mean document it cleanly, with request/reply behavior, and talk about the various transports that are supported.)

1. 6

You are absolutely right that having good documentation is essential, and I’ve created a github issue to start to keep track of the most important bits I can think of off the top of my head. To be honest, I’ve always struggled a bit with documentation, but I’m trying to work on it. In fact, the current CI system will fail if any exported symbols do not have a doc string. :) Thanks for bringing this up.

2. 2

Thanks for the explanation! Are there any plans for writing Python client/server?

1. 4

I do hope to get more languages supported. I created an issue to add Python support. I don’t currently deal with any Python code, so help will probably be needed to make sure it “fits” correctly with existing codebases.

1. 1

I have started looking into the drpcwire code, I will wait for the documentation. Meanwhile, is there any place, like IRC, Discord where drpc/storj developers hang out? The Python issue is tagged help-wanted, so, can I be of any help?

1. 3

The worst part of gRPC is its crappy protobuf notation. This tool doesn’t address anything about that.

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

1. 24

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

Sockpuppeting on Lobsters is heavily frowned upon.

1. 13

promote … with a throwaway account.

That. Don’t use throwaway accounts and shill projects.

1. 4

Do you mean the service definitions being .proto files? If so, DRPC has a very modular design, and all of the functionality is implemented by referring to only the interfaces defined in the main module. Because of that, you can effectively manually write out the code generated glue, or generate it in some other way. Here’s an example of that: https://play.golang.org/p/JQcS2A9S8QX

1. 1

I’m wondering why I got banned when I tried to promote another RPC tool with a throwaway account.

Which tool were you trying to promote?

1. 6

He got banned again so we may never know

1. 13

Who thinks it’s a good idea to tell everybody they’re a spammer?

1. 2

Growth hackers.

2. 4

1. 4

You can check the moderation log for timestamp 2021-04-17 12:40 -0500

1. 1

click through to their username, which includes a ban reason.

1. 8

I haven’t read rsc’s critique of the article yet, but while reading the tables I’m thinking, why are they comparing different CockroachDB versions? Surely, if the point was to find bloat over time due to changes in Go, one would want to look at the same program across Go versions?

1. 25

I took the time to do my best to compile an older and a newer version of the code with many different Go versions and here’s what I got:

        v1.0          v20.2.0
1.8     58,099,688    n/a
1.9     57,897,616    314,191,032
1.10    57,722,520    313,669,616
1.11    48,961,712    233,170,304
1.12    52,440,168    236,192,600
1.13    50,844,048    214,373,144
1.14    50,527,320    212,699,656
1.15    47,910,360    201,391,416
1.16    47,317,624    205,018,136


I couldn’t compile v20.2.0 on go1.8 because of the large use of type aliases that would take a very long time to remove. I already had to make some other smaller changes or backports of standard library code to have it compile, and type aliases crossed the line for me.

The overall binary size has decreased by 20% to 33% which means that all of this “dark code” is just their inability to understand their binary, and all of the growth is due to their code base.

I noticed in the process of getting these numbers that an awful lot of C/C++ compiling and javascript webpacking was going on. It turns out, there’s about 7M of javascript embedded in the Go binary, as well as these archives linked in:

librocksdb.a       69M
libcryptopp.a      28M
libprotobuf.a      22M
libgssapi_krb5.a   3.9M
ibroachccl.a       2.3M
libgeos.so         2.3M
libproj.a          2.2M
libsnappy.a        168K


I have no idea how much they make up of the resulting binary, but I’d be surprised if it was negligible.

1. 8

Thank you!

2. 2

Most likely the current version doesn’t build with old Go. Probably there exists an older version that builds across all Go versions, but then the numbers wouldn’t be as “impressive” :-P

1. 2

Yes this is the usual reason. When I benchmarked how Rust compilation speed changed over the years, it was tricky to find a library that compiled down to Rust 1.0. I ended up using an old version of library to parse URL.

1. 53

1. A minor issue, but the Go program has a data race. The i loop variable is captured in the spawned goroutines while it is mutated by the outer loop. Fixing it doesn’t change the results of the program. It’s also immediately caught when built with the -race flag. Rust being data race free by construction is a super power.

2. Changing the amount of spawned goroutines/threads to 50,000 has a noticeable difference between the two on my machine. The Go program still completes in a timely fashion with ~5x the rss and no negative impacts on the rest of my system:

real 11.25s
user 41.49s
sys  0.70s


but the Rust version immediately causes my system to come to a crawl. UIs stop responding, audio starts stuttering, and eventually it crashes:

thread 'main' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', src/libcore/result.rs:1189:5
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace.
Command exited with non-zero status 101
real 16.39s
user 23.76s
sys  98.50s


So while it may not be significantly lighter than threads as far as this one measure is concerned, there is definitely some resource that threads are taking up that goroutines are not. I checked on some production monitoring systems to see how many active goroutines they had, and they were sitting comfortably in the 20-30k area. I wonder if they were using threads how they would fare.

1. 8

Rust being data race free by construction is a super power.

Well, it also necessarily prohibits entire classes of useful and productive patterns… it was a design decision, with pros and cons, but certainly not strictly better.

1. 4

Strong plus +1. I wish, for application development, it was possible to remove lifetimes, borrow-checker and manual memory management (which are not that useful for this domain) and to keep only fearless concurrency (which is useful even (more so?) in high-level languages). Alas, it seems that Rust’s thread-safety banana is tightly attached to the rest of the jungle.

1. 7

FWIW. Ponylang has exactly this:

• no lifetimes, data is either known statically via simple escape analysis or traced at runtime
• no borrow checker, although it does have a stricter reference capability system
• no manual memory management: uses a deterministically invoked (excluding shared, send data) Mark-and-no-sweep GC
• has the equivalent Send and Sync traits in its reference capability system which provide the same static guarantees

As with everything though it has its own trade offs; Whether that be in its ref-cap system, lack of explicit control in the system, etc.

2. 4

To prevent that Rust error, I believe you need to change vm.max_map_count: https://github.com/rust-lang/rust/issues/78497#issuecomment-730055721

1. 4

That seems to me like the kind of thing Rust should do to some extent on its own!

2. 3

I’d be interested to see how many threads ended up scheduled with the Go example. My guess is not many al all: IIRC the compiler recognizes those sleeps as a safe place to yield control from the goroutines. So I suspect you end up packing them densely on to threads.

1. 25

I think threads are better than a lot of people think, but in this case the measurement strikes me as a bit naive. Go very likely consumes more memory than strictly needed because of the GC overhead. The actual memory occupied by goroutines could be less than half the memory used in total, but here we only see goroutine overhead + GC overhead without a way of separating them.

1. 8

What overhead specifically? Stack maps for GC?

The benchmark doesn’t seem to have many heap objects, and I thought Go used a conservative/imprecise GC anyway so objects don’t have GC space overhead (mark bits, etc.). Although I think the GC changed many times and I haven’t followed all the changes.

1. 7

The GC has been fully precise since Go 1.4.

1. 6

OK that’s what I suspected, but as far as I can tell it’s a moot point because the benchmark doesn’t have enough GC’d objects for it to matter. Happy to be corrected.

2. 5

I think when people speak of GC overhead they’re not talking about the per-object overhead of tracking data. Rather, the total amount of memory space one needs to set aside to cover both memory actually in use, and memory no longer in use but not yet collected: garbage. This can often be quite a lot larger than the working set, especially if there is a lot of churn.

1. 3

Where does that exist in the benchmark?

I see that each goroutine calculates a hash and calls time.Sleep(). I don’t see any garbage (or at least not enough to know what the grandparent comment is talking about)

1. 1

The space can be reserved for future collections anyway. It’s well known that GCs require more space than strictly needed for alive objects.

1. 8

This was an interesting article, it breaks down the issues with net.IP well, and describes the path to the current solution well.

But.

This isn’t a difficult problem. Don’t waste a ton of space, don’t allocate everywhere, make it possible to make the type a key in the language’s standard map implementation. In C++, this would’ve been easy. In Rust, this would’ve been easy. In C, this would’ve been easy (assuming you’re using some kind of halfway decent map abstraction). It doesn’t speak well of Go’s aspiration to be a systems programming language that doing this easy task in Go requires a bunch of ugly hacks and a separate package to make a string deduplicator which uses uintptrs to fool the garbage collector and relies on finalizers to clean up. I can’t help but think that this would’ve been a very straightforward problem to solve in Rust with traits or C++ with operator overloading or even Java with its Comparable generic interface.

That’s not to say that the resulting netaddr.IP type is bad, it seems like basically the best possible implementation in Go. But there are clearly some severe limitations in the Go language to make it necessary.

1. 11

Almost all of the complexity that happened here is related to the ipv6 zone string combined with fitting the value in 24 bytes. Given that a pointer is 8 bytes and an ipv6 address is 16 bytes, you must use only a single pointer for the zone. Then, having amortized zero allocations with no space leaks for the zone portion, some form of interning with automatic cleanup is required.

If this is as easy as you claim in C/C++/Rust/whatever real systems language you want, can you provide a code snippet implementing it? I’d be happy to audit to see if it does meet the same (or better!) constraints.

1. 6

Here’s a C++ version: https://godbolt.org/z/E3WGPb - see the bottom for a usage example.

Now, C++ is a terrible language in many ways. It makes everything look super complicated, and there’s a lot of seemingly unnecessary code there, but almost all of that stems from having to make my own RAII type, which includes writing the default constructor, the move constructor, the copy constructor, the destructor, the move operator= and the copy operator=. That complexity is just par for the course in C++.

One advantage of the netaddr.IP type is that it doesn’t allocate for every zone, just for every new zone, thanks to the “intern” system. My code will allocate space for the zone for every IPv6 address with a zone. One could definitely implement a “zone cache” system for my IPZone type though, maybe using a shared_ptr instead of a raw pointer for refcounting. One would have to look at usage patterns to see whether the extra complexity and potential memory/CPU overhead would be worth it or if zones are so infrequently used that it doesn’t matter. At least you have the choice in C++ though (and it wouldn’t rely on finalizers and fooling the GC).

1. 7

They also had the choice to just make a copy of every string when parsing and avoid all of the “ugly hacks”. Additionally, a shared_ptr is 16 bytes, so you’d have to figure out some other way to pack that in to the IPAddress without allocations. So far, I don’t think you’ve created an equivalent type without any “ugly hacks”. Would you like to try again?

1. 6

I don’t think they had the choice to just copy the zone strings? My reading of the article was that the intern system was 100% a result of the constraint that A) IP addresses with no zone should be no bigger than 24 bytes and B) it should be possible to use IP addresses as keys. I didn’t see concern over the memory usage of an IP address’s zone string. Whether that’s important or not depends on whether zones are used frequently or almost never.

It’s obviously hard to write a type when the requirements are hypothetical and there’s no data. But here’s a version with a zone string cache: https://godbolt.org/z/P9MWvf. Here, the zone is a uint64_t on the IP address, where 0 represents an IPv4 address, 1 represents an IPv6 address with no zone, and any other number refers to some refcounted zone kept in that IPZoneCache class. This is the “zone mapping table” solution mentioned in the article, but it works properly because the IPAddress class’s destructor decrements the reference count.

1. 7

I don’t think they had the choice to just copy the zone strings? My reading of the article was that the intern system was 100% a result of the constraint that A) IP addresses with no zone should be no bigger than 24 bytes and B) it should be possible to use IP addresses as keys.

Indeed, interning is required by the 24 byte limit. That Interning avoids copies seems to be a secondary benefit meeting the “allocation free” goal. It was a mistake to imply that copying would allow a 24 byte representation and that interning was only to reduce allocations.

That said, your first solution gets away with avoiding interning because it uses C style (null terminated) strings so the reference only takes up a single pointer. Somehow, I don’t think that people would be happier if Go allowed or used C style strings, though, and some might consider using them an “ugly hack”.

I didn’t see concern over the memory usage of an IP address’s zone string. Whether that’s important or not depends on whether zones are used frequently or almost never.

One of the design criteria in the article was “allocation free”.

It’s obviously hard to write a type when the requirements are hypothetical and there’s no data. But here’s a version with a zone string cache: https://godbolt.org/z/P9MWvf.

Great! From what I can tell, this does indeed solve the problem. I appreciate you taking the time to write these samples up.

I have a couple of points to make about your C++ version and some hypothetical C or Rust versions as compared to the Go version, though.

1. It took your C++ code approximately 60 lines to create the ref-counted cache for interning. Similarly, stripping comments and reducing the intern package they wrote to a similar feature set also brings it to around 60 lines. Since it’s not more code, I assume the objection is to the kind of code that is written? If so, I can see that the C++ code you provided looks very much like straightforward C++ code whereas the Go intern package is very much not. That said, the authors of the intern package often work on the Go runtime where these sorts of tricks are more common.

2. In a hypothetical C solution that mirrors your C++ solution, it would need a hash-map library (as you stated). Would you not consider it an ugly hack to have to write one of those every time? Would that push the bar for implementing it C from “easy” towards “difficult”? Why should the Go solution not be afforded the same courtesy under the (now valid) assumption that an intern library exists?

3. I’ll note that when other languages gain a library that increases the capabilities, even if that library does unsafe hacks, it’s often viewed as a positive sign that the language is powerful enough to express the concept. Why not in this case?

4. In a hypothetical Rust solution, the internal representation (I think. Please correct me if I’m wrong) can’t use the enum feature because the tag would push the size limits past 24 bytes. Assuming that’s true, would you consider it an ugly hack to hand-roll your own union type, perhaps using unsafe, to get the same data size layout?

5. All of these languages would trivially solve the problem easily and idiomatically if the size was allowed to be 32 bytes and allocations were allowed (this is take 2 in the blog post). Similarly, I think they all have to overcome significant and non-obvious challenges to hit 24 bytes with no allocations as they did.

Anyway, I want to thank you for engaging and writing some code to demonstrate the type in C++. That’s effort you don’t usually get on the internet. This conversation has caused me to update my beliefs to agree more with adding interning or weak references to the language/standard library. Hopefully my arguments have been as useful to you.

2. 4

I agree—if Go is a systems language. But I don’t think it ever was supposed to be. Or if it was, it’s (in my opinion) really bad at it. Definitely worse than even something like C#, for exactly the reasons you’re highlighting.

I think Go was more originally designed to be a much faster language than Python (or perhaps Java), specifically for Google’s needs, and thus designed to compete with those for high-performance servers. And it’s fine at that. And I’ve thought about solving this kind of issue in those languages, too, using things like array in Python for example.

So I agree Go isn’t a good systems language, but I think that was a bit of retcon. It’s a compiled high-level language that could replace Python usage at Google. It’s not competing with Rust, C, Zig, etc.

1. 3

Ok, I can buy that. IIRC, it was originally promoted as a systems language, but it seems like they’ve gone away from that branding as well. There’s a lot of value to something like “a really fast, natively compiled Python”.

But even then, this article seems to demonstrate a pretty big limitation. Something as simple as using a custom IP address type as the key in a map, ignoring everything performance-related, seems extremely difficult. How would you write an IP address struct which stores an IPv4 address or an IPv6 address with an optional zone, which can be used a the key in a map, even ignoring memory usage and performance? Because that would be easy in Python too; just implement __hash__ and __eq__.

This is a problem which isn’t just related to Go’s positioning, be it a “systems language” or a “faster python”. Near the bottom we have C, where an IP address -> whatever map is about as difficult as any other kind of map. Slightly above, we have C++ and Rust, where the built-in types let you use your IP address class/struct as a key with no performance penalty, since you stamp out a purpose-built “IP address to whatever” map using templates. Above that again, we have Java and C#, which also makes it easy, though at a performance cost due to virtual calls (because genetics aren’t templates), though maybe the JIT optimises out the virtual call, who knows. Near the top, we have Python which makes it arguably even more straightforward than Java thanks to duck typing.

Basically, unless you put Go at the very bottom of the stack alongside C, this should be an easy task regardless of where you consider Go to fit in.

1. 3

IIRC, it was originally promoted as a systems language, but it seems like they’ve gone away from that branding as well.

I believe you’re correct about how Google promoted it. I just remember looking at it, thinking “this is absolutely not a systems language; it’s Limbo (https://en.wikipedia.org/wiki/Limbo_(programming_language), but honestly kind of worse, and without the interesting runtime,” and continuing to not use it. So I’m not sure the team itself actually thought they were doing a systems language.

But even then, this article seems to demonstrate a pretty big limitation. Something as simple as using a custom IP address type as the key in a map, ignoring everything performance-related, seems extremely difficult.

I completely agree, but that’s changing the discussion to whether Go is a good language, period. And since I mostly see that devolving into a flame war, I’m just going to just say that I think you have a lot of company, and also that clearly lots of people love the language despite any warts it has.

1. 2

I completely agree, but that’s changing the discussion to whether Go is a good language, period. And since I mostly see that devolving into a flame war, I’m just going to just say that I think you have a lot of company, and also that clearly lots of people love the language despite any warts it has.

My relationship with the language is… Complicated. I often enjoy it, I use it for work, and when I just want to write a small tool (such as when I wrote a process tree viewer) it’s generally my go-to “scripting” language these days. But I hate how the module system puts URLs to random git hosting websites in my source code, there’s a lot of things I dislike about the tooling, and the inability write a datastructure which acts like the built-in datastructures and the inability to write a type which works with the built-in datastructures are both super annoying issues which none of the other languages I use have. I’m hoping Go 2 will fix some of the bigger problems, and I’m always worried about which directions the corporate management at Google will take the language or its tooling/infrastructure.

But you’re right, this is tantamount to flamewar bait so I’ll stop now.

1. 12

It’s nice to bring some nuance to the discussion: some languages and ecosystems have it worse than others.

To add some more nuance, here’s a tradeoff about the “throw it in an executor” solution that I rarely see discussed. How many threads do you create?

Well, first, you can either have it be bounded or unbounded. Unbounded seems obviously problematic because the whole point of async code is to avoid the heaviness one thread per task, and you may end up hitting that worst case.

But bounded has a less obvious issue in that it effectively becomes a semaphore. Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex) and a thread pool of size 1. If you attempt to throw both on a thread pool, and A ends up scheduled and B doesn’t, you get a deadlock.

You don’t even need dependencies between tasks, either. If you have an async task that dispatches a sync task that dispatches an async task that dispatches a sync task, and your threadpool doesn’t have enough room, you can hit it again. Switching between the worlds still comes with edge cases.

This may seem rare and it probably is, especially for threadpools of any appreciable size, but I’ve hit it in production before (on Twisted Python). It was a relief when I stopped having to think about these issues entirely.

1. 3

Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex)

Isn’t this an antipattern for async in general? Typically you’d either a) make sure to release the mutex before yielding, or b) change he interaction to “B notifies A”, right?

1. 4

Changing the interaction to “B notifies A” doesn’t fix anything because presumably A waits until it is notified, taking up a threadpool slot, making it so that B can never notify A. Additionally, it’s not always obvious when one sync task depends on another, especially if you allow your sync tasks to block on the result of an async task. In my experience, that sort of thing happens when you have to bolt the two worlds together.

1. 2

It’s a general problem. It can happen whenever you have a threadpool, no matter whether it’s sync or async.

2. 3

But bounded has a less obvious issue in that it effectively becomes a semaphore. Imaging having two sync tasks, A and B where the result of B ends up unblocking A (think mutex) and a thread pool of size 1. If you attempt to throw both on a thread pool, and A ends up scheduled and B doesn’t, you get a deadlock.

Thread-based work scheduling can, imo, be just as complicated as async scheduling. The biggest difference is that async scheduling makes you pay the cost in code complexity (through function coloring, concurrency runtimes, etc) while thread-based scheduling makes you pay the cost in operational and architectural complexity (by deciding how many thread pools to have, which tasks should run on which pools, how large each pool should be, how long we should wait to retry to grab a thread from the pool, etc, etc). While shifting the complexity to operational and architectural complexity might seem to shift the work up to operators or some dedicate operationalizing phase, in practice the context lost by lifting decisions up to this level can make tradeoffs for pools and tasks non-obvious, making it harder to make good decisions. Also, as workloads change over time, new thread pools may need to be created and these new pools necessitate rebalancing of other pools, which requires a lot of churn. Async has none of these drawbacks (though to be clear, it has its own unique drawbacks.)

1. 8

I’ve never designed a system like this or worked on a system designed like this before. I’ve never had one task depend on the value of another task while both tasks were scheduled simultaneously.

Here’s perhaps a not-unreasonable scenario: imagine a cache with an API to retrieve some value for a key if it exists and otherwise compute, store, and return it. The cache exports an async API and the callback it runs to compute the value ends up dispatching a sync task to a threadpool (maybe it’s a database query using a sync library). We want the cache to be able to be accessed from multiple threads, so it is wrapped in a sync mutex.

Now imagine that an async task tries to use the cache that is backed by a threadpool of size 1. The task disaptches a thread which acquires the sync mutex, calls to get some value (waiting however on the returned future), and assuming it doesn’t exist, the cache blocks forever because it cannot dispatch the task to produce the value. The size of 1 isn’t special: this can happen with any bounded size thread pool under enough concurrent load.

One may object to the sync mutex, but you can have the same issue if the cache is recursive in the sense that producing a value may depend on the cache populating other values. I don’t think that’s very far fetched either. Alternatively, the cache may be a library used as a component of a sync object that is expected to be used concurrently and that is the part that contains the mutex.

In my experience, the problem is surprisingly easy to accidentally introduce when you have a code base that frequently mixes async and sync code dispatching to each other. Once I started really looking for it, I found many places where it could have happened in the (admittedly very wacky) code base.

1. 3

Fair enough that is a situation that can arise. Those situations I would probably reach for either adding an expiry to my threaded tasks or separating thread pools for DB or cache threads from general application threads. (Perhaps an R/W Lock would help over a regular mutex, but I realize that’s orthogonal to the problem at hand here and probably a pedagogical simplification.) The reality is that mixing sync and async code can be pretty fraught if you’re not careful.

2. 2

I have seen similar scenarios without a user visible mutex: you get deadlocks if a thread on a bounded thread pool waits for another task scheduled on the same thread pool

Of course, there are remedies, e.g. never schedule subtasks on the same thread pool. Timeouts help but still lead to abysmall behavior under load because your threads just idle around until the timeout triggers.

1. 1

Note that you can also run async Rust functions with zero (extra) threads, by polling it on your current thread. A threadpool is not a requirement.

1. 3

Isn’t that equivalent to either a threadpool of size 1 or going back to epoll style event loops? If it’s the former, you haven’t gained anything, and if it’s the latter, you’ve thrown out the benefits of the async keyword.

1. 3

Async has always been a syntax sugar for epoll-style event loops. Number of threads has nothing to do with it, e.g. tokio can switch between single and multi-threaded execution, but so can nginx.

Async gives you higher-level composability of futures, and the ease of writing imperative-like code to build state machines.

1. 29

One thing I don’t think I’m ever going to get is how much Go does in comments. In my mind, comments are a way to communicate something to other humans, or, sometimes, a way to communicate something to some external tool (such as a documentation generator). However, in Go, you configure whether the file should be built in comments, you write C code in comments with CGo, you write shell commands for code generation in comments, and, with 1.16, you embed data into your binary using comments.

I’m all for a kind of extensible meta language; a way to have commands in the code which the compiler proper will skip, but which official tooling might interpret. Something like a #pragma. But… in my mind, the official language toolchain shouldn’t extensively parse my comments.

On the positive side though, I think what embed does is really cool. More languages should have a way to embed arbitrary data into the binary (without hacks).

1. 17

If it helps, you can consider the string //go: to be equivalent to the string #pragma and not think of it as a comment. Reusing the comment syntax has the advantage that parsers/formatters don’t have to worry about parsing anything other than comments. Much like how a shebang (#!/usr/bin/env bash) happens to be a comment in a bunch of scripting languages.

1. 17

Yea, if it was only that, I wouldn’t have been too worried. However, you also have a case where a comment changes the semantics of the following import after the comment. The whole thing seems like a giant hack to me; like “changing the grammar is too much work, so we’ll just make the tooling parse your comments so we don’t have to change the compiler”. I’m not saying that’s how the process is; I’m saying that’s how it feels as an observer.

I would’ve probably done something like C, where lines starting with # are obviously going to be interpreted differently from other lines. The compiler could ignore all lines starting with #, so that the tooling could add new directives in a backwards-compatible way, and parsers/formatters could largely ignore them. It just would’ve been a structured thing instead of essentially parsing prose.

1. 10

The whole thing seems like a giant hack to me; like “changing the grammar is too much work, so we’ll just (…)

This is how I feel about imports as strings. I see no good reason why we need quotation marks around the import paths in Go, it should be obvious for a parser how to deal with a path there. In general the Go syntax is… inconsistent at best.

1. 3

I see no good reason why we need quotation marks around the import paths in Go

This is a guess, but it may be to make plumbing easier in Plan 9/Acme. Syntax like <something> and "something" is easier to match than just something.

1. 1

I think the comment you’re replying to is arguing that it’s not just "something" vs. something, but import "something" vs. import something.

1. 3

Yeah, I know, but multiple imports still need quotation marks to be plumbable:

import (
"path/to/package1"
"path/to/package2"
)


works, whereas

import (
path/to/package1
path/to/package2
)


wouldn’t.

1. 2

Not sure why that would be the case. the plumber gets the whole string either way, expanded out to the surrounding whitespace if there’s no selection.

1. 4

But the plumber has to know somehow that it is a Go import path. The quotation mark syntax in Go (and C) makes that clear.

For example, should test, without quotes, really be treated as a Go/C import path? Wouldn’t that be too broad a rule?

1. 1

The plumber also gets a bunch of context, like the place you plumbed from. It should use that.

2. 3

I thought the same at first, but it’s simpler for the lexer/tokenizer to not have to know what mode it’s in. It’ll (presumably) just output a STRING token, and it’s only the parser that has to know that it’s in an import directive. Go is partly about keeping parsing (and lexing) simple and fast.

3. 6

It’s not my most favourite syntax either, but using #pragma or //go: seems like a really minor issue. I’m not sure what the motivation was for choosing //go: over # or #pragma.

a case where a comment changes the semantics of the following import after the comment

I assume you mean package foo // import "bar" type comments? That one, in particular, was a mistake IMO. But also kind of obsoleted by modules, and not really related to //go: directives.

1. 1

I don’t know what package foo // import "bar" does. Could you elaborate on it?

I was talking mostly about CGo, where you have a giant comment with a mix of C code and #cgo pseudo-directives followed by a line with import "C", where the comment (minus the #cgo lines) is compiled by a C compiler and linked with the binary.

1. 3

It enforces that the package is imported as bar; often used for “vanity paths”; e.g. lobste.rs/mypkg instead of github.com/lobsters/mypkg. This prevents some problem where one dependency might use the lobste.rs path, and the other the github.com path. It’s a bit of a hack, and you can do the same with a go.mod file now.

cgo is kind of a tricky beast, yeah; but also fairly rare. And since you can write any C code in that comment, I’m also not sure how else to do it? Overall, it seems to work fairly well and in the grand scheme of things, it seems like a fairly minor issue to me.

2. 37

But it’s a great way to pretend that it’s a small language with few reserved words.

1. 7

Even with the comment syntax, Go has fewer reserved words than C.

1. 7

While I agree with your comment, it’s a non sequitur. Having fewer reserved words and being a smaller language are two different things. LISP has 0 reserved words, therefore it must be the smallest language, right?

1. 4

Go is still a small language.

2. 3

This is the long-standing rationale for not having comments in JSON, and I think it stands the test of time (and complaints of programmers).

1. 16

This was just paternalistic nonsense on Crockford’s part. While I don’t really understand the Go author’s decision to use comments for pragmas, I would never in my life give up all comments to put a stop to it. An absolute textbook example of cutting off one’s nose to spite one’s face.

1. 16

I think it makes perfect sense for JSON’s intended usage as a data interchange format with good interoperability between all sorts of different environments. Extensions have made this much harder than it needs to be on more than a few occasions in the past.

Now, if you want to use JSON for your config files then sure, it’s annoying. But that wasn’t the intended usage. If you’re driving a square peg through a round hole then you shouldn’t really complain about the peg shape but get a different peg instead.

1. 7

I still don’t buy it. If we’re considering the intended usage, one of the goals that gets thrown around is that it is “easy for humans to read and write” – just as long as they never need to convey something to another human being in the form of a comment!

There are so many other under-specified parts of JSON, like what to do with large integers, or what happens when a document contains duplicate keys. It is extremely frustrating that comments were intentionally and unnecessarily stripped out, while the hard problems were apparently just ignored.

1. 8

HTTP or SMTP are easy for humans to read and write, and don’t have comments either because in the intended usage space it’s not really needed. Like those data formats, JSON is primarily intended to send text from one program to the other, and the “easy for humans to read and write” is more as a debugging and testing aid than anything else.

There are so many other under-specified parts of JSON, like what to do with large integers, or what happens when a document contains duplicate keys. It is extremely frustrating that comments were intentionally and unnecessarily stripped out, while the hard problems were apparently just ignored.

Sure, it could be improved, maybe, but as you mention these are not easy things to define in a compatible way since different languages deal with these things in different ways. But I don’t think that’s a good reason to introduce more interoperability issues.

2. 2

The comment syntax is mainly used together with cgo. Including a header instead of inlining C makes it feel less hacky and allows for syntax highlighting of the C code for a larger range of editors: // #include "project.h".

1. 2

IIRC there was discussion of adding pragma syntax, but it was decided that it’s not worth it to add yet another syntax, since there already existed comment-based pragmas that would have to be kept because of the go 1 compat promise.