1. 25
  1.  

  2. 6

    I think this is meant to supersede the proposal at https://github.com/golang/go/discussions/54245. I like it. It resolves a lot of my niggles with the old design.

    1. 6

      It’s worrying that the proposal doesn’t explore how iterators can be combined, chained, or even used beyond the simplest case of a single iterator in a single for loop.

      The push/pull solutions (AKA internal vs external iteration) have very different tradeoffs when you try to use more than one iterator at a time (e.g. merge results from multiple sources into a single collection).

      1. 2

        That was a deliberate choice, since the first discussion was all about creating an iterators library, but there’s some push back in the discussion that we can’t totally design this in isolation from the question of how to chain and reuse.

        1. 3

          By that standard, isn’t Go good because people have just been using things like WalkDirFunc until now?

        2. 7

          There is no standard way to iterate over a sequence of values in Go.

          Go was publicly announced in November 2009, and version 1.0 was released in March 2012.

          I can’t help but feel like something went badly wrong here but what do I know.

          1. 20

            These types of comments really drive me up a wall. It feels like what you are saying is “this is a common feature in other languages, the people behind Go (vague bad thing) since they didn’t add the feature too” which is just not sound reasoning.

            In order to form a judgement about how bad something is, we should consider the consequences of it. The “normalness” of a behavior is an OK pointer, but thats all it is.

            Maybe you can argue that the consequences have been grave and thus this is a grave failure, but that doesn’t seem true to me.

            1. 26

              I can’t argue that any of the things about Go’s design that people have been unproductively grouchy about in comments sections for the past decade have had grave consequences for any given Go adopter or for the widespread adoption of Go. Having written one short program in Go for my own purposes, the lack of a proper idiomatic iteration construct (no map, no iterating for loop, no list comprehension, no yielding to a block, just the apparently-hacky for-range) was flummoxing. Go’s designers are entitled to their priorities but idk I feel like I’m entitled to make fun of those priorities a little bit, especially because they get paid more than I do.

              1. 14

                IMO there is a solid iteration idiom, and they hit on it early in the stdlib, although the fact that they managed to do it several different ways afterwards is a disappointment. It’s the

                for iter.Next() {
                    item := iter.Val()
                    // ...
                }
                

                one. You can do pretty much anything with it, it doesn’t impose any burden on the caller, it doesn’t make the caller ugly, and you can implement it on top of pretty much anything else. With generics you could even codify it as an interface (parameterized on the return type of Val).

                None of which is to say that I opposite this proposal — it looks pretty nice to me. But in 7+ years of writing Go professionally, the lack of an iterator syntax hasn’t been a major thorn in my side — or even a substantial annoyance.

            2. 7

              I’m pretty sure Zig has no concept of iterators either

              https://ziglang.org/documentation/master/

              I saw in the latest blog that the Zig for loop is being changed, but AFAIK there will still be no iterators. You basically do what you do in Go – write your own set of methods on a struct.

              So it seems like Zig will be a ~2025 language without iterators

              I would say languages have different priorities, and it’s harder than you think. Or you could just make vague statements about people doing bad things for no reason

              (edit: this bug seems to confirm what I thought https://github.com/ziglang/zig/issues/6185)

              1. 3

                Zig deliberately has no interfaces or traits whatsoever. You put the methods in the struct and they get called and it will compile if and only if the types work out after comptime propagation. I might be wrong but as far as I understand “iterators” in the language will be a bit of syntax sugar relying on a documented (but informal) interface, and Zig will very much have iterators exactly like, say, Julia or JavaScript or Python have iterators (except those languages check if things work out at runtime instead of compile time).

                On the other hand the major selling point of Go is having interfaces enforced by the compiler. But a fast iteration interface needs to be a generic interface so that wasn’t really possible until recently…

                Hopefully it all works out on both fronts.

                1. 4

                  Eh I don’t see what you’re saying. My point is that, as of now, Go and Zig are the same as far as iterators.

                  As of ~2025, Go might have iterators, and Zig probably won’t. Go is thinking about adding some concept of iterators to the language to enforce consistency.

                  Python’s for loop and list comprehensions understand iterators; I don’t think the same is true of Zig.

                  If you know otherwise, please provide a link to the docs.

              2. 4

                The dominating quality of Go’s development over the past decade has been the most extreme caution when it came to adding features. You can’t have performant, extensible iteration without some kind of generics and they were stuck in place on that issue out of fear of C++ compile times until finally a couple of years ago.

                1. 8

                  You can’t have performant, extensible iteration without some kind of generics

                  It’s even stronger than that: if you do want to map’n’filter, you need a boatload of machinery inside the compiler to make that fast, in addition to significant amount of machinery to make it expressible at all.

                  Rust’s signature for map is roughly

                  trait Iterator {
                    type Item;
                  
                    fn map<T, F>(self, f: F) -> Map<Self, F> 
                    where
                      F: FnMut(Self::Item) -> B;
                  }
                  

                  That is, .map returns a struct, called Map, which is parameterized by the type of the original iterator Self, as well as the type of unnameable closure F. Meditating on this single example for a long time explains half of why Rust looks the way it does.

                2. 3

                  Go’s developers focused on higher priority concerns, such as pretty great performance, a pretty great (though basic) type system, an awesome runtime and compilation model, and fantastic tooling. Go’s feature set (including the features it elided) made developers really, really productive compared with other languages.

                  While there are a few use cases that weren’t feasible without generics, the absence of generics made for some really interesting and compelling properties–like “everyone writes their code the same way (and thus any developer can jump into any other project and be immediately productive)” and “code is very concrete; people don’t usually try to make things overly abstract” which aren’t present in other languages. It wasn’t actually as obvious that generics were the right choice as Go’s critics claim (whose analyses flatly pretended as though there were no disadvantages to generics).

                  The net upside to generics (including iterators) was relatively small, so it makes sense that the decision was deferred.

                  1. 4

                    Go is a Google language. If a proposal helps or is of benefit to Google, it’ll be added. If it’s bad for Google, it will be ignored. If it’s neutral, then the only concern Google has is how well does it externalize training costs for Google.

                    1. 10

                      Google doesn’t really figure in at this level of discussion. The Plan 9 guys who made Go are the relevant actors for this. They were skeptical of generics, so it wasn’t a priority for Go 1.0. With no generics, a generic iterator protocol doesn’t make any sense, so that wasn’t in Go 1.0 either. Now Go has generics as of Feb. 2022, so there is a discussion about the best way to do an iterator protocol. This is the second proposal, which builds off of ideas from the first discussion and some ideas that had been in the issues tracker before that. It’s not really more complicated than that.

                      1. 4

                        You’re obviously right that the decision making an process is entirely about Google’s desires, but I’d hesitate to assume that it’s necessarily utilitarian. Google does a lot of self-sabotage.

                      2. 1

                        There is no standard way to iterate over a sequence of values in Standard ML, which is from circa 1970s/80s depending on who you ask, and is widely considered one of the most elegant of language designs. Something went badly wrong here or…?

                        1. 1

                          After having to deal with Rust iteration for a bit and missing out on Python…. I think the decent explanation here is that in more dynamic languages with stuff like coroutines it’s pretty easy to come up with a nice iterator protocol, but in more serious things it’s harder to come up with one that is both flexible enough for the “right” use cases without being very hard to use.

                          Like C++ has iterators right? And they do the job but they’re kind of miserable to use (or at least were 5+ years back, I’m sure things are better now).

                          Combine that with the perrenial generics things meaning container classes aren’t a thing and “stuff things into arrays and use indices” feels like a pretty OK solution for a long time.

                          1. 2

                            I think C++ iterators are uniquely awkward in their design, and it’s not an inherent design problem or any sort of static typing limitation.

                            C++ iterators are based around emulating pointer arithmetic with operator overloading, with a state awkwardly split between two objects. There’s no reason to do it this way other than homage to C and a former lack of for loop syntax sugar.

                            And C++ iterators aren’t merely tasked with iterating over a set once from start to finish, but double as a general-purpose description of a collection, which needlessly makes both roles harder.

                            1. 2

                              There’s no reason to do it this way other than homage to C and a former lack of for loop syntax sugar.

                              I think this is a little unfair, the primary reason to do it this way is so that code, especially templates work on pointers or iterators, eg being able to have a single implementation for something like std::find work for list or pointers. It’s not a “homage” so much as a source level interoperability consideration.

                              1. 1

                                OK, “homage” is a poor way of phrasing it. But it’s still an “interoperability consideration” with pointer arithmetic and C’s way of doing things, rather than a ground-up iterator design. The messy end result is not because iteration is such a hard problem, but because preserving C legacy is messy.

                                1. 1

                                  Right it’s not inherent to “designing iterators in statically typed language.” Go doesn’t have a different language it’s trying to be incrementally adoptable from.

                          2. -6

                            but what do I know.

                            Not much.

                          3. 2

                            I only looked at this briefly, but at first glance, I like it. It reminds me in a good way of Lua’s generic for.

                            1. 3

                              In Lua, is it impossible to iterate over a container containing nil?

                              1. 2

                                How to handle tables with nils in them has varied over Lua’s history. In current versions, you can use pairs or next. (pairs itself uses next under the hood, I believe.)

                                $ cat nils.lua
                                has_nils = { 1, 3, nil, 4, 5, nil }
                                
                                for _, v in pairs(has_nils) do
                                    print("Working on", v)
                                end
                                
                                $ lua nils.lua
                                Working on	1
                                Working on	3
                                Working on	4
                                Working on	5
                                
                                1. 2

                                  Containers (or rather, tables) in Lua can’t contain nll, which is used to mark ‘does not have a value’ or ‘does not exist’. That’s why bools were added to the language—the only two falsey values in Lua are false and nil.

                                  1. 1

                                    Sure, but you can have a table (being used as a container or object) with a set of keys, some of which can have nil values. I think the question was how would you iterate over the whole container (e.g., table) without the iteration ending early at the first nil. That’s where, pairs or next are handy. But maybe I’m confused.

                                    1. 1

                                      A bit. A Lua table is an associative array. Any undefined key into a table will return a nil. To remove a value, you set the value to nil. There is also the concept of a “sequence”, or what we would normally call an array. The first index is 1, and increases. This: { 6 , 7 , 8 , 9 } is a valid sequence, but this: { 6 , nil , 8 , 9 } is not, because the nil breaks the sequence. It sounds horrible, but in practice, it’s not bad once you get used to it. And yes, you can use pairs() on a sequence, so my second example would return keys 1 , 3, and 4.

                                      1. 1

                                        And yes, you can use pairs() on a sequence, so my second example would return keys 1 , 3, and 4.

                                        Agreed: that was my point. I think the OP’s original question was, in effect, can you iterate over an entire table if the table contains nil values. The answer is yes, using pairs or next.