1. 6

  2. 6

    The answer is never. If you need a generator-like function, you give the generator-like function a callback to call on all the iterables (that can return if it’s done or what not, for break-like behavior).

    Callbacks are always more general. The go developers realized this and were sad about it. I wonder if I can find the mailing list thread when I’m not on my phone, but check out os.Walk in the stdlib.

    That’s the right way. Trying to shoehorn channels here so you can use range is a hack. Someday I’ll write a blog post about how having to spin up a goroutine to service the side of a channel - just so you can use a channel when you could implement it without the goroutine or channel at all - is a huge anti-pattern

    EDIT: can’t find the thread, maybe it was the old Go wiki comments? I’ve looked everywhere, can’t find it.

    1. 8

      Or model it as an iterator:

      for s.Scan() {
      1. 1

        yep, another great model. a bit more boilerplate (create the iterator type if you want to support more than one concurrent iteration) but probably the most flexible and easy to use as a user

        1. 2

          The primary advantage is that you leave control up to the caller. For callbacks I generally allow a boolean or error return value, so that the callback function can signal that the iteration should be interrupted. However, it seems that some libraries use callbacks without a way to stop iteration (except for panic()).

      2. 4

        I think it is worth pointing out explicitly that when you use the Go pattern discussed here you can’t use break to get out of the loop. If you do, the goroutine will not be garbage collected.

        1. 1

          Indeed. You have to close the input reader if you want to cancel early. Python generators have the same issue, which is why you can throw an exception into a generator from the outside.

          1. 1

            No, Python generators do not have the same problem. If they did, then this interactive session would be leaking memory like crazy, and instead its memory usage remains stable at 8 megs (3.6 resident) for the minute-plus I could be bothered to wait:

            $ python
            Python 2.7.3 (default, Mar 14 2014, 11:57:14) 
            [GCC 4.7.2] on linux2
            Type "help", "copyright", "credits" or "license" for more information.
            >>> def x():
            ...  yield 1
            ...  yield 2
            >>> while True:
            ...  for y in x():
            ...   break
            1. 1

              See the old PEP 325 (from 2003), “Resource-Release Support for Generators”, for what I’m talking about.

              1. 1

                That was fixed ten years ago in Python 2.5 with PEP 342, but stalled generators didn’t result in leaking memory in Python the way blocked goroutines do in Golang even before that. It’s just that, until then, finally blocks in generators were not reliable. That’s what PEP 325 and the part of PEP 342 that supplanted it were designed to fix. PEP 342 is the PEP that introduced the .throw method you refer to upthread.

                But Python has never, not even in 2003, suffered from the problem Golang apparently does.

                I haven’t done enough in Golang to know why Golang would have this problem; you’d think that it would be safe to garbage-collect a process that’s blocked on a channel that nothing else has a reference to. I guess you could argue that the orphaned goroutine has nobody to report any possible errors to, but that’s already the case!

                1. 1

                  Yes, PEP 342 helps with throw. What I’m trying to say that finally blocks in generators still aren’t reliable unless you’re careful enough to fully exhaust the generator or throw() an exception into it. It’s similar to how you can force close the input reader to cause a goroutine to exit, in this example.

                  1. 1

                    Even that’s not true. As PEP 342 explains, you can also .close() the generator, which is equivalent to .throw()ing a GeneratorExit into it, and the .__del__() finalizer implicitly invokes .close(), so the only way that a finally or __exit__ can fail to run in a generator is if the generator never gets finalized — if the machine loses power, for example, or you kill -9 the process, or (I think) if the generator is part of a reference cycle.

                    1. 1

                      That’s a great point. The case I’m worried about is the last one (reference cycle or never garbage collected). Like you said, it would be great if Go had a way to do similar garbage collection by terminating a goroutine that’s not needed.

                      1. 1

                        While this is in theory a problem, I’ve never seen it, for these reasons:

                        1. Most of my generators (I use a lot of generators) don’t have with or finally clauses, because they don’t have side effects that they need to undo.
                        2. I do exhaust most of my generators.
                        3. I very rarely have circular references in my generators, and indeed I usually try to avoid circular references in general.
        2. 3

          Yep, I’ve been disillusioned by channels and iterators/generators are one of those subtle problems. If you’re not careful, you end up leaking. I’ve found the iterator pattern as coda suggests a much saner approach.

          Couldn’t have it put better myself @jtolds!

          1. 3

            I posted this comment on the article but I’ll reproduce it here:

            I think “never” is bad advice. The comments in the Reddit thread do a good job of covering the pros and cons of this approach. It’s a set of tradeoffs that depends on what you’re doing, like everything in programming. Sometimes a goroutine/channel is a good fit, sometimes it’s not.

            1. 1

              Not to be persistent here, but do you understand my point in my reply to your comment on the article about how a callback interface is always more general? Trivially, you can use a callback-based interface to implement a channel-based one. Just use the callback to send on a channel.

              I agree “never” is bad advice, but that said 2 will never be greater than 3.

              1. 1

                Isn’t that reversible? If you have a channel, just call the callback with everything that comes out of it.

                1. 1

                  Nope, not reversible. Channels add some unfortunate requirements.

                  1) Channels requires a goroutine servicing the channel, which isn’t trivial overhead unfortunately. This matters most when your API is being called heavily in hotpaths, and it’s frustrating when you’re trying to use an API that creates this overhead for you so you have to do something else.

                  2) Callbacks as discussed in a different thread above make it so you can break out of the iteration, but if it’s a channel you can’t throw a signal back to the generator that it’s time to stop generating. If you stop reading off the channel, unless you have a large enough buffer the producer will block and fail to die, causing resource leaks.