1. 32
  1.  

  2. 32

    If there’s one thing doing formal methods has taught me, it’s that time is evil and I hate it. Time is just an inherently hard domain and there’s only so much languages can do to address concurrency.

    Edit: successfully reproduced the bug in TLA+, verified that moving the channel receive to the worker fixes the deadlock.

    Edit2: TLA+ spec is here

    1. 13

      Give OTP a try sometime. Battle-hardened patterns for concurrency-model demands.

      1. 1

        Do you know of anyone who’s tried to build something like OTP in Go? I think, with the right juggling of channels behind the scenes, it should be generally possible?

        1. 11

          OTP builds on top of Erlang. There is one fundamental rule that you have to follow if you want to have even a vague chance of getting concurrency right: No object may be both mutable and shared between concurrent execution contexts. In Erlang, that is trivial: the only mutable objects are the process dictionaries, which you cannot take a reference to and so cannot share. Everything else is immutable and so safe to share.

          In Go, there is nothing in the type system that helps you avoid accidentally sharing mutable objects. There is no real notion of immutability in the language, so you have to build immutable abstractions by creating objects that expose only read-only accessors for their state.

          For a language that has channels or message passing and also has mutable objects, you want to guarantee that one of two properties holds for any object that you send. It is either immutable, or you gave up your reference to it at the time that you sent it. Pony and Verona enforce this in the type system, Rust tries to build this the standard library, C++ makes it possible to express this via smart pointers (but doesn’t make it easy), Go doesn’t even make it possible to express this in any way that can be statically checked.

          Go is one of the worst languages around for concurrency. It makes it incredibly easy to write concurrent code but gives you no help in writing correct concurrent code.

          1. 2

            Yikes. Thanks for the insights!

          2. 1

            Not off hand. Sounds like a fantastic idea, though. Not familiar with Go, but I know that some of the fundamentals that probably need to be at hand to do something OTP-like are monitors, links, and an error model that’s rooted in message passing.

        2. 9

          I would say especially in Go, since Go makes concurrency pretty hard compared to systems I’m used to…

          1. 11

            What systems are you used to?

            1. 6

              He’s probably thinking of Haskell, maybe secondarily Scala.

              I mostly use Haskell, Rust, and Java and I have to concur.

              1. 2

                Yeah, Rust+Tokio is also pretty good.

              2. 4

                If I want to do concurrency I’ll always reach for Haskell, but also comfortable in Ruby+EventMachine

                1. 2

                  This is super confusing to me. What do you use concurrency for?

                  1. 1

                    High-throughput network servers and clients, mostly.

                    1. 2

                      I think you might be the first person I’ve ever encountered who defaults to Haskell for network servers. This isn’t a criticism in any way, just an expression of mild astonishment.

                      1. 2

                        I certainly didn’t used to, but at this point I haven’t been able to find something else that even comes close in terms of concurrency abilities, especially with the GHC runtime. In something like Ruby+Eventmachine or Rust+Tokio you have to manage your async much more explicitly, whereas in GHC Haskell all IO operations are async all the time within the idiomatic programming model. With lower level systems like Go, you can have thread safety problems and non-atomic operations, wheras in Haskell all IO operations are atomic (unless you use the FFI in unsafe ways) and of course most code is pure and has no possible thread safety problems at all.

                        Probably more reasons, but that’s what comes to my mind.

                        1. 1

                          What kind of RPS and p99 latency do you get with a Haskell service serving HTTP and doing some nontrivial work per request?

                          1. 1

                            Looks like the Haskell web server (warp) comes in at 20th on a webserver benchmark from four months ago.

                            At my last job I did a lightning talk on Python vs Haskell for a simple webapp. I wanted to focus on simplicity of the code, but my coworkers wanted to see benchmarks. Haskell was much faster than Python for 99% of the requests, with laziness and garbage collection putting the last fraction of 1% of responses slower than Python. Python was slow, but consistently slow.

                            1. 1

                              Hmm, I have done some Haskell HTTP stuff, but not for high performance. If you’re really curious about HTTP I’d look up warp benchmarks.

                              1. 1

                                OK, then whatever you’ve done: I’m just trying to get a sense Haskell’s ballpark.

                          2. 1

                            I’m also a fan, lots of benefits to doing network servers in Haskell.

                            Perhaps this tour of go in haskell would help illustrate some benefits?

                2. 9

                  Instead of spinning up one goroutine per task and rate-limiting the speed you create them, a cleaner solution is to spin up N goroutines and have them all read from a shared channel. Something like:

                  func forEachLimit(arr []int, fn func(int) err, parallelism int) err {
                    queue := make(chan int)
                    g := new(errorgroup.Group)
                  
                    for i := 0; i < parallelism; i++ {
                      g.Go(func() {
                        for item := range queue {
                          if err := fn(item); err != nil {
                            return err
                          }
                        }
                        return nil
                      })
                    }
                  
                    for _, item := range arr {
                      queue <- item
                    }
                  
                    close(queue)
                  
                    if err := g.Wait(); err != nil {
                      return err
                    }
                  
                    return nil
                  }
                  

                  FWIW, I do agree that concurrency in Go isn’t easy to get right. There was a paper a while ago where the authors found plenty of real world examples of deadlocks and memory leaks due to the specific behavior of channels. They also found Go programs tend to use way more concurrency than in other languages. I suspect this is because Go makes it so easy to write concurrent code, even when concurrency isn’t necessary.

                  1. 2

                    I suspect this is because Go makes it so easy to write concurrent code, even when concurrency isn’t necessary.

                    In fairness, Rob Pike has warned before that goroutines can be a lot of fun to write and work with, but programmers shouldn’t get carried away with them because sometimes all you need a reference counter

                  2. 9

                    Interesting tangental thought:

                    I saw this article and the URL and was like, “oh its the guy who hates Go again!” But when I looked at the actual website this author actually writes a significant amount about Go, the vast majority not negative. It seems like that the users of lobsters seem to be only interested in posting and upvoting anti-Go articles, and it is we who are biased.

                    1. 4

                      I’m the author of the linked-to article, and as you sort of noticed I’m pretty fond of Go (and use it fairly frequently). If I wasn’t, I wouldn’t write about it much or at all. I suspect that part of the way my writing about Go has come out the way it has is that I think it’s a lot easier to write about problems (including mistakes that you can make) than about things that are good and just work.

                      (We have one core program in our fileserver environment that’s written in Go, for example, but it’s pretty boring; it sits there and just works, and Go made writing it straightforward, with simple patterns.)

                      1. 3

                        I quite fond of Go myself, and I enjoy reading your articles even when I thought you only talked about the problems! 😂

                        I think negative language news (especially regarding “newer” languages) has more success on this site, so there’s a secondary filtering happening as well.

                      2. 0

                        who’s the guy who hates Go?

                      3. 6

                        Go can only provide low level concurrency tools because it’s statically typed. High level patterns require generics or dynamic typing so they can be used by applications. It’s between a rock and a hard place.

                        1. 3

                          The highest-level concurrency patterns I have ever seen are in GHC Haskell, which is pretty much as statically typed as you can get.

                          1. 1

                            The difference is that Haskell also has extremely powerful generics, so you can define a concurrency pattern once and have it work everywhere without having to resort to interface {} hacky nonsense.

                        2. 5

                          Related:

                          Channels Are Not Enough

                          An interesting argument that Go’s out-of-the-box concurrency abstractions are too low-level.

                          1. 3

                            https://godoc.org/golang.org/x/sync/errgroup is very useful for this kind of thing.

                            Here is a pattern I’ve been using a lot lately which made concurrency much easier and safer for this kind of scenario:

                            func DoStuff(ctx context.Context, tasks []Something) (Result, error) {
                            	resultCh := make(chan Result)
                            	g, gCtx := errgroup.WithContext(ctx)
                            
                            	// Can move this task provisioning block into a goroutine also, especially if
                            	// maintaining a max worker limit
                            	for _, t := range tasks {
                            		t := t // Copy value because we're passing it down the closure. A bit of a gotcha
                            		g.Go(func() error {
                            			// Usually bunch of code here, or just pass do_something into g.Go
                            			return do_something(t)
                            		})
                            	}
                            
                            	// This is a bit awkward, I wish errgroup had something like:
                            	// g.Defer(func() { cleanup code that runs just before g.Wait() })
                            	gProcess, _ := errgroup.WithContext(gCtx)
                            	gProcess.Go(func() error {
                            		// This group closes resultCh which stops the accumulation loop below
                            		defer close(resultCh)
                            		return g.Wait()
                            	})
                            
                            	var result Result
                            	for res := range resultCh {
                            		// accumulate result until resultCh is closed
                            	}
                            
                            	return result, gProcess.Wait()
                            }
                            

                            I’ve been using it so much that I’ve been thinking of upstreaming the errgroup.Group.Defer(func()) idea, or maybe maintaining a fork.

                            1. 4

                              Won’t your code still run into the same problem if you were to implement the max goroutine limit here, where it would block if you had too many tasks to start at once, before you started draining the responses from those tasks?

                              1. 1

                                That’s a good point, if you want to do max goroutine management then the task provisioning will need to be in a goroutine also.

                                I often wish errgroup also had max worker throttling too. I wonder if there’s already an implementation that does all of this?

                                1. 1

                                  One can also divide the task slice by the desired number of workers and let each worker process a slice. This is a pattern I use sometimes if limited concurrency is required.

                              2. 2

                                Don’t need errgroup for that :)

                                1. 2

                                  Of course not, I wasn’t making the argument that anything is not possible without errgroup, just that this kind of errgroup pattern adds convenient rails that help avoid the kinds of mistakes which the blog post mentioned. :)

                                  Also encourages me to collect errors and pass contexts around without avoiding it due to laziness.

                                  1. 1

                                    Sure, my point is that the code without errgroup is actually easier to grok than with it.

                                    1. 1

                                      I do see what you mean. :)

                              3. 3

                                Go definitely provides better primitives than just locks for some concurrency patterns, but there are a lot of patterns that need to be built atop of them still, from what I can see. This looks like it could been useful to use something like a threadpool, or a higher order function for limiting the amount of parallelism (like a , though Go doesn’t make writing those sorts of higher level functions easy.

                                That being said, it’s definitely a subtle problem. Concurrency (via Channels or via Aysnc) is definitely something that can be easy to get tripped up on. Usually, I find that modeling problems by treating each part of the process as it’s own mini server can be handy, but supervision (which is necessary for systems over a certain amount of complexity) can still make things fiddly.

                                1. 5

                                  My impression of Go is that it encourages “DIY” concurrency primitives. Channel here, channel there. You can relatively easily make things run concurrently, and even compose a few things together.

                                  But once programs get bigger, I’m getting lost. I’m missing a clear structure: parallel iterators, queues, thread pools. They’re all sort-of there, but all duct-taped from channels. It reminds me of early Node.js where callbacks where the only hammer.

                                  1. 2

                                    Yeah, that’s how I’ve done things in the past (I haven’t programmed in Go in a while, however). One hopes that generics will bring some needed higher-level libraries to the mix with all of this.