1. 3

    One of my big hopes with generics in 1.18 is JSON encoding / decoding gets faster because we can use concrete types and not use reflection (obviously this will take time as the stdlib will be mostly generics-free for some time).

    The encoding/json performance problems in the go standard library are a major issue, not insurmountable though.

    My suggestion to the author is to try and non-stdlib / optimized JSON unmarshaler.

    1. 3

      That’s not going to happen. The way generics work doesn’t allow for compile time execution (or even specialization, although that will probably happen eventually), so there’s no way serialization will ever work with Go generics. For the foreseeable future, if you want to use a concrete type when serializing, it will need to be go:generated.

      1. 3

        Yes, it’s likely that JSON unmarshaling is a performance bottleneck here. For “big data” JSON filtering, this is a real concern. I saw https://github.com/bytedance/sonic just the other day, and I know there are other performance-focused JSON libs.

        That said, I think it’s a stretch to say the “the encoding/json performance problems in the go standard library are a major issue”. For many common use cases like web applications it just doesn’t matter much. I’ve used Go’s encoding/json extensively for small servers as well as reasonably high-throughput servers (10s or 100s of requests per second, not thousands or millions) and it works fine – database query performance was usually the bottleneck.

        1. 2

          The share of issue depends on how much your program is doing in each area; if your program is just processing json, and the json library is slow, it’s an issue (as in, this case).

          The other piece of context here is how the performance is “relatively” to other runtimes; here is where the standard library implementation suffers the most, as compared to other common runtimes it’s significantly slower.

          1. 2

            What other common runtimes are significantly faster than Go, and in which dimensions? As far as I’m aware, Go’s runtime, which is optimized for latency, is as fast or faster than anything on the market, but I’d be happy to see data suggesting otherwise!

            1. 2

              Sorry should have been clearer, *runtime standard library json packages. The terminology (“runtime”) there was used poorly.

              1. 1

                Gotcha! Thanks.

      1. 14

        I/O in Go is not buffered by default (explicit is better than implicit), so fmt.Println calls fmt.Fprintln(os.Stdout, ...) where os.Stdout is unbuffered. This is fine for interactive programs with small amounts of output, but for any performance-oriented filter you want to use fmt.Fprintln(buf, ...), where buf is a bufio.Writer which wraps os.Stdout. You’ll see at least a 10x performance improvement if the bottleneck is writing output. See for example the commit where I fixed this in GoAWK.

        Edit: The author does hint at this in his post, “I suspect that if the buffer fmt.Println is writing to wasn’t flushed so often there could be a dramatic increase in performance.” And thanks to Go’s excellent I/O interfaces, it’s only a two-line fix to do so.

        1. 3

          Thanks for the tip Ben. I just did a quick benchmark with buffering and it was 20% faster on the 1 million-line file. I’ll do a run later today and see what the impact is on the 1.27B dataset.

          1. 5

            Update: bufio took 22 minutes off the 71-minute run time. I’ve updated the post with all the details.

        1. 14

          Nice article.

          Sorry for the tangent, but I wanted to use this as an opportunity to discuss error handling strategies. There was some discussion back on https://lobste.rs/s/yjvmlh/go_ing_insane_part_one_endless_error - but I and others expressed needing specific cases in order to discuss improvements, removing if err != nil { return err } patterns.

          For this code, I’d probably make a structure for encapsulating the state of the decoding process:

          type headerDecoder struct {
            bs []byte
            offset int
            err error
          }
          

          Then for the read helpers, I’d change them so that they fail silently:

          func (dec *headerDecoder) readUint32() (i uint32) {
            if dec.err != nil {
              return
            }
            end := offset + 4
            if end > len(dec.bs) {
              dec.err = errOverranBuffer
              return
            }
            i = binary.LittleEndian.Uint32(dec.bs[dec.offset:end])
            dec.offset = end
            return
          }
          

          Alternatively, can return an extra/ignorable ok bool if the call sites need it.

          Now, your main decoding logic is dramatically cleaner:

          version := dec.readUint16()
          bitFlag := dec.readUint16()
          compression := noCompression
          compressionRaw := dec.readUint16()
          if compressionRaw == 8 {
              compression = deflateCompression
          }
          lmTime := dec.readUint16()
          lmDate := dec.readUint16()
          lastModified := msdosTimeToGoTime(lmDate, lmTime)
          crc32 := dec.readUint32()
          compressedSize := dec.readUint32()
          uncompressedSize := dec.readUint32()
          fileNameLength := dec.readUint16()
          extraFieldLength := dec.readUint16()
          fileName := dec.readString(int(fileNameLength))
          extraField := dec.readBytes(int(extraFieldLength))
          return &localFileHeader{
              signature: signature,
              version: version,
              bitFlag: bitFlag,
              compression: compression,
              lastModified: lastModified,
              crc32: crc32,
              compressedSize: compressedSize,
              uncompressedSize: uncompressedSize,
              fileName: fileName,
              extraField: extraField,
              fileContents: fileContents,
          }, dec.offset, dec.err
          
          1. 3

            Programmable control flow, nice!

            1. 6

              Rob Pike wrote about this pattern a few years ago: https://go.dev/blog/errors-are-values

            2. 2

              I’m a huge fan of this pattern. (As @pims says in another reply, if you don’t know the pattern, check out Rob Pike’s “Errors are values” on the Go blog.)

              Another nice use for it is in command line applications. I define a small helper function like this:

              func (app *App) NoOp() bool {
              	return app.ExitValue != exitSuccess || app.HelpWanted || app.VersionWanted
              }
              

              That function is called at the top of (almost) all the methods for App:

              if app.NoOp() {
                  // return whatever makes sense for the given method, often nothing at all
              }
              
              

              Then the command’s main Run function looks as clean as this:

              func Run(args []string) int {
                  app := &App{ExitValue: exitSuccess}
                  cfg := app.ParseFlags(args)
                  courseData := app.Unmarshal(classFile)
                  app.Validate(cfg, courseData)
                  app.Write(cfg, courseData)
                  return app.ExitValue
              }
              

              Run in turn is called from a very simple main.go. (Happily stolen from Nate Finch, Mat Ryer, and @carlmjohnson.

              func main() {
                  os.Exit(cli.Run(os.Args[1:]))
              }
              
              1. 1

                That’s a nice idea for code like this. I guess it’s a little bit less efficient in the error case, as it’d have to run through all those checks even if the failure is on the first one, but presumably the error case is relatively rare so that doesn’t matter.

                1. 3

                  presumably the error case is relatively rare so that doesn’t matter

                  This is true. Moreover, the worst case performance here is bounded by the performance of the success case.

                  Not super relevant here, but in other cases, this approach is useful for improving error reporting more generally. For example, compilers that give up at the first sign of error are less pleasant to work with than those that attempt to recover and report multiple errors in a single compiler run.

                  1. 1

                    Moreover, the worst case performance here is bounded by the performance of the success case.

                    Yep, good point.

                    For example, compilers that give up at the first sign of error are less pleasant to work with than those that attempt to recover and report multiple errors in a single compiler run.

                    Maybe it depends on one’s working style, but I tend to just fix one (or maybe one or two) issues and recompile often, so I haven’t found this to be the case (Python only shows you the first error, for example). Even in Go, which prints the first few errors (10 I think) I often only fix one or two at a time, then go run again. Then again, I guess it’s strictly better to give you at least a few errors at once – if you want to only fix one at a time, you can, but with Python you don’t get the choice.

              1. 3

                I’m really hoping this will be included as a built-in in a future version of Go, ala the Go issue 45624 that you linked to in the README. I like the spelling &callLike(v), for example &int(42) or &time.Now(), but ptr(42) or new(int, 42) or similar would work too.

                Using a library for this seems leftpad-ish: too small to pull in as a 3rd party dependency, but you might not bother if you have to write func ptr[T any](v T) *T { return &v } in every package where you use it might get a bit old.

                1. 3

                  Yes, as an advocate of avoiding dependencies you could write in an afternoon, this definitely falls under my threshold. Most people should just write their own version if they want to use something like it.

                  That said, I prefer a new generic built in to allowing new(int, 42) or &int(42). The two value new form is unique even though it could just be a normal generic function, and the &int(42) form has problems with confusing results from taking pointers to functions.

                  1. 2

                    Yeah, fair reasoning. I’m good with a generic ptr(x) builtin!

                1. 25

                  Go doesn’t need async/await, it has goroutines and channels.

                  1. 12

                    +1

                    c := make(chan int)      // future
                    go func() { c <- f() }() // async
                    v := <-c                 // await
                    
                    1. 1

                      I had a negative knee-jerk reaction when I saw the async/await naming. But if you rename/rejig the api a bit, it seems a lot less bad. See: https://gotipplay.golang.org/p/IoHS5HME1bm w/context https://gotipplay.golang.org/p/Uwmn1uq5vdU

                    2. 2

                      this uses goroutines and channels under the hood …

                      1. 8

                        why do you need to abstract-away goroutines and channels?

                        1. 4

                          I have no idea. It’s like seeing someone recite a phone book from memory: I can appreciate the enthusiasm without understanding the why

                      2. 1

                        They said Go didn’t need generics either :)

                        I get your point though. Hence why almost every bit of this repo screams “experimental.” I have been just playing around with the pattern in some work/personal projects and seeing how it works ergonomically and seeing if it improves areas with lots of asynchronous operations.

                        But, only a matter of time until more folks begin trying to abstract away the “nitty-gritty” of goroutines/channels with generics. I personally point goroutines/channels out as Go’s greatest features, but I have seen others really want to abstract them away.

                        1. 4

                          Goroutines and channels are there to abstract away asynchronous code.

                          1. 5

                            Goroutines and channels are abstractions that are a marked improvement on the state of the art prior to Go, but I find that they tend to be too low-level for many of the problems that programmers are using them to solve. Structured concurrency (or something like it) and patterns like errgroup seem to be what folks actually need,

                            1. 5

                              Yeah, I also long time ago thought, that one area where generics in Go could hopefully help, would be in abstracting away channel patterns - things like fan-out, fan-in, debouncing, etc.

                              1. 2

                                honestly I just want to be able to call select on N channels where N is not known at compile time. A cool thing about promises is being able to create collections of promises. You can’t meaningfully create collections of channels. I mean sure, you can make a slice of channels, but you can’t call select on a slice of channels. select on a slice of channels is probably not the answer but is a hint at the right direction . Maybe all := join(c, c2) where all three of those values are of the same type chan T. I dunno, just spitballing I haven’t given that much thought, but the ability to compose promises and the relative inability to compose channels with the same expressive power is worth facing honestly.

                                I actually fully hate using async and await in JS but every time I need to fan-out, fan-in channels manually I get a little feeling that maybe there’s a piece missing here.

                                1. 3

                                  I just want to be able to call select on N channels where N is not known at compile time.

                                  You can.

                                  https://golang.org/pkg/reflect#Select.

                                  1. 2

                                    the argument that I’m making is that promises have ergonomics that channels lack, and that although I don’t think Go needs promises, that the project in question is reflective of how promise ecosystems have invested heavily in ergonomics in many scenarios that Go leaves for every developer to solve on their own. Calling reflect.Select is not a solution to a problem of ergonomics, because reflect.Select is terribly cumbersome to use.

                                  2. 1

                                    honestly I just want to be able to call select on N channels where N is not known at compile time

                                    That’s still too low-level, in my experience. And being able to do this doesn’t, like, unlock any exciting new capabilities or anything. It makes some niche use cases easier to implement but that’s about it. If you want to do this you just create a single receiver goroutine that loops over a set of owned channels and does a nonblocking recv on each of them and you’re done.

                                    every time I need to fan-out, fan-in channels manually I get a little feeling that maybe there’s a piece missing here.

                                    A channel is a primitive that must be owned and written to and maybe eventually closed by a single goroutine. It can be received from by multiple goroutines. This is just what they are and how they work. Internalize these rules and the usage patterns flow naturally from them.

                                    1. 2

                                      loops over a set of owned channels and does a nonblocking recv on each of them and you’re done.

                                      How do you wait with this method? Surely it’s inefficient to do this in a busy/polling loop. Or maybe I’m missing something obvious.

                                      Other approaches are one goroutine per channel sending to a common channel, or reflect. Select().

                                      1. 1

                                        Ah, true, if you need select’s blocking behavior over a dynamic number of channels then you’re down to the two options you list. But I’ve never personally hit this use case… the closest I’ve come is the subscriber pattern, where a single component broadcasts updates to an arbitrary number of receivers, which can come and go. That’s effectively solved with the method I suggested originally.

                                      2. 1

                                        I’ve been programming Go for ten years. I know how channels work.

                                        Promises can be composed, and that is a useful feature of promises. Channels cannot be composed meaningfully, and that is rather disappointing. The composition of channels has much to give us. Incidentally, the existence of errgroup and broadly most uses of sync.WaitGroup are the direct result of not having an ability to compose channels, and channel composition would obviate their necessity entirely.

                                        What is it that sync.WaitGroup and errgroup are solving when people generally use them? Generally, these constructs are used in the situation that you have N concurrent producers. A common pattern would be to create a channel for output, spawn N producers, give every producer that channel, and then have all producers write to one channel. The problem being solved is that once a channel has multiple writers, it cannot be closed. sync.WaitGroup is often used to signal that all producers have finished.

                                        This means that practically speaking, producer functions very often have a signature that looks like this:

                                        func work(c chan T) { ... }
                                        

                                        Instead of this:

                                        func work() <-chan T { ... }
                                        

                                        This is in practice very bothersome. In the situation that you have exactly one producer that returns a channel and closes it, you could do this:

                                        for v := range work() {
                                        }
                                        

                                        This is great and wonderfully ergonomic. The producer simply closes the channel when it’s done. But when you have N producers, where N is not known until runtime, what can you do? That signature is no longer useful, so instead you do this:

                                        func work(wg *sync.WaitGroup, c chan T) {
                                            defer wg.Done()
                                            // do whatever, write to c but don't close c
                                        }
                                        
                                        var wg sync.WaitGroup
                                        c := make(chan T)
                                        for i := 0; i < n; i++ {
                                            wg.Add(1)
                                            go work(&wg, c)
                                        }
                                        
                                        done := make(chan struct{})
                                        go func() {
                                            wg.Wait()
                                            close(done)
                                        }()
                                        
                                        for {
                                            select {
                                            case <-c:
                                                // use the result of some work
                                            case <-done:
                                                // break out of two loops
                                            }
                                        }
                                        

                                        That’s pretty long-winded. The producer written for the case of being 1 of 1 producer and the producer written for the case of being 1 of N producers have to be different. Maybe you dispense with the extra done channel and close c, maybe you use errgroup to automatically wrap things up for you, it’s all very similar.

                                        But what if instead of N workers writing to 1 channel, every worker had their own channel and we had the ability to compose those channels? In this case, composing channels would mean that given the channels X and Y, we compose those those channels to form the channel Z. A read on Z would be the same as reading from both X and Y together in a select statement. Closing X would remove its branch from the select statement. Once X and Y are both closed, Z would close automatically. Given this function, we could simply have the worker definition return its own channel and close it when its done, then compose all of those, and then read off that one channel. No errgroup or sync.WaitGroup necessary. Here is an example of what that would look like:

                                        func work() <-chan T {}
                                        
                                        var c <-chan T
                                        for i := 0; i < n; i++ {
                                            c = join(c, work())
                                        }
                                        
                                        for v := range c {
                                            // use the result of some work
                                        }
                                        

                                        Here is a working program that implements this concept at the library level: https://gist.github.com/jordanorelli/5debfbf8dfa0e8c7fa4dfcb3b08f9478

                                        Tada. No errgroup necessary, no sync.WaitGroup, none of that. The producer is completely unaware that it is in a group and the consumer is completely unaware that there are multiple producers. You could use that producer and read its results as if it’s just one, or one of many in the exact same way.

                                        It makes consuming the result of N workers much easier, it makes it so that a worker may be defined in the event that it is 1 of 1 and 1 of N in exactly the same way, and it makes it so that consumers can consume the work from a channel without any knowledge of how many producers that channel has or any coordination outside of seeing the channel closed. Of course, implementing this at the library level and not at the language level means adding an overhead of additional goroutines to facilitate the joining. If it could be implemented at the language level so that joining N channels into 1 does not require N-1 additional goroutines, that would be neat.

                                        This implementation is also subtly broken in that composing X and Y to form Z makes it so that you can’t read off of X and Y on their own correctly now; this is not a full implementation, and there’s certainly a question of implementation feasibility here.

                                        1. 1

                                          Channels cannot be composed

                                          I don’t think I agree. It’s straightforward to build higher-order constructs from goroutines and channels as long as you understand that a channel must be owned by a single producer.

                                          The problem being solved is that once a channel has multiple writers, it cannot be closed.

                                          It doesn’t need to be closed. If you have 1 channel receiving N sends, then you just do

                                          c := make(chan int, n)
                                          for i := 0; i < cap(c); i++ {
                                              go func() { c <- 123 }()
                                          }
                                          for i := 0; i < cap(c); i++ {
                                              log.Println(<-c)
                                          }
                                          

                                          This means that practically speaking, producer functions very often have a signature that looks like func work(c chan T) { ... }

                                          Hopefully not! Your worker function signature should be synchronous, i.e.

                                          func work() T
                                          

                                          and you would call it like

                                          go func() { c <-work() }()
                                          

                                          Or, said another way,

                                          go work(&wg, c)

                                          As a rule, it’s a red flag if concurrency primitives like WaitGroups and channels appear in function signatures. Functions should by default do their work synchronously, and leave concurrency as something the caller can opt-in to.

                                          But what if . . .

                                          If you internalize the notion that workers (functions) should be synchronous, then you can do whatever you want in terms of concurrency at the call site. I totally agree that goroutines and channels are, in hindsight, too low-level for the things that people actually want to do with them. But if you bite that bullet, and understand that patterns like the one you’re describing should be expressed by consumers rather than mandated by producers, then everything kind of falls into place.

                                          1. 1

                                            It’s clear that you didn’t read the gist. Your example falls apart immediately when the workers need to produce more than one value.

                                            Your worker function signature should be synchronous, i.e. func work() T

                                            That’s not a worker. It’s just a function; a unit of work. That’s not at all the problem at hand and has never been the problem at hand. Maybe try reading the gist.

                                            The workers in the gist aren’t producing exactly 1 output. They’re producing between 0 and 2 outputs. The case of “run a function N times concurrently and collect the output” is trivial and is not the problem at hand.

                                            The workers are producing an arbitrary number of values that is not known in advance to the consumer. The workers are not aware that they’re in the pool. The consumer is not aware that they’re reading from the pool. There is nothing shared between producers to make them coordinate, and nothing shared with consumers to make them coordinate. There is no coordination between producers and consumers at all. The consumer is not aware of how many workers there are or how many values are produced by each worker, they are only interested in the sum of all work. The workers simply write to a channel and close it when they’re done. The consumer simply reads a channel until the end. That’s it. No errgroup requiring closures to implement the other half of the pattern, no sync.WaitGroup required to manually setup the synchronization. Just summing channels. The case of 1 worker and 1 consumer is handled by a worker having signature func f() <-chan T. The case of 1 worker and N consumers, N workers and 1 consumer, and N workers and M consumers are all handled with the same worker signature, with no additional coordination required.

                                            1. 1

                                              It’s clear that you didn’t read the gist.

                                              I mean, I did, I just reject the premise :)

                                              That’s not a worker. It’s just a function; a unit of work

                                              Given work is e.g. func work() T then my claim is that a “worker” should be an anonymous function defined by the code which invokes the work func, rather than a first-order function provided by the author of the work func itself.

                                              The workers are producing an arbitrary number of values that is not known in advance to the consumer . . . the consumer simply reads a channel until the end.

                                              Channels simply don’t support the access pattern of N produers + 1 consumer without a bit of additional code. It’s fair to criticize them for that! But it’s not like the thing is impossible, you just have to add a bit of extra scaffolding on top of the primitives provided by the language.

                                2. 2

                                  I think generics will make channels much easier to use correctly. The shenanigans required to handle cancellation, error reporting, fan-out with limits, etc etc etc means that very few programs handle the edge cases around goroutines. Certainly when I wrote go, I wouldn’t follow the patterns needed to prevent infinite go routine leaks, and often I’d decide to panic on error instead of figuring out how to add error channels or result structs with a null error pointer, etc.

                                  What I like about Promise is that it’s a Result[T] - but ideally I’d be able to get the composition boilerplate of structured CSP stuff out of the way with generics instead of adopting the Promise model wholesale.

                                  (My history: I loved writing go for many years but eventually burned out from all the boilerplate and decided to wait for generics)

                              1. 13

                                Love the corrections, and I agree that it is weird that this tutorial uses Gin of all libraries to show of writing APIs with Go - Gin doesn’t even conform to the standard library’s net/http interfaces, you have to wrap any middleware that does conform.

                                One small nit-pick around the IDs, and it’s not even your improvements, but when you’re writing an API you should never (seriously can’t think of one reason why you should) let the user set record IDs. Ideally that should all be in the database (which is why using an in-memory database is an…odd choice).

                                Overall, Russ’s comments about the tutorial irk me a bit - if you want something narrow in scope, don’t choose to do something like a full-on RESTful API. Sure, it’s basic and common like spaghetti and meatballs, but like spaghetti and meatballs it takes a lot of work and a surprising amount of complexity to make it truly good. Like, auth is integral to how APIs work, interacting with a database is a key part of the API. Either give a disclaimer that those parts are going to be glossed over, or change the tutorial title to reflect that you’re focusing on RESTful endpoints or something.

                                I’m probably coming off overly grumpy about this, but I really love Go and I think we can do better when it comes to teaching aspects of the language. The rest of the documentation and (official) tutorials are incredibly good, no reason this can’t be as well.

                                1. 6

                                  These comments resonate with me – thanks.

                                  I agree about the caller not having to generate the ID: I should have made that change too. :-) I’ve added a comment pointing this out and linking here at the bottom of the Unique album IDs section.

                                  1. 2

                                    I’ve been quite happy with using chi.

                                    1. 2

                                      Me too. All new Go work I’ve done that justifies using something beyond the standard library has used Chi.

                                  1. 16

                                    I like this. Fun quote:

                                    Blog.txt supports multiple options for the chronological order of posts. If you start writing new posts below old posts, the default post sort is descending. If you start writing new posts above all the old posts, like I do, then the post sort algorithm will default to ascending. But if the user would like to change the sort order of the posts, they can press the “End” button on their keyboard to reverse the default chronological order!

                                    1. 2

                                      Keyboard shortcuts built in :)

                                    1. 3

                                      I’m the first to urge caution in upgrades, but without highlighting actual breaking changes this seems like fud.

                                      1. 12

                                        Some of us hang out in forums where people literally start posting minutes after a Python release that they don’t understand why NumPy isn’t installing on the new version.

                                        Waiting at least a little bit for the ecosystem to catch up is sound advice.

                                        1. 10

                                          I don’t understand why you say that when the article was very clearly a meta-discussion of how to approach Python version upgrades. It is not asking users to hold off indefinitely, but instead is reacting to the availability and how that plays out with updates throughout the ecosystem.

                                          A “product manager” for Python could take a lot away from how clearly the pain points were laid out. As a platform, it’s advantageous for Python to tackle a lot of the issues pointed out, but it’s hard because of the number of stakeholders for things like packages. Getting a Docker image out more quickly seems like low-hanging fruit, but delaying a few days could perhaps be intentional.

                                          1. 2

                                            For what it is worth, the Docker container, like many very popular containers on the official docker registry, are in fact owned and maintained by the Docker community themselves. I am unsure if it is really their duty to-do that.

                                            Many of the listed things in the article are indeed painful things to deal with, but some of them I’m not sure if the PSF is really the right entity to have had them fixed on launch day.

                                            edit: clarified that is the docker community that maintains it, not Docker the corporate entity.

                                            1. 2

                                              Also, as the author suggested it could be, it’s fixed already:

                                              Digest: sha256:05ff1b50a28aaf96f696a1e6cdc2ed2c53e1d03e3a87af402cab23905c8a2df0
                                              Status: Downloaded newer image for python:3.10
                                              Python 3.10.0 (default, Oct  5 2021, 23:39:58) [GCC 10.2.1 20210110] on linux
                                              Type "help", "copyright", "credits" or "license" for more information.
                                              >>> 
                                              

                                              They had to hit publish pretty quickly to release that complaint while it was still true.

                                          2. 3

                                            Some of concerns seem reasonable, for example the tooling catching up with the new pattern matching syntax blocks (match, case). If you use the popular Black code formatter, for example, it doesn’t yet handle pattern matching (and it looks like it’s going to be a bit of a job to update that).

                                          1. 4

                                            Nice article – it was interesting to me that gorilla/mux has “route reversal” (though I’ve never needed that personally). Note that chi also supports host routing: not the package itself, but with the go-chi/hostrouter middleware that’s part of the chi project (it works as stand-alone middleware too).

                                            For an in-depth look at building your own routing using regexp and other types of custom matching, see my article Different approaches to HTTP routing in Go. I also compare to Axel Wagner’s “ShiftPath” technique, and some of the popular router libraries (chi, gorilla, pat).

                                            1. 3

                                              Heh I made something quite similar, with almost the same values (no JS, server rendered HTML). I store the data in two CSV files, one for open, one for completed tasks. Amusingly I also don’t allow editing—if you need to edit, delete and recreate. I called mine Leaf Tasks and implemented it in Rust: https://github.com/wezm/leaf

                                              1. 2

                                                Nice, thanks for sharing. Yes, some very similar design choices! I’ll peruse the code (not that I know Rust).

                                              1. 7

                                                Congratulations on the project and a great blog post.

                                                You mention that you do not love the verbosity of errors, and give an example.

                                                You are not wrapping returned errors, but pass them up the stack as is. This is not a problem only because your application is tiny, database wrapper never does more than a single operation within a method and call stack extremely shallow. If you would have a complex schema and method GetLists would make several queries, not wrapping error would cause issues. You could not tell where the error comes from.

                                                http.Error(w, err.Error(), http.StatusInternalServerError) is probably never wise to do. Why would you leak out internal application information? What if there is some sensitive data in the error string? What should the user do with the error information anyway?

                                                If you want to always return 500 on error (which is a great oversimplification or error handling) you can use a different handler notation together with a wrapper.

                                                func x(fn func(w http.ResponesWriter, r *http.Request) error) http.HandlerFunc {
                                                    return func(w http.ResponesWriter, r *http.Request) {
                                                        if err := fn(w, r); err != nil {
                                                            log.Printf(err)
                                                            w.WriteHeader(http.StatusInternalServerError)
                                                        }
                                                    }
                                                }
                                                

                                                If you will handle your errors correctly (i.e. return ErrNotFound), your can improve x wrapper to return a proper status:

                                                switch err := fn(w, r); {
                                                case err == nil:
                                                    // Respones was written.
                                                case errors.Is(err, ErrNotFound):
                                                    w.WriteHeader(http.StatusNotFound)
                                                default:
                                                    w.WriteHeader(http.StatusInternalServerError)
                                                }
                                                

                                                Another approach could be to make your errors implement http.Handler interface in order to write error information themselves. This is not the direction I would recommend, but since your project is small in scope an experiment might not be bad :)

                                                type errNotFound struct {
                                                     entityName string
                                                }
                                                
                                                func (e errNotFound) Error() string {
                                                    return e.entityName " was not found"
                                                }
                                                
                                                func (e errNotFound) ServeHTTP(w http.ResponseWriter, r *http.Request) {
                                                    w.WriteHeader(http.StatusNotFound)
                                                    fmt.Fprintf(w, "%s was not found", e.entityName)
                                                }
                                                

                                                Since you should wrap your errors, you will need a helper function to extract the right one from a chain.

                                                1. 3

                                                  I wrote a thing about how I handle errors in Go. The TL;DR is I have an error helper that lets me wrap errors in a status code and user message. If an error makes it to the top level handler without having been wrapped in a status code, the code defaults to 500 and the message defaults to Internal Server Error. I think it’s a pretty good system!

                                                  1. 2

                                                    These are good points, and thanks. Yeah, you’re right that I probably shouldn’t return err.Error() directly to the user – it was definitely a shortcut to avoid thinking about it for this project. :-) I’ve updated the post and code to avoid this.

                                                    Regarding passing errors up directly: yes, it’s usually fine in a tiny app because as you say, the call stack is shallow. In languages with exceptions, the stack trace is always attached to the exception, so you get that “for free” (not in compute cost, but in developer time). I know there are libraries for Go which wrap errors and record a stack trace in the same way, but I haven’t used them. In practice for larger apps, I’d use fmt.Errorf("...: %w", err) or other forms of wrapping.

                                                    Nice idea with the wrapper that wraps an error-returning function: I quite like the looks of that pattern. I might even play with it in this project on a branch to see how it works in practice.

                                                    Thanks again for the interaction!

                                                    1. 4

                                                      In practice for larger apps, I’d use fmt.Errorf("...: %w", err) or other forms of wrapping.

                                                      That is definitely the right way to go, and I totally understand why you wouldn’t bother with that for a tiny app!

                                                      In languages with exceptions, the stack trace is always attached to the exception, so you get that “for free”

                                                      That’s true, but it’s only ever a bare stack trace with no useful context. Wrapping errors in the idiomatic Go way encourages developers to add useful context: I find Go error messages are generally much more useful for debugging than stack traces. This is especially true when working with systems across a network, where a stack trace generated on one node would be almost useless.

                                                      There’s a nice description of error handling in Go here: Error handling in Upspin

                                                      1. 4

                                                        The Upspin article is good, but it predates errors.As, so it’s a bit outdated IMO. I wrote a thing that builds on that article specifically about how to create error domains in a post-errors.As world: https://blog.carlmjohnson.net/post/2020/working-with-errors-as/

                                                        1. 2

                                                          That’s an excellent article, thanks for sharing it.

                                                    2. 1

                                                      A list of suggestions:

                                                    1. 10

                                                      Once we move beyond one-liners, a natural question is why. As in ‘Why not use Python? Isn’t it good at this type of thing?’

                                                      The reasons provided are fine, but for me the main reason is speed. AWK is much, much faster than Python for “line at a time” processing. When you have large files, the difference becomes clear. (perl -p can be a reasonable substitute.)

                                                      Once you are writing long AWK programs, though, it’s time to consider Python or something else. AWK isn’t very fun once data manipulation gets complicated.

                                                      1. 4

                                                        (perl -p can be a reasonable substitute.)

                                                        +1. In my eyes, it’s Awk and then Perl. Perl turns out to be much better for these purposes than other scripting languages. The difference in startup time between Perl and Python is very significant. If you don’t use (m)any modules, Perl scripts usually start just as quickly as Awk scripts.

                                                        1. 2

                                                          I’m sure that’s true for some kinds of scripts, but that doesn’t match my experience/benchmarks here (Python is somewhat faster than AWK for this case of counting unique words). For what programs did you find AWK “much, much faster”? I can imagine very small datasets being faster in AWK because it’s startup time is 3ms compared to Python’s 20ms.

                                                          1. 2

                                                            For what programs did you find AWK “much, much faster”?

                                                            Any time the input file is big. As in hundreds of MGs big.

                                                            I used to have to process 2GB+ of CSV on a regular basis and the AWK version was easily 5x faster than the Python version.

                                                            1. 1

                                                              Was the Python version streaming, or did it read the whole file in at once?

                                                              1. 1

                                                                Streaming.

                                                            2. 2

                                                              Regarding your results, 3.55 under awk is with or without -b?

                                                              I get 1.774s (simple) and 1.136s (optimized) for Python. For simple awk, I get 2.552s (without -b) 1.537s (with -b). For optimized, I get 2.091s and 1.435s respectively. I’m using gawk here, mawk is of course faster.

                                                              Also, I’ve noticed that awk does poorly when there are large number of dictionary keys. If you are doing field based decisions, awk is likely to be much faster. I tried printing first field of each line (removed empty lines from your test file since line.split()[0] gives error for empty lines). I got 0.583s for Python compared to 0.176s (without -b) and 0.158s (with -b)

                                                              1. 4

                                                                Also, I’ve noticed that awk does poorly when there are large number of dictionary keys.

                                                                Same here. If you are making extensive use of arrays, then AWK may not be the best tool.

                                                            3. 2

                                                              Once you are writing long AWK programs, though, it’s time to consider Python or something else. AWK isn’t very fun once data manipulation gets complicated.

                                                              I dunno, I think it’s pretty fun.

                                                              I am consistently surprised that there aren’t more tools that support AWK-style “record oriented programming” (since a record need not be a line, if you change the record separator). I found this for Go, but that’s about it. This style of data interpretation comes up pretty often in my experience. I feel like as great as AWK is, we could do better - for example, what about something like AWK that can read directly from CSV (with proper support for quoting), assigning each row to a record, and perhaps with more natural support for headers.

                                                              1. 2

                                                                You are right. Recently I was mixing AWK and Python in a way that AWK was producing key,value output easily readable and processed later by Python script. Nice, simple and quick to develop.

                                                              1. 11

                                                                So, for example, if you want to write a function that modifies a slice, you might instinctively write

                                                                func modify[T any](in []T) []T
                                                                

                                                                This will take in any slice, but if you define a type like type MySlice []int when you call modify with a MySlice, you will get back a plain []int. The answer is to instead write

                                                                func modify[S constraints.Slice[T], T any](in S) S
                                                                

                                                                This is significantly more of a pain in the ass to write, so I think a lot of code that doesn’t need to work with named types will end up using the simpler []T syntax instead.

                                                                1. 5

                                                                  Agreed! However, there’s an open proposal under consideration that makes this a lot nicer, so hopefully it gets accepted. It will let you write constraints.Slice[T] as just ~[]T inline, so will look like this for your second version:

                                                                  func modify[S ~[]T, T any](in S) S
                                                                  
                                                                  1. 3

                                                                    I read the earlier version of that proposal and was very skeptical of it, but this version looks good. The point about interface{X} being equivalent to interface{interface{X}} is convincing that it’s a good idea.

                                                                    1. 2

                                                                      Interesting, thanks for the link!

                                                                      To clarify a bit for people (like me) who are roughly aware of the coming generics proposal in Go, but not deep in the trenches of all the discussions, syntax, and recent changes — IIUC:

                                                                      • func modify[S constraints.Slice[T], T any](in S) S — under other currently accepted (?) proposals is a “more ergonomic wrapper” (?) for arguably more verbose (yet IIUC syntactically correct as well?) syntax as below:
                                                                      • func modify[S interface{~[]T}, T any](in S) S; whereas:
                                                                      • func modify[S ~[]T, T any](in S) S — is the currently proposed shorter syntax for the above “expanded/verbose” syntax (in accordance with the current title of the proposal being: “allow eliding interface{ } in constraint literals”).

                                                                      Notably, the new proposal, after a tweak making it what it I describe above (vs. an earlier version), quickly got active interest from the Core Team, with a CL by them being promised, which I see as a good (though not 100%) prognostic for a solid chance of it being accepted, in this way or another (in particular, possibly leading to some other, surprising yet quite cool, improvement).

                                                                  1. 17

                                                                    Pattern matching has been available in functional programming languages for decades now, it was introduced in the 70s. (Logic programming languages expose even more expressive forms, at higher runtime cost.) It obviously improves readability of code manipulating symbolic expressions/trees, and there is a lot of code like this. I find it surprising that in the 2020s there are still people wondering whether “the feature provides enough value to justify its complexity”.

                                                                    (The fact that Python did without for so long was rather a sign of closed-mindedness of its designer subgroup. The same applies, in my opinion, to languages (including Python, Go, etc.) that still don’t have proper support for disjoint union types / variants / sums / sealed case classes.)

                                                                    1. 45

                                                                      Pretty much every feature that has ever been added to every language ever is useful in some way. You can leave a comment like this on almost any feature that a language may not want to implement for one reason or the other.

                                                                      1. 14

                                                                        I think it makes more sense in statically typed languages, especially functional ones. That said, languages make different choices. For me, Python has always been about simplicity and readability, and as I’ve tried to show in the article, at least in Python, structural pattern matching is only useful in a relatively few cases. But it’s also a question of taste: I really value the simplicity of the Go language (and C before it), and don’t mind a little bit of verbosity if it makes things clearer and simpler. I did some Scala for a while, and I can see how people like the “power” of it, but the learning curve of its type system was very steep, and there were so many different ways to do things (not to mention the compiler was very slow, partly because of the very complex type system).

                                                                        1. 22

                                                                          For the record, pattern-matching was developed mostly in dynamically-typed languages before being adopted in statically-typed languages, and it works just as well in a dynamically-typed world. (In the ML-family world, sum types and pattern-matching were introduced by Hope, an experimental dynamically-typed language; in the logic world, they are basic constructs of Prolog, which is also dynamically-typed – although some more-typed dialects exist.)

                                                                          as I’ve tried to show in the article, at least in Python, structural pattern matching is only useful in a relatively few cases

                                                                          Out of the 4 cases you describe in the tutorial, I believe your description of two of them is overly advantageous to if..elif:

                                                                          • In the match event.get() case, the example you show is a variation of the original example (the longer of the three such examples in the tutorial), and the change you made makes it easier to write an equivalent if..elif version, because you integrated a case (from another version) that ignores all other Click() events. Without this case (as in the original tutorial example), rewriting with if..elif is harder, you need to duplicate the failure case.
                                                                          • In the eval_expr example, you consider the two versions as readable, but the pattern-version is much easier to maintain. Consider, for example, supporting operations with 4 or 5 parameters, or adding an extra parameter to an existing operator; it’s an easy change with the pattern-matching version, and requires boilerplate-y, non-local transformations with if..elif. These may be uncommon needs for standard mathematical operations, but they are very common when working with other domain-specific languages.
                                                                          1. 1

                                                                            the change you made makes it easier to write an equivalent if..elif version

                                                                            Sorry if it appeared that way – that was certainly not my intention. I’m not quite sure what you mean, though. The first/original event example in the tutorial handles all click events with no filtering using the same code path, so it’s even simpler to convert. I added the Button.LEFT filtering from a subsequent example to give it a bit more interest so it wasn’t quite so simple. I might be missing something, though.

                                                                            In the eval_expr example, you consider the two versions as readable, but the pattern-version is much easier to maintain. Consider, for example, supporting operations with 4 or 5 parameters, or adding an extra parameter to an existing operator;

                                                                            I think those examples are very hypothetical – as you indicate, binary and unary operators aren’t suddenly going to support 4 or 5 parameters. A new operation might, but that’s okay. The only line that’s slightly repetitive is the “attribute unpacking”: w, x, y, z = expr.w, expr.x, expr.y, expr.z.

                                                                            These may be uncommon needs for standard mathematical operations, but they are very common when working with other domain-specific languages.

                                                                            You’re right, and that’s part of my point. Python isn’t used for implementing compilers or interpreters all that often. That’s where I’m coming from when I ask, “does the feature provide enough value to justify the complexity?” If 90% of Python developers will only rarely use this complex feature, does it make sense to add it to the language?

                                                                            1. 3

                                                                              that was certainly not my intention.

                                                                              To be clear, I’m not suggesting that the change was intentional or sneaky, I’m just pointing out that the translation would be more subtle.

                                                                              The first/original event example does not ignore “all other Click events” (there is no Click() case), and therefore an accurate if..elif translation would have to do things differently if there is no position field or if it’s not a pair, namely it would have to fall back to the ValueError case.

                                                                              You’re right, and that’s part of my point. Python isn’t used for implementing compilers or interpreters all that often.

                                                                              You don’t need to implement a compiler for C or Java, or anything people recognize as a programming language (or HTML or CSS, etc.), to be dealing with a domain-specific languages. Many problem domains contain pieces of data that are effectively expressions in some DSL, and recognizing this can very helpful to write programs in those domains – if the language supports the right features to make this convenient. For example:

                                                                              • to start with the obvious, many programs start by interpreting some configuration file to influence their behavior; many programs have simple needs well-served by linear formats, but many programs (eg. cron jobs, etc.) require more elaborate configurations that are DSL-like. Even if the configuration is written in some standard format (INI, Yaml, etc.) – so parsing can be delegated to a library – the programmer will still write code to interpret or analyze the configuration data.
                                                                              • more gnerally, “structured data formats” are often DSL-shaped; ingesting structured data is something we do super-often in programs
                                                                              • programs that offer a “query” capability typically provide a small language to express those queries
                                                                              • events in an event loop typically form a small language
                                                                          2. 14

                                                                            I think it makes more sense in statically typed languages, especially functional ones.

                                                                            In addition to the earlier ones gasche mentioned (it’s important to remember this history), it’s used to pervasively in Erlang, and later Elixir. Clojure has core.match, Racket has match, as does Guile. It’s now in Ruby as well!

                                                                            1. 3

                                                                              Thanks! I didn’t know that. I have used pattern matching in statically typed language (mostly Scala), and had seen it in the likes of Haskell and OCaml, so I’d incorrectly assumed it was mainly a statically-typed language thing.

                                                                              1. 1

                                                                                It is an important feature of OCaml.

                                                                                1. 3

                                                                                  I am aware - was focusing on dynamically typed languages.

                                                                              2. 7

                                                                                For me, it is the combination of algebraic data types + pattern matching + compile time exhaustiveness checking that is the real game changer. With just 1 out of 3, pattern matching in Python is much less compelling.

                                                                                1. 1

                                                                                  I agree. I wonder if they plan to add exhaustiveness checking to mypy. The way the PEP is so no hold barred makes it seem like the goal was featurefulness and not an attempt to support exhaustiveness checking.

                                                                                  1. 2

                                                                                    I wonder if they plan to add exhaustiveness checking to mypy.

                                                                                    I don’t think that’s possible in the general case. If I understand the PEP correctly, __match_args__ may be a @property getter method, which could read the contents of a file, or perform a network request, etc.

                                                                              3. 11

                                                                                I find it surprising that in the 2020s there are still people wondering whether “the feature provides enough value to justify its complexity”.

                                                                                I find it surprising that people find this surprising.

                                                                                Adding features like pattern matching isn’t trivial, and adding it too hastily can backfire in the long term; especially for an established language like Python. As such I would prefer a language take their time, rather than slapping things on because somebody on the internet said it was a good idea.

                                                                                1. 3

                                                                                  That’s always been the Scheme philosophy:

                                                                                  Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary.

                                                                                  And indeed, this pays off: in the Scheme world, there’s been a match package floating around for a long time, implemented simply as a macro. No changes to the core language needed.

                                                                                  1. 4

                                                                                    No changes to the core language needed.

                                                                                    I’m sure you recognize that this situation does not translate to other languages like in this case Python. Implementing it as a macro is just not feasible. And even in Scheme the usage of match macros is rather low. This can be because it is not that useful, but also might be because of the hurdle of adding dependencies is not worth the payoff. Once a feature is integrated in a language, its usage “costs” nothing, thus the value proposition when writing code can be quite different.

                                                                                    1. 7

                                                                                      This is rather unrelated to the overall discussion, but as a user of the match macros in Scheme, I must say that I find the lack of integration into the base forms slightly annoying. You cannot pattern-match on a let or lambda, you have to use match-let and match-lambda, define/match (the latter only in Racket I think), etc. This makes reaching for pattern-matching feel heavier, and it may be a partial cause to their comparatively lower usage. ML-family languages generalize all binding positions to accept patterns, which is very nice to decompose records for example (or other single-case data structures). I wish Scheme dialects would embrace this generalization, but they haven’t for now – at least not Racket or Clojure.

                                                                                      1. 2

                                                                                        In the case of Clojure while it doesn’t have pattern matching built-in, it does have quite comprehensive destructuring forms (like nested matching in maps, with rather elaborate mechanisms) that works in all binding positions.

                                                                                        1. 2

                                                                                          Nice! I suppose (from your post above) that pattern-matching is somehow “integrated” in the Clojure implementation, rather than just being part of the base macro layer that all users see.

                                                                                          1. 2

                                                                                            I think the case is that Clojure core special forms support it (I suppose the implementation itself is here and called “binding-forms”, which is then used by let, fn and loop which user defined macros often end up expanding to). Thus it is somewhat under the base layer that people use.

                                                                                            But bear in mind this is destructuring, in a more general manner than what Python 2.x already supported, not pattern matching. It also tends to get messy with deep destructuring, but the same can be said of deep pattern matches through multiple layers of constructors.

                                                                                2. 8

                                                                                  I agree about pattern matching and Python in general. It’s depressing how many features have died in python-ideas because it takes more than a few seconds for an established programmer to grok them. Function composition comes to mind.

                                                                                  But I think Python might be too complicated for pattern matching. The mechanism they’ve settled on is pretty gnarly. I wrote a thing for pattern matching regexps to see how it’d turn out (admittedly against an early version of the PEP; I haven’t checked it against the current state) and I think the results speak for themselves.

                                                                                  1. 6

                                                                                    But I think Python might be too complicated for pattern matching. The mechanism they’ve settled on is pretty gnarly.

                                                                                    I mostly agree. I generally like pattern matching and have been excited about this feature, but am still feeling out exactly when I’ll use it and how it lines up with my intuition.

                                                                                    The part that does feel very Pythonic is that destructuring/unpacking is already pretty pervasive in Python. Not only for basic assignments, but also integrated into control flow constructs. For example, it’s idiomatic to do something like:

                                                                                    for key, val in some_dictionary.items():
                                                                                        # ...
                                                                                    

                                                                                    Rather than:

                                                                                    for item in some_dictionary.items():
                                                                                        key, val = item
                                                                                        # ...
                                                                                    

                                                                                    Or something even worse, like explicit item[0] and item[1]. So the lack of a conditional-with-destructuring, the way we already have foreach-with-destructuring, did seem like a real gap to me, making you have to write the moral equivalent of code that looks more like the 2nd case than the 1st. That hole is now filled by pattern matching. But I agree there are pitfalls around how all these features interact.

                                                                                  2. 2
                                                                                    for i, (k, v) in enumerate(d.items(), 1): pass
                                                                                    

                                                                                    looks like pattern matching to me

                                                                                    1. 2

                                                                                      Go aims for simplicity of maintenance and deployment. It doesn’t “still don’t have those features”. The Go authors avoided them on purpose. If you want endless abstractions in Go, embedding Lisp is a possibilty: https://github.com/glycerine/zygomys

                                                                                      1. 5

                                                                                        Disjoint sums are a basic programming feature (it models data whose shape is “either this or that or that other thing”, which ubiquitous in the wild just like pairs/records/structs). It is not an “endless abstraction”, and it is perfectly compatible with maintenance and deployment. Go is a nice language in some respects, the runtime is excellent, the tooling is impressive, etc etc. But this is no rational excuse for the lack of some basic language features.

                                                                                        We are in the 2020s, there is no excuse for lacking support for sum types and/or pattern matching. Those features have been available for 30 years, their implementation is well-understood, they require no specific runtime support, and they are useful in basically all problem domains.

                                                                                        I’m not trying to bash a language and attract defensive reactions, but rather to discuss (with concrete examples) the fact that language designer’s mindsets can be influenced by some design cultures more than others, and as a result sometimes the design is held back by a lack of interest for things they are unfamiliar with. Not everyone is fortunate to be working with a deeply knowledgeable and curious language designer, such as Graydon Hoare; we need more such people in our language design teams. The default is for people to keep working on what they know; this sort of closed-ecosystem evolution can lead to beautiful ideas (some bits of Perl 6 for example are very nice!), but it can also hold back.

                                                                                        1. 3

                                                                                          But this is no rational excuse for the lack of some basic language features.

                                                                                          Yes there is. Everyone has a favorite feature, and if all of those are implemented, there would easily be feature bloat, long build times and projects with too many dependencies that depend on too many dependencies, like in C++.

                                                                                          In my opinion, the question is not if a language lacks a feature that someone wants or not, but if it’s usable for goals that people wish to achieve, and Go is clearly suitable for many goals.

                                                                                      2. 3

                                                                                        Ah yes, Python is famously closed-minded and hateful toward useful features. For example, they’d never adopt something like, say, list comprehensions. The language’s leaders are far too closed-minded, and dogmatically unwilling to ever consider superior ideas, to pick up something like that. Same for any sort of ability to work with lazy iterables, or do useful combinatoric work with them. That’s something that definitely will never be adopted into Python due to the closed-mindedness of its leaders. And don’t get me started on basic FP building blocks like map and folds. It’s well known that Guido hates them so much that they’re permanently forbidden from ever being in the language!

                                                                                        (the fact that Python is not Lisp was always unforgivable to many people; the fact that it is not Haskell has now apparently overtaken that on the list of irredeemable sins; yet somehow we Python programmers continue to get useful work done and shrug off the sneers and insults of our self-proclaimed betters much as we always have)

                                                                                        1. 25

                                                                                          It is well-documented that Guido Van Rossum planned to remove lambda from Python 3. (For the record, I agree that map and filter on lists are much less useful in presence of list comprehensions.) It is also well-documented that recursion is severely limited in Python, making many elegant definitions impractical.

                                                                                          Sure, Python adopted (in 2000 I believe?) list comprehensions from ABC (due to Guido working with the language in the 1980s), and a couple of library-definable iterators. I don’t think this contradicts my claim. New ideas came to the language since (generators, decorators), but it remains notable that the language seems to have resisted incorporating strong ideas from other languages. (More so than, say, Ruby, C#, Kotlin, etc.)

                                                                                          Meta: One aspect of your post that I find unpleasant is the tone. You speak of “sneers and insults”, but it is your post that is highly sarcastic and full of stray exagerations at this or that language community. I’m not interested in escalating in this direction.

                                                                                          1. 7

                                                                                            less useful in presence of list comprehension

                                                                                            I’m certainly biased, but I find Python’s list comprehension an abomination towards readability in comparison to higher-order pipelines or recursion. I’ve not personally coded Python in 8-9 years, but when I see examples, I feel like I need to put my head on upsidedown to understand it.

                                                                                            1. 6

                                                                                              It is also well-documented that recursion is severely limited in Python, making many elegant definitions impractical.

                                                                                              For a subjective definition of “elegant”. But this basically is just “Python is not Lisp” (or more specifically, “Python is not Scheme”). And that’s OK. Not every language has to have Scheme’s approach to programming, and Scheme’s history has shown that maybe it’s a good thing for other languages not to be Scheme, since Scheme has been badly held back by its community’s insistence that tail-recursive implementations of algorithms should be the only implementations of those algorithms.

                                                                                              You speak of “sneers and insults”, but it is your post that is highly sarcastic and full of stray exagerations at this or that language community.

                                                                                              Your original comment started from a place of assuming – and there really is no other way to read it! – that the programming patterns you care about are objectively superior to other patterns, that languages which do not adopt those patterns are inherently inferior, and that the only reason why a language would not adopt them is due to “closed-mindedness”. Nowhere in your comment is there room for the (ironically) open-minded possibility that someone else might look at patterns you personally subjectively love, evaluate them rationally, and come to a different conclusion than you did – rather, you assume that people who disagree with your stance must be doing so because of personal faults on their part.

                                                                                              And, well, like I said we’ve got decades of experience of people looking down their noses at Python and/or its core team + community for not becoming a copy of their preferred languages. Your comment really is just another instance of that.

                                                                                              1. 8

                                                                                                I’m not specifically pointing out the lack of tail-call optimization (TCO) in Python (which I think is unfortunate indeed; the main argument is that call stack matters, but it’s technically fully possible to preserve call stacks on the side with TC-optimizing implementations). Ignoring TCO for a minute, the main problem would be the fact that the CPython interpreter severely limits the call space (iirc it’s 1K calls by default; compare that to the 8Mb default on most Unix systems), making recursion mostly unusable in practice, except for logarithmic-space algorithms (balanced trees, etc.).

                                                                                                Scheme has been badly held back by its community’s insistence that tail-recursive implementations of algorithms should be the only implementations of those algorithms.

                                                                                                I’m not sure what you mean – that does not make any sense to me.

                                                                                                [you assume] that the programming patterns you care about are objectively superior to other patterns [..]

                                                                                                Well, I claimed

                                                                                                [pattern matching] obviously improves readability of code manipulating symbolic expressions/trees

                                                                                                and I stand by this rather modest claim, which I believe is an objective statement. In fact it is supported quite well by the blog post that this comment thread is about. (Pattern-matching combines very well with static typing, and it will be interesting to see what Python typers make of it; but its benefits are already evident in a dynamically-typed context.)

                                                                                                1. 4

                                                                                                  and I stand by this rather modest claim, which I believe is an objective statement.

                                                                                                  Nit: I don’t think you can have an objective statement of value.

                                                                                                  1. 4

                                                                                                    Again: your original comment admits of no other interpretation than that you do not believe anyone could rationally look at the feature you like and come to a different conclusion about it. Thus you had to resort to trying to find personal fault in anyone who did.

                                                                                                    This does not indicate “closed-mindedness” on the part of others. They may prioritize things differently than you do. They may take different views of complexity and tradeoffs (which are the core of any new language-feature proposal) than you do. Or perhaps they simply do not like the feature as much as you do. But you were unwilling to allow for this — if someone didn’t agree with your stance it must be due to personal fault. You allowed for no other explanation.

                                                                                                    That is a problem. And from someone who’s used to seeing that sort of attitude it will get you a dismissive “here we go again”. Which is exactly what you got.

                                                                                                2. 4

                                                                                                  This is perhaps more of a feeling, but saying that Rust isn’t adopting features as quickly as Ruby seems a bit off. Static type adoption in the Python community has been quicker. async/await has been painful, but is being attempted. Stuff like generalized unpacking (and this!) is also shipping out!

                                                                                                  Maybe it can be faster, but honestly Python probably has one of the lowest “funding amount relative to impact” of the modern languages which makes the whole project not be able to just get things done as quickly IMO.

                                                                                                  Python is truly in a funny place, where many people loudly complain about it not adopting enough features, and many other loudly complain about it loudly adopting too many! It’s of course “different people have different opinions” but still funny to see all on the same page.

                                                                                                  1. 3

                                                                                                    It is well-documented that Guido Van Rossum planned to remove lambda from Python 3

                                                                                                    Thank you for sharing that document. I think Guido was right: it’s not pythonic to map, nor to use lambdas in most cases.

                                                                                                    Every feature is useful, but some ecosystems work better without certain features. I’m not sure where go’s generics fall on this spectrum, but I’m sure most proposed features for python move it away from it’s core competency, rather than augmenting a strong core.

                                                                                                    1. 1

                                                                                                      We have previously discussed their tone problem. It comes from their political position within the Python ecosystem and they’re relatively blind to it. Just try to stay cool, I suppose?

                                                                                                      1. 6

                                                                                                        I really do recommend clicking through to that link, and seeing just what an unbelievably awful thing I said that the user above called out as “emblematic” of the “contempt” I display to Python users. Or the horrific ulterior motive I was found to have further down.

                                                                                                        Please, though, before clicking through, shield the eyes of children and anyone else who might be affected by seeing such content.

                                                                                                    2. 5

                                                                                                      To pick one of my favorite examples, I talked to the author of PEP 498 after a presentation that they gave on f-strings, and asked why they did not add destructuring for f-strings, as well as whether they knew about customizeable template literals in ECMAScript, which trace their lineage through quasiliterals in E all the way back to quasiquotation in formal logic. The author knew of all of this history too, but told me that they were unable to convince CPython’s core developers to adopt any of the more advanced language features because they were not seen as useful.

                                                                                                      I think that this perspective is the one which might help you understand. Where you see one new feature in PEP 498, I see three missing subfeatures. Where you see itertools as a successful borrowing of many different ideas from many different languages, I see a failure to embrace the arrays and tacit programming of APL and K, and a lack of pattern-matching and custom operators compared to Haskell and SML.

                                                                                                    3. 1

                                                                                                      I think the issue is more about pattern matching being a late addition to Python, which means there will be lots of code floating around that isn’t using match expressions. Since it’s not realistic to expect this code to be ported, the old style if … elif will continue to live on. All of this adds up to a larger language surface area, which makes tool support, learning and consistency more difficult.

                                                                                                      I’m not really a big fan of this “pile of features” style of language design - if you add something I’d prefer if something got taken away as well. Otherwise you’ll end up with something like Perl 5

                                                                                                    1. 35

                                                                                                      return err is almost always the wrong thing to do. Instead of:

                                                                                                      if err := foo(); err != nil {
                                                                                                      	return err
                                                                                                      }
                                                                                                      

                                                                                                      Write:

                                                                                                      if err := foo(); err != nil {
                                                                                                      	return fmt.Errorf("fooing: %w", err)
                                                                                                      }
                                                                                                      

                                                                                                      Yes, this is even more verbose, but doing this is what makes error messages actually useful. Deciding what to put in the error message requires meaningful thought and cannot be adequately automated. Furthermore, stack traces are not adequate context for user-facing, non-programming errors. They are verbose, leak implementation details, are disrupted by any form of indirection or concurrency, etc.

                                                                                                      Even with proper context, lots of error paths like this is potentially a code smell. It means you probably have broader error strategy problems. I’d try to give some advice on how to improve the code the author provided, but it is too abstract in order to provide any useful insights.

                                                                                                      1. 18

                                                                                                        I disagree on a higher level. What we really want is a stacktrace so we know where the error originated, not manually dispensed breadcrumbs…

                                                                                                        1. 32

                                                                                                          maybe you do, but I prefer an error chain that was designed. A Go program rarely has just one stack, because every goroutine is its own stack. Having the trace of just that one stack isn’t really a statement of the program as a whole since there’s many stacks, not one. Additionally, stack traces omit the parameters to the functions at each frame, which means that understanding the error means starting with your stack trace, and then bouncing all over your code and reading the code and running it in your head in order to understand your stack trace. This is even more annoying if you’re looking at an error several days later in a heterogeneous environment where you may need the additional complication of having to figure out which version of the code was running when that trace originated. Or you could just have an error like “failed to create a room: unable to reserve room in database ‘database-name’: request timed out” or something similar. Additionally, hand-crafted error chains have the effect that they are often much easier to understand for people who operate but don’t author something; they may have never seen the code before, so understanding what a stack trace means exactly may be difficult for them, especially if they’re not familiar with the language.

                                                                                                          1. 6

                                                                                                            I dunno. Erlang and related languages give you back a stack trace (with parameters) in concurrently running processes no problem

                                                                                                            1. 5

                                                                                                              It’s been ages since I wrote Erlang, but I remember that back then I rarely wanted a stack trace. My stack were typically 1-2 levels deep: each process had a single function that dispatched messages and did a small amount of work in each one. The thing that I wanted was the state of the process that had sent the unexpected message. I ended up with some debugging modes that attached the PID of the sending process and some other information so that I could reconstruct the state at the point where the problem occurred. This is almost the same situation as Go, where you don’t want the stack trace of the goroutine, you want to capture a stack trace of the program at the point where a goroutine was created and inspect that at the point where the goroutine failed.

                                                                                                              This isn’t specific to concurrent programs, though it is more common there, it’s similar for anything written in a dataflow / pipeline style. For example, when I’m debugging something in clang’s IR generation I often wish I could go back and see what had caused that particular AST node to be constructed during parsing or semantic analysis. I can’t because all of the state associated with that stack is long gone.

                                                                                                          2. 10

                                                                                                            FWIW, I wrote a helper that adds tracing information.

                                                                                                            I sort of have two minds about this. On the one hand, yeah, computers are good at tracking stack traces, why are we adding them manually and sporadically? OTOH, it’s nice that you can decide if you want the traces or not and it gives you the ability to do higher level things like using errors as response codes and whatnot.

                                                                                                            The thing that I have read about in Zig that I wish Go had is an error trace which is different from the stack trace, which shows how the error was created, not the how the error propagates back to the execution error boundary which is not very interesting in most scenarios.

                                                                                                            1. 7

                                                                                                              The nice thing about those error traces is that they end where the stack trace begins, so it’s seamless to the point that you don’t even need to know that they are a thing, you just get exactly the information that otherwise you would be manually looking for.

                                                                                                            2. 8

                                                                                                              In a multiprocess system that’s exchanging messages: which stack?

                                                                                                              1. 2

                                                                                                                see: erlang

                                                                                                              2. 5

                                                                                                                You don’t want stack traces; you want to know what went wrong.

                                                                                                                A stack trace can suggest what may have gone wrong, but an error message that declares exactly what went wrong is far more valuable, no?

                                                                                                                1. 8

                                                                                                                  An error message is easy, we already have that: “i/o timeout”. A stack trace tells me the exact code path that lead to that error. Building up a string of breadcrumbs that led to that timeout is just a poorly implemented, ad-hoc stack trace.

                                                                                                                  1. 5

                                                                                                                    Indeed and I wouldn’t argue with that. I love a good stack trace, but I find they’re often relied upon in lieu of useful error messages and I think that’s a problem.

                                                                                                                    1. 2

                                                                                                                      Building up a string of breadcrumbs that led to that timeout is just a poorly implemented, ad-hoc stack trace.

                                                                                                                      That’s a bit of an over-generalization. A stack trace is inherently a story about the construction of the program that originated the error, while an error chain is a story about the events that led to an error. A stack trace can’t tell you what went wrong if you don’t have access to the program’s source code in the way that a hand crafted error chain can. A stack trace is more about where an error occurred, while an error chain is more about why an error occurred. I think they’re much more distinct than you are suggesting.

                                                                                                                      and of course, if people are just bubbling up errors without wrapping them, yeah you’re going to have a bad time, but I think attacking that case is like suggesting that every language that has exceptions encourages Pokémon exception handling. That’s a bad exception-handling pattern, but I don’t think that the possibility of this pattern is a fair indictment of exceptions generally. Meanwhile you’re using examples of bad error handling practices that are not usually employed by Go programmers with more than a few weeks experience to indict the entire paradigm.

                                                                                                                  2. 4

                                                                                                                    Stack traces are expensive to compute and inappropriate to display to most users. Also, errors aren’t exceptions.

                                                                                                                    1. 1

                                                                                                                      That’s why Swift throws errors instead. Exceptions immediately abort the program.

                                                                                                                    2. 3

                                                                                                                      What really is the “origin” of an error? Isn’t that somewhat arbitrary? If the error comes from a system call, isn’t the origin deeper in the kernel somewhere? What if you call in to a remote, 3rd party service. Do you want the client to get the stack trace with references to the service’s private code? If you’re using an interface, presumably the purpose is to abstract over the specific implementation. Maybe the stack trace should be truncated at the boundary like a kernel call or API call?

                                                                                                                      Stack traces are inherently an encapsulation violation. They can be useful for debugging your internals, but they are an anti-feature for your users debugging their own system. If your user sees a stack trace, that means your program is bugged, not theirs.

                                                                                                                      1. 5

                                                                                                                        I get a line of logging output: error: i/o timeout. What do I do with that? With Ruby, I get a stack trace which tells me exactly where the timeout came from, giving me a huge lead on debugging the issue.

                                                                                                                        1. 6

                                                                                                                          I get a line of logging output: error: i/o timeout. What do I do with that?

                                                                                                                          Well, that’s a problem you fix by annotating your errors properly. You don’t need stack traces.

                                                                                                                          1. 3

                                                                                                                            When your Ruby service returns an HTTP 500, do you send me the stack trace in the response body? What do I do with that?

                                                                                                                            Go will produce stack traces on panics as well, but that’s precisely the point here: these are two different things. Panics capture stack traces as a “better than nothing” breadcrumb trail for when the programmer has failed to account for a possibility. They are for producers of code, not consumers of it.

                                                                                                                          2. 2

                                                                                                                            There’s definitely competing needs between different audiences and environments here.

                                                                                                                            A non-technical end user doesn’t want to see anything past “something went wrong on our end, but we’re aware of it”. Well, they don’t even want to see that.

                                                                                                                            A developer wants to see the entire stack trace, or at least have it available. They probably only care about frames in their own code at first, and maybe will want to delve into library code if the error truly doesn’t seem to come from their code or is hard to understand in the first place.

                                                                                                                            A technical end user might want to see something in-between: they don’t want to see “something was wrong”. They might not even want to see solely the outer error of “something went wrong while persisting data” if the root cause was “I couldn’t reach this host”, because the latter is something they could actually debug within their environment.

                                                                                                                        2. 9

                                                                                                                          This is one reason I haven’t gone back to Go since university - There’s no right way to do anything. I think I’ve seen a thousand different right ways to return errors.

                                                                                                                          1. 10

                                                                                                                            Lots of pundits say lots of stuff. One good way to learn good patterns (I won’t call them “right”), is to look at real code by experienced Go developers. For instance, if you look at https://github.com/tailscale/tailscale you’ll find pervasive use of fmt.Errorf. One thing you might not see – at least not without careful study – is how to handle code with lots of error paths. That is by it’s very nature harder to see because you have to read and understand what the code is trying to do and what has to happen when something goes wrong in that specific situation.

                                                                                                                            1. 6

                                                                                                                              there is a right way to do most things; but it takes some context and understanding for why.

                                                                                                                              the mistake is thinking go is approachable for beginners; it’s not.

                                                                                                                              go is an ergonomic joy for people that spend a lot of time investing in it, or bring a ton of context from other languages.

                                                                                                                              for beginners with little context, it is definitely a mess.

                                                                                                                              1. 9

                                                                                                                                I thought Go was for beginners, because Rob Pike doesn’t trust programmers to be good.

                                                                                                                                1. 19

                                                                                                                                  I’d assume that Rob Pike, an industry veteran, probably has excellent insight into precisely how good the average programmer at Google is, and what kind of language will enable them to be productive at the stuff Google makes. If this makes programming languages connaisseurs sad, that’s not his problem.

                                                                                                                                  1. 9

                                                                                                                                    Here’s the actual quote:

                                                                                                                                    The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                                                                                                                    So I have to wonder who is capable of understanding a “brilliant language” …

                                                                                                                                    1. 8

                                                                                                                                      So I have to wonder who is capable of understanding a “brilliant language” …

                                                                                                                                      Many people. They don’t work at Google at an entry-level capacity, that’s all.

                                                                                                                                      There’s a subtle fallacy at work here - Google makes a lot of money, so Google can afford to employ smart people (like Rob Pike!) It does not follow that everyone who works at Google is, on average, smarter than anyone else.

                                                                                                                                      (edited to include quote)

                                                                                                                                      1. 8

                                                                                                                                        Let’s say concretely we are talking about OCaml. Surely entry-level Googlers are capable of understanding OCaml. Jane Street teaches it to all new hires (devs or not) in a two-week bootcamp. I’ve heard stories of people quickly becoming productive in Elm too.

                                                                                                                                        The real meaning of that quote is not ‘entry-level Googlers are not capable of it’, it’s ‘We don’t trust them with it’ and ‘We’re not willing to invest in training them in it’. They want people to start banging out code almost instantly, not take some time to ramp up.

                                                                                                                                        1. 8

                                                                                                                                          Let’s say concretely we are talking about OCaml. Surely entry-level Googlers are capable of understanding OCaml. Jane Street teaches it to all new hires (devs or not) in a two-week bootcamp.

                                                                                                                                          I suspect that Jane Street’s hiring selects for people who are capable of understanding OCaml; I guarantee that the inverse happens and applicants interested in OCaml self select for careers at Jane Street, just like Erlang-ers used to flock towards Ericsson.

                                                                                                                                          Google has two orders of magnitude more employees than Jane Street. It needs a much bigger funnel and is likely far less selective in hiring. Go is “the law of large numbers” manifest as a programming language. That’s not necessarily bad, just something that is important for a massive software company and far less important for small boutiques.

                                                                                                                                          1. 3

                                                                                                                                            And I remember when Google would require at minimum a Masters Degree before hiring.

                                                                                                                                            1. 1

                                                                                                                                              I had a master’s degree in engineering (though not in CS) and I couldn’t code my way out of a paper bag when I graduated. Thankfully no-one cared in Dot Com Bubble 1.0!

                                                                                                                                            2. 2

                                                                                                                                              applicants interested in OCaml self select for careers at Jane Street,

                                                                                                                                              As I said, they teach it to all hires, including non-devs.

                                                                                                                                              Google has two orders of magnitude more employees than Jane Street. It needs a much bigger funnel and is likely far less selective in hiring

                                                                                                                                              Surely though, they are not so loose that they hire Tom Dick and Harry off the street. Why don’t we actually look at an actual listing and check? E.g. https://careers.google.com/jobs/results/115367821606560454-software-developer-intern-bachelors-summer-2022/

                                                                                                                                              Job title: Software Developer Intern, Bachelors, Summer 2022 (not exactly senior level)

                                                                                                                                              Minimum qualifications:

                                                                                                                                              Pursuing a Bachelor’s degree program or post secondary or training experience with a focus on subjects in software development or other technical related field. Experience in Software Development and coding in a general purpose programming language. Experience coding in two of C, C++, Java, JavaScript, Python or similar.

                                                                                                                                              I’m sorry but there’s no way I’m believing that these candidates would be capable of learning Go but not OCaml (e.g.). It’s not about their capability, it’s about what Google wants to invest in them. Another reply even openly admits this! https://lobste.rs/s/yjvmlh/go_ing_insane_part_one_endless_error#c_s3peh9

                                                                                                                                            3. 3

                                                                                                                                              They want people to start banging out code almost instantly, not take some time to ramp up.

                                                                                                                                              Yes, and? The commodification of software developers is a well-known trend (and goal) of most companies. When your assets are basically servers, intangible assets like software and patents, and the people required to keep the stuff running, you naturally try to lower the costs of hiring and paying salary, just like you try to have faster servers and more efficient code.

                                                                                                                                              People are mad at Rob Pike, but he just made a language for Google. It’s not his fault the rest of the industry thought “OMG this is the bee’s knees, let’s GO!” and adopted it widely.

                                                                                                                                              1. 1

                                                                                                                                                Yes, I agree that the commodification of software developers is prevalent today. And we can all see the result, the profession is in dire straits–hard to hire because of bonkers interview practices, hard to keep people because management refuses to compensate them properly, and cranking out bugs like no tomorrow.

                                                                                                                                              2. 3

                                                                                                                                                on the contrary, google provides a ton of ramp up time for new hires because getting to grips with all the internal infrastructure takes a while (the language is the least part of it). indeed, when I joined a standard part of the orientation lecture was that whatever our experience level was, we should not expect to be productive any time soon.

                                                                                                                                                what go (which I do not use very much) might be optimising for is a certain straightforwardness and uniformity in the code base, so that engineers can move between projects without having to learn essentially a new DSL every time they do.

                                                                                                                                                1. 1

                                                                                                                                                  You may have a misconception that good programming languages force people to ‘essentially learn a new DSL’ in every project. In any case, as you yourself said, the language is the least part of the ramp-up of a new project, so even if that bit were true, it’s still optimizing for the wrong thing.

                                                                                                                                                  1. 1

                                                                                                                                                    no, you misunderstood what i was getting at. i was saying that go was optimisng for straightforwardness and uniformity so that there would be less chance of complex projects evolving their own way of doing things, not that better languages would force people to invent their own DSLs per project.

                                                                                                                                                    also the ramp-up time i was referring to was for new hires; a lot of google’s internal libraries and services are pretty consistently used across projects (and even languages via bindings and RPC) so changing teams requires a lot less ramp up than joining google in the first place.

                                                                                                                                                    1. 1

                                                                                                                                                      i was saying that go was optimisng for straightforwardness and uniformity so that there would be less chance of complex projects evolving their own way of doing things,

                                                                                                                                                      Again, the chances of that happening are not really as great as the Go people seem to be afraid it is, provided we are talking about a reasonable, good language. So let’s say we leave out Haskell or Clojure. The fear of language-enabled complexity seems pretty overblown to me. Especially considering the effort put into the response, creating an entirely new language and surrounding ecosystem.

                                                                                                                                        2. 9

                                                                                                                                          No, Rob observed, correctly, that in an organization of 10,000 programmers, the skill level trends towards the mean. And so if you’re designing a language for this environment, you have to keep that in mind.

                                                                                                                                          1. 4

                                                                                                                                            it’s not just that. It’s a language that has to reconcile the reality that skill level trends toward the mean, with the fact that the way that google interviews incurs a selection/survival bias towards very junior programmers who think they are the shit, and thus are very dangerous with the wrong type of power.

                                                                                                                                            1. 4

                                                                                                                                              As I get older and become, presumably, a better programmer, it really does occur to me just how bad I was for how long. I think because I learned how to program as a second grader, I didn’t get how much of a factor “it’s neat he can do it all” was in my self-assessment. I was pretty bad, but since I was being compared to the other kids who did zero programming, it didn’t matter that objectively I was quite awful, and I thought I was hot shit.

                                                                                                                                            2. 4

                                                                                                                                              Right! But the cargo-cult mentality of the industry meant that a language designed to facilitate the commodification of software development for a huge, singular organization escaped and was inflicted on the rest of us.

                                                                                                                                              1. 4

                                                                                                                                                But let’s be real for a moment:

                                                                                                                                                a language designed to facilitate the commodification of software development

                                                                                                                                                This is what matters.

                                                                                                                                                It doesn’t matter if you work for a company of 12 or 120,000: if you are paid to program – that is, you are not a founder – the people who sign your paychecks are absolutely doing everything within their power to make you and your coworkers just cogs in the machine.

                                                                                                                                                So I don’t think this is a case of “the little fish copying what big bad Google does” as much as it is an essential quality of being a software developer.

                                                                                                                                                1. 1

                                                                                                                                                  Thank you, yes. But also, the cargo cult mentality is real.

                                                                                                                                            3. 3

                                                                                                                                              Go is for compilers, because Google builds a billion lines a day.

                                                                                                                                        3. 2

                                                                                                                                          return errors.Wrapf(err, "fooing %s", bar) is a bit nicer.

                                                                                                                                          1. 13

                                                                                                                                            That uses the non-standard errors package and has been obsolete since 1.13: https://stackoverflow.com/questions/61933650/whats-the-difference-between-errors-wrapf-errors-errorf-and-fmt-errorf

                                                                                                                                            1. 1

                                                                                                                                              Thanks, that’s good to know.

                                                                                                                                            2. 8

                                                                                                                                              return fmt.Errorf("fooing %s %w", bar, err) is idiomatic.

                                                                                                                                              1. 9

                                                                                                                                                Very small tweak: normally you’d include a colon between the current message and the %w, to separate error messages in the chain, like so:

                                                                                                                                                return fmt.Errorf("fooing %s: %w", bar, err)
                                                                                                                                                
                                                                                                                                            3. 1

                                                                                                                                              It makes error messages useful but if it returns a modified err then I can’t catch it further up with if err == someErr, correct?

                                                                                                                                              1. 2

                                                                                                                                                You can use errors.Is to check wrapped errors - https://pkg.go.dev/errors#Is

                                                                                                                                                Is unwraps its first argument sequentially looking for an error that matches the second. It reports whether it finds a match. It should be used in preference to simple equality checks

                                                                                                                                                1. 2

                                                                                                                                                  Thanks! I actually didn’t know about that.

                                                                                                                                                2. 2

                                                                                                                                                  Yes, but you can use errors.Is and errors.As to solve that problem. These use errors.Unwrap under the hood. This error chaining mechanism was introduced in Go 1.13 after being incubated in the “errors” package for a long while before that. See https://go.dev/blog/go1.13-errors for details.

                                                                                                                                              1. 34

                                                                                                                                                when i got to google NYC i was sitting next to an older guy who seemed to spend all the workday talking to his broker. a few weeks in, i finally caught him doing some work, as he cursed at a lengthy awk invocation. seeking to impress him with how plugged-into the NYC scene i was despite just having come up from georgia, i opined “if you’re having trouble with awk, you know, kernighan is a few floors up, and aho is down the street. they’re the a and k in awk.” he stared balefully at me. “the w stands for ‘who-the-hell-knows’, heh.” thunderclouds descended. “i’m peter weinberger. i’m the w in awk.” we didn’t talk much after that, and one of the monthly reorgs moved the go team away soon thereafter.

                                                                                                                                                1. 7

                                                                                                                                                  Ha, that’s hilarious. After I wrote GoAWK, I was chatting offline with Alan Donovan (co-author of The Go Programming Language) and I found out Weinberger worked with him on the Go team, so I asked to meet up with them. We talked about what he worked on then (gopls, I think), what he’d do differently in AWK if writing it now, and we talked about Elm, which I’d just learnt. It was a fun lunch!

                                                                                                                                                  1. 4

                                                                                                                                                    “I’ve forgotten more about AWK than you’ll ever know!”

                                                                                                                                                  1. 4

                                                                                                                                                    Direct link to code (with commentary): https://play.golang.org/p/83fLiHDTSdY (and on the Go2go Playground: https://go2goplay.golang.org/p/83fLiHDTSdY)

                                                                                                                                                    I think I see what he’s doing, building up an interpreter DSL, but man, I have no idea what most of his commentary means. For example:

                                                                                                                                                    I exploit the duality between universal and existential quantification and encode sum types using the existential dual of the Boehm-Berarducci isomorphism.

                                                                                                                                                    1. 11

                                                                                                                                                      Well it’s pretty simple, really. Types are all about the composition of things, so the question is how do we compose what we have (language primitives) to get what we want. We can guess, try things out, but if that doesn’t work out we can use math to derive the result we seek, because these are all simple settled questions in math, and due to the Curry–Howard corespondence we should be able to implement them in our language.

                                                                                                                                                      Alonzo Church showed us how to encode anything into (untyped) lambda calculus, and indeed, we can encode variants using Church encoding like so:

                                                                                                                                                      a+b ≡ λa b f . f a b
                                                                                                                                                      fst ≡ λt . t (λa b . a)
                                                                                                                                                      snd ≡ λt . t (λa b . b)
                                                                                                                                                      

                                                                                                                                                      a+b takes three arguments, but we only supply two when we create it. Then a+b is a pending computation, and fst and snd extract the first or second component by supplying the continuation that tells the sum type what to do!

                                                                                                                                                      Unfortunately, this is not quite what we want, because untyped lambda calculus is untyped, and the whole point of the exercise is to be typed so this encoding won’t cut it. But it’s ok, this captures the procedural aspect of the idea, it tells us what we have to do with the data, but not with the types. Now we have to find a way to determine a typeful encoding.

                                                                                                                                                      Well, it’s pretty easy if we use math. A type like a can always be written like ∀z . (a→z) → z. But we are interested in a+b, so we just plug-in a+b, and we do some trivial algebra and we get:

                                                                                                                                                      ∀z . (a+b →z) →z = ∀z . ((a→z) * (b→z)) →z
                                                                                                                                                      

                                                                                                                                                      This looks suspiciously similar to Church encoding, we can see the two continuations. However, this is a type equation. It doesn’t tells us anything what to do with the values, it’s just a relation between types. But we know what to do with the values from the CPS interpretation of the Church encoding, and now we can implement this in any language that has parametric polymorphism. This is the road to the Boehm-Berarducci encoding, which expands on this idea (except that it goes a step further and encodes it as ∀z . (a→z) → (b→z) →z).

                                                                                                                                                      We can implement this in Haskell.

                                                                                                                                                      {-# LANGUAGE ExistentialQuantification #-}
                                                                                                                                                      {-# LANGUAGE RankNTypes #-}
                                                                                                                                                      
                                                                                                                                                      import Text.Printf
                                                                                                                                                      
                                                                                                                                                      data Variants a b z = Variants (a->z) (b->z)
                                                                                                                                                      type Sum a b = (forall z. Variants a b z -> z)
                                                                                                                                                      
                                                                                                                                                      plus :: Int -> Sum Int Int
                                                                                                                                                      plus x (Variants f _) = f x
                                                                                                                                                      minus :: Int -> Sum Int Int
                                                                                                                                                      minus x (Variants _ f) = f x
                                                                                                                                                      
                                                                                                                                                      neg :: Sum Int Int -> Sum Int Int
                                                                                                                                                      neg v = v (Variants (\x -> minus x) (\x -> plus x))
                                                                                                                                                      
                                                                                                                                                      foo = plus 42
                                                                                                                                                      bar = minus 16
                                                                                                                                                      baz = neg foo
                                                                                                                                                      
                                                                                                                                                      example :: Sum Int Int -> String
                                                                                                                                                      example v = v (Variants
                                                                                                                                                          (\x -> printf "got a plus %d" $ x)
                                                                                                                                                          (\x -> printf "got a minus %d" $ x))
                                                                                                                                                      
                                                                                                                                                      main = do
                                                                                                                                                          putStrLn (example foo)
                                                                                                                                                          putStrLn (example bar)
                                                                                                                                                          putStrLn (example baz)
                                                                                                                                                      

                                                                                                                                                      Prints:

                                                                                                                                                      : z800:aram; a
                                                                                                                                                      got a plus 42
                                                                                                                                                      got a minus 16
                                                                                                                                                      got a minus 42
                                                                                                                                                      

                                                                                                                                                      However, we can’t implement this in Go. Until very recently, Go didn’t have parametric polymorphism at all, and even so, notice that the forall in Haskell is inside the type. We need rank-2 polymorphism which Go doesn’t have. But the fact that we had to enable ExistentialQuantification in Haskell should give us a hint.

                                                                                                                                                      Go interfaces are existential types. In most (all?) languages based on System F/HM existential types are encoded as rank-n universals, but Go is unusual in that Go has always had existential types even before it had parametric polymorphism.

                                                                                                                                                      But universal quantification in logic is dual to existential quantification! We can always switch back and forth between the two (at least in classical logic, but because of intentional limitations of the Go type system, the excluded middle axiom should always hold, so we should be ok here).

                                                                                                                                                      So we can just translate any statement using universal quantifiers into its dual that uses only existential quantifiers, basically through double negation.

                                                                                                                                                      ∀x P(x) ≡ ¬∃x ¬P(x)
                                                                                                                                                      

                                                                                                                                                      So we just do that, and we get what’s implemented in my program!

                                                                                                                                                      Notice that I make the above transformation in logic, not in Go. If I’d made it in Go I would have to assume that the Curry–Howard isomorphism holds in the Go type system, which it may or it might not depending on the semantics of its generics and interfaces. I do not make this assumption. I do not assume we can use that axiom inside Go, but of course, we can use it in the meta-theory.

                                                                                                                                                      I explained all of this starting from Church encoding, going though Boehm-Berarducci, etc, but that’s not how I discovered this at all. I was trying to formalize Go generics, and equations such as the ones above popped out as intermediary results. I then immediately saw the potential of this, and trying to make sure that it works, and that I understood it, I started from the existential representation, and transformed it into the Boehm-Berarducci encoding (which I wasn’t even aware it existed when I started this exercise).

                                                                                                                                                      I only considered ADTs (not GADTs) in this post, but I hope this at least gives you some intuition.

                                                                                                                                                      Hope this helps.

                                                                                                                                                      1. 2

                                                                                                                                                        I appreciate you taking time with this answer, and I did read it all, but unfortunately I just don’t have the math and computer science background to comprehend this. Back to school for me!

                                                                                                                                                        1. 2

                                                                                                                                                          This explanation would be more effective for me and other laypeople if you include a brief tutorial on the math notation you use.

                                                                                                                                                          It’s easier to pick up the Haskell than it is to pick up the math notation.

                                                                                                                                                          1. 2

                                                                                                                                                            Alonzo Church showed us how to encode anything into (untyped) lambda calculus, and indeed, we can encode variants using Church encoding like so:

                                                                                                                                                            a+b ≡ λa b f . f a b
                                                                                                                                                            fst ≡ λt . t (λa b . a)
                                                                                                                                                            snd ≡ λt . t (λa b . b)
                                                                                                                                                            

                                                                                                                                                            a+b takes three arguments, but we only supply two when we create it. Then a+b is a pending computation, and fst and snd extract the first or second component by supplying the continuation that tells the sum type what to do!

                                                                                                                                                            Something is off here. That’s the Church encoding for products, not variants/sums.

                                                                                                                                                            1. 1

                                                                                                                                                              maybe because of this equivalence?

                                                                                                                                                              ∀z . (a+b →z) →z = ∀z . ((a→z) * (b→z)) →z

                                                                                                                                                              1. 2

                                                                                                                                                                It’s true that destructing a sum amounts to providing a product of handlers, one for each case. But that’s not how OP explained it. OP said the provided Church encoding was for variants/sums, which it isn’t, and then didn’t seem to use it at all in the Haskell code, since the Variants constructor is already a product of exponentials and doesn’t need to be Church-encoded. I think it’s just a mistake. It doesn’t detract from the Go demonstration, which is really cool. But Church encoding explanation is wrong, and I can see why people in these comments are confused.

                                                                                                                                                              2. 1

                                                                                                                                                                Thanks! That is what happens when you didn’t sleep for 24 hours. It should have been:

                                                                                                                                                                c₁ ≡ λt f₁ f₂ . f₁ t
                                                                                                                                                                c₂ ≡ λt f₁ f₂ . f₂ t
                                                                                                                                                                 m ≡ λt f₁ f₂ . t f₁ f₂
                                                                                                                                                                

                                                                                                                                                                e.g for usage: swap ≡ λt . m t c₂ c₁

                                                                                                                                                                (I can’t edit the mistake in the original post.)

                                                                                                                                                            2. 5

                                                                                                                                                              It’s like casting a spell. ‘I invoke the elemental magics of Yogg-Zonnroth, and cast the energies of Zum-ra into the ether!’

                                                                                                                                                              1. 1

                                                                                                                                                                Boehm-Berarducci encoding

                                                                                                                                                                Oleg has a good paper about this. Basically you can translate all A(lgebraic)DT operations (construction and destruction (aka pattern-matching)) into lambdas and applications of lambdas, meaning you can embed ADTs in languages that do not have them.

                                                                                                                                                                1. 1

                                                                                                                                                                  The code fails to run in both playgrounds … I’m guessing they are not using the right golang version ?

                                                                                                                                                                  1. 1

                                                                                                                                                                    Yes, it only works in the latest version of the compiler.

                                                                                                                                                                1. 1

                                                                                                                                                                  This is exciting! The biggest thing that has put me off Go so far has been the lack of generics. (I remember the days before generics made it into Java and I’m not too eager to re-live that experience)

                                                                                                                                                                  I understand the rationale for not including generics from the start, but as far as I can tell it leads to a lot of boilerplate and unsafe code. Especially in a language with good support for first-class functions, not having generics always felt like a missed opportunity.

                                                                                                                                                                  1. 6

                                                                                                                                                                    I’ve been writing Go professionally for a few years now, and lack of generics doesn’t result in “unsafe” code, at least not in the Go sense of memory-unsafe. It can result in runtime type assertion failures if interface{} types are used and converted at runtime. It can also result in boilerplate, with repeated (type safe) functions like ContainsInt and ContainsString instead of Contains[T] for slices. However, that happens less often than you’d think, as (unlike early Java), the built-in slice and map types are “generic” already (parameterized by type) … you just can’t add user-defined generic types or functions. This gets most people surprisingly far.

                                                                                                                                                                    That said, I’m cautiously optimistic about generics. They seem to be taking a typically Go-like caution with the features, not overhauling the language or stdlib with the introduction of generics. Still, I suspect there will be a raft of new “functional programming” libraries and other such things at first which won’t really fit in the Go spirit. Then hopefully it’ll settle down.

                                                                                                                                                                    1. 3

                                                                                                                                                                      I hope they don’t take too long with introducing the genetics to the standard lib. They risk a number of third party solutions filling the void. Best case, it will be like Apache commons. Worst case, like perl OOP. (I guess rust async frameworks are somewhere in between)

                                                                                                                                                                      1. 2

                                                                                                                                                                        it’s not entirely clear where you want generics to go in the standard library. the only real place i can think of is the sort library, and last thing i knew that was already being worked on. the go standard library isn’t filled with collections types or whatever like rust or c++. there really are only one or two files in the entire standard library where generics even could be added, much less would be useful. there’s a reason they hadn’t been added yet, and it’s because you don’t need them 99.9% of the time if you’re programming in go idiomatically, which the standard library generally does.

                                                                                                                                                                        my biggest fear with the introduction of generics is what is articulated in the parent of your comment, which is that everyone starts burying all of their code in a thousand layers of generic gibberish. the last thing anyone should want is for go to turn into c++. the fact of the matter is that at the moment if you find yourself “needing” generics in your go project, you’re probably doing something wrong, and you’ll either need to restructure your project in a better way, or, in the very few cases when you can’t do that (because you actually do need generics) you may have to hack around it. when the introduction of generics goes mainstream, you won’t ever have to hack around their absence when you really need them, but that doesn’t mean you shouldn’t refactor around their unnecessary inclusion. more often than not, reflexive use of generics is a matter of premature optimization, which your code (particularly in go) will be far better off without.

                                                                                                                                                                        i think a lot of people have this mistaken impression that all the philosophical “a bit of copying is better than a bit of dependency” etc etc stuff in the go community arose as some sort of coping mechanism for the lack of several important features. the opposite is true in reality: these “missing features” were deliberately left out for the purpose of incentivizing what the language’s developers believed to be a better style of programming. whether the costs incurred are worth the benefits is up for debate. at the very least it appears to be widely understood that some annoyances arise from that, particularly boilerplate, and that’s where generics and the various error handling proposals that aren’t set in stone yet have come from.

                                                                                                                                                                        here’s what that doesn’t mean:

                                                                                                                                                                        • best practices have meaningfully changed
                                                                                                                                                                        • every function needs type parameters
                                                                                                                                                                        • every file needs type parameters
                                                                                                                                                                        • every project needs type parameters

                                                                                                                                                                        if go as an experiment has taught us anything at all it’s that in the vast majority of cases you absolutely can go without all of these things we’ve convinced ourselves we need. if you find yourself thinking you need to use generics for a particular problem, the fact that you now can should not make you any less apt to pause and consider whether there really isn’t any other way to model the solution to your problem.

                                                                                                                                                                    2. 1

                                                                                                                                                                      I will say that Java’s solution to everything was inheritance + downcasting in the days before generics, and I think people mistakenly put all of the inheritance unpleasantness of inheritance into generics when thinking of the bad old days of Java. Go locks generics and that’s not great, but it still feels a whole lot better than Java 1.0. Notable Go also has type inference, first class functions, and structural subtyping which contribute a lot to the overall experience relative to Java 1.0.

                                                                                                                                                                      1. 1

                                                                                                                                                                        I’m mostly leery of generics, because I worry that it will lead to code which is hard to follow and reason about.

                                                                                                                                                                      1. 1

                                                                                                                                                                        How many org ship Golang from HEAD? Only google? I think Uber has their own cherry pick fork?

                                                                                                                                                                        1. 2

                                                                                                                                                                          I don’t even think Google does! Even the Go team at Google runs most of their production servers on the latest released version of Go. I ran into an issue just the other day due to the Go module proxy being on Go 1.16 (but they’re upgrading to 1.17 soon).

                                                                                                                                                                          Edit: that doesn’t mean no one does, though. The Tailscale startup seems to.

                                                                                                                                                                        1. 3

                                                                                                                                                                          It took me a little bit to find it (link was hidden behind the gray Show More “button” on YouTube), so here’s the text proposal and discussion.

                                                                                                                                                                          1. 1

                                                                                                                                                                            Reading through that proposal a bit, I don’t really follow what the advantage of this is over adding “replace example.com/foo => /path/to/foo” in the current go.mod? That works well for me anyway.

                                                                                                                                                                            1. 1

                                                                                                                                                                              I think the idea is that you have multiple related modules and don’t want to accidentally publish a version with the replace statements included.