1. 7

    I can’t put my finger on what I dislike about this story. What’s AWS supposed to do, mail him a cheque? Take him out for dinner?

    If (hypothetically) AWS deployed one of my open source projects right now I’m pretty sure I’d put it in the README and in my resume and feel like I got what I wanted out of it.

    1. 5

      Its not about what is legally required, but what is just nice and collaborative behavior. Nobody is forced by law to be nice to each other, but generally people are. If somebody is a dick there is no law preventing them from being one, but there are still a dick.

      1. 3

        Personally I think it’d be both sensible (nobody knows the code better) and polite (what better way to say thankyou) to offer the author a job working on the ‘as a service’ team.

        1. 3

          They’re supposed to credit him, at minimum. It’s in the article.

          There’s definitely a discussion to be had about what AWS could or should have done beyond that - for example, whether they have should paid him in some way is definitely a gray area since it’s open source and offered free of charge, and the author knew that. But they couldn’t even put the original author’s name anywhere in the product UI? Even if that wasn’t intentional, it’s a symptom of a larger company culture problem inside AWS that causes them to have a relationship with open source that is at times collaborative and what we would generally define as good open source citizenship (as Andi Gutmans points out) and is at other times frankly parasitic.

          Gutmans appears to be asserting that AWS contributes a lot to various upstreams and is a good open source citizen (probably true) and that therefore they couldn’t possibly be copying/stealing from different open source upstreams (probably false). This argument is a complete non sequitor. I mean, am I reading his argument incorrectly or unfairly? Someone correct me if so.

          1. 2

            They could easily pay author some kind of royalty per customer and both sides would be better off for it.

          2. 2

            Quoting the post:

            He said he hadn’t given the license for Headless Recorder a lot of thought because it’s just a browser extension full of client-side code

            I guess this is the issue. If it upsets you so much that a company uses your code with no upstream collaboration, you should actually give more thought to which license you choose for your project. There are licenses out there that prevent exactly this type of situation.

            1. 1

              The cloud protection license is a great development. But yes the article seems to be about people who’re grumpy they didn’t think of it earlier. Amazon isn’t really known to be an ethical company, you can’t expect them to treat you better than any of their employees out of politeness.

              1. 1

                About the only thing that most permissive licenses don’t allow you to do is claim you wrote someone else’s software. In a lot of locales, there’s a notion of ‘moral rights’ in copyright law (it’s per-state in the US and is a complete mess). In the copyright page for a book published in the UK, for example, you will find something like ‘the moral right of the author to be associated with this work has not been infringed’. This is something that’s been baked into copyright law for a long time: the understanding that even if someone else has bought or otherwise acquired the rights to profit from your work, you still retain the right to be credited.

                Amazon has fulfilled the letter of their obligation here by including something in the notices file but they’ve done so in the absolute minimal way possible. It wouldn’t have cost them any more to be a bit more public and say ‘this product is built on top of X, thanks to the author for their great work!’.

                Imagine if you’d written a book and the publisher put someone else’s name on the cover and yours in tiny text on the copyright page. It’s exactly equivalent from a legal and moral perspective: both are fine legally, but doing either without explicit approval from the author makes you a dick.

                1. 1

                  Imagine if you’d written a book and the publisher put someone else’s name on the cover and yours in tiny text on the copyright page.

                  I don’t buy it. The more appropriate analogy is if I put my book on a public book-sharing website, with a license that allowed people to do whatever they wanted with my book, and then complained when people did whatever they wanted with my book.

                2. 1

                  If you’re making money out of someone else’s work, even if it is in public domain, with even no license at all, buy them a gift. It didn’t even need to be Amazon (the company). Members of the very team that built the service could have done so.

                  Basic politeness and courtesy.

                1. 3

                  This oldie is a goodie, make sure to check out the sequel Kind of Like That. Does anyone have any tips or blog to do something similar with audio instead? Feels a bit harder, given that it’s not static in time unlike images.

                  1. 4

                    Do you mean audio fingerprinting? Take a look at https://acoustid.org/chromaprint

                    Here’s an article about using chromaprints that I recently borrowed code from: https://medium.com/@shivama205/audio-signals-comparison-23e431ed2207

                    Or: https://mtg.github.io/essentia-labs/news/2019/09/05/cover-song-similarity/

                    1. 3

                      My naive guess: maybe you can convert the sound to a spectrogram, and then think of it as an image?

                      Feels a bit harder, given that it’s not static in time unlike images.

                      What do you mean? Audio varies over time (unlike images), but an image varies over position (unlike audio). Maybe with audio, the query is a short clip, so you want to find both 1. which song contains this 2. at which offset? That seems analogous to comparing cropped images.

                      1. 1

                        Fair point on images varying on X and Y axis while audio varying on amplitude and time (also two axis). But I think I was referring to the fact that different, more fancy transformations are needed for audio? Like the signal goes up and down really fast instead of being the contour of the sound. I hardly have any knowledge on signals, so I wouldn’t know.

                        Comparing cropped images, much like a short clip of audio in a song, really increases the difficulty as well. Thanks for the comment though, I think this problem is good food for thought.

                        1. 2

                          Like the signal goes up and down really fast instead of being the contour of the sound.

                          That’s a good point! The individual peaks and troughs aren’t very meaningful to us. A spectrogram seems closer to what we perceive as the “contour of the sound”.

                          You can use the Fourier transform to break up a signal into its component frequencies. 3blue1brown has some nice explanations of that (https://www.youtube.com/watch?v=spUNpyF58BY). To get frequency-over-time you could chop the signal up into little slices and take the Fourier transform of each slice (https://en.wikipedia.org/wiki/Short-time_Fourier_transform).

                    1. 3

                      I think the worst variant of this I witnessed was a new/junior-ish teammate who tried to solve it silently themselves for 2 hours (probably from shame) until it came to light that an important database was missing.

                      Mistakes happen. If you drop a prod db, don’t be embarrassed. Blast that shit loud, like a foghorn, on every single Slack channel (or whatever communication channel) you can find. You never know who has a magic script or backup hidden up their sleeve.

                      (less related to the article than just general advice)

                      1. 1

                        I’m trying to figure out how RNNoise works: https://github.com/xiph/rnnoise

                        Overall score: 10/10

                        • README instructions don’t work? Check
                        • Scripts were written as if they couldn’t ever possibly leave the creators’ personal computer (e.g. “/usr/bin/python”?) Check
                        • Hundreds of unanswered GitHub issues where nobody can figure out the authors’ reproducibility claims? Check
                        • Released with lots of fanfare, press, hype, and publication to a big ML conference? Check
                        • Author is active in the GitHub issues until the conference publication, and then conveniently wipes their hands of the entire affair? Check

                        The author is pretty dang prolific, though: https://jmvalin.dreamwidth.org/. I wish I could keep up.

                        1. 48

                          Read the rest of this story with a free account.

                          Nah, rather not.

                          1. 29

                            It’s irritating that this guy has a self-hosted version of this post, but decided to post (or self-promote) the medium version instead.

                            1. 12

                              I wonder what’s the criteria to get banned for self-promotion … I also submit articles I wrote here sometimes – am I on the danger list for that?

                              1. 34

                                He had zero site use outside of promoting his own posts even after I DMd to ask him to stop. I generally give at least one warning PM unless it’s something really abusive like the LoadMill spam ring.

                                1. 6

                                  @pushcx I’m just wondering if the ban was a tad strict?

                                  I looked at their profile and while it’s all true (person only posts own articles, links only to medium though they have their own website versions) they do seem to have a high lobste.rs score (8.27) indicating some popularity here for their content (unless you tell me there has been some gaming of the score, in which case, what is wrong with people?)

                                  I don’t know how medium works, but I guess there may be some financial reason for the author to drive traffic there? It is a gray area, but the posts all seem to have technical merit and generate useful discussion?

                                  Anyhow, just wondering.

                                  1. 6

                                    I think it’s always the case that if you post more links than comments, that your score rises, since more people vote on posts than comments.

                                    1. 1

                                      Perhaps Karma garnered from posts should be balanced by the number of comments, or maybe comparing the ratio of people who click through (obviously this requires obnoxious tracking :( ) to those who upvote.

                                    2. 4

                                      I agree that his posts have generally been pretty good. The part you can’t see is that I sent him a note in June asking him to quit treating the site so exploitatively, and to take a break entirely from posting links to his site. (At this point a pretty formulaic note at this point.) He did not respond and continued.

                                      It’s a deliberate strategy of this author to post Medium links. Medium has a Partner Program incentivizing exactly this behavior: clicks = dollars. Another Medium-hosted spammer I’ve banned posted entreporn to YC News today to make money on the (sanitized) story of how he made money spamming links.

                                      1. 1

                                        @pushcx, thank you for taking the time to explain. I can see the long term danger of this behavior.

                                        1. 2

                                          Thanks for pointing out when I might be making a mistake. I’ve certainly made them before, and I’m glad to have the questioning and feedback.

                                    3. 3

                                      Ah ok, thanks!

                                  2. 6

                                    On the plus side, I can read the medium version without having to enable javascript…

                                    1. 4

                                      Ah, he’s been banned.

                                      1. 1

                                        It’s Time To Say Goodbye to MartinHeinz

                                      2. 1

                                        you mean this black page with “We’re sorry but frontend doesn’t work properly without JavaScript enabled. Please enable it to continue.” ? Sorry but I had to say this..

                                    1. 32

                                      I know these are corporations without feelings etc. etc. but I can’t help but feel bad for Docker.

                                      The “it’s just BSD jails/chroot” argument is like the “Dropbox is just ftp” arguments - Docker made containerization mainstream.

                                      And after Docker created a pretty massive movement to start using and deploying containers, everybody and their mother started writing their own container runtimes. Runc, OCI, CoreOS, cr-io, rkt, podman. Where were these tools before Docker?

                                      1. 45

                                        I see where you’re coming from, but from someone who used containers pre-Docker, and on container tech pre-Docker for a hobby project (2011-2015), I think they did a bad job.

                                        Docker is a pretty low quality piece of software. The superfluous daemon is a major reason (the article mentions this). The security story is another, though they were building on unstable foundations in the kernel.

                                        So yes they made something popular, and created a de-facto standard. But other people have had to swim in their wake and clean up the mess (OCI, etc.).

                                        I like cyphar’s blog on these topics: https://www.cyphar.com/blog/

                                        They also raised a lot of money and hired a lot of people, which I suppose is a good way to build certain things. I’m not sure it is a great way to build container tech, although I’ll take your point that there was little cooperation in the space beforehand.

                                        There is some “blame” to Google, since the kernel features were contributed by them, but user space tools were “missing”:

                                        https://en.wikipedia.org/wiki/Cgroups

                                        But really I think it is more of an issue with the kernel dev model, which is good at copying APIs that AT&T or someone else made, and bad at developing new ones:

                                        https://lwn.net/Articles/679786/

                                        As maintainer Tejun Heo himself has admitted [YouTube], “design followed implementation”, “different decisions were taken for different controllers”, and “sometimes too much flexibility causes a hindrance”. In an LWN article from 2012, it was said that “control groups are one of those features that kernel developers love to hate.”

                                        https://lwn.net/Articles/484251/

                                        1. 14

                                          Docker is a pretty low quality piece of software. The superfluous daemon is a major reason (the article mentions this). The security story is another, though they were building on unstable foundations in the kernel.

                                          Even taken on its own terms (daemon architecture, etc.) Docker is pretty low-quality. The daemon tends to wedge itself in an unresponsive state after running for a few weeks!

                                          “design followed implementation”

                                          I love this quote, and I think this is a factor in Docker as well. The image layer system pretty directly exposes union filesystems, but it’s not a particularly efficient solution to the problem of distributing incremental image updates. I think that this is part of why part of the Docker community pushes for very small base images, like Alpine — it’s a workaround for an inefficient distribution mechanism.

                                          1. 5

                                            Yes, yes and thrice yes. What I also really don’t understand is why there can’t be a big push for build tooling that takes big chunks of work out of the process of building statically linked binary only images so that you don’t need a whole OS inside the container. I mean, I guess I do, because there’s clearly no motivation for any of the big companies, who make tons of money out of hosting per-mb container registry storage, to make containers smaller, let alone to expend effort helping others to do so - and the whole “resources are cheap, developer time is expensive” line that Rails & DHH popularised back in the day makes everyone think “ahh it’s half a gig, who cares? Incremental, anyway, amirite?”. Well, I care. Every single byte of every image contributes to energy usage. It makes me cross.

                                            1. 12

                                              That’s basically bazel, open sourced by Google: https://bazel.build/

                                              Google doesn’t ship OS images inside its containers, in the style of Docker. (And remember as mentioned above, many of the Linux kernel container features were developed by Google).

                                              Instead they use statically linked binaries. However it doesn’t really solve the “gigabyte images” problem. A single static dependency graph tends to blow up as well, and you end up with gigabyte binaries.

                                              Bazel works really well for some cases, namely if most of your code is in C++. It compiles fast and the binaries are reasonably small compared to the optimum.

                                              In other cases it can be the worst of both worlds, because you have to throw out the entire build system (Makefile, autoconf) and rewrite in the Bazel build language. You have a maintenance problem, in other words.


                                              I used to have the same question as you… but then I tried to build containers from scratch, which was one of the primary motivations for Oil.

                                              I guess the short answer is “Do Linux From Scratch and see how much work it is”. The dependency graph is inherently not well curated.

                                              Another answer is “try to build a container that makes a plot in the R language, built from scratch”. This is a very difficult problem, more difficult than anyone thinks without having tried it. (e.g. it depends on Fortran compilers, graphics libraries, etc.)


                                              I think there could be a big push along the lines of “reproducible builds” for “dependency graph curation”, but it is a hard problem, not many people have the expertise, and requires a lot of coordination … It basically requires fiddling with a lot of gross build systems, i.e. adding more hacks upon the big pile that’s already there.


                                              Another thing I learned ~2012 when looking into this stuff: Version resolution is an NP-complete problem.

                                              https://research.swtch.com/version-sat

                                              Also Debian’s package manager is not divorced from its package data. There are hacks in the package manager for specific packages. That is true for most distros as far as I know.

                                              So again, it’s a very hard problem … not just one of motivation.

                                              1. 2

                                                That’s a really insightful and interesting reply. Thank you.

                                          2. 2

                                            So it could be safer to say that their marketed/attractive product, even if not technically the best, may have galvanized cooperation and better developments in the container space?

                                            1. 3

                                              I would say that’s accurate. If raising a bunch of money for a non-commercially-viable company is the only way to do that, then I guess I don’t have any answers … :) But I sure wish there was a better way.

                                              The old way was that AT&T was a monopoly and hired smart people to design software, which was flawed in its own way too (and Xerox PARC too). Google has a similar role now, in that they open source enormous amounts of code like Android, Chrome, and cgroups, which other people build on, including Docker … But yes it is ironic that Docker motivated Google to work on container tech, when Google had the original kernel use cases for their clusters.

                                            2. 2

                                              But really I think it is more of an issue with the kernel dev model, which is good at copying APIs that AT&T or someone else made, and bad at developing new ones:

                                              FreeBSD Jails and Solaris Zones are both better kernel technologies for deploying containers than the cgroup / namespace / seccomp-bpf mess that Linux uses, so it appears that the Linux kernel devs are not actually very good at copying APIs that other people made either. When Solaris copied Jails, they came up with something noticeably better (Jails have now more or less caught up). Linux, with both Jails and Zones to copy from, made something significantly worse in the name of generality and has no excuse.

                                            3. 28

                                              Where were these tools before Docker?

                                              Lacking a marketing department.

                                              1. 32

                                                Like it or not, marketing is a part of software development. The eschew-everything hacker ethos is marketing too, it just has a different target audience.

                                                1. 1

                                                  Well, I despise the former version of marketing that you mention. Software should stand on its own merits only, not because it’s “first” or whatever.

                                                  1. 1

                                                    The “social” in “social coding” seems to be eating the “coding” more than usual as time progresses.

                                                    I suppose there are some that believe that to be a feature.

                                                    1. 3

                                                      This is a subjective judgment, which I think reflects more on the speaker and their biases than on any objective condition. I don’t share it, for the record — I think programming has been a pathologically asocial discipline for most of it’s existence, and we’re just now beginning to “course correct” in a meaningful way.

                                                  2. 8

                                                    LXC did have marketing, though admittedly not the sort of blitz Docker went on.

                                                    I think that the Docker difference is willingness to sacrifice security for UX. With LXC you must type sudo all the time. The Docker way is to add yourself to the docker group. Sure, that makes your user root-equivalent, but now all your docker commands work so easily! This produces good word-of-mouth and lots of Medium tutorials, none of which mention the root-equivalence thing.

                                                    It is probably also important that the Docker CLI runs on a Mac. In that context we might do better to compare Docker to Vagrant than LXC. Even if boot2docker/Docker Desktop/Docker for Mac don’t work that well, they are initially appealing.

                                                    1. 9

                                                      I think by far the biggest problem Docker solved is easy redistribution of images: docker push, docker pull. This is the part that no container system had at the time (AFAIK) and explains much of its popularity.

                                                      Everything else: meh.

                                                      1. 2

                                                        This. I saw one of the initial Docker demos, at PyCon long ago, and this was what made a compelling difference from what LXC provided.

                                                    2. 2

                                                      Pithy and cute, but none of the tools OP mentioned existed before Docker.

                                                  1. 6

                                                    Getting a weird vibe from the comments here, like Linux isn’t kvlt enough for former OSX users and they need to drink straight shots of NixOS/BSD to cure themselves of the shame of having been mainstream for too long.

                                                    Just use a fully featured Linux distribution (e.g. Fedora).

                                                    1. 3

                                                      OSX is related to BSD and Nix has nix-darwin integration for OSX. As a result, transitioning from OSX to BSD is easier than OSX to Linux, and NixOS is more familiar than Fedora.

                                                      1. 1

                                                        Can’t run docker on *BSDs.

                                                        1. 3

                                                          That’s a feature :-)

                                                          1. 2

                                                            Looks more like a tombstone if I’m being honest - and that’s being kind, assuming they’re not already dead.

                                                    1. 12

                                                      After trying a bunch of stuff, I’ve just started forgetting things.

                                                      It sounds dumb, but when you focus on the things that truly matter to remember and understand that there’s some stuff you just won’t know next week, it allows you to focus on the parts of your life/job/technology that really matter and understand what you can and can’t just figure out on the fly. I’ve found that having taken notes isn’t really as helpful as the process of taking notes, which is only useful because it forces you to think about what you’re trying to remember and how to process the information. If you can train that mindfulness for the things that matter and understand how to find the information that matters less, you don’t really need to keep notes anymore.

                                                      1. 3

                                                        The real value to taking notes comes in the processing stage - when you go back and condense and synthesize your notes into your own words and opinions on the subject later on, after you’ve watched the lecture or read the book or whatever. This is always when I have my biggest “connection” moments from notes.

                                                        1. 1

                                                          I noticed that whether or not I take notes, within several weeks or months of having worked on a project, the minutiae are already forgotten (and re-reading notes written during the project are equally incomprehensible).

                                                          So I treat note-taking as a process, rather than a permanent thing. The notes exist alongside the project, and finish with the project. The completed project is the only testament to the notes having existed.

                                                          1. 1

                                                            So from my perspective that’s one aspect of a notes database’s usefulness - maintaining project context, but to your point that’s only useful up to a certain degree and then some amount of context is GOING to be lost, so the key is understanding that and trying to focus on what breadcrumbs you’ll need to rebuild the detailed mental context should you need to go back.

                                                            What I find my notes DB much more useful for personally is all the useful bits and bobs that I’m NEVER going to remember because their utility is applicable to a very narrow sphere of opportunity.

                                                            Some random entries from my personal notes file:

                                                            • How to burn a CD/DVD on OSX with only standard tools
                                                            • Cheat sheet for git bisect
                                                            • How to check out a pull request from Github to your local workspace

                                                            That sort of thing. These aren’t associated with a given project but when I need them again, I’ll be able to find them trivially because (for my setup) all Joplin notes can have tags associated with them and I use that feature studiously.

                                                          2. 1

                                                            After trying a bunch of stuff, I’ve just started forgetting things.

                                                            This, plus searching for things. I’ve never really taken notes in any context. I don’t find summarizing things all that useful… I feel like maybe I should since it’s so popular, but in reality info is pretty easy to find again 80% of the time…

                                                          1. 6

                                                            FWIW, Brendan Gregg (probably one of the top Linux performance engineers) replied in the HackerNews post of this article: https://news.ycombinator.com/item?id=24630395

                                                            Interesting take from somebody who’s definitely written good books on Linux performance engineering:

                                                            I’m pretty sure if I showed up to a performance engineering job interview and said “I wrote the book on performance,” as though that was what mattered most, I’d be shown the door. In my opinion it would be horrendously arrogant. In fact, I interviewed at Google while I was writing Systems Performance for Prentice Hall, and I never mentioned it in the interviews for that reason. In such interviews I’m focused on what value I can bring to the company – what problems I can help them solve in the future. I’m not there to coast on my prior reputation.

                                                            Also, regarding “overcome the hurdles”: those hurdles are now practically non-existent with various publishers. This results in books on the topic that should never have been published.

                                                            I could explain in more detail, but there’s a lot of negative connotations with technical book authorship, which I work to overcome in such interviews and in roles. Does the author care more about working on their reputation than working for the company? Did the author only write the book to make an exaggerated name for themselves for the benefit of job interviews? (As you suggested in your opening sentence :-) I personally only write books when I’m already the expert on the topic, and it’s a way to share my expertise.

                                                            Another interview I had years ago, after telling them about my first book, the interviewer said “so you know the theory, but can you put it in practice?” The assumption being “those that can do, those that can’t write books.” I was in a worse position for letting them know about it.

                                                            And of course, there can be positives for writing books. You are showing a willingness to share your knowledge with others. (You can do that in blog posts as well.)

                                                            1. 2

                                                              Wow, his experience is surprising. I never would have thought that writing a book would hurt you in an interview…

                                                            1. 23

                                                              To be honest I find the whole idea of someone adjudicating the worth of another human’s life work to be morally questionable and even perhaps likely to come from a place of extreme privilege.

                                                              Most of my family spent their entire lives slaving over plastic molding machines or doing even more menial labor like digging ditches.

                                                              They were good people who provided for their family and were generally speaking happy to do so. Were their lives not well spent?

                                                              This topic really resonates with me since my choice of employer has resulted in a TON of negative feedback both from randoms on the internet and people I generally admire and aspire to emulate.

                                                              1. 11

                                                                I don’t think anyone is adjudicating the worth of another person’s life here. Even the least charitable interpretation of the comment this article is responding to would be that Percival could have solved more challenging problems than he has. The worst way of reading it would be that he could have done more valuable work, not that his life has any less worth as a result of his career choice.

                                                                And I think the real point isn’t even that. It’s a question of whether the economic conditions we live under discourage optimal allocation of human capital. While Percival seems to respond with “no”, at least in his particular case, I think it’s still a 100% valid question to ask. While the point that the academia has its share of problems and limits on intellectual freedom is fair, it seems almost hard to argue that there aren’t a significant number of promising computer scientists, mathematicians, psychologists, etc. that spend their careers facilitating selling ads or products that the world would probably be better off without. We don’t need to say anything about the worth of these people to say that an economic system that makes the best interests of many highly trained scientists and mathematicians mildly harmful to the public at large has some issues.

                                                                1. 6

                                                                  And I think the real point isn’t even that. It’s a question of whether the economic conditions we live under discourage optimal allocation of human capital. While Percival seems to respond with “no”, at least in his particular case, I think it’s still a 100% valid question to ask.

                                                                  This is a really excellent point and a hard question I find myself chewing on often in other contexts.

                                                                  Capitalism definitely has some really ugly by-products, and I often wonder what society would look like if scarcity wasn’t a thing and the raison d’etre for money eroded entirely.

                                                                  I’d LIKE to think it’d be like Ian Banks Culture novels (My #1 voted future I’d love to live in BTW) but I don’t think anyone can know.

                                                                2. 5

                                                                  To be honest I find the whole idea of someone adjudicating the worth of another human’s life work to be morally questionable and even perhaps likely to come from a place of extreme privilege.

                                                                  You criticise judging somebody, while poisoning the well with a judgement at the same time.

                                                                  1. 3

                                                                    If this is the “judging judgemental people is also judgemental” paradox, the tie-breaker is easy. Whoever was first to be judgemental deserves to be judged for being judgemental. Fighting fire with fire is good.

                                                                    1. 3

                                                                      You criticise judging somebody, while poisoning the well with a judgement at the same time.

                                                                      A fair point. I suppose that if I zoom out past my own personal hangups around people casting aspersions at my own choices there’s nothing wrong with having the discussion.

                                                                  1. 4

                                                                    This is really cool! Thanks for writing & sharing :)

                                                                    I’m curious about a couple of things:

                                                                    • How do you recommend actually issuing alerts based on the query? I could imagine having a cron job execute the query and text/email/PagerDuty if there’s an anomaly returned for given period, but I’m wondering if there’s a better existing solution here.

                                                                    • You mentioned at the end there are some tools that provide similar functionality, I’m wondering if you could give a few examples? I know Datadog has great alarm tooling but I would imagine those are less general-purpose than a technique like this.

                                                                    1. 3

                                                                      Hey Jeff, glad you liked it.

                                                                      How do you recommend actually issuing alerts based on the query?

                                                                      Just like you said. A cron job executing the query at regular intervals and sending an email/text/whatever if it detects an anomaly. I know that there are some reporting tools (I use Redash for example) that have this ability as well.

                                                                      I’m wondering if you could give a few examples?

                                                                      I imagine any monitoring tool should have this functionality. Datadog and Scout are two that come to mind.

                                                                      The main point I wanted to convey in this article is that you set up a pretty descent monitoring system with plain SQL, zero dependencies and no $$$. From my experience, this simple method can go a long way.

                                                                      1. 2

                                                                        We use https://www.anodot.com/ (somewhere in the org). It’s downstream from our metrics gathering that I maintain (Graphite, Prometheus) but supports both AFAIK. I also don’t know if there’s a free/open source version or it’s a paid tool.

                                                                      1. 8

                                                                        I don’t have much experience with golang, but even then it was helpful to see some common patterns in rust (Result) explained concisely. Thanks for the post!

                                                                        1. 3

                                                                          No problem. Do you have any ideas for topics I can cover in this kind of style? Trying to build up a new backlog.

                                                                          1. 7

                                                                            I’d like to see an article describing the latest accepted/best third-party crates for Rust for some common tasks - e.g. error handling, JSON, HTTP requests. Some (eyre, reqwest, tokio for errors/HTTP requests/async) were already covered.

                                                                            E.g. back when I was writing Rust, there were different choices for “everybody should be using this!” error crates, and now eyre is apparently the good one.

                                                                            1. 5

                                                                              +1 to third-party crates overview!

                                                                              One of my frustrations of learning Rust over and over (coming from Go, but also I tried learning Rust before I started with Go) is that “just use the standard library” is practically not a thing like it is in Go where 80%+ of usecases are covered (http, json, templates, flags, etc – none are perfect, but all more than good enough to get started).

                                                                              Say I’m building a crawler, and I hit up https://crates.io/search?q=http, how do I choose? Having done a bunch of research since then, my instinct says hyper or reqwest (which don’t even show up on the first page!) but I remember running into some crates supporting async, some don’t.

                                                                              I’d love a collection of rust crates that are good for 80% of usecases, kinda equivalent to what’s in the Go standard library. When in doubt, use X.

                                                                              1. 6

                                                                                I’ll definitely work on a “Gopher in Exile” set of crates that about equal the majority usecase of the Go standard library. It’s something I’ve been struggling with too.

                                                                                1. 1

                                                                                  for your usecase: probably reqwest, that’s using hyper internally, supports async and blocking(opt-in via feature flag)

                                                                                2. 4

                                                                                  https://github.com/rust-unofficial/awesome-rust is not a terrible source. The ecosystem gets big enough for different purposes that it’s very difficult to cover exhaustively though, let alone keeping it up to date. Looking at its game engine resources for example, which is where my own work mostly lies, it hits the high points but misses basically all of the interesting secondary stuff.

                                                                                  Error crates are a great example. IMO it’s a problem space that was either solved long ago with failure, or is never going to be solved, depending on what you want. People keep inventing new things and for medium-sized projects at least I’ve yet to see any that are actually worth the mental overhead and compile times.

                                                                                  1. 1

                                                                                    huh, last time I checked everyone went from failure to thiserror/anyhow and I’m using snafu in crates to have optional backtraces on stable, anything new here ?

                                                                              1. 11

                                                                                What’s with this assumption of being a worldwide authority or known entity that’s so popular in blog posts or internet comments?

                                                                                My thoughts on…

                                                                                I’ve always maintained that…

                                                                                Like many people know…

                                                                                People who know me know that…

                                                                                I’m infamous for…

                                                                                Nobody knows who you are - just write your content without the preamble.

                                                                                1. 59

                                                                                  I don’t think a blogger owes anyone being overly modest on their own blog. They can write however they like. I’m going to read it as long as it’s useful and/or entertaining.

                                                                                  1. 29

                                                                                    I thought it was perfectly fine? They have discouraged and discussed this kind of online tooling, and eventually decided to write it up as a blog post. It’s perfectly reasonable to frame this article as the author’s reasoning behind doing what they’re doing.

                                                                                    In fact, I’d argue it’s more modest than something like “why online tooling is bad”. With the author’s framing as it is, they’re not pretending to have the objectively correct view; they’re upfront about how this is their opinion on the matter.

                                                                                    1. 2

                                                                                      That’s true as well. “JWT.io considered harmful” would be a worse post, and the personalisms clearly draw the line between “this is my opinion” and “I speak for everybody”.

                                                                                      My original post probably crossed wires between “authority” vs. “familiarity”

                                                                                    2. 16

                                                                                      You seem like you must be a worldwide authority on how to write preambles for blogs. ;)

                                                                                      1. 1

                                                                                        Why would he need to be?

                                                                                        1. 4

                                                                                          He wouldn’t, but as others have cited, being overly critical of someone’s writing because of their particular language choices and not commenting at all on the content being presented is A) Of questionable constructive value and B) Potentially not in the spirit of this very technology oriented forum.

                                                                                      2. 14

                                                                                        I’m sorry that you read it like so - would you mind explaining what wording made me sound like I think I’m a worldwide authority on the matter?

                                                                                        1. 9

                                                                                          You’re all good, you have your own voice. It’s some BS that this is the feedback you’re getting unsolicited instead of talking about the actual topic.

                                                                                          1. 4

                                                                                            Given the reaction, I’m probably in the wrong here. It’s a minor nit that I magnified into a big deal.

                                                                                            Buuut, to explain my rationale, it’s specifically your first paragraph. The content was fine (and I use jwt.io often).

                                                                                            Something my colleagues know well is how little I trust online tools like jwt.io, online JSON validators or online tooling in general, but it also bleeds into comments I make online and in my blog posts, too, and I thought I’d share a little bit of a reason as to why.

                                                                                            I’m not familiar with your coworkers, your online comments, or your blog posts. And I don’t mean that offensively, like “you’re a nobody” - it’s just that in this case I would rather tailor the article for what’s probably the larger group (people who don’t know you) vs. the smaller group (people who know you).

                                                                                            My reaction was overblown, and contained more of a general rant towards every instance of this I’ve seen, not your specific violation.

                                                                                            To be frank I’m surprised that others disagree (although it’s probably more about me sounding like a dick than my opinion). I genuinely find it strange to read public content that’s written as if I’m this person’s best friend and remember the thing they said in the kitchen last Tuesday.

                                                                                          2. 5

                                                                                            On the other hand, I dislike when people speak for me on my behalf with phrases like “the python community agrees that” or “all java developers think that”.

                                                                                            I also think it’s important to separate things that are factual from opinions. It’s not that my opinion is important but that it’s just an opinion, and everyone can have one. Disagreeing does not necessarily make you wrong when it comes to opinions, but more so when it comes to facts.

                                                                                            1. 1

                                                                                              “Or, as I’ve lately taken to calling it, …”

                                                                                              It’s weird though — these phrases didn’t start out as imitation of people with actual authority, as far as I can tell. A lot of fake authority starts as imitatiion (e.g. dressing like the king/CEO/next-rung manager), but I can’t recall widely respected people like DHH (recently) or JKH (decades ago) ever using phrases like “Like many people know”. These phrases were always fake.

                                                                                            1. 1

                                                                                              Do you know if the same applies to IPP?

                                                                                              Some quick tracing shows that MKL uses a single function mkl_serv_intel_cpu_true to detect whether it is dealing with a genuine Intel CPU. Fortunately, the function is rather trivial, so we can replace it by our own function:

                                                                                              What sort of tracing can I use to find out in IPP? Strace? Thanks.

                                                                                              1. 1

                                                                                                There are a lot of possibilities for tracing. One for them is ltrace. E.g.:

                                                                                                ltrace -l "*mkl*.so" -n 2 ./mt-sgemm
                                                                                                

                                                                                                Traces all calls in libraries that match the given library pattern (-l) and adds two spaces of indentation for nested calls (-n).

                                                                                              1. 11

                                                                                                by god, we’ve done it! we’ve fixed racism.

                                                                                                1. 8

                                                                                                  Not all changes are positive in the fight against racism. I think you’re being sarcastic, but I think superficial things like this sometimes make racism worse by celebrating non-victories and removing motivation for real change.

                                                                                                  Kind of like how joining a gym and paying money is worse than working out because it creates the mental perception of positive action. If everyone who joined a gym and didn’t go would just walk 2 hours a week they would be much better off. Of course, if they went to the gym they’d be better off as well.

                                                                                                  1. 19

                                                                                                    If you don’t even believe in incremental change you probably aren’t using git, so no problem for you either way, right?

                                                                                                    1. 5

                                                                                                      I think the disagreement is more whether or not this is even a “racism” question in the first place, rather than whether or not we should do anything about racism (something pretty more or less everyone agrees on, anyone not on the fringes of the right anyway).

                                                                                                      1. 1

                                                                                                        In my experience whether it was one in the first place is irrelevant, it’s whether it is one now. You can generalize this to bikeshedding moments well known to us developers. Something comes up which is easy to have an opinion about. Either everybody has their say, a decision is made and it gets forgotten. Or it gets tied into office politics and the discussion keeps coming back, but by the 3rd time you realize that it’s not really about whether the button should be in a menu or not, it’s about should we listen more to the UX designer who has a degree or the project managers who’re with customers every day? A decision has to be made which means there will be a “winner”.

                                                                                                        It’s not about whether this is racism in the first place, all racists have banded together behind resisting this change. Lobste.rs is pretty civilized but you should see the garbage being spouted on some sites against those who want to give you the tools to maybe make this change for your own project. The fringes of the right like to turn everything into culture wars and I’m glad they’re losing this one.

                                                                                                        1. 8

                                                                                                          whether it was one in the first place is irrelevant, it’s whether it is one now

                                                                                                          This is an interesting point! It also puts people whose opinion on the entire matter is a mere “meh” – probably the vast majority of people – in an extremely uncomfortable position, as:

                                                                                                          1. you either have to invest time and effort in to what you see as a pointless meaningless change that’s completely ineffective at fighting racism, or;
                                                                                                          2. you give in to the right-wing toxic trolls ranting about Cultural Marxist SJW Nazis, or whatever the lingo these days is.

                                                                                                          The only reason I’d be tempted to rename any branch is as a big “fuck you” to the people from item 2, but at the same time I think the entire thing is also completely meaningless beyond that.

                                                                                                          I don’t think it’s just the “fringes of the right” turning everything in to culture wars. I feel this is a good example of a “culture war” instigated by the left based on some extremely tenuous arm-chair psychology, and the ensuing conflict has left most people in a very awkward position :-/

                                                                                                          1. 3

                                                                                                            It’s not about whether this is racism in the first place, all racists have banded together behind resisting this change

                                                                                                            I buy the explanation that the term “master branch” is more appropriate in the sense of “master copy”, and ambiguity is arguably the defining characteristic of the English language (we’re still fighting about free vs. open vs. libre after 60 years).

                                                                                                            This makes me a racist. Go figure.

                                                                                                            1. 0

                                                                                                              As a programmer you should understand logic. All ducks are birds. Not all birds are ducks. All racists resist the change. Not all who resit the change are racists.

                                                                                                              Why pretend that you do not understand it in this case? What other reason could you have other than to add fuel to a racial fire?

                                                                                                              1. 3

                                                                                                                My purpose is to make sure overly dramatic posts like yours don’t go unchallenged.

                                                                                                                It’s not about whether this is racism in the first place, all racists have banded together behind resisting this change.

                                                                                                                Your initial statement is heavy with connotation that resisting the change and racism go hand in hand. You left no room for non-racist resisting of the change.

                                                                                                                You’ll notice arp242’s reply interprets your message the same way I did, and the same way you intended. Don’t try to weasel your way out of this one.

                                                                                                                1. 1

                                                                                                                  Why are you doing this? Speaking for others, twisting words and reading the worst into it, getting defensive about it… I expect more from Lobsters discussions.

                                                                                                                  Don’t let yourself be gaslighted. I really just meant what I said. Remember that nobody is forcing anything upon you. It is about being given the tools to do what you want with your own repo.

                                                                                                                  The linked article just gives you tools, it only vaguely references the controversy. “In three steps you have renamed a git branch without making a big deal out of it, all while avoiding the wrath of internet reactionaries.” Yet there was wrath. Because, as I was saying, changing the name of the master branch will never ever again be about just that.

                                                                                                    1. 2

                                                                                                      Host your own e-mail, don’t host your own e-mail. Front page ping-pong. There must be more interesting topics than this.

                                                                                                      1. -1

                                                                                                        Can anyone recommend some material describing concrete motivations for adding generics to Go? I’m aware of the abstract idea that you need generics in order to build data structures that work with different types, but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g? My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                        1. 19

                                                                                                          I hope it’s not that hard to imagine wanting different data structures than hash maps; maybe your problem fits better into a binary search tree for example.

                                                                                                          Well, I for one usually don’t feel like implementing my own red-black tree, so I would like to just grab a library. That library will be much nicer to use if I can just make an RBTree<string, MyFoo>. I certainly wouldn’t want to copy/paste an int->string red-black tree into some file(s) and judiciously search/replace until I have a string->MyFoo tree (and then do the same when I need an int->float tree).

                                                                                                          1. 0

                                                                                                            That makes sense, but I am still looking for some grounding in actual programming practice. Is there a use of a red-black tree that would not warrant customizing it for the storage type? Or one where it would make sense to add a library dependency rather than copying the RB tree code?

                                                                                                            1. 7

                                                                                                              How do you write a library that provides a Red-Black tree that can in principle work with many different client types without generics? This isn’t a rhetorical question, I don’t know Go and I genuinely don’t know how you would implement this kind of library in Go without generics.

                                                                                                              1. 6

                                                                                                                Go’s sync.Map (concurrent hashmap) is an actual real world example of this, and it uses interface{}, akin to Java’s Object.

                                                                                                                1. 24

                                                                                                                  Right, that’s a great example. Because it uses interface{}, it:

                                                                                                                  • Requires all keys and values to be heap allocated, leading to worse performance, worse memory usage, and worse memory fragmentation. Requiring two heap-allocated ints to store one value in an int->int concurrent hash map is unacceptable for many uses.
                                                                                                                  • Is less ergonomic, requiring a cast every time you want to use a value.
                                                                                                                  • Provides no type safety. (I imagine this one will be the least convincing to Go programmers, since Go generally expects the programmer to just not make mistakes)
                                                                                                                  1. 3

                                                                                                                    This brings me back to C++Builder 3 back in the 90s. To use a list, you had to create a class derived from a kind of TItem class to be able to store things. Why anyone would want to go back to that in productive code boggles my mind.

                                                                                                                    1. 1

                                                                                                                      I’m using a sync.Map (for its concurrency support - I have many goroutines writing map entries, and another goroutine periodically ranging over the entire map to dump it to a json file).

                                                                                                                      However I know the types I write to the map, I have no need for interface{}.

                                                                                                                      Am I better off with a real typed map + using sync.RWLock/mutex/etc. directly (in a custom struct)? Performance-wise.

                                                                                                                      1. 1

                                                                                                                        I don’t know, you would have to benchmark or measure CPU or memory usage. The sync.Map documentation suggests that using a regular map + mutexes could be better though: https://golang.org/pkg/sync/#Map

                                                                                                                        The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.

                                                                                                                        The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

                                                                                                                        If your usage falls outside of the two use cases which sync.Map is optimized for, it would absolutely be worth looking into replacing your sync.Map with a regular map and a mutex.

                                                                                                                        I suppose it becomes a question of which has the biggest performance penalty for you, heap allocation + indirection with sync.Map or lock contention with regular map + mutex?

                                                                                                                        (Also, in most cases, this probably doesn’t matter; make sure you’re not spending a long time improving performance in a part of your code which isn’t actually a performance issue :p)

                                                                                                                        1. 1

                                                                                                                          Right - the code “just works(TM)” and it takes around 0.5 seconds to render the JSON file every minute (which I track with metrics just to be safe) so it should be fine to keep as is. This is just a for-fun conversation.

                                                                                                                          or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys. In these two cases, use of a Map may significantly reduce lock contention compared to a Go map paired with a separate Mutex or RWMutex.

                                                                                                                          I definitely remember reading this sentence and it made me choose sync.Map because it sounds like my usecase. But like you say if I don’t measure it’ll be hard to tell.

                                                                                                                  2. -1

                                                                                                                    I don’t know and I didn’t think you could. I’m asking for an example use of an RB tree where using a library would make sense.

                                                                                                                    1. 6

                                                                                                                      Here is a popular Go RB tree implementation https://github.com/emirpasic/gods/ that uses Interface{} for the key and value types. Just search github for uses of it… With generics, users of this library would get greater typesafety.

                                                                                                                      https://github.com/search?q=%22github.com%2Femirpasic%2Fgods%2Ftrees%2Fredblacktree%22&type=Code

                                                                                                                      1. -2

                                                                                                                        okay. except i don’t know how to search github for uses of it and your search link brings me to a login page :(

                                                                                                                        1. 4

                                                                                                                          To short-circuit this:

                                                                                                                          At a previous job, I worked on a tool that started various services. The services had different dependencies, each of which needed to be started before the service. We wanted to be able to bring them up with as much parallelism as possible, or have a flag to launch them serially.

                                                                                                                          A simple approach to doing this correctly is modeling the dependencies as an acyclic graph (if it’s a cyclic graph, you’ve got a problem — you can never bring the services up, because they all depend on each other). To launch them in parallel, launch each one that has its dependencies met. To launch them serially, topologically sort the graph into an array/list/whatever and launch them one by one.

                                                                                                                          A generic graph implementation would be very useful, as would a topological sort that worked on generic graphs. With Go, you can’t have one that’s type-safe.

                                                                                                                          Another great use case for graphs: visualizing dependency graphs! You can have an open source graph visualization library, build a graph of whatever it is you’re trying to visualize, and pass it to the library and get a nice visualization of the data.

                                                                                                                          Graph data structures can be quite useful. Supporting generics makes them type-safe, so you catch errors at compile time instead of runtime. Some other examples of the usefulness of graphs:

                                                                                                                          • Graphs of friends at a social network (I currently work at one, and we use generic graph data structures all over the place — graphs of people to people, graphs connecting people and photos they’re tagged in, etc)
                                                                                                                          • Network topology graphs
                                                                                                                          • Graphs of links between documents

                                                                                                                          etc.

                                                                                                                          And it’s not just graphs. How do you write a type-safe function that takes in a list of possibly-null items, and returns a new list with the nulls stripped out? How about a function that takes a map and returns the list of its keys? In Golang, the answer is always copy-paste or give up type safety. In languages with generics, you can trivially write these functions yourself if they’re not in the standard library.

                                                                                                                          1. 1

                                                                                                                            thanks, this is a good motivating example.

                                                                                                                            1. 1

                                                                                                                              Huh. It had not occurred to me that github search would require a login.

                                                                                                                  3. 11

                                                                                                                    To turn the question around, why would you want to manually copy/paste code all over the place when the compiler can do it for you? And while I personally think “DRY” can be over done, not having the same (or very similar) code copy/pasted all over the place seems like a big practical win.

                                                                                                                    As far as customizing specific data structure and type combinations, most languages with generics have a way to do that, and I’d bet the Go designers thought of it.

                                                                                                                    1. 2

                                                                                                                      Copy / paste has got it’s own problems, but it lets you avoid a ton of complexity in the toolchain.

                                                                                                                      Toolchain development is all about tradeoffs. For instance, I use Typescript; the reference implementation is featureful, but slow to boot, so it keeps a background process alive to cache the heavy lifting, which accumulates state and introduces subtle confusions (eg type errors that don’t exist) until it’s restarted.

                                                                                                                      For some problem spaces, the problems introduced by copy/paste pale in comparison to the problems introduced by slow, stateful compilers.

                                                                                                                      1. 7

                                                                                                                        Copy/paste vs generics is unrelated to compiler bugginess.

                                                                                                                        If you carefully pick TypeScript as the comparison point, you can make the case that a buggy toolchain is bad (not that most users care, they just restart the compile process when it starts to go bad).

                                                                                                                        But if you were to pick say ReasonML for comparison, you could say that it’s possible to have a solid generics implementation (much less copy-pasting) and a super-fast, accurate compiler.

                                                                                                                        I.e. you can have both buggy and non-buggy compilers supporting generics. Hence, unrelated.

                                                                                                                        1. 2

                                                                                                                          ReasonML is great!

                                                                                                                          That said, while the relationship is indirect, it’s there. Adding complexity is never free. It didn’t cost ReasonML speed or reliability, but it costs maintainers time and makes every other feature more difficult to add in an orthogonal way.

                                                                                                                          1. 2

                                                                                                                            In the scheme of things, is it more important to have a super-simple compiler codebase, or is it more important to put more power and expressiveness in the hands of users? Note that every mainstream language that started without generics, has now added it.

                                                                                                                            1. 1

                                                                                                                              IMO, there’s such a thing as a right time to do it.

                                                                                                                              In the early years it’s more important to keep the simplicity - there aren’t that many users and you’re still figuring out what you want the language to be (not every feature is compatible with every approach to generics).

                                                                                                                              Once you’re ready to start generics you need to answer questions like - do you want monomorphisation or lookup tables? Is boxing an acceptable overhead for the improved debugging ergonomics?

                                                                                                                              1. 1

                                                                                                                                It seems like Go has been going through exactly the process you’re describing.

                                                                                                                            2. 2

                                                                                                                              I think these comparisons are a bit unfair: isn’t Typescript self hosted, whereas ReasonML is written in OCaml? It seems like Typescript would have a very hard time competing.

                                                                                                                              1. 3

                                                                                                                                Being able to use lots of existing OCaml bits is a huge advantage.

                                                                                                                                Typescript has been able to compete due to the sheer number of contributors - MS pays quite a large team to work on it (and related stuff like the excellent Language Server Protocol, VScode integration).

                                                                                                                                However, large teams tend to produce more complex software (IMO due to the added communications overhead - it becomes easier to add a new thing than find out what existing thing solves the same problem).

                                                                                                                                1. 1

                                                                                                                                  I should clarify my comment was more about comparing performance of the two languages.

                                                                                                                                  OCaml is a well optimized language that targets native machine code so tooling built in OCaml should be more performant than tooling built in Typescript. As a result, it’s hard to compare the complexity of either tool by the performance of the tool. It’s apples and oranges.

                                                                                                                                2. 2

                                                                                                                                  isn’t Typescript self hosted, whereas ReasonML is written in OCaml? It seems like Typescript would have a very hard time competing.

                                                                                                                                  That’s a strange argument. If it were very hard for them to compete why would they not use OCaml as well, especially since its contemporary alternative, Flow, was written in OCaml too? Or why would they not make TypeScript as good as a language for writing TypeScript in as OCaml is?

                                                                                                                                  1. 1

                                                                                                                                    My comment was more about performance, but it wasn’t very clear. It’s hard for Typescript, which is compiled to Javascript and then interpreted/JITed, to create tooling that’s as fast as a language that builds optimized native code.

                                                                                                                                    Given that Typescript is self hosted it has the advantage that community involvement is more seamless and I don’t want to downplay the power that brings.

                                                                                                                              2. 0

                                                                                                                                But compilers that support generics are more likely to be buggy. That’s a relation.

                                                                                                                                1. 2

                                                                                                                                  Any source for this rather surprising assertion?

                                                                                                                                  1. 0

                                                                                                                                    generics are feature that requires code to implement; code can contain bugs.

                                                                                                                                    1. 1

                                                                                                                                      But a self-hosting compiler with generics is likely to be less verbose (because generics) than one without, so it should be less buggy.

                                                                                                                                      1. 1

                                                                                                                                        i guess you can’t prove it either way but IME the complexity of algorithms is more likely to cause bugs than verbosity.

                                                                                                                              3. 5

                                                                                                                                I think Typescript is a straw man. Does this Go implementation of generics slow down the compiler a noticeable amount? There’s nothing inherent to generics that would make compiling them slow.

                                                                                                                                On the other hand, copy/pasted code is an ever increasing burden on developer and compile time.

                                                                                                                              4. -2

                                                                                                                                You are imagining a code base where the same complex data structure is instantiated with two different types. Is that realistic?

                                                                                                                                1. 5

                                                                                                                                  You are imagining a code base where the same complex data structure is instantiated with two different types. Is that realistic?

                                                                                                                                  Realistic enough that the Linux kernel developers went through the hassle of developing generic associative arrays, circular buffers, and other generic data structures using void*.

                                                                                                                                  And so did Gnome with GLib, which provides generic lists, hash tables, and trees, along with several others structures, also using void*.

                                                                                                                                  And the standard libraries of most modern languages include reusable and generic sequence and associative data types, and some times significantly more than that.

                                                                                                                                  For most data structures, though, focusing on a single code base gives too narrow of a view. Generics allow libraries of data structures to be created, so even though a single code base only use one R* tree (or whatever), that R* tree library can be used as-is by any number of projects.

                                                                                                                              5. 8

                                                                                                                                The Abstract and Background sections of the draft design doc touch on the motivations. Additionally, each section describing a dimension of the design usually mentions, at least briefly, the motivation for that feature.

                                                                                                                                1. 8

                                                                                                                                  Here is an example that I’ve wanted for ever, and can finally do. Higher order combinators that you can leverage first class functions with!

                                                                                                                                  Generic map, in go

                                                                                                                                  1. 1

                                                                                                                                    That’s the type of thing I have seen as a justification, but I don’t get why that’s so important. Can’t you just use a for loop?

                                                                                                                                    1. 22

                                                                                                                                      “Can’t you just …” goes forever. “Can’t you just write your for loop with labels and jumps in assembly?”^^

                                                                                                                                      For me, it’s all about abstraction. Having low level combinators, like this, that I can compose to build higher level abstractions in a generic way is wonderful.

                                                                                                                                      ^^: See also whataboutism.

                                                                                                                                      1. 3

                                                                                                                                        I’m not sure that composing from higher level abstractions is always such a good idea. I like both Go (hobby projects) and Rust (work!) but I still fell that most of the time I prefer this level of abstraction:

                                                                                                                                        type Server struct {
                                                                                                                                        ...
                                                                                                                                            Handler Handler // handler to invoke, http.DefaultServeMux if nil
                                                                                                                                        ...
                                                                                                                                        }
                                                                                                                                        type Handler interface {
                                                                                                                                            ServeHTTP(ResponseWriter, *Request)
                                                                                                                                        }
                                                                                                                                        

                                                                                                                                        from this:

                                                                                                                                         pub fn serve<S, B>(self, new_service: S) -> Server<I, S, E>
                                                                                                                                            where
                                                                                                                                                I: Accept,
                                                                                                                                                I::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                                I::Conn: AsyncRead + AsyncWrite + Unpin + Send + 'static,
                                                                                                                                                S: MakeServiceRef<I::Conn, Body, ResBody = B>,
                                                                                                                                                S::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                                B: HttpBody + 'static,
                                                                                                                                                B::Error: Into<Box<dyn StdError + Send + Sync>>,
                                                                                                                                                E: NewSvcExec<I::Conn, S::Future, S::Service, E, NoopWatcher>,
                                                                                                                                                E: H2Exec<<S::Service as HttpService<Body>>::Future, B>,
                                                                                                                                            {
                                                                                                                                            ...
                                                                                                                                        

                                                                                                                                        Don’t get me wrong, i like type level guarantees and I can see flexibility here, but in my experience with c++, rust and haskell is that generic programming often ends up complicating things to a degree that I personally don’t like.

                                                                                                                                        1. 1

                                                                                                                                          I think this is going to be a balance that the community has to find. I don’t regularly program in rust, but I’d be quite surprised if it wasn’t possible to get something close to the Go http API in it. The example you pasted seems complicated for the sake of being complicated. In theory, the Go community has been drilled into thinking in terms of the littlest abstraction that’ll work, which maybe makes it possible to generally avoid generic APIs that don’t actually need to be?

                                                                                                                                        2. 3

                                                                                                                                          “Can’t you just” does not go forever. It is a simpler way to say that the alternative is not significantly harder than what’s proposed. Is there some type of task that would be doable using a generic map but unreasonably hard using for loops?

                                                                                                                                          I feel like Go was designed from the ground up to be written in an imperative style, and composing first order functions is more of a functional style of coding. If I understand, without generics you would be nesting for loops rather than composing map functions, which is no more difficult to understand or write.

                                                                                                                                          I don’t follow the connection to whataboutism.

                                                                                                                                          1. 2

                                                                                                                                            I think it’s fine for your style of writing code to be to use loops and conditionals instead of map and filter. I think it’s a fine way to code that makes more sense in an imperative language. Straight for loops and while loops with if statements inside them is just better, more easily understandable code in an imperative language, in my opinion, than .map(...).filter(...).map(...) etc.

                                                                                                                                            1. -1

                                                                                                                                              Incidentally there is a repo wherein Rob Pike expresses his attitude towards this style of coding:

                                                                                                                                              https://github.com/robpike/filter/

                                                                                                                                              I wanted to see how hard it was to implement this sort of thing in Go, with as nice an API as I could manage. It wasn’t hard.

                                                                                                                                              Having written it a couple of years ago, I haven’t had occasion to use it once. Instead, I just use “for” loops.

                                                                                                                                              You shouldn’t use it either.

                                                                                                                                              1. 2

                                                                                                                                                I mean… that’s like … one man’s opinion… man. See also.

                                                                                                                                                Generics are going to create a divide in the Go community, and it’s going to be popcorn worthy. There’s no point of adding Generics to the language if this filter thing “shouldn’t be used,” and the community rejects the abstractions that Generics provide.

                                                                                                                                                This divide is easily already seen in the community as it relates to test helpers. On the one hand, there’s a set of developers that say “stdlib testing is more than enough.” On the other hand, there are people who want the full testing facilities of junit, with matchers, lots of assert style helpers, etc. Who is right? They all are, because those things work for their respective teams and projects.

                                                                                                                                                This general dogmatic approach to language idioms is why I call it “idiotmatic” Go.

                                                                                                                                                1. -1

                                                                                                                                                  I suppose if Ken and Rob wanted generics they would’ve put them in the original language, and there wouldn’t be this controversy. Time to go back to learning Erlang which seems old and dusty enough to not have big language changes and drama.

                                                                                                                                            2. 16

                                                                                                                                              You can’t pass a for loop to anything, you can only write it where you need it. Sure, toy examples look like toy examples, but the fact remains that Go has first-class functions, which should be a nice thing, but it doesn’t actually have a type system rich enough to express 90% of the things that make first-class functions worth having.

                                                                                                                                              1. -1

                                                                                                                                                You can’t pass a for loop to anything, you can only write it where you need it.

                                                                                                                                                right, so the example code could be done with a for loop no problem. is there a more motivating example?

                                                                                                                                                it doesn’t actually have a type system rich enough to express 90% of the things that make first-class functions worth having.

                                                                                                                                                how do you mean?

                                                                                                                                              2. 3

                                                                                                                                                Consider composing multiple transformations and filters together. With multiple for loops you have to iterate over the array each time, while by composing maps you only need to iterate once.

                                                                                                                                                1. 3

                                                                                                                                                  Just compose the operations inside the loop.

                                                                                                                                                  for x in y:
                                                                                                                                                      ...f(g(x))...
                                                                                                                                                  
                                                                                                                                                  1. 2

                                                                                                                                                    That works in some cases, but it’s pretty easy to find a counter example, too.

                                                                                                                                            3. 7

                                                                                                                                              In terms of collections, the truth is most of the time a map/slice is a good option. Here’s my top two favorite use cases for generics in go:

                                                                                                                                              1. Result<T> and functions that compose over them.
                                                                                                                                              2. Typesafe versions of sync.Map, sync.Pool, atomic.Value, even a rust like Mutex
                                                                                                                                              1. 5

                                                                                                                                                Oh man. I hadn’t even considered a better way to do error handling, eg. a Result type. People are gonna get so mad.

                                                                                                                                                1. 5

                                                                                                                                                  Generics isn’t enough to do what people want to do with error handling 99.99% of the time, which is to return early. For that, you either need a macro, such as the aborted try proposal, or syntactic sugar for chaining such “functions that compose over them” (like Haskell’s do notation).

                                                                                                                                                  Otherwise you end up with callback hell à la JavaScript, and I think nobody wants that in Go.

                                                                                                                                                  1. 4

                                                                                                                                                    I was more thinking of something where the if err pattern is enforced via the type system. You’re still not getting there 100%, you could get reasonably close, with a generic Result type that panics when the wrong thing is accessed, forcing you to check always or risk a panic.

                                                                                                                                                    r := thing()
                                                                                                                                                    if r.HasError() { handleError(r.Err()) }
                                                                                                                                                    else v := r.Val() { handleSuccess(v) }
                                                                                                                                                    

                                                                                                                                                    And of course it’s easy to question why this is interesting until you do chaining of things, and get a full on, type safe Result monad.

                                                                                                                                                    r := thing().andThen(func(i int) { ... }).andThen(func(i int) { ... })
                                                                                                                                                    if r.IsErr() {
                                                                                                                                                       handleErrForWholeComputation(r.Err())
                                                                                                                                                    } else {
                                                                                                                                                       handleSuccessForWholeComputation(r.Val())
                                                                                                                                                    }
                                                                                                                                                    

                                                                                                                                                    The alternative can be seen in things like this where you skirt around the fact that you can’t generically accept a value in one of those called functions. This is also why I said people are going to get so mad. These things are confusing to people who haven’t dealt with them before, and will make Go much more expressive, but less easy to grok without effort.

                                                                                                                                              2. 5

                                                                                                                                                but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g

                                                                                                                                                All of them. Copying code manually is one of the worst things you can do in software development. If it weren’t, why even bother writing functions, ever?

                                                                                                                                                My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                                                                The fact that libraries exist that don’t customise in such a way in languages with generics would disprove that notion.

                                                                                                                                                1. 6

                                                                                                                                                  one of the worst things you can do in software development

                                                                                                                                                  For me that’s “making things unreadable for whoever comes after you”. And sometimes copying a bit of code is the optimal solution for avoid that.

                                                                                                                                                  1. 0

                                                                                                                                                    but is there a real-world setting where this is actually better than just copying code and doing s/type1/type2/g

                                                                                                                                                    All of them. Copying code manually is one of the worst things you can do in software development. If it weren’t, why even bother writing functions, ever?

                                                                                                                                                    I disagree with your implication that the use of functions means code should never be copied. For example if you want to use strlcpy() in a portable C program, it makes more sense to put a copy in your source tree rather than relying on an external library. An extra dependency would add more headache than just copying the code.

                                                                                                                                                    My sense is that projects that use complex data structures almost always customize the data structures in a way that depends on the data type being stored.

                                                                                                                                                    The fact that libraries exist that don’t customise in such a way in languages with generics would disprove that notion.

                                                                                                                                                    That’s why I said “almost always.” And remember that the existence of a library doesn’t mean it is used with any frequency.

                                                                                                                                                  2. 3

                                                                                                                                                    Suppose you have a structure parametrizable by types T1, T2. You’re writing it in Go, so you assume that it’s ok if T1=string, T2=int. Also, in some of the places, you were using int for purpose unrelated to T2 (ie. if T2=foo, then there are still ints left in the source). Another programmer wants to copy-paste the code and change some types. How does he do it?

                                                                                                                                                    1. 2

                                                                                                                                                      I think “this would make copy-pasting code harder” is not so compelling an argument. One of the major points of introducing generics is that it would eliminate much of the present need for copy/paste in Go.

                                                                                                                                                      1. 2

                                                                                                                                                        Yes it would be harder than a search-and-replace, but that is still abstract and unrelated to any real-world use case.

                                                                                                                                                        Yes, I’m just counterpointing the parent commenter’s argument. I know the value of generic structures.

                                                                                                                                                      2. -1

                                                                                                                                                        Yes it would be harder than a search-and-replace, but that is still abstract and unrelated to any real-world use case.

                                                                                                                                                    1. 1

                                                                                                                                                      You haven’t lived if you haven’t sat next to a person who likes crunching ice cubes throughout the day.

                                                                                                                                                      1. 1

                                                                                                                                                        I don’t think that an attempt to make a community more welcoming to marginalized groups is dramatic or aggressive. I don’t think forking a project after a maintainer has decided not to address an issue is dramatic or aggressive.

                                                                                                                                                        In the original GitHub issue, you posted this:

                                                                                                                                                        I wonder how all of you would feel if suddenly a ton of people who haven’t done anything for the project appeared here, told you’re a horrible person and started bullying you into doing what they want…

                                                                                                                                                        This is in reference to 4 or 5 comments that expressed disagreement, most of which were couched in “I feel” or “I think.” Up to this point - and please correct me if I’m wrong - the discussion actually seems quite polite. People were passionate because police violence is an important topic, but I didn’t see anyone calling you names or dismissing your efforts even when they disagreed with you. But after you made your post, people came to your defense by characterizing the original issue as “nonsense,” “execrable,” “SJW,” “absurd,” “whining,” “ridiculous,” and so on.

                                                                                                                                                        1. 34

                                                                                                                                                          I acknowledge any mistakes on my part and I regret and denounce the negativity invoked in my defence. My initial reaction was directed at some Twitter comments, but I failed to make this clear, which is on me. As noted I wrote this at 7 am having woken up a few minutes earlier. Clearly not my brightest moment. :-)

                                                                                                                                                          1. 14

                                                                                                                                                            After reading the entire thread, you did nothing wrong. I found your replies to be reasonable.

                                                                                                                                                            I would have put a “No” and closed/disabled further discussion in the very beginning. Engaging with this type of person is a waste of time. Pretending as if forking and running a sed command is “doing most of the hard work”.

                                                                                                                                                            Ultimately this seems to be the consequence of writing a successful tool with a big userbase.

                                                                                                                                                            1. 5

                                                                                                                                                              If someone forks it, just sit back and enjoy them fixing all your bugs for you, or giving up at the effort.

                                                                                                                                                              1. 2

                                                                                                                                                                You have done absolutely nothing wrong and have nothing to apologise for.

                                                                                                                                                              2. 17

                                                                                                                                                                What would your response be if that happened to you? If someone opened up an issue with grunt-simple-mocha requesting you change the name because “grunt” is a US slang term for “soldier,” which implies US hegemony and violence against other countries? And a lot of other people supported that?

                                                                                                                                                                1. 6

                                                                                                                                                                  I’d take the request seriously. I’d wonder why they were taking it up with me rather than the authors of Grunt. I’d wonder if they opened the PR as a form of harassment/retaliation. But assuming their concerns were legitimate, I’d consider changing the name.

                                                                                                                                                                  I’d definitely delete any needlessly dramatic comments that called other contributors SJWs, bullies, aggressive, or censors for expressing concerns about bigotry in an industry which has repeatedly proven itself to be structurally unequal and unwelcoming to large groups of people. But I already do that thrice daily after washing my hands for 20 seconds with soap.

                                                                                                                                                                  1. 7

                                                                                                                                                                    It was an example of a project whose name might be taken in a bad context. I could have used, for example, your funny-fingers repo as offending someone because that is mocking people with deformed hands. You might think that’s a silly example, but until two weeks ago, there wasn’t an issue with the word “cop”.

                                                                                                                                                                  2. 4

                                                                                                                                                                    I wonder this, as well. Where do we draw the line?

                                                                                                                                                                    Should our industry lean towards plain-English definitions for names when it comes to software?

                                                                                                                                                                    Is there demonstrable evidence that “RuboCop” has caused anyone, anywhere, undue distress? Moreso than would be required to rename a highly-used library?

                                                                                                                                                                    1. 9

                                                                                                                                                                      I think too many people are taking George Orwell’s 1984 more as a manual than a warning.

                                                                                                                                                                      1. 1

                                                                                                                                                                        If I’m not mistaken, it was written more as a polemic; an argument against the dangers of Stalinist government over-reach in 1940s Russia.

                                                                                                                                                                  3. 1

                                                                                                                                                                    But after you made your post, people came to your defense by characterizing the original issue as “nonsense,” “execrable,” “SJW,” “absurd,” “whining,” “ridiculous,” and so on.

                                                                                                                                                                    Which is fine, because the original issue is absurd, ridiculous whining from an SJW concerned, ironically, more with policing the language of others than with doing anything useful.

                                                                                                                                                                  1. 2

                                                                                                                                                                    https://teachyourselfcs.com/

                                                                                                                                                                    Found this site some time ago. It can be helpful for you.

                                                                                                                                                                    1. 6

                                                                                                                                                                      Not sure why this would be their first port of call when it essentially covers what they’ve just studied in their degree.

                                                                                                                                                                      1. 1

                                                                                                                                                                        Studying it once isn’t enough to act natural when pretending you didn’t memorize your whiteboard questions. Gotta hit that leetcode.com too.

                                                                                                                                                                        1. 2

                                                                                                                                                                          Cheers you’ve just reminded me why I have no inclination to throw myself into the big tech meat grinder when I graduate.

                                                                                                                                                                          1. 2

                                                                                                                                                                            OK, upon hearing you haven’t graduated yet, I’ll amend my first statement.

                                                                                                                                                                            I may be exaggerating. I’m a person who has A) done (and failed) at least 50 leetcode-style whiteboard interviews, B) had (so far in ~7 years) rewarding jobs without leetcode interviews, where I had good coworkers and learned lots.

                                                                                                                                                                            Maybe by the 20th time I had to draw a graph or binary tree in a Google doc (like actually typing out symbols like A—–B—–C to draw edges and vertices) to answer a network flow question in 30 min over a scratchy phone line while “vocalizing my thought process”, it got pretty old.

                                                                                                                                                                            I know a common feel-good statement around leetcode is that you’ll only get them when you’re trying to get into the best of the best. That’s a joke too, leetcode will surprise you anywhere and everywhere. I’ve had the most absurd questions for run of the mill “DevOps Engineer” jobs at companies that were most definitely not big tech (or even close).

                                                                                                                                                                            tl;dr: fuck leetcode, hate leetcode, but study leetcode. Just in case

                                                                                                                                                                            1. 1

                                                                                                                                                                              I’ve been working as a developer for six years, I’ve probably interviewed with 20 companies across a few job hunts. I only had to whiteboard a few times and it was always about architecture rather than algorithms.

                                                                                                                                                                              Granted I haven’t interviewed with any of the biggest name brand companies, but it’s definitely not a common phenomena for your average interview in my experience.

                                                                                                                                                                              Most interviewers are aware of the “invert a b-tree on a whiteboard” meme, those who persist don’t have many excuses.