1. 37

    TL,DR - Twilio lost half their developers due to terrible build systems; they added a platform engineering team.

    1. 3

      Rereading How to Suppress Women’s Writing and having an uncomfortable conversation.

      1. 7

        There’s some value in supporting odd platforms, because it excercises portability of programs, like the endianness issue mentioned in the post. I’m sad that the endian wars were won by the wrong endian.

        1. 5

          I’m way more happy about the fact that the endian wars are over. I agree it’s a little sad that it is LE that won, just because BE is easier to read when you see it in a hex dump.

          1. 4

            Big Endian is easy for us only because we ended up with some weird legacy of using Arabic (right-to-left) numbers in Latin (left-to-write) text. Arabic numbers in Arabic text are least-significant-digit first. There are some tasks in computing that are easier on little-endian values, none that are easier on big-endian, so I’m very happy that LE won.

            If you want to know the low byte of a little-endian number, you read the first byte. If you want to know the top byte of a little-endian number, you need to know its width. The converse is true of a big-endian number, but if you want to know the top byte of any number and do anything useful with it then you generally do know its width because otherwise ‘top’ doesn’t mean anything meaningful.

            1. 2

              Likewise, there are some fun bugs only big endian can expose, like accessing a field with the wrong size. On little endian it’s likely to work with small values, but BE would always break.

          2. 2

            Apart from “network byte order” looking more intuitive to me at first sight, could you eloborate why big endian is better than little endian? I’m genuinely curious (and hope this won’t escalate ;)).

            1. 10

              My favorite property of big-endian is that lexicographically sorting encoded integers preserves the ordering of the numbers itself. This can be useful in binary formats. Since you have to use big-endian to get this property, a big-endian system doesn’t need to do byte swapping before using the bytes as an integer.

              Also, given that we write numbers with the most significant digits first, it just makes more “sense” to me personally.

              1. 5

                Also, given that we write numbers with the most significant digits first, it just makes more “sense” to me personally.

                A random fact I love: Arabic text is right-to-left, but writes its numbers with the same ordering of digits as Latin texts… so in Arabic, numbers are little-endian.

                1. 3

                  Speaking of Endianness: In Arabic, relationships are described from the end farthest from you to the closest, as in if you were to naively describe the husband of a second cousin, instead of saying “my mother’s cousin’s daughter’s husband” you would say “the husband of the daughter of the cousin of my mother” and it makes it insanely hard to hold it in your head without a massive working memory (because you need to reverse it to actually grok the relationship) but I always wonder if it’s because I’m not the most fluent Arabic speaker or if it’s a problem for everyone that speaks it.

                  1. 2

                    My guess is that it is harder for native speakers as well, but they don’t notice it because they are used to it. A comparable case I can think of is a friend of mine who is a native German speaker who came to the States for a post-doc. He commented that after speaking English consistently for a while, he realized that German two digit numbers are needlessly complicated. Three and twenty is harder to keep in your head than twenty three for the same reason.

                    1. 2

                      German has nothing to Danish.

                      95 is “fem og halvfems” - “five and half-five”, where the final five refers to five twentys (100), and the “half” refers to half of 20, i.e. 10, giving 90.

                      It’s logical once you get the hang of it…

                      In Swedish it’s “nittiofem”.

              2. 3

                Little-endian vs. big-endian has a good summary of the trade-offs.

                1. 3

                  I wondered this often and figured everyone just did the wrong thing, because BE seems obviously superior. Just today I’ve been reading RISC-V: An Overview of the Instruction Set Architecture and noted this comment on endianness:

                  Notice that with a little endian architecture, the first byte in memory always goes into the same bits in the register, regardless of whether the instruction is moving a byte, halfword, or word. This can result in a simplification of the circuitry.

                  It’s the first time I’ve noticed something positive about LE!

                  1. 1

                    From what I hear, it mostly impacted smaller/older devices with small buses. The impact isn’t as big nowadays.

                  2. 2

                    That was a bit of tongue-in-cheek, so I don’t really want to restart the debate :)

                    1. 2

                      Whichever endianness you prefer, it is the wrong one. ;-)

                      Jokes aside, my understanding is that either endianness makes certain types of circuits/components/wire protocols easier and others harder. It’s just a matter of optimizing for the use case the speaker cares about more.

                    2. 1

                      Having debugged on big-endian for the longest time, I miss “sane” memory dumps on little-endian. It takes a bit more thought to parse them.

                      But I started programming on the 6502, and little-endian clearly makes sense when you’re cascading operations 8 bits at a time. I had a little trouble transitioning to the big-endian 16-bit 9900 as a result.

                    1. 4

                      The Most Dangerous Code in the World has some other examples of TLS security vulnerabilities caused by poorly designed APIs.

                      1. 73

                        Apparently I need to remind people that:

                        1. No, it doesn’t matter if a cryptography blog has cartoons on it.
                        2. No, it’s off topic to rehash this meta discussion. Again. You can click the domain to find it on several earlier posts.
                        3. No, it’s not appropriate to flame the author because their blog has cartoons. This has been the final straw in at least one ban so far.
                        4. No, it’s not appropriate to flag the story for mod attention because it feels like you get to punish cartoons. Reread the previous point and consider whether you want mod attention. You can click ‘hide’ below the title if you want to not see the story or any comments to it.
                        1. 4

                          I feel this and I agree with you. However, I have flagged this as spam because this person keeps posting their own blog over and over. Maybe that’s not how that flag is intended to be used? I’m open to criticism about that. In any case I wanted to be clear about my motives for flagging this since it seems like there’s some other thing going on that I didn’t know about until reading this.

                          1. 46

                            A qualitatively good article every two weeks is hardly spam in my opinion.

                            1. 38

                              Spam is low effort, low quality, high volume. This is none of those things.

                              1. 28

                                It’s well written work, yes it’s posted regularly, but that doesn’t make it blog spam. The submissions tend to see engagement, so I think there’s some level of consensus there amongst the community.

                                1. 3

                                  there’s a previous thread somewhere discussing if self-posting should be permitted or not, there’s definitely arguments for both sides. some users routinely post their own site and it’s just really uninteresting stuff (including this post), but I don’t think that’s spam— it’s just content I’m not interested in personally.

                                  Also, there’s a user script for hiding posts from a domain/user/whatever. someone linked it to me in a previous thread but the story was nuked.

                                  1. 6

                                    Here’s the Greasemonkey script for blocking domains, stories, or users.

                                    1. 3

                                      Yeah, this is more in line with what my opinion is. I’m not arguing whether or not it’s well written and it’s certainly on topic for this site. I’m not trying to come in here to argue with folks. I flag something as spam if someone keeps posting “self-promotion” things like this. That still seems like spam to me but I’m fine being told I’m wrong as it’s clear what the consensus is. I won’t reply to each person that replied but I appreciate them all the same.

                                      1. 9

                                        If it helps to add context, I do submit things other than what I’ve written, if I believe they belong on this site. :)

                                        1. 4

                                          Yes, I see 4 out of your 15 posts aren’t something you authored. One of which comes after this thread started. Either way, my opinions aren’t personal attacks or facts… they’re just opinions. I mean you no disrespect.

                                          1. 7

                                            If it’s on-topic then it’s on-topic, and does it really matter who submitted it? In general I prefer it when authors submit their own stuff, because if I have some comment or criticism I can write “I don’t agree with you there”, “I think you are mistaken”, “I think this is unclear”, etc. and can then have a conversation with the author themselves, instead of “I don’t agree with the author”.

                                            1. 3

                                              Obviously, I think it matters or I wouldn’t have flagged it. Seems we prefer different things.

                                              1. 6

                                                If you don’t personally like it, even though it’s on topic and there’s nothing wrong with it, use the hide feature.

                                1. 4

                                  Is passlist and blocklist usage really so hard?

                                  1. 23

                                    Is it really so hard to realize many people are simply unaware of the controversy?

                                    Is it really so hard to realize people are creatures of habit and do not even realize they are using non-preferred nomenclature?

                                    Is it really so hard to realize that scolding them like this is counterproductive, because they will automatically become defensive?

                                    I could have phrased this nicer and more constructive. Could you?

                                    1. 16

                                      100% – I’m leaving my comment up as it was a silly and unconstructive knee jerk reaction. Thank you.

                                    2. 4

                                      This page is at least 9 years old; it was submitted to HN in 2011.

                                    1. 8

                                      I guess variants / disjoint-sum-types / algebraic datatypes will have to wait for PHP 81.0.

                                      1. 2

                                        PHP has unions since 8.0: https://www.php.net/manual/en/language.types.declarations.php#language.types.declarations.union

                                        (I realise that one cannot name such a union yet.)

                                        Did you mean something different by “disjoint sum type”?

                                        1. 2

                                          Yes, “union” are not necessarily disjoint. See the not-very-good wikipedia page where it is called “tagged union”. Basically an enumeration where each case can carry (typed) parameters, not just a name.

                                          If you don’t know of this feature, you are missing a fundamental concept about programming, which should be as primitive and regularly-used as pairs/tuples, but isn’t well-known because many programming languages ignore it completely, which is a shame. You should have a look!

                                          1. 3

                                            Yep, dynamic languages don’t have the syntax for it, but the use case of course exists, e.g. discriminating with a “kind” or “type” field. Thanks for the pointer though.

                                            Looking more into it, PHP has an RFC for tagged unions: https://wiki.php.net/rfc/tagged_unions and it seems that there is a larger effort to bring more features for algebraic types to PHP: https://wiki.php.net/rfc/adts

                                            1. 5

                                              Not-directly-related note: Some languages argue that because they already have constructs that enable dispatch-on-case, typically Go interfaces, they “don’t need tagged unions” which are a strict subset of their feature. I understand the argument (in the good cases, the dispatching constructs allow an encoding of typical pattern-matching that is tolerable, although it’s easy to overlook nested matching and other niceties). But I think it misses the point that tagged unions enables static exhaustiveness checking (does this switch/match cover all cases?), which is a huge win usability-wide: when I add a new case, the checker walks me through all the definitions that need to be updated. (Just like adding a new method in a class.) Scala realized this early, encoding algebraic datatypes as sealed classes. (Of course, this argument is moot for languages whose users are averse to static checking of their code.)

                                              1. 1

                                                Russ Cox’s post and the GitHub issue show some of the difficulties with adding sum types to Go.

                                      1. 1

                                        The “Open” part of the name continues to irk me.

                                        For anyone who doesn’t already know: It isn’t Open Source.

                                        1. 8

                                          The Open in OpenVMS refers to open systems and predates the term open source.

                                        1. 7

                                          I would have liked to read this, but I didn’t want to make an account. Is this for monetization so that the author can get money? I’m ok with that but do sites really need me to create an account for that? Can’t they just track views?

                                          1. 6

                                            Whenever I see a page on medium, I just automatically go to archive.is and look for a cached copy.

                                            Here’s this one.

                                            If there’s no cached copy there, I request one and it takes about two minutes.

                                            1. 5

                                              Sadly that’s Medium… no idea why people use it, you can always read it with Incognito Mode in your browser.

                                              1. 4

                                                I’ve just started adding medium.com to my ad blocklist. I do wish there was a way to filter specific sites on lobste.rs like you can filter tags[1].

                                                [1]: I realize it wouldn’t work for this url in particular, because the linked site uses several redirects (with cookies) under the hood, as well as cloudflare – e.g. medium “custom domain”. Still would filter out quite a few though.

                                                1. 3

                                                  There’s a Greasemonkey script for blocking domains and stories.

                                                  1. 1

                                                    That looks pretty sweet. Thanks! <3

                                              1. 21

                                                The article never mentions the, in my humble opinion, most important part of good logging practices and that is structured logging. Without it you end up with weird regexes or other hacks trying to parse your log messages.

                                                1. 4

                                                  As a sibling post notes, if you use structured logging you’re mostly throwing away the idea that the entries must be easily parsable by a human. If that’s the case, and we’ll need a custom method of displaying the structured logs in a human friendly way, I believe we should forego plain text all together and gain the benefits of logging directly to binary.

                                                  1. 5

                                                    You can do human readable structured logging if you use key="value" formats inside text messages. Some people still prefer json, but there is a middle ground.

                                                    1. 2

                                                      If you need just key=value, that’s not really structured in my opinion.

                                                      1. 4

                                                        Why not?

                                                        1. 2

                                                          Because the amount of information added by this format would be infinitesimal over a line based logger with manual tokenization. The reason why you’d want a structured logger is to allow proper context to a message. Unless you’re working with simple cases, the structure that would offer such context is more than one level deep.

                                                          1. 3

                                                            Hmm, definitely not.

                                                            Structured logging is about decorating log events with just enough of a schema to make them machine parseable, so that searching, aggregating, filtering, etc. can more than a crapshoot. Deeply nested events significantly increase the complexity of that schema, and therefore the requirements of the consumer.

                                                            By default, structured logs should be flat key/value pairs. It gets you the benefits of richer parseability, without giving up the ability to grep.

                                                  2. 2

                                                    Excellent point. That’s become such second nature to me by now, that I forgot to even mention it!

                                                    1. 1

                                                      On top of that, structured logger if implemented properly, can often be faster and be operated at granular levels (like the other comments pointed out, sometimes you do want to on-fly turn on some logs at some locations, not all logs at all locations).

                                                      1. 1

                                                        I love structured logging, with one caveat: the raw messages emitted (let’s assume JSON) are harder for me to scan when tailing directly (which I usually only do locally as we have better log querying tools in the cloud), in contrast to a semi-structured simple key-value format. Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                                        1. 5

                                                          At Joyent, we used the Bunyan format. Each line in the file was a separate JSON object with standard properties, some mandatory and some optional, and freeform additional properties. We shipped a tool, bunyan, that was capable of acting as a filter that would render different human readable views of the JSON. For example, you would often run something like:

                                                          tail -F $(svcs -L manatee) | bunyan -o short
                                                          

                                                          It also had some rudimentary filtering options. It also had a relatively novel mode that would, instead of reading from a file or standard input, use DTrace probes for different log levels to allow you to dynamically listen for DEBUG and TRACE events even when those were not ordinarily present in the log files. The DTrace mode could target a particular process, or even all processes on the system that emitted Bunyan logs.

                                                          1. 1

                                                            Hi, what were the required fields? Was it just a unique request ID? Thanks for sharing about bunyan. Even though it’s been out for a while I was unaware of it.

                                                          2. 5

                                                            Do you all use a different format than JSON? Or a tool that transforms structured logs to something more friendly to humans, eg. with different log levels displayed in different appropriate colors, eg. JSON syntax characters diminished, for local tailing?

                                                            We use JSON and the only tools I use are grep and jq. And although I am pretty much still a novice with these two, I found that with the power of shell piping I can do almost anything I want. Sometimes I reach for the Kibana web interface, get seriously confused and then go back to the command line to figure out how to do it there.

                                                            I wrote a simple tutorial for the process, just a couple of weeks ago.

                                                            1. 1

                                                              If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager? I feel like the systemd folks had the right idea even when everyone was making fun of them.

                                                              1. 3

                                                                I don’t think the average employer would be happy subsidizing an employee writing a log pager instead of implementing something that would bring a tangible result to the business. The potential money savings by using binary logs probably doesn’t outweigh the new subs/increased profits of churning out more features.

                                                                1. 1

                                                                  To me that sounds like an excuse. The world is not made up of only software that is beholden to the all mighty shareholder.

                                                                  1. 1

                                                                    I mean, yes, if you’re developing something in your personal time, go bananas on what you implement.

                                                                    But I also know my manager would look at me funny and ask why I’m not just shoving everything into CloudWatch/<cloud logging service>

                                                                2. 2

                                                                  I’m sure most problems with systemd journals are fixable, but they’ve left a very bad taste in my mouth for two main reasons: if stuff gets deleted from under them they apparently never recover (my services continue to say something like “journal was rotated” until I restart them), and inspecting journals is incredibly slow. I’m talking magnitudes slower than log files. This is at its worst (I often have time to make a cup of tea) when piping the output into grep or, as journalctl already does by default, less, which means every byte has to be formatted by journalctl and copied only to be skipped over by its recipient. But it’s still pretty bad (I have time to complain on IRC about the wait) when giving journalctl filters that reduce the final output down to a few thousand lines, which makes me suspect that there are other less fundamental issues.

                                                                  I should note that I’m using spinning disks and the logs I’m talking about are tens to hundreds of GB over a few months. I feel like that situation’s not abnormal.

                                                                  1. 1

                                                                    If you rely on external tools to be able to make sense of your logs, why not go all the way, gain the speed and size benefits that binary logs would bring, and write your own log pager?

                                                                    It’s hard to imagine a case at work where I could justify writing my own log pager.
                                                                    Here are some of the reasons I would avoid doing so:

                                                                    • Logs are an incidental detail to the application.
                                                                    • Logs are well understood; I can apply a logging library without issues.
                                                                    • My application isn’t a beautiful and unique snowflake. I should use the same logging mechanisms and libraries as our other applications unless I can justify doing something different.
                                                                    • JSON is boring, has a specification, substantial library support, tooling, etc.
                                                                    • Specifying, documenting, and testing a custom format is a lot of work.
                                                                    • Engineering time is limited; I try to focus my efforts on tasks that only I can complete.
                                                                    1. 2

                                                                      Logs are an incidental detail to the application.

                                                                      I think this is trivially disproved by observing that if the logs stop working for your service, that is (hopefully!) a page-able event.

                                                                      Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                                      1. 1

                                                                        Logs are a cross-cutting concern, but as essential as any other piece of operational telemetry.

                                                                        I rely heavily on logging for the services I support but the applications I wrote for work have only error reporting. They are used by a small audience and problems are rare; I might get a crash report every 18 months or so.

                                                                        1. 1

                                                                          Ah, yeah, I presume the context here is services.

                                                                  2. 1

                                                                    Agreed. jq is a really nice tool. It made the decision to transition to using JSON for logging very easy.

                                                                  3. 3

                                                                    Don’t use JSON, use logfmt.

                                                                    1. 1

                                                                      Yes! Logfmt is the good stuff. But it’s only semi-structured. Why not use JSON and a tool to transform to logfmt (with nested data elided probably) when needing to scan as a human?

                                                                      1. 1

                                                                        Logfmt is fully structured, it just doesn’t support nesting, which is an important feature! Structured logs should be flat.

                                                                  4. 1

                                                                    I’m surprised it wasn’t mentioned, but the larger advantage of passing a logger around to constructors is the ability to then have nested named loggers, such as

                                                                    Battery.ChargingStatus.FileReader: Failed to open file { file: "/tmp/battery charge", error: ... }
                                                                    Battery.ChargingStatus: Failed to access status logs, skipping report
                                                                    
                                                                  1. 1

                                                                    Trace scheduling and VLIWs might qualify; Josh Fisher’s PhD thesis Optimization of horizontal microcode within and beyond basic blocks was published in 1979. VLIW research continued at Yale as the ELI-512 project and Multiflow Computer was founded in 1986 to commercialize this work. Multiflow’s designs had substantial ILP - they executed 7, 14, or 28 instructions simultaneously.

                                                                    Here are a few other possibilities:

                                                                    1. 1

                                                                      I can’t think of any examples of “just in time” compilation prior to the 1990s, so that might be one new thing.

                                                                      1. 3

                                                                        The Alto used reprogrammable microcode to convert between OS-specific bytecode formats and the base machine code instructions, which (while hardware-assisted) is arguably comparable to the kind of JIT compilation performed in Java; Alto Smalltalk also compiled smalltalk code in real time so that running code could be edited live.

                                                                        However, JIT compilation is also more or less a performance enhancement. Had moore’s law not run out, we could reasonably imagine it not mattering at all, & having machines that had no compiled code at all other than some kind of interpreter living in firmware. There are all sorts of performance tricks that have been invented, & some are very clever, but their purpose is to increase the speed of conventional programs rather than change the kind of programs one can imagine using or writing – they are quantitative rather than qualitative improvements.

                                                                        1. 2

                                                                          The BBC Micro had a dialect of BASIC that had an inline assembler that could take a string, assemble it, and give you back a pointer that you could jump to from BASIC. One of the learn-to-program books at my school had a toy language that you could write and compile in a REPL.

                                                                          1. 2

                                                                            The term JITs may go back to the 1990s but the idea is much older; the 1974 thesis Adaptive systems for the dynamic run-time optimization of programs describes a JIT.

                                                                          1. 110

                                                                            I tell anyone asking me for career advice the same two things.

                                                                            The first: the deeper in the world’s dependency tree you are, the less frequently things will churn, and the longer your skills will last. TCP doesn’t change very often. Theoretical skills may be applicable for your entire career. Human skills are more durable than any technical skill. Kernels don’t change very often (but more than TCP). Databases don’t change very often (but more than kernels). There is a spectrum of skill durability, and you will burn out faster if you find that all of your skills become worthless after a very short time. Dependency forces things not to change their interface, which causes the work to shift toward performance and reliability among other things that some people find far more rewarding over time.

                                                                            The second: the more people who do what you do, the worse you will be treated, the more BS you will have to put up with, the worse your pay will be, the faster you will be fired, the harder it will be to find a job that values you, etc… etc… Supply and demand applies to our labor market, and if you want to be happier, you should exploit this dynamic as heavily as possible. Avoid competition like the plague. But don’t avoid funding. How do you avoid competition without going off into the wilderness where there is no money to be made? Hype drives funding, but it also drives a lot of competition. However, using rule #1 above, the hyped things depend on other things. Many of these dependencies are viewed as “too hard” for one reason or another. That’s the best place to be. Go where other people are afraid, but nevertheless have a lot of money depending on.

                                                                            All hyped things rely on things that for one reason or another are not commonly understood, and tend not to change quickly. That’s a good place to find work involving durable skills that tend to have lower competition. Go where the dependency is high but the competition is low, and you have a better chance of being happy than people who go where the competition is high or the dependency is low. Bonus points if it’s actually “hard” because then you won’t get bored as quickly.

                                                                            There are areas of front-end that are high-dependency, durable, slow-changing, and low-competition. That’s where engineers are likely to be happiest. But these two principles apply to every field or zooming out to any business generally. I’m pretty happy working on new distributed systems and database storage engines for the time being. But I’m always looking for the things that are viewed as hard while also receiving significant investment, as these are the things that will ultimately give me more opportunities to live life on my own terms.

                                                                            1. 10

                                                                              Go where other people are afraid, but nevertheless have a lot of money depending on.

                                                                              There is an old Yorkshire saying: “where’s there’s muck, there’s brass”.

                                                                              1. 7

                                                                                This is so true, I’ve gone to my car to fetch my laptop just to upvote and comment. It’s an exceptionally important piece of advice I wish I had understood as early as possible in life, but I didn’t.

                                                                                CS (pure) and Math degrees are so good because they teach you really basic theories that are mostly timeless. Whenever I’ve gravitated towards more trendy or applied skills, either in coursework or in jobs, there’s always been a really poor and transient ROI.

                                                                                […] using rule #1 above, the hyped things depend on other things. Many of these dependencies are viewed as “too hard” for one reason or another. That’s the best place to be.

                                                                                What are some examples of these dependencies right now, or in the near future?

                                                                                1. 6

                                                                                  Thank you very much for this post. Great distillation of essential career advice, especially the part about the durability of human skills. So many developers would derive far more value from a single public speaking or improv class than from learning yet another new programming language.

                                                                                  1. 3

                                                                                    Oh man, the number of times I’ve echoed this exact same message to others almost verbatim in the first two paragraphs. Thanks for posting this. Thinking about my career in this way a few years ago was probably the most valuable change I made.

                                                                                    1. 4

                                                                                      Thank you for the kind words :)

                                                                                      Large-scale ML, blockchain, IoT, serverless, k8s, etc… are all areas recently flooded by newly minted experts in the topical skills, but like the Australian poop diver who claims to have never worked a day in his life, there are great opportunities for high-respect jobs in the dirty internals of the systems :) Particularly with this set of hyped tech, there are very few people who seem to specialize in getting bugs to jump out in distributed systems. Everybody is still writing almost all of them in a way that assumes the happy path. But there are techniques for building all of these systems in ways that encourage the race conditions to jump out. The few people who take pride in building correct distributed systems will have their plates full for a while!

                                                                                      Another reason why this kind of bug hunting and prevention is not all that popular may be that the sorta-similar yet way-cooler-seeming field of systems security tends to absorb a lot of the people who may have otherwise been predisposed to this type of work.

                                                                                    2. 1

                                                                                      love this comment. like the other commenter, do you have any examples of “hard, but trendy” in frontend or elsewhere?

                                                                                      1. 7

                                                                                        Spitballing, I’m going to google “hyped programming field” and see what comes up, then I’ll try to break it into areas for investigation. Ok, my results on the first page seemed to be about AI, python, golang. I mentioned high-scale ML depending on distributed systems skills and correctness skills above, so I’ll think about the others. Python feels harder for me to answer so let’s dig into that.

                                                                                        Python is immensely popular and I think that becoming an expert in alleviating any friction point that people often hit will be lucrative. But what about things that SOUND really boring? When I think about things that suck about Python, the most dreadful thing that comes to my mind is migrating large applications from Python 2 to Python 3. Python 2 stuff is still everywhere, and it’s increasingly hazardous over time because its support has run out. But people don’t want to touch things that are still ticking along. Legacy systems, like geriatric medical care, becomes more important every day.

                                                                                        But despite being extremely important and an excellent place to apply interesting mechanical (yet supervised) translation and correctness analysis during modernization, so many people have been burned by consent issues at work where their managers forced them to work on legacy despite them not wanting to work on them. So much of the overall programming population has been burned by non-consensual legacy engineering that almost everyone wants to stay as far away as possible. The only question is how to find the companies who realize that modernizing their legacy systems is actually something you want to pay a specialist to do instead of forcing the task on junior devs with no negotiating power. Banks are well-known for paying consultants for this kind of work, but to be honest I’m not sure.

                                                                                        Maybe Python 2 modernization isn’t exactly a golden ticket due to the difficulty in finding companies who are willing to hire specialists to perform this task. Maybe it’s too easy. But may it’s not. I’m not sure. In any case, this maybe demonstrates the general search technique, and can be used to illuminate further areas for research. If general Python 2 modernization is “too easy” then you can slap on more filters. Python 2 modernization for automotive codebases. That sounds much more correctness-critical and likely to pay a specialist to accomplish.

                                                                                        Anyway, the general pattern is: what feels repulsive and hard? Maybe that sense of repulsion creates a dynamic where the few people who do the thing are extremely well treated due to their scarcity. If that scarcity of talent aligns with an extremely popular overall field, there’s a good chance that there are folks who require a specialist to address this niche.

                                                                                        1. 6

                                                                                          Anyway, the general pattern is: what feels repulsive and hard? Maybe that sense of repulsion creates a dynamic where the few people who do the thing are extremely well treated due to their scarcity. If that scarcity of talent aligns with an extremely popular overall field, there’s a good chance that there are folks who require a specialist to address this niche.

                                                                                          Here’s an example of scarcity working out well for an expert. Another company in my industry was converting to a different accounting system. These projects are incredibly complex and can last years. Part of the process required recreating hundreds of templates for the new system. One contractor handled the template conversion; she retired at a young age after the project was complete.

                                                                                          1. 3

                                                                                            non-consensual legacy engineering

                                                                                            Have you written about this anywhere? I’d be curious to hear more about it. It immediately struck a chord with me, and probably does with most software engineers.

                                                                                      1. 14

                                                                                        I just discovered that this can be detected with -Wpartial-fields, which is nice, but not part of -Wall.

                                                                                        Aaaaargh. Why even bother having a -Wall if it doesn’t have all warnings?

                                                                                        1. 3

                                                                                          That’s weird in GCC as well; it has -Wall and -Wextra, where the latter enables warnings not enabled by the former.

                                                                                          1. 4

                                                                                            IIRC the rationale is that -Wall enables everything uncontroversial, but the maintainers also wanted to include possibly controversial warnings, without making teams fight and in the end turn off -Wall.

                                                                                            Put differently, if some teams’ sensible code styles says to do x, then -Wall is indended to not warn about x. Or put differently again, if a new diagnostic couldn’t be added to a compiler because it would make -Wall too loud, then that is added but outside -Wall.

                                                                                            1. 1

                                                                                              gcc’s -Wall is rarely extended because many projects use -Werror and -Wall. Adding more warnings to -Wall can break their builds and results in angry messages to the GCC mailing list.

                                                                                            2. 2

                                                                                              Agreed! I think -Wall should be named -Wrecommended instead.

                                                                                              Haskell has -Weverything which really does turn on every warning. Amusingly it enables a warning that’s less strict than another warning, so you have to turn that one off. https://gitlab.haskell.org/ghc/ghc/-/issues/14794

                                                                                            1. 3

                                                                                              “It ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. Because the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new.”
                                                                                              – Niccolo Machiavelli, The Prince

                                                                                              Here are a few questions I would ask before attempting to change the approach to documentation.

                                                                                              • Do other people agree that documentation should be handled in a different manner?
                                                                                              • How does your management feel about this potential change?
                                                                                              • Is there a pilot project that you can demonstrate value on?
                                                                                                For example, a project that no one wants to touch or the existing documentation is a maze of twisty little passages.
                                                                                              • Is there a team or group that you can pilot this change with?
                                                                                                Some teams will have a higher ratio of innovators or early adopters and are probably more open to new ideas.
                                                                                              1. 1

                                                                                                Do other people agree that documentation should be handled in a different manner?

                                                                                                Nearly every people yes. Too easily some people say “write documentation”… but we all agree that people do not read it.

                                                                                                How does your management feel about this potential change?

                                                                                                They encourage us.

                                                                                                Is there a pilot project that you can demonstrate value on?

                                                                                                Yes completely. It is a Rails app. We already made for example commands like bin/start, bin/update or bin/setup very efficient. We removed 80 lines in our README with that change.

                                                                                                Is there a team or group that you can pilot this change with?

                                                                                                I think we have to demonstrate the efficiency on 1 project, then invite or force other projects to follow the same recommendations. We could write something short similar to “12 factors apps” about documentation.

                                                                                              1. 14

                                                                                                A bit offtopic, but could someone explain why there is always such a fuss around rolling out a new tag?

                                                                                                1. 10

                                                                                                  Categories and ontologies are inherently controversial; I saw the same behavior at work when we changed the ticketing system categories.

                                                                                                  There are at least the following points of view:

                                                                                                  • Categories should be narrow, e.g. left-handed avocado farmers.
                                                                                                  • Categories should be broad, e.g. operating systems
                                                                                                  • Categories are undesirable or the complexity of the domain make them useless
                                                                                                  • Categories should be defined based on a completely different basis
                                                                                                  • Categories should be user definable
                                                                                                  • Categories should be consistent even if the topic isn’t
                                                                                                  • Categories are a necessary evil and should be minimized to reduce cognitive load
                                                                                                  • Categories should cover the common cases
                                                                                                  • Categories should cover every possible case

                                                                                                  My rule of thumb is that a category has the right scope when you get complaints that it’s both too broad and too narrow.

                                                                                                1. 3

                                                                                                  This C library provides prepared statements for the system() function. It’s under the Apache license.

                                                                                                  1. 18

                                                                                                    TL,DR - some of these complaints are inaccurate.

                                                                                                    This rant came up 3 months ago and I spent some time checking it out. The rest of this post is derived from that post.

                                                                                                    • Some screen sharing and screen recording apps aren’t yet/won’t be ported to Wayland.
                                                                                                      This was equally true for other windowing system transitions - some apps weren’t ported from NeWS or SunView either. It doesn’t mean that Wayland is unsuitable for general use.
                                                                                                    • Wayland GTK4 breaks global menus in Gnome.
                                                                                                      This is a GTK issue. GTK4 removed support for a feature that is required by global menus. Wayland isn’t responsible for the development practices of the GNOME project.
                                                                                                    • Wayland broke global menus with KDE platformplugin.
                                                                                                      This issue has been resolved.
                                                                                                    • Wayland breaks global menus with non-KDE Qt platformplugins.
                                                                                                      This may have been fixed in 5.11.
                                                                                                    • Wayland breaks AppImages that don’t ship a special Wayland Qt plugin.
                                                                                                      This is another 5.11 issue, it may have been fixed.
                                                                                                    • A Youtube video from someone who prefers X11.
                                                                                                    1. 1

                                                                                                      If you’re looking for something more complex or interesting, here are a few possibilities:

                                                                                                      • learn a business domain
                                                                                                      • data science/machine learning
                                                                                                      • mobile development
                                                                                                      • fintech
                                                                                                      • HPC
                                                                                                      • embedded systems
                                                                                                      • systems programming
                                                                                                      • distributed systems
                                                                                                      1. 7

                                                                                                        I don’t want to 💩on the author’s writeup here, because it is a decent one. I’m using it to launch another public objection to Go Generics.

                                                                                                        A lot of proposals for and write ups about Go Generics seem to miss that there’s a very large group of Go users who object to Generics, and for good reason. It’s not because this group questions the efficacy of generics in solving very specific problems very well – objectors are generally well attuned to Generics’ utility. What’s objected to is the necessity of Generics. The question that we pose is do we need generics at all? Are the problems that Generics solve so important that Generics should pervade the language?

                                                                                                        From the author’s conclusion

                                                                                                        I was able to solve a problem in a way that was not previously possible.

                                                                                                        Being able to solve problems in new ways isn’t always valuable; it can even be counter-productive.

                                                                                                        1. 24

                                                                                                          Nothing is necessary except an assembler. Well, you don’t even need the assembler, you can just flip the bits yourself.

                                                                                                          Go has an expressiveness gap. It has some kind of big classes of algorithms that can’t be made into libraries in a useful way. Most people advocate just rewriting basically the same code over and over forever, which is kind of crazy and error-prone. Other people advocate code-generation tools with go generate, which is totally crazy and error-prone, even with the decent AST tools in the stdlib. Generics close the gap pretty well, they’re not insanely complex, and people have had decades to get used to them. If you don’t want to use them yourself, don’t use them, but accept that there are people for whom, say, the ability to just go get a red-black tree implementation that they can use with a datatype of their own choosing, without loss of type-safety or performance, will greatly improve the usefulness of the language.

                                                                                                          Plus, from a purely aesthetic standpoint, it always seemed criminal to me to have a language that has first-class functions, and lexical closure, but in which you can’t even write map because its type is inexpressible.

                                                                                                          1. 9

                                                                                                            Go has an expressiveness gap.

                                                                                                            That’s true. You’ve identified some of the costs. Can you identify some of the benefits, too?

                                                                                                            1. 12

                                                                                                              Easy: not having a feature protects you from bright idiots that would misuse it.

                                                                                                              Honestly though, that’s the only argument I can make against generic. And it’s not even valid, because you could say this about almost any feature. It’s a fully general counter argument: give people hammers, some will whack each other’s heads instead of hitting nails.

                                                                                                              Assuming basic competency of the users and assuming they were designed from the ground up, generics have practically no downsides. They provide huge benefits at almost no marginal cost. There is a sizeable up-front cost for the language designer and the compiler writer, but they were willing to pay that kind of price when they set out to build a general purpose languages, didn’t they?

                                                                                                              1. 2

                                                                                                                They provide huge benefits at almost no marginal cost.

                                                                                                                If this huge benefit is only one in a minor part of the project, or even, in a minority of projects, then it has to be balanced and thought through.

                                                                                                                Right now, I don’t know many people that work Go daily, telling me that not having generics makes their day a pain.

                                                                                                                Most of them told me that it’s sometimes painful, but that’s actually pretty rare.

                                                                                                                There is a sizeable up-front cost for the language designer and the compiler writer, but they were willing to pay that kind of price when they set out to build a general purpose languages, didn’t they?

                                                                                                                Is the burden really on them? To me the it is on the program writer.

                                                                                                                1. 8

                                                                                                                  There’s likely a survivorship bias going on there.

                                                                                                                  I used Go as a programming language for my side projects for years. The thing that finally got me to give it up was the lack of generics. In writing PISC, the way I had approached it in Go ended up causing a lot of boilerplate for binding functions.

                                                                                                                  Go is something I’d happily write for pay, but I prefer expressiveness for my side projects now, as the amount of effort that goes into a side project is a big determining factor in how much I can do in one

                                                                                                                  1. 3

                                                                                                                    There is a sizeable up-front cost for the language designer and the compiler writer, but they were willing to pay that kind of price when they set out to build a general purpose languages, didn’t they?

                                                                                                                    Is the burden really on them? To me the it is on the program writer.

                                                                                                                    Assuming we are a collaborative species (we mostly are, with lots of exceptions), then one of our goals should be minimizing total cost. Either because we want to spend our time doing something else, or because we want to program even more stuff.

                                                                                                                    For a moderately popular programming language, the users will far outnumber and outproduce the maintainers of the language themselves. At the same time, the languages maintainers’ work have a disproportionate impact on everyone else. To such a ludicrous extent in fact that it might be worth spending months on a feature that would save users a few seconds per day. Like compilation speed.

                                                                                                                    Other stuff like generic will affect fewer users, but (i) it will affect them in a far bigger way than shaving off a few seconds of compilation time would have, and (ii) those particular users tend to be library writers, and as such they will have a significant impact on the rest of the community.

                                                                                                                    So yes, the burden really is on the language creators and compiler writers.


                                                                                                                    Note that the same reasoning applies when you write more mundane software, like a train reservation system. While there is rarely any monetary incentive to make that kind of thing not only rock solid, but fast and easy to work with, there is a moral imperative not to inflict misery upon your users.

                                                                                                                2. 5

                                                                                                                  I haven’t used Go in anger but here are some benefits from not including generics.

                                                                                                                  • Generics are sometimes overused, e.g. many C++ libraries.
                                                                                                                  • The type system is simpler.
                                                                                                                  • The compiler is easier to implement and high quality error messages are easier to produce.
                                                                                                                  • The absence of generics encourages developers to use pre-existing data structures.
                                                                                                                3. 2

                                                                                                                  If red-black trees and map were just built in to Go, wouldn’t that solve 90% of the problem, for all practical purposes?

                                                                                                                  What I really miss in Go is not generics, but something that solves the same problems as multiple dispatch and operator overloading.

                                                                                                                  1. 3

                                                                                                                    Sort of, but no. There’s too many data structures, and too many useful higher-order functions, to make them all part of the language. I was just throwing out examples, but literally just a red-black tree and map wouldn’t solve 90% of the problem. Maybe 2%. Everyone has their own needs, and Go is supposed to be a small language.

                                                                                                                    1. 1

                                                                                                                      Data structures and higher-order functions can already be implemented in Go, though, just not by using generics as part of the language.

                                                                                                                4. 15

                                                                                                                  Technically Go does have generics, they just aren’t exposed to the end developer, except in the form of the builtin map and array types, and are only allowed for internal developers. So in a sense, Go does need generics and they already pervade the language.

                                                                                                                  I don’t personally have a horse in this race and don’t work with Go, but from a language-design perspective it does seem strange to limit user-developed code in such a way. I’d be curious what your thoughts on why this discrepancy is OK and why it shouldn’t be fixed by adding generics to the language.

                                                                                                                  1. 14

                                                                                                                    I don’t personally have a horse in this race and don’t work with Go, but from a language-design perspective it does seem strange to limit user-developed code in such a way.

                                                                                                                    Language design is all about limiting user defined code to reasonable subsets of what can be expressed. For a trivial example, why can’t I name my variable ‘int’? (In Myrddin, as a counterexample, var int : int is perfectly legal and well defined).

                                                                                                                    For a less trivial example, relatively few languages guarantee tail recursion – this also limits user developed code, and requires programmers to use loops instead of tail recursion or continuation passing style.

                                                                                                                    Adding generics adds a lot of corner cases to the type system, and increases the complexity of the language a good deal. I know. I implemented generics, type inference, and so on in Myrddin, and I’m sympathetic to leaving generics out (or, as you say, extremely limited) to put a cap on the complexity.

                                                                                                                    1. 3

                                                                                                                      I see only two legitimate reasons to limit a user’s capabilities:

                                                                                                                      1. Removing the limitation would make the implementer’s life harder.
                                                                                                                      2. Removing the limitation would allow the user to shoot themselves in the foot.

                                                                                                                      Limiting tail recursion falls squarely in (1). There is no way that guaranteeing tail recursion would cause users to shoot themselves in the foot. Generics is another matter, but I strongly suspect it is more about (1) than it is about (2).

                                                                                                                      Adding generics adds a lot of corner cases to the type system, and increases the complexity of the language a good deal.

                                                                                                                      This particular type system, perhaps. This particular language, maybe. I don’t know Go, I’ll take your word for it. Thing is, if Go’s designers had the… common sense not to omit generics from their upcoming language, they would have made a slightly different language, with far fewer corner cases they will inevitably suffer now that they’re adding it after the fact.

                                                                                                                      Besides, the complexity of a language is never a primary concern. The only complexity that matters is that of the programs written in that language. Now the complexity of a language does negatively impact the complexity of the programs that result from it, if only because language space is bigger. On the other hand, this complexity has the potential to pay for itself, and end up being a net win.

                                                                                                                      Take C++ for instance. Every single feature we add to it increases the complexity of the language, to almost unbearable levels. I hate this language. Yet, some of its features definitely pay for themselves. Range for for instance, while it slightly complicates the language, makes programs that use it significantly cleaner (although only locally). That particular feature definitely pays for itself. (we could discuss other examples, but this one has the advantage of being uncontroversial.)

                                                                                                                      As far as I can tell, generics tend to massively pay for themselves. Not only do they add flexibility in many cases, they often add type safety (not in C++, they don’t). See for instance this function:

                                                                                                                      foo : (a -> b) -> [a] -> [b]
                                                                                                                      

                                                                                                                      This function has two arguments (where a and be are unknown types): a function from a to b, and a list of a. It returns a list of b. From this alone, there is a lot we can tell about this function. The core idea here is that the body of the function cannot rely on the contents of generic types. This severely constraints what it can do, including the bugs it can have.

                                                                                                                      So, when we write let ys = foo f xs, here’s what we can expect before we even look at the source code:

                                                                                                                      • Assuming f is of type a->b, then xs is a list of a, and the result ys is a list of b.
                                                                                                                      • The elements of ys, if any, can only come from elements of xs.
                                                                                                                        • And they must have gone through f.
                                                                                                                        • Exactly once.
                                                                                                                      • The function f itself does not affect the number or order of elements in the result ys
                                                                                                                      • The elements of xs do not individually affect the number or order of elements in the result ys
                                                                                                                      • The only thing that affects the number or order of elements in the result ys is the size of xs (and the code of foo, of course).

                                                                                                                      This is quite unlike C++, or other template/monomorphisation approaches. Done right, generics have the opportunity to remove corner cases in practice. Any language designer deciding they’re not worth their while better have a damn good explanation. And in my opinion, the explanations offered for Go weren’t satisfactory.

                                                                                                                      1. 4

                                                                                                                        Complexity of a language is the primary concern!

                                                                                                                        Languages are tools to express ideas, but expressiveness is a secondary concern, in the same way that the computer is the secondary audience. Humans are the primary audience of a computer program, and coherence is the primary concern to optimize for.

                                                                                                                        Literary authors don’t generally invent new spoken languages because they’re dissatisfied with the expressive capability of their own. Artful literature is that which leverages the constraints of it’s language.

                                                                                                                        1. 4

                                                                                                                          Literary authors don’t generally invent new spoken languages because they’re dissatisfied with the expressive capability of their own. Artful literature is that which leverages the constraints of it’s language.

                                                                                                                          Eh, I have to disagree here. Literary authors try to stretch and cross the boundaries the of their spoken languages all the time, specifically because they search ways to express things that where not yet expressed before. To give some uncontroversial examples, Shakespeare invented 1700 new words and Tolkien invented not one, but a couple of whole new languages.

                                                                                                                          I am but a very low level amateur writer, but I can tell you: the struggle with the tool to express your ideas is as real with spoken languages as it is with programming languages. It is an approach from another direction, but the results from spoken languages turn out to be as imperfect as those from programming ones.

                                                                                                                          1. 1

                                                                                                                            I’d argue that constrained writing is more common, if nothing else than showing ones mastery of a shared language is more impressive than adding unknown elements.

                                                                                                                            Tolkien’s Elvish languages, while impressively complete, are simply used as flavor to the main story. The entire narrative instead leans heavily on tropes and language patterns from older (proto-English) tales.

                                                                                                                            1. 1

                                                                                                                              Yes, you have a point. I mentioned Tolkien because that was the first writer that created a new language that I could come up with. But in the end, if you want to express an idea, then your audience must understand the language that you use, otherwise they will not get your message. So common language and tropes can help a lot.

                                                                                                                              However, I think your mention of constrained writing is interesting. Because in a way, that Go does not have generics, is similar to the constraint that a sonnet must follow a particular scheme in form and content. It is perfectly possible to add generics to Go, the same way as it is very possible to slap another tercet at the end of a sonnet. Nothing is stopping you, really, Expect that then it would no longer be a sonnet. Is that a bad thing? I guess not. But still almost no-one does it.

                                                                                                                              I’d say that the rules, or the constraints, are a form of communication too. If I read a sonnet, I know what to expect. If I read Go, I know what to expect. Because some things are ruled out, there can be more focus on what is expressed within the boundaries. As a reader you can still be amazed. And, the same as in Go, if what you want to express really does not fit in the rules of a sonnet, or if it is not worth the effort to try it, then you can use another form. Or another programming language.

                                                                                                                            2. 1

                                                                                                                              Your points don’t conflict with my points, and I agree with them.

                                                                                                                            3. 2

                                                                                                                              Can we agree that the goal of programming languages is to reduce costs?

                                                                                                                              • Cost of writing the program.
                                                                                                                              • Cost of errors that may occur.
                                                                                                                              • Cost of correcting those errors.
                                                                                                                              • Cost of modifying the program in the face of unanticipated new requirements.

                                                                                                                              That kind of thing. Now we must ask what influences the costs. Now what about increased expressiveness?

                                                                                                                              A more expressive language might be more complex (that’s bad), more error prone (that’s bad), and allow shorter programs (that’s good), or even clearer programs (that’s good). By only looking at the complexity of the language, you are ignoring many factors that often matter a whole lot more.

                                                                                                                              Besides, that kind of reasoning quickly breaks down when you take it to its logical extreme. No one in their right mind would use the simplest language possible, which would be something like the Lambda Calculus, or even just the iota combinator. Good luck writing (or maintaining!) anything worth writing in those.

                                                                                                                              Yes, generics makes a language more complex. No, that’s not a good enough argument. If it was, the best language would only use the iota combinator. And after working years in a number of languages (C, C++, OCaml, Ptython, Lua…), I can tell with high confidence that generics are worth their price several orders of magnitudes over.

                                                                                                                              1. 2

                                                                                                                                I agree with you that generics can be hugely net positive in the cost/benefit sense. But that’s a judgment that can only be made in the whole, taking into account the impact of the feature on the other dimensions of the language. And that’s true of all features.

                                                                                                                                1. 1

                                                                                                                                  Just popping in here because I have minimal experience with go, but a decent amount of experience in languages with generics, and I’m wondering: if we set aside the implementation challenge, what are some examples of the “other dimensions” of the language which will be negatively impacted by adding generics? Are these unique to go, or general trade offs in languages with generics?

                                                                                                                                  To frame it in another way, maybe a naive take but I’ve been pretty surprised to see generics in go being rejected due to “complexity”. I agree that complexity ought to be weighed against utility but can we be a little more specific? Complexity of what specifically? In what way will writing, reading, compiling, running, or testing code become more complicated when my compiler supports generics. Is this complexity present even if my own code doesn’t use generics?

                                                                                                                                  And just a final comparison on language complexity. I remember when go was announced, the big ticket feature was its m:n threaded runtime and support for CSP-style programming. These runtimes aren’t trivial to implement, and certainly add “complexity” via segmented stacks. But the upside is the ability to ergonomically express certain kinds of computational processes that otherwise would require much more effort in a language without these primitives. Someone decided this tradeoff was worth it and I haven’t seen any popular backlash against it. This feature feels very analogous to generics in terms of tradeoffs which is why I’m so confused about the whole “complexity” take. And like, maybe another naive question, but wouldn’t generics be significantly less tricky to implement than m:n threads?

                                                                                                                                  1. 5

                                                                                                                                    It isn’t just implementation complexity of generics itself. It’s also sure to increase the complexity of source code itself, particularly in libraries. Maybe you don’t use generics in your code, but surely some library you use will use generics. In languages that have generics, I routinely come across libraries that are more difficult to understand because of their use of generics.

                                                                                                                                    The tricky part is that generics often provides some additional functionality that might not be plausible without it. This means the complexity isn’t just about generics itself, but rather, the designs and functionality encouraged by the very existence of generics. This also makes strict apples-to-apples comparisons difficult.

                                                                                                                                    At the end of the day, when I come across a library with lots of type parameters and generic interfaces, that almost always translates directly into spending more time understanding the library before I can use it, even for simple use cases. That to me is ultimately what leads me to say that “generics increases complexity.”

                                                                                                                                    1. 2

                                                                                                                                      what are some examples of the “other dimensions” of the language which will be negatively impacted by adding generics?

                                                                                                                                      From early golang blog posts I recall generics add substantial complexity to the garbage collector.

                                                                                                                                      The team have always been open about their position (that generics are not an early priority, and they will only add them if they can find a design that doesn’t compromise the language in ways they care about). There have been (numerous proposals rejected)[https://github.com/golang/go/issues?page=3&q=generics++is%3Aclosed+label%3AProposal] for varied reasons.

                                                                                                                                      Someone decided this tradeoff was worth it and I haven’t seen any popular backlash against it

                                                                                                                                      There’s no backlash against features in new languages, because there’s nobody to do the backlash.

                                                                                                                                      Go has already got a large community, and there’s no shortage of people who came to go because it was simple. For them, adding something complex to the language is frightening because they have invested substantial time in an ecosystem because of its simplicity. Time will tell whether those fears were well-founded.

                                                                                                                                2. 1

                                                                                                                                  No, expressiveness is the only reason for languages to exist. As you say, humans are the primary audience. With enough brute force, any language can get any task done, but what we want is a language that aids the reader’s understanding. You do that by drawing attention to certain parts of the code and away from certain parts, so that the reader can follow the chain of logic that makes a given program or function tick, without getting distracted by irrelevant detail. A language that provides the range of tools to let an author achieve that kind of clarity is expressive.

                                                                                                                                  1. 2

                                                                                                                                    I think we are using “expressive” differently. Which is fair, it’s not really a well-defined term. But for me, expressiveness is basically a measure of the surface area of the language, the features and dimensions it offers to users to express different ideas, idioms, patterns, etc. Importantly, it’s also proportional to the number of things that it’s users have to learn in order to be fluent, and most of the time actually exponentially proportional, as emergent behaviors between interacting features are often non-obvious. This is a major cost of expressiveness, which IMO is systemically underestimated by PLT folks.

                                                                                                                                3. 3

                                                                                                                                  I implemented generics. You’re trying to convince me that it’s worth implementing generics. Why?

                                                                                                                                  Besides, the complexity of a language is never a primary concern.

                                                                                                                                  I disagree. I think implementation matters.

                                                                                                                              2. 2

                                                                                                                                That’s an intersting observation; thanks for sharing it.

                                                                                                                                they just aren’t exposed to the end developer

                                                                                                                                I think this supports my point better than I’m able to. Language design is just as much about what is hidden from developers as what is exposed. That generics are hidden from end users is something I greatly appreciate about Go. So when I refer to generics, I’m referring to generics used by every day developers.

                                                                                                                                I’d be curious what your thoughts on why this discrepancy is OK and why it shouldn’t be fixed by adding generics to the language.

                                                                                                                                In my opinion the greatest signal that Go doesn’t need generics is the wonderfully immense corpus of code we have from the last decade – all written without generics. Much of it written with delight by developers who chose Go over other langauges for it’s pleasant simplicity and dearth of features.

                                                                                                                                That is not to say that some of us offasionally could have written less code if generics were available. Particularly developers writing library or framework code that would be used by other developers. Those developers absolutely would have been aided by generics. They would have written less code; their projects may have cost less to initially develop. But for every library/framework developer there are five, ten, twenty (I can’t pretend to know) end user application developers who never had the cognitive load of genericized types foisted on them. And I think that is an advantage worth forgoing generics. I don’t think I’m particularly smart. Generics make code less readable to me. They impose immense cognitive load when you’re a new developer to a project. I think there are a lot of people like me. After years of Java and Scala development, Go to me is an absolute delight with its absence of generics.

                                                                                                                                1. 6

                                                                                                                                  In my opinion the greatest signal that Go doesn’t need generics is the wonderfully immense corpus of code we have from the last decade

                                                                                                                                  I don’t have a ready example, but I’ve read that the standard library itself conspicuously jumped through hoops because of the lack of generics. I see it as a very strong sign (that’s an understatement) that the language has a dire, pervasive, need for generics. Worse, it could have been noticed even before the language went public.

                                                                                                                                  If you had the misfortune of working with bright incompetent architects astronauts who used generics as an opportunity to make an overly generic behemoth “just in case” instead of solving the real problem they had in front of them, well… sorry. Yet, I would hesitate to accuse the language’s semantics for the failings of its community.

                                                                                                                              3. 7

                                                                                                                                I don’t remember exact details, it was super long ago, but I once wanted to write an editor centered around using a nontrivial data structure (“table chain” or “string table” or whatever was the name). Also the editor had some display aspect structures (~cells of terminal). At some point I needed to be able to experiment with rapidly changing the type of the object stored both in the “cells” and “chains” of the editor (e.g. to see if adding styles etc. per character might make sense from architectural point of view). If you squint, those are both kind of “containers” for characters (haskeller would maybe say monads? dunno). I had to basically either manually change all the places where the original “character” type was used, or fall back to interface{} losing all benefits of static typing that I really needed. Notably this was long before type aliases which would have possibly allowed me to push a bit further, though it’s hard for me to recall now. But the pain and impossibility of rapid prototyping at this point was so big I didn’t see it possible to continue working on the project and abandoned it. Not sure if immediately then or some time later I realized that this is the rare moment where generics would be valuable in letting me explore designs I cannot realistically explore now.

                                                                                                                                In other words, what others say: nontrivial/special-purpose “containers”. You don’t need them until you do.

                                                                                                                                Until then I fully subscribed to “don’t need generics in Go” view. Since then I’m in “don’t need generics in Go; except when do”. And I had one more hobby project afterwards that I abandoned for exactly the same reason.

                                                                                                                                And I am fearful and do lament that once they are introduced, we’ll probably see everyone around abusing them for a lot of unnecessary purposes, and that this will be a major change to the taste of the language. That makes me respect the fact that the Team are taking their time. But I do miss them since, and if the Team grudgingly accepts the current draft as passabke, this is such a high bar that it makes me extremely excited for what’s to come, that it will be one of the best ways how this compromise can be introduced. Given that most decisions in languages are some compromises.

                                                                                                                                1. 6

                                                                                                                                  Yeah, Go is very much not a language for rapid prototyping. It expects you to come to the table with a design already in mind.

                                                                                                                                  1. 2

                                                                                                                                    Umm, what? Honestly not sure if you’re meaning this or being sarcastic (and if yes, don’t see the point). I prototyped quite a lot of things in Go no problem. I actually hold it as one of the preferred languages for rapid prototyping if I expect I might want to keep the result.

                                                                                                                                    1. 5

                                                                                                                                      I’m being totally serious. Go is chock full of stuff that makes typical rapid prototyping extremely difficult. A lack of a REPL. Compiler errors on unused variables. Verbose error handling. And so on. All of these things combine to make it harder to “design on the fly”, so to speak, which is what rapid prototyping frequently means.

                                                                                                                                      With that said, Go works great for prototyping in the “tracer bullet” methodology. That’s where your prototype is a complete and production quality thing, and the iteration happens at a higher level.

                                                                                                                                      1. 1

                                                                                                                                        Got it, thanks! This made me realize that I reach for different languages in different cases for prototyping. Not yet really sure why now. But I feel that sometimes the dynamic types of Lua make me explore faster, whereas sometimes static types of Go or Nim make me explore faster.

                                                                                                                                2. 4

                                                                                                                                  I’m going to assume you’re arguing in good faith here, but as a lurker on the go-nuts mailing list, I’ve seen too many people say “I don’t think generics are necessary” or “I haven’t heard a good enough reason for the complexity of generics”. It’s worth pointing out the Go team has collected feedback. Ian Lance Taylor (one of the current proposal’s main authors) spends a large portion of time responding to emails/questions/objections.

                                                                                                                                  I read a comment from someone who was on the Kubernetes team that part of the complexity of the API (my understanding is they have a pseudo-type system inside) is based on the fact that proto-Kubernetes was written in Java and the differences between the type systems compounded with a lack of generics created lots of complexity. (NOTE I don’t remember who said this, and I am just some rando on the net, but that sounds like a decent example of the argument for generics. Yes, you can redesign everything to be more idiomatic, but sometimes there is a compelling need to do things like transfer a code base to a different language)

                                                                                                                                  1. 1

                                                                                                                                    Ouch, I was wondering why the Kubernetes API looks so painfully like Java and not like Go. TIL that’s because it was literally a dumb translation from Java. :/ As much as I’m a pro-generics-in-Go guy, I’m afraid that’s a bad case for an argument, as I strongly believe it is a really awful and unidiomatic API from Go perspective. Thus I by default suspect that if its authors had generics at their disposal, they’d still write it Java-style and not Go-style, and probably still complain that Go generics are different from Java generics (and generally that Go is not Java).

                                                                                                                                  2. 3

                                                                                                                                    I don’t know if the author’s example was a good one to demonstrate the value of generics, but a cursory look at the diff would suggest he didn’t really gain anything from it. I always thought a huge benefit of generics was it saved you 10s or even 100s of lines of code because you could write one generic function and have it work for multiple types. He ended up adding lines. Granted, the author said it was mostly from tests, but still there doesn’t seem to be any dramatic savings here.

                                                                                                                                    1. 3

                                                                                                                                      I recommend taking more than a cursory look. The value here is very much in the new library interface. In effect, the package provides generalize channels, and before the change, that generalization meant both a complicated interface, and losing compiler-enforced type safety.