1. 3

    Under “Cons” for the foreign key approach:

    Additional tables to maintain — which is a SIGNIFICANT cost

    Let’s imagine adding a new table just to store the list of genders

    I don’t see why creating and populating a reference table with a fixed set of values is significantly more costly to maintain than adding an enum type. Yes, it’s a CREATE TABLE and an INSERT rather than a single CREATE TYPE, but then… you’re done. One additional SQL statement in a migration script doesn’t seem like a significant maintenance burden to me.

    My hunch is that the author was thinking about the “Foreign keys mean users can maintain the list of values” aspect and the significant cost would mostly be adding user-facing functionality to edit the table. But the cost there is in the additional functional requirement (user-maintainable lists of values), not the fact that you’re using a table instead of a type. If you had the same functional requirement with an enum-based implementation, it would be even more costly.

    1. 1

      Tables are also less efficient — accessing a table, even by an efficient primary key type, costs IO in a way an enum or function call does not. And, so does looking up and enforcing foreign keys. As your data grows, foreign keys are the first thing to go — it’s usually not worth the read amplification. But still, I think using a table for this is worth it if you want user-configurable enums and you’re not expecting huge amounts of data.

      1. 3

        In the case of small tables, it’s going to be in the page cache essentially every time. It would be a rare configuration that actually required a syscall, let alone a disk read.

    1. 3

      From the comparison page of the linked compiler it looks like the free edition can only be used to create GPL licensed software. Is there a good free Ada compiler that does not restrict how you license what you write?

      1. 6

        You can use GCC. GNAT Pro is not GCC.

        Edit: Explanation of how this works with GPL’s “no further restrictions”. GNAT standard library is copyrighted by FSF and distributed under GPL with linking exception. Thanks to linking exception, there is no restriction to linking standard library. AdaCore redistributes FSF-copyrighted GNAT standard library under GPL, without linking exception. Just as any other software licensed under GPL, your application should be GPL if you link AdaCore distributed GNAT standard library.

        GPL forbids further restrictions, but GPL has no problem whatsoever with dropping exceptions. You get the same code with same copyright, but only FSF gives you exception, AdaCore does not give you exception unless you pay. Instead of paying, you could get exception from FSF, but while AdaCore spends efforts to advertise, FSF does not. AdaCore especially does not advertise you can get the same exception from FSF without paying AdaCore. As you have shown, apparently this “revenue by obscurity” is very effective in practice.

        Edit 2: Explanation of how this works with GPL’s “you must show users these terms so they know their rights”. GPL requires you to include GPL in any redistributions. But even if you received it with exceptions, there is no requirement whatsoever you should let your users know about exceptions. GPL exceptions are not rights, so they are not covered by “know their rights” provision.

        1. 13

          As you have shown, apparently this “revenue by obscurity” is very effective in practice.

          Disclaimer: I work at AdaCore.

          The reason companies pay AdaCore is that they want support, not that they want to be able to write proprietary software. According to the people in our sales team, “Why should we pay you instead of using the FSF’s GNAT?” is a question that often comes up during negotiations and the answer always is “Because you won’t get access to experts that can answer your questions, fix your bugs in a timely manner and provide insurance in case something goes wrong if you don’t”. Companies chose to pay because in the safety-critical world you can’t afford to not have support for the tools you rely on.

          It’s true that the license situation is confusing though. As far as I understand this is why AdaCore is planning on discontinuing GNAT community and instead will start helping linux distributions ship up to date versions of its tools.

          1. 2

            AdaCore is planning on discontinuing GNAT community

            Don’t forget the Windows users. Given the current dependency on Makefiles, support already isn’t super great, and this would probably incentivize me to go back to writing Rust or C++17.

            Companies chose to pay because in the safety-critical world you can’t afford to not have support for the tools you rely on.

            This is true in all software, not just safety-critical. I’ve seen this happen many times, and companies don’t understand that it’s better to not burn $50-100+/hr per developer on a team when tools don’t work as intended.

            1. 2

              AdaCore is planning on discontinuing GNAT community

              Don’t forget the Windows users.

              First please don’t take my word as an absolute truth - I am not involved in any of the circles that actually decide what should happen regarding AdaCore’s involvement with the Ada community and I may have misunderstood some of the things I heard. In the end, I don’t really know what’s going to happen wrt Gnat Community.

              If what I understood is correct, the plan for windows users would be to support them with Mingw (or maybe another distribution of linux tools for windows whose name escapes me). I remember that one of the other alternatives discussed was to rely on Alire for toolchain management (kind of like rustup/cargo). The other tools (gprbuild, GNAT Studio) that won’t be shipped with MinGW would still be available from AdaCore’s website. I think one of the other goals (aside from clearing the license confusion) of this move is to have the Ada community be more self-reliant, so that people would stop seeing AdaCore as “owning” the ecosystem and rather as just one of its actors.

              1. 3

                Disclaimer: I don’t work for AdaCore, or anyone Ada-related. I’m a C++ grunt.

                aside from clearing the license confusion

                This is a great move since one of my major gripes is the usage of GPL, of which many companies like to steer well clear.

                so that people would stop seeing AdaCore as “owning” the ecosystem and rather as just one of its actors.

                I think this is a really good goal. Alire is neat. Mingw doesn’t really cut it though, WSL is ok, but native support would be the best. Yeah, I get it’s a lot of toolchain work. It’d be nice if Microsoft built it into Visual Studio, which would be a legitimate option since there’s a formal spec and the ACATS test suite.

                I wouldn’t be working at all in Ada if not for the work groundwork for an “Ada Renaissance” so to speak, laid by AdaCore in the last few years (language server, llvm compiler, libadalang, ada language server, the learning site, and quality youtube videos.

                Anyone who felt like they missed the bus on getting involved on the ground floor of a language (like on Alire) definitely has huge opportunities in Ada.

                1. 1

                  What’s wrong with Mingw?

              2. 1

                companies don’t understand that it’s better to not burn $50-100+/hr per developer on a team when tools don’t work as intended.

                I think this is somewhat situational. Having good support for a tool can indeed save a lot of developer time. But paying for support doesn’t guarantee that the support will be useful when you need it.

                Low-quality support is one problem, but response time can be an issue too: as a developer, if a tool issue is blocking me from getting my work done, it’s often the case that I can dig into it and figure out a workaround or a fix in less time than it takes to get an initial response, let alone a resolution, from a vendor’s support people. I’m costing the company just as much money when I’m twiddling my thumbs waiting for vendor support as I would if I were digging into the problem myself.

                Obviously that depends hugely on which tools we’re talking about and on my level of expertise; it won’t be true for all tools and all developers.

                That said, it’s been my experience that tool issues generally fall into three buckets: things I can figure out on my own in a reasonable amount of time, things I can’t figure out on my own but the vendor could solve in a reasonable amount of time, and things that would take the vendor a long time to solve. And the middle bucket is almost always much smaller than the other two.

                None of which is to say that paying for support is a waste of money. But I think depending on the situation, it can also be rational to decide that the net cost is lower if developers figure out tool issues on their own.

              3. 1

                Thank you for your comment, it put things a bit in perspective for me.

                I don’t purport to know if getting free tools in the hands of as many potential developers as possible is a good business decision or not, but I fear that if you manage to scare away even a percentage of potential new users, it may hurt the chances for Ada, the language, to grow.

              4. 2

                GPL forbids further restrictions, but GPL has no problem whatsoever with dropping exceptions

                This is not true. The GPL explicitly forbids redistributing with weaker restrictions. If this were not the case, you could combine a GPL’d file in a BSDL library and distribute the result under the BSDL, then incorporate this in a proprietary product, defeating the point of the GPL. You can; however, create a new license that is the GPL + some exceptions. This is what the FSF does with the GCC linking exemption, for example[1].

                This distinction is important. Your phrasing suggests that it would be possible to take an existing GPL’d file and incorporate it into the GNAT standard library. Doing so would violate the GPL (if you distributed the result) unless the copyright holder agreed to the relicensing. The FSF works around this by requiring copyright assignment.

                [1] Note that you have to do this by adding exemptions to the end of the GPL, rather than modifying the GPL because, somewhat ironically, the text of the GPL is copyrighted by the FSF and distribution with modifications is not permitted by the license of the license text itself.

              5. 4

                The GNAT version from the FSF (which your distro ships) allows writing software under any license. The GNAT version from AdaCore and the one from FSF are basically the same.

              1. 10

                Interesting that the discussion around this all sort of seems to assume that the maintainers are choosing not to fix bugs and therefore must be bad at running their projects.

                That’s probably true pretty often. But I think it’s also often true that open-source projects have a small, more-or-less-fixed time budget for maintenance that does not expand at the same rate as the growth of the user base, whereas the volume of bug reports does expand along with the user base. Over the years I’ve frequently been surprised to learn that some piece of software I use constantly was the product of one person working solo and it wasn’t even their day job.

                If I publish something I wrote in my spare time to scratch an itch, and it happens to get popular enough to generate more bug reports than I can handle in the few hours a week I’m able to spend on the thing, what are my options, other than quitting my job and working full-time on bug fixing? Do I delete the popular repo that people are finding useful enough to file lots of bug reports about, in the name of reducing the number of unhealthy projects in the world? Do I just let the bug list expand to infinity knowing I’ll never have time to clear the backlog? Do I spend my project time reviewing historical bug reports rather than working on the code?

                In theory, of course, a project that gets bug reports should have lots of people volunteering to fix the bugs. But the quantity and quality of community contributions seems to vary dramatically from one project to the next.

                1. 23

                  It’s not about not fixing bugs. That’s expected, normal, fine. Maintainer does not owe me labour. Closing issues that aren’t fixed is just an FU to the users, though. Let the issue stay until someone fixes it, hopefully someone being affected by it.

                  1. 7

                    Exactly this. I was raised in a world where SFTW or RTFM was an acceptable response, and so put effort into providing good issues. Having a bot drive by and close/lock my still open issue is a slap in the face with a wet fish.

                    1. 1

                      SFTW

                      I’ve never seen this acronym before? What does it mean?

                      1. 1

                        It was meant to be STFW, but I can’t type.

                        1. 2

                          Search The Fucking Web

                          (I had still never seen this one and had to look it up … by searching the web)

                          1. 3

                            Ironically this becomes more difficult as the GitHub bug pages show up first and are locked/closed due to being old.

                  2. 8

                    What’s the downside of letting the open bug list expand?

                    1. 14

                      Eh. Caveats first:

                      1. I’m not a fan of simple auto-closing stale bots (roughly: I think the incentives are misaligned, and ~stale is a not-quite-right proxy of a few real problems, and it is trying to work around some limitations of using the GH issue system for triage)

                      2. but I am a big fan of having a way to punt unactionable issues back to the reporter, and let automation close the issue if the reporter never responds or return it to the triage queue if they do

                      That out of the way: in a busy long-term project, a stack of open bug reports comes with ongoing logistical overhead. GH’s UI/X exacerbates this, but some of it just comes with the territory.

                      • This cost is an aggregate of: maintainer time spent managing the requests, the opportunity-cost of that time, the motivation it saps from the maintainers as they spend more time doing paperwork and less time making electrons dance, and the despair that comes from feeling obliged to play Sisyphus on an ever-growing mountain of reports (or the guilt that comes with having to ignore the reports).
                      • Each person helping with bug triage will be paying a pretty similar cost unless you have tools/processes for de-duplicating logistical effort (unlikely for most volunteer efforts).
                      • This cost is somewhat proportional to the amount of time each triager is able to invest. If you have 100 contributors doing 1 hour of triage a week, there’s going to be a lot more duplicate effort in the reports they read than if you have 2 people doing 50 hours of triage a week.
                      • As the pile grows, efficiently pairing people with reports that are the best use of their time will get both harder and more important to your success.
                      • At scale, the high-leverage maintainers most-likely to have the authority (whether literally the permission, or just the reputational capital) to go around closing reports manually will usually have more critical things to invest their time in. Those the task actually falls to aren’t as likely to be empowered to weed the garden.
                      • Unless you’re willing to declare bug-bankruptcy or use automation (i.e., ideally #2 above–though obviously also a stale-bot), weeding out old dead-weight reports with the same care/consideration you’d usually run triage with can be (ironically?) a massive ongoing timesink in its own right.

                      Reports don’t have to be bad or obsolete to contribute to this cost. 10 or 20 brilliant suggestions may be an asset if you can find time to implement 1 a year for the next 5 years. 1000 brilliant suggestions may be an albatross if the extra logistical overhead fritters away the time and motivation you need to implement one.

                      It doesn’t, of course, apply to all reports equally. It’s better to have a single long-lived “darkmode pls” issue with gobs of comments and reactions than to deal with someone opening a new one every few days.

                      1. 7

                        Similar to not cleaning up your house. Increased cognitive load and ugliness.

                        1. 8

                          This. I am constantly surprised how many people dismiss the paralyzing effect of 2.4k open issues, even if they all make sense in some way. Resources are limited.

                          1. 8

                            Wouldn’t auto-closing issues be like just hiding everything under the bed in this analogy?

                            This all looks like the consequence of bad tooling to me.

                            1. 1

                              Yes. The issue trackers we have need the following features.

                              1. accept issues
                              2. comments on issues
                              3. occasional attachments to issues (need? Maybe not)
                              4. assigning people to issues
                              5. lifecycles as makes sense for your team and organization (open, closed, or on the other extreme: open, ready, in work, in testing, merged, in production, closed)
                              6. tags on issues for classifications
                              7. filtering on reporter, tag, assignee, and status, including default filtering.

                              You do that, you fit 99% of people’s needs.

                              Imagine if github had a tag “maintainers will not address” and the automatic filter (currently set to only show issues where their lifecycle is at “open”) only showed “open & not(tag:will-not-address-but-might-merge)”

                              People would be happier to tune that automatic filter for their own needs, and this would allow a tag to banish the issue from a maintainer’s headspace.

                          2. 5

                            Mostly that there will be a certain number of old bugs that get fixed or rendered irrelevant as a side effect of some other change, and searching through the bug database will become less useful if a sizable portion of the bug reports reflect an obsolete reality.

                            I’ve occasionally run into this on existing projects, in fact: I will search for some error message I’m seeing, find a bug report with some discussion that includes workarounds people used, and only after wasting time trying those workarounds and digging into why they’re not working for me do I discover that the bug they were talking about is in a part of the code that no longer even exists.

                            Of course, in those kinds of cases, issues being auto-closed by a stale bot wouldn’t have helped much; I’d have still found them and probably still tried the obsolete workarounds. I don’t have a perfect answer for this either!

                            1. 8

                              I haven’t heard a good case for a stale-bot. In the article it says “to reduce the number of open tickets”, but it doesn’t explain why that is a good thing – the number could be reduced to zero by not having a bug tracker at all!

                              Perhaps it is a UI thing, and Github should display the number of “active” open tickets, for some activity metric.

                              1. 4

                                I turned a stale bot on for a popular project because I thought it would be a more honest communication of the likely fate of old GitHub reports - “this is stale and we aren’t planning to triage or address it further”. Few others perceived it that way and, after a few months, the bot was removed.

                                The triage work is substantial - we spend easily 120 hours a week on community question/support/defect/feature-request triage. I’d love to do more but of course there are fixed dollars and hours to allocate across many priorities.

                                Anyway, that was my reasoning, which turned out to be incorrect.

                        1. 7

                          TLDR: Kotlin is nice but the fact that you can have a full dev environment only in IntelliJ makes it hard to adopt in the teams.

                          There is an independent Kotlin language server but the repository says right off the bat:

                          This repository needs your help!

                          As for JetBrains position on the topic:

                          We have no plans to support LSP at this time. Instead, we’re focusing on providing the best possible Kotlin development experience in our own IDE, IntelliJ IDEA.

                          While I understand JetBrains’ motivation and I really like where language is going (I actually really want to try their new Compose for Desktop project) there might be some tensions when introducing the language. In fact I have recently removed a few Kotlin classes from our Java project to speed up build times and remove a lot of pain for me (I work in Emacs and use Eclipse’s language server for Java which knows nothing about Kotlin classes).

                          As for the null safety example I guess it is is a common practice to use java.util.Optional nowadays:

                          int subLength = 0;
                          if (obj != null) {
                            if (obj.subObj != null) {
                              subLenth = obj.subObj.length();
                            }
                          }
                          

                          could be written as

                          int subLength = Optional.ofNullable(obj).map(o -> o.subObj)
                              .map(s -> s.length).orElse(0);
                          

                          Obviously, it is not as short as in Kotlin but a huge improvement over if/else nesting.

                          1. 2

                            Interesting – in Java, if I were trying to be concise, I’d probably turn to a ternary operator first, like

                            int subLength = obj != null && obj.subObj != null ? obj.subObj.length() : 0;
                            

                            Still less concise than the Kotlin example but (to me, totally subjective) clearer than the Optional call chain and doesn’t require instantiating a series of intermediate objects.

                            1. 1

                              You can do the null checking in a less concise way in Java, but it becomes a pain for highly nested types from third party libraries. Additionally, having the optional types in Kotlin make it really easy to enforce when something might be null vs will never be with syntax that’s shorter than Java. I know Java has @Nullable and @NotNull, but I never got it to enforce the checks as easily as Kotlin. If you are working a big project and you don’t have a standard way to determine if something is nullable or not, you end up with null checks everywhere for safety, which is a pain. For new developers to Kotlin you can just tell them not to use !! in production code, and then you don’t get NPEs.

                              There are other languages that have nicer optionals, but I think Kotlin does a good job while still working well with existing JVM libraries written in Java.

                          1. 24

                            I am confused about why the Rest crowd is all over grpc ant the likes. I thought the reason why Rest became a thing was that they didn’t really thought RPC protocols were appropriate. Then Google decides to release an binary (no less) RPC protocol and all of the sudden, everyone thinks RPC is what everyone should do. SOAP wasn’t even that long ago. It’s still used out there.

                            Could it be just cargo cult? I’ve yet to see a deployment where the protocol is the bottleneck.

                            1. 14

                              Because a lot of what is called REST wends up as something fairly close to an informal RPC over HTTP in JSON, maybe with an ad-hoc URI call scheme, and with these semantics, actual binary rpc is mostly an improvement.

                              (Also everyone flocks to go for services and discover that performant JSON is a surprisingly poor fit for that language)

                              1. 14

                                I’I imagine that the hypermedia architectural constraints weren’t actually buying them much. For example, not many folks even do things like cacheability well, never mind building generic hypermedia client applications.

                                But a lot of the time the bottleneck is usually around delivering new functionality. RPC style interfaces are cheapter to build, as they’re conceptually closer to “just making a function call” (albeit one that can fail half way though), wheras more hypermedia style interfaces requires a bit more planning. Or at least thinking in a way that I’ve not seen often.

                                1. 10

                                  There has never been much, if anything at all, hypermedia specific about HTTP, It’s just a simple text based stateless protocol on top of TCP. At this day an age, that alone buys anyone more than any binary protocol. I cannot reason as to why anyone would want to use a binary protocol over a human readable (and writeable) text one, except for very rare situations of extreme performance or extreme bandwidth optimisations. Which I don’t think are common to encounter even among tech giants.

                                  Virtually every computing device has a TCP/IP stack these days. $2 microcontrollers have it. Text protocols were a luxury in the days where each kilobyte came with high costs. We are 20-30 years pst that time. Today even in the IoT world HTTP and MQTT are the go to choices for virtually everyone, no one bothers to buy into the hassle of an opaque protocol.

                                  I agree with you, but I think the herd is taking the wrong direction again. My suspicion is that the whole Rest histeria was a success because of being JSON over HTTP which are great easy to grasp and reliable technologies. Not because of the alleged architectural advantages as you well pointed out.

                                  SOAP does provide “just making a function call”, I think the reason why it lost to Restful APIs, was because requests were not easy to assemble without resourcing to advanced tooling. And implementations in new programming languages were demanding. I do think gRPC suffers from these problems too. It’s all fun and games while developers are hyped “because google is doing”, once the hype dies out, I’m picturing this old embarrassing beast no one wants to touch, in the lines of GWT, appengine, etc.

                                  1. 9

                                    I cannot reason as to why anyone would want to use a binary protocol over a human readable (and writeable) text one, except for very rare situations of extreme performance or extreme bandwidth optimisations.

                                    Those are not rare situations, believe me. Binary protocols can be much more efficient, in bandwidth and code complexity. In version 2 of the product I work on we switched from a REST-based protocol to a binary one and greatly increased performance.

                                    As for bandwidth, I still remember a major customer doing their own WireShark analysis of our protocol and asking us to shave off some data from the connection setup phase, because they really, really needed the lowest possible bandwidth.

                                    1. 2

                                      hypermedia specific about HTTP

                                      Sure, but the framing mostly comes from Roy Fielding’s thesis, which compares network architectural styles, and describes one for the web.

                                      But even then, you have the constraints around uniform access, cacheability and a stateless client, all of which are present in HTTP.

                                      just a simple text based stateless protocol

                                      The protocol might have comparatively few elements, but it’s just meant that other folks have had to specify their own semantics on top. For example, header values are (mostly) just byte strings. So for example, in some sense, it’s valid to send Content-Length: 50, 53 in a response to a client. Interpreting that and maintaing synchronisation within the protocol is hardly simple.

                                      herd is taking the wrong direction again

                                      I really don’t think that’s a helpful framing. Folks aren’t paid to ship something that’s elegant, they’re paid to ship things that work, so they’ll not want to fuck about too much. And while it might be crude and and inelegant, chunking JSON over HTTP achived precisely that.

                                      By and large gRPC succeeded because it lets developers ignore a whole swathe of issues around protocol design. And so far, it’s managed to avoid a lot of the ambiguity and interoperability issues that plagued XML based mechanisms.

                                  2. 3

                                    Cargo Cult/Flavour of the Week/Stockholm Syndrome.

                                    A good portion of JS-focussed developers seem to act like cats: they’re easily distracted by a new shiny thing. Look at the tooling. Don’t blink, it’ll change before you’ve finished reading about what’s ‘current’. But they also act like lemmings: once the new shiny thing is there, they all want to follow the new shiny thing.

                                    And then there’s the ‘tech’ worker generic “well if it works for google…” approach that has introduced so many unnecessary bullshit complications into mainstream use, and let slide so many egregious actions by said company. It’s basically Stockholm syndrome. Google’s influence is actively bad for the open web and makes development practices more complicated, but (a lot of) developers lap it up like the aforementioned Lemming Cats chasing a saucer of milk that’s thrown off a cliff.

                                    1. 2

                                      Partly for sure. It’s true for everything coming out of Google. Of course this also leads to a large userbase and ecosystem.

                                      However I personally dislike Rest. I do not think it’s a good interface and prefer functions and actions over (even if sometimes very well) forcing that into modifying a model or resource. But it also really depends on the use case. There certainly is standard CRUD stuff where it’s the perfect design and it’s the most frequent use case!

                                      However I was really unhappy when SOAP essentially killed RPC style Interfaces because it brought problems that are not inherent in RPC interfaces.

                                      I really liked JSON RPC as a minimal approach. Sadly this didn’t really pick up (only way later inside Bitcoin, etc.). This lead to lots of ecosystems and designs being built around REST.

                                      Something that has also been very noticeable with REST being the de-facto standard way of doing APIs is that oftentimes it’s not really followed. Many, I would say most REST-APIs do have very RPC-style parts. There’s also a lot of mixing up HTTP+JSON with REST and RPC with protobufs (or at least some binary format). Sometimes those “mixed” pattern HTTP-Interfaces also have very good reasons to be like they are. Sometimes “late” feature additions simply don’t fit in the well-designed REST-API and one would have to break a lot of rules anyways, leading to the questions of whether the last bits that would be worth preserving for their cost. But that’s a very specific situation, that typically would only arise years into the project, often triggered by the business side of things.

                                      I was happy about gRPC because it made people give it another shot. At the same time I am pretty unhappy about it being unusable for applications where web interfaces need to interact. Yes, there is “gateways” and “proxies” and while probably well designed in one way or another they come at a huge price essentially turning them into a big hack, which is also a reason why there’s so many grpc-alikes now. None as far as I know has a big ecosystem. Maybe thrift. And there’s many approaches not mentioned in the article, like webrpc.

                                      Anyways, while I don’t think RPC (and certainly gRPC) is the answer to everything I also don’t think restful services are, nor graphql.

                                      I really would have liked to see what json-rpc would have turned to if it got more traction, because I can imagine it during for many applications that now use REST. But this is more a curiosity on an alternative reality.

                                      So I think like all Google Project (Go, Tensorflow, Kubernetes, early Angular, Flutter, …) there is a huge cargo cult mentality around gRPC. I do however think that there’s quite a lot of people that would have loved to do it themselves, if that could guarantee that it would not be a single person or company using it.

                                      I also think the cargo cult is partly the reason for contenders not picking up. In cases where I use RPC over REST I certainly default to gRPC simply because there’s an ecosystem. I think a competitor would have a chance though if it would manage a way simpler implementation which most do.

                                      1. 1

                                        I can’t agree more with that comment! I think the RPC approach is fine most of the time. Unfortunately, SOAP, gRPC and GraphQL are too complex. I’d really like to see something like JSON-RPC, with a schema to define schemas (like the Protobuf or GraphQL IDL), used in more places.

                                        1. 2

                                          Working in a place that uses gRPC quite heavily, the primary advantage of passing protobufs instead of just json is that you can encode type information in the request/response. Granted you’re working with an extremely limited type system derived from golang’s also extremely limited type system, but it’s WONDERFUL to be able to express to your callers that such-and-such field is a User comprised of a string, a uint32, and so forth rather than having to write application code to validate every field in every endpoint. I would never trade that in for regular JSON again.

                                          1. 1

                                            Strong typing is definitely nice, but I don’t see how that’s unique to gRPC. Swagger/OpenAPI, JSON Schema, and so forth can express “this field is a User with a string and a uint32” kinds of structures in regular JSON documents, and can drive validators to enforce those rules.

                                    1. 6

                                      Very interesting but I remain unconvinced. It seems as though once you start adding the required features for productionization the models converge.

                                      The obvious way to improve throughout in a pull based model is to the have producer prefetch a buffer. This mimics the buffer that would exist on the DAG consumers in the push-based model. (In fact it may be better as it is naturally shared amount consumers.) In many real-world systems you will need to add back pressure to the producer in a push-based model, leading to it being basically equivalent to the pull-based model with buffering. (again, except the buffers are in the consumers).

                                      The author does raise an interesting point though, PostgreSQL and CockroachDB materialize the entire table when using with classes. In the past I have heard is that this is treated as some sort of optimization hint in PostgreSQL so I wonder if there is something fundamental about the model that is holding it back, or if it is a “feature” that it works this way.

                                      1. 3

                                        PostgreSQL used to materialize WITH clauses unconditionally, and it was often used as a mechanism for hand-optimizing queries. But as of PostgreSQL 12, by default it only materializes them if they are used more than once, are recursive, or have side effects. Otherwise it folds them into the main query for purposes of generating a query plan. If you still want to use them for manual optimization, you can explicitly say whether to materialize them.

                                        1. 2

                                          +1 Insightful. This is approximately my take too. I’ve written query engines and I opted for something that’s kind of a mix: Next() and Check() being the respective primitive operations to pull and push. The buffer you mention is a materialize operator that’s a logical noop to the plan but something the optimizer can add where it sees fit.

                                          That said, there may be something to leaning more push than I have been, specifically for DAGs, which require hacks in a pull model. Furthermore, in a distributed setting, push works a better because there’s much less back-and-forth. Tradeoff there is throwing a ton of data over the network. Still needs a hybrid.

                                          If you find it interesting, let me know what specifically, maybe I’ll put it on my blogging stack.

                                        1. 16

                                          I think its speed is one of the thing which makes apk (and therefore alpine) so well suited to containers.

                                          It used to be that the slowness of apt wasn’t a huge issue. You would potentially have to let apt spin in the background for a few minutes while upgrading your system, and, once in a blue moon when you need a new package right now, the longer-than-necessary wait isn’t a huge issue. But these days, people spin up new containers left right and center. As a frequent user of Ubuntu-based containers, I feel that apt’s single-threaded, phase-based design is frequently a large time cost. It’s also one of the things which makes CI builds excruciatingly slow.

                                          1. 4

                                            distri really can’t happen fast enough… the current state of package management really feels stuck in time.

                                            1. 1

                                              I feel like speed could be a non issue if the repository state was “reified” somehow. Then you could cache installation as a function, like

                                              f(image state, repo state, installation_query) -> new_image_state
                                              

                                              This seems obvious but doesn’t seem like the state of the art. (I know Nix and guix do better here, but I also need Python/JS/R packages, etc.)

                                              The number of times packages are installed in a container build seems bizarre to me. And it’s not even that; right now every time I push to a CI on Travis and sourcehut it installs packages. It seems very inefficient and obviously redundant. I guess all the CI services run a package cache for apt and so forth, but I don’t think that is a great solution. I use some less common package managers like CRAN, etc.

                                              1. 2

                                                Part of it is no doubt that hosted CI platforms don’t do a great job of keeping a consistent container build cache around. You usually have to manually manage saving and restoring the cache to some kind of hosted artifact repository, and copying it around can add up to a nontrivial chunk of your build time.

                                                At my previous job, that was a big part of our motivation for switching to self-hosted build servers: with no extra fussing, the build servers’ local Docker build caches would quickly get populated with all the infrequently-changing layers of our various container builds.

                                              2. 1

                                                This sounds reasonable, until you realise it means that containers are constantly being rebuilt rather than just persisted and loaded when needed.

                                                1. 3

                                                  Yeah, but they are. Look at any popular CI - TravisCI, CircleCi, builds.sr.ht, probably many, many others. They all expect you to specify some base image (usually Debian, Ubuntu or Alpine), a set of packages you need installed on top of the base, and some commands to run once the packages are installed. Here’s an example of the kind of thing which happens for every commit to Sway: https://builds.sr.ht/~emersion/job/496138 - spin up an Alpine image, install 164 packages, then finally start doing useful work.

                                                  I’m not saying it’s good, but it’s the way people are doing it, and it means that slow package managers slow things down unreasonably.

                                                  1. 2

                                                    If you’re rebuilding your OS every time you want to test or compile your application, it’s not the package manager making it slow, no matter what said package manager does.

                                                  2. 1

                                                    Persistence can be your enemy in testing environments.

                                                    1. 2

                                                      Sure re-deploy your app, but rebuild the OS? I understand everybody does it all the time (I work in the CI/CD space), but that doesn’t mean it’s a good idea.

                                                1. 3

                                                  I’ve never been a fan of UML just because I’m not a visual thinker, but I definitely agree that formal specifications have fallen out of favor.

                                                  My charitable take is that a big part of it is because our tools are getting better, such that the cost of being somewhat wrong is often considerably lower than the cost of trying to be precisely correct. When iterating on a piece of software meant a new months- or years-long development cycle, it really really paid to make sure you got it right the first time. But now? “Sure, we should be able to change how that works in time for the weekly production release.”

                                                  My less-charitable take is that it’s because in enterprise and consumer software, we’ve sort of decided that specs should be written by product managers who often have little or no training or background in rigorously specifying a system and are instead focused almost exclusively on high-level “user stories” that don’t require thinking in fine detail, or on UI mockups that don’t require writing down precise business rules.

                                                  That said, though, there are still contexts where you see people take up-front design discussion pretty seriously. Of course there are obvious candidates like aerospace, but also, for example, the processes some programming languages have in place to manage their evolution (JEP, PEP, etc.) involve a lot of design discussion before anyone is willing to review any code.

                                                  1. 1

                                                    specs should be written by product managers who often have little or no training or background in rigorously specifying a system

                                                    When you say “project manager”, I expect someone whose job duties and training are in:

                                                    • keeping track of dependencies between tasks so people aren’t asked to do things for which the prerequisites aren’t available yet
                                                    • keeping up to date estimates of cost and timeline
                                                    • getting people to work on the tasks that are on the critical path to shipping
                                                    • being a personable business-savvy “face” for the project

                                                    None of those are requirements gathering?

                                                    To be fair, you need PMs to be in the requirements gathering anyway because reqs impact schedule and cost, so leaving them out is counter productive.

                                                    1. 3

                                                      specs should be written by product managers who often have little or no training or background in rigorously specifying a system

                                                      When you say “project manager”, I expect someone

                                                      Mismatch. They said “product manager”, not “project manager”. A product manager should be the major source of requirements.

                                                      1. 1

                                                        Ah! Thanks for pointing this out. I did indeed misread.

                                                  1. 3

                                                    This is a well-thought-out list. I’ve seen nearly all of these at one point or another in my career, and the situations have often played out the way the paper predicts.

                                                    I’m a bit disappointed that they take it as an absolute given, not even worth mentioning, that everyone is using Scrum or something like it; while many of the high-level patterns have nothing to do with a specific agile methodology, there are a lot of “look for thing X at point Y in your sprints to see if this is happening” guidelines that make no sense for, say, a team that’s using Kanban where there’s no concept of a “sprint.”

                                                    The other assumption I’d call out, though I think this is just an artifact of the document being written for the kind of audience that would buy Pluralsight’s products, is that the team is large and is part of a mature organization with clearly-defined roles. A number of the patterns have pretty different implications if your engineering organization is very small, if the developers are also the product owners, or if you’re exploring a problem space rather than working to implement a well-understood solution to a well-understood problem before a well-understood deadline.

                                                    But those are minor complaints; generally this is a nice piece of work.

                                                    1. 6

                                                      In my personal experience I’ve seen a fair number of very talented, but young, developers combine hoarding and domain championing as a way to establish themselves and, I think, stave off boredom and burnout. In these cases, I suspect that the answer is to make sure these individuals are challenged and push them to work more closely with their (perhaps less skillful or less motivated) teammates.

                                                      1. 6

                                                        I think you’d need to be pretty careful about judging the personalities involved. Making sure they’re challenged is almost certainly the right call, but if someone is already altering their work style to stave off boredom and burnout, pushing them to work more closely with less-skillful or less-motivated teammates may cause them to become even more bored and burned out rather than becoming energized by the opportunity to mentor.

                                                        Obviously domain championing and hoarding are not healthy for a team, though.

                                                      1. 49

                                                        As the saying goes:

                                                        The first rule of distributing systems is don’t, if you can avoid it.

                                                        What’s been amazing to me has been the amount of time and effort spent on this technique outside of the small fraction of places it makes sense, all while it is trivially obvious for anybody who stops and thinks for teo minutes what the problems are gonna be.

                                                        The conference-industrial complex sold snake oil and everybody just lapped it up. Such is the quality of “engineering” in software.

                                                        1. 28

                                                          And worse, the snake oil often claims to cure a disease most people don’t have in the first place: the need to scale up dramatically on extremely short notice.

                                                          1. 31

                                                            I read a joke somewhere (I cannot remember where): “kubernetes. An ancient Greek word for more containers than customers.”

                                                            1. 4

                                                              I believe @srbaker coined that phrase.

                                                              1. 3

                                                                Corey from Screaming in the Cloud (https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/) has a variant of this (if I’m remembering well):

                                                                Kubernetes, the Greek god of spending money in the cloud

                                                                1. 2

                                                                  Boss wants to market horizontally scaling when vertical would be 10x cheaper :)

                                                            1. 27

                                                              I took a job once where there was a performance bonus that went up to something like 30% of my base salary (tens of thousands of dollars in addition to the money I was making there).

                                                              The actual performance rubric (had I paid closer attention to it) for upper levels required basically being a luminary in the field–and the sort of work we were doing, at the size of team and problem domain we had, frankly would never provide the opportunity. So, right out of the gate the game was rigged and I was too blinded by my own greed to recognize it.

                                                              Next, the bonus was judged by a rubric where we did a self-review, and then our manger (in my case, the CTO) did their own review, and then we would truthseek to some reasonable amount. Now, if you have half a brain you see the obvious strategy on my side is to give myself full marks regardless of how I actually felt about it–because they’re never going to give you more than you give yourself. So, it’s just this toxic thing.

                                                              Wouldn’t you know it, when my first year and bonus came around, the CTO with great sadness–apparently, he had warned the young CEO specifically against this sort of nonsense–explained that I only qualified for something like 8K of the large bonus.

                                                              This was compounded by:

                                                              • Some deal where I could take the bonus in cash, or if I took it in options I could get twice as much! Being a doofus, I figured I’d go for the company scrip.
                                                              • Upon receiving my company scrip, I discovered that their calculation of options and my calculation differed–you can either divide the scrip value by the option strike price (so, 1K of scrip for 1USD strike means 1000 options) or by the stock price (so, 1K of scrip converted into 2USD/share gets 500 options). Obviously, I liked the former calculation because it meant more options/shares–the CEO, being a shrewd businessperson preferred the latter. I was basically given the option to take their calculation (which was a haircut of something like 1/2-2/3 of the value i could’ve gotten) or pound sand. With frustration, I took their offer.
                                                              • Said CEO took ages and ages to finalize my purchase, which was just annoying as hell. The CFO finally stepped in and did the needful, and during this conversation I discovered that the company valuation had actually dropped the share price to something like the former strike price.

                                                              The second year I was there–again, in spite of shipping major features and rewrites and sturm und drang…and being the only surviving engineer of the senior cadre that had been hired into that arrangement (oh yes, everyone hired after me had no bonus structure)–I was informed by the CTO I did not qualify for any bonus whatsoever. I was sacked a month or so later, one day right after an all-hands meeting praising my work.

                                                              Anyways, point is: if the bonus looks too good to be true, it probably is. There’s always free cheese in a mousetrap.

                                                              1. 11

                                                                Next, the bonus was judged by a rubric where we did a self-review, and then our manger (in my case, the CTO) did their own review, and then we would truthseek to some reasonable amount. Now, if you have half a brain you see the obvious strategy on my side is to give myself full marks regardless of how I actually felt about it–because they’re never going to give you more than you give yourself. So, it’s just this toxic thing.

                                                                These self-review things are incredibly frustrating :-D.

                                                                A while back, I ended up in a big $megacorp project that was pretty much in its dying phase (the team size and overly-optimistic predictions didn’t quite give it away but the day-to-day activity did). And, this being $megacorp and all, of course it had a pretty bureaucratic performance review process, which included a self-review phase.

                                                                This self-review phase, being meant for “mere employees”, not managers, didn’t really leave much margin for actually useful reviewing. There were 10-15 things (teamwork, domain expertise, whatever) that you had to rate yourself for, with ratings between “needs improvement” and “excels at”. In order to save face about this being useful somehow, you were also expected to come up with some sort of yearly development plan, so that you could improve the things that “need improvement” and further build upon things that you excelled at and so on.

                                                                Trouble was, the team development strategy in that department could essentially be summarised as “nope”. We didn’t even have access to hardware reference manuals for some of the ASICs in our hardware because the company didn’t want to pay for the required support level. The team was also awkwardly understaffed. So anything that involved the company paying for anything, or getting even basic guarantees about what you would work on for more than 2-3 weeks in advance so that you could learn something on the job, was just not going to happen.

                                                                So if you did the self-review in good faith, you quickly sabotaged your chance at a bonus. If you genuinely thought you sucked at X and had to improve, every avenue for you to do that was closed.

                                                                So those of us who ended up getting bonuses did the only reasonable thing: we filed it backwards.

                                                                We said we “need improvement” at the things that we knew we had to do anyway. For example, I knew I was going to hold most of the interviews for our team because I was the friendliest person on the team, and I’d worked in some other fields, too, so it was easier for me to strike up a conversation. So despite being pretty good at this, I made sure my review form said I sucked at it and therefore had to either attend trainings (that never happened – company wouldn’t pay for it) or just do more of it and learn on the job. When the yearly evaluation come, I could always say I worked super hard to improve myself, and had an year’s worth of new colleagues to back me up and say the interviews were pretty good.

                                                                And, of course, we said we were great at the things we knew we had no way of improving. My review form always said I was a domain expert (and my direct manager agreed wholeheartedly!) even though I absolutely sucked at what we actually did. If I’d actually admitted to sucking at it on the form, my self-development plan would’ve been a dead end: training was out of the question, projects were always behind so assigning the guy without domain expertise to do the hard things was also out of the question, and everyone was not only busy putting out fires but also missing hardware docs for newer hardware generations – just like I did – so mentoring wasn’t gonna happen, either. Then – no matter how well I’d done under the circumstances – someone would be sure to point out that I’d done nothing to improve since our last evaluation. But since I was already at my best, there was no need to improve things, which just so happened to be very well aligned with the company’s goals, because the higher-ups hadn’t approved anything related to professional development in almost a decade.

                                                                So basically I got a bonus for keeping quiet about how much my skills were rotting.

                                                                1. 4

                                                                  It’s not just a big-company thing, either. I’ve seen the exact same problem over the years at multiple small companies, and as someone who tends to try to write self-reviews in good faith, it is quite demoralizing to look back and realize that at no point did the company do anything about a single one of the things I said I wanted to improve. I come away thinking, why are you wasting my time on this useless crap? Just don’t ask if the answer doesn’t matter.

                                                                  But it’s also a “careful what you wish for” situation. In some ways, the company ignoring my “needs improvement” areas is the desirable outcome, because the other outcome that I’ve had happen to me a couple times is that I say I want to improve X and the company signs me up for a mandatory training course on X that they chose with no input from me and that turns out to be of no actual use.

                                                                  1. 4

                                                                    Oh, of course – training is just the thing that most easily gets thrown around, but in that particular case, it was also the only way to acquire some, uh, particular knowledge (tl;dr some big companies, including the one who developed some of the stuff we were using, does not really publish documentation – it gives the people who attend their trainings access to said documentation, and the documentation is good enough to generally write code by, but not that good that you don’t need a beefy support package).

                                                                    Except for such super-justified cases I really try to avoid trainings and workshops. I’ve been to dozens and I can think of only one or two that were actually useful – and even those were, IMHO, terribly inefficient.

                                                                    The option I usually try to sell first is “buy me these two books and give me two weeks’ leave”. I’ve found that to be immensely more useful than any training I’ve ever attended, and it’s usually cheaper, too, so it’s not that tough to sell. On the other hand, it doesn’t work for everything. E.g. one of the two useful trainings I attended was Wayland-related, nearly five years ago, and that’s because there was (and still isn’t) any useful documentation there, it’s all word of mouth.

                                                              1. 25

                                                                So basically we finally arrived at the “make your app a web page” that Apple demanded when launching the iPhone

                                                                1. 30

                                                                  Yes, and the latest trend in web development is to render content on the server. Everything old is new again!

                                                                  1. 7

                                                                    I think it’s better this time, because phones and network are fast enough that doing everything in the browser isn’t limited by UMTS speeds.

                                                                    1. 3

                                                                      The original iPhone didn’t even support UMTS (3G), it was GPRS (2G) EDGE (2.5G). A load of mobile providers who had already rolled out large UMTS had to go and deploy older hardware to support the iPhone without it falling back to GPRS. The latency on GPRS was awful (500ms RTTs were common, making it unusable for anything interactive).

                                                                    2. 2

                                                                      I have noticed this and had the very same reaction a few weeks ago.

                                                                    3. 13

                                                                      To be fair: when Apple announced this, React did not exist, Vue did not exist, precursors like Backbone didn’t even exist, and most critically, most of the technologies and tools we use in 2021 to do SPAs, let alone offline webapps, did not exist. Hell, I think the dominant offline storage solution was WebSQL, which was never standardized and is not (AFAIK) supported in any contemporary browser, and no equivalent of web workers existed unless you had Google Gears installed. You also had nothing like WebGL, or web sockets, or even widespread contemporary CSS that would make reasonable, cross-platform styling feasible. So what Apple was offering at the time was morally equivalent to having bookmark links on the home screen.

                                                                      (Yeah, I’m very aware of the pile of old meta tags you could use to make the experience be better than that in a literal sense, but that doesn’t resolve anything else I highlighted.)

                                                                      Speaking purely for myself, I found the initial announcement infuriating, not because I didn’t believe in the web (I did! Firefox was growing! Safari was proving the viability of KHTML! IE was on the decline!), but because Apple’s proposal was just so damn far from what doing that seriously would’ve actually looked like that it felt condescending. The Palm Pre, which notably came out two years later, was dramatically closer to what I’d have expected if Apple were being sincere in their offer. (And even there, webOS, much as I love it, is more an OS that happens to have JavaScript-powered apps than a genuine web app platform in the 2021 sense.)

                                                                      1. 5

                                                                        Even at the time, Apple’s stance felt to me like, “We aren’t finished with our native SDK yet and it’s so far from ready for public consumption that we’re going to just pretend it doesn’t exist at all.” I remember talking about the iPhone with my coworkers when it first came out and everyone just assumed native apps would be coming at some point.

                                                                        Even webOS (which I also loved) ended up supporting native apps eventually, despite having a much more feature-rich platform for JavaScript code.

                                                                        Games seem to be the killer app category that pushes mobile OS vendors to support native code. They’re one of the few categories of application where a lack of native code support can make an app impossible to implement, rather than just making it a bit slower or clunkier but still basically workable.

                                                                      2. 3

                                                                        Even Firefox OS was too early in the game for that (besides other problems of FFOS).

                                                                        1. 6

                                                                          If it was timed right, Mozilla would have found another way to run it into the ground. ;-)

                                                                          1. 1

                                                                            Could not agree more !

                                                                      1. 2

                                                                        This feels about right to me, but I wonder to what extent the numbers of searches of each type are skewed by the huge variance in difficulty of getting good results for different queries.

                                                                        For example, when I search for API documentation, the official docs are very often what I’m looking for and they’re very often in the top 2-3 results if I search for “<language/library name> <function name>”. One search and I’m done.

                                                                        But for troubleshooting? Oh boy. Paste the error message into the search bar. Click on all the links on the first page to find they are all unrelated problems that happened to cause a similar error message. Add some keywords to try to narrow it down. Repeat half a dozen times until I either hit the magic combination of words or conclude that nobody else has ever posted a question or a bug report about whatever is causing the problem for me.

                                                                        Looking at the raw numbers, you’d conclude that I search for error message help far more than I search for API docs, but in reality it’s just that one of those is much more efficient to search for than the other.

                                                                        Same with “how to do X” questions: when I’m asking that about some technology that I’m already somewhat proficient with, it often takes me several attempts to get past the beginner questions that happen to have a lot of keyword overlap with what I’m trying to do. (Of course, when I’m searching about technology I’m not already familiar with, the beginner questions are sometimes just what I want.)

                                                                        1. 1

                                                                          That’s a very valid point! To properly take into account how many searches it takes you to find what you’re looking for, we should have classified whole search sessions instead of individual searches.

                                                                          In this case, we did cluster the searches into sessions to look at query refinement strategies, but we couldn’t make the assumption that the category of the session was the same as the category of an individual query in that session.

                                                                          That would have been interesting to look into! Our dataset was too small for any real statistical analysis, though… I did report the numbers, but the meatier discussion was around the qualitative stuff.

                                                                        1. 9

                                                                          This resonates with me, though as others have noted it’s missing one critical detail. Willingness to go shovel the crap until it’s all gone is one of the huge differentiators between the merely-smart developers and the ones that get “most productive member of the team” and “stop it, you’re making the rest of us look useless” feedback over and over again.

                                                                          The missing detail is that you need to be able to pick or make the right shovel so your efforts are effective. To strain the analogy further: grit and determination and a shovel will get the job done better than grit and determination and a teaspoon. Accurately identifying cases where spending time building a tool and using that tool will get you through the immediate set of tedious work faster than doing the work without building the tool is more of an art than a science, but it’s a skill worth cultivating. “Always build a tool to automate every repetitive procedure so you save effort over the long haul” is often not the right answer, but it’s an easy trap to fall into.

                                                                          1. 13

                                                                            Author here. Ask me anything!

                                                                            1. 9

                                                                              just wanted to say congratulations on what you’ve got so far; it was interesting for sure, and I’m looking forward to future parts.

                                                                              it makes me wonder if you’re basically modeling parallel dataflow; linear types probably means each stack can be executed pretty safely in parallel, yes? the moving values into a new stack feels very “spawn a thread” to me.

                                                                              1. 5

                                                                                Thanks!

                                                                                Yes, that’s absolutely correct. Operations on separate stacks could trivially be parallelized by a properly designed runtime. Eventually, high-level Dawn code could be compiled to OpenCL or CUDA, similar to in Futhark. Alternatively, a compiler to assembly for a superscalar CPU could interleave operations on the different stacks to take advantage of instruction-level parallelism.

                                                                              2. 6

                                                                                This looks very interesting! I’ll eagerly await the next installment and look forward to playing with the language.

                                                                                Reading the post, I wondered how the stack names are scoped. In the example code, one can reasonably (I think?) guess that $x, $y, and $z are scoped to the function whose {spread} operation created them, or maybe they are bound to the current stack. But looking at the comment you linked to in an earlier discussion about I/O, it seemed like there were references to named stacks like $stdout from some external scope.

                                                                                Perhaps I’m wrong to assume that “scope” is even a relevant concept. But presumably there has to be some way to avoid stack name collisions.

                                                                                (Edit) …Huh, maybe I am wrong to assume that scope matters. I am still trying to wrap my brain around it, but maybe if the language can guarantee linearity across the whole application, you don’t even have to care about name collisions because if there is already a $x stack from some calling function, it doesn’t matter if you reuse it and push values onto it as long as you’re guaranteed to have removed them all by the time your caller looks at it again (assuming your caller wasn’t expecting you to leave behind a value on that stack, but then “leaves behind one value on $x” would be part of your function signature).

                                                                                I’m not quite sure that actually works, but now I’m even more eager to read the next post.

                                                                                1. 4

                                                                                  But presumably there has to be some way to avoid stack name collisions.

                                                                                  Yes, there is. Take for example the expression: {$a swap} where swap is defined as

                                                                                  {fn swap => $a<- $b<- $a-> $b->}
                                                                                  

                                                                                  Expanded, this becomes

                                                                                  {$a $a<- $b<- $a-> $b->}
                                                                                  

                                                                                  Or, equivalently,

                                                                                  {$a {$a push} {$b push} {$a pop} {$b pop}}
                                                                                  

                                                                                  Without some mechanism to handle the stack collision between the inner and outer $a stack, this would not behave properly. Since we want functions to behave the same way regardless of what stack context they are executed from, that would be unacceptable. So this is handled by checking for nested stack name collisions and renaming the inner stack. So the latter would effectively be rewritten to

                                                                                  {$a {$$a push} {$b push} {$$a pop} {$b pop}}
                                                                                  

                                                                                  In the existing type checker prototype, renamed stacks are distinguished by a prefix of more than one $. Then, if one of these temporary stack names escapes up to the inferred type for a user-defined function, an error is raised. This ensures expected behavior while ensuring we don’t need to monomorphize each function to operate on different stacks.

                                                                                2. 4

                                                                                  Your introduction to Dawn was clear and compelling, really excited to follow along with Dawn’s development.

                                                                                  {spread $x $y $z} is syntactic sugar

                                                                                  Do you intend to expose a macro system to Dawn programmers?

                                                                                  Conceptually, these named stacks can be considered to be different parts of one big multi-stack.

                                                                                  How are Dawn’s named stacks represented internally? I don’t have much familiarity with stack-based languages, but it seems like it would be straightforward to understand how the machine runs a program just by reading the source. Is that lost with the introduction of named stacks, or is there a mental mapping that can be made?

                                                                                  Dawn is really exciting!

                                                                                  1. 2

                                                                                    Thanks!

                                                                                    I’m undecided on syntactic macros, but the plan is absolutely to provide meta-programming. I haven’t prototyped this at all yet, but I hope and expect for compiler passes to be implemented in something I’ve been calling meta-Dawn in my notes—a form of staged compilation. The plan is for Dawn to be its own intermediate representation. We’ll see how that works out in practice, of course.

                                                                                    1. 2

                                                                                      And to answer your other questions, in the existing interpreter the named stacks are a Map String (Stack Val). The first compiler, which will be through translation to C, will use static analysis (basically the inferred types) to turn all stack slots, regardless of which stack they are on, into function parameters.

                                                                                      Eventually, I plan to write an assembler in/for Dawn, in which stack names will correspond to architecture-specific registers.

                                                                                    2. 2

                                                                                      I’m looking forward to the rest of the series. Is there any change you could put an rss/atom feed on the site? It’s a much nicer experience than subscribing to a mailing list.

                                                                                        1. 1

                                                                                          Thanks!

                                                                                        2. 2

                                                                                          Thanks for the suggestion. I’ll take a look at what that entails.

                                                                                        3. 1

                                                                                          First of all, it’s a really interesting concept! One question about the linear typing aspect though: What happens if multiple functions want to access the same read-only data structure? I assume that for small values clone just copies by value, but what if the data structure is prohibitively expensive to copy? Are some values cloned by reference? If yes, don’t you need to track how many readers / writers access the data so that you know when to free the memory?

                                                                                          I guess another way of phrasing this question would be: How do you handle the more complex cases of borrowing data with the semantics described in the post? Rust’s borrow checker is of course useful even for simple cases of stack-allocated values, but it really shines (and this is where its complexity lies) in cases of more complex heap-allocated data structures. (And of course, even Rust with its borrow semantics needs Rc/RefCell as escape hatches for situations where these guarantees cannot be statically checked, is there something comparable in Dawn?)

                                                                                          1. 1

                                                                                            Great question! I’m going to get to that in a future post, but there is a solution, and it doesn’t require a separate borrow checker.

                                                                                          2. 1

                                                                                            Nice! Also looking forward to future parts.

                                                                                            a way to bind values to local named variables. Unfortunately, this breaks one of the primary advantages of concatenative notation: that functions can be trivially split and recombined at any syntactic boundary—i.e. their factorability.

                                                                                            You gave an example of how Dawn still has this but can you give an example of why Factor or Kitten do not? Or more generally, why a concatenative with bind values to local named variables cannot.

                                                                                            Here’s a example of Fibonnaci from Flpc (Disclaimer: I’m the author).

                                                                                                [ 1 return2 ] bind: base-case
                                                                                                [ newfunc1 assign: i
                                                                                                  pick: i pushi: 3 < pushf: base-case if
                                                                                                  pick: i 1 - fib pick: i 2 - fib + return1
                                                                                                  ] bind: fib
                                                                                            

                                                                                            Any part of the body can be split off. For example

                                                                                                [ 1 return2 ] bind: base-case
                                                                                                [ pick: i 1 - ] bind: one-less
                                                                                                [ pick: i 2 - ] bind: two-less
                                                                                                [ newfunc1 assign: i
                                                                                                  pick: i pushi: 3 < pushf: base-case if
                                                                                                  one-less fib two-less fib + return1
                                                                                                  ] bind: fib
                                                                                            
                                                                                                [ 1 return2 ] bind: base-case
                                                                                                [ pick: i s21 - ] bind: recurse-fib
                                                                                                [ newfunc1 assign: i
                                                                                                  pick: i pushi: 3 < pushf: base-case if
                                                                                                  1 recurse-fib 2 recurse-fib + return1
                                                                                                  ] bind: fib
                                                                                            

                                                                                            For your example, this would be

                                                                                            [ newfunc3 assign: z assign: y assign: x
                                                                                              pick: y square pick: x square + pick: y abs - return1 ] bind: f
                                                                                            

                                                                                            (z needs not be dropped explicitly since return1 takes care of that.)

                                                                                            1. 1

                                                                                              In your example of splitting up fib, what happens if you reuse one-less in another function? Does the source essentially get inlined, so that pick: i refers to the assign: i? If so, then this appears to be similar to Dawn, but without the linearity restriction.

                                                                                              In Factor and Kitten, I don’t believe local variables work like that, though. I believe they behave more like they do in most existing languages, e.g. Python, where the pick: i in one-less would be an undefined variable error.

                                                                                              1. 2

                                                                                                In your example of splitting up fib, what happens if you reuse one-less in another function? Does the source essentially get inlined, so that pick: i refers to the assign: i?

                                                                                                Yes, that’s exactly right.

                                                                                                In Factor and Kitten, I don’t believe local variables work like that, though. I believe they behave more like they do in most existing languages, e.g. Python, where the pick: i in one-less would be an undefined variable error.

                                                                                                Oh I see. I’m wondering if it’s because of scoping. With what I’m doing and maybe what you’re doing, you can’t (easily) get lexical scoping whereas the other way you could.

                                                                                          1. 23

                                                                                            I’ve come up with a simple analogy to help explain to non-programmers, that I think really gets to the heart of the issue: cooking.

                                                                                            Is it hard to make a meal? Well starting out, it’s easy to learn how to boil an egg, cook some vegetables, or even make a basic stir fry and rice. But how long would it take you to practice and learn so that you could serve a quality 5 course meal to 8 people at once in a limited amount of time?

                                                                                            This analogy is of course terrible and an apples-to-oranges comparison, but pretty much everyone has some experience with cooking - so it works particularly well at conveying the effort involved. They both require significant time, dedication, and practice.

                                                                                            It’s “easy” to learn how to write a small program that you yourself use. But writing workable software for others to pick up and enjoy? That’s hard.

                                                                                            1. 7

                                                                                              I think this analogy hits a lot of important points. Like cooking, programming is more difficult when there’s limited time, when you have to combine multiple seemingly-easy tasks, and although the output (taste) should be all that matters, presentation is also important.

                                                                                              1. 3

                                                                                                And also, doing it well when it’s just you takes a very different set of skills and disciplines and tools than doing it well as part of a team, but the beginner lessons only ever cover the solo stuff.

                                                                                              2. 5

                                                                                                This is a great analogy. Some programmers are content to be perceived as wizards of a special caste of people who were born to code. The cooking analogy pulls back the curtain. As Gusteau said in Ratatouille, “Anyone can cook.” The question is, do you enjoy it?

                                                                                                1. 5

                                                                                                  I like the analogy too, but I have a very different read.

                                                                                                  A lot of people want clear instruction, approachable and easy to operate stations, and a large number of people just like them they can ask for help, but don’t realise that means they’re working at McDonalds.

                                                                                                  Real gourmet isn’t going to have any of those things; Michelin doesn’t rate restaurants on how “readable” their recipe cards are, but on the standards of the experience patrons have enjoying the restaurant.

                                                                                                  So why do most arguments about programming ignore the final product? Can people agree coding is like cooking but still think python and java make a tasty meal? Doesn’t anyone else feel bloated trying to eat a fifty megabyte hello world, and think maybe there’s something wrong in the kitchen?

                                                                                                  1. 2

                                                                                                    This analogy is a goldmine!

                                                                                                    So why do most arguments about programming ignore the final product? Can people agree coding is like cooking but still think python and java make a tasty meal?

                                                                                                    This depends a ton on what you think “the final product” is. Is it the flavor or is it the calorie count or is it the rare ingredients? Are you critiquing how the sausage was made, or how the sausage feels in your mouth? Any of those answers could be correct depending on who you’re asking.

                                                                                                    A skilled chef can come into your home and make an amazing meal out of the cheap leftover ingredients you have in your fridge and pantry. (And you can watch it on TV!) A novice chef will make a mediocre meal at best if you put them in a fully-stocked three-Michelin-star restaurant kitchen and tell them to go nuts.

                                                                                                    If your tool is PHP, maybe you write some hobby web site full of SQL injection vulnerabilities. Or maybe you write Wikipedia.

                                                                                              1. 10

                                                                                                At my job we develop applications in Clojure to be deployed on a JVM platform. Our experience jibes with Norvig in that we can write a complete program in Clojure that is around 90% smaller than the equivalent Java code. A program that might require dozens or even hundreds of Java class files can be written in one or two Clojure modules.

                                                                                                I do agree with @moodyharsh that optimizing for understandability is probably a better goal than reducing up-front development time. The applications we deliver are expected to be in service for many years, with the oldest of these at 8+ years and counting. There are now around 200 people who program in this environment and any of us might be expected to maintain or modify code authored by another. In aggregate they are an average group of programmers. (But highly adept at delivering business value—most just aren’t formally trained in CS or software engineering.)

                                                                                                I would say that optimizing for code size isn’t generally advantageous when you’re talking about minor reductions in lines-of-code, but I think there’s a clear benefit to choosing a language that affords an order-of-magnitude reduction. (If I had my way, we’d all be programming in APL, which I’ve found to be another 10x more expressive than Clojure, but that’s for another time.)

                                                                                                As for readability, I don’t believe that the syntax makes for a significant impediment although the choices made in formatting can have an effect. Rather, I think the major challenges arise from complex business logic and data modeling. The lightweight syntax and operators provided by Clojure for modeling ad-hoc data as vectors or hash maps are far simpler than Java, comparable to Python which is the language the majority of our staff has the most prior experience. But I’ve not found that the choice of language really makes a big difference in simplifying the business logic. A poorly designed program will be hard to understand or modify regardless of its implementation language.

                                                                                                One area of particular challenge for us is reasoning about the asynchronous parts of our programs. Thus far, I haven’t seen any language that makes asynchronous programming as simple as it needs to be for the average business programmer.

                                                                                                1. 2

                                                                                                  One area of particular challenge for us is reasoning about the asynchronous parts of our programs. Thus far, I haven’t seen any language that makes asynchronous programming as simple as it needs to be for the average business programmer.

                                                                                                  Might be worth checking out Elixir. It has a rep for being Ruby on the BEAM, but the language actually has Clojure and (obviously) Erlang influence.

                                                                                                  1. 2

                                                                                                    What kind of asynchronous programming are you referring to? I find there are two kinds of asynchronous behavior I need to reason about: async-as-implementation-detail and async-as-business-logic.

                                                                                                    On the implementation side, I like Kotlin’s approach, which is to default to awaiting unless the code says not to. Flipping the default behavior is a huge quality-of-life improvement. I dislike writing asynchronous code in languages like JavaScript where nearly every line of your code starts with “await” and if you leave it out by mistake, you’ve probably just introduced a hard-to-diagnose intermittent bug. Await-by-default means you can write async code as if it were synchronous.

                                                                                                    But the business logic side is the tricky one. Event-based architectures have this a lot and it can be a pain to work with. Event A triggers event B which triggers event C, all indirectly delivered through a message queue, so tracing the source of a weird value in event C is never as simple as just looking at a stack trace.

                                                                                                    1. 1

                                                                                                      Mostly implementation details, such as a call out to an API service or database query that returns a future. A lot of the trouble comes around deciding how soon in the program can we schedule the future, and how far later can we wait to block on its result, while maintaining understandable code that doesn’t spread its concerns all over he place.

                                                                                                  1. 11

                                                                                                    Operator overloading is interesting to me in that it demonstrates how much the programming culture around a language can affect whether the language’s features turn out well in practice.

                                                                                                    No statistics to back this up, but I think a lot of people who recoil in horror at the prospect of operator overloading probably arrived at that opinion after working with C++ code. Operator overloading seems to be widely thought of as abused in idiomatic C++.

                                                                                                    But other languages have operator overloading too and their idioms have evolved to use it more sparingly. To take the article’s specific example: Kotlin on JVM overloads the + and * operators for the Java BigDecimal class. Idiomatic Kotlin code adds and multiplies BigDecimals using those operators just like the author wants to do. But I have yet to run across any Kotlin code that, say, overloads bit-shift operators to perform I/O operations. Nothing in the language prevents you from doing that kind of thing if you choose, but it’s not considered good style by the community.

                                                                                                    Of course, as a language designer you’re rolling the dice to some extent. You can’t know if your language’s community will take some new language feature and run with it in a horrible direction. My point is mostly just that it’s not a given that things like operator overloading will be commonly abused just because they can be.

                                                                                                    1. 4

                                                                                                      The answer to this might be as simple as whether you overload operators by name (e.g. the “inc” method) or by symbol (operator ++(..)). The former discourages you from changing the operation’s semantics.

                                                                                                      1. 2

                                                                                                        I think this is one of C++’s biggest issues.

                                                                                                        In my opinion, you should provide the language with information about how to do something — this is how you add two of this type together, this is how you move this type, this is how you copy this type, this is how you dereference this type — and the language should decide when to do those things. But instead, you tell the language what should happen when the + operator is used, that should happen when the -> operator is used, when the * operator is used, you provide the T(&&T) constructor and the T(const &T) constructor and the =(&&T) operator and the =(const &T) operator.

                                                                                                        Not only does this result in a lot of boilerplate (two ways to move, two ways to copy, two ways to add, two special ways to add 1, two ways to subtract, two ways to subtract 1, ….); it also encourages operator overloading abuse because you’re overloading operators and not semantics.

                                                                                                        Who said using “<<“ for writing to a stream is wrong? You’re just overloading the symbols “<<“; nothing is suggesting that “<<“ should be any kind of left shift. (Except that the precedence rules for “<<“ as a stream write operator are completely bonkers.)

                                                                                                        1. 2

                                                                                                          It’s worth noting that there are sometimes cases where you want to define ++ and not + (for example, ++ on an object representing a date would mean ‘the next date’, but you can’t sensibly add dates) or + but not ++ (matrices, vectors).

                                                                                                          1. 2

                                                                                                            I think that’s more a matter of wanting to define + on a Date only if the right hand side is an integer and not a date, ++ meaning +1. C++ does this kind of overloading very “well”.

                                                                                                      2. 2

                                                                                                        I don’t know for sure, but I suspect the opposite is true: language features determine the culture. There are different ways to implement operator overloading, and they lead to different results.

                                                                                                        One of the core design tenants of C++ is that user-defined types should do everything that a built in types do. Hence, it’s operator overloading supports everything. You can overload ,, ->, =, and implicit conversions.

                                                                                                        In Scala and Haskell, one of the goals is to be able to “write abstract math”, so there you can define your own custom operators (and even custom precedence, in Haskell).

                                                                                                        In Kotlin and (to a lesser extent) Python, operator overloading is very tame and scoped only to overload enough syntax to make BigInt and the like work.

                                                                                                        But I have yet to run across any Kotlin code that, say, overloads bit-shift operators to perform I/O operations.

                                                                                                        Counter-example: a bunch of pre-Compose UI frameworks which didn’t have access to compiler plugins overrode unary plus or call without parameters to mean “add the current thing to the enclosing UI container”.

                                                                                                        1. 2

                                                                                                          Operator overloading is one of those features that is not intrinsically evil, but which doesn’t compose well with some others. In particular, it really doesn’t play well with operator precedence. a + b * c as an arithmetic expression may make sense to execute as a + (b * c) because that’s how people are taught arithmetic, but what if a and b are strings, operator+ on string means concatenate, c is an integer and operator* on string and integer is repeat? Should "foo" + "bar" * 2 evaluate to "foobarfoobar" or "foobarbar"? The latter is consistent with arithmetic, but is probably surprising to a lot of readers who know the types and don’t think of the + and * as related.

                                                                                                          In Verona, we are supporting operator overloading (and any word or symbol can be an infix operator) but not precedence. Any sequence of the same operator is applied left to right. Any sequence of different operators is a parse error and requires explicit brackets.

                                                                                                          1. 1

                                                                                                            I think Rust got it right in this respect - have just a bunch of overloadable operators represented by traits covering the basic arithmetic operators that would help reduce boilerplate (as in the case of Java’s BigInteger and BigDecimal classes), but not unbridled overloading like in C++.

                                                                                                          1. 29

                                                                                                            I agree with the points raised in the article. One thing to add is that many tools built in academia are there to support the paper, not to take a life of their own. Given the short life of a thesis, there is not much one could do to make these tools gain much traction (all examples in the article show this).

                                                                                                            There is another weird thing in our industry — I’ve seen many companies embracing the latest shiny tools and frameworks (k8s, react, containers, …), yet when it comes to thinking and implementing ways to speed up developer work by improving build times or reorganize dependencies, that is always a task for the intern, the least suitable person to handle such a thing.

                                                                                                            1. 10

                                                                                                              yet when it comes to thinking and implementing ways to speed up developer work by improving build times or reorganize dependencies, that is always a task for the intern, the least suitable person to handle such a thing.

                                                                                                              That has not been my experience. I’ve been tasked with that sort of work many times, and given permission to pursue it many more. Currently a senior UI engineer at Microsoft, but previously worked at a 500 person EdTech company for several years, and had a brief, but enjoyable stint at Todoist. All three were very happy to give that work to an experienced developer.

                                                                                                              Have I had managers who were averse to that? Yes. But it has been the exception.

                                                                                                              1. 3

                                                                                                                Agreed with this. My experience at both large and small tech firms has been that when a senior person wanted to work on developer productivity, it was almost always welcomed. Maybe it’s different at companies whose product is something other than what the developers are building, though.

                                                                                                              2. 7

                                                                                                                The problem in academia is that it is controlled by where the professor can get money from. There is a lot more grant money for making shiny new tools than for maintaining and adapting old tools, and your academic life depends really strongly on how much grant money you can bring in (determines how many students you can support, how many conferences you can send your students to, and what equipment you can give your students). (New papers are also indirectly about grant money. You need new papers to bring in grant money, and you can’t write new papers simply by maintaining or updating existing tools).

                                                                                                                1. 3

                                                                                                                  I agree, and it’s also about promotion criteria; it would be hard to get tenure if all you ever worked on was maintaining old tools. I think the solution is to create a new tenure-track academic role which is evaluated, promoted, and funded based on the maintenance and creation of useful open-source artifacts (rather than the traditional goal of the discovery of novel knowledge).