1. 2

    I used it here and there. It was never very good, but it was a neat trick to run Finder on a Unix.

    I first used it around 1996, at work. I didn’t have to do much Unixy stuff on it; I get the impression now that it was there because of a military contract. I later used it in 1998-99 for fun in college, and was struck by how much better NetBSD was on the same machines.

    Now, I have an SE/30 and a Quadra 840av. A/UX doesn’t run on the latter, and I’d prefer not to waste the former on SVR2 :)

    1. 3

      Maybe I’m a bit crazy but I feel like if you’re programing elixir with message passing in mind, then you’re doing something wrong. 99% on your code should be purely functional and you should not be thinking about message passing (which is fundamentally stateful/effectful). Sure, it’s great to keep in mind that it’s happening behind the scenes, and that grokking the mechanism can give you confidence about the robustness of your code, but I do not architect the bulk of my code considering message passing, except if I’m digging really deep and writing e.g. a low-level tcp protocol implementation (which I almost never am).

      Is it different in the erlang community?

      1. 8

        Elixir and Erlang do not have a “pure FP” philosophy. Evaluation is eager, typing is dynamic, and side effects are basically never pushed into a monad in practice. Some of the core libraries even proudly expose global/cluster-wide state.

        The parts of FP that made it in (first-class functions, immutable data structures, certainly some other aspects I am missing) are there because they are useful for building systems that are resilient, debuggable, repairable, upgradable, and evolvable with ~zero downtime.

        That is the primary goal of Erlang, and Elixir inherits this legacy. It’s a goal, not a philosophy, so you may find competing ideas next to each other, because they’ve been shown to work well together in this context.

        1. 7

          total nitpick, but pure FP requires neither static typing nor lazyness.

          1. 2

            Also effects in FP languages can be, and usually are modeled using process calculi which are exactly what Erlang offers!

            That being said, Erlang also has side effects apart from message passing.

          2. 2

            never claimed it is pure FP. The VM is nice in that gives you places to breach FP pureness in exactly the right spots and makes it very hard or ugly to breaching pureness where doing so is dangerous.

            1. 4

              My mistake, I thought you were surprised at message passing from a pure-FP point of view.

              Another reason to think of message passing, and more broadly genservers/processes, in particular is that they can become bottlenecks if used carelessly. People talk about genservers as a building block of concurrency, which isn’t false, but from another point of view they are Erlang’s units of single-threaded computation. They only process one message at a time, and this is a feature if you know how/when to use it, but a drawback at other times. Effective Elixir or Erlang development must keep in mind the overall pattern of message passing largely to avoid this issue (and, in highly performance-sensitive cases, to avoid the overhead of message passing itself).

              1. 1

                Love reminding people that genservers are a unit of single threaded ness.

                1. 1

                  I still can’t find a good word for it! https://twitter.com/gamache/status/1390326847662137355

          3. 6

            I can only offer my own experience (5 years of Erlang programming professionally).

            Message passing, like with FP, is a tool that can be (mis)used. Some of the best uses of multiple processes or message passing that I see in code are:

            • Enforcing a sequence on unordered events, or enforcing the single writer principle.
            • Bounded concurrency. Worker pools, queues, and more.
            • Implementing protocols. Protocols for passing messages between processes serve to standardize and abstract. Suppose you have a service on TCP that you want to extend to work over Websockets. The well-architected solution for this has 3 kinds of processes. 1 process that receives Erlang terms, and 2 processes that receive data along some transport (TCP, Websockets, etc.), and send Erlang terms. Structuring Erlang code in this way is an amazing aid in keeping code simple and organized.

            I’ll generally come across problems that are solved by processes/message passing when writing libraries. When writing application code that uses those libraries, it’s usually far less common.

            1. 4

              my advice is typically:

              • are you writing a library? You probably don’t need a genserver (except for odd things like “I need a “fire and forget” genserver to wrap an ets table, well, yeah).
              • ok so you still think you need a genserver? did you try Task (this is elixir-specific)
              • are you wrapping a stateful communication protocol? then go ahead use genserver.
              • are you creating a ‘smart cache’ for something IRL or external to the vm? then go ahead and use genserver
              • are you creating temporary shared state between users (like a chat room?) then go head and use genserver

              I like the bounded concurrency one. Should probably add it to my list. Are you creating a rate limiter or resource pool? then use genserver.

              1. 3

                There is nothing wrong with using gen_server in library. The thing is that in most cases it is not you who should start and manage that process - leave it up to the user. The “problem” is that there are 3 “kinds” of projects in Erlang world:

                • “libraries” which should not start their own supervisor in general and should leave all process management up to the end user
                • “library applications” which should be mostly self contained, and are mostly independent from the main application, for example OpenTelemetry, systemd integration, Telemetry Poller, etc.
                • “end-user applications” your application, where you handle all the data and processes on your own

                In each of these parts there are different needs and message passing will be used more or less, depending on the needs.

                1. 1

                  150% this. Dave Thomas got this trichotomy right in his empex talk, it’s just infuriating to me that his choice of “names” for these things is unnecessarily confusing.

                  1. 1

                    Sorry I mistyped… It should be “are you writing a library? If not, you should probably not be writing a genserver.

              2. 2

                In my experience (having done Elixir for 8 years), folks that don’t understand/think about messages and genservers in Elixir are at a severe disadvantage when debugging and grokking why their code is behaving some way.

                It’s a lot like people who in the previous generation of webdevs learned Rails but not Ruby…which is fitting, since those are the folks driving adoption of Phoenix.

                (There are also certainly people who reach for a genserver for everything, and that’s a whole additional annoying thing. Also the people who use Tasks when they really, really shouldn’t.)

                1. 1

                  I’ve not used Elixir, but it sounds from your description as if it has some abstractions that give you Erlang (technically Beam, I guess) processes but abstract away the message-passing details? The last time I wrote a non-trivial amount of Erlang code was about 15 years ago, but at least back then Erlang didn’t have anything like this. If you wanted concurrency, you used processes and message passing. If you didn’t want concurrency, there were several functional programming languages with much faster sequential execution than Erlang, so you’d probably chosen the wrong tool.

                  1. 1

                    Processes are the core feature that provides both error isolation and fault tolerance. As you build more robust systems in Elixir, the number of times you use processes increases.

                    Often it is correct to abstract this away behind a module or API, but you’re still passing messages.

                  1. 14

                    All of the negative observations about the CSV format in this article are true. What is not mentioned is that Excel, Google Sheets, and the rest of the serious spreadsheet applications work around these problems as a matter of course, so no one cares, nor should they have to. Emit reasonably-formatted fields (ISO 8601 for datetime, etc), quote everything properly, and it will be perfectly usable on the other end if it belongs in a spreadsheet at all.

                    1. 4

                      Yet here I am, working with the local equivalent of Fortune 500 companies that send me ; separated ‘CSV’s that use the , as the decimal separator.

                      1. 2

                        Emit reasonably-formatted fields (ISO 8601 for datetime, etc), quote everything properly, and it will be perfectly usable

                        But people don’t do those things. Some people do, but certainly not all of them. That’s the entire point.

                      1. 4

                        Here’s a little context on Delta and the linked blog post:

                        Delta is a format to describe documents’ contents and how it changes over time. This is a core piece of technology at Slab, that powers our real-time collaboration engine, thanks to the built-in support for Operational Transform (think multiple users working together in Google docs).

                        Though we’ve been using it internally for almost 4 years now, we’re finally open-sourcing it to the wider Elixir community.

                        Feel free to reach out if you have any questions or feedback!

                        1. 1

                          What kind of networking does the OT use underneath? e.g. are you using some form of epidemic broadcast?

                          1. 1

                            This library just implements the core OT format and algorithms, and not any networking component.

                            Though at Slab, we use this with Phoenix Channels on top of a custom GenServer & Supervisor and it has worked out great for us.

                          2. 1

                            Do Google docs still use OTs? I thought they moved away from them about a decade ago because they hit the problem everyone hits just after they’ve invested a lot in OTs: that they hit combinatorial problems when you have a lot of operations and they all need to compose.

                            1. 2

                              AFAIK Google Docs still uses OT, though they have probably moved away from the extremely rich data model supported by Wave early on. A visible artifact of this is that for each doc, there is a server coordinating updates, and with a large number of clients it can get overloaded (“Editing is temporarily disabled”). OT requires a centralized server to handle the propagation of updates.

                              CRDTs, conversely, allow a flatter topology in which the server is more like just another client. As you can imagine, having a centralized authority simplifies things, so CRDTs are generally more complex than OTs with equivalent operations, and were for a long time considered infeasible for real-time editing and similar use cases.

                              1. 1

                                Yes AFAWK they still use OT and never moved for realtime text editing.

                                To implement transform for OT you do need to consider all interactions between the different operations so the complexity is n^2 where n is the number of operations (although many combinations may be similar). Google Wave ran into this issue and was a common criticism even amongst their own engineering team. That’s why our design only includes 3 operations: insert, delete, and retain. A tradeoff example is: there is no replace operation. To effectively implement replace you would use insert + delete. There are subtle differences between a replace and a insert+delete but we did not feel they were worth the complexity it would create.

                            1. 3

                              I find the COBOL code on the site much more readable than the mess of brackets that is Elixir.

                              1. 2

                                Do you mean the 4 curly braces in 20 lines of code? The translated code looks fine to me. Would have a lot more braces if it had C based syntax.

                                1. 2

                                  In general, the thing I like most with COBOL is the (relative) lack of special characters. Sure, it could be worse.

                                2. 2

                                  I did have the same thought. I had never actually seen COBOL before this article. It has a very SQL look to it, which is nice and clean. The contrast between the clean COBOL and the typical bracket-y language syntax is very stark.

                                  1. 2

                                    Readability was never one of the common criticisms of COBOL. Quite the reverse: it’s mainly criticised for being verbose. There was a joke that there would be an OO extension to COBOL, just as C++ was to C, called ADD ONE TO COBOL RETURNING COBOL. COBOL, by design, doesn’t have a lot of the syntactic sugar that other languages provide that make things more terse and so is very easy to read but more effort to write (though probably less so with a modern editor that can do a lot of autocompletion).

                                  2. 1

                                    Elixir accepts pull requests; perhaps you should work up a patch to banish tuple and map syntax to instead live under DATA DIVISION.

                                    1. 1

                                      I doubt they’d accept it.

                                  1. 3

                                    Python is really a shit language for obfuscating things. In the Perl community, obfuscated scripts that print “Just another Perl hacker” have been a tradition for decades, and the richness of the language really pays off when you’re trying to be a jerk.

                                    Fresh from 2009 or so, here’s my best take on the genre – this produces a rendering of my email sig at the time. I’m particularly proud of the base 26 conversion :)

                                    $q=q((pete gamache));$_=fpeoiaglclivdlcgglvqhcbwclemhcbwflclvqivcrgqhcbwafak
                                    ;sub A{for(($a=1)..pop){$a*=$a*=$a+$a*$a;$a%=999999}$a%128}sub b{@A=a..pop}$
                                    qq.=chr A(uc b($1))while s/(..)//;$qq="($qq)";&z;print"$q $qq 14 /Courier ";
                                    sub c{$x=$_[0];$x-=6.28until$x<6.28;my$a;for(0..10){$a+=($x**(2*$_)*(-1)**$_
                                    )/_(2*$_)}$a*2**(-$_[0]/8)}sub _{$_[0]<1?1:$_[0]*_($_[0]-1)}print join (' ',
                                    '66 666 translate 510 0 moveto 0 dup dup lineto 50 moveto',map{$_,c($_/8)*50
                                    ,'lineto'}(0..444)),' stroke findfont exch scalefont setfont 222 -30 moveto'
                                    ,' show 222 20 moveto show showpage';sub z{$q=~s/(g.+E)/$1, $1\@gmail.com/i}
                                    
                                    1. 3

                                      This has been my .signature for ages:

                                      #!/usr/bin/perl                             https://domm.plix.at
                                      for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}
                                      
                                      1. 3
                                      1. 8

                                        I kind of want a new tag for urbit just so I don’t need to see the political fight every single time it comes up.

                                        1. 15

                                          It’d be cool if we had a community notm to avoid politics when discussing tech, but there is a sizable contingent of Lobsters here who cannot function that way and prioritize their desire to talk about politics over talking about the tech.

                                          1. 12

                                            Lots of projects have creators with shitty opinions, but few of those projects represent and implement those opinions. Urbit is tech that implements a political view, and to consider the tech separately from the politics is as inane as considering the politics separately from the tech.

                                            1. 12

                                              The politics of Urbit have been discussed many times over. There’s literally nothing new to talk about there. The tech, however, is ever evolving. Why not talk about that instead? Surely technical perspectives should be prioritized, at least here on Lobsters.

                                              1. 12

                                                The tech is “ever-evolving” I guess, but this article is a survey of the Urbit landscape, not a changelog.

                                                Anyway: go ahead, talk about the tech. But in this project’s case, the tech and the politics are deliberately tangled, so don’t be surprised or disappointed when the politics of the project come up for discussion too.

                                              2. 11

                                                An interesting and enlightening discussion around Urbit’s politics would talk about what those politics are, how they have influenced the tech behind Urbit, and finally to what degree Urbit represents a successful realization of those politics. This is similar to how we evaluate deeply political technology projects like Project Cybersyn.

                                                This is emphatically not the style of discussion we get here. Instead, people do the shallow (and perhaps even lazy) thing and make uninteresting (and often unsubstantiated and incoherent) moral claims, and then you just get a bunch of Lobsters clacking angrily at each other while high on self-righteousness. It’s all so tiresome.

                                                1. 5

                                                  Such a discussion would be granting the project a legitimacy that it hasn’t earned and doesn’t deserve.

                                                  1. 3

                                                    When was the last time Lobsters talked about Cybersyn? When I searched I see a handful of posts with a handful of comments. That’s not really a discussion. So I don’t know where this deep evaluation is happening.

                                                    1. 5

                                                      I found this in my DB of all submissions to lobste.rs, it got 6 upvotes and 1 comment:

                                                      https://www.jacobinmag.com/2015/04/allende-chile-beer-medina-cybersyn/

                                                      Interestingly it’s one of six submissions from Jacobin, the last one was in Nov 2018.

                                                      Including this one, 23 submissions have “urbit” in their title.

                                                      1. 3

                                                        So I don’t know where this deep evaluation is happening.

                                                        That would be my point–it ain’t happening here, because in the vast majority of lobsters (quite probably myself included) are incapable of objective, dispassionate policy discussion.

                                                        1. 9

                                                          because in the vast majority of lobsters (quite probably myself included) are incapable of objective, dispassionate policy discussion.

                                                          “Objective, dispassionate policy discussion” isn’t some Platonic ideal that’s fit for all subject matter. When the topic is an elaborate prank schemed up by some Neoreactionary narcissist, the appropriate response is manifestly not a contemplative survey of its politics or positions.

                                                          1. 5

                                                            I’m also reasonably sure political discussion is somewhat off-topic here. Further, if a technical topic is sufficiently politicised I think the technical discussion itself becomes off-topic since no one can reliably talk about it in a neutral way that doesn’t in some ways refer back to the political context of the work.

                                                        2. 2

                                                          I don’t disagree that some (lots) of the political disussion that ensues after an Urbit posting is shallow, lazy, tiresome, content-free drivel. But the reason it doesn’t belong here isn’t because politics don’t belong in Lobste.rs discussions.

                                                          1. 3

                                                            The politics of urbit itself match your descriptors. Pointing that out as an FYI may be a simple observation, but that doesn’t make it unimportant.

                                                            People can still ignore those threads and discuss urbits’s whacky-ass tech as is currently the top post. Nobody is preventing that discussion from occurring.

                                                        3. 3

                                                          I never understood the “He’s right wing so these are the choices he made” argument. (oversimplified? Yes. But I’m OK with that.) The first time I read it, sure, interesting, engaging. The second time, I tried again, and again it seemed there was a massive cultural gap between what people thought and what I could understand. Third time, similar… and then I stopped caring. Urbit seemed to not be going anywhere, and caring about a random person’s political opinions because he happened to be a tech author seemed like a waste of time.

                                                          An 18 second sketch that illustrates what this all seems like to me: https://www.youtube.com/watch?v=79GNnfDrgWM

                                                        4. 5

                                                          You’re active on all political discussion I’ve seen here in recent memory, including sharing your own politics opinions. I chime in on this stuff too because it interests me. Why pretend to be above it? Clearly people want to discuss the tech implications of politically-adjacent stuff like this. It’s not virtuous to remove extra-technical context from the forum, it just makes the discussion less informed.

                                                          1. 1

                                                            I don’t understand your point. People want to flame about politics, not talk about the tech related aspects, or I’d have fewer problems with the chatter.

                                                            1. 3

                                                              I’m not understanding how your pushback here relates to my point. But to respond - there’s plenty of tech discussion here by the people who want to be having it. I don’t see the political threads preventing people from engaging. The top two threads are about technical details. I don’t suppose that anyone extra would have jumped into the techy bits had nobody commented on the extra-techy bits. The site is designed to accommodate multiple threads of discussion. So the whole ‘politics distracts from the technical discussion’ is not something I’m buying in this case.

                                                      1. 6

                                                        Zig article -> 44 upvotes, Scala 3 -> 14 upvotes.

                                                        I am not sure what Scala is used for anymore. F# / Rust / Kotlin and probably OCaml ate away much of the user base over time. If you are an ex Scala coder, what language did you switch to?

                                                        1. 11

                                                          There’s 18 Apache projects written in Scala

                                                          • Apache CarbonData
                                                          • Apache Clerezza
                                                          • Apache Crunch (in the Attic)
                                                          • Apache cTAKES
                                                          • Apache Daffodil
                                                          • Apache ESME (in the Attic)
                                                          • Apache Flink
                                                          • Apache Hudi
                                                          • Apache Kafka
                                                          • Apache Polygene (in the Attic)
                                                          • Apache PredictionIO (in the Attic)
                                                          • Apache Samza
                                                          • Apache ServiceMix
                                                          • Apache Spark
                                                          • Apache Zeppelin

                                                          There’s some big names there (Spark, Kafka, Flink, Samza), especially in data movement. Also Netflix has atlas (time-series DB), Microsoft has hyperspace. Seems like most Scala code is associated to Spark in one way or another.

                                                          1. 2

                                                            Huh, I thought Kafka, Flink and Samza were written in Java. Shows what I know. Neat link!

                                                            1. 2

                                                              This overstates the case for Scala a bit. Check the founding years of these projects. Kafka and Spark, which are two of the most popular projects in this list, were created in 2011 and 2014, at the height of Scala popularity. Both projects were written in Scala, but had to put significant effort into engineering a first-class pure Java API. Kafka team even rewrote the clients in Java eventually. GitHub repo analysis has the Kafka codebase as 70% Java and 23% Scala.

                                                              It’s true that Spark does use Scala a bit more. GitHub there has Scala as 70% of codebase, with Python 13% and Java 8%. But Spark might just be the “perfect” use case for a language like Scala, being as focused as it is on concurrency, immutability, parallel computing, higher-order programming, etc.

                                                              I also closely tracked development of Apache Storm (created 2011), and it started as a Clojure project, but was eventually rewritten (from scratch) in Java. There are lots of issues with infrastructure software not sticking with vanilla Java (or other “systems” languages like C, Go, etc.). Apache Cassandra and Elasticsearch stuck with pure Java, and had fewer such issues. Durability, simplicity, and ecosystem matter more than programming language features.

                                                          2. 8

                                                            It’s still pretty big in data engineering. Apache Spark was written in Scala.

                                                            1. 6

                                                              The company I work for uses Scala for data engineering. I don’t think that team has any plans to move away from it. I suspect that the use of Scala is a product of the time: the company is about ten years old; Scala was chosen very early on. It was popular back then.

                                                            2. 7

                                                              Elixir and loving it for almost 6 years now. I miss a couple of Scala’s features; in particular, implicits were nice for things like DI, execution contexts, etc. I don’t miss all the syntax and I certainly don’t miss all the Haskell that creeps in in a team setting.

                                                              1. 6

                                                                If you are an ex Scala coder, what language did you switch to?

                                                                Python, now Go. At one point I figured I could ship entire features while waiting for my mid-sized Scala project to compile.

                                                                I hope they addressed that problem.

                                                                1. 3

                                                                  YAML in mid 2015 (I took a detour in infrastructure) but Clojure since 2019 now that I’m a dev again.

                                                                  FWIW I liked Scala as a “better Java” when I started using it around mid 2012, until early 2015 when I left that gig.

                                                                  I remember that I found it very difficult to navigate Scala’s matrix-style documentation; and that I hated implicits, and the operator precedence. I loved case classes, and I think I liked object classes (not sure that’s the right terminology). And I liked that vals were immutable.

                                                                  Compile times didn’t bother me that much, perhaps because I worked on early-stage greenfield projects with one or two other devs. (So didn’t have lots of code.)

                                                                  I liked programming with actors (we used Akka) but we found it difficult to monitor our services. Some devs were concerned about loss of type safety when using Akka actors.

                                                                  1. 2

                                                                    Akka actors are type-safe now for several years now.

                                                                    1. 2

                                                                      Off topic: there’s a lot of conversation/blog posts to be had about devs finding themselves in infrastructure. I’m there at the moment, and it’s a very different world.

                                                                    2. 3

                                                                      I’m using Scala for my hobby projects. It does all I need. I like multiple argument lists and ability to convert last argument to a block. Implicits are cool, though they need to be controlled, because sometimes they’re confusing; for example, I don’t really understand how uPickle library works inside, even though I know how to use it. Compilation times are not that bad as some people say; maybe they were bad in some early Scala versions, but they’re not as bad as e.g. C++. It works with Java libraries (though sometimes it’s awkward to use Java-style libs in Scala, but it’s the same story as using C from C++ – it’s a matter of creating some wrappers here and there).

                                                                      1. 3

                                                                        I wrote several big scala projects a decade ago (naggati, scrooge, kestrel) but stopped pretty much as soon as I stopped being paid to. Now I always reach for typescript, rust, or python for a new project. Of the three, rust seems the most obviously directly influenced by (the good parts of) scala, and would probably be the most natural migration.

                                                                        Others covered some of the biggest pain points, like incredibly long compile times and odd syntax. I’ll add:

                                                                        Java interoperability hurt. They couldn’t get rid of null, so you often needed to watch out and check for null, even if you were using Option. Same for all other java warts, like boxes and type erasure.

                                                                        They never said “no” to features. Even by 2011 the language was far too big to keep in your head, and different coders would write using different subsets, so it was often hard to understand other people’s code. (C++ has a similar problem.) Operator overloading, combined with the ability to make up operators from nothing (like <+-+>) meant that some libraries would encourage code that was literally unreadable. Implicits were useful to an extent, but required constant vigilance or you would be squinting at code at 3am going “where the heck did this function get imported from?”

                                                                        1. 2

                                                                          2 days later, Scala 3 -> 44 upvotes ;)

                                                                          1. 1

                                                                            I’m currently in Scala hiatus after nearly eight years as my primary stack with a splash of Rust, Ruby, Groovy, and a whole lot of shell scripting across several products at three companies. This included an IBM product that peaked near $100M/yr in sales. The major component I managed was JVM-only and 3/4 of it was Scala.

                                                                            For the last few months, I’m working in Python doing some PySpark and some Tensorflow and PyTorch computer vision stuff. While I concede that the Python ecosystem has certainly matured in the 15 years since I did anything material with it, my preference would be to rewrite everything I’m writing presently in Scala if the key libraries were available and their (re)implementations were mature.

                                                                          1. 5

                                                                            My Elixir deployment is mostly on ARM servers these days, so I have to wait a little bit before I can fully enjoy the JIT. But I am enjoying the ArgumentError improvements already – what a difference! Congrats to the team.

                                                                            1. 8

                                                                              Congrats to the team! I love seeing the steady progress on the project and seeing these ideas explored on the BEAM. In particular, making all assignments expressions instead of statements is going to be a big quality-of-life improvement.

                                                                              1. 6

                                                                                I like lisp but macros should be a last resort thing. Is it really needed in those cases, I wonder.

                                                                                1. 18

                                                                                  I disagree. Macros, if anything, are easier to reason about than functions, because in the vast majority of cases their expansions are deterministic, and in every situation they can be expanded and inspected at compile-time, before any code has run. The vast majority of bugs that I’ve made have been in normal application logic, not my macros - it’s much more difficult to reason about things whose interesting behavior is at run-time than at compile-time.

                                                                                  Moreover, most macros are limited to simple tree structure processing, which is far more constrained than all of the things you can get up to in your application code.

                                                                                  Can you make difficult-to-understand code with macros? Absolutely. However, the vast majority of Common Lisp code that I see is written by programmers disciplined enough to not do that - when you write good macros, they make code more readable.

                                                                                  1. 3

                                                                                    “Macros, if anything, are easier to reason about than functions, because in the vast majority of cases their expansions are deterministic, and in every situation they can be expanded and inspected at compile-time, before any code has run. The vast majority of bugs that I’ve made have been in normal application logic”

                                                                                    What you’ve just argued for are deterministic, simple functions whose behavior is understandable at compile time. They have the benefits you describe. Such code is common in real-time and safety/security-critical coding. An extra benefit is that static analysis, automated testing, and so on can easily flush bugs out in it. Tools that help optimize performance might also benefit from such code just due to easier analysis.

                                                                                    From there, there’s macros. The drawback of macros is they might not be understood instantly like a programmer will understand common, language constructs. If done right (esp names/docs), then this won’t be a problem. Next problem author already notes is that tooling breaks down on them. Although I didn’t prove it out, I hypothesized this process to make them reliable:

                                                                                    1. Write the code that the macros would output first on a few variations of inputs. Simple, deterministic functions operating on data. Make sure it has pre/post conditions and invariants. Make sure these pass above QA methods.

                                                                                    2. Write the same code operating on code (or trees or whatever) in an environment that allows similar compile-time QA. Port pre/post conditions and invariants to code form. Make sure that passes QA.

                                                                                    3. Make final macro that’s a mapping 1-to-1 of that to target language. This step can be eliminated where target language already has excellent QA tooling and macro support. Idk if any do, though.

                                                                                    4. Optionally, if the environment supports it, use an optimizing compiler on the macros integrated with the development environment so the code transformations run super-fast during development iterations. This was speculation on my part. I don’t know if any environment implements something like this. This could also be a preprocessing step.

                                                                                    The resulting macros using 1-3 should be more reliable than most functions people would’ve used in their place.

                                                                                    1. 2

                                                                                      What you’ve just argued for are deterministic, simple functions whose behavior is understandable at compile time.

                                                                                      In a very local sense, I agree with you - a simple function is easier to understand than a complex function.

                                                                                      However, that’s not a very interesting property.

                                                                                      A more interesting question/property is “Is a large, complex system made out of small, simpler functions easier to manipulate than one made from larger, more complex functions?”

                                                                                      My experience has been that, when I create lots of small, simple functions, the overall accidental complexity of the system increases. Ignoring that accidental complexity for the time being, all problems have some essential complexity to them. If you make smaller, simpler functions, you end up having to make more of them to implement your design in all of its essential complexity - which, in my experience, ends up adding far more accidental complexity due to indirection and abstraction than a smaller number of larger functions.

                                                                                      That aside, I think that your process for making macros more reliable is interesting - is it meant to make them more reliable for humans or to integrate tools with them better?

                                                                                      1. 1

                                                                                        “A more interesting question/property is “Is a large, complex system made out of small, simpler functions easier to manipulate than one made from larger, more complex functions?”

                                                                                        I think the question might be what is simple and what is complex? Another is simple for humans or machines? I liked the kinds of abstractions and generative techniques that let a human understand something that produced what was easy for a machine to work with. In general, I think the two often contradict.

                                                                                        That leads to your next point where increasing the number of simple functions actually made it more complex for you. That happened in formally-verified systems, too, where simplifications for proof assistants made it ugly for humans. I guess it should be as simple as it can be without causing extra problems. I have no precise measurement of that. Plus, more R&D invested in generative techniques that connect high-level, human-readable representations to machine-analyzable ones. Quick examples to make it clear might be Python vs C’s looping, parallel for in non-parallel language, or per-module choices for memory management (eg GC’s).

                                                                                        “is it meant to make them more reliable for humans or to integrate tools with them better?”

                                                                                        Just reliable in general: they do precisely what they’re specified to do. From there, humans or tools could use them. Humans will use them as they did before except with precise, behavioral information on them at the interface. Looking at contracts, tools already exist to generate tests or proof conditions from them.

                                                                                        Another benefit might be integration with machine learning to spot refactoring opportunities, esp if it’s simple swaps. For example, there’s a library function that does something, a macro that generates an optimized-for-machine version (eg parallelism), and the tool swaps them out based on both function signature and info in specification.

                                                                                  2. 7

                                                                                    Want to trade longer runtimes for longer compile times? There’s a tool for that. Need to execute a bit of code in the caller’s context, without forcing boilerplate on the developer? There’s a tool for that. Macros are a tool, not a last resort. I’m sure Grammarly’s code is no more of a monstrosity than you’d see at the equivalent Java shop, if the equivalent Java shop existed.

                                                                                    1. 9

                                                                                      Java shop would be using a bunch of annotations, dependency injection and similar compile time tricks with codegen. So still macros, just much less convenient to write :)

                                                                                      1. 1

                                                                                        the equivalent Java shop

                                                                                        I guess that would be Languagetool. How much of a monstrosity it is is left as an exercise to the reader, mostly because it’s free software and anybody can read it.

                                                                                      2. 7

                                                                                        This reminds me of when Paul Graham was bragging about how ViaWeb was like 25% macros and other lispers were kind of just looking on in horror trying to imagine what a headache it must be to debug.

                                                                                        1. 6

                                                                                          The source code of the Viaweb editor was probably about 20-25% macros. Macros are harder to write than ordinary Lisp functions, and it’s considered to be bad style to use them when they’re not necessary. So every macro in that code is there because it has to be. What that means is that at least 20-25% of the code in this program is doing things that you can’t easily do in any other language.

                                                                                          It’s such a bizarre argument.

                                                                                          1. 3

                                                                                            I find it persuasive. If a choice is made by someone who knows better, that choice probably has a good justification.

                                                                                            1. 11

                                                                                              It’s a terrible argument; it jumps from “it’s considered to be bad style to use [macros] when they’re not necessary” straight to “therefore they must have been necessary” without even considering “therefore the code base exhibited bad style” which is far more likely. Typical pg arrogance and misdirection.

                                                                                              1. 3

                                                                                                I don’t have any insight into whether the macros are necessary; it’s the last statement I take issue with. For example: Haskell has a lot of complicated machinery for working with state and such that doesn’t exist in other languages, but that doesn’t mean those other languages can’t work with state. They just do it differently.

                                                                                                Or to pick a more concrete example, the existence of the loop macro and the fact that it’s implemented as a macro doesn’t mean other languages can’t have powerful iteration capabilities.

                                                                                                1. 1

                                                                                                  One hopes.

                                                                                          1. 2

                                                                                            While not in the same league, one may think of Kubernetes (also written in Go, BTW) as an alternative to the BEAM capable of orchestrating macro-scale processes (pods, services).

                                                                                            I have noticed this too. Can anyone from elixir confirm this ? I wanted to learn elixir just to avoid kubernetes but erlang concepts are just weird. There are people who tried to port more BEAM style solutions to go on github.

                                                                                            I am still not sure how different elixir is to deploying with lets say ansible + supervisord + python (with workers) on multiple servers, through an nginx proxy. Isn’t the real challenge of scaling database sharding, which is already solved by nosql ?

                                                                                            1. 8

                                                                                              While not in the same league, one may think of Kubernetes (also written in Go, BTW) as an alternative to the BEAM capable of orchestrating macro-scale processes (pods, services).

                                                                                              I have noticed this too. Can anyone from elixir confirm this ? I wanted to learn elixir just to avoid kubernetes but erlang concepts are just weird. There are people who tried to port more BEAM style solutions to go on github.

                                                                                              Think of Erlang as its own OS and you’re not far off. It’s not weird, per se, it just doesn’t resemble the other OSes you’ve used before. Also, it’s not like k8s isn’t a little weird too :)

                                                                                              I am still not sure how different elixir is to deploying with lets say ansible + supervisord + python on multiple servers, through an nginx proxy. Isn’t the real challenge of scaling database sharding, which is already solved by nosql ?

                                                                                              Deploying Elixir is a non-issue. Throw it in a Docker container and do what you want with it, if you like. There is no single “real challenge of scaling”, but Elixir is there not to be the challenge.

                                                                                              1. 6

                                                                                                So. I worked with a system that used to be ansible + uwsgi + python and I rebuilt it in elixir. It was a vm orchestrator (think bespoke ec2 for a special class of machine + customer). Maintaining the system with python was awful since it doesn’t really have a sane way to handle tracking state concurrently, especially when you are multidatacenter. Python forces you to think synchronously. When your objects go out of sync with reality you wind up in trouble (we had zombie vms all the time). Elixir by contrast makes you distrust any local data as soon as it’s obtained and guides you to smart strategies to handle conflict (crash the process and start over with a fresh pull of data).

                                                                                                However I had to leave when I was knocked down the org chart and the cto that was put between me and the ceo could not understand this.

                                                                                                1. 1

                                                                                                  When your objects go out of sync ?

                                                                                                  How can this happen ? Can’t we use database/redis as the source of state ?

                                                                                                  1. 3

                                                                                                    Yeah that’s the problem. The CTO wanted to treat the database as authoritative, when the source of truth was things IRL. It’s the object oriented mindset. And then I got blamed when stuff was not in sync, and the cto refused to create something that triggered a celery job to periodically poll my component to update the database.

                                                                                                    And I couldn’t have my component write to the database, because it was in a different data center (not that that would have been a good idea, due to separation of concerns).

                                                                                                    1. 1

                                                                                                      That sucks. I can see how elixir can make you think carefully about such state. That’s something I got out of a bit of clojure.

                                                                                                  2. 1

                                                                                                    Where was your authoritative data though? I can’t imagine that elixir somehow does away with a need for some source of truth.

                                                                                                    1. 1

                                                                                                      The authoritative information about the state of the vms lives at the edge, on the host machines. The authoritative information about customer intent lives in the database. So already there is something dangerous because information is not centralized. And it gets complicated. Suppose a customer does sudo shutdown from inside the vm instead of terminating from the dashboard (to be fair this was a huge problem for end users in the early ec2 days too). You have to think hard about reconciling these differences (and tolerating other corner cases, like netsplits or network disruptions, or orchestrator software redeploys, that might for reasons look like vm shutdown depending on how you are checking).

                                                                                                      Elixir doesn’t do away with that need but it just surfaces these concerns early because it doesn’t try to hide them behind artificial abstractions like objects. As an elixir dev you get into the habit of distrusting (in the sense of is it stale?) any data that comes your way. Along with that mentality it provides a very convenient way of dealing with inconsistency or confusion: log it, crash your process, and have it restarted with a fresh pull of data, so you also don’t have to think about reconciling any data dependencies, because they will be populated in the way that you expect from the fresh pull.

                                                                                                  3. 5
                                                                                                    1. 2

                                                                                                      The section on partial failures is enlightening. Having faced something like it in python, I can totally see how it works … but itsn’t the crash and fail model similar to apache + php in that case ?

                                                                                                      1. 5

                                                                                                        The key difference is the Erlang OTP model is often modelled in long running processes usually as gen_servers. Those processes are supervised and those processes are grouped such that failures are isolated. This structure makes it easy to reason about and isolate partial failures in your application.

                                                                                                        You can certainly provide the same isolation in PHP, but it not part of the core PHP platform.

                                                                                                        1. 1

                                                                                                          It is similar to many things:

                                                                                                          • system supervisor restarting your application
                                                                                                          • ASG restarting your VM
                                                                                                          • CGI when retrying request

                                                                                                          The main difference between BEAM platform and others, is that BEAM makes it layered and basis for most of its work. So restarting is very quick, that in many cases you shouldn’t even notice that. Additional thing there is that the failure is highly isolated - it causes only single process to fail - no other parts of your system are affected at all (for example failure in one HTTP request do not cause any troubles in other, concurrent, requests in the same system).

                                                                                                      2. 3

                                                                                                        BEAM provides you with fault tolerance at the scale of a server rack, whilst K8S provides region failover. They do not operate at the same scale and this is by design. Now if you want, you can base an a big infrastructure on BEAM clusters, but you’ll be doing most of the hard work.

                                                                                                      1. 9

                                                                                                        This is a very unconvincing argument, literally so – you spend the precious few bytes asserting, not convincing or explaining or hinting or anything that would help the reader translate these abstract statements into something precise. This isn’t a koan, it’s 1/5th of a blog post.

                                                                                                        1. 2

                                                                                                          It has been useful on its own for some. There are concrete examples linked at the end if you don’t find it sufficient. Would you prefer if those concrete examples were included directly at the end of the post? I feel like that wouldn’t really be aesthetically appropriate…

                                                                                                          1. 9

                                                                                                            In my opinion (and I can’t state that this is my opinion strongly enough), Alan Kay’s early talks were more often about leading an audience to a conclusion rather than a conclusion outright. He then spent the remaining years yelling at the industry for failing to read his mind correctly (or failing to get his conclusions). Koans aren’t always appropriate. Often times, it’s hard enough to get people to accept good ideas stated directly.

                                                                                                        1. 1

                                                                                                          Long long ago, when both I and my college’s EE department were too cheap to purchase a PIC programmer, I made a two-switch input device. BIT (toggle) and CLK (momentary, debounced) are enough to slam code into a PIC, if you’re spiteful enough (and if you are using PIC asm, you are filled with spite). It worked and everything.

                                                                                                          Funny hack, but ultimately I ponied up for a BASIC Stamp and the project got a lot easier.

                                                                                                          1. 31

                                                                                                            If the author is here and open to changing the post title, Rust is for everyone, and good for professionals too! The title as is sounds like it may be exclusionary toward people who don’t identity as professionals, or feel intimidated by Rust, which is contrary to Rust’s core goal of opening up systems programming to everyone!

                                                                                                            1. 39

                                                                                                              I, the author, am here.

                                                                                                              I appreciate the constructive feedback and can see your point. Choosing titles is hard. I too was concerned about the exclusionary potential for the title. However, I thought I had it sufficiently mitigated by the introduction paragraphs. Apparently I could have done better.

                                                                                                              I’m not going to change the title of this post because I don’t want to deal with the hassle of rewriting URLs. However, I will try to consider your feedback for future posts I author.

                                                                                                              1. 6

                                                                                                                I appreciated the way you took the feedback. Just wanted to say thanks for listening and considering it.

                                                                                                                1. 3

                                                                                                                  I’ve programmed in the following languages: C, C++ (only until C++11), C#, Erlang, Go, JavaScript, Java, Lua, Perl, PHP, Python, Ruby, Rust, shell, SQL, and Verilog.

                                                                                                                  To me, Rust introduced a number of new concepts, like match for control flow, enums as algebraic types, the borrow checker, the Option and Result types/enums and more.

                                                                                                                  How did you use Erlang without pattern matching?

                                                                                                                  1. 2

                                                                                                                    I think the order is alphabetical so its possible that author used Erlang after he learned Rust.

                                                                                                                    Have a nice day!

                                                                                                                    1. 1

                                                                                                                      Doesn’t add up, he says that Rust introduced pattern matching to him :-)

                                                                                                                  2. 3

                                                                                                                    My usual way of framing it is “Rust is an industrial programming language”. That leaves the door open for practitioners of all kinds. It’s more like a cars are (in general) a utility and not a plaything or research endeavor, but are open to enthusiasts and amateur practitioners as well.

                                                                                                                    1. 2

                                                                                                                      Thanks! I liked the disclaimer, and think predicting how people will respond can be tough. Appreciate you putting in the time to write this, and agree that Rust can be used in professional / production environments successfully, and that it can feel like a breath of fresh air for many!

                                                                                                                    2. 22

                                                                                                                      Or, alternately, when someone writes 13,000 words about how nice Rust is, how ergonomic and human-centric and welcoming and humane the language and tooling and community are, they can keep the title without someone telling them they’re maybe being exclusionary.

                                                                                                                      1. 16

                                                                                                                        It’s constructive criticism. If you post something on the public web, it’s fair game to criticize it- especially if done in a polite manner.

                                                                                                                        1. 23

                                                                                                                          It’s criticism, but just because it’s delivered politely does not mean it is constructive. There is a cost to spending the effort to write a thirteen thousand word love letter to a language, just to have someone nipping at your ankles about how the first four words might exclude someone. Objectively speaking, this is a blogger that Rust would benefit from sticking around – but what is the signal they’ve received here?

                                                                                                                          1. 11

                                                                                                                            Agree 💯.

                                                                                                                            Recently a friend submitted his static site generator which is optimised for math heavy sites on HN.

                                                                                                                            He made the fatal mistake of calling it “beautiful” in the description and the amount of hangups that people had. It was done in a mostly polite manner, but the amount of time and energy wasted in discussing just the first sentence with barely any technical discussion of the internals, was a form of pathological bikeshedding that I was surprised to see. It was just that “nipping at his ankles” as you’ve described, which can take its toll.

                                                                                                                            Vowed to never take that site seriously again.

                                                                                                                            1. 1

                                                                                                                              It was done in a mostly polite manner, but the amount of time and energy wasted in discussing just the first sentence with barely any technical discussion of the internals, was a form of pathological bikeshedding that I was surprised to see. It was just that “nipping at his ankles” as you’ve described, which can take its toll.

                                                                                                                              Agreed. Yet here we are- arguing about arguing. I need a life… xD

                                                                                                                              1. 1

                                                                                                                                Right, in my experience, HN often attracts uninteresting, unhelpful, or even mean-spirited commentary. Sometimes I am surprised by the opposite, though. In any case, understanding the dynamic really helps. A lot of commentary, seems to me, are upset at some level in some way and take it out in their commentary.

                                                                                                                              2. 3

                                                                                                                                The author thanked them and described the feedback as constructive. So if they feel like they’ve taken something away from it, who are we to decide otherwise? My own hopefully constructive feedback here is that this subthread strikes me as projecting conflict into an interaction between two parties who themselves seem to have had a positive experience with it.

                                                                                                                                1. 2

                                                                                                                                  Yeah, I get where you’re coming from and I don’t really disagree.

                                                                                                                                  But I also think that the person who offered the criticism is just trying to help the author get more views.

                                                                                                                                  It’s fair to suggest that we all should consider our criticisms carefully, though. Because you’re right- if the criticism isn’t that important, we may just be disincentivizing someone from future contributions.

                                                                                                                                  I also wonder if you’re maybe reading a little too hard into the choice of the word “exclude” used by the criticizer, though… I think that term has a little bit of extra charge and connotation in today’s social environment and I don’t believe the criticizer meant it in that way. I think they simply meant that a person who isn’t writing enterprise software might pass over the article because they believe they aren’t the target audience. I agree with that conclusion as well, because I do write enterprisey software and I thought the article was going to be about that specific point of view- I thought I would see arguments about productivity and developer costs and cross-platform support and other less-sexy stuff. But, it was really- as you said, just a love letter to everything about Rust.

                                                                                                                              3. 8

                                                                                                                                I felt this lead-in made it a friendly suggestion and not harsh:

                                                                                                                                If the author is here and open to changing the post title

                                                                                                                                1. 4

                                                                                                                                  FWIW first third of those thousands of words is irellevant chitchat. The First actual claim on why is rust good is rust makes me giddy. I’m not sure “giddy” describes profesionals. It’s not no, but for me definitely not a yes either - giddy is just…giddy. Then all the technical points, borrow checkers, race conditions and whatnot - none of that tells me why is it a thing for professionals.

                                                                                                                                  What I mean is, I don’t know if rust is for pros or not, I just think that the title is a bit click-baity, and “rust is good for everyone, even pros” like alilleybrinker suggested would work much better.

                                                                                                                                  1. 7

                                                                                                                                    The First actual claim on why is rust good is rust makes me giddy. I’m not sure “giddy” describes profesionals.

                                                                                                                                    There’s a layer of extra irony here in that the word “amateur” comes from a root meaning “love”; in other words, amateurs are people who do something because they love it. So taken literally, this is very unprofessional!

                                                                                                                                2. 18

                                                                                                                                  It seems relevant to note that the author does address this in the post itself:

                                                                                                                                  The statement Rust is for Professionals does not imply any logical variant thereof. e.g. I am not implying Rust is not for non-professionals. Rather, the subject/thesis merely defines the audience I want to speak to: people who spend a lot of time authoring, maintaining, and supporting software and are invested in its longer-term outcomes.

                                                                                                                                  1. 10

                                                                                                                                    Yes, except the title doesn’t have a disclaimer, and will be taken in the way they say they don’t want. Rather than leaving it ambiguous, they could change the title to remove the ambiguity.

                                                                                                                                    1. 28

                                                                                                                                      This side-steps the issue “people don’t read past the title” - which is the real problem. It’s bad, and unrealistic, to expect every person who writes on the internet to tailor each individual sub-part of an article (including the title) to an audience that is refusing to read the (whole) article itself before making judgements or assumptions. That’s purely the fault of the audience, not the writer, and changing the article title is merely addressing the symptoms, instead of the root problem, which will allow it to just become worse.

                                                                                                                                      We have an expression “judging a book by its cover” specifically for this scenario, and in addition to that, any reasonably-thoughtful reader is aware that, once a sufficient amount of context has been stripped away, any statement loses its meaning - which logically implies that full context must be absorbed before the reader has any chance of understanding the author’s intended meaning.

                                                                                                                                      People should not have to write hyper-defensively on the internet, or anywhere, because civil discussion collapses when your audience isn’t acting honestly.

                                                                                                                                      Reading the title and judging before reading the content itself is not acting honestly.

                                                                                                                                      1. 4

                                                                                                                                        Thank you for writing this. I’m sick of how much culture is warped by toxic social media sites.

                                                                                                                                      2. 7

                                                                                                                                        Agreed. It’s a poor choice for a title, even with a disclaimer. In fact, that he knew he needed a disclaimer should have been a big clue that it wasn’t a good title.

                                                                                                                                        A better title would be: “Professional devs should love Rust”, or “Rust isn’t just for enthusiasts and hipsters”

                                                                                                                                        1. 0

                                                                                                                                          I don’t disagree.

                                                                                                                                      3. 18

                                                                                                                                        Responses like this make me hesitant to try Rust, because they make me feel that I’d have to tiptoe around every word I put into a chat room, issue tracker, or mailing list.

                                                                                                                                        1. 10

                                                                                                                                          I’m not sure this is a Rust issue as much as a general issue of recent years. Be delicate and excessively sensitive about everything because everyone’s an egg shell. It’s contrary to anti-fragility which I would suggest is a far better approach; and the opposite approach. “Rust is for professionals” would have had me targeting Rust so that I could level up, not scare me away. Show me a challenge,

                                                                                                                                        2. 5

                                                                                                                                          I used rust for three years outside of work, but I’m afraid I stopped last year because I find it too big for a hobbyist. If I used it all day every day - no problem. It’s a great language, but I agree with the title.

                                                                                                                                          1. 2

                                                                                                                                            This was my first thought, as well, before even reading the article. Obviously I should read before judging, but the author may want to consider that the title could turn away potential readers.

                                                                                                                                            I made the same mistake in my own My staff engineering reading list. There was no reason for me to needlessly exclude people earlier in their careers from my list.

                                                                                                                                            And it’s good feedback that the author of this piece may not want to unintentionally exclude people from their article.

                                                                                                                                          1. 29

                                                                                                                                            I love Postgresql, and I’m really grateful for posts like this to balance my opinion and present a well-argumented counter-argument. However, when I read posts like this I mention a pattern: all these downsides seem to be relevant to really huge, perfomance-sensitive projects with enormous loads. The kind of projects where you likely have over a hundred of developers, and probably should move away from one master relational database to micro-services and message queues as a backbone of your architecture.

                                                                                                                                            What I’m saying, this kind of criticism just highlights the fact that Postgresql is probably the best choice if you’re smaller than that, both in terms of load and team size.

                                                                                                                                            1. 19

                                                                                                                                              I almost hit the XID wraparound (I was 2-3 days from an outage; it was bad enough that AWS emailed me) when my company only had 4-5 devs and 100 customers. And I’ve hit connections-related performance issues at least four or five times through the last five years, at relatively modest scale (<100 application servers, in an overwhelmingly read-heavy workload). This affected us as recently as yesterday, as we are bringing a Timescale DB (which is a Postgres extension) into production and we are faced with the prospect of tiered pgBouncers or breaking the Timescale access into its own microservice.

                                                                                                                                              I love Postgres but these are real and they can hit earlier than you expect.

                                                                                                                                              1. 4

                                                                                                                                                I love Postgres, and I was strongly considering moving to it…but for our use case, it simply requires too much care and feeding. We ship databases to customers in appliance form, meaning all the maintenance has to be invisible and automatic and Postgres simply isn’t there.

                                                                                                                                                1. 6

                                                                                                                                                  Having worked for a company that did that and been responsible for Postgres tuning, I say it can be done. In nearly 15 years of shipping out a postgresql db as part of an appliance, I have not seen any of these problems.

                                                                                                                                                  Edit: Except for long upgrade times. That one is a PITA.

                                                                                                                                                  JADP.

                                                                                                                                                  1. 4

                                                                                                                                                    I’d love to hear about your experience if you have time.

                                                                                                                                                    Also, I’m drawing a blank on “JADP”…

                                                                                                                                                    1. 3

                                                                                                                                                      Just Another Data Point?

                                                                                                                                                      1. 1

                                                                                                                                                        Just A Data Point. A weird acronym I picked up from old-timey online fora like USENET, and The Well.

                                                                                                                                                        I probably can’t say too much more about my experience postgres tuning as.

                                                                                                                                                        1. It was for a company, and might be considered propietary information.
                                                                                                                                                        2. It was about 5 years ago and I really don’t recall all that well what I did.

                                                                                                                                                        sorry, just know that these are really rare problems if you’re dealing with the limited scale inherent in incorporating postgresql as part of an appliance. Most of them deal with syndication, or ginormous tables. They’re web-scale problems, not appliance scale problems

                                                                                                                                                    2. 2

                                                                                                                                                      what do you use instead?

                                                                                                                                                      1. 1

                                                                                                                                                        What are you planning on using instead?

                                                                                                                                                        1. 1

                                                                                                                                                          I do this as well. There’s definitely a discovery period but I’ve reached the point that I almost never have to check anything on the database side for roughly 200 customers, running varying versions from 9.6 to 11.9.

                                                                                                                                                        2. 4

                                                                                                                                                          Definitely echo that these problems (and others) can hit you way before you get to 100 devs. We were running into the pains mentioned in this article (which admittedly is a more general crritique of SQL databases, and lands on MySQL over Postgres) at more like 10 developers.

                                                                                                                                                          It absolutely isn’t that hard to run into the pitfalls of SQL databases at relatively small scale, especially if you’re using them for OLTP workloads in services where uptime/response times matter.

                                                                                                                                                        3. 5

                                                                                                                                                          all these downsides seem to be relevant to really huge, perfomance-sensitive projects with enormous loads. The kind of projects where you likely have over a hundred of developers

                                                                                                                                                          A number of these issues affect “normal” users of PostgreSQL as well:

                                                                                                                                                          • Replication is something you may want even on smaller use cases.

                                                                                                                                                          • “Wasted” space from the “update is really delete + insert”-paradigm can be a problem even on fairly small use cases (i.e. tens of millions of rows). It can make some simple operations rather painful.

                                                                                                                                                          • Lack of query hints is pretty annoying, especially for smaller users who don’t have a dedicated DBA with a Ph.D. in the PostgreSQL query planner. It’s also a massive help in development; want to try a new index? Now it’s a drop index, create index, wait, analyse`, wait some more, run some queries, discover that didn’t do what you expected, drop the index, create a different one, wait, etc. It’s very time-consuming and much of the time is spent waiting.

                                                                                                                                                          1. 5

                                                                                                                                                            Nice example: “complaining” that it is hard to tune it for a million concurrent connections.

                                                                                                                                                            Haven’t read it to the end yet, almost hoping to see an ending like “of course I’m happy to have a free DB that gets me in trouble for a million concurrent connections instead of breaking my bank at 1000 connections or when somebody touches advanced debugging like Oracle or

                                                                                                                                                            1. 5

                                                                                                                                                              FYI you should have read it through to the end, as the article does end on that note.

                                                                                                                                                          1. 12

                                                                                                                                                            If there’s a path from Aphyr fanfic to better databases, I’m all for it.

                                                                                                                                                            1. 21

                                                                                                                                                              This mindset of replacing a language to remove a class of errors is naive at best.

                                                                                                                                                              I hate null with a passion and I do think Rust memory safety is a valuable feature. But lets take the biggest problem of that class as an example, the heartbleed bug. If you look at the vulnerability code, it is a very basic mistake. If you took an introductory course in C, you would learn how not to do that.

                                                                                                                                                              To argue that it was just a matter of using a language that doesn’t allow for that kind of error is the solution is to defend an impossible solution. Without doubting the good intentions of whomever wrote that piece of code, let us call a spade a spade, it was objectively poor code with basic flaws.

                                                                                                                                                              You don’t solve bad engineering by throwing a hack at it such as changing the language. It will manifest itself in the form of other classes of bugs and there is no evidence whatsoever that the outcome isn’t actually worse than the problem one is trying to fix.

                                                                                                                                                              Java doesn’t allow one to reference data by its memory address, precisely to avoid this whole class of problems, why isn’t everyone raving about how that magically solved all problems? The answer is: because it obviously didn’t.

                                                                                                                                                              I love curl and use it intensively, but this post goes down that whole mindset. Running scripts to find bugs and so on.

                                                                                                                                                              1. 67

                                                                                                                                                                I’m not convinced by this argument. Large C and C++ projects seem to always have loads of memory vulns. Either they’re not caused by bad programming or bad programming is inevitable.

                                                                                                                                                                I think the core question of whether memory unsafe languages result in more vulnerable code can probably be answered with data. The only review I’m aware of is this fairly short one by a Rust contributor, but there are probably others: https://alexgaynor.net/2020/may/27/science-on-memory-unsafety-and-security/

                                                                                                                                                                1. 17

                                                                                                                                                                  Good article, the writer sums it up brilliant:

                                                                                                                                                                  Until you have the evidence, don’t bother with hypothetical notions that someone can write 10 million lines of C without ubiquitious memory-unsafety vulnerabilities – it’s just Flat Earth Theory for software engineers.

                                                                                                                                                                  1. 14

                                                                                                                                                                    There should be a corollary: until you have the evidence, don’t bother with hypothetical notions that rewriting 10 million lines of C in another language would fix more bugs than it introduces.

                                                                                                                                                                    1. 9

                                                                                                                                                                      Agreed. But nuance is deserved on “both sides” of the argument.

                                                                                                                                                                      It’s fair to say that rewriting 10 million lines of C in a memory safe language will, in fact, fix more memory bugs than it introduces (because it fix them all and wont introduce any).

                                                                                                                                                                      It’s also fair to acknowledge that memory bugs are not the only security bugs and that security bugs aren’t the only important bugs.

                                                                                                                                                                      It’s not fair to say that it’s literally impossible for a C program to ever be totally secure.

                                                                                                                                                                      My tentative conclusion is this: If your C program is not nominally related to security, itself, then it very likely will become more secure by rewriting in Rust/Zig/Go/whatever. In other words, if there are no crypto or security algorithms implemented in your project, then the only real source of security issues is from C, itself (or your dependencies, of course).

                                                                                                                                                                      If you C program is related to security in purpose, as in sudo, a crypto library, password manager, etc, then the answer is a lot less clear. Many venerable C projects have the advantage of time- they’ve been around forever and have lots of battle testing. It’s likely that if they stay stable and don’t have a lot of code churn that they wont introduce many new security bugs over time.

                                                                                                                                                                      1. 1

                                                                                                                                                                        if there are no crypto or security algorithms implemented in your project, then the only real source of security issues is from C, itself

                                                                                                                                                                        I don’t think this is true. All sorts of programs accept untrusted input, not just crypto or security projects, and almost any code that handles untrusted input will have all sorts of opportunities to be unsafe, regardless of implementation language.

                                                                                                                                                                        1. 1

                                                                                                                                                                          Theoretically yes. But, in practice, if you’re not just passing a user-provided query string into your database, it’s much, MUCH, harder for bad input to pose a security threat. What’s the worst they can do- type such a long string that you OOM? I can be pretty confident that no matter what they type, it’s not going to start writing to arbitrary parts of my process’s memory.

                                                                                                                                                                          1. 1

                                                                                                                                                                            It’s not just databases, tho, it’s any templating or code generation that uses untrusted input.

                                                                                                                                                                            Do you generate printf format strings, filesystem paths, URLs, HTML, db queries, shell commands, markdown, yaml, config files, etc? If so, you can have escaping issues.

                                                                                                                                                                            And then there are problems specific to memory unsafety: buffer overturns let you write arbitrary instructions to process memory, etc.

                                                                                                                                                                            1. 1

                                                                                                                                                                              Did you forget that my original comment was specifically claiming that you should not use C because of buffer overruns? So that’s not a counter-point to my comment at all- it’s an argument for it.

                                                                                                                                                                              My overall assertion was that if you’re writing a program in C, it will almost definitely become more secure if you rewrote it in a memory-safe language, with the exception of programs that are about security things- those programs might already have hard-won wisdom that you’d be giving up in a rewrite, so the trade-off is less clear.

                                                                                                                                                                              I made a remark that if your C program doesn’t, itself, do “security stuff”, that the only security issues will be from the choice of C. That’s not really correct, as you pointed out- you can surely do something very stupid like passing a user-provided query right to your database, or connect to a user-provided URL, or whatever.

                                                                                                                                                                              But if that’s the bar we’re setting, then that program definitely has no business being written in C (really at all, but still). There’s certainly no way it’s going to become less secure with a rewrite in a memory-safe language.

                                                                                                                                                                2. 63

                                                                                                                                                                  Your argument is essentially a form of “victim shaming” where we slap the programmers and tell them to be better and more careful engineers next time.

                                                                                                                                                                  It is an escapism that stiffles progress by conveniently opting to blame the person making the mistake, rather than the surrounding tools and environment that either enabled, or failed to prevent the error.

                                                                                                                                                                  It can be applied to all sorts of other contexts including things such as car safety. You could stop making cars safer and just blame the drivers for not paying more attention, going too fast, drink driving, etc…

                                                                                                                                                                  If we can improve our tools of the trade to reduce or - better yet - eliminate the possibility of mistakes and errors we should do it. If it takes another whole language to do it then so be it.

                                                                                                                                                                  That’s similar to a car manufacturer using a different engine or chassis because somehow it reduces the accidents because of the properties that it has.

                                                                                                                                                                  The way we can make that progress is exactly by blaming our “tools” as the “mistake enablers”. Not the person using the tools. Usually they’ve done their best in good faith to avoid a mistake. If they have still made one, that’s an opportunity for improvement of our tools.

                                                                                                                                                                  1. 38

                                                                                                                                                                    Your argument is essentially “you can’t prevent bad engineering or silly programmer errors with technical means; this is a human problem that should be fixed at the human level”. I think this is the wrong way to look at it.

                                                                                                                                                                    I think it’s all about the programmer’s mental bandwidth; humans are wonderful, intricate, and beautiful biological machines. But in spite of this we’re also pretty flawed and error-prone. Ask someone to do the exact same non-trivial thing every Wednesday afternoon for a year and chances are a large amount of them will fail at least once to follow the instructions exactly. Usually this is okay because most things in life have fairly comfortable error margins and the consequences of failure are non-existent or very small, but for some things it’s a bit different.

                                                                                                                                                                    This is why checklists are used extensively in aviation; it’s not because the pilots are dumb or inexperienced, it’s because it’s just so damn easy to forget something when dealing with these complex systems, and the margin for error is fairly low if you’re 2km up in the sky and the consequences can be very severe.

                                                                                                                                                                    C imposes fairly high mental bandwidth: there are a lot of things you need to do “the right way” or you run in to problems. I don’t think anyone is immune to forgetting something on occasion; who knows what happened with the Heartbleed thing; perhaps the programmer got distracted for a few seconds because the cat jumped on the desk, or maybe their spouse asked what they wanted for dinner tonight, or maybe they were in a bad mood that day, or maybe they … just forgot.

                                                                                                                                                                    Very few people are in top form all day, every day. And if you write code every day then sooner or later you will make a mistake. Maybe it’s only once every five years, but if you’re working on something like OpenSSL the “make a silly mistake once every five years” won’t really cut it, just as it won’t for pilots.

                                                                                                                                                                    The code is now finished and moves on to the reviewer(s); and the more they need to keep in mind when checking the code the more chance there is they may miss something. Reviewing code is something I already find quite hard even with “easy” languages: how can I be sure that it’s “correct”? Doing a proper review takes almost as much time as writing the code itself (or longer!) The more you need to review/check for every line of code, the bigger the chance is that you’ll miss a mistake like this.


                                                                                                                                                                    I don’t think that memory safety is some sort of panacea, or that it’s a fix for sloppy programming. But it makes it frees up mental bandwidth and mistakes will be harder and their consequences less severe. It’s just one thing you don’t have to think about, and now you have more space to think about other aspects of the program (including security problems not related to memory safety).

                                                                                                                                                                    @x64k mentioned PHP in another reply, and this suffers from the same problem; I’ve seen critical security fixes which consist of changing in_array($list, $item) to in_array($list, $item, true). That last parameters enable strict type checking (so that "1" == 1 is false). The root cause of these issues is the same as in C: it imposes too much bandwidth to get it right, every time, all the time.

                                                                                                                                                                    NULLs have the same issue: you need to think “can this be NULL?” every time. It’s not a hard question, but it’s sooner or later you’ll get it wrong and asking it all the time takes up a lot of bandwidth probably best spent elsewhere.

                                                                                                                                                                    1. 30

                                                                                                                                                                      Java does magically solve all memory problems. People did rave about garbage collection: garbage collection is in fact revolutionary.

                                                                                                                                                                      1. 5

                                                                                                                                                                        That was a long time ago so lots of people don’t remember what OP is talking about anymore. The claim wasn’t that Java would magically solve all memory problems. That was back when the whole “scripting vs. systems” language dichotomy was all the rage and everyone thought everything would be written in TCL, Scheme or whatever in ten years or so. There was a more or less general expectation (read: lots of marketing material, since Java was commercially-backed, but certainly no shortage of independent tech evangelists) that, without pointers, all problems would go away – no more security issues, no more crashes and so on.

                                                                                                                                                                        Unsurprisingly, neither of those happened, and Java software turned out to be crash-prone in its own unpleasant ways (there was a joke about how you can close a Java program if you can’t find the quit button: wiggle the mouse around, it’ll eventually throw an unhandled exception) in addition to good ol’ language-agnostic programmer error.

                                                                                                                                                                      2. 29

                                                                                                                                                                        If you took an introductory course in C, you would learn how not to do that.

                                                                                                                                                                        Yet somehow the cURL person/people made the mistake. Things slip by.

                                                                                                                                                                        Java doesn’t allow one to reference data by its memory address, precisely to avoid this whole class of problems, why isn’t everyone raving about how that magically solved all problems? The answer is: because it obviously didn’t.

                                                                                                                                                                        That, actually, was one of the the biggest selling points of Java to C++ devs. It’s probably the biggest reason that Java is still such a dominant language today.

                                                                                                                                                                        I also take issue with your whole message. You say that you can’t fix bad engineering by throwing a new language at it. But that’s an over generalization of the arguments being made. You literally can fix bad memory engineering by using a language that doesn’t allow it, whether that’s Java or Rust. In the meantime, you offer no solution other than “don’t do this thing that history has shown is effectively unavoidable in any sufficiently large and long-lived C program”. So what do you suggest instead? Or are we just going to wait for heartbleed 2.0 and act surprised that it happened yet again in a C program?

                                                                                                                                                                        Further, you throw out a complaint that we can’t prove that rewriting in Rust (or whatever) won’t make things worse than they currently are. We live in the real world- you can’t prove lots of things, but is there any reason to actually suspect that this is realistically possible?

                                                                                                                                                                        1. 27

                                                                                                                                                                          This mindset of replacing a language to remove a class of errors is naive at best.

                                                                                                                                                                          I’d rather say that your post is, charitably, naive at best (and it continuing to dominate the conversation is an unfortunate consequence of the removal of the ability to flag down egregiously incorrect posts, sadly).

                                                                                                                                                                          I hate null with a passion and I do think Rust memory safety is a valuable feature. But lets take the biggest problem of that class as an example, the heartbleed bug. If you look at the vulnerability code, it is a very basic mistake. If you took an introductory course in C, you would learn how not to do that.

                                                                                                                                                                          Do you really believe that the OpenSSL programmers (whatever else you can say about that project) lack an introductory knowledge of C? Do you feel the linux kernel devs, who have made identical mistakes, similarly lack an introductory knowledge of C? Nginx devs? Apache? Etc, etc.

                                                                                                                                                                          This is an extraordinary claim.

                                                                                                                                                                          You don’t solve bad engineering by throwing a hack at it such as changing the language.

                                                                                                                                                                          Probably one of the most successful fields in history at profoundly reducing, and keeping low, error rates has been the Aviation industry, and the lesson of their success is that you don’t solve human errors by insisting that the people who made the errors would have known not to if they’d just taken an introductory course they’d already long covered, or in general just be more perfect.

                                                                                                                                                                          The Aviation industry realized that humans, no matter how well tutored and disciplined and focused, inevitably will still make mistakes, and that the only thing that reduces errors is looking at the mistakes that are made and then changing the system to account for those mistakes and reduce or eliminate their ability to recur.

                                                                                                                                                                          When your dogma leads to you making extraordinary (indeed, ludicrous) claims, and the historical evidence points the polar opposite of the attitude you’re preaching being the successful approach, it’s past time to start reconsidering your premise.

                                                                                                                                                                          1. 13

                                                                                                                                                                            The Aviation industry realized that humans, no matter how well tutored and disciplined and focused, inevitably will still make mistakes, and that the only thing that reduces errors is looking at the mistakes that are made and then changing the system to account for those mistakes and reduce or eliminate their ability to recur.

                                                                                                                                                                            I’d like to stress that this is only one part of Aviation’s approach, at least as driven by the FAA and the NTSB in the US. The FAA also strives to create a culture of safety, by mandating investigations into incidents, requiring regular medical checkups depending on your pilot rating, releasing accident findings often, incentivizing record-keeping on both aircraft (maintenance books) and pilots (logbooks), encouraging pilots to share anonymous information on incidents that occurred with minor aircraft or no passenger impact, and many more. This isn’t as simple as tweaking the system. It’s about prioritizing safety at every step of the conversation.

                                                                                                                                                                            1. 7

                                                                                                                                                                              A fair point. And of course all of this flies directly in the face of just yelling “be more perfect at using the deadly tools!” at people.

                                                                                                                                                                              1. 4

                                                                                                                                                                                Yup, I meant this more to demonstrate what it takes to increase safety in an organized endeavor.

                                                                                                                                                                              2. 1

                                                                                                                                                                                Dropping the discussion of “the problem is human nature” in this comment. I’m explicitly not rhetorically commenting on it or implying such.

                                                                                                                                                                                These “other parts”, and culture of safety - how would we translate that across into programming? Actually, come to think of it that’s probably not the first question. The first question is, is it possible to translate that across into programming?

                                                                                                                                                                                I think it’s fair to say that in e.g. webdev people flat-out just value developer velocity over aerospace levels of safety because (I presume) faster development is simply more valuable in webdev than it is in aerospace - if the thing crashes every tuesday you’ll lose money, but you won’t lose that much money. So, maybe it’s impractical to construct such a culture. Maybe. I don’t know.

                                                                                                                                                                                But, supposing it is practical, what are we talking about? Record-keeping sounds like encouraging people to blog about minor accidents, I guess? But people posting blogs is useless if you don’t have some social structure for discussing the stuff, and I’m not sure what the analogous social structure would be here.

                                                                                                                                                                                “Prioritizing safety at every step of the conversation” sounds like being able to say no to your boss without worry.

                                                                                                                                                                                “This isn’t as simple as tweaking the system” sounds like you’re saying “treat this seriously and stop chronicly underserving it both financially and politically”, which sounds to me like “aim for the high-hanging fruit of potential problems”, which I don’t think anyone with the word “monetize” job description will ever remotely consider.

                                                                                                                                                                                What are the low-hanging fruit options in this “stop excessively focusing on low-hanging fruit options” mindset you speak of?

                                                                                                                                                                                Actually, it sounds like that sort of thing would need sort of government intervention in IT security or massive consumer backlash. Or more likely both, with the latter causing the former.

                                                                                                                                                                                1. 1

                                                                                                                                                                                  The first question is, is it possible to translate that across into programming?

                                                                                                                                                                                  It most certainly is. The “easiest” place to see evidence of this is to look into fields of high-reliability computing. Computing for power plants, aviation, medical devices, or space are all good examples. A step down would be cloud providers that do their best to provide high availability guarantees. These providers also spend a lot of engineering effort + processes in emphasizing reliability.

                                                                                                                                                                                  But, supposing it is practical, what are we talking about? Record-keeping sounds like encouraging people to blog about minor accidents, I guess? But people posting blogs is useless if you don’t have some social structure for discussing the stuff, and I’m not sure what the analogous social structure would be here.

                                                                                                                                                                                  Thing of the postmortem process posted on the blogs of the big cloud providers. This is a lot like the accident reports that the NTSB releases after accident investigation. I think outside of the context of a single entity coding for a unified goal (whether that’s an affiliation of friends, a co-op, or a company), it’s tough to create a “culture” of any sort, because in different contexts of computing, different tradeoffs are desired. After all, I doubt you need a high reliability process to write a simple script.

                                                                                                                                                                                  “This isn’t as simple as tweaking the system” sounds like you’re saying “treat this seriously and stop chronicly underserving it both financially and politically”, which sounds to me like “aim for the high-hanging fruit of potential problems”, which I don’t think anyone with the word “monetize” job description will ever remotely consider.

                                                                                                                                                                                  You’d be surprised how many organizations, both monetizing and not, have this issue. Processes become ossified; change is hard. Aiming for high-hanging fruit is expensive. But a mix of long-term thinking and short-term thinking is always the key to making good decisions, and in computing it’s no different. You have to push for change if you’re pushing against a current trend of unsafety.

                                                                                                                                                                                  What are the low-hanging fruit options in this “stop excessively focusing on low-hanging fruit options” mindset you speak of?

                                                                                                                                                                                  There needs to be a feedback mechanism between failure of the system and engineers creating the system. Once that feedback is in place, safety can be prioritized over time. Or at least, this is one way I’ve seen this done. There are probably many paths out there.

                                                                                                                                                                                  I think it’s fair to say that in e.g. webdev people flat-out just value developer velocity over aerospace levels of safety because (I presume) faster development is simply more valuable in webdev than it is in aerospace - if the thing crashes every tuesday you’ll lose money, but you won’t lose that much money. So, maybe it’s impractical to construct such a culture. Maybe. I don’t know.

                                                                                                                                                                                  This here is the core problem. Honestly, there’s no reason to hold most software to a very high standard. If you’re writing code to scrape the weather from time to time from some online API and push it to a billboard, meh. What software needs to do is get a lot better about prioritizing safety in the applications that require it (and yes, that will require some debate in the community to come up with applications that require this safety, and yes there will probably be different schools of thought as there always are). I feel that security is a minimum, but beyond that, it’s all application specific. Perhaps the thing software needs the most now is just pedagogy on operating and coding with safety in mind.

                                                                                                                                                                                  1. 1

                                                                                                                                                                                    A step down would be cloud providers that do their best to provide high availability guarantees. These providers also spend a lot of engineering effort + processes in emphasizing reliability.

                                                                                                                                                                                    Google’s SRE program and the SRE book being published for free are poster examples of promoting a culture of software reliability.

                                                                                                                                                                            2. 18

                                                                                                                                                                              You don’t solve bad engineering by throwing a hack at it such as changing the language.

                                                                                                                                                                              Yes, you absolutely do. One thing you can rely on is that humans will make mistakes. Even if they are the best, even if you pay them the most, even if you ride their ass 24 hours a day. Languages that make certain kinds of common mistakes uncommon or impossible save us from ourselves. All other things being equal, you’d be a fool not to choose a safer language.

                                                                                                                                                                              1. 8

                                                                                                                                                                                I wrote a crypto library in C. It’s small, only 2K lines of code. I’ve been very diligent every step of the way (save one, which I paid dearly). I reviewed the code several times over. There was even an external audit. And very recently, I’ve basically fixed dead code. Copying a whopping 1KB, allocating and wiping a whole buffer, wasting lines of code, for no benefit whatsoever. Objectively a poor piece of code with a basic flaw.

                                                                                                                                                                                I’m very careful with my library, and overall I’m very proud of its overall quality; but sometimes I’m just tired.

                                                                                                                                                                                (As for why it wasn’t noticed: as bad as it was, the old code was correct, so it didn’t trigger any error.)

                                                                                                                                                                                1. 7

                                                                                                                                                                                  All programmers are bad programmers then, otherwise why do we need compiler error messages?

                                                                                                                                                                                  Software apparently can’t just be written correctly the first time.

                                                                                                                                                                                  1. 15

                                                                                                                                                                                    I’m half joking here but, indeed, if language-level memory safety were all it takes for secure software to happen, we could have been saved ages ago. We didn’t have to wait for Go, or Rust, or Zig to pop up. A memory-safe language with no null pointers, where buffer overflow, double-frees and use-after-free bugs are impossible, has been available for more than 20 years now, and the security track record of applications written in that language is a very useful lesson. That language is PHP.

                                                                                                                                                                                    I’m not arguing that (re)writing curl in Go or Rust wouldn’t eventually lead to a program with fewer vulnerabilities in this particular class, I’m just arguing that “this program is written in C and therefore not trustworthy because C is an unsafe language” is, at best, silly. PHP 4 was safer than Rust and boy do I not want to go back to dealing with PHP 4 applications.

                                                                                                                                                                                    Now of course one may argue that, just like half of curl’s vulnerabilities are C mistakes, half of those vulnerabilities were PHP 4 mistakes. But in that case, it seems a little unwise to wager that, ten years from now, we won’t have any “half of X vulnerabilities are Rust mistakes” blog posts…

                                                                                                                                                                                    1. 13

                                                                                                                                                                                      Language-level anything isn’t all it takes, but from my experience they do help and they help much more than “a little”, and… I’ll split this in two.

                                                                                                                                                                                      The thing I’ve done that found the largest number of bugs ever was when I once wrote a script to look for methods (in a >100kloc code base) that that three properties: a) Each method accepted at least one pointer parameter b) contained null in the code and c) did not mention null in the documentation for that method. Did that find all null-related errors? Far from it, and there were several false positives for each bug, and many of the bugs weren’t serious, but I used the output to fix many bugs in just a couple of days.

                                                                                                                                                                                      Did this fix all bugs related to null pointers? No, not even nearly. Could I have found and fixed them in other ways? Yes, I could. The other ways would have been slower, though. The script (or let’s call it a query) augmented my capability, in much the same way as many modern techniques augment programmers.

                                                                                                                                                                                      And this brings me to the second part.

                                                                                                                                                                                      We have many techniques that do do nothing capable programmers can’t do. (I’ve written assembly language without any written specification, other documentation, unit tests or dedicated testers, and the code ran in production and worked. It can be done.)

                                                                                                                                                                                      That doesn’t mean that these techniques are superfluous. Capable programmers are short of time and attention; techniques that use CPU cycles, RAM and files, and that save brain time are generally a net gain.

                                                                                                                                                                                      That includes safe languages, but also things like linting, code queries, unit tests, writing documentation and fuzzing (or other white-noise tests). I’d say it also includes code review, which can be described as using other ream members’ attention to reduce the total attention needed to deliver features/fix bugs.

                                                                                                                                                                                      Saying “this program is safe because it has been fuzzed” or “because it uses unit tests” doersn’t make sense. But “this program is unsafe because it does not use anything more than programmer brains” makes sense and is at least a reasonable starting assumption.

                                                                                                                                                                                      (The example I used above was a code query. A customer reported a bug, I hacked together a code query to find similar possible trouble spots, and found many. select functions from code where …)

                                                                                                                                                                                      1. 2

                                                                                                                                                                                        PHP – like Go, Rust and many others out there – also doesn’t use anything more than programmer brains to avoid off-by-one errors, for example, which is one of the most common causes of bugs with or without security implications. Yet nobody rushes to claim that programs written in one of these languages are inherently unsafe because they rely on nothing but programmer brains to find such bugs.

                                                                                                                                                                                        As I mentioned above: I’m not saying these things don’t matter, of course they do. But conflating memory safety with software security or reliability is a bad idea. There’s tons of memory-safe code out there that has so many CVEs it’s not even funny.

                                                                                                                                                                                        1. 17

                                                                                                                                                                                          But conflating memory safety with software security or reliability is a bad idea. There’s tons of memory-safe code out there that has so many CVEs it’s not even funny.

                                                                                                                                                                                          Who is doing this? The title of the OP is explicitly not conflating memory safety with software security. Like, can you find anyone with any kind of credibility conflating these things? Are there actually credible people saying, “curl would not have any CVEs if it were written in a memory safe language”?

                                                                                                                                                                                          It is absolutely amazing to me how often this straw man comes up.

                                                                                                                                                                                          EDIT: I use the word “credible” because you can probably find a person somewhere on the Internet making comments that support almost any kind of position, no matter how ridiculous. So “credible” in this context might mean, “an author or maintainer of software the other people actually use.” Or similarish. But I do mean “credible” in a broad sense. It doesn’t have to be some kind of authority. Basically, someone with some kind of stake in the game.

                                                                                                                                                                                          1. 7

                                                                                                                                                                                            Just a few days ago there was a story on the lobster.rs front page whose author’s chief complaint about Linux was that its security was “not ideal”, the first reason for that being that “Linux is written in C, [which] makes security bugs rather common and, more importantly, means that a bug in one part of the code can impact any other part of the code. Nothing is secure unless everything is secure.” (Edit: which, to be clear, was in specific contrast to some Wayland compositor being written in Rust).

                                                                                                                                                                                            Yeah, I’m tired of it, too. I like and use Rust but I really dislike the “evangelism taskforce” aspect of its community.

                                                                                                                                                                                            1. 9

                                                                                                                                                                                              I suppose “nothing is secure unless everything is secure” is probably conflating things. But saying that C makes security bugs more common doesn’t necessarily. In any case, is this person credible? Are they writing software that other people use?

                                                                                                                                                                                              I guess I just don’t understand why people spend so much time attacking a straw man. (Do you even agree that it is a straw man?) If someone made this conflation in a Rust space, for example, folks would be very quick to correct them that Rust doesn’t solve all security problems. Rust’s thesis is that it reduces them. Sometimes people get confused either because they don’t understand or because none are so enthusiastic as the newly converted. But I can’t remember anyone with credibility making this conflation.

                                                                                                                                                                                              Like, sure, if you see someone conflating memory safety with all types of security vulnerabilities, then absolutely point it out. But I don’t think it makes sense to talk about that conflation as a general phenomenon that is driving any sort of action. Instead, what’s driving that action is the thesis that many security vulnerabilities are indeed related to memory safety problems, and that using tools which reduce those problems in turn can eventually lead to more secure software. While some people disagree with that, it takes a much more nuance argument and it sounds a lot less ridiculous than the straw man you’re tearing down.

                                                                                                                                                                                              Yeah, I’m tired of it, too. I like and use Rust but I really dislike the “evangelism taskforce” aspect of its community.

                                                                                                                                                                                              I’m more tired of people complaining about the “evangelism taskforce.” I see a lot more of that than I do the RESF.

                                                                                                                                                                                              1. 7

                                                                                                                                                                                                Sorry, I think I should have made the context more obvious. I mean, let me start with this one, because I’d also like to clarify that a) I think Rust is good and b) that, as far as this particular debate is concerned, I think writing new things in Rust rather than C or especially C++ is a good idea in almost every case:

                                                                                                                                                                                                (Do you even agree that it is a straw man?)

                                                                                                                                                                                                What, that experienced software developers who know and understand Rust are effectively claiming that Rust is magic security/reliability dust? Oh yeah, I absolutely agree that it’s bollocks, I’ve seen very few people who know Rust and have more than a few years of real-life development experience in a commercial setting making that claim with a straight face. There are exceptions but that’s true of every technology.

                                                                                                                                                                                                But when it comes to the strike force part, here’s the thing:

                                                                                                                                                                                                If someone made this conflation in a Rust space, for example, folks would be very quick to correct them that Rust doesn’t solve all security problems.

                                                                                                                                                                                                …on the other hand, for a few years now it feels like outside Rust spaces, you can barely mention an OS kernel or a linker or a window manager or (just from a few days ago!) a sound daemon without someone showing up saying ugh, C, yeah, this is completely insecure, I wouldn’t touch it with a ten-foot pole. Most of the time it’s at least plausible, but sometimes it’s outright ridiculous – you see the “not written in Rust” complaint stuck on software that has to run on platforms Rust doesn’t even support, or that was started ten years ago and so on.

                                                                                                                                                                                                Most of them aren’t credible by your own standards or mine, of course, but they’re part of the Rust community whether they’re representative of the “authoritative” claims made by the Rust developers or not.

                                                                                                                                                                                                1. 4

                                                                                                                                                                                                  Fair enough. Thanks for the reply!

                                                                                                                                                                                                  1. 4

                                                                                                                                                                                                    …on the other hand, for a few years now it feels like outside Rust spaces, you can barely mention an OS kernel or a linker or a window manager or (just from a few days ago!) a sound daemon without someone showing up saying ugh, C, yeah, this is completely insecure, I wouldn’t touch it with a ten-foot pole. Most of the time it’s at least plausible, but sometimes it’s outright ridiculous – you see the “not written in Rust” complaint stuck on software that has to run on platforms Rust doesn’t even support, or that was started ten years ago and so on.

                                                                                                                                                                                                    As I mentioned in my below comment on this article, this is a good thing. I want people who decide to write a novel sound daemon in C to see those sorts of comments, and (ideally) rethink the decision to write a novel C program to begin with. Again, this doesn’t necessarily imply that Rust is the right choice of language for any given project, but it’s a strong contender right now.

                                                                                                                                                                                                    1. 4

                                                                                                                                                                                                      Even now though, there still is significant tension between “don’t use C” and “make it portable”. Especially if you’re targetting embedded, or unknown platforms. C is still king of the hill as far as portability goes.

                                                                                                                                                                                                      What we really want is dethrone C at its own game: make something that eventually becomes even more portable. That’s possible: we could target C as a backend, and we could formally specify the language so it’s clear what’s a compiler bug (not to mention the possibility of writing formally verified compilers). Rust isn’t there yet.

                                                                                                                                                                                                      1. 6

                                                                                                                                                                                                        One of the (many) reasons I don’t use C and use Rust instead is because it’s easier to write portable programs. I believe every Rust program I’ve written also works on Windows, and that has nearly come for free. Certainly a lot cheaper than if I had written it in C. I suppose people use “portable” to mean different things, but without qualification, your dichotomy doesn’t actually seem like a dichotomy. I suppose the dichotomy is more, “don’t use C” and “make it portable to niche platforms”?

                                                                                                                                                                                                        1. 3

                                                                                                                                                                                                          I think we (as in both me and the parent poster) were talking about different kinds of portability. One of the many reasons why most of the software I work on is (still) in C rather than Rust is that, while every Rust program I’ve written works on Windows, lots of the ones I need have to work on architectures that are, at best, Tier 2. Proposing that we ship something compiled with a toolchain that’s only “guaranteed to build” would at best get me laughed at.

                                                                                                                                                                                                          1. 9

                                                                                                                                                                                                            Yes. The point I’m making is that using the word “portable” unqualified is missing the fact that Rust lets you target one of the most popular platforms in the world at a considerably lower cost in lots of common cases. It makes the trade off being made more slanted than what folks probably intend by holding up “portability” as a ubiquitously good thing. Well, if we’re going to do that, we should acknowledge that there is a very large world beyond POSIX and embedded, and that world is primarily Windows.

                                                                                                                                                                                                            1. 5

                                                                                                                                                                                                              For the record, if I were writing desktop GUI applications or games, of course the only relevant platforms are Windows, Linux, and MacOSX. Or Android and iOS, if the application is meant for palmtops. From there “portability” just means I chose middleware that have backends for all the platforms I care about. Rust, with its expanded standard library, does have an edge.

                                                                                                                                                                                                              If however I’m writing a widely applicable library (like a crypto library), then Rust suddenly don’t look so good any more. Because I know for a fact that many people still work on platforms that Rust doesn’t support yet. Not to mention the build dependency on Rust itself. So either I still use C, and I have more reach, or I use Rust, and I have more safety (not by much if I test my C code correctly).

                                                                                                                                                                                                              Well, if we’re going to do that, we should acknowledge that there is a very large world beyond POSIX and embedded, and that world is primarily Windows.

                                                                                                                                                                                                              Of course, my C library is also going to target Windows. Not doing so would defeat the point.

                                                                                                                                                                                                              1. 6

                                                                                                                                                                                                                I don’t think I strongly disagree with anything here. It’s just when folks say things like this

                                                                                                                                                                                                                C is still king of the hill as far as portability goes.

                                                                                                                                                                                                                I would say, “welllll I’m not so sure about that, because I can reach a lot more people with less effort using Rust than I can with C.” Because if I use C, I now need to write my own compatibility layer between my application and the OS in order to support a particularly popular platform: Windows.

                                                                                                                                                                                                                And of course, this depends on your target audience, the problem you’re solving and oodles of other things, as you point out. But there’s a bit of nuance here because of how general the word “portable” is.

                                                                                                                                                                                                                1. 3

                                                                                                                                                                                                                  Yeah, I was really talking about I/O free libraries. I believe programs should be organised in 3 layers:

                                                                                                                                                                                                                  • At the bottom, you have I/O free libraries, that depend on nothing but the compiler and maybe the standard library. That lack of dependency can make them extremely portable, and easy to integrate to existing projects. The lack of I/O makes them easy to test, so they have the potential to be very reliable, even if they’re written in an unsafe language.
                                                                                                                                                                                                                  • In the middle, you have, the I/O compatibility layer. SDL, Qt, Libuv, Rust’s stdlib, even hand written, take your pick. That one cannot possibly be portable, because it has to depend on the quirks of the underlying platform. But it can have several backends, which make the users of this compatibility layer quite portable.
                                                                                                                                                                                                                  • At the top, you have the application, that depends on the I/O free library and the compatibility layer. It cannot target platforms the compatibility layers doesn’t target, but at least is should be fairly small (maybe 10 times smaller than the I/O free libraries?), so if a rewrite is needed it shouldn’t be that daunting.

                                                                                                                                                                                                                  I believe C is still a strong contender for the bottom layer. There specifically, it is still the king of portability. For the middleware and the top layer however, the portability of the language means almost nothing, so it’s much harder to defend using C there.

                                                                                                                                                                                                                  Also note that the I/O free libraries can easily be application specific, and not intended for wider distribution. In that case, C also loses its edge, as (i) portability matters much less, and (ii) it’s still easier to use a safe language than write a properly paranoid test suite.

                                                                                                                                                                                                              2. 0

                                                                                                                                                                                                                So does C#. Targeting popularity does not make you portable.

                                                                                                                                                                                                            2. 3

                                                                                                                                                                                                              I was talking about “runs on a 16-bit micro controller as well as an 64-bit monster”. The kind where you might not have any I/O, or even a heap allocator. The kind where you avoid undefined behaviour and unspecified behaviour and implementation defined behaviour.

                                                                                                                                                                                                              Hard, but possible for some programs. Crypto libraries (without the RNG) for instance are pure computation, and can conform to that highly restricted setting. I’ll even go a bit further: I think over 95% of programs can be pure computation, and be fully separated from the rest (I/O, system calls, networking and all that).

                                                                                                                                                                                                              If you want to print stuff on a terminal, portability drops. If you want to talk to the network or draw pixels on the screen, portability in C is flat out impossible, because the required capabilities aren’t in the standard library. I hear Rust fares far better in that department.

                                                                                                                                                                                                              I suppose the dichotomy is more, “don’t use C” and “make it portable to niche platforms”?

                                                                                                                                                                                                              Open BSD has tier 3 support for Rust, which basically means no support. You assess how “niche” OpenBSD really is, especially in a security context.

                                                                                                                                                                                                              1. 8

                                                                                                                                                                                                                Yes, I would absolutely say OpenBSD is a niche platform. I generally don’t care if the stuff I write works on OpenBSD. I just don’t. I care a lot more that it runs on Windows though. If more people used OpenBSD, then I’d care more. That’s the only reason I care about Windows. It’s where the people are.

                                                                                                                                                                                                                Niche doesn’t mean unimportant.

                                                                                                                                                                                                                To pop up a level, I was making a very narrow point on a particular choice of wording. Namely, that “Rust isn’t as portable as C” is glossing over some really significant nuance depending on what you’re trying to do. If all you’re trying to do is distribute a CLI application, then Rust might not let you target as many platforms as easily as C, but it might let you reach more people with a lot less effort.

                                                                                                                                                                                                          2. 4

                                                                                                                                                                                                            I want people who decide to write a novel sound daemon in C to see those sorts of comments, and (ideally) rethink the decision to write a novel C program to begin with.

                                                                                                                                                                                                            What in the world makes you think they haven’t considered that question and concluded that C, for all its shortcomings, was nonetheless their best option?

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              If the conclusion is C, their thinking is wrong.

                                                                                                                                                                                                          3. 1

                                                                                                                                                                                                            stuck on software that has to run on platforms Rust doesn’t even support

                                                                                                                                                                                                            Porting Rust to a platform sounds more achievable than writing correct software in C, so the only thing ridiculous is that people think “I haven’t ported it” is a valid excuse.

                                                                                                                                                                                                      2. 2

                                                                                                                                                                                                        Who is doing this?

                                                                                                                                                                                                        Lots of people. Search for “heartbleed C” or “heartbleed memory safety” or “heartbleed rust”.

                                                                                                                                                                                                        Are there actually credible people saying, “curl would not have any CVEs if it were written in a memory safe language”?

                                                                                                                                                                                                        They are not credible to me if they make such absurd claims, but they exist in very large numbers. They won’t claim that all problems would go away, but they all point out that heartbleed woulnd’t happen if openssl was written in Rust (for example). Yes, there are hundreds of such claims on the web. Thousands probably. As if a basic error like the one that led to heartbleed could only take the form of a memory safety problem.

                                                                                                                                                                                                        As for credibility. I don’t find your definition very useful. There is a lot of software used by millions, much of it genuinely useful that is still badly engineered. I don’t think popularity is a good indicator for credibility.

                                                                                                                                                                                                        1. 6

                                                                                                                                                                                                          Can you actually show me someone who is claiming that all security vulnerabilities will be fixed by using Rust or some other memory safe language that would meet your standard of credibility if it weren’t for that statement itself?

                                                                                                                                                                                                          I tried your search queries and I found nobody saying or implying something like, “using a memory safe language will prevent all CVEs.”

                                                                                                                                                                                                          but they all point out that heartbleed woulnd’t happen if openssl was written in Rust (for example)

                                                                                                                                                                                                          This is a very specific claim though. For the sake of argument, if someone were wrong about that specific case, that doesn’t mean they are conflating memory safety for all security vulnerabilities. That’s what I’m responding to.

                                                                                                                                                                                                          As for credibility. I don’t find your definition very useful. There is a lot of software used by millions, much of it genuinely useful that is still badly engineered. I don’t think popularity is a good indicator for credibility.

                                                                                                                                                                                                          So propose a new one? Sheesh. Dispense with the pointless nitpicking and move the discussion forward. My definition doesn’t require something to be popular. I think I was pretty clear in my comment what I was trying to achieve by using the word “credible.” Especially after my edit. I even explicitly said that I was trying to use it in a very broad sense. So if you want to disagree, fine. Then propose something better. Unless you have no standard of credibility. In which case, I suppose we’re at an impasse.

                                                                                                                                                                                                          1. 1

                                                                                                                                                                                                            I tried your search queries and I found nobody saying or implying something like, “using a memory safe language will prevent all CVEs.”

                                                                                                                                                                                                            I never made such claim. You are insisting in the whole “prevent all CVEs”. That is an extreme point that I never made, nor did any other people in this thread. If you take to that extreme, then sure you are right. I never claimed that people say that Rust will magically do their laundry either. Please let’s keep the discussion to a level of reasonability so it stays fruitful.

                                                                                                                                                                                                            FWIW, for “heartbleed rust”, google returns this in the first page:

                                                                                                                                                                                                            • Would the Cloudbleed have been prevented if Rust was used
                                                                                                                                                                                                            • How to Prevent the next Heartbleed
                                                                                                                                                                                                            • Would Rust have prevented Heartbleed? Another look

                                                                                                                                                                                                            All these are absurd. It is not my point to shame or blame. I have no idea who the author of the heartbleed offending code is. And in all honest we all have made mistakes. But let’s not take relativism to the absurd. Let’s be clear, it was objectively very poorly written code with a trivial error. A seasoned engineer should look at that and immediately see the problem. If you think that level of quality is less problematic when you use a ‘safer’ language, you are in for very bad surprises. It was objectively bad engineering, nothing less. The language had nothing to do with that. A problem with the same severity would have the same probability to occur in Rust, it would just take other form. The claims on the titles I quoted from my google are silly. If you jump from a plane without a parachute you also prevent the whole class of accidents that happen when the screen opens up. I am sure people understand that that is a silly claim.

                                                                                                                                                                                                            This is a very specific claim though. For the sake of argument, if someone were wrong about that specific case, that doesn’t mean they are conflating memory safety for all security vulnerabilities. That’s what I’m responding to.

                                                                                                                                                                                                            Again, no one is claiming that rust community is conflating memory safety with “all safety vulns”. I have no clue where you got that from. But to the point, it is as specific as it is pointless, as is the parachute example.

                                                                                                                                                                                                            1. 6

                                                                                                                                                                                                              I never made such claim.

                                                                                                                                                                                                              I didn’t say you did. But that’s the claim I’m responding to.

                                                                                                                                                                                                              You are insisting in the whole “prevent all CVEs”.

                                                                                                                                                                                                              I didn’t, no. I was responding to it by pointing out that it’s a straw man. It sounds like you agree. Which was my central point.

                                                                                                                                                                                                              nor did any other people in this thread

                                                                                                                                                                                                              No, they did:

                                                                                                                                                                                                              But conflating memory safety with software security or reliability is a bad idea. There’s tons of memory-safe code out there that has so many CVEs it’s not even funny.

                                                                                                                                                                                                              The rest of your comment is just so far removed from any reality that I know that I don’t even know how to engage with it. It sounds like you’re in the “humans just need to be better” camp. I’m sure you’ve heard the various arguments about why that’s not a particularly productive position to take. I don’t have any new arguments to present.

                                                                                                                                                                                                              1. 3

                                                                                                                                                                                                                No, they did:

                                                                                                                                                                                                                Specifically, I (sort of) claimed that and expanded upon it here. And I’m just going to re-emphasise what I said in that comment.

                                                                                                                                                                                                                I see from your profile that you’re a member of the Rust library team – I imagine most of the interactions you have within the Rust community are with people who are actively involved in building the Rust environment. That is, people who have the expertise (both with Rust and other aspects software development), the skill, and the free time to make substantial contributions to a game-changing technology, and who are therefore extremely unlikely to claim anything of that sort.

                                                                                                                                                                                                                So I understand why this looks like a straw man argument to you – but this is not the Rust-related interaction that many people have. I was gonna say “most” but who knows, maybe I just got dealt a bad hand.

                                                                                                                                                                                                                Most of “us” (who don’t know/use Rust or who, like me, don’t use it that much) know it via the armchair engineering crowd that sits on the sides, sneers at software like the Linux kernel and casually dismisses it as insecure just for being written in C, with the obvious undertone that writing it in Rust would make it secure. Like this or like this or like this.

                                                                                                                                                                                                                They’re not dismissing it as memory unsafe and the undertone isn’t that (re)writing it in Rust would plug the memory safety holes. When they propose that some 25 year-old piece of software be rewritten in Rust today, the idea, really, is that if you start today, you’ll have something that’s more secure, whatever the means, in like one or two years.

                                                                                                                                                                                                                That’s why there are people who want RIIR flags. Not to avoid useful discussion with knowledgeable members of the Rust community like you, but to avoid dismissive comments from the well ackshually crowd who thinks about something for all of thirty seconds and then knows exactly where and why someone is wrong about a project they’ve been working on for five years.

                                                                                                                                                                                                                1. 5

                                                                                                                                                                                                                  I imagine most of the interactions you have within the Rust community are with people who are actively involved in building the Rust environment.

                                                                                                                                                                                                                  Not necessarily. It depends on the day. I am also on the moderation team. So I tend to get a narrow view on library matters and a very broad view on everything else. But I also frequent r/rust (not an “official” Rust space), in addition to HN and Lobsters.

                                                                                                                                                                                                                  That is, people who have the expertise (both with Rust and other aspects software development), the skill, and the free time to make substantial contributions to a game-changing technology, and who are therefore extremely unlikely to claim anything of that sort.

                                                                                                                                                                                                                  Certainly. I am under no illusion about that. I don’t think you or anyone was saying “core Rust engineers have made ridiculous claim Foo.” That’s why I was asking for more data. I wanted to hear about any credible person who was making those claims.

                                                                                                                                                                                                                  FWIW, you didn’t just do this with Rust. You kinda did it with Java too in another comment:

                                                                                                                                                                                                                  There was a more or less general expectation (read: lots of marketing material, since Java was commercially-backed, but certainly no shortage of independent tech evangelists) that, without pointers, all problems would go away – no more security issues, no more crashes and so on.

                                                                                                                                                                                                                  I mean, like, really? All problems? I might give you that there were maybe some marketing materials that, by virtue of omission, gave that impression that Java solved “all problems.” But, a “general expectation”? I was around back then too, and I don’t remember anything resembling that.

                                                                                                                                                                                                                  But, like, Java did solve some problems. It came with some of its own, not all of which were as deeply explored as they are today.

                                                                                                                                                                                                                  See, the thing is, when you say hyperbolic things like this, it makes your side of the argument a lot easier to make. Because making this sort of argument paints the opposing side as patently ridiculous, and this in turn removes the need to address the nuance in these arguments.

                                                                                                                                                                                                                  So I understand why this looks like a straw man argument to you – but this is not the Rust-related interaction that many people have. I was gonna say “most” but who knows, maybe I just got dealt a bad hand.

                                                                                                                                                                                                                  Again. If you see someone with a misconception like this—they no doubt exist—then kindly point it out. But talking about it as a sort of general phenomenon just seems so misguided to me. Unless it really is a general phenomenon, in which case, I’d expect to be able to observe at least someone building software that others use on the premise that switching to Rust will fix all of their security problems. Instead, what we see are folks like the curl author making a very careful analysis of the trade offs involved here. With data.

                                                                                                                                                                                                                  Like this or like this or like this.

                                                                                                                                                                                                                  RE https://news.ycombinator.com/item?id=25921917: Yup, that’s a troll comment from my perspective. If I had seen it, I would have flagged it.

                                                                                                                                                                                                                  RE https://news.ycombinator.com/threads?id=xvilka: I can’t tell if they’re a troll, but they definitely post low effort comments. I’d downvote most of them if I saw them. I downvote almost any comment that is entirely, “Why didn’t you write it in X language?” Regretably, it can be a legitimate question for beginners to ask, since a beginner’s view of the world is just so narrow, nearly by definition.

                                                                                                                                                                                                                  RE https://news.ycombinator.com/item?id=26398042: Kinda more of the above.

                                                                                                                                                                                                                  I note that the first and third links you gave were downvoted quite a bit. So that seems like the system is working. And that there aren’t hordes of people secretly in favor of comments like that and upvoting them.

                                                                                                                                                                                                                  FWIW, I don’t recognize any of these people as Rust community members. Or rather, I don’t recognize their handles. And as a moderator, I am at least passively aware of pretty much anyone that frequents Rust spaces. Because I have to skim a lot of content.

                                                                                                                                                                                                                  I’m not sure why we have descended into an RESF debate. For every RESF comment you show, I could show you another anti-RESF Lobsters’ comment.

                                                                                                                                                                                                                  It’s just amazing to me that folks cannot distinguish between the zeal of the newly converted and the actual substance of the idea itself. Like, look at this comment in this very thread. Calling this post “RIIR spam,” even though it’s clearly not.

                                                                                                                                                                                                                  They’re not dismissing it as memory unsafe and the undertone isn’t that (re)writing it in Rust would plug the memory safety holes. When they propose that some 25 year-old piece of software be rewritten in Rust today, the idea, really, is that if you start today, you’ll have something that’s more secure, whatever the means, in like one or two years.

                                                                                                                                                                                                                  But that’s very different than what you said before. It doesn’t necessarily conflate memory safety with security. That’s a more nuanced representation of the argument and it is much harder to easily knock down (if at all). It’s at least true enough that multiple groups of people with a financial stake have made a bet on that being true. A reasonable interpretation of “more secure” is “using Rust will fix most or nearly all of the security vulnerabilities that we have as a result of memory unsafety.” Can using Rust also introduce new security vulnerabilities unrelated to memory safety by virtue of the rewrite? Absolutely. But whether this is true or not, and to what extent, really depends on a number of nuanced factors.

                                                                                                                                                                                                                  That’s why there are people who want RIIR flags. Not to avoid useful discussion with knowledgeable members of the Rust community like you, but to avoid dismissive comments from the well ackshually crowd who thinks about something for all of thirty seconds and then knows exactly where and why someone is wrong about a project they’ve been working on for five years.

                                                                                                                                                                                                                  The “well ackshually” crowd exists pretty much everywhere. Rust perhaps has a higher concentration of them right now because it’s still new. But they’re always going to be around. I’ve been downvoting and arguing with the “well ackshually” crowd for years before I even knew what Rust was.

                                                                                                                                                                                                                  If you see a dismissive comment that isn’t contributing to the discussion, regardless of whether it’s about RIIR or not, flag it. I have absolutely no problem with folks pointing out low effort RESF comments that express blind enthusiasm for a technology. Trade offs should always be accounted for. My problem is that the anti-RESF crowd is not doing just that. They are also using it as a bludgeon against almost anything that involves switching to Rust. This very post, by Curl’s author, is not some low effort RESF bullshit. (I should say that not all RESF bullshit is trolling. I’ve come across a few folks that are just new to the world of programming. So they just don’t know how to see the nuance in things yet, even if it’s explicitly stated. There’s only so much novel signal a brain can take in at any point. Unfortunately, it’s difficult to differentiate between sincere but misguided beginners and trolls. Maybe there are people other than trolls and beginners posting RESF bullshit, but I don’t actually know who they are.)

                                                                                                                                                                                                                  Either way, if we get RIIR flags, then we should get anti-RIIR flags. See where that leads? Nowhere good. Because people can’t seem to differentiate between flippant comments and substance.

                                                                                                                                                                                                                  Sorry I got a bit ranty, but this whole thread is just bush league IMO.

                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                    This very post, by Curl’s author, is not some low effort RESF bullshit.

                                                                                                                                                                                                                    No, I’m aware of that. But the RESF bullshit posters have a bit of a history with Curl: https://daniel.haxx.se/blog/2017/03/27/curl-is-c/ .

                                                                                                                                                                                                                    Every once in a while someone suggests to me that curl and libcurl would do better if rewritten in a “safe language”. Rust is one such alternative language commonly suggested. This happens especially often when we publish new security vulnerabilities.

                                                                                                                                                                                                                    I try to keep away from these bush league threads myself but, uh, sometimes you just go with it, and this was one of those cases precisely because of that context.

                                                                                                                                                                                                                    I’ve been slowly trying to nudge people into using Rust on embedded systems ever since I gave up on Ada, so I’m not trying to dismiss it, I have quite an active interest in it. Yet I’ve been at the receiving end of “you should rewrite that in a safe language” many times, too, like most people writing firmware. And I don’t mean on lobster.rs (which has a level-headed audience, mostly :P), I mean IRL, too. Nine times out of ten these discussions are bullshit.

                                                                                                                                                                                                                    That’s because nine times out of ten they’re not carried out with people who are really knowledgeable about Rust and firmware development. E.g. I get long lectures about how it’ll vastly improve firmware reliability by eliminating double frees and dangling pointers. When I try to point out that this is true in general, and that Rust’s memory model is generally helpful in embedded systems (e.g. the borrow checker is great!) but this particular problem is a non-issue because this is an embedded system and all allocations are static and we never get a double free because we don’t even malloc! I get long lectures about how two years from now everything will be AArch64 anyway and memory space won’t be an issue.

                                                                                                                                                                                                                    (Edit: to be clear – I definitely don’t support “RIIR” flags or anything of the sort, and indeed “the system” works, as in, when one of the RIIR trolls pop up, they get downvoted into oblivion, whether they’re deliberately trolling or just don’t know better. I’m just trying to explain where some of the negativity comes from, and why in my personal experience it’s often try to hold back on it even when you actually like Rust and want to use it more!)

                                                                                                                                                                                                                    I mean, like, really? All problems? I might give you that there were maybe some marketing materials that, by virtue of omission, gave that impression that Java solved “all problems.” But, a “general expectation”? I was around back then too, and I don’t remember anything resembling that.

                                                                                                                                                                                                                    Oh, yeah, that was my first exposure to hype, and it gave me a healthy dose of skepticism towards tech publications. I got to witness that as a part of the (budding, in my part of the world) tech journalism scene (and then, to some degree, through my first programming gigs). The cycle went basically as follows:

                                                                                                                                                                                                                    There were lots of talks and articles and books on Java between ’95 and ’97-‘98 (that was somewhat before my time but that’s the material I learned Java from later) that always opened with two things: it’s super portable (JVM!) and there are no pointers, so Java programs are less likely to crash due to bad memory accesses and are less likely to have security problems.

                                                                                                                                                                                                                    These were completely level-headed and obviously correct. Experienced programmers got it and even those who didn’t use Java nonetheless emulated some of the good ideas in their own environments.

                                                                                                                                                                                                                    Then 2-4 years later we got hit by all the interns and lovely USENET flamers who’d grown up on stories they didn’t really understand about Java and didn’t really qualify these statements.

                                                                                                                                                                                                                    So then I spent about two years politely fending off suggestions about articles on how net appliances and thin clients are using Java because it’s more secure and stable, on why everyone is moving to Java and C++ will only be used for legacy applications and so on – largely because I really didn’t understand these things well enough, but man am I glad I let modesty get the better of me. Most of my colleagues didn’t budge, either, but there was a period during which I read a “Java programs don’t crash” article every month because at least one – otherwise quite respectable – magazine would publish one.

                                                                                                                                                                                                                    1. 4

                                                                                                                                                                                                                      Aye. Thanks for sharing. I totally get your perspective. Talking about your projects with folks only to have to get into an exhausting debate that you’ve had umpteen times already is frustrating. Happens to me all the time too, for things outside of Rust. So I know the feeling. It happens in the form of, “why didn’t you do X instead of Y?” Depending on how its phrased, it can feel like a low brow dismissal. The problem is that that line of questioning is also a really great way of getting a better understanding of the thing you’re looking at in a way that fits into your own mental model of the world. Like for example, at work I use Elasticsearch. If I see a new project use SOLR, I’m going to be naturally curious as to why they chose it over Elasticsearch. I don’t give two poops about either one personally, but maybe they have some insight into the two that I don’t have, and updating my mental model would be nice. The problem is that asking the obvious question comes across as a dismissal. It’s unfortunate. (Of course, sometimes it is a dismissal. It’s not always asked in good faith. Sometimes it’s coupled with a healthy dose of snobbery, and those folks can just fuck right off.)

                                                                                                                                                                                                                      It’s harder to do IRL, but the technique I’ve adopted is that when someone asks me questions like that, I put about as much effort into the response as they did the question. If they’re earnest and just trying to understand, then my hope is that they might ask more follow up questions, and then it might become a nice teaching moment. But most of the time, it’s not.

                                                                                                                                                                                                                      I guess I would just re-iterate that my main issue with the anti-RIIR crowd is that it’s overbroad. If it were just some groaning about trolls, then fine. But it’s brought up pretty much any time Rust is brought up, even if bringing Rust up is appropriate.

                                                                                                                                                                                                                      But I suppose that’s the state of the Internet these days. Tribes are everywhere and culture wars can’t be stopped.

                                                                                                                                                                                                              2. 2

                                                                                                                                                                                                                And in all honest we all have made mistakes. But let’s not take relativism to the absurd. Let’s be clear, it was objectively very poorly written code with a trivial error. A seasoned engineer should look at that and immediately see the problem. If you think that level of quality is less problematic when you use a ‘safer’ language, you are in for very bad surprises. It was objectively bad engineering, nothing less. The language had nothing to do with that. A problem with the same severity would have the same probability to occur in Rust, it would just take other form.

                                                                                                                                                                                                                What form would it take? Would it be in the “all private keys in use on the internet can be leaked”-form? Probably not, I think?

                                                                                                                                                                                                                Anyway, let’s forget about security for a minute; why wouldn’t you want the computer to automate memory management for you? Our entire job as programmers is automate things and the more things are automated, the better as it’s less work for us. This is why we write programs and scripts in the first place.

                                                                                                                                                                                                                Traditionally automated memory management has come with some trade-offs (i.e. runtime performance hits due to GCs) and Rust attempts to find a solution which automates things without these drawbacks. This seems like a good idea to me, because it’s just more convenient: I want the computer to do as much work for me as possible; that’s its job.

                                                                                                                                                                                                                Back to security: if I have a door that could be securely locked by just pressing a button vs. a door that can be securely locked by some complicated procedure, then the first door would be more secure as it’s easier to use. Sooner or later people will invariable make some mistake in the second door’s procedure. Does that mean the first door guarantees security? No, of course not. You might forget to close the window, or you might even forget to press that button. But it sure reduces the number of things you need to do for a secure locking, and the chances you get it right are higher.

                                                                                                                                                                                                    2. 4

                                                                                                                                                                                                      Kinda sorta. PHP is/was largely written in C itself and IIRC had its share of memory related security bugs from just feeding naughty data to PHP standard library functions.

                                                                                                                                                                                                      I don’t know what PHP is like today from that point of view.

                                                                                                                                                                                                      So, I take issue when you say:

                                                                                                                                                                                                      I’m just arguing that “this program is written in C and therefore not trustworthy because C is an unsafe language” is, at best, silly. PHP 4 was safer than Rust and boy do I not want to go back to dealing with PHP 4 applications.

                                                                                                                                                                                                      I still think it’s completely justifiable to be skeptical of a program written in C. Just because another language may also be bad/insecure/whatever does not invalidate the statement or sentiment that C is a dangerous language that honestly brings almost nothing to the table in the modern era.

                                                                                                                                                                                                      1. 3

                                                                                                                                                                                                        Interestingly, there’s a different language from a very similar time as C, that has more safety features, and its name is Pascal. And there was a time when they were kinda competing, as far as I understood. This was maybe especially at the time of Turbo C and Turbo Pascal, and then also Delphi. Somehow C won, with the main argument being I believe “because performance”, at least that’s how I remember it. My impression is that quite often, when faced with a performance vs. security choice, “the market” chooses performance over security. I don’t have hard data as to whether code written in Pascal was more secure than that in C; I’d be curious to see some comparison like that. I seem to have a purely anecdotal memory, than when I felt some software was remarkably stable, it tended to show up to be written in Pascal. Obviously it was still totally possible to write programs with bugs in Pascal; I think Delphi code had some characteristic kind of error messages that I saw often enough to learn to recognize them. Notably, it also actually still required manual memory management - but I believe it was better guarded against buffer overruns etc. than C.

                                                                                                                                                                                                        1. 2

                                                                                                                                                                                                          I thought the reasons for C’s wide adoption were Unix and the university system. I.E., the universities were turning out grads who knew Unix and C. I’ve only heard good things about the performance of Turbo Pascal.

                                                                                                                                                                                                          Pascal is safer, but it was certainly possible to write buggy Pascal. Back in the early 90s I hung out on bulletin boards and played a lot of Trade Wars 2002. That was written in Turbo Pascal, and it had a few notable and widely exploited bugs over the years. One such was a signed overflow of a 16-bit integer. I won a couple Trade Wars games by exploiting those kinds of bugs.

                                                                                                                                                                                                      2. 17

                                                                                                                                                                                                        You are arguing that eliminating one class of bugs doesn’t make sense, because there are other classes of bugs? That reminds me of the mental gymnastics of untyped language proponents.

                                                                                                                                                                                                        1. 7

                                                                                                                                                                                                          This cartoon deserves an entire blog post! But I’ll just list out the gymnastic routines of statically-typed language proponents:

                                                                                                                                                                                                          1. 2

                                                                                                                                                                                                            I think no one cares about this.

                                                                                                                                                                                                            1. 3

                                                                                                                                                                                                              Well, not when they only spend 3 minutes per link. You might have to read and reflect.

                                                                                                                                                                                                              1. 1

                                                                                                                                                                                                                The thing is that you are completely missing the point (and your kind of nerd-contrarianism doesn’t make you look as smart as you think it does).

                                                                                                                                                                                                                1. 5

                                                                                                                                                                                                                  Oh! I don’t do this to look smart. I do this because I see in you the same tribalism that I once had, and I only grew past it because of evidence like the links I shared with you. I’m not trying to say that static typing is wrong; I’m trying to expand and enrich your knowledge about type theory. Once you’ve reached a certain altitude and vantage, then you’ll see that static and dynamic typing are not tribes which live in opposition, but ways of looking at the universal behaviors of computation.

                                                                                                                                                                                                                  Please, read and reflect. Otherwise this entire thread was off-topic: Your first post is not a reply to its parent, but a tangent that allowed you to display your tribal affiliation. I don’t mind being off-topic as long as it provides a chance to improve discourse.

                                                                                                                                                                                                                  1. 2

                                                                                                                                                                                                                    The (popular) mistake you are making is that you pretend things are equal when they are not, just like shitting your pants (untyped) and trying to not shit your pants (typed) are not positions with similar merit.

                                                                                                                                                                                                              2. 1

                                                                                                                                                                                                                No those are all serious refutations of why statically-typed languages is a panacea and is a lot more insightful than a silly comic someone made to find affirmation among their Twitter followers.

                                                                                                                                                                                                                1. 1

                                                                                                                                                                                                                  Can you show me where I claimed that “typed languages [are] a panacea”?

                                                                                                                                                                                                                  Anyway, have fun stomping on that strawman.

                                                                                                                                                                                                          2. 6

                                                                                                                                                                                                            Here are the only economically viable solutions I see to the problem of “too much core infrastructure has recurring, exploitable memory unsafety bugs” problem (mainly because of C):

                                                                                                                                                                                                            • Gradually update the code with annotations – something like Checked C. Or even some palatable subset/enhancement of C++.
                                                                                                                                                                                                            • Distros evolve into sandboxing based on the principle of least privilege (DJB style, reusing Chrome/Mozilla sandboxes, etc.) – I think this is one of the most economically viable solutions. You still have memory unsafety but the resulting exploits are prevented (there has been research quantifying this)
                                                                                                                                                                                                            • The product side of the industry somehow makes a drastic shift and people don’t use kernels and browsers anymore (very unlikely, as even the “mobile revolution” reused a ton of infrastructure from 10, 20, 30, 40 years ago, both on the iOS and Android side)
                                                                                                                                                                                                            • (leaving out hardware-based solutions here since I think hardware changes slower than software)
                                                                                                                                                                                                            • (I have looked at some of the C to Rust translators, and based on my own experience with translating code and manually rewriting it, I’m not optimistic about that approach. The target language has to be designed with translation in mind, or you get a big mess.)

                                                                                                                                                                                                            Manually rewriting code in Rust or any other language is NOT on that list. It would be nice but I think it’s a fantasy. There’s simply too much code, and too few people to rewrite it.

                                                                                                                                                                                                            Moreover with code like bash, to a first approximation there’s 1 person who understands it well enough to rewrite it (and even that person doesn’t really understand his own code from years ago, and that’s about what we should expect, given the situation).

                                                                                                                                                                                                            Also, the most infamous bash vulnerability (ShellShock) was not related to memory unsafety at all. Memory unsafety is really a subset of the problem with core infrastructure.

                                                                                                                                                                                                            Sometime around 2020 I made a claim that in 2030 the majority of your kernel and your browser will still be in C or C++ (not to mention most of your phone’s low level stack, etc.).

                                                                                                                                                                                                            Knowing what the incentives are and the level of resources devoted to the problem, I think we’re still on track for that.

                                                                                                                                                                                                            I’m honestly interested if anyone would take the opposite side: we can migrate more than 50% of our critical common infrastructure by 2030.


                                                                                                                                                                                                            This says nothing about new projects written in Rust of course. For core infrastructure, the memory safety + lack of GC could make it a great choice. But to a large degree we’ll still be using old code. Software and especially low level infrastructure has really severe network effects.

                                                                                                                                                                                                            1. 1

                                                                                                                                                                                                              I agree with you. It’s just not going to happen that we actually replace most of the C code that is out there. My hope, however, is that C (and C++) becomes the next COBOL in the sense that it still exists, there are still people paid to work on systems written in it, but nobody is starting new projects in it.

                                                                                                                                                                                                              Along the same lines as your first bullet point, I think a big step forward- and the best “bang for our buck” will be people doing fuzz testing on all of these old C projects. There was just recently a story making the rounds here and on HN about some bug in… sudo? maybe? that lead to a revelation of the fact that the commit on the project that caused the regression was a bugfix to a bug for which there was no test, before or after the change. So, not only did the change introduce a bug that wasn’t caught by a test that didn’t exist- we can’t even be sure that it really fixed the issue it claimed to, or that we really understood the issue, or whatever.

                                                                                                                                                                                                              My point is that these projects probably “should” be rewritten in Rust or Zig or whatever. But there’s much lower hanging fruit. Just throw these code bases through some sanitizers, fuzzers, whatever.

                                                                                                                                                                                                              1. 2

                                                                                                                                                                                                                Yeah that’s basically what OSS Fuzz has been doing since 2016. Basically throwing a little money at projects to integrate continuous fuzzing. I haven’t heard many updates on it but in principle it seems like the right thing.

                                                                                                                                                                                                                https://github.com/google/oss-fuzz

                                                                                                                                                                                                                As of January 2021, OSS-Fuzz has found over 25,000 bugs in 375 open source projects.

                                                                                                                                                                                                                The obvious question is what percent of curl’s vulnerabilities could be found this way. I googled and found this:

                                                                                                                                                                                                                https://github.com/curl/curl-fuzzer

                                                                                                                                                                                                                which doesn’t look particularly active, and doesn’t seem to show any results (?).

                                                                                                                                                                                                                Rather than talk about curl and “RIIR” (which isn’t going to happen soon even if the maintainer wants it to), it would be better to talk about if curl is doing everything it can along the other lines.

                                                                                                                                                                                                                Someone mentioned that curl is the epitome of bad 90’s C code, and I’ve seen a lot of that myself. There is a lot of diversity in the quality of C code out there, and often the sloppiest C projects have a lot of users.

                                                                                                                                                                                                                Prominent examples are bash, PHP, Apache, etc. They code fast and sloppy and are responsive to their users.

                                                                                                                                                                                                                There’s a fundamental economic problem that a lot of these discussion are missing. Possible/impossible or feasible/infeasible is one thing; whether it will actually happen is a different story.


                                                                                                                                                                                                                Bottom line is that I think there should be more talk about projects along these lines, more talk about sandboxing and principle of least privilege, and less talk about “RIIR”.

                                                                                                                                                                                                              2. 1

                                                                                                                                                                                                                Manually rewriting code in Rust or any other language is NOT on that list.

                                                                                                                                                                                                                Nah. This kinda assumes that the rewrite/replacement/whatever would happen due to technical reasons.

                                                                                                                                                                                                                It certainly wouldn’t. If a kernel/library/application gets replaced by a safer implementation, it’s for business reasons, where it just happens that the replacement is written in e. g. Rust.

                                                                                                                                                                                                                So yes, I fully expect that a certain amount of rewrites to happen, just not for the reasons you think.

                                                                                                                                                                                                              3. 3

                                                                                                                                                                                                                I mean, there is no silver bullet, right? We can all agree with that? So, therefore, “just apply more sagacious thinking” isn’t going to fix anything just as “switch to <Rust|D|C#|&C>” won’t? The focus on tools seems to miss the truth that this is a human factors problem, and a technological solution isn’t going to actually work.

                                                                                                                                                                                                                1. 2

                                                                                                                                                                                                                  One of curl’s main advantages is its ubiquity; I can run it on an OpenWRT router, a 32bit ARMv7 OpenBSD machine, a POWER9 machine, and even Illumos//OpenIndiana. It’s a universal toolkit. It also runs in extremely constrained and underpowered environments.

                                                                                                                                                                                                                  Do you know of a memory-safe language that fits the bill (portability and a tiny footprint)? Rust fails on the former and Java fails on the latter. Go might work (gccgo and cgo combined have a lot of targets and TinyGo can work in constrained environments), but nowhere as well as C.

                                                                                                                                                                                                                  1. 3

                                                                                                                                                                                                                    Java fails on [a tiny footprint]

                                                                                                                                                                                                                    There is java for smartcards . . .

                                                                                                                                                                                                                    Do you know of a memory-safe language that fits the bill (portability and a tiny footprint)

                                                                                                                                                                                                                    Nim, ats.

                                                                                                                                                                                                                  2. 2

                                                                                                                                                                                                                    How would you deal with plan interference? The linked paper requires an entirely new language in order to even talk about this class of bugs!

                                                                                                                                                                                                                    1. 1

                                                                                                                                                                                                                      Global state as the dominant paradigm? “JavaScript interoperability”? Sounds like today is a Fuck You Friday.