1. 4

    Honestly, point # 8 should be directly in the Rust book. And the whole article is amazing.

    Corollary – has anyone ever explored the possibility of an editor plugin or code formatter that graphically, explicitly shows the lifetimes inferred by the compiler? (I imagine it would be much more complex than it sounds … maybe not even feasible).

    1. 3

      I have wanted this. If the error string can show you the borrow points, you should be able to get that data in the editor.

    1. 17

      Since nobody else has written about Rust yet, and you mentioned it, I’ll bite.

      I guess one of the problems of waxing poetic about Rust is that it has kind of already been beaten to death. Oodles of people are excited about Rust. Since it’s still growing, and since there is no zeal like the zeal of the newly converted, there are a lot of people singing its praises. To such an extent that an entire culture war has evolved around it, including right here on Lobsters. I’m a biased observer, and I get my hands dirty in it, but if I try to pop up a level, it really is an interesting phenomenon. But that’s a topic for a different day…

      I am not new to Rust, so I think if I was ever touched by the zeal of the newly converted, it has assuredly worn off by now. With that in mind, I’ll try to avoid just regurgitating the stuff other people say about Rust, but that’s hard.

      I think the Happy State of Rust is its entire package. I’ve written code in a lot of programming languages and ecosystems (with Windows and the APL lineage being at least two of my major blind spots), and I feel reasonably safe in the conclusion that Rust’s complete package is fairly unique. My main shtick is being relentless about being upfront about trade offs, and there are few ecosystems out there that let you express the range of trade offs offered by Rust. On the one hand, you can dip down deep into high performing generic data structures that benefit from the amazing work of optimizing compilers like LLVM, but on the other hand, you can also stay in the relative comfort of a high level application oriented language when you want to. The former might require arcane knowledge of what exactly is and isn’t undefined behavior while the latter might cause you to say, “UB? What’s that?”

      If you pop up a level, that’s a really impressive range to encapsulate in one ecosystem. It’s not like that range hasn’t been done before. Look at Python for example. The typical way you might see that range being expressed is with modules written in C with a nice, comfy and safe Pythonic API layered on top of it. But there’s just all sorts of trade offs there. First, there’s the friction of dealing with Python’s C module boundary. Second, there’s the inherent cost in providing that boundary in the frist place. And thirdly, there’s the cost of Python itself.

      I am perhaps running the risk of putting up a straw man, but that’s OK. The fact that “write low level code in C and wrap it up in a Python module” is a very popular approach to expressing the range of trade offs I’m referring to is I think enough. An example might help.

      Consider something as simple as counting the number of fields in a CSV file. The Python code is deliciously simple:

      import csv
      import sys
      
      count = 0
      for record in csv.reader(sys.stdin):
          count += len(record)
      print(count)
      

      The Rust code is a smidge bigger, but not materially so:

      fn main() -> Result<(), Box<dyn std::error::Error>> {
          let mut rdr = csv::ReaderBuilder::new()
              .has_headers(false)
              .from_reader(std::io::stdin());
          let mut count = 0;
          for result in rdr.byte_records() {
              let record = result?;
              count += record.len();
          }
          println!("{}", count);
          Ok(())
      }
      

      Running the program on a 445MB CSV file from the Open Policing project with the Python program:

      $ time python happy-state.py < /m/sets/csv/openpolicing/MA-clean.csv
      78620877
      
      real    5.844
      user    5.777
      sys     0.060
      maxmem  8 MB
      faults  0
      

      And now Rust:

      $ time ./target/release/happy-state < /m/sets/csv/openpolicing/MA-clean.csv
      78620877
      
      real    0.915
      user    0.873
      sys     0.040
      maxmem  6 MB
      faults  0
      

      Despite the fact that Python’s CSV parser is written in C, it just can’t compete. Rust’s CSV parser (disclaimer: I wrote it, I’m using an example I know well) is itself a very complex beast. But as in the Python ecosystem, all of this is well hidden from typical users of the library. It’s got some gnarly internals, but users are presented with an API that is at least close to the simplicity of Python’s API, except it’s also 6 times faster.

      And this is just the beginning. Since Rust’s range of trade offs can really be cashed in almost everywhere, you can make small tweaks to your program to make it run even faster, if you’re willing to put up with a bit more complexity:

      fn main() -> Result<(), Box<dyn std::error::Error>> {
          let mut rdr = csv::ReaderBuilder::new()
              .has_headers(false)
              .from_reader(std::io::stdin());
          let mut record = csv::ByteRecord::new();
          let mut count = 0;
          while rdr.read_byte_record(&mut record)? {
              count += record.len();
          }
          println!("{}", count);
          Ok(())
      }
      

      And running it:

      $ time ./target/release/happy-state < /m/sets/csv/openpolicing/MA-clean.csv
      78620877
      
      real    0.769
      user    0.729
      sys     0.037
      maxmem  6 MB
      faults  0
      

      The speed boost is small, but things like this can add up as your data set grows. And keeping in mind this is just an example. The point is that you just can’t do stuff like this with Python because Python doesn’t really let you easily amortize allocation. You’d likely need to drop all the way back down into C to get these kinds of optimizations. That’s a lot of friction to pay for.

      OK, so why am I picking on Python? Surely, my point is not to say, “See, look, Rust is faster than Python!” That’s silly. My point is more holistic than that. It’s more like, “See, look, you can be in a comfy, safe high level language, but are also free to trade code complexity for faster programs without a lot of friction.”

      Other ecosystems like modern C++ start to approach this. But obviously there are some critical distinctions there. The main one being that C++ is “unsafe by default everywhere.” You don’t get that same assurance of a safe and comfy high level language that you do in Rust. IMO, anyway.

      And then there are all the other little things that Rust does well that makes me as a pragmatic programmer very happy. The built in unit testing and API documentation tooling is just lovely. It’s one of those things that Rust didn’t invent, of course, but it feels like I’m constantly missing it (in some form, to varying degrees) from almost every other language or ecosystem I use. There’s always some detail that’s missing or not done well enough or not ubiquitous.

      I think I’ll stop it here. I wrote this comment with rose tinted glasses. Rust is not a panacea and there are lots of things about it that are inferior to other languages. For example, I presented one particular example where Rust and Python code have similar levels of complexity, but obviously, that does not necessarily generalize. There are other things that make Rust inferior in certain cases, such as compile times, platform support and certification. Language complexity is hard to fix, but I largely see the other things as at least having the potential to improve significantly. And if they don’t—or don’t improve as much as folks hope—then that’s OK. Rust doesn’t have to be All Things to All People at All Times. The rosy picture I painted above can still be true for men(and hopefully many others) while not being true for others. Trade offs aren’t just purely technical endeavors; costs that may not matter as much to me might matter a lot more to others.

      1. 1

        Typo that gives the completely wrong idea: “ can still be true for men”

        I’m sure you meant “me” instead of “men”

        1. 2

          Hah, yes! Thanks for catching that. (The edit timeout has passed, otherwise I’d fix it.)

      1. 27

        So AFAICT this is the tradeoff the author consciously rejects, and the one that Svelte consciously chooses:

        • Choose writing in a JS framework that is compiled ahead of time, over writing in a JS framework that is interpreted at runtime.

        The disadvantages of this tradeoff that weigh on the author’s mind:

        • If your language is compiled, debugging is harder because the compiled code that is run does not resemble the source code you have to fix.
          • They also make the point that it’s confusing that Svelte code is Javascript, but it needs to be compiled to run it, which may change its behaviour. (To me that’s not that different from frontend framework code that is valid JS/HTML, but needs the framework runtime to run it, which may change its behaviour.)
        • If in the future more front-end-systems compile to Javascript instead of writing in it, it becomes harder to glue them together.

        I think it’s interesting to look how Elm solved these, because like Svelte, it is compiled ahead of time to small and fast JavaScript that doesn’t resemble the source code.

        Elm’s solution to ‘you have to choose between debugging the runtime JS or the source code’ is to go all-in on making it easy to debug the source code. In Elm’s case, it is an ML-family language with a type system that guarantees zero runtime errors (but won’t save you from domain mistakes, obv.), and with compilation error messages that are so helpful that they have inspired many other languages.

        Svelte, presumably, wants to remain Javascript, so a lot of error prevention becomes harder. They mentioned they want to add Typescript support. Or they could add source maps that relate compiled JS to the original Svelte code? Also, debugging compiled projects is a very old craft, it only really gets onerous if the problem is low-level or compilation is slow. I also note that Svelte compilation has a ‘dev’ flag that produces named functions, and also extra code that performs runtime checks and provides debugging info.

        Elm’s solution to the interoperation problem: an Elm module can expose ports (blog post, docs that external JS can send messages into, or that JS can subscribe to. So the ports form an Elm module’s public API.

        That still leaves the interop problem of styling the created components. If it’s you writing the Svelte, you can let Svelte do the styling (if Svelte is the whole system), or specify the right class names on the created DOM (if you’re writing the Svelte component as a subsystem). But if you’re reusing sombody else’s Svelte component, I’m not sure how easy it is to pass in the class names you’d like the component to use. Perhaps ‘support for caller-specified class names’ is even an open problem / blind spot in frontend frameworks in general?

        1. 8

          Good summary.

          In one sense, all of the hard-fought web knowledge that people may subconsciously pride themselves on knowing now becomes a stumbling block. Technologies like Svelte treat the underlying web runtime as something to be papered over and ignored. Much like compiled C, being able to debug the generated code is a needed skill, but the 1-to-1 correspondence between source code and generated code is not a guarantee, and it can be disconcerting to let go of that.

          I’m all for it. We largely ignore x86/x64 by using higher level languages and our code is better for it, even if slightly inefficient.

          Web devs love to talk of developer experience and progress in tooling. Something something…Cambrian explosion? ;)

          1. 10

            I think the author’s problem isn’t so much with it being compiled, but the fact that the source code looks like JS, but your assumptions don’t hold because there’s a lot happening to that JS so the end result isn’t anything like what you typed.

            1. 4

              Yes, I agree. Elm is a language that has its own semantics which are adhered to by its compiler. But Svelte takes the semantics of an existing language (JS) and changes them.

              I have that concern about Svelte too, though it’s not strong enough to change the fact that I’m still a fan and excited to see how Svelte evolves.

              1. 3

                Reminds me very much of the Duck Test https://en.m.wikipedia.org/wiki/Duck_test. Svelte walks like JS and talks like JS but isn’t JS. This is typically seen as a positive for those who judge their tools, at least partly, based on familiarity.

                1. 1

                  That makes the article make more sense. That would be difficult to reckon with.

              2. 5

                Or they could add source maps that relate compiled JS to the original Svelte code?

                Svelte creates JS and CSS source maps on compilation - https://svelte.dev/docs#svelte_compile

                There’s also the @debug helper for templates - https://svelte.dev/docs#debug

                In practice I’ve found debugging Svelte to be mostly trivial and sometimes difficult. Dev tools will help close the gap but they’re not mature.

                For styling, style encapsulation is what I’ve seen the devs recommend, but nothing stops you from passing classes as props to components that accept them. (I do that a lot because I like utility CSS libraries like Tailwind) The biggest open RFC right now is about passing CSS custom properties (CSS vars) to components. https://github.com/sveltejs/rfcs/pull/13

                1. 1

                  I think the tradeoff here isn’t about source mapping and the sort, but instead that if you take it as given that you’re going to compile your language, then you might as well through more language safety features in (a la Elm).

                  That might be true, but the other sacrifice is familiarity. Svelte can be learned by a frontend dev very quickly and without much “relearning” fear. Instead, you get the cognitive dissonance problem of it being almost what you expect but, then, not quite.

                  1. 2

                    if you take it as given that you’re going to compile your language, then you might as well through more language safety features in (a la Elm).

                    There’s a big leap from Svelte to Elm beyond just compiling the language. Elm has tremendous benefits, definitely, but it gives up seamless interop with the DOM, mutable web APIs, JS libraries, and future web standards. Elm has JS interop but only asynchronously through its ports. Purity and soundness are great but at what cost? (that’s a rhetorical holy war question, not worth discussing here IMO!)

                    I think TypeScript has been gaining so much adoption largely because it makes pragmatic compromises everywhere, not just because people are resistant to learning Elm/Reason/PureScript/Haskell/etc, and when support lands in Svelte I’ll be less shy about recommending it to people.

                    1. 2

                      Yeah, I think for most people in most cases, familiarity and ease are the bigger win. I’m not arguing one should use Elm, just laying out that continuum.

                      1. 2

                        Thanks for emphasizing that point. I think I underestimate the impact of familiarity and ease for many people.

                      2. 1

                        By seamless interop do you mean synchronous? I started trying out Svelte yesterday and found the dom interop to not be seamless as I was confused by the difference between <input value={val} /> and <input on:value={val} />

                        I think from memory that’s how you get an interactive input.

                        1. 1

                          I meant seamless but that’s overstating it. (except for mutable web APIs and JS libraries - interop there is generally seamless because of how Svelte extends JS) Anything that isn’t plain HTML/CSS/JS is going to have seams. Svelte minimizes them to a degree that some other frameworks don’t, like React and especially Elm. Vue is on similar footing as Svelte.

                          The nice thing about Svelte’s seams is they often reduce the complexity and verbosity of interacting with the DOM. In your example, it sounds like you want:

                          <input bind:value={val} />

                          (or simply <input bind:value /> if that’s the name)

                          At the same times Svelte gives you the flexibility to optionally use a “controlled input” like React:

                          <input value={val} on:input={updateValue} />

                          The equivalent in plain HTML/JS is not as pleasant. Elm abstracts away the DOM element and events.

                1. 5

                  We are currently doing a mandatory WFH test run where our office is closed today and everyone is working from home. So far it’s been interesting, lots of zoom meetings (context: I work for Blend, our SF office is around 300 people).

                  We are also allowing people to electively WFH for the foreseeable future.

                  1. 10

                    Go you! So many places are all “I guess we’ll deal with it when it happens”, but there’s nothing like a test run.

                    1. 2

                      Similar. I work from home normally though, but part of a big office in London. We’ve been rotating departments the last week, each department testing their team working from home. We’re in public health though so we’re trigger happy mitigating anything that will mess with business continuity.

                      1. 2

                        I’m in ops, and our WFH game is 100% cos we have to be able to do our entire job from a laptop as needed when on call. Our devs are not on call, but they are similarly well set-up.

                        The rest of the company is being dragged into the present quite nicely! We have a shiny new OpenVPN setup that’s doing the job of connection.

                        The sticking point is phone meetings - suitable software (RingCentral sorta sucks; Google Hangouts really sucks; Slack’s audio is way better, but only tech is on Slack), and having the people in the room give a damn about making sure the microphone can hear them.

                        1. 3

                          Zoom any good? Their name seems to crop up in this context a lot.

                          1. 1

                            Zoom has been excellent the few times I’ve used it this year – far above all other solutions I’ve experienced besides Blue Jeans (I’d say they’re tied for top of the heap).

                            1. 1

                              RingCentral is actually a fork of Zoom, I think. I’ve only used Zoom myself for podcast recording, and it was fine?

                              1. 2
                        1. 2

                          This is impressive work. I’d love to find out more about how you approached using Python to metaprogram C++. If you talk about that anywhere, links would be much appreciated. And if not, just know you have an eager audience should you decide to!

                          1. 2

                            I was writing a lot about this early in the project, but I dropped it for lack of time. The three main code generators are ASDL, re2c, and now mycpp, but I haven’t written anything about mycpp.

                            Here is some portions:

                            http://www.oilshell.org/blog/2019/12/22.html#appendix-a-oils-lexer-uses-two-stages-of-code-generation

                            http://www.oilshell.org/blog/tags.html?tag=ASDL#ASDL

                            Your best bet is read Zulip, I post details there, and feel free to ask questions. There are a ton of threads about mycpp and the translation process, which I may link in the next post.

                            https://oilshell.zulipchat.com/#narrow/stream/121539-oil-dev/topic/Naming.20convention.20for.20metaprogramming (requires login)

                            As mentioned in the January blog posts, I’m cutting a lot out of the project because it got too big. But hopefully it will get reasonably done and then I can write more about it.

                            1. 1

                              BTW I wrote a blog post that links some relevant threads. Feel free to ask questions on Zulip!

                              http://www.oilshell.org/blog/2020/03/recap.html#mycpp-the-good-the-bad-and-the-ugly

                              1. 3

                                Thanks. I found that article’s worked example a bit too trivial, but it linked to a paper that has a more complex take on the topic, which I’m currently digesting: https://kataskeue.com/gdp.pdf

                              1. 2

                                Anyone know when this was published? Late in 2019, or early?

                                I looked around for a pub date but didn’t see one.

                                EDIT – I found it from a blog post. November 2019.

                                1. 3

                                  This reminds me of Apple product designations: “JavaScript, early 2019”.

                                  1. 1

                                    Yep! They tend to run the survey around… October-ish I think? And then publish it a bit later in the year, so it’s always kind of a year-end roundup.

                                  1. 1

                                    “Technical debt” is a metaphor for all software design choices that turn out to be suboptimal, no longer valid, or just plain wrong. They incur a cost on future development. The shortcuts taken today will later slow you down, until you “pay back” the debt by fixing the problems. And it’s not just code: artifacts like architecture, documentation, tests, and domain models, can all suffer from technical debt.

                                    Do people actually categorize technical debt this way? I’m stuck in a rut with my org because the technical debt definition is too broad. Since we measure and allocate a specific amount to technical debt, it’s important IMO.

                                    One that I heard that I prefer is “Did we knowingly create that technical debt?” If so it’s tech debt, otherwise it’s not. Any opinions?

                                    1. 1

                                      Good point. I have the same quibble with that definition (though I liked the article a lot).

                                      I don’t think it matters at all if we knowingly created the debt … it only matters that there is something that slows us down that we know we could improve.

                                      In other words, “debt” is too limiting. Technical “opportunities” also exist, and you’d want to manage them pretty much in the same way – by identifying them, trying to size the potential benefit, and then pitching them to whoever controls the roadmap.

                                      1. 1

                                        I’ve rarely run into technical debt that was created knowingly. I don’t think most development is that self-aware, or able to predict the future. Often the decisions are perfectly reasonable at the time. This is where I think the “debt” word in “technical debt” is misleading: when you take on debt in the real world you know you’re doing it. I would best most technical debt in the software world was never taken on intentionally. And wasn’t even debt at the time - it only became debt maybe years later.

                                        My personal definition is any previous technical decisions - whether it’s code that was written or choices of technologies - that are now burdensome through producing more bugs, having extra complexity compared to alternatives, creating greater mental load or annoyance, or just plain slowing us down.

                                        Here are some examples of things I consider tech debt, at my current employer. Some or all of these decisions made sense at the time (all of them precede me, though I suspect if I would have made the same call on at least some of them)

                                        1. One of our teams chose Storm way back in the day. I expect that at the time it was a good decision, and it’s generally served us well, but Storm has been superseded, and is looking pretty long in the tooth, and now it’s holding us back and we want to get off it (or, so I’m told).

                                        2. The first developer they hired didn’t know what they were doing re: databases and now we’re stuck with Mongo. It’s a mess, but we’re essentially locked in at this point and I can’t see us ever getting off it

                                        3. A lot of our major application is written in Coffeescript. At the time this was probably a reasonable choice, but it’s now a dead language whose best features have been rolled into Javascript. We have tens of thousands of lines of coffee code, with no tests, that no-one wants to work on. New developers don’t want to learn Coffeescript (even though it’s fairly easy).

                                        4. We have an application built with NWjs. At the time this was chosen it was probably not at all obvious that Electron would become the clear winner, and that in 2020 would be leagues ahead in terms of features. NWjs putters on (barely) but we (and probably no-one else) would choose it over Electron today.

                                        1. 1

                                          That makes sense to me. Thanks for those examples. I guess my definition is a bit limited.

                                          I want to ask you another question though based on your definition - do you consider bugs technical debt?

                                          1. 2

                                            do you consider bugs technical debt?

                                            Not usually, though I might consider the root cause to be technical debt. For example, we might have a bug because we have a document in Mongo that’s missing a field that our application code is expecting to exist. The top-level cause is the missing field, but the root cause (at least, in my mind) is that Mongo doesn’t support schemas or constraints and if we were using a DB appropriate to the application we wouldn’t be able to have this kind of error at all.

                                            1. 2

                                              Got it. Thanks for the conversation!

                                      1. 1

                                        The explanation for the final pattern, “Siteless UIs,” is not very clear to me. Also, the architectural diagram for it is missing.

                                        1. 1

                                          Sorry - can you try https://dev.to/florianrappl/microfrontends-from-zero-to-hero-3be7 (it should be displayed there).

                                          Since Piral is an implementation of Siteless UIs maybe docs.piral.io provides a better explanation? Let me know if it does - or what is unclear / could be described in more details. Thanks - I’ll try to improve!

                                        1. 12

                                          While I find the writing of Martin Fowler to be good for getting ideas on new patterns or existing ones, better expressed, I strongly disagree calling this page a software architecture guide. Software architecture is a lot more than these patterns and approaches.

                                          I’ve been building a few large distributed systems and been in the design loop of many more the past few years, within Uber. Software design/architecture of these systems resembled anything but what the articles talk about.

                                          We did not use patterns or find the best architecture matches. Instead, we looked at the business case, the problem and the constraints. We whiteboarded, argued and discussed approaches, trade offs and short-medium-long-term impacts. We prototyped, took learnings, documented the whiteboard approach that made sense, then circulated the proposal. We debated some more over comments and in-person, then built an MVP, shipped it and went on with the same approach to the next phase, all the way to migrating over to the new system.

                                          Nowhere in this process did we talk the kind of patterns this post talks about. We talked tradeoffs and clashed ideas all the time though.

                                          Talking with friends in the industry, the same applies for much of the large systems built by tech companies across Uber, Amazon, Facebook, Google. If you read the design documents, they use little to no jargon, simple diagrams, little to no reference to patterns and are full of back-and-forth comments.

                                          Maybe I live and work in a different environment than Martin Fowler. But where his thoughts of architecture end is where my world just about begins.

                                          Also, as my view is so different on this topic than this post, I’ll likely write up my thoughts/experience in a longer form.

                                          1. 6

                                            I agree, what you describe is the way to go. You start with the business case, you think it through collaboratively, and you figure out the best plan you can for (a) getting it done, so that (b) you can maintain it with velocity from then on.

                                            But I don’t think Fowler advocates anything against that.

                                            I think, if you and some teammates were doing that, and Fowler was around, he’d be listening, understanding, and identifying patterns that he could document. Then he’d be remarkably good at explaining them to management and to the Product team. He also might overhear a conversation and say, “We struggled with that at xyz. Here’s what we did.” And he might even point to one of his own articles in that case.

                                            I think what often goes wrong is developers often start with a blank whiteboard, panic, and then grab Fowler’s or some other author’s works, and try to make them fit.

                                            Rather, the process should be: start with a blank whiteboard, think hard, sketch out some ideas and identify challenges, then maybe see if any documented patterns or ideas might help.

                                            1. 3

                                              Do you ever retroactively try to identify what architectural patterns you ended up with, for instance to mention in documentation? In the end I would say patterns are mainly useful as a convenient shorthand to communicate parts of the design, more than a toolbox to try and apply.

                                            2. 5

                                              Maybe I live and work in a different environment than Martin Fowler.

                                              This touches on something I’ve been thinking about for quite a while. That there are broad categories of sofware application, and that programmers typically have a perspective only on those they have actually experienced. I’m not talking here in your case, but more generally.

                                              For many new in the industry, there is a perspective that software is limited to problems in the computer and data sciences, and those involving computing infrastructure. Often these undertakings have a focus largely on what may be considered non-functional requirements, where functional requirements (and how to represent them in code) is of relatively little importance. Issues such as how to scale, be reliable, available, accessible and so forth are paramount.

                                              On the other hand, there are a large proportion of applications where how to model a complex problem domain into code is the issue. Typically a problem domain where expertise is outside of the development team. How to discover, manage and implement functional requirements into code is the core issue. It is this later world that Martin Fowler primarily lives in and writes about.

                                              Is such a categorization of any value?

                                              1. 2

                                                Possibly, but the example you give sounds to me like it intermingles the datamodel / domain structure / domain architecture with the software architecture, while the former, in my view, is mostly one of many constraints on the latter and one that can be used to ‘encode/compress’ part of the functional requirements. However, you can’t design a good software architecture without taking all functional requirements into account, while at the same time you also need to take all other requirements into account: after all, there is only one software architecture.

                                                1. 1

                                                  I think this is a good categorization.

                                              1. 2

                                                “it even comes with a built-in game engine for kids”

                                                Can you (or anyone) provide more info? I wasn’t able to find it with some quick Googling .

                                                1. 3

                                                  Yes, sorry I should have linked it originally! The link is fixed now. It’s big-bang/universe. Realm of Racket is a good book for teaching how to program by building games (sadly its not libre documentation, but it is good)