1. 2

    Preparing my presentation for Trends in Functional Programming (at LambdaDays). It’s my first time giving such a presentation and I want to do well, so I’m a bit nervous. The project is only a small part of what I’ve been doing during my PhD, but it’s the first tangible result that I’m going to go out and show someone else.

    1. 3

      I was curious what the practical implications of having an undecidable type system are. Apparently, it means you can make a program that makes the compiler crash during type checking: https://bugs.openjdk.java.net/browse/JDK-6558545

      1. 4

        I don’t think that’s a requirement. It can also mean that the type checker can go into an infinite loop (without crashing).

        1. 3

          The compiler crashing is rather a side-effect of the underlying undecidability.

          Concretely, undecidability means that there is no universal algorithm which, given a candidate Java program to compile, correctly terminates within some finite time (dependent on the input length), i.e. it may loop infinitely for some programs. It may also require infinite computing resources, such as stack space in this instance, which leads to the compiler crashing. But as @munksgaard correctly said, it can also “just” loop infintely.

          1. 3

            Practically, there is none. As soon as compilation takes too long, you’ll cancel it and rewrite the code anyway.

            1. 1

              I agree with @munksgaard and @bfieldler, crashing during type checking is just one way in which the undecidability can show.

            1. 4

              From the summary, it sounds like async is always as good or better than true threads. So when should I use a thread instead of async?

              1. 10

                I think this is a limited view. What I’m seeing here is that threads, an arguably simpler concurrency primitive, can scale quite competitively on modern operating systems.

                Therefore, while you should continue to use async/epoll etc. when there is a true need (e.g. building web facing software like haproxy, nginx, envoy, caddy), many use cases will run competitively using a simpler blocking threads model and it might be worth evaluating whether you’d prefer working in that paradigm.

                1. 7

                  Async requires separating I/O-bound tasks from CPU-bound ones (you need to specifically spawn a task on a threadpool if it’s going to hog the CPU). If you can’t reliably isolate the two workloads, you’re going to have I/O latency problems. In that case it’s better to sacrifice threads for mixed workload.

                  Similarly if you’re dealing with FFI or non-async libraries, you have no guarantee what they’ll do, so it’s best to use a real thread just in case.

                  async/await in Rust creates a single state machine for every possible await point. If your code is complex with lots of on-stack state intermixed with awaits, then this state machine can get large. Recursion requires heap allocation (because it ceases to have compile-time-fixed call graph depth, so the state becomes unbound). If your code is complex, with lots of state and recursion, a real thread with a real stack may be better.

                  1. 3

                    If I understand rust’s async correctly or rather that it follows other languages’ async/concurrency patterns, you use async to improve single thread/core performance, it allows you to continue if the thread hits something like an io wait or similar non-cpu task. Multi-threading can do this but is better when processing for durations, like a browser.

                    cli tools that are intended to do one thing and exit quickly like ls for example may degrade performance by adding threads, but may improve performance by adding async. Now if as mentioned you have a browser and you have already maxed out what a single thread can do using async it’s time to split using threads to allow you to use more of the available cpu cores to do its job.

                    golang explanation of concurrency vs parallelism: https://blog.golang.org/waza-talk

                    1. 2

                      Performance is only one metric. It is indeed true, to the first approximation, that async is not worse than threads in terms of performance. The most convincing argument for the opposite is that async might have worse latency, as it’s easier to starve tasks, but I haven’t seen this conclusively demonstrated for Rust.

                      Another important metric is ergonomics or “is programming model sane?”. In this regard, async, at least in Rust, is not a sane programming model. Stack traces do not make sense, you get extra “failure modes” (any await might not return (when the corresponding task is dropped)), and there are expressively restrictions (recursion, dynamic dispatch, and abstraction are doable, but awkward).

                      1. 2

                        To play the devil’s advocate, sync, blocking model is also far from perfect. It doesn’t handle cancellation or selection between several sources of events. That is, if you do a blocking read call, there isn’t a nice way to cancel it.

                        With async, because you need to redo the entirety of IO anyway, you can actually go and add proper support for cancellation throughout.

                      2. 1

                        When a syscall/FFI blocks, or when you might have to crunch a lot of numbers are good cases.

                      1. 9

                        My logic behind making my NixOS config repo a public one is to act as an example for others to take inspiration from. I also wanted to make it harder for me to let the config drift. It also gives me a bunch of fodder for this blog.

                        I’ve definitely used your blog as a resource in the process of learning about NixOS and using it in my own home network, so thanks for writing this stuff up.

                        1. 12

                          I’m going to eventually end up writing a book about administering NixOS servers. What kinds of things would you like to see in such a book, or as future articles on my blog?

                          1. 5
                            • the process for creating a new VM that uses NixOS is still pretty manual for me, even if I can configure it automatically once I use Nixops for the first time and it generates a key. I would like to see information in how you solve this problem on your infrastructure.
                            • Nixops itself seems kind of clumsy. I gather that there’s an upcoming 2.0 release that will change a lot of things about the software, but its still not ready to go yet. I’d be interested in reading about how to best use Nixops today; or if there are other tools for applying a nix configuration to many remote boxes. Also, is there any way to query the state of a NixOS machine to see what’s deployed on it?
                            • Hints about how to take software that isn’t yet packaged for nix and making it run on your NixOS infrastructure. Some of the services I run are apparently hard to package for Nix (peertube being a good example), and I’m not sure what I should do to work around this.
                            1. 1

                              if there are other tools for applying a nix configuration to many remote boxes

                              There’s a bunch of other tools (deploy-rs, krops, morph), each with their specific use-case and strengths.

                              I ended up using nixos-rebuild with flakes, and it works really well for me (apart from currently not being able to build the derivation remotely). Admittedly, I manage very few hosts, so I can afford to run it for each individual machine.

                              But it should be pretty easy to write a little bash script that iterates over a few hosts. Maybe even borrowing some ideas from TFA (which I haven’t yet read).

                            2. 4

                              As another point of reference, I refer to your blog posts all the time, and mostly use NixOS because of those. I’d buy your book in a heartbeat!

                          1. 4

                            Does anyone use Jami for chatting or calling? What are your experiences?

                            1. 3

                              I’ve been trying it out with a couple of crustaceans today, and at least on android, basic two-person chats seem to work well. They don’t have group conversations yet, but work is underway: https://jami.net/swarm-introducing-a-new-generation-of-group-conversations/

                              1. 3

                                Also, while you can link your account to several devices, messages sent from one device does not show up on the linked devices. Pretty frustrating.

                                1. 1

                                  It’s on the way, swarm chat give the ability to sync messages across devices (cf the article from munksgaard)

                                  1. 1

                                    Sounds good. Thanks.

                                2. 2

                                  caution: video conference are done since the beginning. There’s only 1-1 text chat though.

                                  (I got fooled too and was corrected by a Jami dev: https://lobste.rs/s/gm1xqu/gnu_jami_together_release_enriched_video#c_hxknps)

                                3. 2

                                  I tried it yesterday, there is no way to explicitly reply to a certain message and afaict the only emoji is the thumbs up emoji but besides that it was surprisingly good, it just worked.

                                  There was a sync issue at one point and I was like “this sucks I am spoiled” but it turns out the person on the other end had closed the app for a second and since it’s literally p2p it’s not surprising that the messages failed to deliver as instantly as they had been when we were both present.

                                  Looking forward to group conversations, I think using a DHT for routing, TLS for e2e connections and git for the underlying sync model is brilliant. All of this technology is super battle tested and the result seems to work quite good.

                                  Edit: feel free to add me, I have the same username here as there.

                                1. 7

                                  What about GNU Jami? It’s a truly decentralized messaging system based on OpenDHT.

                                  1. 2

                                    I’d never heard of Jami before! Definitely looks interesting. Have you tried it? What has been your experience with it?

                                    1. 2

                                      I have installed, but I don’t have anyone to chat with. I tried to chat between my phone and my PC, but it’s just a test.

                                      Anyway, I think the technology is quite interesting and promising.

                                      1. 3

                                        I’ve just created a user with the name munksgaard. Feel free to contact me on there, I’m curious to try it out.

                                        1. 3

                                          I’ve just contacted you in Danish. :)

                                          1. 2

                                            Takeaway:

                                            Jami — formerly Ring — is distributed and E2E encrypted, and you can link several devices to one account.

                                            However, while text messages sent from contacts are echoed to every device, my own messages only appear on the device from where it was sent. This is a highly frustrating experience.

                                            Also, Jami does not support groups yet.

                                            If the latter two issues were fixed, I would probably prefer Jami to both Signal and Telegram when I ditch WhatsApp soon.

                                  1. 2

                                    7⁄8 ths of an iceberg are famously below the water line and functionally invisible.

                                    Huh? Isn’t it more like 9/10?

                                    Other than that, I mostly agree with what the author is saying, although I’m missing some more actionable items in the post.

                                    1. 1

                                      What’s that processing thing they keep mentioning?

                                      1. 1

                                        processing.org (and sibling project p5.js) - there’s a flavor of this that fits in a tweet, which might be what the author is referring to

                                      1. 5

                                        This article nicely illustrates how easy the git email flow is for developers/contributors, but how about for maintainers?

                                        For instance, how do I easily apply a patch sent to me, if I use the gmail/fastmail web interface? If web-mail is frowned upon, does it work if I use thunderbird/outlook/something-else? Or does it basically require me to use mutt?

                                        How about managing different patches from different sources at the same time? Github conveniently suggests you create a local copy of the PR’ed branch, meaning you can easily change between different PRs. How does this work with git email flow? Do I just create those branches myself, or is there an easier way to do it?

                                        What about very large patches? I recently applied a code formatter to a project I contribute to, resulting in a 4MB diff. Not all email clients/servers handle arbitrarily large files well.

                                        I’ve seen enough descriptions about how git-send-email works from a contributors perspective, but I would really appreciate it, if someone would write down how they then receive and review patches in this flow.

                                        1. 6

                                          Although his workflow is obviously not representative of most maintainers, Greg K-H’s writeup was nonetheless fascinating.

                                          1. 1

                                            That does indeed look fascinating, and like it addresses some of my questions. Will take a closer look later.

                                          2. 6

                                            As long as your e-mail client/interface doesn’t somehow mangle the original message, the simplest solution would probably be to simply copy the message as a whole and then run something like xsel -ob | git am in the repository. I’d reckon this is much easier than setting up some more UNIX-like e-mail client.

                                            1. 3

                                              For instance, how do I easily apply a patch sent to me, if I use the gmail/fastmail web interface? If web-mail is frowned upon, does it work if I use thunderbird/outlook/something-else? Or does it basically require me to use mutt?

                                              You can certainly use mutt or Gnus if you want. Most projects using git-by-email use mailing lists, some are fancy with download buttons to get the patches. Most of the time you can pass the output of curl fetching the mailing list page directly to patch. Quoting the manual

                                              patch tries to skip any leading garbage, apply the diff, and then skip any trailing garbage. Thus you could feed an article or message containing a diff listing to patch, and it should work.

                                              I’ve done this as recently as this week, when I found an interesting patch to u-boot.

                                              If a mailing list or other web view isn’t available, then you can either set one up (patchwork is the defacto standard) or find a client that doesn’t break plaintext emails. Last I checked, receiving patches via Gmail’s web interface was difficult (but sometimes possible if you hunt down the “raw message” link), I haven’t checked Fastmail. If you don’t want to use a TUI or GUI email client, you can configure getmail to only fetch emails from specific folders or labels, and that will create mbox files you can use with git am.

                                              How about managing different patches from different sources at the same time? Github conveniently suggests you create a local copy of the PR’ed branch, meaning you can easily change between different PRs. How does this work with git email flow? Do I just create those branches myself, or is there an easier way to do it?

                                              Branches are lightweight, create them for however many things you want to work on at the same time. I usually don’t use branches at all, instead using stgit, and just import patches and push and pop them as needed.

                                              What about very large patches? I recently applied a code formatter to a project I contribute to, resulting in a 4MB diff. Not all email clients/servers handle arbitrarily large files well.

                                              You can breakup that work into smaller chunks, to avoid too much disruption to other patches that might be in flight. Nothing stops you from linking to an external patch, though I would probably prefer instructions on how to run the tool myself, and just forgo the patch in that case.

                                              1. 3

                                                What about very large patches?

                                                git request-pull can be used for this purpose, it generates emails with a URL from which the reviewer can pull the changes. It’s generally used for subsystem maintainers in large projects to merge their (independent) histories upstream, but it can also be used to handle large changes which would be unwieldly in patch form.

                                                1. 2
                                                  1. 1

                                                    I wondered about the same thing. After a very long unfruitful search on the internet, @fkooman pointed me to git am - Apply a series of patches from a mailbox, which pretty much takes care of everything. It would have been helpful if git-send-email.io had a pointer to that, or maybe I missed it..

                                                  1. 10

                                                    Amazing article, thank you @ahu

                                                    A few questions, if I may — and please understand that I don’t even know what I don’t know about this sort of thing, so forgive me if these are stupid or silly questions.

                                                    1. If we’re modifying our own cells to make the spike proteins, how do we stop all our cells from getting modified? Do the modified cells not replicate? Is the mRNA “used up?”

                                                    2. With the advent of these technologies, are we basically able to create vaccines on demand very rapidly, or are these techniques only applicable to SARS-CoV-like viruses (not that that isn’t enough!)?

                                                    3. Did previous vaccines modify our existing cells like this, or is this new to the mRNA technique?

                                                    4. Replacing U with psi makes our immune system ignore the protein. Is there any chance evolution will give us a virus with U replaced with psi in nature, making a virus that is just completely ignored by our immune system? That seems like it would be bad.

                                                    And finally, I am not a religious man, but…I definitely have a sense of wonder looking at how DNA and RNA work and encode information, and I am in absolute awe at what humankind can do now. As someone said, we’re halfway between apes and God and seem to be heading in the right direction…

                                                    1. 14

                                                      1: Yes, explicitly. The mRNA is “one shot”, and it degrades quickly. By design. 2: Yes and no. This technology has been tried against influenza for example, but that is a very very clever virus. But it appears that mRNA will be extremely useful for coming to the universal flu vaccine. And even if we can’t do the universal one, we could quickly print the most useful one for.. this month :-) 3: Previous vaccines injected dead or living or modified existing viruses. Some of these did the very same thing - get our cells to build proteins. That part is not new. It is just a lot more surgical now. 4: Good point, but there is happy news. The whole “architecture of life” can’t make these psi things. And viruses ride that architecture. This needs a lab.

                                                      And yes, I am not religious either but the sheer sense of AWE can be overwhelming!

                                                      1. 5

                                                        May I double-check my understanding of #4 with you? Biology is not my field but the way I think of it is that cells and organisms are basically factories for assembling/moving around/ripping apart chunks of molecules, of various types. So it sounds like this Ψ faux-RNA sequence is a piece of molecule that never appears in any living organism, but is the right shape that it slots into the “build protein” machinery and tells it to make the same thing that a U would in its place. But if you feed that RNA sequence it into the cell’s “copy DNA/RNA” machinery then it doesn’t work, because the molecular machinery doesn’t know how to handle it, and because there’s no molecular machinery in any living organism for manufacturing the Ψ piece-of-molecule and bringing it to the right place. So to teach a cell how to handle a Ψ in its genetic sequence you’d have to make a whole metabolism that could build it, tear it apart, and get it to the right places in the right amounts. Am I understanding this properly?

                                                        Pretty cool, if so. Basically sounds like genetic ROM.

                                                        1. 5

                                                          You got that right. To change this is definitely a forklift operation :-)

                                                        2. 3

                                                          Could someone artificially enhance an existing virus by replacing U with Ψ as some sort of biological warfare?

                                                          Edit: Also, thank you for making this so understandable! This was probably the best thing I’ve read all year.

                                                        3. 5

                                                          For a sense of scale on #2 at least… they mention in the article that they had to make some alterations to the protein the cell synthesizes this way to make it actually work properly and not just fold up into a mess. These alterations were based on research that other people did in 2017 on similar viruses in the family. So even if it’s “easy” to use this technology to tell cells to make new things, there’s more than a little finesse in making things that do what you want.

                                                        1. 7

                                                          While I know that this will always be up for debate, I think the author is wrong to suggest that parsing should not return a Result. Always verify when you’re handling stuff coming from outside of your program. Also, all those unwraps can be avoided by using ?.

                                                          Other than that, I think there is valuable feedback here.

                                                          1. 2

                                                            Always verify when you’re handling stuff coming from outside of your program

                                                            The case mentioned was not for parsing the HTML – it was for “parsing” the CSS selector the author had hard-coded into the program. Which raises a question about whether it really was “coming from outside your program”. From the perspective of the scraper authors, sure. From the perspective of the programmer using scraper, though, it wasn’t.

                                                            I suspect the static-typist’s ideal alternative would be a language that could express grammatically-valid CSS selectors as a type and thus expose a Selector interface that just statically rejects anything that isn’t a CSS selector[1]. But as a matter of programmer convenience unless/until that ideal world arrives, there might be room to explore alternatives that don’t require the extra Selector::parse for a trusted value.


                                                            [1] This is mostly humor. One can write parsers and validators for the grammar of CSS selectors, but would gain very little real-world safety from doing so – the risks associated with untrusted input would not be mitigated by a requirement that all selectors be grammatically valid, any more than SQL injection vulnerabilities would be mitigated by only accepting user input that is grammatically-valid SQL.

                                                            1. 6

                                                              This is an important lesson to learn as soon as possible when working with failures encoded in return values. It’s 100% acceptable, and in fact more correct, to panic/throw/crash on errors that are the fault of the author of the function.

                                                              You (generally, as a rule of thumb) return an error value if the inputs of the function cause failure or if one of the dependencies of the function failed (database returned a duplicate key error, etc).

                                                              IMO, of course.

                                                              I like to reference the last section of this page: https://dev.realworldocaml.org/error-handling.html

                                                              1. 2

                                                                Another important lesson is: when you disagree with someone, don’t automatically assume it’s because they’re ignorant and don’t know what you know.

                                                                It’s entirely possible that someone could be perfectly well familiar with your preferred pattern and still disagree that that pattern is the best possible one for this case.

                                                                This is, also, why functional programming gets kind of a bad name – the built-in assumption of “everything about this is so obviously objectively superior that only someone who doesn’t know about it would disagree” is off-putting.

                                                                1. 1

                                                                  I’m not exactly sure why I deserved this response. I thought it was clear that I was simply advocating for a particular “philosophy” around when to return an error and when to throw an exception.

                                                                  Yes, I framed it as a “lesson” that one might learn from working with languages that return error values, but l feel like I left room for myself not being omniscient (the part where I said “IMO”).

                                                                  It was also agreeing with you. Since the person using the library was hard coding the values used for selectors, it’s better to unwrap and crash than to bubble up an error value that only says “the author of this function wrote it wrong”.

                                                                2. 1

                                                                  Thanks for linking that page - after reading it I think it makes some great points that are certainly applicable outside of OCaml. The final sentence makes for a nice TL;DR:

                                                                  In short, for errors that are a foreseeable and ordinary part of the execution of your production code and that are not omnipresent, error-aware return types are typically the right solution.

                                                            1. 1

                                                              One thing that’s still not supported (as far as I understand) is display mirroring.

                                                              1. 4

                                                                GNOME and KDE both support display mirroring.

                                                                1. 1

                                                                  Ah, I guess you’re right. I am using sway, so I wouldn’t know.

                                                              1. 2

                                                                currently supports packages hosted on GitHub and GitLab.

                                                                Locking into git is a bad idea… but locking it into GitHub and GitLab only is even worse! I can’t see why namespacing should have anything to do with repo URLs, and why it must not be possible to self-host packages. OPAM supports multiple URL schemas including git, mercurial, darcs, and HTTP.

                                                                Also, I wonder if Poly/ML support is feasible. It’s the only one that has both a native code compiler and a REPL. MLton and MLKit have their strong sides, but no REPL.

                                                                1. 1

                                                                  Poly/ML doesn’t use mlb format. The way I’ve considered handling both is to have a pre-processing step that maps a standard configuration (possibly mlb) to both mlbs and Poly/ML style includes.

                                                                  1. 1

                                                                    Yeah, I was thinking of something like that. Then again, I wonder if Poly/ML’s maintainer is against .mlb, or a patch for supporting both styles can be accepted.

                                                                    And then there’s a third option: a retargetable equivalent of OCaml’s findlib, separate from the package manager.

                                                                    1. 2

                                                                      A quick search doesn’t turn anything up. These programs have existed for so long I assume David has a reason not to have added mlb support, but I can’t remember seeing anything.

                                                                      To my surprise (I must have forgotten), SML/NJ has their own style too: CM files.

                                                                      So for now, a pre-processing step is probably the least friction approach to supporting the major implementations.

                                                                  2. 1

                                                                    I don’t think that anything inherent to the design of either smlpkg or futhark-pkg (which smlpkg is built on) locks you to GitHub og GitLab. They are supported right now, for pragmatic reasons, but there’s nothing stopping you (or someone else) from adding another repository, as far as I’ve understood.

                                                                    Additionally, I’ll point out that this package manager is pretty compiler-agnostic. The current implementation might only compile using MLKit or MLton, but the executable can be used to manage packages for projects written for any distribution of ML.

                                                                    Finally, I’ll point out that MosML also has both a native code compiler and a REPL.

                                                                  1. 4

                                                                    I just use pass + oathtool.

                                                                    For instance, in “2fa/github”, I store:

                                                                    oathtool --totp --base32 MY_KEY
                                                                    

                                                                    In my .bashrc I have:

                                                                    eval $(pass 2fa/$1)
                                                                    

                                                                    and then when I need a code I use:

                                                                    2fa github
                                                                    

                                                                    I should write a blog article to explain that. :)

                                                                    EDIT: I just did.

                                                                    1. 2

                                                                      How is this different from using the pass-otp extension to pass? I don’t have experience with either, but I use pass to store my regular passwords.

                                                                      1. 1

                                                                        I just didn’t know pass-otp. It also uses oathtool and zbar, and it is packaged for Arch. I will add it to my post.

                                                                      2. 1

                                                                        Using pass-otp, it’s just a matter of:

                                                                        pass otp 2fa/github

                                                                        And storing the otpauth string in he pass file:

                                                                        otpauth://totp/GitHub:USERNAME?secret=SECRETSECRET&issuer=GitHub

                                                                        1. 1

                                                                          worth noting that the pass iOS app can scan QR codes to add the otpauth to your entries, quite neat

                                                                      1. 5

                                                                        While I really enjoy Rust, and believe that it is a language for the future, I am also concerned about the pace that Rust is developing at. I fear that people who only occasionally work with the language are scared away by the rapid changes in idiomatic style and eco-system (which web server library should I use this month?), and I also fear that Rust will become bloated with so many features that you never truly understand the language. One of the reasons I like Standard ML so much, is that it is simple enough that you can learn the whole language in a few days, while still being powerful enough that you can express anything succinctly.

                                                                        I have the utmost respect for the Rust developers, and I’ll probably continue to use the language, I just hope that the developers are mindful that not everyone will see constant change in the language as a strength.

                                                                        1. 8

                                                                          The example you picked, web servers, is because async/await was just recently stabilized. If you’re using Rust for writing web servers, then you should absolutely be prepared for churn and rapid evolution, like any good early adopter.

                                                                          If you picked a different topic, like say, automatic serialization, then it’d be fair to say that it has been quite stable without much change for years at this point.

                                                                          Language features are added at a more rapid pace than, say, Go, but it’s not that often. Since stabilization of ?, what’s considered “idiomatic” has largely remained unchanged.

                                                                          1. 5

                                                                            I think Rust is becoming simpler over time. There’s a recurring pattern of:

                                                                            Q: Why doesn’t this code compile!?

                                                                            A: Complex reasons.

                                                                            …new rust comes out…

                                                                            A: It compiles now!

                                                                            So for outsiders it may seem like Rust just keeps adding and adding stuff, but the insider perspective is that Rust takes stuff that has been postponed before 1.0, unfinished, or clunky, or difficult to use — and finishes it.

                                                                            I used to have to write lots of error handling boilerplate. Now it’s mostly a single token. I used to have to strategically place extra braces to fit inflexible borrow checking rules. Now I don’t have to. I used to need to wrangle with differences between ref and & regularly. Now it’s pretty much gone. Async code used to require a lot of move closures, Arc wrapping, nesting Either wrappers. Now it’s almost as easy as sync code.

                                                                            1. 3

                                                                              I agree that using the language has become simpler over time. I think I’m more worried that understanding the language becomes harder. You definitely need to wrap your head around more concepts in order to have a full understanding of the language now than you did when 1.0 came out. Yes, that made some things more verbose and clunky, it also meant that all code was explicit and easy to follow.

                                                                              1. 6

                                                                                Yes, that made some things more verbose and clunky, it also meant that all code was explicit and easy to follow.

                                                                                Rust has a much more mature approach to explicitness than “tedious noisy syntax == easy”: https://boats.gitlab.io/blog/post/2017-12-27-things-explicit-is-not

                                                                                For example, Go’s error handling famously much more “explicit” than Rust’s, but it doesn’t make code easier to follow. It adds code that distracts. It adds code that needs to be checked for correctness. Rust’s terse error handling is more robust: it can’t be as easily ignored/forgotten, and catches more types of issues at compile time.

                                                                                There is a nuanced relationship between simplicity of the language and ease of understanding programs, so I generally disagree with the implied conclusion that bigger, more featured languages are more difficult to grasp. (simple = consisting of few parts) != (simple = easy to understand). Brainfuck is the extreme example of this. Another example is C, which is usually considered small and simple, but “is this a valid C program free of UB?” whooo boy. That’s a difficult question.

                                                                                I’m afraid that modern C++ has been such a shining beacon of ever-growing unfixable accidental complexity that it has put a stain on the mere notion of language evolution. I think C++ is an outlier, and languages aren’t doomed to end up being C++. PHP, JS, and Python have been evolving, and (ignoring Python’s migration logistics) they ended up being much better than what they started with.

                                                                                1. 2

                                                                                  Hm, thant kinda started a thought in my mind, that Rust might be getting into a “hard to write, easy to read” territory, as a kinda polar opposite to how I consider Perl to be “easy to write, hard to read”

                                                                                  1. 1

                                                                                    I don’t know a lot about Go, but from what I do know, I wouldn’t call Go’s error handling more “explicit” than Rust’s. It’s definitely noisier, but it’s also very weak.

                                                                                    That being said, I agree with most of what you’re saying.

                                                                              2. 3

                                                                                Putting my library team hat aside, yes, I like Standard ML for the same reasons. But it’s effectively a “dead” language outside of academia. Ocaml might be a more fair measuring stick in terms of comparing apples to apples. Although, Ocaml has been around for a long time, so I suppose there’s no truly fair comparison.

                                                                                So I guess, here’s a question for: what would you do differently? What would you give up?

                                                                                1. 3

                                                                                  So I guess, here’s a question for: what would you do differently? What would you give up?

                                                                                  I don’t know, and I certainly don’t know that what the Rust team (you included, thank you for all your hard work) is doing is wrong either.

                                                                                  I think there’s a case to be made for a Rust–: Essentially C with algebraic data types, pattern matching and the borrow checker. Perhaps also traits. But something that could be standardized and implemented by other compilers, perhaps even formalized in a proper spec (disregarding RustBelt for now).

                                                                                  I realize that much of the complexity in Rust comes from the fact that Rust is trying to do everything in the right way: If we have borrow checking, it should also cover concurrent code; if we have concurrent code, we should try to use an asynchronous programming model, because that’s the way these things are done nowadays; if we have traits, we should also be able to have lists of trait objects; if we have trait objects, the syntax shouldn’t hamper us; if we have algebraic data types, we should be able to use monadic return types (Option, Result) without too much syntactic overhead; and so on. Every time we encounter a problem because of some recently added feature, we introduce a new feature to handle that. All of the steps make sense, but the end result is increased complexity and a bigger language. But perhaps that is necessary.

                                                                                  1. 4

                                                                                    Thanks for the response. My thinking is similar.

                                                                                    With respect to a spec, FYI, Ferrous Systems is starting down this path with Sealed Rust.

                                                                                    1. 4

                                                                                      I heavily use (and teach) concurrency with rust, but I almost never use rust’s async functionality. It’s an abstraction that both slows down implementations and makes codebases worse for humans for many things other than load-balancer-like workloads where the CPU costs per kernel process scheduling decision are very very low. None of the publicly available async schedulers pay any attention to modern scheduler theory, and they lock you into suboptimal latency-throughput trade-offs.

                                                                                      Async is a sharp knife to really only be used for load-balancer-like workloads.

                                                                                      1. 1

                                                                                        I’m interested in hearing more about this, if you have the time. I’m sure you’ve got sufficient experience with these trade-offs in building Sled, but I’ve found the publicly available schedulers in Rust to be excellent mechanisms to structure concurrent programs and tasks—better than plain threads, in my experience.

                                                                                        1. 8

                                                                                          I think it’s a beautiful way to compile certain simple state machines. But they have made a lot of decisions that make it difficult to actually build a high quality scheduler that can run them, due to needing to shoehorn everything into the Poll interface.

                                                                                          For instance, it’s generally well accepted that for request-response workloads you want to prioritize work in this order:

                                                                                          • first run things that are ready to write, as they signify work that is finished
                                                                                          • things that are ready to read, as they are work that has been accepted and the timer is ticking for
                                                                                          • only accept based on a desired queue depth based on your latency/throughput position. If you care about latency above everything, never accept unless all writes and reads are serviced and blocked. If you care about throughput above all else, you want to oversubscribe and accept a lot more work to reduce the frequency that your system bottoms out and has no work to do. You don’t want to accept work that you’re not servicing though if latency is a priority, and you want a smaller TCP backlog that will fill up and provide backpressure for your load balancer so it can do its job.

                                                                                          With Rust, it’s impossible to write a general purpose scheduler that does the right thing here, because there is no way to tell a Context / Waker that your task is blocked due to a write/read/accept-based readiness event. You have to write a custom scheduler that is aware of the priorities of your workload. You have to write your own Future that also feeds back information through something like a thread local variable, out-of-band.

                                                                                          There are very few reasons to use an async task. Memory usage is not one of them, because async tasks usually compile to about the same size as a large stack anyway, easily taking up megabytes of space. It’s the most pessimistic stack. Having it precompiled and existing separately from something that runs on a stack also has cache implications that are sometimes pretty negative compared to a stack that is always hot in cache.

                                                                                          The one time you could benefit from async tasks are when you are doing almost 0 compute per request, because only then do context switches become measurable compared to the actual workload. However, this effect is distorted by the way that people tend to run microbenchmarks which don’t do any real work, making it seem like the proportion of CPU budget consumed by context switches is large, but for anything de/serializing json, the context switch is already noise in comparison.

                                                                                          Then there are the human costs. These dramatically improved with async/await. But the impact of blocking is high, and all of the non-blocking libraries try to look as similar to blocking std stuff as possible for usability, but this is also a hazard that increases the chances that the blocking std version will be used, causing your perf to tank due to no longer being concurrent. Compared to threads, there are more implementation details that are in flux that will change performance over time. With threads, you can reason about the system because it’s familiar and it’s the same one many people have been working against for their entire careers as systems engineers. The need to use profiling tools at every dev stage dramatically rises any time you use async stuff because of the various performance pitfalls that they introduce.

                                                                                          An async task is a worse thread in almost every way, I find. They remove control, introduce naive scheduling decisions, bloat the memory use, pollute cache, make compilation slower, increase uncertainty, lower throughput due to shoehorning everything through Poll, increase latency by making it difficult to avoid oversubscribing your system, etc etc etc… Only good for load balancers that don’t do any CPU work anyway, which is exactly what the company behind tokio does. async-std got hijacked by ferrous to try to sell more Rust trainings, because it introduces so many hazards that you need to be educated to avoid. Those 2 businesses may not be aligned with yours.

                                                                                          1. 2

                                                                                            Thanks for taking the time to write this all up! I’m still processing it. I’ll try to dig into each point on my own. What you said is pretty valuable!

                                                                                1. 2

                                                                                  I use SSH and TRAMP quite a lot to access a GPU cluster at the university. The cluster is managed by the IT-admins, and for some reason they’ve decided to disallow ~/.ssh/authorized_keys to work properly. Frustrated by having to input my randomly generated 20 character long password dozens of times per day, I recently found out about auth-source.el, which let’s you hard-code the password for a given host so that you don’t need to input it through TRAMP all the time.

                                                                                  Now, I have the following in ~/.emacs.d/authinfo:

                                                                                  machine remote_server_name login my_username password my_password port ssh
                                                                                  

                                                                                  and a simple addition to my .emacs:

                                                                                  (setq auth-sources '("/home/munksgaard/.emacs.d/authinfo"))
                                                                                  

                                                                                  Of course, this is horribly insecure, and really quite silly, but in this case convenience wins over security.

                                                                                  I have yet to find a similar solution for regular ssh in the terminal. sshpass seems quite finicky for some reason.

                                                                                  1. 5

                                                                                    Emacs has built in encryption support for all files, including its authinfo files. C-x C-w your authinfo to /home/munksgaard/.emacs.d/authinfo.gpg and source that instead, and when you try to access something Emacs should ask you for the decryption password.

                                                                                    I’ve only used this for email information so ymmv, but I don’t see why it shouldn’t work.

                                                                                    1. 1

                                                                                      That’s very helpful indeed, thank you!

                                                                                      1. 2

                                                                                        Nææh, en DIKUfant?

                                                                                        Get a hold of a hardware key like Yubikey and this solution gets even more awesome. With one of those you can store your GPG key in hardware, and set up the Yubikey so that decryption/signing requires you to touch a button on the fob.

                                                                                        You can derive an SSH key from your GPG key, drop that into ~/.ssh/authorized_keys and now all SSH access will require a touch. This way you can avoid the security issues with having your SSH private key in an unlocked keyring/ssh-agent, as an attacker will need to have physical access to your computer to be able to decrypt files/SSH. It also works with gpg-agent/ssh-agent forwarding.

                                                                                        1. 1

                                                                                          I’ve been considering it for a while, but I’m confused about which model to choose, and how to know if it interacts well with my Linux install. But I guess I’ll have to investigate more.

                                                                                          1. 2

                                                                                            I’ve used it on Linux for some 3-4 years now, works great. I’m currently using a YubiKey 4 series (full size USB A) and a YubiKey 5 (small size, USB A; permanently occupying a port in my laptop).

                                                                                            1. 1

                                                                                              Any idea if it works with fully-free distros? I’ve been looking into them as a better way to manage my keys, but don’t want to spend the money if it’s not definitely going to work. Also if you know of any blog posts about people’s experience please slide them my way; I’ve struggled to find many from Linux-y individuals (as opposed to mainstream Windows-oriented tech sites).

                                                                                              1. 1

                                                                                                Yes, everything is free as in libre. Works just fine on Debian. The Yubikey implements a smartcard and also works as a USB HID (for HOTP/U2F).

                                                                                  1. 9

                                                                                    I like the idea of Nix, even though I haven’t used it much at this point. However, all this extra tooling and setup concerns me. Niv, lorri, direnv, naersk? Separate configuration files for each? Isn’t this what we we’re supposed to be able to avoid by using Nix?

                                                                                    1. 8

                                                                                      However, all this extra tooling and setup concerns me.

                                                                                      There is a lot of extra tooling. However, the nice thing is that if you pass such a project to others, they could still just use nix-build/nix-shell. No need to use direnv or lorri.

                                                                                      Niv

                                                                                      niv has some overlap with the proposed the Nix flakes functionality, which is currently under development:

                                                                                      https://github.com/tweag/rfcs/blob/flakes/rfcs/0049-flakes.md

                                                                                      With flakes, Nix allows you to lock dependencies, etc.

                                                                                      1. 3

                                                                                        Nix flakes hasn’t been released yet. Niv has.

                                                                                        1. 4

                                                                                          Sure, this was not a criticism. I also use Niv. I just wanted to point out that while you now need an extra tool such as Niv, Nix will have similar functionality in the future. So, Niv will not be necessary as additional tooling in the future.

                                                                                        2. 3

                                                                                          The flake branch is also caching the evaluation outputs which makes lorri mostly redundant.

                                                                                        3. 4

                                                                                          They each do one thing very well and work to integrate with eachother.

                                                                                          • Niv is like git submodules but maintainable and uses the nix store to your advantage
                                                                                          • Lorri caches nix-shell generations (because making your shell/editor lag for ~5-10 minutes at worst is unacceptable)
                                                                                          • Direnv exposes lorri’s data into your shell
                                                                                          • Naersk parses Cargo manifests and puts every rust dependency in the Nix store

                                                                                          It looks like a lot from the outside, but it’s not actually that much from the inside. It’s about building on top of things in logical increments. Traditional package managers have the same kind of problem (but do a far worse job of handling it in my opinion) where they need libraries and tooling to handle Rust, Go, Haskell and Perl programs. Nix just lets the user handle those things the way they want. I could have used cargo2nix or something, but naersk is significantly less setup/configuration because it parses the cargo manifest/lock file.

                                                                                          1. 2

                                                                                            Naersk parses Cargo manifests and puts every rust dependency in the Nix store

                                                                                            Well, it realizes all the dependencies in together in the Nix store (in a single store path). This is better than buildRustPackage, but it is not optimal, since compiled dependencies cannot be reused between projects and/or of the dependencies of the projects change.

                                                                                            buildRustCrate realizes every crate (dependency) into a separate store path, meaning that compiled dependencies can be reused between projects. Also, if you change a dependency of a project, only that dependency is (re)compiled. I’d recommend crate2nix to use buildRustCrate. By using IFD, crate2nix can also be used to parse Cargo.lock without explicit code generation.

                                                                                            The downside of buildRustCrate is that it reimplements Cargo in Nix (it does not use Cargo at all), this can sometimes result in subtle differences in behavior compared to Cargo and some esoteric features are not implemented. But generally it results in very fast builds.

                                                                                            1. 2

                                                                                              Thanks for writing this blog post, it reminded me of all the steps we currently have to go through to setup new projects. IMO this is just the beginning and a lot of trimming down will happen. If I had to guess the future:

                                                                                              • Niv and Lorri become redundant because of Nix flakes
                                                                                              • Maybe somebody will rewrite a minimal version of direnv that only handles nix projects.
                                                                                              • Naersk will be part of nixpkgs, or be more easily composable thanks to flakes. If it’s part of the flake registry it will be easier to discover.
                                                                                              • A nix init command that auto-detects the project layout and fills with sensible defaults.

                                                                                              Ideally the default surface of nixified projects would be 2 files so it’s not too intrusive.

                                                                                          1. 2

                                                                                            “Pure LISP” never existed - LISP had assignmentand goto before it had conditional expressions and recursion.

                                                                                            I’m not sure about the choice of the word “before”. As far as I remember, conditional expressions were one of the core concepts (ie if) of lisp, even before it’s first implementation.

                                                                                            1. 1

                                                                                              Yeah, I was a bit confused by this statement as well. Skimming the original paper, McCarthy mentions both COND (a generalized conditional expression) and recursive functions, but also mentions that Lisp contains a

                                                                                              “program feature” [that] allows programs containing assignment and go to statements in the style of ALGOL

                                                                                            1. 2

                                                                                              I’m a little miffed that, five years or so after last trying to use CL seriously, Quicklisp is still not the de-facto package system and still does not have easy packages in every Linux distro.

                                                                                              1. 4

                                                                                                I’m pretty sure Quicklisp is the de-facto package system. ASDF is the build system, and is used by quicklisp. See also the question “How is Quicklisp related to ASDF?” here: https://www.quicklisp.org/beta/faq.html

                                                                                                1. 3

                                                                                                  Perhaps what @icefox means (and what I sometimes long for) is that implementations do not ship with it in their image. This would make using packages in scripts a lot easier.

                                                                                                  1. 2

                                                                                                    Is that partly due to the fact that Quicklisp blurs the line between upgrades to Quicklisp itself, and upgrades to the collection of libraries it can install? To be clear, I used to write CL a decade ago, but I’ve not kept up; if my question doesn’t make sense, please call me out on it.

                                                                                                    1. 1

                                                                                                      I don’t know. I do not feel that line is very blurry :-)