Threads for vlmutolo

    1. 15

      You had me in the first half, not gonna lie!

      Elixir is made up of very delightful layers of functionality. One time I wanted to submit a proposal to make the pipeline operator more general. I wanted to be able to pass in an underscore so you could specify where the piped argument would go. Something like 1 |> subtract(2, _). I was able to download the repo, find the code I needed to change and fix up the test cases in like an hour or two. It was very readable and easy to build. Would recommend checking it out.

      1. 9

        There is built in way for that using Kernel.then/2 function in form 1 |> then(&subtract(2, &1)). I also have created magritte library that extend |> to support 1 |> subtract(2, ...) form.

        1. 4

          Magritte, such a good name for a pipe library 😍

          1. 4

            The library tagline on Hex is (of course):

            Ceci n’est pas une pipe - extended pipe operator

        2. 1

          That’s awesome! I didn’t think of replacing the operator. I figured that wouldn’t be an option.

      2. 2

        I’ve had that same operator idea since forever. The restriction on where the argument goes with the pipe is relatively easy to work around but never feels general enough.

        1. 4

          We’re in good company! There is a SFRI that does this for scheme:

          https://srfi.schemers.org/srfi-197/srfi-197.html

      3. 2

        I had that same pipe/underscore idea! It seems so arbitrary that the first argument is always the one filled in by the pipe.

    2. 1

      Are there any papers or posts that describe the current Pijul data model? I’d be interested in reading about the underlying format. I know it’s undergone some revisions over the past few years.

      1. 1

        No, not really. The underlying format is complicated, mostly because it relies on Sanakirja, which is itself complicated.

    3. 4

      What is the reason to not def send_emails(list_of_recipients, bccs=[]):?

      From my Clojure backend I wonder what is sending the emails (and how you can possibly test that unless you are passing in something that will send the email), but perhaps there is something I miss from the Python perspective?

      1. 17

        Hi! Thanks for reading! 🙂

        The answer is mutable default arguments; if you’re coming from Clojure, I can see why this wouldn’t be an issue 😛

        When Python is defining this function, it’s making the default argument a single instance of an empty list, not “a fresh empty list created when invoking the function.” So if that function mutates the parameter in some way, that mutation will persist across calls, which leads to buggy behavior.

        An example might be (with a whole module):

        from sendgrid import actually_send_emails
        from database_functions import get_alternates_for
        
        def send_emails(list_of_recipients, bccs=[]):
          # suppose this app has some settings where users can specify alternate
          # addresses where they always want a copy sent. It's
          # contrived but demonstrates the bug:
        
          alternate_recipients = get_alternates_for(list_of_recipients)
          bccs.extend(alternate_recipients) # this mutates bccs
        
          actually_send_email(
            to=list_of_recipients,
            subject="Welcome to My app!",
            bccs=bccs,
            message="Thanks for joining!")
        

        What happens is: every time you call it without specifying a second argument, the emails added from previous invocations will still be in the default bccs list.

        The way to actually do a “default to empty list” in Python is:

        def send_emails(list_of_recipients, bccs=None):
          if bccs is None:
            bccs = []
          # rest of the function
        
        1. 16

          This is horrifying. TIL, but I almost wish I hadn’t.

          1. 3

            It makes sense outside of the pass-by-reference/pass-by-value dichotomy but it’s still a massive footgun.

        2. 2

          As for how you test it, it’s probably with mocking. Python has introspection/monkey patching abilities, so if you relied on another library to handle actually sending the email (the example above I pretended sendgrid had an SDK with a function called actually_send_email, in Python you would usually do something like

          # a test module
          from unittest.mock import patch
          
          def test_send_emails(): 
            with patch('my_module.sendgrid.actually_send_email') as mocked_email_send:
              to = ['pablo@example.com']
              bccs = ['pablo_alternate@example.com']
              send_emails(to, bccs)
              mocked_email_send.assert_called_once_with(
                to=to,
                subject="Welcome to My app!",
                bccs=bccs,
               message="Thanks for joining!")
          

          This mucks with the module at runtime to convert sendgrid.actually_send_email to instead record its calls instead of what the function did originally.

          docs here

      2. 9

        It’s not about sending emails, but the mutable default argument: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

        Python will initialize the function argument variables on module import, rather than whenever the function is called.

    4. 6

      The embedded computation looks like a trap which will make papers harder to reproduce. Is there support for directly embedding scripts from other languages?

      1. 4

        There are built-in ways to use JSON data. I think for complex or expensive computations, users could leverage their preferred language, output JSON, and then pick it up as input in Typst.

    5. 2

      What’s the editor support like? I assume the best thing to do would be create a tree-sitter grammar for this, unless it already exists?

      1. 4

        Not an answer to your question, but related: they also develop an online collaborative editor that looks very interesting: https://typst.app/

      2. 3

        I suspect that the moment Helix supports this format, I will start converting all my notes from Markdown.

        I absolutely love the look of this, it seems to be everything I’ve dreamt of in a LaTeX replacement. The web UI is really nice too, although it seems to be struggling a bit under the load post launch announcement.

        1. 1

          Are you writing lots of markdown in Helix? Whenever I try, I get annoyed by the lack of soft wrap. I suspect I’d have the same problem writing LaTeX or Typst with Helix.

          1. 4

            Helix has soft wrap in HEAD since #5893 was merged two weeks ago. If you run nightly, just add editor.soft-wrap.enable = true to your config. Else of course wait for the next release, though I don’t know if there are plans yet for when it will drop.

            Oh, and according to (nightly)hx --health, Helix doesn’t support Typst yet. I guess it will be added quickly once there is a tree sitter for it.

            1. 1

              Oh wow, that’s great news! This could definitely change my workflow.

          2. 1

            Personally I always use :reflow or pipe to fmt to hard wrap lines when I write Markdown. But it’s also good to hear that soft wrap is coming soon!

      3. 1

        Nonexistent :’]

        Honestly the syntax is not the worst thing to write without highlighting, but I’d love an $EDITOR plugin.

    6. 3

      It’s weird to see a Rust UI post that doesn’t at least mention Druid and related projects. Raph Levien and others have spent a long time thinking about why Rust GUI is “hard” and the right data model to tackle the problem.

      1. 1

        Raph’s “Ergonomic APIs for hard problems” keynote at RustLab 2022 is also worth a look if you haven’t seen it (recording, slides).

    7. 2

      To me it’s not very clear what problem does Helix try to solve and what are the shortcomings of Vi/Vim/Neovim for solving that problem? I admit that I did not read very carefully the documentation of this project but I was expecting to get that information from the front page of the project.

      1. 11

        I think the four axis of improvement are:

        • better editing model (see https://kakoune.org/why-kakoune/why-kakoune.html#_improving_on_the_editing_model for why it is better)
        • code editor, rather than text editor: syntax tree is used for “text” editing and navigation (programming language aware “text objects”), built-in LSP for semantics-aware features
        • more out-of-the-box: default config is reasonable, more features are built-in, editor tries to help the user by providing more introspective contextual help on how to use the thing.
        • more hackable, by virtue of being implemented in a collaboration-friendly language.
        1. 2

          Sorry but I’m not convinced that those are real improvements. Let’s take them one by one. First, I see no numbers to hold the claim that the new model is better than the old model. The thing that they give as an example in the link can be done in Vim with Visual Mode, more or less. So I would take it as a proposal for improvement, not as an actual improvement. Second, I would be interested in the complexity of implementing such a code editor. My guess is that it needs to “recompile” the code every time you make a change so that it can “reason” about your changes. This looks to me as an overkill for large files or large projects. More than that, old systems cannot afford this type of “recompilation”. Third, I don’t understand your point here. Why more is better? Fourth, call me unagreeable, but I don’t buy the Rust propaganda. Every project in whatever language it is implemented is “hackable” and all programming languages are “collaboration-friendly”. I’m curious what you think about this.

          1. 9

            I am not trying to convince you, you do sound like a person quite happy with vi, and that’s perfectly fine! It feels like helix solves the problems you personally don’t have. But I am pretty sure these are real problems for quite a few people (and I personally definitely acutely feel all of them).

            Re editing model: I personally wouldn’t rely on numbers to make a decision here. To me, it’s blindingly obvious that what kakoune/helix are doing is much better. I’ve also heard enough stories from vim users who tried kakoune and got addicted. But, if you want numbers, it is better than vim at vimgolf: https://news.ycombinator.com/item?id=29984176

            Re code editor complexity&performance: we’ve been doing that since early 2000’s in Java perfectly fine, there are no insurmountable technical or performance challenges there. It’s just due to open source micro economical reasons that no one actually bothered to go beyond regexes until Microsoft did the thing (see why LSP). As another data point, I started solving IDE problem for Rust in 2015, and by 2021 it was thoroughly solved, twice (by IntelliJ Rust and rust-analyzer). Some docs for that are in https://github.com/rust-lang/rust-analyzer/blob/master/docs/dev/architecture.md, https://rust-analyzer.github.io/blog, https://www.youtube.com/playlist?list=PLhb66M_x9UmrqXhQuIpWC5VgTdrGxMx3y)

            Why do that? To be honest, for me the biggest reason is aesthetics: everytime someone opens an .html file in a text editor, html gets parsed with regexes, and that’s just horrible. See why an IDE for the list of most impactful things semantic understanding of the language provides.

            Every project in whatever language it is implemented is “hackable” and all programming languages are “collaboration-friendly”. I’m curious what you think about this.

            I do think that there are various degrees of collaboration-friendliness. On this axis, I think Rust (and Go as well, though I don’t have direct experience with that) is ahead of the average by a wide margin. One significant contribution is tooling: git clone $rust_repo && cd $rust_repo && cargo test works for 90% of rust projects without extra setup. For most other languages, it is “go read README to learn how we do things here, and then spend some time creating a non-hermetic dev environment”. The second significant contribution is that Rust is much better at interfaces (as “software components”, not as “dynamic dispatch”). This is most visible if we compare memory management between Rust and other sans-GC languages. In something like C++, every codebase has certain rules for lifecycles of objects, and its up to the programmer to ensure that distant parts of code agree on who frees stuff and when. This is a very significant chunk of project-specific knowledge, and is a barrier for contribution. In Rust, there are also project-specific rules about lifetimes, but then the compiler just tell you about them, you don’t need to bug the maintainer to learn how things work. But this goes beyond just memory: module&crate system is build around library as a first-class concept, different from a module, which helps tremendously with slicing large projects into independent chunks, and keeping the separation in place throughout refactors.

            On a more anecdotal note, IntelliJ Rust and rust-analyzer are similarly very contribution friendly projects, but setting that up with Rust was loads easier than with Kotlin, as gradle is a trainwreck.

            1. 1

              you do sound like a person quite happy with vi

              It doesn’t matter very much what editor I use, because I try to view the problem from an objective perspective rather than entering the war on text editors. If we fight over which editing style/mode is better based on user preferences then what we are doing is politics, not science.

              Complexity and performance are crucial aspects in developing software. It didn’t work “perfectly fine” because since 2000 a lot of hardware was deprecated to leave space to solutions that did not take into account complexity and performance as their fundamental problems. See Google et al. for reference. The micro-economical factors that you are talking about are there for a reason, i.e. to let everyone use the software on whatever software/hardware stack they want. Aesthetics is a second tier problem in this case.

              Your collaboration-friendliness point is interesting. On one hand you can argue that it’s easier for users to “just use” the project as-is without making modifications to the project, and on the other hand you have people that want to make modifications to the project and they need to go to the README in order to actually understand how the project works under the hood. It depends very much on who is your audience. But I would argue that the idea of open-source works better with the second approach.

          2. 1

            Second, I would be interested in the complexity of implementing such a code editor. My guess is that it needs to “recompile” the code every time you make a change so that it can “reason” about your changes.

            Helix uses tree-sitter under the hood, which is a fairly new project designed to enable incremental, fallible parsing. Basically, on every keystroke, Helix (through tree-sitter) efficiently updates the syntax tree it has parsed. There’s no compilation involved. Edits and selections work on ranges of text that represent pieces of syntax, like a whole function.

            more out-of-the-box: default config is reasonable, more features are built-in, editor tries to help the user by providing more introspective contextual help on how to use the thing.

            Third, I don’t understand your point here. Why more is better?

            More is better because it lowers the barrier to entry for picking up the editor. I personally don’t enjoy spending a long time figuring out what Vim plugins to use and developing a config file hundreds of lines long just to get reasonable defaults that come standard with, e.g. VSCode and now Helix. My Helix config file is like four or five lines of TOML that change two config options.

    8. 6

      Out of curiosity:

      Say I need a ordered list of things. Multiple actors (threads, coroutines, whatever) will hold references to things and the list, and will add new entries either before or after the entry they hold. The insertions must be fast and avoid memory copy. A different type of actor will periodically read the list, so order must be maintained. It has to all be in memory, and in process, because latency must be minimum.

      What would the rust-native solution for this?

      Update: just to clarify, I don’t have a actual problem that matches to this constraints, I just tried to come up with something that would be optimally solved with linked lists. Just a thought experiment =)

      1. 6

        Could you describe a problem that leads to this sort of set of requirements? My instinct upon hearing this is to dig deeper and see if the problem admits an entirely different kind of solution.

        1. 2

          Oh, there is no problem, I just tried to come up with a set of constraints that would be best suited to be solved with a linked list.

          1. 2

            Each actor has a local sorted vector. The reader thread iterates over them for a total ordered view.

            1. 1

              Why three?

              1. 2

                Oh, I said three because of the three items in multiple actors. One vector per thread. I’ll edit to clarify.

          2. 2

            At that point you’re just begging the question. ‘Given this problem that’s been specifically designed to cater for solution X and only solution X, what other solutions are there?”

            1. 1

              Well, it’s not only solvable with linked lists, people came up with several interesting possibilities.

              I didn’t want to “make linked lists win”, I just wanted to see if they could be beaten on their on turf, so to speak, and looks like they can, at least sometimes.

      2. 4

        How many things? How large are they? Are they comparable for order or is this insertion order only?

        Avoiding a memory copy implies chasing pointers which implies latency.

        Echoing @justinpombrio, what’s the problem we’re solving?

      3. 4

        I’d personally use the im crate and use its immutable vectors, and have the threads send new vectors to each other on update. Having multiple writers holding references to the same structure is kinda cursed no matter what you’re doing.

        1. 1

          I was thinking something similar. You could even combine this with arc-swap so you could have more precise control over the latency of receiving updates to that data structure.

      4. 4

        The most idiomatic Rust data structure for linked lists is std::collections::LinkedList.

        Alternatively, if your question is about alternatives to linked lists in general, I’ve had success writing code involving multiple concurrent sequenced writers using (1), a ring buffer of pointers, (2) a deque with indexes (instead of pointers), and (3) a hash map of vectors. That’s a much bigger problem space than what the article’s topic, which is (IMO mostly just griping) about people writing libraries that the author thinks aren’t necessary.

        1. 5

          std::LinkedList doesn’t have cursors on stable (and doesn’t have multiple cursors on nightly), so it doesn’t solve the problem.

      5. 4

        I don’t know how to solve this in concurrent case, but for a single-threaded case one example that comes to mind is the mutable flavor of rust-analyzer syntax trees (for refactors mutable api,where you hold several “cursors” into the tree and mutate around them, is most natural). We just do linked list there: https://github.com/rust-analyzer/rowan/blob/master/src/sll.rs.

      6. 3

        Even in this obviously-contrived scenario to favor a linked list, I wonder whether this “periodic low-latency” access won’t actually be relatively costly and high latency due to pointer chasing and randomly touching cache lines belonging to different cores. Those atomic pointer swaps will be chatty, and you’ll be paying alloc per object.

        In modern CPUs there is ridiculously large cost difference between local CPU work and a pipeline stall due to touching cross-core caches and going all the way to the RAM. This is why we have hyperthreading — the RAM-waiting core is stalled for so long that it has time to switch to another thread, do some work there, and switch back before it hears back from RAM.

        I’d try each worker thread append to a non-sorted vector sized appropriately for the cache, then periodically sort it, and send it away to the reader. This amortizes allocations, and is super cache-friendly. The reader then can use merge sort to combine multiple vectors (or iterate without copying). This way data remains owned by one thread most of the time, first sort pass will be multi-threaded but also happen locally in each thread’s cache, and the reader will have mostly-sorted mostly-contiguous snapshot of the data all for itself.

        1. 1

          You might not believe me, but I swear cache did cross my mind while I was writing my, indeed, very and purposely contrived example XD

          All these answers are almost making me wonder where linked lists are good for anything anymore, really.

          1. 4

            I am fairly confident that liked list are the most overused data structure out there, but it doesn’t mean that they are entirely useless.

            Off the top of my head:

        2. 1

          You can have ‘fat’ linked lists analogous to cdr coding wherein each node has a fixed capacity greater than 1 (and a variable occupancy).

    9. 9

      The thread on LKML about this work really doesn’t portray the Linux community in a good light. With a dozen or so new kernels being written in Rust, I wouldn’t be surprised if this team gives up dealing with Linus and goes to work on adding good Linux ABI compatibility to something else.

      1. 26

        I dunno, Linus’ arguments make a lot of sense to me. It sounds like he’s trying to hammer some realism into the idealists. The easter bunny and santa claus comment was a bit much, but otherwise he sounds quite reasonable.

        1. 19

          Disagreement is over whether “panic and stop” is appropriate for kernel, and here I think Linus is just wrong. Debugging can be done by panic handlers, there is just no need to continue.

          Pierre Krieger said it much better, so I will quote:

          Part of the reasons why I wrote a kernel is to confirm by experience (as I couldn’t be sure before) that “panic and stop” is a completely valid way to handle Rust panics even in the kernel, and “warn and continue” is extremely harmful. I’m just so so tired of the defensive programming ideology: “we can’t prevent logic errors therefore the program must be able to continue even a logic error happens”. That’s why my Linux logs are full of stupid warnings that everyone ignores and that everything is buggy.

          One argument I can accept is that this should be a separate discussion, and Rust patch should follow Linux rule as it stands, however stupid it may be.

          1. 7

            I think the disagreement is more about “should we have APIs that hide the kernel context from the programmer” (e.g. “am I in a critical region”).

            This message made some sense to me: https://lkml.org/lkml/2022/9/19/840

            Linus’ writing style has always been kind of hyperbolic/polemic and I don’t anticipate that changing :( But then again I’m amazed that Rust-in-Linux happened at all, so maybe I should allow for the possibility that Linus will surprise me.

          2. 1

            This is exactly what I still don’t understand in this discussion. Is there something about stack unwinding and catching the panic that is fundamentally problematic in, eg a driver?

            It actually seems like it would be so much better. It recovers some of the resiliency of a microkernel without giving up the performance benefits of a monolithic kernel.

            What if, on an irrecoverable error, the graphics driver just panicked, caught the panic at some near-top-level entry point, reset to some known good state and continued? Seems like such an improvement.

            1. 5

              I don’t believe the Linux kernel has a stack unwinder. I had an intern add one to the FreeBSD kernel a few years ago, but never upstreamed it (*NIX kernel programmers generally don’t want it). Kernel stack traces are generated by following frame-pointer chains and are best-effort debugging things, not required for correctness. The Windows kernel has full SEH support and uses it for all sorts of things (for example, if you try to access userspace memory and it faults, you get an exception, whereas in Linux or FreeBSD you use a copy-in or copy-out function to do the access and check the result).

              The risk with stack unwinding in a context like this is that the stack unwinder trusts the contents of the stack. If you’re hitting a bug because of stack corruption then the stack unwinder can propagate that corruption elsewhere.

              1. 1

                With the objtool/ORC stuff that went into Linux as part of the live-patching work a while back it does actually have a (reliable) stack unwinder: https://lwn.net/Articles/728339/

                1. 2

                  That’s fascinating. I’m not sure how it actually works for unwinding (rather than walking) the stack: It seems to discard the information about the location of registers other than the stack pointer, so I don’t see how it can restore callee-save registers that are spilled to the stack. This is necessary if you want to resume execution (unless you have a setjmp-like mechanism at the catch site, which adds a lot of overhead).

                  1. 2

                    Ah, a terminological misunderstanding then I think – I hadn’t realized you meant “unwinding” specifically as something sophisticated enough to allow resuming execution after popping some number of frames off the stack; I had assumed you just meant traversal of the active frames on the stack, and I think that’s how the linked article used the term as well (though re-reading your comment now I realize it makes more sense in the way you meant it).

                    Since AFAIK it’s just to guarantee accurate stack backtraces for determining livepatch safety I don’t think the objtool/ORC functionality in the Linux kernel supports unwinding in your sense – I don’t know of anything in Linux that would make use of it, aside from maybe userspace memory accesses (though those use a separate ‘extable’ mechanism for explicitly-marked points in the code that might generate exceptions, e.g. this).

                    1. 2

                      If I understand the userspace access things correctly, they look like the same mechanism as FreeBSD (no stack unwinding, just quick resumption to an error handler if you fault on the access).

                      I was quite surprised that the ORC[1] is bigger than DWARF. Usually DWARF debug info can get away with being large because it’s stored in separate pages in the binary from the file and so doesn’t consume any physical memory unless used. I guess speed does matter for things like DTrace / SystemTap probes, where you want to do a full stack trace quickly, but in the kernel you can’t easily lazily load the code.

                      The NT kernel has some really nice properties here. Almost all of the kernel’s memory (including the kernel’s code) is pageable. This means that the kernel’s unwind metadata can be swapped out if not in use, except for the small bits needed for the page-fault logic. In Windows, the metadata for paged-out pages is stored in PTEs and so you can even page out page-table pages, but you can then potentially need to page in every page in a page-table walk to handle a userspace fault. That extreme case probably mattered a lot more when 16 MiB of RAM was a lot for a workstation than it does now, but being able to page out rarely-used bits of kernel is quite useful.

                      In addition, the NT kernel has a complete SEH unwinder and so can easily throw exceptions. The SEH exception model is a lot nicer than the Itanium model for in-kernel use. The Itanium C++ ABI allocates exceptions and unwind state on the heap and then does a stack walk, popping frames off to get to handlers. The SEH model allocates them on the stack and then runs each cleanup frame, in turn, on the top of the stack then, at catch, runs some code on top of the stack before popping off all of the remaining frames[2]. This lets you use exceptions to handle out-of-memory conditions (though not out-of-stack-space conditions) reliably.

                      [1] Such a confusing acronym in this context, given that the modern LLVM JIT is also called ORC.

                      [2] There are some comments in the SEH code that suggest that it’s flexible enough to support the complete set of Common Lisp exception models, though I don’t know if anyone has ever taken advantage of this. The Itanium ABI can’t support resumable exceptions and needs some hoop jumping for restartable ones.

            2. 4

              What you are missing is that stack unwinding requires destructors, for example to unlock locks you locked. It does work fine for Rust kernels, but not for Linux.

        2. 7

          Does the kernel have unprotected memory and just rolls with things like null pointer dereferences reading garbage data?

          For errors that are expected Rust uses Result, and in that case it’s easy to sprinkle the code with result.or(whoopsie_fallback) that does not panic.

          1. 4

            As far as I understand, yeah, sometimes the kernel would prefer to roll with corrupted memory as far as possible:

            So BUG_ON() is basically ALWAYS 100% the wrong thing to do. The argument that “there could be memory corruption” is [not applicable in this context]. See above why.

            (from docs and linked mail).

            null derefernces in particular though usually do what BUG_ON essentially does.

            And things like out-of-bounds accesses seem to end with null-dereference:

            https://github.com/torvalds/linux/blob/45b588d6e5cc172704bac0c998ce54873b149b22/lib/flex_array.c#L268-L269

            Though, notably, out-of-bounds access doesn’t immediately crash the thing.

            1. 8

              As far as I understand, yeah, sometimes the kernel would prefer to roll with corrupted memory as far as possible:

              That’s what I got from the thread and I don’t understand the attitude at all. Once you’ve detected memory corruption then there is nothing that a kernel can do safely and anything that it does risks propagating the corruption to persistent storage and destroying the user’s data.

              Linus is also wrong that there’s nothing outside of a kernel that can handle this kind of failure. Modern hardware lets you make it very difficult to accidentally modify the kernel page tables. As I recall, XNU removes all of the pages containing kernel code from the direct map and protects the kernel’s page tables from modification, so that unrecoverable errors can take an interrupt vector to some immutable code that can then write crash dumps or telemetry and reboot. Windows does this from the Secure Kernel, which is effectively a separate VM that has access to all of the main VM’s memory but which is protected from it. On Android, Halfnium provides this kind of abstraction.

              I read that entire thread as Linus asserting that the way that Linux does things is the only way that kernel programming can possibly work, ignoring the fact that other kernels use different idioms that are significantly better.

              1. 5

                Reading this thread is a little difficult because the discussion is evenly spread between the patch set being proposed, some hypothetical plans for further patch sets, and some existing bad blood between the Linux and Rust community.

                The “roll with corrupted memory as far as possible” part is probably a case of the “bad blood” part. Linux is way more permissive with this than it ought to be but this is probably about something else.

                The initial Rust support patch set failed very eagerly and panicked, including on cases where it really is legit not to panic, like when failing to allocate some memory in a driver initialization code. Obviously, the Linux idiom there isn’t “go on with whatever junk pointer kmalloc gives you there” – you (hopefully – and this is why we should really root for memory safety, because “hopefully” shouldn’t be a part of this!) bail out, that driver’s initialization fails but kernel execution obviously continues, as it probably does on just about every general-purpose kernel out there.

                The patchset’s authors actually clarified immediately that the eager panics are actually just an artefact of the early development status – an alloc implementation (and some bits of std) that follows safe kernel idioms was needed, but it was a ton of work so it was scheduled for later, as it really wasn’t relevant for a first proof of concept – which was actually a very sane approach.

                However, that didn’t stop seemingly half the Rustaceans on Twitter to take out their pitchforks, insists that you should absolutely fail hard if memory allocation fails because what else are you going to do, and rant about how Linux is unsafe and it’s riddled with security bugs because it’s written by obsolete monkeys from the nineties whose approach to memory allocation failures is “well, what could go wrong?” . Which is really not the case, and it really does ignore how much work went into bolting the limited memory safety guarantees that Linux offers on as many systems as it does, while continuing to run critical applications.

                So when someone mentions Rust’s safety guarantees, even in hypothetical cases, there’s a knee-jerk reaction for some folks on the LKML to feel like this is gonna be one of those cases of someone shitting on their work.

                I don’t want to defend it, it’s absolutely the wrong thing to do and I think experienced developers like Linus should realize there’s a difference between programmers actually trying to use Rust for real-world problems (like Linux), and Rust advocates for whom everything falls under either “Rust excels at this” or “this is an irrelevant niche case”. This is not a low-effort patch, lots of thinking went into it, and there’s bound to be some impedance mismatch between a safe language that tries to offer compile-time guarantees and a kernel historically built on overcoming compiler permisiveness through idioms and well-chosen runtime tradeoffs. I don’t think the Linux kernel folks are dealing with this the way they ought to be dealing with it, I just want to offer an interpretation key :-D.

          2. 1

            No expert here, but I imagine linux kernel has methods of handling expected errors & null checks.

        3. 6

          In an ideal world we could have panic and stop in the kernel. But what the kernel does now is what people expect. It’s very hard to make such a sweeping change.

          1. 6

            Sorry, this is a tangent, but your phrasing took me back to one of my favorite webcomics, A Miracle of Science, where mad scientists suffer from a “memetic disease” that causes them to e.g. monologue and explain their plans (and other cliches), but also allows them to make impossible scientific breakthroughs.

            One sign that someone may be suffering from Science Related Memetic Disorder is the phrase “in a perfect world”. It’s never clearly stated exactly why mad scientists tend to say this, but I’d speculate it’s because in their pursuit of their utopian visions, they make compromises (ethical, ugly hacks to technology, etc.), that they wouldn’t have to make in “a perfect world”, and this annoys them. Perhaps it drives them to take over the world and make things “perfect”.

            So I have to ask… are you a mad scientist?

            1. 2

              I aspire to be? bwahahaa

            2. 2

              Hah, thanks for introducing me to that comic! I ended up archive-bingeing it.

          2. 2

            What modern kernels use “panic and stop”? Is it a feature of the BSDs?

            1. 8

              Every kernel except Linux.

            2. 2

              I didn’t exactly mean bsd. And I can’t name one. But verified ones? redox?

              1. 1

                I’m sorry if my question came off as curt or snide, I was asking out of genuine ignorance. I don’t know much about kernels at this level.

                I was wondering how much an outlier the Linux kernel is - @4ad ’s comment suggests it is.

                1. 2

                  No harm done

        4. 4

          I agree. I would be very worried if people writing the Linux kernel adopted the “if it compiles it works” mindset.

        5. 2

          Maybe I’m missing some context, but it looks like Linus is replying to “we don’t want to invoke undefined behavior” with “panicking is bad”, which makes it seem like irrelevant grandstanding.

        6. 2

          The part about debugging specifically makes sense in the “cultural” context of Linux, but it’s not a matter of realism. There were several attempts to get “real” in-kernel debugging support in Linux. None of them really gained much traction, because none of them really worked (as in, reliably, for enough people, and without involving ritual sacrifices), so people sort of begrudgingly settled for debugging by printf and logging unless you really can’t do it otherwise. Realistically, there are kernels that do “panic and stop” well and are very debuggable.

          Also realistically, though: Linux is not one of those kernels, and it doesn’t quite have the right architecture for it, either, so backporting one of these approaches onto it is unlikely to be practical. Linus’ arguments are correct in this context but only insofar as they apply to Linux, this isn’t a case of hammering realism into idealists. The idealists didn’t divine this thing in some programming class that only used pen, paper and algebra, they saw other operating systems doing it.

          That being said, I do think people in the Rust advocacy circles really underestimate how difficult it is to get this working well for a production kernel. Implementing panic handling and a barebones in-kernel debugger that can nonetheless usefully handle 99% of the crashes in a tiny microkernel is something you can walk third-year students through. Implementing a useful in-kernel debugger that can reliably debug failures in any context, on NUMA hardware of various architectures, even on a tiny, elegant microkernel, is a whole other story. Pointing out that there are Rust kernels that do it well (Redshirt comes to mind) isn’t very productive. I suspect most people already know it’s possible, since e.g. Solaris did it well, years ago. But the kind of work that went into that, on every level of the kernel, not just the debugging end, is mind-blowing.

          (Edit: I also suspect this is the usual Rust cultural barrier at work here. The Linux kernel community is absolutely bad at welcoming new contributors. New Rust contributors are also really bad at making themselves welcome. Entertaining the remote theoretical possibility that, unlikely though it might be, it is nonetheless in the realm of physical possibility that you may have to bend your technology around some problems, rather than bending the problems around your technology, or even, God forbid, that you might be wrong about something, can take you a very long way outside a fan bulletin board.)

        7. 1

          easter bunny and santa claus comment

          Wow, Linus really has mellowed over the years ;)

    10. 1

      It will be interesting to see how cargo vet fits into this space once it’s announced officially.

      Previous discussion: here.

    11. 1

      This sounds like it’s fulfilling the same use case as cargo-crev. Are there fundamental architectural differences between the two tools? Will “reviews” from one cross-pollinate to the other?

      The linked page proposes some good ideas for how to make the audit process simpler. Especially “deferred audits”. I just wonder why these ideas are implemented in a new crate instead of being upstreamed to cargo-crev.

      1. 1

        AFAIU, the focus is on having your own source of trusted audits in tree and not rely on a public database like cargo-crev. But I need to admit that my understanding of both approaches is quite limited.

    12. 6

      I really enjoy when someone comes up with a solution I’ve just never even come close to thinking of. The surprise is fun.

      Here, it was the realization that we can try to solve the “updating non-optional fields” challenge by treating readers and writers separately. Time will tell if that’s actually a good idea, but it’s at least a new point in the design space. Very cool.

      Also this library seems surprisingly well fleshed-out. I expected more alpha-quality software, but it seems like everything’s in order. Great documentation.

      I wonder how the performance stacks up against other encoding frameworks. It would be cool to see it added to this comparison benchmark:

      https://github.com/djkoloski/rust_serialization_benchmark

      1. 1

        Thanks for the kind words! It was a pleasant surprise waking up and seeing Typical on Lobsters this morning.

        1. 1

          I learnt about it on ZuriHac. Tie, which allows generation of Haskell server stubs from OpenAPI was presented there, and given that we at work generate OpenAPI from applicative style Haskell schemas (like autodocodec, but not type class based), I am curious about this space.

          Still looking for comprehensive literature on this. I heard that Designing Data-Intensive Applications covers Avro, but I’d like to see some coverage comparing more options.

    13. 21

      I use this a lot in conjunction with a json blob. This allows for document storage without super strict structure enforcement:

      Something like:

      CREATE TABLE releases (
        "raw" BLOB,
        "uri" TEXT GENERATED ALWAYS AS (json_extract(raw, '$.uri')) VIRTUAL,
        "date" timestamp GENERATED ALWAYS AS (json_extract(raw, '$.date')) VIRTUAL,
        "name" TEXT GENERATED ALWAYS AS (json_extract(raw, '$.name')) VIRTUAL
      );
      
      1. 5

        I’ve been doing this in Postgres and it’s worked out well. We had a source of JSON events and wanted to store them for analysis, but didn’t know ahead of time what indices we’d need. We just knew we’d probably want the ability to do that analysis in the future.

        So we ingested everything as jsonb and now I can add generated columns and indices retroactively where needed for query performance. So far, no regrets.

        1. 2

          Indeed, the fact that both sqlite and postgres support virtual columns is great that it allows a path for minimal scaling up from the former to the later if desired.

      2. 3

        What is stored if the field is missing?

        1. 4

          NULL is stored:

          sqlite> select json_extract('{}', '$.uri') is null;
          1
          
    14. 2

      The Grep family is one of those tools I always lose competency with after a few months. I only reliably use it for simple searches and exclusions on STDIN.

      I can never remember the difference between grep, egrep, fgrep, and how those versions differ between GNU grep and BSD (macOS) grep. It’s the same with sed - trying to keep invocations straight between the BSD and GNU variants means I remember neither effectively. find to a lesser extent.

      I am very, very happy that ripgrep and fd solved searching file contents and file names for me. Is there a similar replacement for sed? Or a more general modern tool for recursive find-replace file editing on a directory tree? That task is often the sole reason I open an IDE.

      1. 2

        I never use egrep or fgrep. From my perspective, these look like relics of the past that almost nobody usea anymore.

        Grep does search into multiple files and has support for PCRE.

        But yeah, there are BSD and macOS versions of grep that share the name but their utility and performance are not to be compared. They don’t even have support for PCRE, which I use 50% of the time.

      2. 1

        I use sd all the time and I love it.

    15. 2

      I used a few of these crappy hubs over the years and always had problems. Tried a Lenovo Thunderbolt 3 dock as mentioned by another commenter, but maybe the one I had was bad because I had issues with it as well.

      Finally gave up and bought the CalDigit TS3+ for a ridiculous amount of money (even used off of ebay) and I’ve had zero problems since. This thing is rated one of the best for a reason and now that I never have to mess with these again I’m happy I spent the money.

      Unfortunately if you want a portable option, the CalDigit TS3+ does not apply. It’s solid enough that I’d consider their other devices though.

      1. 1

        I’ve been looking into docs recently, and don’t know much about them. Should I be considering Linux compatibility? Do you know if the CalDigit TS3/4 specifically is compatible with Linux?

        1. 2

          I can only speak to the the TS3. It works just as well on my Linux Thinkpad as it does with a Mac. I swap back and forth almost daily with no issues. I’ve used a few distros but am settled on Arch.

    16. 1

      I’ve been putting off learning Rust macros because they feel too hard. Compare that to Crystal macros for example.

      1. 1

        Rust’s declarative macros (macro_rules) seem pretty similar. They’re a bit more verbose, sure, and a little harder to read maybe. But fundamentally similar in that they’re declarative.

        Procedural macros, where you write arbitrary code to produce an AST from an AST, are definitely more difficult—but also more powerful.

        1. 1

          proc macros are token -> token conversions instead of AST -> AST conversions, it’s one of the reasons you need quote/syn for most macro crates. The rfc explains the reasoning here, the big one being so changes to the AST don’t break procedural macros.

    17. 4

      The core message of the blog post - that you can get away with the worst kind of password if you’re using hardware MFA - is right on the money.

      But! Password re-use is what lets people log into all of your (non-MFA) accounts after finding one password in a hacked dump from 2013 that they’ve been chewing on for as long as it takes. Some online vendors probably still aren’t salting passwords correctly, making password recovery almost trivial with modern hardware / usual end user password lengths. It won’t show up in these password scan analyses, but can be far more devastating: I still think it’s good advice to not re-use passwords. Write them down in a book, use a password manager, whatever.

      & to reiterate that core message: using a 2FA hardware key is the single most effective thing you can do to secure your online accounts. Even if you can’t be bothered with 2FA on most of your accounts (which is fair enough) using it on your primary email account will make a huge, huge difference. Anyone with access to the email you used to create other accounts can use that email to reset every account you used it for, so it’s your most important online identity.

      The cheap Yubico keys are £25 or so. How much will a compromise of your Gmail (Outlook / whatever) cost you?

      1. 3

        On one hand, using a Yubikey (or other FIDO device) is extremely easy. Plug it in, touch it. The end. We really just need to expand server-side support.

        On the other hand, how do people keep their hardware key always on them? I find this essentially impossible.

        1. 3

          You can make it easier by having multiple Yubikeys, including one permanently attached to each device.

        2. 2

          Same as the house keys - on a keyring. If that’s not in my pocket, something’s wrong.

          1. 1

            Do you carry your keyring around your house/apartment? Maybe my problem is that I’m unbearably lazy and hate having to get up and walk over to go get my Yubikey from across the room

            1. 3

              It’s in my pocket literally all the time I’m not in bed or in water. Same with the phone. I’d keep forgetting them otherwise.

            2. 1

              The first thing I do after putting on my trousers is put wallet, phone and keys in my pockets, watch on my wrist and wedding ring on my finger.

              If I left my keys out of my pocket, I would be worried about locking myself out of the house accidentally.

            3. 1

              If you’re using a YubiKey to log into a web site, the odds are that it’s using WebAuthn these days. You don’t need a separate token with WebAuthn, you can use the TPM to protect your credentials and have the OS unlock them with biometrics. Most of the works stuff I use is like this: I tough a fingerprint reader on my machine to log in. I might not have my keys with me, but I definitely do have a computer with me when I want to log in. When I want to log in from another machine, my phone can be used to authorise a new device (either registering a new private key or temporarily) by sending a notification to my phone that requires the same process (touch the fingerprint reader and then approve) to log in.

        3. 1

          On the other hand, how do people keep their hardware key always on them? I find this essentially impossible.

          Mine lives on my keyring, so I always have it on me.

    18. 3

      Working on a new markup language for prose (working name: Sona). Example here. There’s no parser for this latest version. Still trying to nail down the syntax. Still trying to get rid of the braces. Can’t seem to figure out how and still preserve clear hierarchy in a sane way.

    19. 2

      The article claims that Rust doesn’t have seamless structured concurrency. Rayon (and crossbeam’s scoped threads under the hood) solves most of this problem.

      We can use its trait implementations:

      use rayon::iter::*;
      

      Define some expensive functions:

      /// A number p is prime if iff: (p-1)! mod p = p-1
      /// (apparently this is called "Wilson's theorem")
      fn is_prime(p: u64) -> bool {
          factorial_mod_n(p - 1, p) == p - 1
      }
      
      fn factorial_mod_n(k: u64, n: u64) -> u64 {
          (1..=k).reduce(|acc, item| (acc * item) % n).unwrap()
      }
      

      And then pretty quickly max out every core on your computer finding primes.

      fn main() {
          // Find primes up to N.
          const MAX: u64 = 200000;
      
          // This works.
          let primes: Vec<u64> = (2..=MAX).into_par_iter().filter(|&n| is_prime(n)).collect();
      }
      

      We can even reference data on the stack. This works because crossbeam’s “scoped threads” library enforces that the thread joins back into its parent thread before the reference becomes invalid.

          let my_cool_prime = 503;
          primes.as_slice().par_iter().for_each(|&prime| {
              if prime == my_cool_prime {
                  println!("found my prime!");
              }
          });
      
      1. 3

        As mentioned in sidenote 10, one can only sometimes reference data on the stack in parent scopes, only things that happen to be Sync. One will often find that their data doesn’t implement Sync, because of some trait object buried within. In that situation, one has to refactor to make it Sync somehow.

        If one has to refactor existing code, it’s not seamless, IMO.

        1. 3

          Something not being Sync usually means it uses interior mutability without a mutex. It’s pretty hard to make something magically have a mutex in it; even if you can, it’s a 50/50 chance that it’ll be heavily contested and you’ll need to restructure it anyway. The other main time I’ve had something not be Sync is when it contains an OS resource or thread-local or such that is explicitly not allowed to be shared between threads, and you generally can’t fix that just by find-and-replacing it with Mutex<T>.

          Sometimes complicated things are just complicated, I guess.

          1. 1

            See the Rust example linked from the article (I think it was in a side note), something as simple as having a regular trait object somewhere in your data is enough to make something not Sync.

            1. 2

              Trait objects can be sync if you write the type as dyn Trait + Sync.

              This needs to be written explicitly, because otherwise the trait allows non-sync implementations, and you could have a data race.

              1. 1

                Yep, that’s precisely the issue that’s in the way of Rust’s seamless concurrency.

                1. 2

                  Oh, come on. If it doesn’t accept buggy code it’s not seamless? Running of single-threaded code on multiple threads should not be “seamless”.

                  1. 1

                    Excuse me? I didn’t say anything about accepting buggy code.

                    1. 2

                      You’ve complained that non-Sync (i.e. non-thread-safe) trait objects are prevented from being used in multi-threaded code. If they were allowed, that would be unsound, and “seamlessly” allow data race bugs.

                      Just to be clear: Rust trait objects can be compatible with multi-threaded code. You don’t need to eliminate them or work around them. They just come in two flavors: thread-safe and non-thread-safe. If you have a non-thread-safe trait object, then of course you can’t “seamlessly” use it, as that would be incorrect. But when you have dyn Trait + Send + Sync, it just works in both single-threaded and multi-threaded contexts.

                      1. 1

                        That’s a pretty big misinterpretation of my words. I’m happy to clarify what I’ve said.

                        I am not suggesting any changes to Rust to make safe seamless concurrency possible; I’m not sure that’s even possible, without introducing a lot of complexity to the language. I’m not saying it should change its rules to allow use of non-Sync/non-Send data in multi-threaded code. I’m not saying anything at all about changing Rust.

                        I’m simply stating that, because Rust requires refactoring existing code (for example deeply changing trait objects and other existing data to be Sync), it cannot do this seamless concurrency. If you’re really curious about the root cause of why Rust requires Sync and Send, and therefore why it can’t do seamless concurrency, let me know and I can enlighten. All I mean so far is that Rust cannot do this, because of its own decisions and limitations.

                        1. 3

                          But by this definition, languages that care about correctness of multi-threaded code can’t be “seamless”. Because otherwise you can always be in a situation where you’re dealing with code that either actually isn’t thread-safe, or incorrectly declares itself to be thread-unsafe, and you need to fix it instead of having compiler ignore the problem and let you run it anyway. From Rust’s perspective the refactoring you’re describing is a bug fix, because this situation would not happen in properly written code.

                          1. 1

                            That’s not true; the article shows an example where a language can safely have multiple threads concurrently reading the same data, without refactoring.

                            It also links to an example showing that it is not possible to do safely in Rust, without refactoring. Given that it is possible, but not in Rust, it indicates that the problem is with Rust here.

                            From Rust’s perspective the refactoring you’re describing is a bug fix, because this situation would not happen in properly written code.

                            Yes, if we tried this in Rust, it would be a bug in that Rust code, because Rust can’t safely express this pattern. That doesn’t mean it would be a bug in other languages.

                            1. 2

                              You’re disqualifying Rust for not being able to safely express this pattern? But you include OpenMP, which doesn’t even have these guarantees in the first place? How is that not favoring lack of safety checks as seamlessness?

                              1. 1

                                I see you’re bringing up OpenMP suddenly, when our conversation so far hasn’t been comparing C to Rust, it has been about safe languages.

                                Perhaps our conversation’s context didn’t make it clear enough, so I’ll be more specific: Rust cannot support seamless fearless structured concurrency, but the article shows that we can have seamless fearless structured concurrency in another language, such as Vale.

                                I’m sure you’re discussing in good faith, and not intentionally straw-manning what I’m saying, so I’m happy to keep clarifying. I always enjoy exploring what’s possible in language design with those who are honest and curious.

                                1. 1

                                  I’m just contesting what can be called “seamless”, which is more of argument about semantics of the word, and tangential to the Vale language, so it’s probably not worth digging this further. Thank you for your responses.

            2. 2

              Yeah, it would be interesting to have trait objects follow the same rules as deriving traits, same way that new types are automatically Send+Sync+Drop when they can be unless you specify otherwise. ie, saying dyn Trait usually means dyn Trait+Send+Sync+Drop. It would probably be a breaking change to do that, so will probably never happen, but I wonder whether it’s one of those things would abruptly spiral out of control and make everything in the world break, or whether it would be a minor inconvenience to implement and then mostly just work everywhere.

    20. 6

      DNS is full of confusing jargon. At $work we do R&D closely related to DNS and run into this all the time.

      Example: what is a “base domain name”? If you go by ICANN, the BDN for “alice.github.io” is “io”, but if you go by the Public Suffix List, the “base domain” (more often called a “site” in this context), is “github.io” because that definition is more relavent for security and privacy on The Web.

      A more accurate name that someone was kind enough to invent is TLD+1 (vs eTLD+1) so it’s clear which we mean. I’ve seen “BDN” refer to both, but more commonly TLD+1.

      But somehow, in the midst of all this jargon, we still don’t have a reliable way to refer to the “+1” label in TLD+1, even though it’s the most important label in the group. In “apple.com”, what is “apple”? At work we call it the “primary name label”, though that phrase also misses the subtlety of distinguishing between TLDs and eTLDs, and isn’t widely used as far as I know. (If someone knows official terminology for this, please tell me.)

      And then there’s the hostname/FQDN/domain triplet that all pretty much mean the same thing (until they don’t).

      And then there’s the difference between a DNS proxy (sometimes called a forwarding resolver) and a DNS server, which is fairly close to the point made in the article. As far as I can tell, the difference is in whether or not the server caches responses. Or at least that’s where I think the difference should be. Some people distinguish between proxies and resolvers. Others don’t.

      Also “label” vs “subdomain”. That’s a good one. Is the “root domain” a subdomain? It’s usually considered to be an empty label. But if root is a subdomain, shouldn’t it be a subdomain of something?

      There’s more fun in DNS (think about how domains are supposed to be case-insensitive in a Unicode world (also domains aren’t necessarily Unicode and can be arbitrary bytes)), but I’ll leave it there before I get too much further off-topic.