1. 46
  1. 15

    How does it handle data structures that are not trees? It big problem with Rust is that it uses linear types as the sole memory management strategy in the language, so anything that requires aliases that have non-lexical lifetimes is impossible to implement. Rust works around this with unsafe blocks that allow you to ignore linearity, but any of these can introduce memory safety bugs. My biggest concern about Rust is that it each one of these unsafe blocks may be locally valid but not safe in the presence of composition because the type system is not expressive enough to capture the invariants that these depend on (the Oakland paper that found 200+ CVEs in Cargo packages looking for three idioms around incorrect use of locally-safe unsafe blocks that I shared here last year adds some weight to that worry). In skimming the article,

    I didn’t see anything that lets me safely implement the kinds of structures that I need in systems programming. For example, a thread object may need to be on a list of threads for a process and on one or more sleep queues. This can’t be implemented with linear types unless you add an indirection layer (and now temporal safety issues still exist, they just result in instance confusion and can still lead to critical CVEs), which adds the kind of overhead that is often unacceptable in systems programming.

    1. 7

      As far as I can tell, Austral is not for you. It is for people who want “simple” Rust, not for people who want more complex Rust.

      I think simplicity is just a deranged goal for a programming language, so I am utterly uninterested in Austral. Austral defines simplicity to be short description length. Why does Austral have if and while statement then? Isn’t goto simpler?

      1. 8

        I think simplicity is just a deranged goal for a programming language

        I don’t know if I agree with this take or not, but it feels so against the current discourse on programming languages that I have no choice but to like it XD

        1. 6

          The word “simple” is being used here with a precise technical meaning, and gotos are not simpler than control structures with single entry and single exit. “Short description length” refers a formal technical description that fully and exhaustively describes the semantics, including in all corner cases.

          1. 4

            A short description is not necessarily simple though. There is no precise meaning to the word “simple.”

          2. 5

            It is for people who want “simple” Rust, not for people who want more complex Rust.

            I don’t want the language to be simpler (in the sense used here). I want using the language to be simpler. Those aren’t the same; Brainfuck and 6502 assembly are good counterexamples.

            One of the main reasons I find Rust hard to use is because it tries to force me into single-ownership patterns. A language like Rust that didn’t have that kind of restriction would be simpler to me, even if its specification were longer.

            1. 1

              Simplicity here has a straightforward definition: it is the amount of information it takes to describe a system.

              That seems like a reasonable thing to optimize for. Imagine if Java tried to follow it…

              edit: However, seems like some of its decisions go against that:

              There are no implicit type conversions anywhere. More generally: there are no implicit function calls. If it’s not in the source code, it’s not happening, and you’re not paying the cost of it.

            2. 6

              Austral has unsafe code, but yuck, using that feature undermines the design goals (safety, ease of reasoning about programs).

              I think the answer to your data structure question may lie in adopting the “data oriented programming” and “entity component system” style of programming that is used for high performance systems programming in video games. I’ve read some articles written by Rust people about exploring this style of programming in Rust.

              The author wants a simple language. Trees and DAGs are simpler than cyclic graphs. One article I read advocating this position called pointers the “gotos” of data structures. But to use this kind of language means abandoning programming idioms we are accustomed to (you said “the kinds of structures I need in systems programming”) and using new idioms that I think we are still in the process of discovering. It’s obvious that Austral is a research project and not a drop-in replacement for the style of systems language you are used to, and I think part of the needed research is developing and perfecting these new idioms.

              a thread object may need to be on a list of threads for a process and on one or more sleep queues

              This structure can be described using a DAG, I think. You don’t need a cyclic graph? You aren’t forced to use strict linear semantics for all resources, there is borrowing, so DAGS can be constructed as long as lifetimes are nested. I think Austral would be forced by its design goals to adopt structured concurrency, where thread lifetimes are strictly nested (unlike in Rust, which doesn’t require this).

              1. 8

                This structure can be described using a DAG, I think

                Correct, but it cannot be expressed with linear pointers because each structure appears on multiple lists.

                You aren’t forced to use strict linear semantics for all resources, there is borrowing, so DAGS can be constructed as long as lifetimes are nested

                That’s the problem. The lifetimes for these structures are not lexical. A thread object remains on a sleep queue for as long as it is sleeping, but that queue may be traversed by another thread (with appropriate synchronisation).

                Some things are unavoidably cyclic. The FreeBSD kernel actually includes a tracing garbage collector because file descriptors passed via UNIX domain sockets can create cycles of the file structures. That problem is unavoidable if you want POSIX semantics (I don’t really, but unfortunately I do want to run a load of code that does).

                1. 3

                  It sounds like you are talking about kernel programming: implementing threads inside a kernel like FreeBSD. I have ideas about that.

                  Austral is very similar to a language I’m developing. In my language, there are pure immutable values (which cannot contain references to mutable objects), and pure functions (which are values), and there are also references to objects containing mutable state, but these references are subject to linear semantics, and are not values. Unlike C (which is used for all modern kernel programming) there is no shared mutable state.

                  The absence of shared mutable state means you must use a very different coding style to write a Unix kernel (in my language). I think it would be a message passing microkernel system. So think about Erlang, or QNX. An unsafe systems language with shared mutable state (like assembly language, C, Zig) must be used to implement the trusted “microkernel” or “language runtime”, central to which is the implementation of threads, and message passing between threads. This microkernel is small, because it is written in an unsafe language, and needs to be carefully audited. The majority of the Unix kernel is written in the safe language. I would want to keep the unsafe microkernel or runtime code separate from the safe code, and maintained by a select group of people, rather than have a coding culture where anybody can drop unsafe code blocks into whatever they are writing. This makes the unsafe code easier to audit and prevents unchecked proliferation. That’s why I talk about using 2 different systems languages, rather than a single language with unsafe blocks.

                  Cyclic structures are implemented using data oriented programming techniques. Instead of a graph of mutable objects allocated on a heap, where the arcs in the graph are machine pointers, you represent a collection of objects as an array or a structure of arrays (SOA), and object references become array indexes or the “handles” of data oriented programming. Data oriented designs are faster, because there are fewer heap allocations and an SOA memory layout can be optimized to provide better cache locality. Some gaming systems languages have language support for SOA and data oriented programming, which makes it easier.

                  1. 4

                    I agree with most of what you say, but to me a systems programming language is one that can be used to implement:

                    • A memory allocator
                    • A scheduler

                    Ideally, it should have very few type-system escape hatcher for this (mostly to some assembly for capturing the register state on context switches) and be able to provide safe idioms for these things.

                    I then see something one step up, where you need fine-grained control over memory lifetimes and communication. This is infrastructure programming, where you might implement device drivers (so you need an atomic model for accessing memory-mapped I/O via typed regions), network protocols (so you need to be able to efficiently parse and generate binary data), but where everything that you do should be expressible within your language’s type system. At this level, you should never need an unsafe escape hatch, you should be able to do absolutely everything in the type system. Linear types at the object level are still not sufficient here because code at this level is very often performance (throughput / latency / jitter) sensitive and so you requiring more complex data structures to express an idea is not adequate.

                    Above this, you have things where programmer experience and efficiency is the primary goal, where you’re happy to put up with some memory (and performance) overhead and some jitter in exchange for reduced cognitive load. This is application programming.

                    For Verona, we are targeting infrastructure programming. The runtime (scheduler, memory allocator) is written in C++, but the only FFI to unsafe code is via a sandboxing model, which prevents the foreign code from being able to corrupt the safe-language data structures. We build a concurrency model into the language, because you cannot implement a concurrency model in a safe language (by definition, it requires shared mutable state). Objects are all allocated in a region. Regions have a sentinel object that dominates all live objects within that region. Pointers to the sentinel from outside are linear, so we can safely move a region between threads by assigning that pointer (or move-capturing it in a lambda) but within a region we can have arbitrary object graphs. A region can have its choice of memory-management model, for example a traced GC if you want complex cyclical data structures, reference counted if you want DAGs (or simple cyclic data structures, such as doubly linked lists, where you will do explicit cycle braking), linear ownership if you want only trees, or bump allocation with bulk free if everything has roughly the same lifetime. We can also freeze a region, which consumes the linear capability to the sentinel and returns an immutable capability to the same object graph. Immutable objects can be shared between concurrent owners (cowns), our unit of concurrency. Work can be asynchronously scheduled over a set of cowns, allowing things like two-phase commit, which are essential to efficient infrastructure programming but incredibly painful with actor-model programming (our concurrency model is a generalisation of the actor model).

                  2. 1

                    I’m super curious about this too. It’s rough enough in Rust, and maybe there’s a better way forward?

                    if you want POSIX semantics (I don’t really, but unfortunately I do want to run a load of code that does)

                    After trying to actually implement POSIX filesystem sematics once upon a time, I’m stealing this saying for the future. :)

                    1. 4

                      The approach that we’re taking in Verona is to move the linearity up one layer. Every object is in precisely one region (an abstract machine concept, not necessarily a contiguous address range). Every region has a single object that dominates all live (reachable) objects in the region. Pointers from the outside of a region to that sentinel object are linear. This gives you the nice concurrency properties (transferring the linear pointer is an ownership transfer of the whole region) but also lets you have different memory management strategies within a region. A region might use a tracing GC, reference counting, or simple bump allocation and bulk free, depending on the use.

              2. 3

                Looks cute. I really like this trend of new programming languages treating Rust as the bar to clear rather than older worse designed languages.

                One small syntactic criticism: the “borrowing” section doesn’t show an example of that syntax.

                One point of confusion: I believe that it should be okay to use linear values inside loops as long as each iteration preserves the linearity? while f() { consume(linear_thing); linear_thing = produce(); } kinda stuff (and borrowing ofc) could be permissible because they don’t send the quantity of linear_thing below 0 or above 1 and every loop iteration both begins and ends with quantity 1.

                1. 3

                  It was shared a while ago but it seems like it has advanced a bunch, and the blog post I found very interesting with the examples.

                  1. 3

                    This is the first language I’ve been excited about in quite a while. It looks to check all the boxes for me!

                    1. 2

                      executable YAML

                      What is this? This sounds horrific.

                      1. 6

                        A lot of deployment tools require you to define the deployment steps in YAML files:

                        These tools end up reimplementing programming language features like variables, conditionals, and function calls within the data structures supported by YAML. They support using YAML’s alias nodes feature to emulate variables or functions, but might additionally define their own variable feature that supports string interpolation.

                        The configuration languages CUE and Dhall seem to provide a better foundation for deployment tools than YAML does.

                        1. 4

                          Another turn around the configuration complexity clock, wheeeeeee! http://mikehadlow.blogspot.com/2012/05/configuration-complexity-clock.html

                          1. 1

                            Wow, I’ve never heard of alias nodes. YAML is way more complicated than I thought. I use GH actions but never really considered how configuring it is kind of like a language in itself