This is the handout for a workshop I gave this morning at the NICAR data journalism conference. The handout is designed to be useful completely independently of the in-person workshop itself.
I wrote a bit more about this workshop here, including notes on the various new pieces of software I developed in advance of teaching at NICAR.
I originally planned to include notes on running local models for structured data extraction - but I ran into a problem with length limits. For data extraction you need to be able to input quite a lot of text, and the current batch of local model solutions (at least that I’ve tried) tend to stop working well once you go above a few thousand input tokens. Meanwhile OpenAI handles 100,000 and Gemini handles 1-2 million.
In journalism you’re often working with public data, in which case I see no problem at all feeding it into models.
I had some interesting conversations at the conference about the much larger challenge of analyzing private leaks. Those are cases where dumping them into an online model isn’t acceptable - the best solution right now may be investing in a new $10,000 Mac Studio and running capable long context models on that.
Those are cases where dumping them into an online model isn’t acceptable - the best solution right now may be investing in a new $10,000 Mac Studio and running capable long context models on that.
What are the most capable long context models nowadays that you can run locally? And why do you specifically need a Mac Studio for that — I’m assuming for the ludicrous amounts of shared memory? How much exactly of it would one need?
That’s something I’ve been trying to figure out. The problem is that longer context requires a LOT of both processing and memory.
I ran a survey about this yesterday on Twitter and on Bluesky - in both cases I asked:
Anyone had much success running long context prompts through local LLMs? On my M2 64GB Mac I’m finding that longer prompts take an unreasonably long time to process, am I holding it wrong? Any models or serving platforms I should try out that might respond reasonably quickly?
Prompt processing is compute bound (raw FLOPS) instead of memory bandwidth bound like token generation. M2 Max is 13TFLOPS. Nvidia 3090 is 35TFLOPS. It’s just the Mac’s GPU being small.
Various suggestions to try things like flash attention which I clearly need to learn more about.
I should give fish another try. Reading this made me realise I rely on ^S, ^X, as well as frequent use of ^Z, jobs and fg: habits I learned in a very different time, With better equivalents now
Apple oscillates a bit here. They either do nothing, point FreeBSD committers at patches they may want, or actively push things. They’ve never been particularly consistent about it and it changes over time. They funded the TrustedBSD MAC work, for example, which landed in both FreeBSD and XNU and provides the kernel bit of the iOS / macOS sandboxing framework. In contrast, they extended all of their core utilities to support -h and -H consistently for decimal and binary SI prefixes and didn’t upstream those. They offered to relicense some bits of Darwin that we were considering bringing into the FreeBSD base system.
There’s often a reasonable overlap between folks in the Apple CoreOS team and FreeBSD committers, which helps. I wouldn’t take this as a shift in Apple policy to treating FreeBSD explicitly as an upstream as they do with LLVM, though that would be very nice.
They funded parts of TrustedBSD, but they’re deprecating its subsystems on macOS since at least 2019. For example, OpenBSM is already deprecated since 2022, MAC Framework is deprecated since 2019.
As I understand it, their sandbox framework is built on top of the MAC framework. They have largely removed support for third-party kernel modules, which makes it impossible to use the MAC framework’s lower levels, only their higher-level abstractions.
It’s a shame that they never merged Capsicum, since it was designed to support things that look like their sandboxing model.
They have built something on the MAC framework (e.g. EndpointSecurity is currently built on it, for the time being), but the thing is that Apple didn’t allow using it much earlier than when kexts were deprecated. In fact, developers started to use it when Apple has released headers of MACF by mistake, not because Apple considered MACF to be a valid kernel API.
I’m not sure what point you’re trying to make here. Indeed, Apple never officially considered the MAC Framework a public API in macOS/XNU, so why do you consider it deprecated since 2019? And what does that have to do with Apple’s FreeBSD contributions? MACF was never supported on macOS in the first place. In fact, breaking API/ABI changes were frequent long before 2019. They just started cracking down hard on kexts in general in 2019.
The point is that even if Apple has funded creation of subsystems mentioned above, it’s not possible to use them on their platform, because either they’re deprecated and will be removed from the OS altogether, or were never allowed to be used by thirdparties. OP’s comment suggested that Apple has funded them and they’re available to be used, and that’s not true today for OpenBSM, and it was never true for MACF.
I never said (or intentionally implied) that Apple would expose or intend something for third-party developers on XNU just because they funded its development in FreeBSD. I did suggest that, if they’re in the open source bits of Darwin then it’s often possible to pull fixes from there into FreeBSD, but that’s very different.
I want to get more comfortable with low level stuff. I do a lot of work in Python and Java, but things like communicating directly with hardware or modifying older video games written in assembly have always been a mystery to me (I think a big part of it is from being self taught).
If anyone has recommendations for books or reasonably doable projects, I’d really appreciate it!
Have you ever written an interpreter for a language? That exercise will hammer home what assembly really is.
Now, that’s wholly distinct from actually talking with hardware! But it’s a great object lesson in the most fundamental pattern of computing, namely, ascribing meaning to data, aka interpreting it.
If you want to go more the hardware route you could try something like PICO-8. Not really hardware but will probably have the flavor of real hardware, annoying quirks and all.
I’m also looking to learn this sort of thing, and I’m working my way through the exercises in Understanding Software Dynamics by Richard Sites. It’s been slow going, but really eye-opening in terms of how hardware and software design interact.
It’s probably being developed as a smoother off-ramp for the “Shopify Scripts” feature they are removing. They have another WASM-based feature called “Shopify Functions”. This could bridge the gap.
No types is a major pain; I spend so much time grepping around nixpkgs for stuff because I have no idea what form an argument takes, so I have to find the call site, figure out what value was passed in, and that value is usually the result of some other function call, so I need to find that function definition and figure out what value it returns, which is itself the result of some function call, and so on until the heatdeath of the universe all to answer a question that should be easily answered by hovering my cursor over the original variable. :)
Relatedly, the nix folks don’t seem to like documentation or informative variable names, but that’s not strictly a language problem.
It’s all part of the ecosystem of the language. A proper language server would probably also be able to track down any value through all the files and show it.
I have yet to understand what it is Flakes are exactly, even after reading about it multiple times over some time. Can you explain what it is Flakes provide you?
Sure I would like to figure that out too. Lets say I have a project and I want development shell that bring in all the dependency to build the project AND those dependencies are not in guix’s tree, they are in independent channels.
dev shell
In guix I think you would make a channels.scm and do guix time-machine -C channels.scm something? to enter your development shell?
In nix you specify what packages you want in your dev shell and the other flakes which are dependencies in flakes.nix as inputs. To enter the dev shell nix develop
updating
To update your project’s dependencies in nix you do
nix flake update
guix is guix pull --channels=./my-channels.scm and guix describe --format=channels > ./channels.scm
(im not 100% those are equivalent).
distribution
If I want distribute whatever I was working on in nix I just need to add package output to my flake.nix. Then other’s can include my flake in there flake.
In guix I have to create an channel which also references the other channels that my project depend on. Do the channels that where pinned in ‘./channels.scm’ stay pinned when other people use my channel?
How does guix deal with conflicting package between channels? In nix dependent flakes are arguments to a function (example) and you have to manually join the package sets, or just reference the argument variable.
Overall all my impression is that flakes are more composable then guix channels. And I think this composablity is important. I don’t want a giant repository of packages, I would just rather have one channel or flake per package. In nix that is easy to do and I would love to know how to do it in guix. I like scheme and guix’s bootstrap story a lot.
Seems correct, guix time-machine -C channels.scm -- shell hello will get you a development environment with the package hello in it as specified by the channel version specified in channel.scm. It seems like you could combine this with inferior guixes to pull packages out of arbitrary revisions of channels, but that API is not stable yet.
But thank you for your explanation of flakes. They seems to be Russian nesting dolls of packages from specific channel version. I’m on foot now but if I remember I will ask around in IRC when I get to a real computer if its possible to compose channel files like flakes.
About conflicts: In Scheme code you can adress them uniquely through their inferior parentes (what an amazing sentance btw). In the produced shell environment the latest specified package takes precedence.
I don’t have any interest in an OS (also doing my best to ignore NixOS as much as possible). The scheme based go at the same thing is interesting but Nix is already borderline too niche for me.
Nix on MacOS is a relatively common and discussed usage of Nix.
Yeah, it’s just that you have to wade through half the comms being about NixOS.
I talked to a colleague where they had Nix adopted in a part of a large company and every time somebody mentioned it, first they had to create clarity what it was that the person was talking about (language, packages, OS or something else entirely).
Doesn’t Elixir generate BEAM bytecode? If so, I’d expect it to also benefit from this. That said, I was a big confused by reading the announcement because a lot of the things that they were talking about sounded a lot like things from the original HiPE paper, and I thought HiPE had been merged around 15 years ago.
Reclaiming 900GB+ of disk space on a Pg server after dropping a JSONB column from a table with 2.5B rows. We’re using pg_repack to avoid extended downtime. After that, burying my work computer in the garden while I take a week off.
Race a dinghy on Sunday morning, and celebrate my birthday in the afternoon.
using SQLAlchemy (which makes it hard for developers to understand what database queries their code is going to emit, leading to various situations that are hard to debug and involve unnecessary operational pain, especially related to the above point about database transaction boundaries)
I’m also a fan of raw SQL. If I would be asked for mistakes I have made, I would as well name the database abstraction (in my case peewee). Two things that annoy me: The database is now mixed into my models and peewee has a very global connection management. I would much rather like the model to be totally independent from the database layer and do explicit connection management. E.g.
# no active record, no implicit connection tied to the model
my_object = Customer(name='John Doe')
# CustomerDbLayer must contain all the logic to understand
# how to store a Customer into the DB
# (as well as how to read one from the DB)
cdb = CustomerDbLayer(db_connection)
cdb.save(my_object)
It’s a bit surprising that at the end of the article they mention GraphQL and Kubernetes as some of their good decisions, two technologies which I would definitely not list in the section boring. At least a few years ago both seemed quite the hype to me.
Kubernetes reminds me a lot of way back in the day when I was first learning about sendmail and was like “wait, this thing’s config is so complex people have to use and configure another tool just to generate the config file for it?” Except with Kubernetes it’s often multiple layers of that going on.
The specific case with SQLAlchemy there is one that I’ve seen be a problem at a previous employer (where control of exactly when, what, and in what order queries ran was important for regulatory compliance), but is more an issue with the session/unit-of-work/magic-optimization approach than with ORM or DB libraries in general. Plenty of ORMs and DB libraries give you much more fine-grained control of when queries run, when things get flushed to DB, etc.
just wanted to say congratulations on what you’ve got so far; it was interesting for sure, and I’m looking forward to future parts.
it makes me wonder if you’re basically modeling parallel dataflow; linear types probably means each stack can be executed pretty safely in parallel, yes? the moving values into a new stack feels very “spawn a thread” to me.
Yes, that’s absolutely correct. Operations on separate stacks could trivially be parallelized by a properly designed runtime. Eventually, high-level Dawn code could be compiled to OpenCL or CUDA, similar to in Futhark. Alternatively, a compiler to assembly for a superscalar CPU could interleave operations on the different stacks to take advantage of instruction-level parallelism.
This looks very interesting! I’ll eagerly await the next installment and look forward to playing with the language.
Reading the post, I wondered how the stack names are scoped. In the example code, one can reasonably (I think?) guess that $x, $y, and $z are scoped to the function whose {spread} operation created them, or maybe they are bound to the current stack. But looking at the comment you linked to in an earlier discussion about I/O, it seemed like there were references to named stacks like $stdout from some external scope.
Perhaps I’m wrong to assume that “scope” is even a relevant concept. But presumably there has to be some way to avoid stack name collisions.
(Edit) …Huh, maybe I am wrong to assume that scope matters. I am still trying to wrap my brain around it, but maybe if the language can guarantee linearity across the whole application, you don’t even have to care about name collisions because if there is already a $x stack from some calling function, it doesn’t matter if you reuse it and push values onto it as long as you’re guaranteed to have removed them all by the time your caller looks at it again (assuming your caller wasn’t expecting you to leave behind a value on that stack, but then “leaves behind one value on $x” would be part of your function signature).
I’m not quite sure that actually works, but now I’m even more eager to read the next post.
But presumably there has to be some way to avoid stack name collisions.
Yes, there is. Take for example the expression: {$a swap} where swap is defined as
{fn swap => $a<- $b<- $a-> $b->}
Expanded, this becomes
{$a $a<- $b<- $a-> $b->}
Or, equivalently,
{$a {$a push} {$b push} {$a pop} {$b pop}}
Without some mechanism to handle the stack collision between the inner and outer $a stack, this would not behave properly. Since we want functions to behave the same way regardless of what stack context they are executed from, that would be unacceptable. So this is handled by checking for nested stack name collisions and renaming the inner stack. So the latter would effectively be rewritten to
{$a {$$a push} {$b push} {$$a pop} {$b pop}}
In the existing type checker prototype, renamed stacks are distinguished by a prefix of more than one $. Then, if one of these temporary stack names escapes up to the inferred type for a user-defined function, an error is raised. This ensures expected behavior while ensuring we don’t need to monomorphize each function to operate on different stacks.
Your introduction to Dawn was clear and compelling, really excited to follow along with Dawn’s development.
{spread $x $y $z} is syntactic sugar
Do you intend to expose a macro system to Dawn programmers?
Conceptually, these named stacks can be considered to be different parts of one big multi-stack.
How are Dawn’s named stacks represented internally? I don’t have much familiarity with stack-based languages, but it seems like it would be straightforward to understand how the machine runs a program just by reading the source. Is that lost with the introduction of named stacks, or is there a mental mapping that can be made?
I’m undecided on syntactic macros, but the plan is absolutely to provide meta-programming. I haven’t prototyped this at all yet, but I hope and expect for compiler passes to be implemented in something I’ve been calling meta-Dawn in my notes—a form of staged compilation. The plan is for Dawn to be its own intermediate representation. We’ll see how that works out in practice, of course.
And to answer your other questions, in the existing interpreter the named stacks are a Map String (Stack Val). The first compiler, which will be through translation to C, will use static analysis (basically the inferred types) to turn all stack slots, regardless of which stack they are on, into function parameters.
Eventually, I plan to write an assembler in/for Dawn, in which stack names will correspond to architecture-specific registers.
I’m looking forward to the rest of the series. Is there any change you could put an rss/atom feed on the site? It’s a much nicer experience than subscribing to a mailing list.
First of all, it’s a really interesting concept! One question about the linear typing aspect though: What happens if multiple functions want to access the same read-only data structure? I assume that for small values clone just copies by value, but what if the data structure is prohibitively expensive to copy? Are some values cloned by reference? If yes, don’t you need to track how many readers / writers access the data so that you know when to free the memory?
I guess another way of phrasing this question would be: How do you handle the more complex cases of borrowing data with the semantics described in the post? Rust’s borrow checker is of course useful even for simple cases of stack-allocated values, but it really shines (and this is where its complexity lies) in cases of more complex heap-allocated data structures. (And of course, even Rust with its borrow semantics needs Rc/RefCell as escape hatches for situations where these guarantees cannot be statically checked, is there something comparable in Dawn?)
a way to bind values to local named variables. Unfortunately, this breaks one of the primary advantages of concatenative notation: that functions can be trivially split and recombined at any syntactic boundary—i.e. their factorability.
You gave an example of how Dawn still has this but can you give an example of why Factor or Kitten do not? Or more generally, why a concatenative with bind values to local named variables cannot.
Here’s a example of Fibonnaci from Flpc (Disclaimer: I’m the author).
[ 1 return2 ] bind: base-case
[ newfunc1 assign: i
pick: i pushi: 3 < pushf: base-case if
pick: i 1 - fib pick: i 2 - fib + return1
] bind: fib
Any part of the body can be split off. For example
[ 1 return2 ] bind: base-case
[ pick: i 1 - ] bind: one-less
[ pick: i 2 - ] bind: two-less
[ newfunc1 assign: i
pick: i pushi: 3 < pushf: base-case if
one-less fib two-less fib + return1
] bind: fib
[ 1 return2 ] bind: base-case
[ pick: i s21 - ] bind: recurse-fib
[ newfunc1 assign: i
pick: i pushi: 3 < pushf: base-case if
1 recurse-fib 2 recurse-fib + return1
] bind: fib
For your example, this would be
[ newfunc3 assign: z assign: y assign: x
pick: y square pick: x square + pick: y abs - return1 ] bind: f
(z needs not be dropped explicitly since return1 takes care of that.)
In your example of splitting up fib, what happens if you reuse one-less in another function? Does the source essentially get inlined, so that pick: i refers to the assign: i? If so, then this appears to be similar to Dawn, but without the linearity restriction.
In Factor and Kitten, I don’t believe local variables work like that, though. I believe they behave more like they do in most existing languages, e.g. Python, where the pick: i in one-less would be an undefined variable error.
In your example of splitting up fib, what happens if you reuse one-less in another function? Does the source essentially get inlined, so that pick: i refers to the assign: i?
Yes, that’s exactly right.
In Factor and Kitten, I don’t believe local variables work like that, though. I believe they behave more like they do in most existing languages, e.g. Python, where the pick: i in one-less would be an undefined variable error.
Oh I see. I’m wondering if it’s because of scoping. With what I’m doing and maybe what you’re doing, you can’t (easily) get lexical scoping whereas the other way you could.
I would love to read a thread where someone explains why the app is so gigantic to begin with. It does not do a whole lot, so what is all this code doing?
The app looks simple from the user perspective. But on the global scale the business rules are insanely complicated. Every region has custom rules/regulations. Different products have different workflows. Some regions have cash payments. Every airport has different pick rules…
See also the people replying to that discussion, they talk about how unreliable networking means they can’t do this stuff on the backend.
they talk about how unreliable networking means they can’t do this stuff on the backend.
Uber is infamous for having over 9000 microservices. They even had to “re-invent” distributed tracing, because NIH syndrome or something (Jaeger, now open-tracing). I really, really doubt they do it all on the client. They have to make sure the transaction is captured anyway and without network, how can you even inform drivers?
Presumably they verify everything on the backend anyway because you can’t trust the client. My guess is that the network is reliable enough to submit a new trip, but not reliable enough to make every aspect of the UI call out to the backend. Imagine opening the app, waiting for it to locate you, then waiting again while the app calls out to the network to assess regulations for where you are. Same thing picking a destination. That latency adds up, especially perceptually.
Yeah that’s the part that gets me, it’s like the dril tweets about refusing to spend less on candles. Just write less code! Don’t make your app the Homer Simpson car! What is the 1mb/week of new shit your app does good for anyways? It’s easy, just play Doom instead of typing in Xcode.
I don’t have ｈｙｐｅｒｇｒｏｗｔｈ mentality though, I guess that’s why Uber rakes in all the profits.
I recall a few years ago someone disassembled Facebook’s (I think) Android app and found it had some stupendously huge number of classes in it. Which led to a joke about “I guess that’s why their interviews make you implement binary trees and everything else from scratch, every developer there is required to build their own personal implementation every time they work on something”.
Which is cynical, but probably not too far off the mark – large organizations tend to be terrible at standardizing on a single implementation of certain things, and instead every department/team/project builds their own that’s done just the way they want it, and when the end product is a single executable, you can see it in the size.
I wonder if there could also be some exponential copy-paste going on, where instead of refactoring some existing module to be more general so it satisfies your need, you copy-paste the files and change the bits you need. Then somebody else comes along and does the same to the package that contains both your changed copy and the original package, so there are now 4 copies of that code. The cancerous death of poorly managed software projects.
I like it! I’d put “overflow: auto” on “figure.highlight pre” to make the scrollbars only appear when required on code snippets. Otherwise cool idea :)
My experience so far with Clojure’s spec library is that it provides exactly the experience Michael describes here. Start at the REPL to get a sketch together, then “finish” it with specifications of arguments/return values and property-based testing.
This is the handout for a workshop I gave this morning at the NICAR data journalism conference. The handout is designed to be useful completely independently of the in-person workshop itself.
I wrote a bit more about this workshop here, including notes on the various new pieces of software I developed in advance of teaching at NICAR.
He wrote the guide for investigative reporters at a data journalism conference.
Yeah, I have no problem at all with the ethics of helping data journalists scrape things that really don’t want to be scraped. News is what somebody does not want you to print.
I originally planned to include notes on running local models for structured data extraction - but I ran into a problem with length limits. For data extraction you need to be able to input quite a lot of text, and the current batch of local model solutions (at least that I’ve tried) tend to stop working well once you go above a few thousand input tokens. Meanwhile OpenAI handles 100,000 and Gemini handles 1-2 million.
In journalism you’re often working with public data, in which case I see no problem at all feeding it into models.
I had some interesting conversations at the conference about the much larger challenge of analyzing private leaks. Those are cases where dumping them into an online model isn’t acceptable - the best solution right now may be investing in a new $10,000 Mac Studio and running capable long context models on that.
What are the most capable long context models nowadays that you can run locally? And why do you specifically need a Mac Studio for that — I’m assuming for the ludicrous amounts of shared memory? How much exactly of it would one need?
That’s something I’ve been trying to figure out. The problem is that longer context requires a LOT of both processing and memory.
I ran a survey about this yesterday on Twitter and on Bluesky - in both cases I asked:
Quite a few interesting answers, including this one:
Various suggestions to try things like flash attention which I clearly need to learn more about.
I should give fish another try. Reading this made me realise I rely on ^S, ^X, as well as frequent use of ^Z, jobs and fg: habits I learned in a very different time, With better equivalents now
I use ^Z, jobs and
fgall day. What are these better equivalents you mention?Oh I mean stuff like screen, tmux, terminal-inside-neovim, even simple things like “multiple terminals”. But old habits die hard.
“Better” is too strong of a word that I wouldn’t use, but I imagine using multiple windows/tabs/split panes is what OP had in mind.
Is the basis of their new “Private Compute Cloud” macOS, or could it be FreeBSD?
It’s probably the fact macOS coreutils is based on BSD ones, and Apple has perhaps decided to be nicer and upstream bug fixes.
Apple oscillates a bit here. They either do nothing, point FreeBSD committers at patches they may want, or actively push things. They’ve never been particularly consistent about it and it changes over time. They funded the TrustedBSD MAC work, for example, which landed in both FreeBSD and XNU and provides the kernel bit of the iOS / macOS sandboxing framework. In contrast, they extended all of their core utilities to support
-hand-Hconsistently for decimal and binary SI prefixes and didn’t upstream those. They offered to relicense some bits of Darwin that we were considering bringing into the FreeBSD base system.There’s often a reasonable overlap between folks in the Apple CoreOS team and FreeBSD committers, which helps. I wouldn’t take this as a shift in Apple policy to treating FreeBSD explicitly as an upstream as they do with LLVM, though that would be very nice.
They funded parts of TrustedBSD, but they’re deprecating its subsystems on macOS since at least 2019. For example, OpenBSM is already deprecated since 2022, MAC Framework is deprecated since 2019.
As I understand it, their sandbox framework is built on top of the MAC framework. They have largely removed support for third-party kernel modules, which makes it impossible to use the MAC framework’s lower levels, only their higher-level abstractions.
It’s a shame that they never merged Capsicum, since it was designed to support things that look like their sandboxing model.
They have built something on the MAC framework (e.g. EndpointSecurity is currently built on it, for the time being), but the thing is that Apple didn’t allow using it much earlier than when kexts were deprecated. In fact, developers started to use it when Apple has released headers of MACF by mistake, not because Apple considered MACF to be a valid kernel API.
I’m not sure what point you’re trying to make here. Indeed, Apple never officially considered the MAC Framework a public API in macOS/XNU, so why do you consider it deprecated since 2019? And what does that have to do with Apple’s FreeBSD contributions? MACF was never supported on macOS in the first place. In fact, breaking API/ABI changes were frequent long before 2019. They just started cracking down hard on kexts in general in 2019.
The point is that even if Apple has funded creation of subsystems mentioned above, it’s not possible to use them on their platform, because either they’re deprecated and will be removed from the OS altogether, or were never allowed to be used by thirdparties. OP’s comment suggested that Apple has funded them and they’re available to be used, and that’s not true today for OpenBSM, and it was never true for MACF.
I never said (or intentionally implied) that Apple would expose or intend something for third-party developers on XNU just because they funded its development in FreeBSD. I did suggest that, if they’re in the open source bits of Darwin then it’s often possible to pull fixes from there into FreeBSD, but that’s very different.
Nathan Sobo is a great guy! If you’re interested in knowing more about him as a person, I have an interview with him here
This is a fantastic interview. Very personal and weirdly inspirational. Thanks to you and Nathan for it.
Good list, but it’s missing transactions.
I want to get more comfortable with low level stuff. I do a lot of work in Python and Java, but things like communicating directly with hardware or modifying older video games written in assembly have always been a mystery to me (I think a big part of it is from being self taught).
If anyone has recommendations for books or reasonably doable projects, I’d really appreciate it!
Have you ever written an interpreter for a language? That exercise will hammer home what assembly really is.
Now, that’s wholly distinct from actually talking with hardware! But it’s a great object lesson in the most fundamental pattern of computing, namely, ascribing meaning to data, aka interpreting it.
If you want to go more the hardware route you could try something like PICO-8. Not really hardware but will probably have the flavor of real hardware, annoying quirks and all.
Thank you for the suggestions! That’s super helpful
I’m also looking to learn this sort of thing, and I’m working my way through the exercises in Understanding Software Dynamics by Richard Sites. It’s been slow going, but really eye-opening in terms of how hardware and software design interact.
edit: asked my non-programmer gf the same thing since I wanted to see how a person external to this world thinks about software and she said
the reason mostly because they allow her to do her work for free, interesting how different it is from our community but not that much
Have you been able to integrate Magit and git-branchless?
I wrote some custom extensions to integrate it yeah, have kind of a graph interface for git prev and git next based on git sl
I want to know more about what they are using it for internally. Shopify is pushing the envelope with Ruby, it’s inspirational.
It’s probably being developed as a smoother off-ramp for the “Shopify Scripts” feature they are removing. They have another WASM-based feature called “Shopify Functions”. This could bridge the gap.
No mention of it being a successor to Nix-the-language. Is that off the table?
Nope it seems not: https://discourse.nixos.org/t/nickel-1-0-release/28252 and from there to https://github.com/nickel-lang/nickel-nix
Given how things progress, that is going to be… a process.
Still I look forward to it even though Nickel is quite esoteric, it does not make my eyes bleed like Nixlang does.
How does Nix make your eyes bleed?
inherit,with, etc.).Like, it does the/a job but it’s never going to win any prize for beauty or elegance.
No types is a major pain; I spend so much time grepping around nixpkgs for stuff because I have no idea what form an argument takes, so I have to find the call site, figure out what value was passed in, and that value is usually the result of some other function call, so I need to find that function definition and figure out what value it returns, which is itself the result of some function call, and so on until the heatdeath of the universe all to answer a question that should be easily answered by hovering my cursor over the original variable. :)
Relatedly, the nix folks don’t seem to like documentation or informative variable names, but that’s not strictly a language problem.
I just found a band-aid, the nil language server is a big help.
It displays lots of helpful information. I can’t find a screenshot, but it’s worth trying.
Oh, cool. Thanks for the tip. If I’m ever brave enough to venture back into Nix again, I will have to check this out! :)
It’s all part of the ecosystem of the language. A proper language server would probably also be able to track down any value through all the files and show it.
Have you tried Guix? (Shameless plug of my favorite config lang.)
guile is nice to work with but nix has flakes
I have yet to understand what it is Flakes are exactly, even after reading about it multiple times over some time. Can you explain what it is Flakes provide you?
Reading the following post it seems you could achieve the same effect with
guix time-machineand some flags: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00022.htmlI haven’t found anything better than this, which seems somewhat outdated: https://serokell.io/blog/practical-nix-flakes
I’m working on adding flakes coverage to nix-shorts.
Sure I would like to figure that out too. Lets say I have a project and I want development shell that bring in all the dependency to build the project AND those dependencies are not in guix’s tree, they are in independent channels.
dev shell- In guix I think you would make a channels.scm and do
- In nix you specify what packages you want in your dev shell and the other flakes which are dependencies in
updatingguix time-machine -C channels.scm something?to enter your development shell?flakes.nixasinputs. To enter the dev shellnix developTo update your project’s dependencies in nix you do
- guix is
distributionnix flake updateguix pull --channels=./my-channels.scmandguix describe --format=channels > ./channels.scm(im not 100% those are equivalent).How does guix deal with conflicting package between channels? In nix dependent flakes are arguments to a function (example) and you have to manually join the package sets, or just reference the argument variable.
Overall all my impression is that flakes are more composable then guix channels. And I think this composablity is important. I don’t want a giant repository of packages, I would just rather have one channel or flake per package. In nix that is easy to do and I would love to know how to do it in guix. I like scheme and guix’s bootstrap story a lot.
Seems correct,
guix time-machine -C channels.scm -- shell hellowill get you a development environment with the package hello in it as specified by the channel version specified in channel.scm. It seems like you could combine this with inferior guixes to pull packages out of arbitrary revisions of channels, but that API is not stable yet.But thank you for your explanation of flakes. They seems to be Russian nesting dolls of packages from specific channel version. I’m on foot now but if I remember I will ask around in IRC when I get to a real computer if its possible to compose channel files like flakes.
About conflicts: In Scheme code you can adress them uniquely through their inferior parentes (what an amazing sentance btw). In the produced shell environment the latest specified package takes precedence.
Nix flakes are derivations that build a reproducible output based on some input, all the inputs are hashed, tracked by git revision etc.
I don’t have any interest in an OS (also doing my best to ignore NixOS as much as possible). The scheme based go at the same thing is interesting but Nix is already borderline too niche for me.
Neither Nix nor Guix are operating systems (or Linux distributions)
Depends who you ask…I find that the projects aren’t exactly in a hurry to create clarity on the matter.
“Guix is a distribution of the GNU operating system.”
You could fool me with this!
Nix on MacOS is a relatively common and discussed usage of Nix.
That is fair enough, you could fool me with that too - It definitely used to refer to Guix as a transactional package manager.
Yeah, it’s just that you have to wade through half the comms being about NixOS.
I talked to a colleague where they had Nix adopted in a part of a large company and every time somebody mentioned it, first they had to create clarity what it was that the person was talking about (language, packages, OS or something else entirely).
This would work great in tandem with Kent Beck’s test && commit || revert workflow.
Are there plans for the Elixir compiler to leverage these optimizations at some point?
Doesn’t Elixir generate BEAM bytecode? If so, I’d expect it to also benefit from this. That said, I was a big confused by reading the announcement because a lot of the things that they were talking about sounded a lot like things from the original HiPE paper, and I thought HiPE had been merged around 15 years ago.
HIPE has been deprecated in favour of this approach
Anything generating BEAM byte code benefits at runtime
Reclaiming 900GB+ of disk space on a Pg server after dropping a JSONB column from a table with 2.5B rows. We’re using
pg_repackto avoid extended downtime. After that, burying my work computer in the garden while I take a week off.Race a dinghy on Sunday morning, and celebrate my birthday in the afternoon.
I’m pretty sure that JSONB column is my fault, happy to see you’ve found a way to get rid of it!
Never again
Understandable :)
I’m also a fan of raw SQL. If I would be asked for mistakes I have made, I would as well name the database abstraction (in my case peewee). Two things that annoy me: The database is now mixed into my models and peewee has a very global connection management. I would much rather like the model to be totally independent from the database layer and do explicit connection management. E.g.
It’s a bit surprising that at the end of the article they mention GraphQL and Kubernetes as some of their good decisions, two technologies which I would definitely not list in the section boring. At least a few years ago both seemed quite the hype to me.
Kubernetes reminds me a lot of way back in the day when I was first learning about sendmail and was like “wait, this thing’s config is so complex people have to use and configure another tool just to generate the config file for it?” Except with Kubernetes it’s often multiple layers of that going on.
The specific case with SQLAlchemy there is one that I’ve seen be a problem at a previous employer (where control of exactly when, what, and in what order queries ran was important for regulatory compliance), but is more an issue with the session/unit-of-work/magic-optimization approach than with ORM or DB libraries in general. Plenty of ORMs and DB libraries give you much more fine-grained control of when queries run, when things get flushed to DB, etc.
I’ve found out about https://github.com/nackjicholson/aiosql last week and it hits is the perfect balance imho. I knew about a similar lib in clojure from few years back ( https://github.com/krisajenkins/yesql) but I hadn’t found anything equivalent for python.
Have you seen PugSQL?
GraphQL is just an old school RPC protocol. How is it not boring?
GraphQL doesn’t have a well-defined story for caching or error handling, both of which are otherwise boring HTTP built-ins.
Author here. Ask me anything!
just wanted to say congratulations on what you’ve got so far; it was interesting for sure, and I’m looking forward to future parts.
it makes me wonder if you’re basically modeling parallel dataflow; linear types probably means each stack can be executed pretty safely in parallel, yes? the moving values into a new stack feels very “spawn a thread” to me.
Thanks!
Yes, that’s absolutely correct. Operations on separate stacks could trivially be parallelized by a properly designed runtime. Eventually, high-level Dawn code could be compiled to OpenCL or CUDA, similar to in Futhark. Alternatively, a compiler to assembly for a superscalar CPU could interleave operations on the different stacks to take advantage of instruction-level parallelism.
This looks very interesting! I’ll eagerly await the next installment and look forward to playing with the language.
Reading the post, I wondered how the stack names are scoped. In the example code, one can reasonably (I think?) guess that
$x,$y, and$zare scoped to the function whose{spread}operation created them, or maybe they are bound to the current stack. But looking at the comment you linked to in an earlier discussion about I/O, it seemed like there were references to named stacks like$stdoutfrom some external scope.Perhaps I’m wrong to assume that “scope” is even a relevant concept. But presumably there has to be some way to avoid stack name collisions.
(Edit) …Huh, maybe I am wrong to assume that scope matters. I am still trying to wrap my brain around it, but maybe if the language can guarantee linearity across the whole application, you don’t even have to care about name collisions because if there is already a
$xstack from some calling function, it doesn’t matter if you reuse it and push values onto it as long as you’re guaranteed to have removed them all by the time your caller looks at it again (assuming your caller wasn’t expecting you to leave behind a value on that stack, but then “leaves behind one value on $x” would be part of your function signature).I’m not quite sure that actually works, but now I’m even more eager to read the next post.
Yes, there is. Take for example the expression:
{$a swap}whereswapis defined asExpanded, this becomes
Or, equivalently,
Without some mechanism to handle the stack collision between the inner and outer
$astack, this would not behave properly. Since we want functions to behave the same way regardless of what stack context they are executed from, that would be unacceptable. So this is handled by checking for nested stack name collisions and renaming the inner stack. So the latter would effectively be rewritten toIn the existing type checker prototype, renamed stacks are distinguished by a prefix of more than one
$. Then, if one of these temporary stack names escapes up to the inferred type for a user-defined function, an error is raised. This ensures expected behavior while ensuring we don’t need to monomorphize each function to operate on different stacks.Your introduction to Dawn was clear and compelling, really excited to follow along with Dawn’s development.
Do you intend to expose a macro system to Dawn programmers?
How are Dawn’s named stacks represented internally? I don’t have much familiarity with stack-based languages, but it seems like it would be straightforward to understand how the machine runs a program just by reading the source. Is that lost with the introduction of named stacks, or is there a mental mapping that can be made?
Dawn is really exciting!
Thanks!
I’m undecided on syntactic macros, but the plan is absolutely to provide meta-programming. I haven’t prototyped this at all yet, but I hope and expect for compiler passes to be implemented in something I’ve been calling meta-Dawn in my notes—a form of staged compilation. The plan is for Dawn to be its own intermediate representation. We’ll see how that works out in practice, of course.
And to answer your other questions, in the existing interpreter the named stacks are a Map String (Stack Val). The first compiler, which will be through translation to C, will use static analysis (basically the inferred types) to turn all stack slots, regardless of which stack they are on, into function parameters.
Eventually, I plan to write an assembler in/for Dawn, in which stack names will correspond to architecture-specific registers.
I’m looking forward to the rest of the series. Is there any change you could put an rss/atom feed on the site? It’s a much nicer experience than subscribing to a mailing list.
Added! https://www.dawn-lang.org/feed.xml
Thanks!
Thanks for the suggestion. I’ll take a look at what that entails.
First of all, it’s a really interesting concept! One question about the linear typing aspect though: What happens if multiple functions want to access the same read-only data structure? I assume that for small values
clonejust copies by value, but what if the data structure is prohibitively expensive to copy? Are some values cloned by reference? If yes, don’t you need to track how many readers / writers access the data so that you know when to free the memory?I guess another way of phrasing this question would be: How do you handle the more complex cases of borrowing data with the semantics described in the post? Rust’s borrow checker is of course useful even for simple cases of stack-allocated values, but it really shines (and this is where its complexity lies) in cases of more complex heap-allocated data structures. (And of course, even Rust with its borrow semantics needs Rc/RefCell as escape hatches for situations where these guarantees cannot be statically checked, is there something comparable in Dawn?)
Great question! I’m going to get to that in a future post, but there is a solution, and it doesn’t require a separate borrow checker.
Nice! Also looking forward to future parts.
You gave an example of how Dawn still has this but can you give an example of why Factor or Kitten do not? Or more generally, why a concatenative with bind values to local named variables cannot.
Here’s a example of Fibonnaci from Flpc (Disclaimer: I’m the author).
Any part of the body can be split off. For example
For your example, this would be
(
zneeds not be dropped explicitly sincereturn1takes care of that.)In your example of splitting up
fib, what happens if you reuseone-lessin another function? Does the source essentially get inlined, so thatpick: irefers to theassign: i? If so, then this appears to be similar to Dawn, but without the linearity restriction.In Factor and Kitten, I don’t believe local variables work like that, though. I believe they behave more like they do in most existing languages, e.g. Python, where the
pick: iinone-lesswould be an undefined variable error.Yes, that’s exactly right.
Oh I see. I’m wondering if it’s because of scoping. With what I’m doing and maybe what you’re doing, you can’t (easily) get lexical scoping whereas the other way you could.
Completing my Ruby bindings to macOS Big Sur’s Virtualization.framework. Then I can hopefully get started on a Vagrant provider.
I would love to read a thread where someone explains why the app is so gigantic to begin with. It does not do a whole lot, so what is all this code doing?
The author explains this in a reply here: https://mobile.twitter.com/StanTwinB/status/1337055778256130062
See also the people replying to that discussion, they talk about how unreliable networking means they can’t do this stuff on the backend.
Uber is infamous for having over 9000 microservices. They even had to “re-invent” distributed tracing, because NIH syndrome or something (Jaeger, now open-tracing). I really, really doubt they do it all on the client. They have to make sure the transaction is captured anyway and without network, how can you even inform drivers?
Presumably they verify everything on the backend anyway because you can’t trust the client. My guess is that the network is reliable enough to submit a new trip, but not reliable enough to make every aspect of the UI call out to the backend. Imagine opening the app, waiting for it to locate you, then waiting again while the app calls out to the network to assess regulations for where you are. Same thing picking a destination. That latency adds up, especially perceptually.
Yeah that’s the part that gets me, it’s like the dril tweets about refusing to spend less on candles. Just write less code! Don’t make your app the Homer Simpson car! What is the 1mb/week of new shit your app does good for anyways? It’s easy, just play Doom instead of typing in Xcode.
I don’t have ｈｙｐｅｒｇｒｏｗｔｈ mentality though, I guess that’s why Uber rakes in all the profits.
Fixed it for ya ;-)
I would love to see Chuck Moore’s
(RIP)take on 1 MB/week. He might just have a seizure.Thankfully, Chuck Moore is alive.
Ah, man, well I have egg on my face. I feel quite silly. I’m going to slink into a corner somewhere.
I recall a few years ago someone disassembled Facebook’s (I think) Android app and found it had some stupendously huge number of classes in it. Which led to a joke about “I guess that’s why their interviews make you implement binary trees and everything else from scratch, every developer there is required to build their own personal implementation every time they work on something”.
Which is cynical, but probably not too far off the mark – large organizations tend to be terrible at standardizing on a single implementation of certain things, and instead every department/team/project builds their own that’s done just the way they want it, and when the end product is a single executable, you can see it in the size.
I wonder if there could also be some exponential copy-paste going on, where instead of refactoring some existing module to be more general so it satisfies your need, you copy-paste the files and change the bits you need. Then somebody else comes along and does the same to the package that contains both your changed copy and the original package, so there are now 4 copies of that code. The cancerous death of poorly managed software projects.
Scary… and probably very true.
https://theinternate.com
Looks more or less exactly how it’s made: Markdown on a Solarized Light background
I like it! I’d put “overflow: auto” on “figure.highlight pre” to make the scrollbars only appear when required on code snippets. Otherwise cool idea :)
I’m in
My experience so far with Clojure’s spec library is that it provides exactly the experience Michael describes here. Start at the REPL to get a sketch together, then “finish” it with specifications of arguments/return values and property-based testing.