Threads for safinaskar
-
icefox
1 month ago
|
link
| on:
Novel way of handling integer overflow, which actually follows Liskov Substitution Principle (2021)
Looks very clever, but IIRC range subtyping is very difficult, if not impossible??? I could very well be wrong.
That said, lots of things are very difficult but still have Good Enough practical solutions, so.
-
but IIRC range subtyping is very difficult, if not impossible???
This proposal separates range and representation, and thus makes range subtyping possible
-
-
safinaskar
1 month ago
|
link
| on:
Novel way of handling integer overflow, which actually follows Liskov Substitution Principle (2021)
Also I recommend reading other posts in this blog. They are very insightful. For example, this article https://blog.polybdenum.com/2023/03/05/fixing-the-next-10-000-aliasing-bugs.html proves that any language with mutability (i. e. all popular languages except for Haskell) should have something like Rust’s borrow checker. All languages, not only low-level ones or performance-oriented. You should have borrow checker. Not for speed, but for correctness. Also this gives argument why Rust is best language. Not “best within fast languages”. Simply “best”. Best of all languages with mutability
-
safinaskar
1 month ago
|
link
| on:
Novel way of handling integer overflow, which actually follows Liskov Substitution Principle (2021)
We want i8 to be true subtype of i16. When I say subtype I mean true subtype as in PLT, i. e. I want true Liskov Substitution Principle. But in real programming language this is not true. Because i8 and i16 represented differently. I. e. you cannot use struct containing i8 everywhere you use same struct, but with i16. Because of representation issues. But in this article novel way is proposed: decouple range from representation! Now you get true subtypes (as in PLT). And in the same time you get better overflow management! And in the same time you keep decent performance! And no any undefined behavior! BRILLIANT!
-
Sounds like integer from common lisp: http://clhs.lisp.se/Body/t_intege.htm#integer
-
-
-
safinaskar
1 month ago
|
link
| on:
Novel way of handling integer overflow, which actually follows Liskov Substitution Principle (2021)
This article is brilliant! One of the best articles I have read for a long time.
I want to create my own language. Like many people here. :) And this article gave me a lot of insight, I will certainly use this ideas in my language
-
I’m looking for a Linux distro to switch from increasingly commercial Ubuntu (ads in motd, ads in apt, apt forcibly replaced with the snap fiasco where I never know if I should use
--classicwithout trial and error). Ideally, I’d like to have the same system on my laptop, VMs and containers. For me, systemd is a downside for Debian and I’ve tried using Alpine recently but it was too time consuming.After creating a Dockerfile for an alpine-based container with Python, PyTorch and few other things, PyTorch would not import. There’s an over a year old thread[1] about this on Stack Overflow with the same error. Apparently,
pthread_attr_setaffinity_npis not available on Alpine. However a Python module attempting to use it exists.I don’t need bleeding-edge but definitely want to avoid being forced to use outdated software when newest versions work ok. For now I find Alpine inefficient for my work. I think I’ll end up with Debian.
[1] https://stackoverflow.com/questions/70740411/cannot-import-pytorch-in-alpine-docker-container
-
What about GUIX? Being a GNU project, it’s decidedly non-commercial, and there’s plenty of active development so you won’t get outdated packages unless you pin them, and you can easily mix and match different versions. Unlike NixOS (which is tightly linked to systemd), it uses its own init system (GNU shepherd).
Best of all, if you use derivations in your dev projects you won’t even need containers and VMs. But it should run inside those just as well, if you want to use them.
-
-
-
Use Debian. Yes, it has systemd, so what? I use Debian for many years, and I’m happy. (Sometimes there are some driver issues, wi-fi, hibernation, but this is same for every distro). Systemd is cool. It allows me to create systemd unit file, start it and then query systemd whether the service is started. Systemd keeps state. Compare this with sysvinit.
If you really-really hate systemd, consider devuan.
Also, keep in mind one particular problem with debian (possibly applies to all distros): when you run new debian release in docker in old debian release, sometimes everything breaks, such as here: https://github.com/debuerreotype/docker-debian-artifacts/issues/122 . Just add
"seccomp-profile": "/etc/docker/seccomp.json"to/etc/docker/daemon.jsonand put{}to/etc/docker/seccomp.json(I can add more details).
-
-
Yes this is a great paper. It’s a perspective from kernel implementers, and the summary is that fork() is only good for shells these days, not other apps like servers :)
It reminds me that I bookmarked a comment from the creator of Ninja about using posix_spawn() on OS X since 2016 for performance:
https://news.ycombinator.com/item?id=30502392
I think basically Ninja can do it because it only spawns simple processes like the compiler.
It doesn’t do anything between fork() and exec(), which is how a shell sets up pipelines, redirects, and does job control (just implemented by Melvin in Oil, with setpgid() etc. )
Also, anything like OCI / Docker / systemd-nspawn does a bunch of stuff between fork() and exec()
It would be cool if someone writes a demo of that functionality without fork(), makes a blog post, etc.
Related from the Hacker News thread:
As a way to help alternative kernel implementers, I’d be open to making https://www.oilshell.org optionally use some different APIs, but I’m not sure of the details right now
-
It doesn’t do anything between fork() and exec(), which is how a shell sets up pipelines, redirects, and does job control (just implemented by Melvin in Oil, with setpgid() etc. )
Most of those things can be done with posix_spawn_file_actions_t.
posix_spawn_file_actions_t a,b; int p[6];pid_t c[2]; pipe(p),pipe(p+2),pipe(p+4); posix_spawn_file_actions_adddup2(&a,*p,0); posix_spawn_file_actions_adddup2(&a,p[3],1) posix_spawn_file_actions_adddup2(&b,p[2],0); posix_spawn_file_actions_adddup2(&b,p[5],1); posix_spawn(c,"a",&a,NULL,...); posix_spawn(c+1,"b",&b,NULL,...); // write to p[1] to put into head of pipeline (a) read the end of the pipelline (b) from p[4] // wait for *c(a) and c[1](b) for status code.You can get setpgid with POSIX_SPAWN_SETPGROUP, and the signal mask with posix_spawnattr_setsigmask.
-
Oh cool, I didn’t know that .. Are there any docs or books with some history / background on posix_spawn()?
When I google I get man pages, but they sort of tell “what” and not “why”
And random tweets, the poorest medium for technical information :-( https://twitter.com/ridiculous_fish/status/1232889391531491329?lang=en
I guess this kind of thing is what I’m looking for
4/ Linux does posix_spawn in user space, on top of fork - but sometimes vfork (an even faster, more dangerous fork). The vfork usage is what can make posix_spawn faster on Linux.
-
and not “why”
The “why” was largely political, IMO: The spawn() family of functions were introduced into POSIX so that Windows NT could get a POSIX certification and win government contracts, since it had spawn() and not fork().
Tannenbaum’s Modern Operating Systems was written around this time, and you might find its discussion of process-spawning APIs interesting: He doesn’t mention performance, and indeed Linux’s fork+exec was faster than NT’s CreateProcess so I find it incredibly unlikely NT’s omission for fork() was for performance, but more likely to simplify other parts of the NT design.
I guess this kind of thing is what I’m looking for
The suggestion to run a subprocess that calls tcsetpgrp before exec isn’t a bad one, and maybe obviates some of the performance benefits you get from posix_spawn, but it might not be so bad because that subprocess can be a real simple tiny static binary that does what it needs to and calls exec(). One day maybe we won’t have to worry about this.
Another option is to just wait+WIFSTOPPED and then kill+SIGCONT it if it’s supposed to be in the foreground.
-
since it had spawn() and not fork()
Very strange claim. AFAIK it was possible to implement fork on top of Windows NT native API right from the beginning. (I can try to find links.) And early Windows POSIX subsystems (including Interix) actually implemented fork. (This was long before WSL happened.) And Interix actually directly implemented fork on top of Windows NT native API, as opposed to very hacky Cygwin’s fork implementation.
Also IIRC the very first Windows POSIX subsystem happened before posix_spawn was added to POSIX. (Windows had a lot of different official POSIX subsystems authored by Microsoft, WSL is the last one.)
-
AFAIK it was possible to implement fork on top of Windows NT native API right from the beginning.
I think you’re thinking of zwCreateSection, but I don’t think this was a win32 API call (or even well-documented), and it takes a careful reading to see how fork could be implemented with it, so I don’t think this is the same as having
fork()– after all, there’s got to be lots of ways to get a fork-like API out of other things, including:as opposed to very hacky Cygwin’s fork implementation.
I remembered they claimed they had reasons for not using zwCreateSection but I don’t know enough to know what problems they ran into though.
-
-
He doesn’t mention performance, and indeed Linux’s fork+exec was faster than NT’s CreateProcess so I find it incredibly unlikely NT’s omission for fork() was for performance, but more likely to simplify other parts of the NT design.
It’s not quite that clear cut. Modern versions of NT have a thing called a picoprocess, which was originally added for Drawbridge but is now used for WSL. These are basically processes that start (almost) empty. Creating a new process with
CreateProcess[Ex]creates a new process very quickly, but then maps a huge amount of stuff into it. This is the equivalent ofexecve+ld-linux.sorunning and that’s what takes almost all of the time.Even on Linux,
vforkinstead offorkis faster (especially on pre-Milan x86, where fork needs to IPI other cores for TLB synchronisation).
-
-
Actually that tweet thread by the author of fish is very good.
This is exactly what Melvin just added, so looks like we can’t use it ? On what platforms?
8/ What’s missing? One hole is the syscall you’ve never heard of: tcsetpgrp(), which hands-off tty ownership. The “correct” usage with fork is a benign race, where both the parent (tty donator) and child (tty inheritor) request that the child own the tty.
9/ There is no “correct” tcsetpgrp usage with posix_spawn: no way to coax the child into claiming the tty. This means, when job control is on, you may start a process which immediately stops (SIGTTIN) or otherwise. Here’s ksh getting busted: https://mail-archive.com/ast-developers
-
so looks like we can’t use it ?
https://github.com/ksh93/ksh/blob/dev/src/lib/libast/comp/spawnveg.c is a good one to look at; you can see how to use posix_spawn_file_actions_addtcsetpgrp_np and if/when POSIX_SPAWN_TCSETPGROUP shows up you can see how it could be added.
-
Hm in the Twitter thread by ridiculous_fish he points to this thread:
https://www.mail-archive.com/ast-developers@research.att.com/msg00718.html
The way I’m reading this is that ksh has a bug due to using posix_spawn() and lack of tcsetpgrp(). And they are suggesting that the program being CALLED by the shell can apply a workaround, not the shell itself!
This seems very undesirable.
I think we could use posix_spawn() when job control is off, but not when it’s on.
And I actually wonder if this is the best way to port to Windows? Not sure
-
-
-
XNU has a lot of extensions to POSIX spawn that make it almost usable. Unfortunately, they’re not implemented anywhere else. The biggest problem with the API is that they constrained it to permit userspace implementations. As such, it is strictly less expressive than vfork + execve. That said, vfork isn’t actually that bad an API. It would be even better if the execve simply returned and vfork didn’t return twice. Then the sequence would be simply vfork, setup, execve, cleanup.
With a language like C++ that supports RAII, you can avoid the footguns of vfork by doing
pid_t pid = vfork(); if (pid == 0) { { // Set up the child } execve(…); pid = -1; }This ensures that anything that you created in between the setup is cleaned up. I generally use a
std::vectorfor theexecvearguments. This must be declare in the enclosing scope, so it’s cleaned up in the parent. It’s pretty easy to wrap this in a function that takes a lambda and passes it a reference to theargvandenvpvectors, and executes it before theexecve. This ensures that you get the memory management right. As a caller, you just pass a lambda that does any file descriptor opening and so on. The wrapper that I use also takes a vector of file descriptors to inherit, so you can open some files before spawning the child, the do arbitrary additional setup in the child context (including things like entering capability mode or attaching to a jail).-
It would be even better if the execve simply returned and vfork didn’t return twice. Then the sequence would be simply vfork, setup, execve, cleanup.
So, a few very common cases that would break:
-
Redirecting stdin/stdout/stderr. How do you preserve the parent’s stdin/stdout/stderr for the cleanup step without also passing it to the child?
-
Changing UID/GID. Whoops the parents is now no longer root, and can’t change back.
-
Entering a jail/namespace. Again, the parent is now in that jail, so it break out without also leaving the child with an escape hatch.
Basically anything that locks down the child in some way, will also affect the parent now.
-
I don’t understand how any of those use cases break. In between
vforkandexecve, you are running in the child’s context, just as you are now. You can drop privileges, open / close / dup file descriptors, and so on. The only difference is that you wouldn’t have the behaviour whereexecveeffectivelyjongjmps back to thevfork, you’d just return to running in the parent’s context after you started the child.-
Ok, so I think I see what you mean now.
At the point of vfork, the kernel creates two processes using the same page tables. One is suspended (parent), and the other isn’t (child).
The child continues running until execve. At that point the child process image is replaced with the executable loaded by exec. The parent process is then resumed but with the register/stack state of the child.
That would actually work pretty nicely.
-
Exactly. The vfork call just switches out the kernel data structures associated with the running thread but leaves everything else in place, the execve would switch back and doesn’t do any of the saving and restoring of register state that makes vfork a bit exciting. The only major change would be that execve would return to the parent process’ kernel state even in case of failure.
-
-
-
-
-
-
posix_spawn is documented in POSIX: https://pubs.opengroup.org/onlinepubs/9699919799/ , specially here: https://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_spawn.html . This manpage contains big “RATIONALE” section. And “SEE ALSO” section with
posix_spawnattr_*functions -
Linux does posix_spawn in user space, on top of fork - but sometimes vfork
Glibc and musl add more and more optimizations over time, allowing posix_spawn to use vfork (as opposed to fork) in more and more cases. It is quite possible that recent versions of glibc and musl call vfork in all cases.
AFAIK this glibc bug report https://sourceware.org/bugzilla/show_bug.cgi?id=10354 is resolved using patchset, which makes glibc always use vfork/CLONE_VFORK.
Using vfork is not simple. This article explains how hard it is: https://ewontfix.com/7/
-
-
-
I can post code of my
unshare(1)analog. It seems I can simply add CLONE_VFORK option to list ofcloneoptions and everything will work-
That would be cool! What do you use it for?
I would like to do some container-like stuff without Docker, e.g. I’ve looked at bubblewrap, crun, runc, etc. a little bit
We have a bunch of containers and I want to gradually migrate away from them
I also wonder if on Linux at least a shell should distribute a bubblewrap-like tool ! Although I think security takes a long time to get right on Linux, so we probably don’t want that responsibility
-
Here is my util, I call it “asjail” for “Askar Safin’s jail”:
https://paste.gg/p/anonymous/4d26975181eb4223b10800911255c951
It is public domain. I’m sorry for Russian comments.
Compile with
gcc -o asjail asjail.c. My program is x86 Linux specific. Run like so:asjail -p bash.Program has options
-imnpuU, which correspond tounshare(1)options (see its manpage). (Also, actual unshare has option-r, I have no such cool option.)My program usually requires root privileges, but you can specify
-Uflag, which creates user namespace. So, you can runasjail -pU bashas normal user, this will create new user namespace and then create PID namespace inside it. (Again:unshare -pr bashis even better.)But user namespace requires that they should be enabled in kernel. In some distros they are enabled by default, in others - not.
I wrote this util nearly 10 years ago. Back then I wanted some lightweight container solution. I was not aware of
unshare(1)util.unshare(1)fully subsumes my util. (Don’t confuse withunshare(2)syscall.) Also,unshare(1)is low-level util, it’s lower then bubblewrap, runc, etc.I don’t remember some details in this code, for example I don’t remember why I need this signal mask manipulations.
Today I use docker and I’m happy with it. Not only docker provide isolation, it is also allows you to write Dockerfiles. And partial results in dockerfiles are cached, so you can edit some line in dockerfile, and docker will rebuild exactly what is needed and no more. Dockerfiles are perfect for bug reports, i. e. you can simply send dockerfile instead of “steps to reproduce” section. The only problem with docker is inability to run systemd inside it. I have read that this is solved by podman, but I didn’t test it. Also, dockerfiles are not quite reproducible, because they are often rely on downloading something from internet. I have read that proper solution is Nix, but I didn’t test it
-
Additional comments on asjail (and other topics).
You need to add
--to make sure options are processed as part of target command, not asjail itself, i. e.asjail -p -- bash -c 'echo ok'.asjail was written by careful reading of
clone(2)manual page.asjail is not complete solution. It doesn’t do chrooting, mounting needed directories (/proc, /dev, etc). So back then 10 years ago I wrote bash script called
asjail-max-1, which does these additional steps.asjail-max-1was written by careful reading of http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface and soasjail-max-1(together withasjail) can run systemd in container!asjail-max-1is screen sized bash script.But, unfortunately, to make all this work, you also need C program, which emulates terminal using
forkpty(3). So I wrote such a program and I called itpty. It is some screens sized C program. Togetherasjail,asjail-max-1andptygives you complete small solution for running containers. In something like 5 screens of code.I can post all this code.
But all this is not needed, because all this is subsumed by existing tools.
asjailis subsumed byunshare. Andasjail-max-1+asjail+ptyis subsumed bysystemd-nspawn.Today I don’t use any of these my tools. When I need to run container I use docker. If I need to run existing file tree I use systemd-nspawn
-
Also, all these tools I discussed so far are designed for maximal isolation. I. e. to prevent container from accessing host resources. But sometimes you have opposite task, i. e. you want to run some program and give it access to host resources, for example to X server connection, to sound card etc. So I have another script, which is simple wrapper for chroot, which does exactly this: https://paste.gg/p/anonymous/84c3685e200347299cac0dbd23d31bf3
-
-
-
Also a good paper about old POSIX APIs not being a great fit for modern applications (Android and OS X both use non-POSIX IPC throughout, etc.)
POSIX Abstractions in Modern Operating Systems: The Old, the New, and the Missing (2016)
https://roxanageambasu.github.io/publications/eurosys2016posix.pdf
https://news.ycombinator.com/item?id=11652609 (51 comments)
https://lobste.rs/s/jhyzvh/posix_has_become_outdated_2016 (good summary comment)
https://lobste.rs/s/vav0xl/posix_abstractions_modern_oses_old_new (1 comment 6 years ago)
-
I think basically Ninja can do it because it only spawns simple processes like the compiler.
It doesn’t do anything between fork() and exec(), which is how a shell sets up pipelines, redirects
This is not true. Ninja redirects output of command so that outputs of multiple parallel commands don’t mix together
-
I want to write my own shell some day. And I don’t want to use
forkthere, I will use posix_spawn instead. But this will be tricky. How to launch subshell? Using/proc/self/exe? What if/procis not mounted? I will possibly try to send a patch to Linux kernel to make/proc/self/exeavailable even if/procis not mounted. Busybox docs contain (or contained in the past) such patch, so theoretically I can simply copy it from there. I can try to find a link-
BTW my reading of this thread above
https://lobste.rs/s/smbsd5/fork_road#c_qlextq
is that if you want a shell to have job control (and POSIX requires job control), you should probably use fork() when it’s on.
Also I’m interested in whether posix_spawn() is the best way to port a shell to Windows or not …. not sure what APIs bash uses on Windows.
Is posix_spawn() built on top of Win32? How do they do descriptors, etc. ?
-
-
Also, anything like OCI / Docker / systemd-nspawn does a bunch of stuff between fork() and exec()
Surprisingly, we can combine posix_spawn speed with needs of docker and systemd-spawn. Let me tell you how.
First of all, let’s notice that merely fork is not enough for systemd-nspawn. You need to also put child in new mount/utc/etc namespace. For this you need
cloneorunshare(2).Fortunately,
clonehas flagCLONE_VFORK, which allows us to get vfork-like behavior, i. e. our program will be faster than with fork.So, to summarize, we have
two optionsone option to combine posix_spawn speed with features systemd-nspawn needs(either one will be enough):- Use
clonewith all namespacing flags we need (such as CLONE_NEWNS) and CLONE_VFORK Create new process using usual posix_spawn and then callunshareto put the process into new namespace
I didn’t tested any of this, so it may be possible something will go wrong.
Also: I’m author of my own
unshare(1)util analog (don’t confuse withunshare(2)syscall). My util doesn’t do any speed up tricks, I just use plainclonewithout CLONE_VFORK - Use
-
Also, I wrote a Rust program, which spawns program using posix_spawn, redirects its in, out and err using posix_spawn_file_actions_adddup2, collects its out and err, waits for finish and reports status
-
-
I was reminded of this by the discussion about problems with unix process APIs. In general I like the idea of a capability-oriented process API, which is basically what pidfd is. Then if process-manipulating syscalls require a process capability, operating on other processes is just like operating on your own, which also solves the issue of awkwardly huge
spawnAPIs, as in section 6 “Replacingfork- Low-level: Cross-process operations.”I think there’s a contradiction in this paper, tho: I think in principle if it’s possible to explicitly donate a share of parts of the current process’s state to a new empty process, then it’s possible to implement
forkin userland, which is basically what that subsection of section 6 says. But that contradicts section 5 “Implementingfork- Fork infects an entire system” which claims “an abstraction, fork fails to compose: unless every layer supports fork, it cannot be used.”-
I don’t think there’s a contradiction. A userland fork would also not compose with other things, but having a feature in one library that doesn’t compose with features in other libraries is not a new problem: it’s inherent to the UNIX shared library model (and a regression from MULTICS).
-
basically what pidfd is
Modern versions of Linux’s clone syscall allow CLONE_PIDFD flag, which allows to immediately get pidfd!
-
operating on other processes is just like operating on your own, which also solves the issue of awkwardly huge spawn APIs, as in section 6 “Replacing fork - Low-level: Cross-process operations.”
This problem is solved using io_uring_spawn (which is AFAIK not mainline yet). See https://lwn.net/Articles/908268/ and my comment below: https://lobste.rs/s/smbsd5/fork_road#c_vf4hnp
-
-
I’m excited to see if
io_uring_spawngets momentum. (https://lwn.net/Articles/908268/)-
io_uring_spawn is very good thing!!
If you like this article (“fork in the road”), you will like io_uring_spawn, too. I think io_uring_spawn solves all fork problems, while being faster than all other solutions, even faster than vfork and posix_spawn. Slides above include table with time comparison.
Also, “fork in the road” includes this statement:
…clean-slate designs [e.g., 40, 43] have demonstrated an alternative model where system calls that modify per-process state are not constrained to merely the current process, but rather can manipulate any process to which the caller has access. This yields the flexibility and orthogonality of the fork/exec model, without most of its drawbacks: a new process starts as an empty address space, and an advanced user may manipulate it in a piecemeal fashion, populating its address-space and kernel context prior to execution, without needing to clone the parent nor run code in the context of the child. ExOS [43] implemented fork in user-mode atop such a primitive. Retrofitting cross-process APIs into Unix seems at first glance challenging, but may also be productive for future research.
io_uring_spawn is exactly this design!!!
-
-
The article mentions (IIRC) that
forkinteracts badly withsysctl vm.overcommit_memory=2on Linux. This is very bad, becausesysctl vm.overcommit_memory=2is what you (from perfectionist view) should do always (unfortunately, this breaks some software in real world). You can read about overcommit here: https://ewontfix.com/3/
-
Zillions of C functions are incompatible with fork, including… printf!!!!! Consider this code:
#include <stdio.h> #include <unistd.h> int main () { printf ("Hello"); fork (); }This code prints “Hello” twice on my machine (Linux x86_64). Because userspace buffer is duplicated
-
-
The static typing idea is something that “seems obvious”, but will fall down at the first implementation / deployment.
Shell is about composing tools at the operating system level, and even C and C++ are not statically typed at the OS level!
Linux distributions use ABI versioning, not API versioning, e.g.
https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
They must do this because they deploy different components at different times. When you do
apt-get upgrade, components are selectively replaced without recompilation. (Compilation is the thing that enforces static types!)Windows does the same thing. When you do a Windows update, it replaces DLLs. COM and whatever the modern equivalent is are dynamic, not static. You query for interfaces at runtime.
So static typing doesn’t scale even to a single machine – on either Linux or Windows – let alone multiple machines, which is where I use shell a lot. (OS X or Windows talking to Linux is another very common use of shell!)
This is one of the main points in my post A Sketch of the Biggest Idea in Software Architecture
Another good recent post about it: https://faultlore.com/blah/c-isnt-a-language/
So shell has the same issue as C/C++ – components can be replaced independently without global versioning, and you need to evolve in a compatible way. (BTW there used to be this thing called Linux FHS, which would solve some analogous issues for shell as ABIs do for C++ – https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard – however it doesn’t seem to have reached its goals)
So basically, if you have a statically typed shell, you could only use it with a Google-like monorepo with a global version number and static linking.
Though even Google can’t redeploy every binary in a service at once, which is why protobufs have optional fields queried at runtime, not compile time. e.g. see Maybe Not by Hickey: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey)
So Google would still need a dynamically typed shell. Also, feature flags are the idiomatic way to upgrade services there as well. Yes, literal command line flags to server binaries, even though they have a monorepo and protobuf definitions.
https://featureflags.io/feature-flags/
So shells aren’t statically typed for the same reason that the Internet isn’t statically typed: you can’t upgrade everything at once. You don’t download the latest version of TCP/IP and link it into your programs like you do a gRPC definition. Instead you use the protocols dynamically (binary in the case of TCP/IP; text in the case of HTTP, HTML, JSON, etc.)
-
OP is talking about static types in the language level, and you seem to be talking about static vs dynamic linking, which … I think are kind of almost unrelated?
-
A shell is intrinsically a dynamic linker, just not in the sense that the term is routinely used. The purpose of a shell is to run other programs. Those other programs have interfaces (command-line flags, supported input and output) that change over time. You cannot map them into a static type system without also baking in the supported version. New tools are installed more frequently than a shell is upgraded and add to the space of shell types.
The .NET team had some interesting ideas about adding a section to binaries that described the tool’s interfaces and allowed them to be surfaced in PowerShell, but even then a static type system can express only the tools installed when the program is written.
-
-
-
static typing doesn’t scale even to a single machine – on either Linux or Windows – let alone multiple machines, which is where I use shell a lot
https://www.unison-lang.org/ tries to scale static typing to internet level. They avoid compatibility problems using novel method: code (and types) stored in binary form in content addressable storage (I think this is Merkle tree). If old code (and types) are used, they are not deleted and are kept around forever. (Unless you stop to use them, then they are AFAIK garbagge-collected.)
-
Unison definitely has some cool ideas, and I’m glad someone is trying it
But it’s very far from how computers actually work today, and ironically their theory is to NOT have a shell at all, whether statically typed or not.
They’ll certainly have to solve open computer science problems in order to scale to even what a single Windows or Linux machine has today – e.g. ~30M to 100M lines of code. Let alone writing something like Google Maps, Twitter, etc. I’d guess they’re 100x to 1000x slower right now, or maybe exponentially slower, though I’d be interested in some real data
-
-
-
SirJosh
4 months ago
|
link
| on:
EcmaScript spec is ready-to-use interpreter. It is attribute affix grammar. It is PLT paradise
I’d have to strongly disagree with the warm feelings towards the grammar, speaking as a beginner that tried to hack around with it. It definitely has its strong points, and like anything else could improve - but I firmly disagree with the idea of it being a ready-to-paste interpreter.
Most notably, the tree-walking nature of the grammar hurts to comprehend. Particularly, the “Runtime Semantics”, “Static Semantics”, and possibly other Syntax-Directed Operations are hard to grapple with due to their loosely defined almost polymorphic nature.
For example, the
ExpectedArgumentCounthas a piece-wise definition over a set of parse nodes, in particularFormalParameters. It defines static semantics for only these two:FormalParameters : [empty] FunctionRestParameter 1. Return 0. FormalParameters : FormalParameterList , FunctionRestParameter 1. Return ExpectedArgumentCount of FormalParameterList.However, the
FormalParametersis defined as these possible cases:FormalParameters[Yield, Await] : [empty] FunctionRestParameter[?Yield, ?Await] FormalParameterList[?Yield, ?Await] FormalParameterList[?Yield, ?Await] , FormalParameterList[?Yield, ?Await] , FunctionRestParameter[?Yield, ?Await]It forgets about defining a production for
FormalParameterListandFormalParameterList ,! You can encounter this behavior being necessary with a function such asfunction f(x) {}. I had to add cases for these two scenarios in my code, without the corresponding ECMAScript specification to back me up.It seems innocuous here, but this is just one strong example I have. The rest of my criticism stems from my mental frustration regarding it being difficult for an onlooker to piece together where productions and static semantics fit in code inside an interpreter. Essentially a criticism of the polymorphic nature of being able to stick on new static rules, as opposed to something more static/concrete that can easily fit in, such as seeing a pseudo-code switch statement.
My second point is regarding the entrypoint of the spec. Where does an interpreter begin? Where do realms and agents begin/get initialized? I roughly pieced together an idea after combing through the spec, but it was not apparent in the slightest.
Essentially, the routine for my interpreter goes:
- Parse the AST into a ton of runtime objects so we can tree-walk them
- Call
InitializeHostDefinedRealm - Create any host-defined hooks (e.g. setting values in the global object/realm)
- Call
ParseScript - Call
ScriptEvaluation
I fail to see any references to ScriptEvaluation, and no significant ones to
InitializeHostDefinedRealmorParseScriptthat would lend itself to this initialization routine. Oh, and I didn’t implement any early bailing errors properly, I just assume the script is valid. Good luck combining all the static semantics in relation to parse errors!
My last point is regarding async and generators. I was happily implementing things, until I came across some scary lines of text that had huge implications for the entire design of my interpreter. Specifically, I’m thinking about the
GeneratorStart,GeneratorResume, andGeneratorYieldfunctions. They contain(ed[^1]) lines of pseudocode such as these:-
GeneratorStart: 4. Set the code evaluation state of genContext such that when evaluation is resumed for that execution context the following steps will be performed: -
GeneratorResume: 9. Resume the suspended evaluation of genContext using NormalCompletion(value) as the result of the operation that suspended it. Let result be the value returned by the resumed computation. -
GeneratorYield: 7. Set the code evaluation state of genContext such that when evaluation is resumed with a Completion resumptionValue the following steps will be performed:
Thankfully, these are able to be implemented with closures. I looked to engine262 and saw it use
yield*to implement these, and was deeply concerned. It remains to be known if something like this will happen in the future.
Of course, this is just my take on it after trying to implement enough to get function calls barely working. I’m by no means an expert, and this is just my take on some of the negative sides of the specification. Additionally my criticisms of the specification stem from the draft version present circa June 21, 2021 published on https://tc39.es/ecma262/ and it’s possibly things have changed. In-fact, they have:
[^1]: Looks like as of today, these lines no longer exist and they implement them with closures. That’s great as that’s how I decided to implement them as well!
-
You can tell them your criticism to bug tracker ( https://github.com/tc39/ecma262/issues ).
My original point was this: ECMAScript spec is a lot more precisely specified than other specs, such as c and c++ spec or rust reference and even rust ferrocene spec. ECMAScript is (despite its shortcomings) the most precisely specified language among popular ones. ECMAScript sets high bar
-
wezm
4 months ago
|
link
| on:
EcmaScript spec is ready-to-use interpreter. It is attribute affix grammar. It is PLT paradise
Not sure I share this enthusiasm for the ECMAScript spec. Last year I extended the JavaScript implementation in Prince with a number of features including lexical bindings and for-of. I found that the description of behaviour was spread all over the spec making it very hard to know when all behaviours had been captured—and let me assure you there is no shortage of edge cases in JavaScript. Thank goodness for the test262 test suite.
-
I wouldn’t have a job today if I hadn’t decided to mess around and do my dissertation in Prince instead of focusing on my studies. Thanks, Prince rules.
-
-
Yes.
-
-
-
I meant that ECMAScript spec has a lot more rigor than, say, C or C++ spec. ECMAScript is somewhat “executable”. It is easy to convert it to actual (slow) interpreter and then compare practical (fast) implementations against it. This is not possible with C or C++ spec
-
-
Doesn’t work
d-user@comp:~$ curl -s https://skiqqy.xyz/skiqqy.1 ^C d-user@comp:~$Hangs
-
curl -s https://skiqqy.xyz/skiqqy.1
Wonder if skiqqy’s machine is down mmmmh
-
-
It does not matter how many times I try, I always forget the order of the parameters for the SSH tunnels. I should put this as a desktop background to see if it finally sticks.
-
“-L 1111:aaa:2222” means “listen on local port 1111”
“-R 1111:aaa:2222” means “listen on remote port 1111”
-
Here is another solution to described problems: Niko’s moro ( https://github.com/nikomatsakis/moro/ ). It allows one to spawn futures in parallel (using
scope.spawn), thus preventing many deadlocks, and it the same time referencing local variables! (When you poll one of those futures started usingscope.spawn, you automatically poll all other futures.)