I wonder if, instead of parsing the x86 instructions, you could preserve the closure’s LLVM IR until runtime. Then to specialize a closure, you’d grab the IR and the closed-over values and feed them back in to LLVM’s optimizer.
You’d be very tied to LLVM, but in return you’d be less tied to x86.
(I’m the author)
That’s essentially what the holyjit project I linked does. It vendors a custom rustc so that it can also emit the LLVM bitcode with executables that you emit, and then uses the bitcode in order to feed it back into LLVM at runtime for JITting arbitrary interpreters.
Truth be told, I’m not entirely sure why development stalled on it. I think they had problems with vendoring rustc and using LLVM at runtime, since I imagine you have less control over what’s going on if you’re only able to shove a bitcode blob at LLVM and hope it does the right thing. There’s a very good chance that specializing it towards closure generation compilers like Lineiform does instead would make it more tractable, if that was even the issue, and that I should’ve done that instead since x86 dynamic recompilation is definitely a pretty weird approach.
Fun fact, I actually did want to try targeting ARM instead once I hit the Cranelift issues, since I think it would avoid most of the processor flag behavior (and the reduced instruction set would make writing a lifter a lot easier). I tried signing up for the Oracle Cloud Ampere instances, but they were at capacity, and I don’t have a good ARM computer myself, so I just figured I’d stick to x86.
That’s very close to what Google’s NaCL did.
In the past, I wondered why closure generation compiler is not widely known, given that it is very well explained in SICP 4.1.7. (SICP cites the same paper, Marc Feeley 1987.) The answer, obvious in retrospect, is that approximately no one has read SICP. When I was young, I thought many have read SICP, since it is so highly praised.