Has someone figured out how does this work?
Linux can emulate of real mode within a protected mode task. After the DOS program first jumps to Linux, the return to DOS happens inside of that environment, and there Linux can handle software interrupts to receive the commands from the DOS program and run them. (And make console output work etc.)
(BTW I’m pretty sure this kind of emulation was commonly used in early, Userspace Mode Setting video drivers)
Also the dosemu program.
I used the vm86() system call to implement just enough of MS-DOS to run a program. I couldn’t use Dosbox because that didn’t allow me to redirect the input/output of the MS-DOS program to a Unix program. It works on x86-32 bit Linux systems, I don’t know if that system call is available anymore on an x86-64 bit systems.
I’ve thought about doing the opposite: a DOS Linux program loader that traps int 80h and implements most of the Linux syscall ABI.
I looked into this, but I think it’s no longer viable because 1) you’d have to jump into pmode, and 2) modern linux programs tend to use the syscall instruction instead of int 80h. I guess maybe if you compile it for i386 & use the pmode-friendly later MS-DOS that shipped with windows 98…
(I haven’t looked at this in 15+ years though, nor have I done substantial x86 assembly development since then, so my memory may be warped; also, I was a young teenager back then, so I might have misunderstood in the first place.)
I mean, you’d have to use an extender/TSR, I’m sure of that. You could handle things in there. I think you’d also have to do things like translating paths, not supporting fork (then you’re just implementing Linux), etc.
Not supporting fork is going to break basically all actually-existing linux executables, save the handful already present in (say) DJGPP – especially if not implementing fork means not implementing exec. I figure that to have a meaningful linux-on-DOS experience you’d need to at least implement trivial task switching in userspace for all the linux tools that are invoked simultaneously on a pipeline.