Or maybe shells were pretty much built to be primitive User interfaces and not serious “typed” languages. Stop trying to see shells as “scripting” and a lot of your problems pretty much become secondary.
Seriously, just use a REAL scripting language, then you can readdir() or whatever without having to depend on ls and all the weird ways you can invoke it.
While I understand your argument, you should consider that we use the term “script” instead of, say, “command” exactly because scripts were originally a sequence of shell commands saved in a file for future convenience.
So there is a continuous between the glue provided by a shell and a full interpreted programming language.
Shell were designed to glue small programs providing specific features into larger ones.
Interpreted language are designed to write larger programs in the first place, by composing the available libraries that provide the specific features.