My take on this is that languages and libraries are working on fragments of the bigger issue from different angles.
What’s needed in my opinion is an approach which separates the composition of those “bulk operations” (I’m using that term due to the lack of a better one) from the way they are evaluated.
I’d really like to see an approach which satisfies the following requirements:
Efficient by design (no “let’s hope the optimizer/compiler rewrite rules/… figure this out”). This means not creating intermediate data structures and not running operations prematurely.
Unified API which can support all implementations and input sources. I think it’s a shame that many languages have APIs which work on different data types which share the same operations, but have no common type.
Supports both push and pull models (event streams, collections, …).
Reification of operations (to allow reasonable queries against databases as an input source). This means it very likely required to have dual representation of code available: Both as code itself and as an AST, so that the right representation can be picked by the implementation, (i. e. collections use the code, databases use the AST).
I think you’re asking for the moon there. Unifying the advantages of the haskell/scala/clojure approaches is difficult enough without adding all those extra requirements.
I think the requirements are reasonable.
Will it be easy to design such an API? No.
Will it be hard to implement? Absolutely.
Nevertheless, I think the current state of the art is embarrassing. Scala is one of the most expressive languages out there, and it fails utterly at providing a common abstraction for bulk operations.
I think we can do better than that.