I love stream fusion, but whenever I load anything in pandas, it’s to do a lot of different operations and do “Excel”-y operations where having the entire data in memory is usually what I want.
If you’re dealing with streams in python, pandas shouldn’t really be what you’re grabbing at. Honestly, a simple file readline will do this stuff pretty well (there are likely better tools too!)
Seriously, though, Haskell’s stream fusion tech is utterly amazing. I want it everywhere.
I love stream fusion, but whenever I load anything in pandas, it’s to do a lot of different operations and do “Excel”-y operations where having the entire data in memory is usually what I want.
If you’re dealing with streams in python,
pandasshouldn’t really be what you’re grabbing at. Honestly, a simple filereadlinewill do this stuff pretty well (there are likely better tools too!)Seriously, though, Haskell’s stream fusion tech is utterly amazing. I want it everywhere.