I think this is actually a good demonstration of pipes, because it shows that even if you don’t know the -c flag you can kludge together the same thing with pipes. That makes them a lot more accessible!
This seems like a great idea at first, but for anything in production it’s much better to build a library of primitives and write scripts in something with better error checking than bash.
What ends up happening is you end up with hundreds of these tiny scripts like in Moses and it quickly becomes very cumbersome to refactor and debug.
It’s a good point that command-line pipes won’t scale with training complexity. An escape hatch here is the fact that learn.ts is just a regular TypeScript file that can be used for more advanced logic (import model from './learn' will work with everything shown in the post).
The target use cases (for my work at least) are very much akin to the second example presented. Lots of simple data + small models that can be used for embedded heuristics. These certainly have their place in production!
I think for prototyping it’s great, and definitely a slick alternative to using a notebook. Maybe the best of both worlds is writing libraries but if you must have commands make them amendable to piping like in your post.
This looks amazing. I worked in production ML systems and this looks like it could power online learning models. I like how it looks nicer than writing your own input and output routines in Python.
As everyone knows, this is the worst demonstration of pipes:
grep -c '^a..$' /usr/share/dict/wordsI think this is actually a good demonstration of pipes, because it shows that even if you don’t know the
-cflag you can kludge together the same thing with pipes. That makes them a lot more accessible!I’d say this is an even worse demonstration of pipes 😉
If you happen to have a nice introductory demonstration of pipes that you’d be willing to share, I’d really appreciate it!
Related: https://blog.sanctum.geek.nz/useless-use-of-cat/
This seems like a great idea at first, but for anything in production it’s much better to build a library of primitives and write scripts in something with better error checking than bash.
What ends up happening is you end up with hundreds of these tiny scripts like in Moses and it quickly becomes very cumbersome to refactor and debug.
It’s a good point that command-line pipes won’t scale with training complexity. An escape hatch here is the fact that
learn.tsis just a regular TypeScript file that can be used for more advanced logic (import model from './learn'will work with everything shown in the post).The target use cases (for my work at least) are very much akin to the second example presented. Lots of simple data + small models that can be used for embedded heuristics. These certainly have their place in production!
I think for prototyping it’s great, and definitely a slick alternative to using a notebook. Maybe the best of both worlds is writing libraries but if you must have commands make them amendable to piping like in your post.
This looks amazing. I worked in production ML systems and this looks like it could power online learning models. I like how it looks nicer than writing your own input and output routines in Python.
Would have loved a more practical example.