I’ve been going in the other direction. I now use libraries like lens and lens-aeson inside of ghci more and more often. These provide a consistent API for whatever data I’m working with.

I can work with YAML, TOML, JSON, CSV, XML, etc. with the exact same functions. No need to learn separate tools like jq, yq, xmlstarlet, etc.

What I’ve observed is that sometimes these shell pipelines are only executed once, and then writing them saves a lot of time. But sometimes someone wants to run them again. Or even a few times. Occasionally, someone puts them in a crontab somewhere to be run repeatedly until further notice. And then maybe, just maybe, they become part of a production system and they are supposed to run in various environments.

At some point along this chain of events, it would have been easier from a maintenance perspective to just write the program to begin with. But I’m still terrible at judging ahead of time whether my shell pipelines are truly one-off tasks, or if they’re going to grow into a part of the production system.

“But why don’t you just write the shell pipeline first, and then port it to a proper program at some point in the future?” I hear you ask. That’s a perfectly valid solution. It’s also a really boring task to translate a shell pipeline into a program – all the fun of the implementation design work has already been done, and what’s left is a grudging mechanical task of translation. So I tend to put it off…

I do prefer it over jq, but the = and ; in the output are annoying.

serious question: what does “doing some awk in Ruby” look like? This might be a pretty big motivator for me to finally figure out Ruby for scripting (I’m more of a Python guy myself but awk works nicely for small scripts on line-oriented stuff when I want a one-liner)

Compare:

# Official way of naming Go-related things:
$ grep -i ^go /usr/share/dict/* | cut -d: -f2 | sort -R | head -n1
goldfish

Versus Ruby:

puts(Dir['/usr/share/dict/*-english'].map do |f|
  File.open(f)
    .readlines
    .select { |l| l[0..1].downcase == 'go' }
end.flatten.sample.chomp)

Simple example, but I think it demonstrates that doing various basic and common tasks are quite a bit more complex to do in Ruby than in the shell.

That doesn’t mean I’m always in favour of shell scripts – I got that example from an article I wrote saying you shouldn’t use shell scripts – but there are definitely reasons shell scripting persists, even though we have things like Perl and Ruby.

In that article I wrote “I regret writing most shell scripts [..] and my 2018 new year’s pledge will be to not write any more”. I’ve mostly failed at that new years’ pledge, and have happily continued shelling about. I have started rewritting shell script prototypes to other languages at the first sign of getting hairy though, and that seems like a middle ground that is working well for me (I should update/ammend that article).

Because code in sed or awk that worked a decade ago (or, hell, two years) still works. Ruby code seems to bit rot faster than any other language I’ve use for nontrivial work.

Also, with awk, I could put it down for a year, then use it again, and everything I’d need to be productive fits in a small man page. (The same seems to be true of sed, though I don’t use it much.) The Ruby ecosystem moves a lot faster, and if you haven’t been following it closely, catching up will add extra work. (Whether it’s actually going anywhere is neither here nor there.)

Yes, awk is a more limited language, but that’s a trade-off – there are upsides, and I know which I’d prefer.

I don’t know Ruby. But for me these are the reasons why I am writing more and more bash programs:

• Bash is my command line. So I am doing a lot of small steps, file modifications, comparing, searching analysing. At some point I can see that some of the steps can be composed and I pull them out of the history, try them out on the console and at some point put them into a script. If Ruby would have a REPL in which I can do all the operations that I am doing on the command line with less typing and more comfort, I would maybe give it a try.

• Bash is on every Linux box. Ruby is not.

Mostly, because picking up enough jq, awk and sed to be useful is faster than learning the ins and outs of Ruby?

I suppose you could make a similar argument about learning Ruby one-liners, but by the time I’m writing a very long bash script, I’m probably writing a larger program anyway, either in Go or Python. Ruby as a language doesn’t have much appeal to me, at least at the moment.

Awk, at least, fits very nicely into a small space right next to regex. jq is a bit fiddilier to pick up, but very nice for basic stuff. Sed, I still don’t have down very well, but also is nicely regex adjacent.

I regularly write sed one liners to do refactorings on my Ruby code. Usually the sed call is fed by the result of grep or find. I could write a Ruby one liner to do the same, but it would be a much longer line and escaping would be much more difficult. Ruby is simply not a replacement for the convenience of sed.

And maintainability is a red herring here: the whole point of something like sed is that you use it for one-off commands.

I’m not that experienced with jq, but when it comes to awk (and sed), one of their benefits is that you can easily write a program in the shell, since they act as glue between pipe operations.

For example, to filter out all lines that have less than 4 characters, all you have to write is

... | awk 'length >= 5' | ...

no imports or types required. It was made for stuff like this, which makes it easy to use. I’ve only read a book about Ruby a few years ago, but to process stdin/out this was should require a bit more overhead, shouldn’t it?

One part of your history lesson is missing: Paul McCarthy and Steve Russell saw what was going to happen and pre-emptively invented Lisp. And yes, you can do everything in Lisp, in one line if you must, that you can do in bash, awk, sed, jq, perl… but in a more powerful and maintainable form.

;)

Ruby equivalents of the basic awk and sed examples from the article, as examples of Ruby one-liner structure:

• AWK: awk '{print $1}' logs.txt
  • Ruby: cat logs.txt | ruby -ne 'puts $_.split[0]'
  • Ruby: cat logs.txt | ruby -ane 'puts $F[0]'
• sed: sed 's/^[^ ]*//' logs.txt |sed 's/"[^"]*"$//'
  • Ruby: cat logs.txt | ruby -ne 'puts $_.gsub(/^[^ ]*/, "").gsub(/"[^"]*"$/, "")'