A lot of programmers haven’t ever really learnt about the shell, which is a shame. They’ll use it for git commands and make -k but won’t know how to use POSIX utils to optimise their workflow. I wonder if there’s a way to encourage people to learn about shell?
I wonder if there’s a way to encourage people to learn about shell?
Every Friday from 4pm to 5pm, our engineering team at work has an (optional) activity: one person presents on a topic of their choice, while the rest drink a beer and wind down for the weekend. I personally gave presentations on using awk, jq, and git. In the Unix realm, we’ve also had presentations on the basic Unix stdin/stdout model, on the pitfalls of bash, on perf, etc.
If you have a venue like this, prepare a presentation on one or two Unix tools (don’t over-do it and make sure people can leave with actionable knowledge). If you don’t have a venue like this, see if you can’t make one happen. Sometimes, people just don’t know what they don’t know; they need someone to step up and say “here, let me teach you something that is simple, useful, and fun!”
I think the article is a bit black and white. Between the shell and the cloud, writing a small python / ruby / perl script was also a valid solution. AWK is a fairly large and powerful programming language in itself, and heck, Bash is too. So the takeaway is more: try to create a simple local solution first, with whatever you know?
I write bash scripts from time to time, and consider myself proficient, though I rarely touch AWK. I have colleagues who don’t write bash scripts at all outside of sequencing some simple commands, and wouldn’t even touch sed, but I still consider them proficient (in the way they handle interactive mode).
How I learned shell, and how I see other people around me learn shell, is pretty much through experience, googling, and finding tricks in articles somewhere, so the discoverability seems pretty awful. I also rarely see colleagues reach for manpages, and info must be the place where documentation goes to die or something.
On the other hand, python / ruby / perl all have a far better story to tell. And these cloud services have too, which is why people sometimes reach for them without a clear need.
I’m not sure what learning shell would look like, besides getting a book (online or offline). There’s a lot of love for books here on lobsters and elsewhere, but what I actually see people do around me is be very picky about when and which books they actually buy. The shell is probably not an attractive enough subject.
P.S.: People often complain that learning the web these days is actually learning HTML + JS + CSS + a wealth of tooling. Learning Unix is not that different: Bash, regular expressions, sed, awk, and maybe even Perl. Plus, these tools are unattractive because they have so much history, baggage, arcane syntax, and often differ between Unices and even Linux distributions. Arguably, JS these days is far more beautiful than any of the Unix tooling, and JS is at the lower end of beauty to start with.)
The nice part about a Unix environment is that you can start with a simple shell for loop, and then more or less seamlessly add stuff like awk. Perl or Python when you reach the limits of shell (which happens pretty quickly in most of my cases).
The most complicated shell script I’ve written in recent memory is one which loops over zip archives, creates a subdir for each, unzip them into the directory, runs unrar -x on the contents, cleans up, and moves on.[1]
This stuff is perfect for shell - it automates a bunch of repetitive work that would take ages of clicking in a GUI, and it was straightforward to test.
If I’d had to deal with the contents of the archives (edit them in place, extract data, performed calculations etc) I’d have reached for Perl first.
[1], yes the zip files off the back of an internet truck.
I think it’s underestimated how easy Ruby and Perl make this. Both have backtick syntax to execute a shell command and capture its output.
But I agree, and personally reach for shell scripts as well. Also because I don’t really know Perl well, and the availability and compatibility of Bash seems more reliable than Ruby on various systems. (Though I’ve also been bitten by differing Bash installs.)
I also feel like maybe we’re a minority. I think my colleagues would sooner look for some way to do it in Node.js, PHP, or some task runner.
I spent a month of free time learning about the posix shell. A year later, I only remember a small part of what I learned. That small part is what I actually have cause to use. I find this to be true of all kinds of knowledge: I retain most what is most practically useful. The key part is recognizing when I have a need that exceeds my knowledge, when I should go learn more. But simply sitting down to study shell programming just because this might be useful at some point wasn’t a good use of my time.
Note that you don’t have to study things in depth, that’s time-consuming. It is relatively cheap to survey a field at a high level and know what’s there, so when you do need it you can reach right for it.
Lazy evaluation: not just for programming languages.
This applies to more than programmers. There are a lot of instances where I have been out in the field and there has been a litany of proposals for setting up overwrought systems and applications in order to do something as trivial as merging a few csv files and outputting a few trend charts. You have a room full of people on Linux and MacOS laptops and none of them have ever used the terminal, let alone understand that there are tools sitting right there on their laptop that with a bit of effort and time they wouldn’t have to install all this other stuff. Windows users are even more far removed as the Windows operating for so long hasn’t evangelized the powershell and cmd.exe or anything remotely command-like except when you need to do something complicated.
As far as I can tell from that side (and probably from the developer-side as well to some extent) it’s “the fear”. A button in even the most complicated of UIs is a lot less scary than typing seemingly cryptic incantations into a bash shell when you don’t understand the context that you’re operating in when the terminal pops open. Even if that buttons operation is failure-prone and buggy, it’s a confidence boost that “this button is here to do this thing that I want to do and that alone”. To someone whose not used to it (even developers), something as simple as ls -la or cd ../ looks like some kind of low-level black magick nonsense that only Neo and Trinity can understand.
For instance, we have a generation of web developers who have come up through things like dreamweaver and other GUI-oriented workflows, over the past 5-6 years they have begun to understand that *nix systems are more than just an apache/nginx server and a place to put index.html and a mysql instance. A lot of graphic designers made that transition, from designing web sites to actually implementing them, from a heavily GUI-based workflow they expected to move to just a different set of GUIs. So people built GUIs to do those things, like a GUI to install/manage a LAMP stack on macos, or manage a MySQL server. Or for instance macos homebrew, a pretty simple command-line utility to manage packages on macos, something so simple didn’t need a GUI, but someone built one, and people use it. Because if something has a GUI it MUST be stable, it MUST be safe, a GUI is sort of seen as a badge imbuing some stability to a system.
So I think the more important point is that the basics aren’t being evangelized at all, and we’re not building the confidence in people that they need in order to be comfortable in a shell. Forget complex awk and sed workflows. All the operating systems contain tools and libraries that can do amazing things if you understand them and the context they operate in, but if they don’t understand what they’re looking at when they use cd and that their computer isn’t going to light on fire from running it, then we can’t expect people to start writing awk scripts.
My generation (and probably a lot of us here on lobste.rs) grew up on terminal-based computers and eventually MS-DOS in school computers. For me Windows was my world for years because of the jump from MS-DOS to a full on desktop, but I had those years using a terminal, even on Windows with BBS’s and Telnet. I moved specifically to linux then to macos later in life to have access to unix-based tools without having to deal with linux config stuff (nothing against linux, i’m just slightly lazy). We forget that everyone coming after this breakpoint in history may have NEVER had to deal with a terminal or shell of any kind except for opening one by mistake.
Just look at the running joke about exiting vim, even here on lobste.rs we’ve made a joke about it in the form of a plugin that auto-quits vim. It’s funny, but it’s also a symptom and a canary down the mine that something is getting lost that people are that afraid of things in the shell.
People think macos, Windows and Ubuntu ARE the computer, that the Windows UI IS the system, they don’t realize that Apple Finder is just an app and not the filesystem itself. Even if they’re software developers. The OSes are built to make you feel this way. Not specifically to hide the shell, but as a natural byproduct of designing UIs the way that we do, we’ve hidden away the powerful tools and systems that underpin them. You see this with git GUIs, so many developers use them and have no idea about the commands that actually do the work behind the scenes.
Years ago, if you astalavista’d or yahoo’d for “How do I get my files back from a broken drive” for instance, you would end up with loads of articles explaining different commands and workflows to try to pull your data off an old drive. A GUI app may have represented some small part of that process still but it wasn’t the whole, now if you google similar questions the answers are a litany of free, paid and malware/adware-riddled desktop gui applications that do the same thing but show a progress bar. This makes sense for people who aren’t technical, but the offshoot has been that the communal knowledge in these older articles, which would have triggered more learning through the commands peppered throughout, has been lost. It’s not that the whole process has been replaced by a GUI or higher-level abstractions, it’s that the prompts and triggers that made us dig in to learn more beyond the commands to fix the drive have been completely lost.
What we need is for kids, teenagers and young adults who now spend their time on Twitter, Reddit, YouTube and other platforms as they’re whole internet, to have more ready access to these learning prompts to drive them to understand in a way that builds their confidence in these lower-level tools, concepts and contexts, we need cultural prompts, things like War Games, Hackers, The Matrix and all that modernized, coupled with a safe environment to try these things without repercussions or reprimand, we need people to understand that their laptop isn’t a bomb that could go off if they enter the wrong commands.
Sorry, long spiehl over, but I’ve seen this so much across the spectrum of system users and proficiencies. That went deeper than I intended but I think it’s not an issue of for instance, lazy programmers or anything, I think that as a community (developers not lobste.rs) we’ve just failed at passing down the knowledge we have and imbuing the confidence in our successors that we should, partly because we don’t even have access to those successors (they have to find us through stack overflow, lobste.rs, IRC, etc…. there’s no concept of apprenticeship and mentoring is rare, googles algorithms are skewed), and because the internet has evolved and changed and we just aren’t the ones standing on the soapbox with the megaphone anymore and we haven’t been for a long time, so instead of finding ways to reach people and disseminate our knowledge, we write short snipey articles subtly labelling people as stupid for choosing the tooling they do.
I don’t get it. These blog posts appear periodically, but what kind of data do authors process with shell pipelines? You can’t even reliably read and write CSV with sed/awk. You can’t connect to Postgres and read data from it. You can’t use json and xml at all. I can take any programming language which has map/filter/reduce and do the same in it. What are advantages of hacky string processing with weird operations and weird corner cases? Why should I write shell scripts and fear that space or any other special character gets unescaped, if I can use ruby/python/etc with real lists and real strings, not lists represented as strings separated by spaces?
“Available on any Unix system”? I don’t run data processing on random hacked Vac OS/9000/UX machines. And if I did, it would have General Electric awk, not GNU awk installed, which I’m accustomed to use.
Maybe I’ve missed something here, but you certainly can process JSON (with jq) and connect to Postgres to read/write data from/to it in the shell. The psql client will happily take stuff from STDIN, or slurp from a file.
The obvious take away is that developers shouldn’t be afraid of the shell. Certainly true, and there’s no shortage of these types of articles.
Another less obvious and perhaps more pertinent idea here is that humans have zero intuition for numbers. Is five million a big number? It seems big, so one might intuit that something more “sophisticated” would be necessary, and then you might reach for that first.
Generally, I’d advise approaching every problem with the dumbest solution possible first.
A lot of programmers haven’t ever really learnt about the shell, which is a shame. They’ll use it for git commands and make -k but won’t know how to use POSIX utils to optimise their workflow. I wonder if there’s a way to encourage people to learn about shell?
Every Friday from 4pm to 5pm, our engineering team at work has an (optional) activity: one person presents on a topic of their choice, while the rest drink a beer and wind down for the weekend. I personally gave presentations on using awk, jq, and git. In the Unix realm, we’ve also had presentations on the basic Unix stdin/stdout model, on the pitfalls of bash, on perf, etc.
If you have a venue like this, prepare a presentation on one or two Unix tools (don’t over-do it and make sure people can leave with actionable knowledge). If you don’t have a venue like this, see if you can’t make one happen. Sometimes, people just don’t know what they don’t know; they need someone to step up and say “here, let me teach you something that is simple, useful, and fun!”
That sounds like a fun place to work :)
I think the article is a bit black and white. Between the shell and the cloud, writing a small python / ruby / perl script was also a valid solution. AWK is a fairly large and powerful programming language in itself, and heck, Bash is too. So the takeaway is more: try to create a simple local solution first, with whatever you know?
I write bash scripts from time to time, and consider myself proficient, though I rarely touch AWK. I have colleagues who don’t write bash scripts at all outside of sequencing some simple commands, and wouldn’t even touch
sed, but I still consider them proficient (in the way they handle interactive mode).How I learned shell, and how I see other people around me learn shell, is pretty much through experience, googling, and finding tricks in articles somewhere, so the discoverability seems pretty awful. I also rarely see colleagues reach for manpages, and
infomust be the place where documentation goes to die or something.On the other hand, python / ruby / perl all have a far better story to tell. And these cloud services have too, which is why people sometimes reach for them without a clear need.
I’m not sure what learning shell would look like, besides getting a book (online or offline). There’s a lot of love for books here on lobsters and elsewhere, but what I actually see people do around me is be very picky about when and which books they actually buy. The shell is probably not an attractive enough subject.
P.S.: People often complain that learning the web these days is actually learning HTML + JS + CSS + a wealth of tooling. Learning Unix is not that different: Bash, regular expressions, sed, awk, and maybe even Perl. Plus, these tools are unattractive because they have so much history, baggage, arcane syntax, and often differ between Unices and even Linux distributions. Arguably, JS these days is far more beautiful than any of the Unix tooling, and JS is at the lower end of beauty to start with.)
What is nice about awk over she’ll scripts is that you get automatic tokenisation in a much more predictable way for free. And built in regexes.
So any time I also need to “parse” something, I reach for awk before a shell script.
The nice part about a Unix environment is that you can start with a simple shell
forloop, and then more or less seamlessly add stuff like awk. Perl or Python when you reach the limits of shell (which happens pretty quickly in most of my cases).The most complicated shell script I’ve written in recent memory is one which loops over zip archives, creates a subdir for each, unzip them into the directory, runs
unrar -xon the contents, cleans up, and moves on.[1]This stuff is perfect for shell - it automates a bunch of repetitive work that would take ages of clicking in a GUI, and it was straightforward to test.
If I’d had to deal with the contents of the archives (edit them in place, extract data, performed calculations etc) I’d have reached for Perl first.
[1], yes the zip files off the back of an internet truck.
I think it’s underestimated how easy Ruby and Perl make this. Both have backtick syntax to execute a shell command and capture its output.
But I agree, and personally reach for shell scripts as well. Also because I don’t really know Perl well, and the availability and compatibility of Bash seems more reliable than Ruby on various systems. (Though I’ve also been bitten by differing Bash installs.)
I also feel like maybe we’re a minority. I think my colleagues would sooner look for some way to do it in Node.js, PHP, or some task runner.
I spent a month of free time learning about the posix shell. A year later, I only remember a small part of what I learned. That small part is what I actually have cause to use. I find this to be true of all kinds of knowledge: I retain most what is most practically useful. The key part is recognizing when I have a need that exceeds my knowledge, when I should go learn more. But simply sitting down to study shell programming just because this might be useful at some point wasn’t a good use of my time.
Note that you don’t have to study things in depth, that’s time-consuming. It is relatively cheap to survey a field at a high level and know what’s there, so when you do need it you can reach right for it.
Lazy evaluation: not just for programming languages.
I highly recommend installing TLDR, a man-pages complement that makes it easy to learn the basic usage of common commands.
It reminds me of the Doug McIlory pipe vs the program Knuth wrote.
This applies to more than programmers. There are a lot of instances where I have been out in the field and there has been a litany of proposals for setting up overwrought systems and applications in order to do something as trivial as merging a few csv files and outputting a few trend charts. You have a room full of people on Linux and MacOS laptops and none of them have ever used the terminal, let alone understand that there are tools sitting right there on their laptop that with a bit of effort and time they wouldn’t have to install all this other stuff. Windows users are even more far removed as the Windows operating for so long hasn’t evangelized the powershell and cmd.exe or anything remotely command-like except when you need to do something complicated.
As far as I can tell from that side (and probably from the developer-side as well to some extent) it’s “the fear”. A button in even the most complicated of UIs is a lot less scary than typing seemingly cryptic incantations into a bash shell when you don’t understand the context that you’re operating in when the terminal pops open. Even if that buttons operation is failure-prone and buggy, it’s a confidence boost that “this button is here to do this thing that I want to do and that alone”. To someone whose not used to it (even developers), something as simple as
ls -laorcd ../looks like some kind of low-level black magick nonsense that only Neo and Trinity can understand.For instance, we have a generation of web developers who have come up through things like dreamweaver and other GUI-oriented workflows, over the past 5-6 years they have begun to understand that *nix systems are more than just an apache/nginx server and a place to put index.html and a mysql instance. A lot of graphic designers made that transition, from designing web sites to actually implementing them, from a heavily GUI-based workflow they expected to move to just a different set of GUIs. So people built GUIs to do those things, like a GUI to install/manage a LAMP stack on macos, or manage a MySQL server. Or for instance macos homebrew, a pretty simple command-line utility to manage packages on macos, something so simple didn’t need a GUI, but someone built one, and people use it. Because if something has a GUI it MUST be stable, it MUST be safe, a GUI is sort of seen as a badge imbuing some stability to a system.
So I think the more important point is that the basics aren’t being evangelized at all, and we’re not building the confidence in people that they need in order to be comfortable in a shell. Forget complex
awkandsedworkflows. All the operating systems contain tools and libraries that can do amazing things if you understand them and the context they operate in, but if they don’t understand what they’re looking at when they usecdand that their computer isn’t going to light on fire from running it, then we can’t expect people to start writingawkscripts.My generation (and probably a lot of us here on lobste.rs) grew up on terminal-based computers and eventually MS-DOS in school computers. For me Windows was my world for years because of the jump from MS-DOS to a full on desktop, but I had those years using a terminal, even on Windows with BBS’s and Telnet. I moved specifically to linux then to macos later in life to have access to unix-based tools without having to deal with linux config stuff (nothing against linux, i’m just slightly lazy). We forget that everyone coming after this breakpoint in history may have NEVER had to deal with a terminal or shell of any kind except for opening one by mistake.
Just look at the running joke about exiting vim, even here on lobste.rs we’ve made a joke about it in the form of a plugin that auto-quits vim. It’s funny, but it’s also a symptom and a canary down the mine that something is getting lost that people are that afraid of things in the shell.
People think macos, Windows and Ubuntu ARE the computer, that the Windows UI IS the system, they don’t realize that Apple Finder is just an app and not the filesystem itself. Even if they’re software developers. The OSes are built to make you feel this way. Not specifically to hide the shell, but as a natural byproduct of designing UIs the way that we do, we’ve hidden away the powerful tools and systems that underpin them. You see this with git GUIs, so many developers use them and have no idea about the commands that actually do the work behind the scenes.
Years ago, if you astalavista’d or yahoo’d for “How do I get my files back from a broken drive” for instance, you would end up with loads of articles explaining different commands and workflows to try to pull your data off an old drive. A GUI app may have represented some small part of that process still but it wasn’t the whole, now if you google similar questions the answers are a litany of free, paid and malware/adware-riddled desktop gui applications that do the same thing but show a progress bar. This makes sense for people who aren’t technical, but the offshoot has been that the communal knowledge in these older articles, which would have triggered more learning through the commands peppered throughout, has been lost. It’s not that the whole process has been replaced by a GUI or higher-level abstractions, it’s that the prompts and triggers that made us dig in to learn more beyond the commands to fix the drive have been completely lost.
What we need is for kids, teenagers and young adults who now spend their time on Twitter, Reddit, YouTube and other platforms as they’re whole internet, to have more ready access to these learning prompts to drive them to understand in a way that builds their confidence in these lower-level tools, concepts and contexts, we need cultural prompts, things like War Games, Hackers, The Matrix and all that modernized, coupled with a safe environment to try these things without repercussions or reprimand, we need people to understand that their laptop isn’t a bomb that could go off if they enter the wrong commands.
Sorry, long spiehl over, but I’ve seen this so much across the spectrum of system users and proficiencies. That went deeper than I intended but I think it’s not an issue of for instance, lazy programmers or anything, I think that as a community (developers not lobste.rs) we’ve just failed at passing down the knowledge we have and imbuing the confidence in our successors that we should, partly because we don’t even have access to those successors (they have to find us through stack overflow, lobste.rs, IRC, etc…. there’s no concept of apprenticeship and mentoring is rare, googles algorithms are skewed), and because the internet has evolved and changed and we just aren’t the ones standing on the soapbox with the megaphone anymore and we haven’t been for a long time, so instead of finding ways to reach people and disseminate our knowledge, we write short snipey articles subtly labelling people as stupid for choosing the tooling they do.
edited: fixed some typos and clarified a bit
I don’t get it. These blog posts appear periodically, but what kind of data do authors process with shell pipelines? You can’t even reliably read and write CSV with sed/awk. You can’t connect to Postgres and read data from it. You can’t use json and xml at all. I can take any programming language which has map/filter/reduce and do the same in it. What are advantages of hacky string processing with weird operations and weird corner cases? Why should I write shell scripts and fear that space or any other special character gets unescaped, if I can use ruby/python/etc with real lists and real strings, not lists represented as strings separated by spaces?
“Available on any Unix system”? I don’t run data processing on random hacked Vac OS/9000/UX machines. And if I did, it would have General Electric awk, not GNU awk installed, which I’m accustomed to use.
Maybe I’ve missed something here, but you certainly can process JSON (with jq) and connect to Postgres to read/write data from/to it in the shell. The psql client will happily take stuff from STDIN, or slurp from a file.
The obvious take away is that developers shouldn’t be afraid of the shell. Certainly true, and there’s no shortage of these types of articles.
Another less obvious and perhaps more pertinent idea here is that humans have zero intuition for numbers. Is five million a big number? It seems big, so one might intuit that something more “sophisticated” would be necessary, and then you might reach for that first.
Generally, I’d advise approaching every problem with the dumbest solution possible first.