Hey! I want to learn more about how Linux works, but I feel that most of the books go really deep. I usually prefer a different approach when I learn something new: understanding the big picture and how everything works together first, and then go deeper and deeper.
Which resources would you recommend to learn more about Linux?
Writing a simple shell is a hard way, but touches a lot of Unix concepts (syscalls, libc, forking/zombies, signals, pipes, file descriptors, ttys/sessions, etc) in a broad but hands-on way.
That’s actually a good idea. I did a really simple shell many years ago. I’m pretty sure I can do something more complex now.
This is a very good idea. We did this for our operating systems class at uni, and I thought it was a lot of fun. We also wrote a very simple file system implementation which had a C API much like POSIX (read, write, open, close, seek, iirc).
Nowadays you could do the same in userland, and then maybe write a FUSE implementation around it and then maybe convert it into a kernel module later.
I’m assuming you’re asking about using linux as an operating system and not understand the intricacies of the kernel.
This may sound like an odd recommendation, but the simple act of compiling the tools you use from source is surprisingly effective at exposing you to large swathes of how linux systems work. I’ll leave the explanation of why that’s true as an exercise for the reader :)
It’s more about the intrincates of the kernel (and some user space libraries to debug and trace the system)
Don’t know if this has anything you’re looking for, but there’s a Humble Bundle with Linux books.
https://www.humblebundle.com/books/linux-bsd-bookshelf-2019-books
My learning path:
+1 for a Cheap VPS. I’ve learned Tmux, server admin, how to set up weechat, I’ve used it for an ssh tunnel to both RPD and/or ssh into other machines though it, and I’ve learned a fair amount of bash. I’ve also deployed something like 6 or so websites/apps to said VPS for my personal use.
get a cheap VPS and use it for everything you can. Website, SSH, chat, programming, writing, whatever. This makes you know how to do everything through the console.
Not if you use Emacs + Tramp ;).
use arch Linux, ‘cause it’s less cleanly packaged and more fast moving, and this encourages you to tinker more… Because you have to, to keep it working.
I think that this is an excellent recommendation. Additionally, (even though it seems contradictory to the title) you could try to work with a BSD for a while. They generally provide less hand-holding, have better man pages, and less beaten paths (searching the web is less likely to find you a solution to a problem).
Familiarize yourself with composition: how Linux looks like in a disk image, a chroot, a container, etc. Things like Packer, mkosi, casync, welder, debootstrap, etc. This will lead into package management and servicing (updates, etc.)
Then systemd. From Mattias [1], Enrico [2] or Lennart’s blog. You might want to follow the systemd releases page. You can use Debian or Fedora, but even if you use Debian I suggest you track the Fedora changelog.
A good organized collection of readings is [3] and focused in internals.
People are already giving you great eBPF resources. New titles are coming out. I would suggest you experiment with Sysdig’s Falco.
I’ve also learned a thing or three from Julia Evans’ zines and blog posts [4] and I bought her prints. And in terms of actual books, consider Kerrisk’s “The Linux Programming Interface” and the latest Nemeth et al. “UNIX and Linux System Administration Handbook”
I hope this helps. I’ve been using Linux for over 15 years, creating distros, packaging, operating large fleets of it, using it as a desktop and more. I’m surprised how much of it evolves and reinvents itself. It keeps me an eternal learner and a perennial novice!
[1] https://ma.ttias.be/learning-systemd/ [2] https://www.enricozini.org/blog/2017/debian/systemd-01-intro/ [3] https://0xax.gitbooks.io/linux-insides/ [4] https://jvns.ca/
wow, thank you so much for sharing this!
Well, I’m biased on this, but I worked on How Linux Works and I’m rather proud of the resulting book.
strace+mancan yield very good building blocks. I am mostly curious about one nitpicky thing so this might be the fastest route. After googling the hell out of it.Documentation/index.rstin the kernel sourcetree.Start with http://www.man7.org/tlpi/index.html
Regarding the performance analysis you mentioned in a comment, read http://www.brendangregg.com/bpf-performance-tools-book.html
There is also a new version of Gregg’s Systems Performance book coming out soon.
I’m looking forward to reading the BPF book, but I’ve not seen any announcements about the new edition of Systems Performance other than the statement at the end of http://www.brendangregg.com/sysperfbook.html. Do you have any more information?
He mentioned at the end of his LISA 2019 presentation to keep an eye out, but I don’t have any information beyond that.
This is good for XDP: https://github.com/xdp-project/xdp-tutorial
For clarification: Linux the kernel or Linux as an operating system (Linux kernel + userland)?
Kernel + userland. I want to learn more about the kernel, but also about troubleshooting Linux problems, and how to use tools like ftrace, strace, eBPF, perf, etc.
I also think there is a dimension of learning “modern Linux”. I know a bunch about
”*nix”, but it’s not easy to update that to”Linux, 2019”, and even more difficult to update that to”Fedora/Ubuntu, 2019”As controversial as systemd is, I think it’s a good starting point for learning “modern Linux”.
I read O’Reilly’s Understanding The Linux Kernel (2nd ed) and it was really informative, especially read alongside the Tanenbaum Operating Systems textbook. I mostly came to understand the userland through using it over the course of 10ish years. Distributions wikis help a lot, I’m fond of Arch’s and Gentoo’s.
I tried to use the 3rd edition of that book when trying the Eudyptula challenge (when that was still a thing..), and found that the kernel driver interface for, e.g., USB had changed so much from 2.6 (covered in the book) and what I was using at the time (IIRC, one of the later 3.x kernels) that the book was pretty much useless for using as a reference to build a driver for that subsystem. It’s probably so far outdated now that it could only be useful as a high level overview of what the kernel does (where the outdated piece would be the implementation covered in the book..).
Are they planning to release a new edition?
Do you use it daily?
It really depends on where you are and where you want to go.
Have you started up Linux in a VM and tried to use it for email and lobster.rs? I find that you can slowly find the edges of the world as you try to do more and more. You could say “I know how to do x in a gui on Linux, I’ll search how to do the same thing in the terminal”. You also don’t hit the very hard edges of hardware support and failure in this mode.
You could then increase the difficulty by choosing a more from scratch distribution or trying it on physical hardware. I learned so much when trying to get a random thing working or from trying to fix something. I also experienced how Linux is very much like Legos, in that you can switch out many things and still make it work.
Try something like Arch, Gentoo, or Linux from scratch at some point, trying to replicate your current working system in a new VM. One advantage of this is you see the trade offs that many system integrators go through. “Ok, it works with xorg, let me try the same thing with Wayland.” Or “okay, it works with Arch, let me try the same thing with Debian”. There are so many choices to make with Linux that it’s good to understand some of the defaults that are common across distributions. Filesystems, display systems, process managers, container managers, mail servers, web servers, etc.
If you have more specifics about where you are in your journey, maybe one of us could give deeper insight into what we ran into and how we got past our problems.