It’s not about Linux per se, but it does relate to how operating systems work and a similar kernel: The Design and Implementation of the 4.4BSD Operating System. I bought it on the recommendation of John Carmack and the depth it goes into is great. Every chapter also has little quizzes without answers, so you can confirm to yourself you know how the described system components should work.

some modern linux exercises if you haven’t narrowed down your own yet, and want to just jump into kernel hacking:

• start with a small loadable kernel module that logs a message with pr_info that you can read from dmesg
• have your kernel module create a procfs entry that can be modified from userspace to get information into the module
• modify kernel/sched/core.c (formerly sched.c) to read a number from procfs and bias the scheduling algorithm to blacklist it from ever running. test by starting a userspace process that print in a loop, write its pid to procfs, see if it stops or not
• introduce a subtle bug in the scheduler (if you were fortunate enough not to have in the last step) and use qemu + gdb to peer into it at runtime
• implement a new syscall, call it from userspace
• intentionally introduce a memory corruption bug and use kasan to catch it

kernel dev is more of a human political problem than a technical one, and every kernel subsystem has its own way of doing things, but there’s a lot of overlap. whatever subproject you want to join or create, learn enough about its human operations so that you can be respectful. this will involve reading a bit of process documents, learning new tooling, and still probably leaving you with a sense of uncertainty about where to jump in. some projects are more newcomer friendly than others.

it’s easy to do cool hard stuff by yourself, but with kernels it’s much more of a coordination problem, so respect the people by doing your homework before spending their time, then you can have the confidence to ask them for advice knowing that you have done your homework (you can also ask people HOW to do your homework so you can prevent wasting more of their time if a subproject doesn’t make it easy to find) and you’ll find someone who will be willing to point you in the right direction

As @WilhelmVonWeiner mentioned, I would invest in a copy of the The Design and Implementation of the 4.4BSD Operating System or The Design and Implementation of the BSD Operating System.

And I’d suggest starting my writing your own simple operating system kernel. Personally, I moved from there to studying Minix (this was many years ago). Minix was (and I hear still is) great for learning from. It’s different than Linux in that it is a microkernel architecture, that said you will learn a lot from it because it’s easy to read through and understand. Between writing your own and Minix, you’ll be off to a good start.

I still find the source for the various BSDs easier to follow than Linux and would suggest making that your next big move; graduate to a BSD if you will and from there, you could leap to Linux.

One other thing to consider. Picking a less used kernel to start hacking on when you feel comfortable might be a good idea if: you can find folks in that community to mentor you. In the end, the code base matters less than having people who are grateful for your assistance and want to help you learn. In my experience, smaller communities are more likely be ones you can find mentors in. That said, your mileage my vary greatly.

There’s also this pretty thorough description of many of the kernel elements.

https://github.com/0xAX/linux-insides

I’m not a kernel dev, but I wrote a toy Linux kernel module and it definitely gave me a better understanding of how the kernel works. So if you haven’t done that I would recommend it!

For Unix in general, the code for xv6 is also easily buildable and runnable. I ran it in an emulator and did a little hacking on it:

https://pdos.csail.mit.edu/6.828/2012/xv6.html

I would try porting Brainfuck or tinypy to it inside the emulator and running Mandelbrot :) That is, it has a tiny shell and tiny user space. You should be able to compile a new program and boot the system with it.

For some background, I would also suggest The Design of the UNIX Operating System (1986), by Maurice Bach.

I am not a Linux kernel hacker, but here is the route I would take, in addition to all the other great advice in this thread.

• Read lwn/kernel mailing lists
• Have some changes or bugs in mind that you would like addressed
• Start writing linux kernel modules
• Figure out how to debug, build and test your kernels. Inject bugs into a test kernel and then figure out how to find it.
• Have a robust, stable, fast, repeatable dev environment. Don’t hack main kernel on your laptop, get skilled at QEMU and GDB
  • https://www.kernel.org/doc/html/v4.11/dev-tools/gdb-kernel-debugging.html
  • http://sysprogs.com/VBoxGDB/tutorial/
  • https://nickdesaulniers.github.io/blog/2018/10/24/booting-a-custom-linux-kernel-in-qemu-and-debugging-it-with-gdb/

Lastly, get good at statistics, charting and benchmarking. Lots of changes are very subtle, and do to solid engineering, testing and confirming the result is going to be the vast majority of the time spent. The nature of the domain isn’t one of hammering out 1000 lines in a weekend. And even if it was, it will still requires weeks of analysis to confirm the result.

I maintain a book on kernel modules, which can be a good place to start. The examples should all compile with the latest kernel. https://code.freedombone.net/bashrc/LKMPG

May want to start with a device driver to get your feet wet. Linux Device Drivers 3rd Edition is the gold standard for that.

HTML version, easy to navigate: http://www.makelinux.net/ldd3/ PDF: https://bootlin.com/doc/books/ldd3.pdf

The Linux kernel is massive, and learning how it all works is a pretty big undertaking. I learned far more by trying to add to the functionality than reading a book.

If you want to learn how operating systems work, I’d recommend starting with something super simple, like a unikernel, RTOS, or the like. For example there’s FreeRTOS. You can buy a compatible development board and a JTAG (hardware debugger) and step through the boot process to initialize hardware and watch it do things like interrupt handling and context switching. FreeRTOS typically runs on hardware with only a memory protection unit (IE no virtual memory) which is much more simple. Once you understand that I would move on to Linux or something else full featured.

As for Linux proper, is there something in particular you want to learn about? There are device drivers, architecture specific code, the network stack, the block layer, the scheduler, etc…

I took a class on device drivers and the book for that was very good: https://www.oreilly.com/openbook/linuxdrive3/book/ . It’s ancient but the concepts are all still generally applicable. I think it was easier to start trying to add a device driver than anything else.

The kernel has a built-in gdb stub called KGDB that typically works over a serial port, and it also has a simplified front-end called KDB. Stepping through code is sometimes more easy than just trying to read it. Some hardware specific IDEs are also capable of stepping through kernel source (TI’s code composer is the only one that I know about, and the hobbyist SBC of choice for them is the beagleboard https://beagleboard.org/static/beaglebone/a3/Docs/ccs-jtag-simple.htm.)

If at some point you just want to jump in and hack, check out https://kernelnewbies.org/KernelJanitors though I never did it myself.

There’s also an IRC room for kernelnewbies https://kernelnewbies.org/IRC

Try these challenges: http://eudyptula-challenge.org

Nvm, they are closed. Maybe there’s an alternative?

I went through the path of going over bugs and just taking a stab at them. I also had mentors that didn’t mind answering my questions.

kernel panic -> there’s a panic command generating that output so you can find the relevant code. Go look for it, go look around.

In one of my favorite bugs I went over the boot process from the start symbol and seeing where it goes and figuring out what it is doing. I strongly recommend trying that out one time, but I don’t know where it goes for linux :-)