1. 86
  1.  

  2. 24
    sudo bash -c 'echo 1 > /proc/sys/vm/drop_caches'
    

    The cache hierarchy is so damned steep these days, unless you’re very careful, all you’re measuring when you benchmark is whether your caches are hot.

    So if you’re benchmarking, make sure you system is quiet (exit your windowing environment, browsers, memory hogs like eclipse etc. etc.)

    Flush your virtual memory cache’s with command above, and run your benchmark (at least) 4 times. The first measurements tells you how long it takes to scrape your data and programs off spinning rust… the average of the next three tells you how fast your program is.

    1. 15

      Note, this does not flush CPU caches. It drops filesystem caches only, and echoing 1 might not drop everything you expect .See proc(5).

      1. 9

        One thing I’ve heard, which seems to make a lot of sense, is that your want to run the benchmark a bunch of times and then look at the times for the quickest runs. After all, any random event like a network interrupt or disk read interrupt will slow down your program, not speed it up, so the quickest run will be the run affected by the least random external factors.

        1. 5

          I’ve had a very reliable microbenchmarking experience using a statistically inclined benchmarking suite called criterion, some have even gone as far as to make it an executable in lieu of time because it offers so much more bang-for-buck than more pedestrian choices.

          The amount of variance in your performance is an informative data point in itself, as merely taking the fastest results may not indicate typical performance or even an accurate lower bound, misleading the developer. You thought you made a function faster in your second run, but the lower bound has noise confounding it, for example. I’ve certainly fell victim to this approach. Statistical methods iron out these wobbly results.

          1. 1

            Thank you! We need to be much more informed of the statistics required to handle performance measurements. Unfortunately, what’s really common even in “statistically inclined” benchmarking tools is to present the sample mean and standard deviation. This implicitly assumes the samples are drawn from a normal distribution, which is rarely the case when it comes to performance measurements (which tend to be heavy-tailed). Do you happen to know if Criterion makes this mistake as well?

            Edit: Based on the tutorial you referenced, it looks like Criterion sort of tries to dodge the issue by pretending that outliers are noise. Not ideal, given that outliers are the best signal we can get from a small sample out of a heavy tailed distribution.

            1. 2

              In the absence of more time today (I intended to write something more substantial) I can share the author’s announcement post in which he states:

              The performance of real-world applications doesn’t follow some kind of tidy statistical pattern such as the normal distribution, especially when people are measuring in realistic, noisy environments.

              I think that outliers are not all considered noise, but rather outliers are checked to see whether they affect the variance in a significant way, and in this case are flagged up to inform the developer that confidence is low.

              Does that add anything to the discussion?

          2. 2

            I suppose if you do this there’s not really a need to flush caches first, then.

            1. 4

              Does add a touch more variability, as the kernel tries hard not to go to disk. ie. If something else has a dirty page, it will try not to write that out, leaving less ram for your app.

            2. 2

              It really depends on what you’re doing. When I’m trying to figure out what about my algorithms or data structures need improvements (e.g. in a long-running server app), yes, I absolutely want to do multiple runs and generally analyze the fastest time so that I’m looking at what about my code and my algorithms is slow, rather than what about the testing environment is slow. But for e.g. tracking a consumer app’s startup behavior, or (historically) figuring out how to lay out data on a spinning platter for maximum performance, it’s entirely reasonable to flush caches. Like all things in profiling, it’s not one-size-fits-all.

            3. 4

              Side tip, when writing to kernel files, I like to use tee instead of something like sudo bash -c:

              echo 1 | sudo tee /proc/sys/vm/drop_caches
              
              1. 1

                Apart from being able to hit several kernel files with one command… what is the advantage?

                1. 3

                  Less complexity, easier to type, fewer things running as root.

                  1. 2

                    Less complexity … fewer things running as root.

                    My brain went, whirr whirr click, “He is Right! (it probably runs everything in .bashrc)”, then whirr whirr, “but is he?”

                    So using my favouritest tool, strace I proved to myself that, no, not really, you’re wrong about number of things running as root (answer 1 in both cases). Slightly right about less complexity (bash invokes more operating services, mostly about signal handling but tee opened 1 more file).

                    If you want less complexity, replace bash by dash.

                    But hey, you’re right about the easier to type.

                    1. 1

                      Yeah, true that these are all built-in stuff in bash/dash. But tee is even less complex than dash.

                      1. 3

                        Certainly you’re right by size of each beast…

                        du -h `which bash` `which tee` `which dash`
                        1.1M	/bin/bash
                        36K	/usr/bin/tee
                        120K	/bin/dash
                        

                        ..but curiously the runtime behaviour of dash was simpler…. 139 OS calls by bash 59 OS calls by tee 47 OS calls by dash

                        Ah well, all a very minor point, and in this case, I think your “it’s easier to type” is the best reason.

                        1. 1

                          I’m not too surprised about that; GNU coreutils tend to be “overengineered” (well, more like correct and efficient in many corner cases) compared to other software. On my FreeBSD system, /usr/bin/tee is reported as 12 k, but I don’t know how to dtrace it to get a number of OS calls comparable to yours.

                  2. 1

                    You don’t have to think about quotes (primarily useful when doing something more complex than echo 1).

              2. 15

                A lot happens in /proc of course

                Valuable info includes:

                • /proc/$pid/exe -> always gives you the executable, even if it is inside a docker container
                • xargs -0 < /proc/$pid/env
                • xargs -0 < /proc/$pid/cmdline
                1. 3

                  procfs is one of my favourite things in Linux! I miss it so much when I have to use macOS.

                  1. 6

                    My favouritest thing is strace….

                    …which tells me a lot of things under the hood go frootling about in /proc….

                    1. 3

                      For example…. Consider the very very useful lsof (list open files)

                      strace -e openat lsof

                      or even
                      

                      strace -e openat ps

                  2. 2

                    Also /proc/$pid/wchan to see quickly which syscall your code is up to

                  3. 14

                    A personal favourite is that Linux to this day maintains in-kernel support for AX.25, a completely different mode of networking from TCP/IP, primarily for amateur radio. Once you create an interface with a KISS-enabled TNC (modem) and external radio you can write ordinary C networking software (disclosure: my blog) - except with AF_AX25 and SOCK_SEQPACKET to establish connections.

                    You know it’s a real socket too because you can look at it with netstat!

                    $ netstat -A ax25
                    Active AX.25 sockets
                    Dest       Source     Device  State        Vr/Vs    Send-Q  Recv-Q
                    *          VK7NTK-1   ax0     LISTENING    000/000  0       0     
                    

                    Or if you like, you can assign an IP address to the interface and Linux will do TCP/IP over connectionless AX.25 frames. (Note: AX.25 is hardly unique in being an old technology that still exists in Linux, but it’s fun that a handful of people still use it.)

                    1. 3

                      A similar situation is the CAN bus - https://www.elinux.org/CAN_Bus. This is used for a bunch of things, but primarily shows in cars.

                    2. 13

                      flock(1) is a command-line utility that implements lockfiles with the flock system call. It’s easy to use in shell scripts and the locking is reliable.

                      1. 1

                        Too cool! I wish I knew about that earlier.

                      2. 11

                        script(1) is a really interesting command too.

                        1. 4
                          1. 3

                            I had soooo much trouble installing that package.

                            My brain was convinced it was “ascii enema”.

                            1. 2

                              Also tee(1)

                              1. 1

                                Caution: Don’t play with this doing anything / listing anything that is private….

                                By default it squirts your terminal session into the cloud… (which, in theory, is private until you sign up and explicitly make it private)

                              2. 2

                                Use with the -t timings-file argument, then replay with scriptreplay.

                                This can be used to drive, oh, say, as a random example, phosphor(6x), to replay console sessions as a screensaver.

                                This can be actually useful for system set-up / configuration. If you’re accessing a new box over a serial interface (serial-over-IP, IPMI, ILOM, etc.), you can record the entire sequence including boot, BIOS, bootloader, and kernel boot of a new system install. You can also choose the playback speed to accelerate the replayed session.

                              3. 8

                                ~/.inputrc:

                                "\33\33[A": "cd ..\n"
                                

                                If you press Alt + up arrow, it goes to the parent directory

                                1. 8

                                  I tried to document some things I learned about filesystems on Linux and macOS in a document: fsdoc.pdf. It’s not complete and a bit shitty, but if you have anything more that is interesting, do leave me a pull request :)

                                  1. 1

                                    Cool write-up! While scrolling, I read:

                                    For this reason, adoption of acl is not widespread.

                                    I don’t think that’s true, SELinux and AppArmor have this and are default enabled on a number of distributions.

                                  2. 4

                                    Brilliant! Thanks for posting about /dev/full, I had no idea it existed.

                                    Sadly it’s one of those features that I’ll probably forget about next time it could be super helpful for testing.

                                    I think my favourite little-known feature is probably shopt -s extglob from bash:

                                           If the extglob shell option is enabled using the shopt builtin, several
                                           extended  pattern  matching operators are recognized.  In the following
                                           description, a pattern-list is a list of one or more patterns separated
                                           by a |.  Composite patterns may be formed using one or more of the fol‐
                                           lowing sub-patterns:
                                    
                                                  ?(pattern-list)
                                                         Matches zero or one occurrence of the given patterns
                                                  *(pattern-list)
                                                         Matches zero or more occurrences of the given patterns
                                                  +(pattern-list)
                                                         Matches one or more occurrences of the given patterns
                                                  @(pattern-list)
                                                         Matches one of the given patterns
                                                  !(pattern-list)
                                                         Matches anything except one of the given patterns
                                    

                                    It’s handy for doing things like rm !(*.jpg|*.jpeg|*.gif) to remove all non-image files from a directory, for example.

                                    1. 4

                                      /dev/full is a new one to me!

                                      Several years ago I set up a twitter feed for *nix tips and will recalibrate my expectations for what is of interest based on this thread. Here is an RSS version.

                                      Specific to Linux: there’s a sysctl kernel.random.boot_id that is generated once per boot, so you don’t have to guess at whether the system has rebooted based on uptime or ‘who -b’. This is now how ansible detects a reboot on Linux.

                                      1. 3

                                        If you ever want to print a message to your terminal (output redirection be damned), write it to /dev/tty. I have this script on my server because of a few tools insisting on opening Firefox (or another web browser):

                                        #!/bin/sh
                                        # firefox
                                        
                                        echo $* > /dev/tty
                                        
                                        1. 4

                                          I don’t get this. How is that script preventing tools from opening firefox? And what horrible tools are we talking about here, anyway? :D

                                          1. 2

                                            Well, on a headless server without a GUI, opening firefox is kind of useless. The main tool I use that does this evilness is Go’s proof (mostly when opening SVG’s of things, which is understandable to be honest), but I’ve seen other tools do it too.

                                          2. 1

                                            Is this script called anything in particular?

                                            1. 2

                                              firefox!

                                              1. 1

                                                Ah, that makes a lot more sense. (I think both dacav and I missed that bit)

                                                1. 1

                                                  I edited the comment to make this more obvious.

                                          3. 4

                                            The full and empty devices, also zero. Relatively well-known. /dev/null, of course. /dev/random and /dev/urandom, and their (important!) differences.

                                            /bin/true and /bin/false

                                            In some builds, Bash’s built in virtual TCP and UDP devices. These are shell rather than filesystem entities. Debian and derived distros generally disable this in build.

                                            The /proc and /sys trees have simply tons of goodness. You can write your own ps clone from basic tools (shell, awk, etc.) by parsing entries, in a pinch (useful on embedded distros, many modems and routers as examples).

                                            Among the potentially more useful items:

                                            • /proc/self
                                            • /dev/stdin (/proc/self/fd/0
                                            • /dev/stdout (/proc/self/fd/1)
                                            • /dev/stderr (/proc/self/fd/2)

                                            (Also: /proc/$$, for your shell’s process.)

                                            /proc/net/ has useful entries (arp, dev, etc.)

                                            /proc/mounts is quite useful when attempting to sort out what filesystems are actually mounted, especially when root is mounted read-only. This means that /etc/mtab, if a static file, isn’t updated. (On more recent systems, that’s now a symlink to /proc/self/mounts.)

                                            There are several system profiling tools which are little more than wrappers around queries of /proc files.

                                            Too many other idiosyncracies to mention. /etc/nsswitch can be an interesting source of confusion in cases, just off the top of my head.

                                            1. 2

                                              /etc/nsswitch.conf is tricky. Especially if you’re on a system with say MUSL and you also run something that expects (via static link) /etc/nsswitch.conf to exist in order to DNS resolve. It happens sometimes to docker containers.

                                              1. 2

                                                Bash’s built in virtual TCP and UDP devices

                                                have played a crucial role in some of those rm -rf recovery meme posts

                                                1. 1

                                                  Nice.

                                                  I’ve managed recovery from not quite so hosed systems. Having, say, your initrd envrionment available (on Debian, includes the dash shell and busybox) can be quite useful. It’s often mounted, though I don’t know if that will survive an rm -rf. If mounted ro it should.

                                                  Figuring out how to get bits onto the box is the hard part. A serial or parallel console link is another option, assuming you can launch something that will accept bytes sent there. I’ve not gone quite that far, but did once bootstrap a potato Debian install (the last to have a bootable tarball image) by uuencoding and zmodeming packages across a serial link using minicom. First to get the parallel drivers on (56 kbps is a hell of a lot better than 9600 baud), and then PLIP (multiple connections via SSH beats a single 24x80 terminal). Educational if not always precisely useful.

                                              2. 2

                                                binfmt_misc, it’s what makes anything that’s not an executable interpreted by the kernel directly work.

                                                Check those out:

                                                1. 2

                                                  Not exactly kernel related, but this is what I like about Linux ecosystem.

                                                  while true; do if [[ `wmctrl -l | grep Lobsters` ]]; then spd-say "I see you're reading Lobsters. What are your thoughts on: `python3 -c 'import requests; from bs4 import BeautifulSoup; import random; print(random.choice(BeautifulSoup(requests.get("https://lobste.rs").text, "lxml").find_all("a", {"class": "u-url"})).text)'`"; spd-say "`zenity --text 'So what do you think?' --entry`? `echo -e 'Well, everyone has their opinion.\nYep, I agree with you on that one!\nCommon, are you serious?!' | shuf -n 1`"; sleep 60; fi; sleep 1; done

                                                  Pro-tip: Keep this running in a background.

                                                  1. 2

                                                    I think you mean “Come on”, not “Common”?

                                                    1. 1

                                                      You’re right, thanks. Sadly, I can no longer update that comment.

                                                  2. 1

                                                    If you want to write to for example stderr from laguage which doesn’t support stderr redirection (awk), you can instead write it to /dev/stderr and it will redirect to stderr automagically!

                                                    This also means you don’t have to remember the awful sh syntax for redirecting echo output stream to 2nd file descriptor, you can just do echo 'error' > /dev/stderr.

                                                    1. 1

                                                      check if remote port is open

                                                      timeout 1 bash -c 'cat < /dev/null > /dev/tcp/10.11.12.13/5432'; echo $?
                                                      
                                                      1. 1

                                                        How about /proc/pid/syscall?

                                                        This is great, because it exposes the process’ currently executed system call, argument registers, and stack pointer in a lightweight way without having to resort to PTRACE attach. At work we use this trick to monitor Ruby EventMachine workers to determine how much time they’re spending blocked on select(2) versus doing actual I/O. Pretty cool.