1. 15

  2. 10

    Oh, I opened this particular can of worms several years ago, when implementing VFS for rust-analyzer. That was bad. Ultimately, I decided to implement a good enough (that is, wrong) solution, as I wasn’t ready to pick that battle. What l’ve learned:

    • If you think about reliable file watching, the first stop is formulating the criterion of correctness. It turns out out to be a fairly complicated thing, something along the lines of “the input to the watcher is a predicate over file system paths. The watcher guarantees that, if the system is in quiescent state, you eventually learn about the contents of all the paths.
    • “walking” and “watching” are not separate operations. To have consistency, you need to fold both into a single atomic “I want to be kept up-to-date regarding the contents of these files”
    • key point: providing this API requires a statefull service. This should be a part of an OS, but it isn’t. Low-level OS APIs are inadequate for consistent file watching.
    • this explains design of watchman as a daemon. When I was starting to look into this, watchman initially seemed like an over-engineered solution: surely, you want just a library binding OS APIs, right? But the thing is, what you really want is a poly-fill for a missing chunk of an OS, and making it a stand-alone service just makes sense, design wise.
    • watchman gets consistency right. It’s very useful to peruse its docs if only to learn how to define consistency.
    • that beeing said, having a self-contained (Rust) library here would be nice!
    1. 3

      don’t forget that the events given by file systems vary between each OS, and some file systems may not report any information (like inotify in linux when it’s actually a WSL instance and you’re looking at windows-owned folders). Or that you get free recursive folder watching on windows, but will have to traverse the whole folder on linux, adding towards the limit for each file. And if you miss a newly created folder or file, you miss events for that. It’s a really bad state. And then comes linux and documents that it’s kind of impossible to guarantee that you actually get every event. Rename events are also tricky, if you can even call them that. And editors may behave differently in the way they “save” files.

      that beeing said, having a self-contained (Rust) library here would be nice!

      I’m open for contributions, I have too many bugs to slam and not enough time or hardware ;)

      1. 3

        And the biggest trap arising from that: don’t conflate files and paths. *NIX file systems have hard links and inotify will not flag a file as changed if you are watching a directory but a file is modified through a different hard link. On Windows, there are no hard links and every file has a canonical location so this isn’t a problem. Apple tries to provide an abstraction that overlaps both and so the filesystem events framework tends to do a better job.

      2. 2

        All very true. Just for a bit of context, originally Turbowatch was based on Watchman, but was later made to work with any backend due to limitations of Watchman (and other backends) that I discovered along the way. Turbowatch focus is how you process change events regardless of the backend that alerted about them.