ReadDir is optional so that ordinary files (full of bytes) don’t have to implement dummy ReadDir methods. It would be a bit odd to omit ReadDir on an actual directory, because then any traversal tools won’t work. For example the hypothetical key-value store without ReadDir will not work with fs.Walk (or fs.Glob).
Most code shouldn’t need to define things like readDirStatFS or even refer to the ReadDirFS and StatFS interfaces. It should instead use the helper functions fs.ReadDir and fs.Stat, which use those interfaces internally as appropriate. Code that instead requires, say, the readDirStatFS interface will not work with valid FS implementations that don’t provide a direct fs.Stat method (fs.Stat calls fs.Open+f.Stat+f.Close in that case).
fstest.TestFS does more than just assert that a few files exist. It walks the entire file tree in the file system you give it, checking that all the various methods it can find are well-behaved and diagnosing a bunch of common mistakes that file system implementers might make. For example it opens every file it can find and checks that Read+Seek and ReadAt give consistent results. And lots more. So if you write your own FS implementation, one good test you should write is a test that constructs an instance of the new FS and then passes it to fstest.TestFS for inspection.
One of the most common mistakes you might make in an FS implementation would be to mess up ReadDir on Open(”.”) somehow so that it returns no files at all. If you did this, then TestFS’s walk of the entire file system would find nothing to test, so nothing would fail any tests, and the overall TestFS call would incorrectly pass. The list of files passed to TestFS removes this failure mode by saying “look, when you do the walk looking for things to test, make sure you at least see this list of files, or else ReadDir is broken”.
Again, thanks for taking the time to write the post. It was fun to read.
I am looking forward to its use with the //go:embed directive for embedding static files in the compiled binary.
While the third-party solutions work well (e.g. go-bindata), I usually only need 1 or two files embedded, so having that supported out of the box is a welcome addition, especially when I can combine it with the io.FS interfaces so the files don’t need writing to disk for occasional operations which only know how to read from real disks.
A few comments as the author of a filesystem abstraction library for Go (github.com/twpayne/go-vfs):
io/fs will surely be useful in some circumstances, but its lack of support for writes is a major omission that limits its usefulness.
Afero’s MemMapFs attempts to simulate a filesystem in memory but the simulation is buggy. One bug I encountered is that file system access is not correct: if you make chmod 000 a directory, Afero’s MemMapFs still allowed you to read subdirectories of that directory, whereas a real filesystem doesn’t.
Furthermore, the filesystem semantics vary from operating system to operating system and from filesystem to filesystem. For example, on Windows you cannot delete a file while another process has opened it. Different filesystems have different behavior when it comes to filename limitations (e.g. FAT), case sensitivity, and whether they are case preserving. MemMapFs - and generally any simulated filesystem - will invariably have bugs where it does not behave in the same way as the real filesystem and therefore is not useful for testing.
For this reason, github.com/twpayne/go-vfs wraps Go’s os functions, rather than trying to simulate them. This means that for testing it behaves the same way as the underlying filesystem, and has significantly less code (and therefore hopefully fewer bugs).
Afero’s MemMapFs attempts to simulate a filesystem in memory but the simulation is buggy. One bug I encountered is that file system access is not correct: if you make chmod 000 a directory, Afero’s MemMapFs still allowed you to read subdirectories of that directory, whereas a real filesystem doesn’t.
This is exactly the kind of metadata finicking that made them stick to read-only and no writes :P
Speaking from experience with strange bigcorp data storage systems, I have to say that specific writing interface + generic reading interface works surprisingly well. As a concrete example, append-only filesystems unlock a lot of performance boosts. Having separate PosixWritableFS and WindowsWritableFS interfaces seems prudent.
The first time I heard about io/fs was in adding something like go-bindata to the default toolchain. Is that still the plan, or was that delayed/canceled.
Nice post, thanks. A couple clarifications:
os.File is not an interface
ReadDir is optional so that ordinary files (full of bytes) don’t have to implement dummy ReadDir methods. It would be a bit odd to omit ReadDir on an actual directory, because then any traversal tools won’t work. For example the hypothetical key-value store without ReadDir will not work with fs.Walk (or fs.Glob).
Most code shouldn’t need to define things like readDirStatFS or even refer to the ReadDirFS and StatFS interfaces. It should instead use the helper functions fs.ReadDir and fs.Stat, which use those interfaces internally as appropriate. Code that instead requires, say, the readDirStatFS interface will not work with valid FS implementations that don’t provide a direct fs.Stat method (fs.Stat calls fs.Open+f.Stat+f.Close in that case).
fstest.TestFS does more than just assert that a few files exist. It walks the entire file tree in the file system you give it, checking that all the various methods it can find are well-behaved and diagnosing a bunch of common mistakes that file system implementers might make. For example it opens every file it can find and checks that Read+Seek and ReadAt give consistent results. And lots more. So if you write your own FS implementation, one good test you should write is a test that constructs an instance of the new FS and then passes it to fstest.TestFS for inspection.
One of the most common mistakes you might make in an FS implementation would be to mess up ReadDir on Open(”.”) somehow so that it returns no files at all. If you did this, then TestFS’s walk of the entire file system would find nothing to test, so nothing would fail any tests, and the overall TestFS call would incorrectly pass. The list of files passed to TestFS removes this failure mode by saying “look, when you do the walk looking for things to test, make sure you at least see this list of files, or else ReadDir is broken”.
Again, thanks for taking the time to write the post. It was fun to read.
Thanks for the response! I updated the original post to include these clarifications.
I am looking forward to its use with the
//go:embeddirective for embedding static files in the compiled binary.While the third-party solutions work well (e.g. go-bindata), I usually only need 1 or two files embedded, so having that supported out of the box is a welcome addition, especially when I can combine it with the
io.FSinterfaces so the files don’t need writing to disk for occasional operations which only know how to read from real disks.It’s pretty cool. Cleans things up nicely!
Great write-up!
A few comments as the author of a filesystem abstraction library for Go (
github.com/twpayne/go-vfs):io/fswill surely be useful in some circumstances, but its lack of support for writes is a major omission that limits its usefulness.MemMapFsattempts to simulate a filesystem in memory but the simulation is buggy. One bug I encountered is that file system access is not correct: if you makechmod 000a directory, Afero’sMemMapFsstill allowed you to read subdirectories of that directory, whereas a real filesystem doesn’t.MemMapFs- and generally any simulated filesystem - will invariably have bugs where it does not behave in the same way as the real filesystem and therefore is not useful for testing.For this reason,
github.com/twpayne/go-vfswraps Go’sosfunctions, rather than trying to simulate them. This means that for testing it behaves the same way as the underlying filesystem, and has significantly less code (and therefore hopefully fewer bugs).This is exactly the kind of metadata finicking that made them stick to read-only and no writes :P
Speaking from experience with strange bigcorp data storage systems, I have to say that specific writing interface + generic reading interface works surprisingly well. As a concrete example, append-only filesystems unlock a lot of performance boosts. Having separate PosixWritableFS and WindowsWritableFS interfaces seems prudent.
Great write-up! I hadn’t realized it was going to be in the 1.16 release.
The first time I heard about io/fs was in adding something like go-bindata to the default toolchain. Is that still the plan, or was that delayed/canceled.
That’s there: https://tip.golang.org/pkg/embed/