Make is a file based build system too. The difference appears to be that tup basically inverts the description of the dependency tree. I don’t want to say more because I’ve never used it, but this certainly seems like something worth exploring.
Eh, still annoying, especially if you want to do it automatically, without manually specifying path dependencies. I know I ended up with lots of sentinel files whenever I ended up trying that.
I used to use Tup before switching to Rust. A surprisingly powerful and well-designed little tool, I much preferred using it over make/cmake/etc. The fact that it’s language-agnostic is perfect for those odd little projects with non-standard setups, like hobby OS kernels.
One thing you may have noticed is that both C files include the square.h header, but we haven’t specified it as an input to the command. You may be surprised to see that we can still change the header and cause both files to recompile to pick up the change […] The trick is that tup instruments all commands that it executes in order to determine what files were actually read from (the inputs) and written to (the outputs).
With what, strace? Automatically determining what files are read could end up causing a lot of unnecessary rebuilds right? Perhaps I’m missing something.
Tup uses FUSE to discover dependencies. A FUSE filesystem which proxies all requests to the underlying filesystem is mounted, and reads and writes are recorded as dependencies. See https://github.com/gittup/tup/tree/master/src/tup/server for how it is done.
Pretty sure it uses a FUSE filesystem to essentially enforce the dependency graph. If a build process for a target file accesses an asset, that asset is silently marked as a dependency, causing a rebuild of the target if that asset is modified. Such a rebuild is (axiomatically) desirable, or why would the build process have read the file?
Similarly, when a build process creates extra files, those are marked as outputs. If a future run of the build process doesn’t touch those outputs, they’re automatically cleaned up (which is desirable, or the build process would have accessed them).
Finally, if these undeclared input or output files are also defined as (unrelated!) targets, the build halts and complains that the dependency graph you’ve defined is insufficient: it needs to know which targets to build first, that’s the whole point of the graph. It forbids violations of the graph you’ve defined and demonstrable insufficiencies within that graph, based on material observations of the build process, which it then uses to intelligently keep build outputs clean and up-to-date.
I loved playing with it, had to really bend it’s rules to get it to parse files for declared imports (it has strong opinions), and am glad to see that it’s not as harsh about having strictly one Tupfile per directory level of outputs as it used to be, that was a bit of a pain (re: strongly opinionated). Still a little confusing to know what $PWD is going to be if you’re using a Tupfile.default file. Lua extension is really cool.
Jeez, this is way more informative than the linked article. I could have used 80% less cheerleading about how great they are and 100% more description of what it actually is and does. Thanks!
If a build process for a target file accesses an asset, that asset is silently marked as a dependency, causing a rebuild of the target if that asset is modified. Such a rebuild is (axiomatically) desirable, or why would the build process have read the file?
Because it needs it for some part of the functionality that doesn’t affect the output? Say translations of the diagnostics messages.
We’ve actually ran into this with environment variables: quite often tools query environment variables that have nothing to do with their output (verbosity, color output, etc). So if you want to track changes accurately, you not only need to get the list of the environment variables (which is often a pain by itself), but you actually need to understand their semantics.
Similarly, when a build process creates extra files, those are marked as outputs. If a future run of the build process doesn’t touch those outputs, they’re automatically cleaned up (which is desirable, or the build process would have accessed them).
Again, I can think of a scenario where a tool produces multiple files and on a re-run checks one of them to determine if anything changes and skips touching the rest if not. Say a sentinel file guarding a directory full of files?
The underlying theme here is that in software builds relying on implicit knowledge and assumptions is asking for trouble.
[…] for some part of the functionality that doesn’t affect the output? Say translations of the diagnostics messages.
Ah, thanks! I should have mentioned that, because this FUSE FS is mounted over your build directoy, it doesn’t catch reads of eg. system locales. Configuration files affecting colors, verbosity, or translations within your repository are ofc significant (they’re part of you’re program), and changes to them should naturally cause rebuilds and be manually tested for correctness.
I can think of a scenario where a tool produces multiple files and on a re-run checks one of them to determine if anything changes and skips touching the rest
Sounds like the kind of caching a build system would do :p
One probably shouldn’t nest build systems, but in practice, the way to do this would be to run the secondary build system before or after (probably before) running Tup, which begs the question “what if I need to run it in the middle?”, in which case the answer is to “not nest build systems” and migrate / break-up that build process into Tup rules ¯\_(ツ)_/¯
If you’ve got something that does this sort of caching, isn’t a build system, and can’t be broken up into a non-batch process via flags or options, then yeah you got me there.
Make is a file based build system too. The difference appears to be that tup basically inverts the description of the dependency tree. I don’t want to say more because I’ve never used it, but this certainly seems like something worth exploring.
Yeah, just used that title as it’s the official tagline. Several things stood out to me though:
I did a small experiment with it, and managed to take a 30-ish line Makefile into a (more functional!) 3-line Tupfile.
It is? Just use an order-only dependency.
Eh, still annoying, especially if you want to do it automatically, without manually specifying path dependencies. I know I ended up with lots of sentinel files whenever I ended up trying that.
I don’t get how you end up with sentinel files. Just do
[Comment removed by author]
I used to use Tup before switching to Rust. A surprisingly powerful and well-designed little tool, I much preferred using it over make/cmake/etc. The fact that it’s language-agnostic is perfect for those odd little projects with non-standard setups, like hobby OS kernels.
With what, strace? Automatically determining what files are read could end up causing a lot of unnecessary rebuilds right? Perhaps I’m missing something.
Tup uses FUSE to discover dependencies. A FUSE filesystem which proxies all requests to the underlying filesystem is mounted, and reads and writes are recorded as dependencies. See https://github.com/gittup/tup/tree/master/src/tup/server for how it is done.
Pretty sure it uses a FUSE filesystem to essentially enforce the dependency graph. If a build process for a target file accesses an asset, that asset is silently marked as a dependency, causing a rebuild of the target if that asset is modified. Such a rebuild is (axiomatically) desirable, or why would the build process have read the file?
Similarly, when a build process creates extra files, those are marked as outputs. If a future run of the build process doesn’t touch those outputs, they’re automatically cleaned up (which is desirable, or the build process would have accessed them).
Finally, if these undeclared input or output files are also defined as (unrelated!) targets, the build halts and complains that the dependency graph you’ve defined is insufficient: it needs to know which targets to build first, that’s the whole point of the graph. It forbids violations of the graph you’ve defined and demonstrable insufficiencies within that graph, based on material observations of the build process, which it then uses to intelligently keep build outputs clean and up-to-date.
I loved playing with it, had to really bend it’s rules to get it to parse files for declared imports (it has strong opinions), and am glad to see that it’s not as harsh about having strictly one Tupfile per directory level of outputs as it used to be, that was a bit of a pain (re: strongly opinionated). Still a little confusing to know what
$PWDis going to be if you’re using aTupfile.defaultfile. Lua extension is really cool.Jeez, this is way more informative than the linked article. I could have used 80% less cheerleading about how great they are and 100% more description of what it actually is and does. Thanks!
Because it needs it for some part of the functionality that doesn’t affect the output? Say translations of the diagnostics messages.
We’ve actually ran into this with environment variables: quite often tools query environment variables that have nothing to do with their output (verbosity, color output, etc). So if you want to track changes accurately, you not only need to get the list of the environment variables (which is often a pain by itself), but you actually need to understand their semantics.
Again, I can think of a scenario where a tool produces multiple files and on a re-run checks one of them to determine if anything changes and skips touching the rest if not. Say a sentinel file guarding a directory full of files?
The underlying theme here is that in software builds relying on implicit knowledge and assumptions is asking for trouble.
Ah, thanks! I should have mentioned that, because this FUSE FS is mounted over your build directoy, it doesn’t catch reads of eg. system locales. Configuration files affecting colors, verbosity, or translations within your repository are ofc significant (they’re part of you’re program), and changes to them should naturally cause rebuilds and be manually tested for correctness.
Sounds like the kind of caching a build system would do :p
One probably shouldn’t nest build systems, but in practice, the way to do this would be to run the secondary build system before or after (probably before) running Tup, which begs the question “what if I need to run it in the middle?”, in which case the answer is to “not nest build systems” and migrate / break-up that build process into Tup rules ¯\_(ツ)_/¯
If you’ve got something that does this sort of caching, isn’t a build system, and can’t be broken up into a non-batch process via flags or options, then yeah you got me there.
Sounds like a portability nightmare as well.
From what I’ve seen in the examples, the lines of a Tupfile can easily grow very long, yet the manual doesn’t talk about line continuations,
Does anyone know if lines can be broken up with a backslash, as in a shell script?
Missed opportunity to call it
comasince system tools can’t be six characters long.Edit: this thing is really(!) cool, makes me want to find use for it.
Wait, what?
No clue. Maybe they are referring to the super old 8 character limitation on FAT? https://en.wikipedia.org/wiki/8.3_filename
https://unix.stackexchange.com/a/214801
Not into category theory and crossword puzzles?
… oh, like co-make!
Ding ding