On the other hand, although I’ve known about this for many years and always been curious, I’ve never actually tried to use it, whereas I could wrap your concept around almost anything pretty trivially, so :)
Thanks. This work mostly came out of being frustrated with js build scripts. I didn’t want to understand what broccoli.js was doing so I figured treating it as a black box and just grabbing the inputs and outputs would be good enough for caching purposes. I’ve since used the same trick at a few more workplaces and it does wonders for frontend build processes.
Now let’s look at what files the script was reading from (there is again some extraneous cruft involved but those can be easily filtered out if desired)
I think the filtering might get quite complicated quite quickly once you start building and using dynamic libraries.
Hard time writing an algorithm that separates actual input files for the compilation from all other files that are being opened by the compilation process. Maybe though for 99% of the time it really doesn’t matter if you just hash everything that gets opened.
Yes, you can write heuristics but it fundamentally requires some understanding of the build process. I usually manually inspect the trace and then group everything according to what I know about the build. It usually works out but is a manual process that I haven’t figured out how to fully automate.
A while back, I was searching around if there is a way to get CMake to use such a technique to validate the build config. Does anybody know something like that?
I’m not sure. I think all language dependent build systems must do some kind of dependency tracking but whether they expose that information or not is a different issue. I think D does expose this information and can even generate a make file if you pass in the right flags but I’m not an expert so someone else will have to chime in.
“Fabricate” is a build system that uses strace to automatically discover dependencies in this way.
https://github.com/SimonAlfie/fabricate/blob/master/README.md
Nice. Looks like they beat me to this idea by a wide margin. Just goes to show there is not much new under the sun.
On the other hand, although I’ve known about this for many years and always been curious, I’ve never actually tried to use it, whereas I could wrap your concept around almost anything pretty trivially, so :)
Thanks. This work mostly came out of being frustrated with js build scripts. I didn’t want to understand what broccoli.js was doing so I figured treating it as a black box and just grabbing the inputs and outputs would be good enough for caching purposes. I’ve since used the same trick at a few more workplaces and it does wonders for frontend build processes.
I think the filtering might get quite complicated quite quickly once you start building and using dynamic libraries.
What issues do you foresee?
Hard time writing an algorithm that separates actual input files for the compilation from all other files that are being opened by the compilation process. Maybe though for 99% of the time it really doesn’t matter if you just hash everything that gets
opened.Yes, you can write heuristics but it fundamentally requires some understanding of the build process. I usually manually inspect the trace and then group everything according to what I know about the build. It usually works out but is a manual process that I haven’t figured out how to fully automate.
A while back, I was searching around if there is a way to get CMake to use such a technique to validate the build config. Does anybody know something like that?
I’m not sure. I think all language dependent build systems must do some kind of dependency tracking but whether they expose that information or not is a different issue. I think D does expose this information and can even generate a make file if you pass in the right flags but I’m not an expert so someone else will have to chime in.