Build systems are hard because building software is complicated.
Maybe it’s the first commit in brand new repository and all you have is foo.c in there. Why am I telling the compiler what to build? What else would it build??
Compilers should not be the build system, their job is to compile. We have abstractions, layers, and separation of concerns for a reason. Some of those reasons are explained in http://www.catb.org/~esr/writings/taoup/html/ch01s06.html. But the bottom line is if you ask a compiler to start doing build system things, you’re going to be frustrated later on when your project is complex and the build system/compiler mix doesn’t do something you need it do.
The good news is that for trivial projects, writing your own build system is likewise trivial as well. You could do it in a few lines of bash if you want. The author did it in 8 lines of Make but still thinks that’s too hard? I mean, this is like buying a bicycle to get you all around town and then complaining that you have stop once a month and spend 5 minutes cleaning and greasing the chain. Everyone just looks at you and says, “Yes? And?”
The author could have done it in two if he knew Make. And no lines if he just has a single file project. One of the more complex projects I have uses only 50 lines of Make, with 6 lines (one implicit rule, and 5 targets) doing the actual build (the rest are various defines).
I tried writing it with implicit rules. Unless I missed something, they only kick in if the source files and the object files are in the same directory. If I’m wrong, please enlighten me. I mentioned the build directory for a reason.
First off, it would build the executable in the same location as the source files. Sadly, I eventually gave up on a separate build directory to simplify the makefile. So with that out of the way:
make could totally handle this project with a single line actually:
foo: foo.c main.c foo.h
That’s more than enough to build the project (replace .c with .o if you want the object files to be generated). Having subdirectories would make it more complex indeed, but for building simple project, we can use a simple organisation!
Implicit rules are made for a case where source and include files are in the same directory as the Makefile. Now we could argue wether or not it’s a good practice or not. Maybe make should have implicit rules hardcoded for src/, include/ and build/ directories. Maybe not.
In your post you say that Pony does it the good way by having the compiler be the build system, and build project in a simple way by default
Maybe ponyc is aware of directories like src/ and include/, and that could be an improvement over make here. But that doesn’t make its build system simple.
When you go the the ponylang website, you find links to “real-life” pony projects. First surprise, 3 of them use a makefile (and what a makefile…): jylis, ponycheck, wallaroo + rules.mk.
One of them doesn’t, but it looks like the author did put some effort in his program organisation so ponyc can build it the simple way.
As @bityard said, building software is complex, and no build system is smart enough to build any kind of software. All you can do is learn to use your tool so you can make a better use of them and make your work simpler.
Disclaimer: I never looked at pony before, so if there is something I misunderstood about how it works, please correct me.
Build systems are hard because building software is complicated.
Some software? Yes. Most software? No. That’s literally the point of the first paragraph of the blog.
Compilers should not be the build system
Disagree.
We have abstractions, layers, and separation of concerns for a reason
Agree.
But the bottom line is if you ask a compiler to start doing build system things, you’re going to be frustrated later on when your project is complex and the build system/compiler mix doesn’t do something you need it do.
Agree, if “the compiler’s default behaviour is the only option. Which would be silly, since the blog’s first paragraph argues that some projects need more than that.
The good news is that for trivial projects, writing your own build system is likewise trivial as well
I think I showed that’s not the case. Trivial is when I don’t have to tell the computer what it already knows.
The author did it in 8 lines of Make but still thinks that’s too hard?
8 lines is infinity times the ideal number, which is 0. So yes, I think it’s too hard. It’s infinity times harder. It sounds like a 6 year old’s argument, but it doesn’t make it any less true.
I have a few projects at work that embed Lua within the application. I also include all the modules required to run the Lua code within the executable, and that includes Lua modules written in Lua. With make I was able to add an implicit rule to generate .o files from .lua files so they could be linked in with the final executable. Had the compiler had the build system “built in” I doubt I would have been able to do that, or I still would have had to run make.
Your examples are ridiculous: using shell invocation and find is far, far from the simplest way to list your source, objects and output files. As other pointed out, you could use implicit rules. Even without implicit rules, that was 2 lines instead of those 8:
foo: foo.c main.c foo.h
gcc foo.c main.c -o foo
Agree, if “the compiler’s default behaviour is the only option.
Ah, then you want the compiler to embed in its code a way to be configured for every and all possible build that it could be used in? This is an insane proposition, when the current solution is either the team writing the project configuring the build system as well (could be done in shell, for all that matters), or thin wrappers like Rust and Go are using around their compilers: they foster best practices while leaving the flexibility needed by heavier projects.
You seem so arrogant and full of yourself. You should not.
Ah, then you want the compiler to embed in its code a way to be configured for every and all possible build that it could be used in?
That’s not at all what he’s asking for.
This is an insane proposition
I think this is probably true.
You seem so arrogant and full of yourself. You should not.
Disagree. He’s stated his opinion and provided examples demonstrating why he believe’s his point is valid. Finally, he has selectively defended said opinion. I don’t think that’s arrogance at all. This, for example, doesn’t read like arrogance to me.
I don’t appreciate the name calling and I don’t think it has a place here on lobste.rs.
What is mostly arrogant is his dismissal of “dumb” tools, simple commands that will do only what they are asked to do and nothing else.
He wants his tools to presume his intentions. This is an arrogant design, which I find foolish, presumptuous, uselessly complex and inelegant. So I disagree on the technical aspects, certainly.
Now, the way he constructed his blog post and main argumentation is also extremely arrogant or in bad faith, by presenting his own errors as normal ways of doing things and accusing other people to build bad tools because they would not do things his way. This is supremely arrogant and I find it distasteful.
Finally, his blog is named after himself and seems a monument to his opinion. He could write on technical matters without putting his persona and ego into it, which is why I consider him full of himself.
My critic is that beside his technical proposition, which I disagree with, the form he uses to present them makes him a disservice by putting people he interacts with on edge. He should not if he wants his writings to be at all impactful, in my opinion.
I think Cargo works well at this. It’s a wrapper for the compiler, but it feels so well-integrated that the distinction doesn’t matter. I’ve never had trouble with stale files with Cargo, or force-built like I’ve had to with Make.
Rustc does as much of the ‘build system’ stuff as Cargo. rustc src/main.rs finds all the files that main.rs needs to build, and builds them all at once. The only exception (i.e. all Cargo has to do) is pointing it at external libraries.
With external libraries, if you have a extern crate foo in your code rustc will deal with that automagically as well if it can (searches a search path for it, you can add things to the search path with -L deps_folder). Alternatively regardless of whether or not you have an extern crate foo (as of Rust2018, prior to that it was always necessary) you can define the dependency precisely by --extern foo=path/to/foo.rlib.
All cargo does, is download dependencies, build them to rlibs as well, and add those --extern foo=path/to/foo declarations (and other options like -C opt-level=3) to a rustc command line based on a config file.
GHC tried to do this. I don’t personally feel that it was a good idea, or that it worked out very well. Fortunately, it wasn’t done in a way that interfered with the later development of Cabal.
Having written a bunch of Nix code, to invoke Cabal, to set up ghc-pkg, to invoke ghc, I would say the situation is less than ideal (and don’t get me started on Stack..) ;)
Did you read the blog? The reason I want the compiler to be involved is to have dependencies to be calculated at the AST node level. That’s definitely not what Go does.
I read it; I was under the impression that your main point was that build systems should “Just Work” without all sorts of Makefile muckery, and that “the compiler [should be] the build system”. The comment about AST based dependencies seemed like a footnote to this.
The go command already works like that. I suppose AST based dependencies could be added to the implementation, but I’m not sure if that would be a major benefit. The Go compiler is reasonably fast to start with (although not as fast as it used to be), and the build cache introduced in Go 1.10 works pretty well already.
I want the compiler … to [calculate dependencies] at the AST node level. That’s definitely not what Go does.
Technically the go tool isn’t the Go compiler (6c/8c), but practically it is, and since the introduction of modules, it definitely parses and automatically resolves dependencies from source. (Even previous, separate tools like dep parsed the AST and resolved the dep graph with a single command.)
Even with languages that don’t have a built in build system, newer build tools do discovery of source files. Examples in the JVM world include gradle, sbt and lein.
If I touch any header, all source files that #include it will be recompiled. This happens even if it’s an additive change, which by definition can’t affect any existing source files.
This isn’t correct, plenty of additive changes could necessitate recompiling other translation units.
Just off the top of my head:
Adding a field to a struct or class
Adding a visibility modifier before a field on a struct or class
Adding a virtual function
Adding a destructor
Adding a copy constructor
Adding the notation that specifically tells the compiler to not automatically generate either of the above
Adding a new function overload
Adding a new template specialization
Adding any code to a template body
Adding a * after a type in any function declaration
I think an “additive change” here refers to adding something completely new (that nothing existing could possibly already depend on); this is in contrast to the addition of text which could modify the semantics of existing functions or types, as I believe most of your examples do in some way.
No build system is going to be able to, in general, distinguish those two scenarios. Consider that your definition of an additive change to foo.h depends not just on the contents of foo.h but the contents of every other header and compilation unit referenced by any compilation unit referencing foo.h, taking into account both macro expansions and compile-time Turing-complete template semantics.
The net effect is that you need to re-compile the entire tree of compilation units that ever reference foo.h just to determine whether or not you need to re-compile them, anyways. Otherwise how do you know whether or not int bar(short x) {return x} is a purely additive change introducing a completely never before seen function bar, or some more-specific overload of some other bar defined in one of the compilation units that included foo.h? You can’t a-priori rule that out.
I’m almost positive that even adding a simple variable doesn’t meet the “unambiguously additive” definition because you could construct some set of templates such that SFINAE outcomes would change in its presence. Ditto typedefs.
There are C macro constructs that also let you alter what the compiler sees based on whether or not a variable/function/etc exists, so even if you had a database of every single thing a compilation unit referenced on last compile and can rule out that foo.c couldn’t see any bar at last compilation it’s going to be impossible to know whether any given addition of a bar to a header would cause something new to be seen by the compiler on a subsequent compilation of foo.c, be it via macro trickery or template metaprogramming.
Your assertion is incorrect, or at least you’re not understanding the semantics of C macros and/or C++ template metaprogramming. Holding the AST of foo.c in memory is not sufficient to determine whether or not any change to foo.h is “additive”, because any change to foo.h can in the extreme lead to a completely different AST being constructed from the textual contents of foo.c on the next compile, in addition to executing an arbitrary amount of Turing-complete c++ compile time template metaprogramming.
You need to reconstruct the AST from foo.c based on the new contents of foo.h to determine whether or not the change was “additive”. That’s a recompilation of foo.c. You save nothing.
Compilers are not magic. They have to process things, compile them, if you will, to know what will happen at the end. Now maybe you can just go to the AST (if your compiler does that) and know from there what changed, but you still need to compile everything to an AST again, and diffing the AST to know what comes next can be complicated and more expensive than just turning the AST into the output. Maybe, just maybe, it only makes sense for massive projects, and not your proposed 10 file project
That’s not what I meant, and I used C instead of C++ for a reason. I meant additive in terms of adding to the API without changing what came before it. I could have been clearer.
If I touch any header, all source files that #include it will be recompiled. This happens even if it’s an additive change, which by definition can’t affect any existing source files.
Is this true? I take additive change to mean one that simply adds lines, without modifying any existing ones. What if the “additive change” is #defining something that is then checked elsewhere and affects the final build?
Putting #if 0 / #endif around blocks are additive changes as well :)
And you definitely want your source to be recompiled after that “addition”.
You can also have portions of code using #ifndef statement that would be impacted by an addition.
I think the author’s point does not stand here. make only cares about timestamp, it doesn’t try to be smart and decide for you wether your changes have an impact or not. Let this job to the compiler, and you’ll get optimizations as a bonus.
Let this job to the compiler, and you’ll get optimizations as a bonus.
Agree. The theoretical “smart build system” that can tell if a header change is “additive” or not is going to end up resembling the compiler (in terms of complexity & workload) very closely. Most of the promised savings will probably vanish.
I think ccache is as “clever” as viable solutions for avoiding unnecessary C or C++ recompiles can get, in this regard. (side note: ccache is fantastic!)
yes, this is one the design flaws of the C language. The complexity of makefiles isn’t essential complexity.
I don’t really like the idea that every language should have its own build system though. I would like a single package manager that can handle all languages.
Launching; it chews up a lot of RAM. I’m a big fan of Bazel’s language too but the implementation feels a bit heavy to me. Fine for a large project but overkill for something small.
Probably it’s due to default -Xms, -Xmx and similar keys to start jvm instance. As an experiment, it’s possible to change these parameters. However, these problems will not go away easily soon, JVM is still not a good choice for command-line utilities, not sure about GrallVM. Bazel have to use daemon because of slow startup time.
Last time when I tried to use it for small project, I faced that it’s too much opinionated: that dependencies should be copied to your repository, that it should be “monorepo”. Almost no one except Google and Facebook do it that way. Or maybe that was Buck, not Bazel, I tried both at that time and may confuse them. I had to retreat to using CMake, which I hated, but it worked (I had to delete build directory few times per day).
Dependencies can be Git submodules (Buck, Bazel) or whole Git repos (Bazel) or HTTP archives (Bazel) or fetched via a package manager (various tools).
Buck and Bazel do not require a monorepo. In fact, they give more flexibility in module layouts than CMake due to the extra level of indirection (“cells” for Buck, “workspaces” for Bazel).
Buck supports directory output for genrules.
Bazel genrules can have multiple output files, which is almost the same thing.
Maybe it’s the first commit in brand new repository and all you have is foo.c in there. Why am I telling the compiler what to build? What else would it build??
make foo
Works though, and you don’t have to write anything.
If you go further with the author’s reflexion, the same reflexion could be made for many programs actually. Why doesn’t vi open the only file in you directory? Wouldn’t it make sense for rmdir to delete the current directory by default? Maybe man should open the man(1) page by default?
Stupid tools are boring, because they will never surprise you, either in a good or bad way. I’m glad that the tool I use are not trying to guess what I want to do, this way they will never fail for another reason than me being bad at expressing what I want. The author push the Pony build system forward as an example solution, and the link provided states this:
Does the name of the directory matter? Yes, it does. It’s the name of your program!
This means that the build system isn’t deterministic at all, this sounds even worse to me than failing with a meaningful error!
It means that if I want to fetch a program in pony, and run the equivalent of make install, I will end up with programs like foo-3.6-258-beta in my path, and trying new versions will make it even worse.
This might simplify the programmer’s life, but the cost for the packager is huge…
“Sure but you can change the bin name with -b!”
… so you put that in a shell script shipped with your code, say build.sh, so you can be sure that your users all get the same program right?
Build systems are important, especially because they’re independant from the compiler. Makefiles are good, simple, and work for ANY programming language. One just has to learn how to write and use it correctly, but that is a totally different topic :)
Something that seems fundamental to this, and that I never seem to see people talking about, is that C and C++ must use an external build system. In languages like Rust, C#, Go, etc there are modules with a particular structure that the compiler can investigate the program and construct dependencies between files without human help, and even find what external libraries to use. You can’t do this in C because there are no modules, just header files. And header files are subject to the whims of macros, ifdefs, and so on, plus everyone organizes them more or less as they feel fit. So there’s no possible way to go from a compiler loading main.c to the compiler knowing what else to build and link.
You could try to dictate a module system via convention, something like “if you include libfoo.h anywhere the program should get linked with libfoo.a”. But the time to make that sort of decision was probably 1975 or so; now you couldn’t do it without breaking nearly every existing program.
This is basically why single-header libraries exist, why make/ninja is always necessary, and so on. Contrast with Rust, where even big and complex programs like Servo are built basically just with Cargo. There’s nothing special about Rust either, besides that it’s had the chance to start fresh and learn from the mistakes of others.
You could try to dictate a module system via convention, something like “if you include libfoo.h anywhere the program should get linked with libfoo.a”. But the time to make that sort of decision was probably 1975 or so; now you couldn’t do it without breaking nearly every existing program.
MSVC has something sorta-kinda like this. You can put #pragma comment(lib, "foo") in libfoo.h. Any file that includes this header will then tell the linker to all try to pull in the “foo” library. Of course the linker still has to be able to find that library somehow. If that library ships with MSVC then it isn’t too much of a problem. But if it is a third-party library you might need to specify the directory where it is located which means it no longer feels like it just works automatically.
I’m not sure why this would break every existing program. I’m sure there would be some problems. For instance where a program includes a header file from a library and only wants to use some macros from that header so the program doesn’t bother linking against the actual library. And maybe the actual library isn’t available in the linker’s search path. In that case then adding a pragma like the above to the header file would mean that the program would now fail to link. But this seems like it would be a rare case.
Don’t forget that when C was invented, the goal was to provide a language more comfortable than assembly.
Languages like rust, go, … had more time to look at all the alternatives, think about the problems existing elsewhere and find good improvements over it. During that time, C tools could only be patched and upgraded to improve usability. I think it was done well because tools like make that were invented afterward but still integrates well to the building process, and can still compete well with the new tools/features from the new languages.
Oh, certainly. In context, C can’t be blamed for this… looking at languages made around similar times, such as Pascal or BCPL, they do basically the same thing. (I thought BLISS was on this list too, but apparently not…) It wasn’t until a decade later that it becomes common that you get things like Ada, Modula-2, Common Lisp, and other languages that make modules an integral part of the language.
https://github.com/apenwarr/redo is so much better than make, or cmake or any other build tool I have tried. It is not much effort to support properly incremental builds in C, and your build scripts can be in any language you like.
On the topic of alternatives to Make, there is also tup, which gains efficiency by mounting a fuse overlay filesystem on top of your source tree so it can update dependencies more efficiently (without scanning the whole source directory on ever build).
I agree we can do better, and a custom language specific solutions will almost always beat out a language agnostic one. However, I didn’t see the author mention things like optimizing techniques like inlining and Link Time Optimizations (LTO), which I believe are key reasons why portions that are seemingly disparate actually end up intertwined and thus require rebuilding.
Good point on LTO and optimisation. The thing is, building an optimised binary something is something I rarely do and don’t particularly care about. It’s all about running the tests for me.
If you want to go straight from build sources in a directory (or subdirectories) to build artifacts, Qt’s qmake‘s project generation can actually do a reasonable job, even if you’re not using Qt.
The basic pipeline is
Sources in a directory → Qt Project file → Makefile → Binary
Normally, the Qt project file MyProject.pro would be considered part of the build sources, while the Makefile is (when using qmake) considered a throwaway artefact of the build process. However, you could also consider the generated project file to be disposable in a similar way.
So something like the following should get you straight from sources to a binary, for a sufficiently simple project:
cd MyProject && qmake -project -r && qmake && make
(I believe qmake will guess “MyProject” as the output name. I can’t remember whether qmake forces out of tree builds, in which case you’d need to cd to an appropriate build directory between the two qmake calls)
I’ve not tried using this for more than trivial projects, but if you have qmake at hand and just want to compile some sources, not write a Makefile from scratch, it can work.
The reason for that is that as a developer, I want to rebuild the minimum possible amount of code that is required after my edits to run my tests.
Ninja seems to do this already. You still have to tell it what to build, but I don’t necessarily agree that having a compiler go off and try to guess what the user wants to build, and it what order, is a good thing.. I may not want an entire feature to get built in a project, for example.
Build systems are hard because building software is complicated.
Compilers should not be the build system, their job is to compile. We have abstractions, layers, and separation of concerns for a reason. Some of those reasons are explained in http://www.catb.org/~esr/writings/taoup/html/ch01s06.html. But the bottom line is if you ask a compiler to start doing build system things, you’re going to be frustrated later on when your project is complex and the build system/compiler mix doesn’t do something you need it do.
The good news is that for trivial projects, writing your own build system is likewise trivial as well. You could do it in a few lines of bash if you want. The author did it in 8 lines of Make but still thinks that’s too hard? I mean, this is like buying a bicycle to get you all around town and then complaining that you have stop once a month and spend 5 minutes cleaning and greasing the chain. Everyone just looks at you and says, “Yes? And?”
The author could have done it in two if he knew Make. And no lines if he just has a single file project. One of the more complex projects I have uses only 50 lines of Make, with 6 lines (one implicit rule, and 5 targets) doing the actual build (the rest are various defines).
What are the two lines?
I’m unsure what the two lines could be, but for no lines I think spc476 is talking about using implicit rules (http://www.delorie.com/gnu/docs/make/make_101.html) and just calling “make foo”
I tried writing it with implicit rules. Unless I missed something, they only kick in if the source files and the object files are in the same directory. If I’m wrong, please enlighten me. I mentioned the build directory for a reason.
Right, the no lines situation only applies for the single file project setup. I don’t know what are the 2 lines for the example given in the post.
First off, it would build the executable in the same location as the source files. Sadly, I eventually gave up on a separate build directory to simplify the
makefile. So with that out of the way:If you want dependencies, then four lines would suffice—the two above plus these two (and I’m using GNUMake if that isn’t apparent):
The target
dependwill modify themakefilewith the proper dependencies for the source files. Okay, make that GNUMake andmakedepend.Structure:
Makefile:
Build it:
Could you please post said two lines? Thanks.
makecould totally handle this project with a single line actually:That’s more than enough to build the project (replace .c with .o if you want the object files to be generated). Having subdirectories would make it more complex indeed, but for building simple project, we can use a simple organisation! Implicit rules are made for a case where source and include files are in the same directory as the
Makefile. Now we could argue wether or not it’s a good practice or not. Maybemakeshould have implicit rules hardcoded for src/, include/ and build/ directories. Maybe not.In your post you say that Pony does it the good way by having the compiler be the build system, and build project in a simple way by default Maybe
ponycis aware of directories like src/ and include/, and that could be an improvement overmakehere. But that doesn’t make its build system simple. When you go the the ponylang website, you find links to “real-life” pony projects. First surprise, 3 of them use a makefile (and what a makefile…): jylis, ponycheck, wallaroo + rules.mk. One of them doesn’t, but it looks like the author did put some effort in his program organisation soponyccan build it the simple way.As @bityard said, building software is complex, and no build system is smart enough to build any kind of software. All you can do is learn to use your tool so you can make a better use of them and make your work simpler.
Disclaimer: I never looked at pony before, so if there is something I misunderstood about how it works, please correct me.
Some software? Yes. Most software? No. That’s literally the point of the first paragraph of the blog.
Disagree.
Agree.
Agree, if “the compiler’s default behaviour is the only option. Which would be silly, since the blog’s first paragraph argues that some projects need more than that.
I think I showed that’s not the case. Trivial is when I don’t have to tell the computer what it already knows.
8 lines is infinity times the ideal number, which is 0. So yes, I think it’s too hard. It’s infinity times harder. It sounds like a 6 year old’s argument, but it doesn’t make it any less true.
I have a few projects at work that embed Lua within the application. I also include all the modules required to run the Lua code within the executable, and that includes Lua modules written in Lua. With
makeI was able to add an implicit rule to generate.ofiles from.luafiles so they could be linked in with the final executable. Had the compiler had the build system “built in” I doubt I would have been able to do that, or I still would have had to runmake.Please, do not ever write a compiler.
Your examples are ridiculous: using shell invocation and find is far, far from the simplest way to list your source, objects and output files. As other pointed out, you could use implicit rules. Even without implicit rules, that was 2 lines instead of those 8:
Ah, then you want the compiler to embed in its code a way to be configured for every and all possible build that it could be used in? This is an insane proposition, when the current solution is either the team writing the project configuring the build system as well (could be done in shell, for all that matters), or thin wrappers like Rust and Go are using around their compilers: they foster best practices while leaving the flexibility needed by heavier projects.
You seem so arrogant and full of yourself. You should not.
I’d like to respectfully disagree with you here.
That’s not at all what he’s asking for.
I think this is probably true.
Disagree. He’s stated his opinion and provided examples demonstrating why he believe’s his point is valid. Finally, he has selectively defended said opinion. I don’t think that’s arrogance at all. This, for example, doesn’t read like arrogance to me.
I don’t appreciate the name calling and I don’t think it has a place here on lobste.rs.
What is mostly arrogant is his dismissal of “dumb” tools, simple commands that will do only what they are asked to do and nothing else.
He wants his tools to presume his intentions. This is an arrogant design, which I find foolish, presumptuous, uselessly complex and inelegant. So I disagree on the technical aspects, certainly.
Now, the way he constructed his blog post and main argumentation is also extremely arrogant or in bad faith, by presenting his own errors as normal ways of doing things and accusing other people to build bad tools because they would not do things his way. This is supremely arrogant and I find it distasteful.
Finally, his blog is named after himself and seems a monument to his opinion. He could write on technical matters without putting his persona and ego into it, which is why I consider him full of himself.
My critic is that beside his technical proposition, which I disagree with, the form he uses to present them makes him a disservice by putting people he interacts with on edge. He should not if he wants his writings to be at all impactful, in my opinion.
Pot, meet Kettle.
Mirrors invite the strongest responses.
yeah. on the flip side we have that too much configuration makes overcomplicated build systems. For me, there’s a sweet spot with cmake.
Doesn’t Go already do this ?
I think Cargo works well at this. It’s a wrapper for the compiler, but it feels so well-integrated that the distinction doesn’t matter. I’ve never had trouble with stale files with Cargo, or force-built like I’ve had to with Make.
Rustc does as much of the ‘build system’ stuff as Cargo.
rustc src/main.rsfinds all the files that main.rs needs to build, and builds them all at once. The only exception (i.e. all Cargo has to do) is pointing it at external libraries.With external libraries, if you have a
extern crate fooin your coderustcwill deal with that automagically as well if it can (searches a search path for it, you can add things to the search path with-L deps_folder). Alternatively regardless of whether or not you have anextern crate foo(as of Rust2018, prior to that it was always necessary) you can define the dependency precisely by--extern foo=path/to/foo.rlib.All cargo does, is download dependencies, build them to rlibs as well, and add those
--extern foo=path/to/foodeclarations (and other options like-C opt-level=3) to a rustc command line based on a config file.Oh, right! That’s neat. I did wonder whether Cargo looked through the module tree somehow, and the answer is that it doesn’t even need to.
GHC tried to do this. I don’t personally feel that it was a good idea, or that it worked out very well. Fortunately, it wasn’t done in a way that interfered with the later development of Cabal.
Having written a bunch of Nix code, to invoke Cabal, to set up ghc-pkg, to invoke ghc, I would say the situation is less than ideal (and don’t get me started on Stack..) ;)
Not in the way I describe, no.
How so?
go installpretty much behaves asmake installfor the vast majority of projects?Did you read the blog? The reason I want the compiler to be involved is to have dependencies to be calculated at the AST node level. That’s definitely not what Go does.
I read it; I was under the impression that your main point was that build systems should “Just Work” without all sorts of Makefile muckery, and that “the compiler [should be] the build system”. The comment about AST based dependencies seemed like a footnote to this.
The
gocommand already works like that. I suppose AST based dependencies could be added to the implementation, but I’m not sure if that would be a major benefit. The Go compiler is reasonably fast to start with (although not as fast as it used to be), and the build cache introduced in Go 1.10 works pretty well already.Technically the
gotool isn’t the Go compiler (6c/8c), but practically it is, and since the introduction of modules, it definitely parses and automatically resolves dependencies from source. (Even previous, separate tools likedepparsed the AST and resolved the dep graph with a single command.)[Comment removed by author]
Some languages that do this well:
rust (with cargo)
d (with dub)
go (kinda) – build process itself is easy enough but the entire infrastructure around building go packages is a mess.
I write D for a living. Dub is a good package manager but it’s a terrible build system. Utterly dire.
Even with languages that don’t have a built in build system, newer build tools do discovery of source files. Examples in the JVM world include gradle, sbt and lein.
This isn’t correct, plenty of additive changes could necessitate recompiling other translation units.
Just off the top of my head:
I think an “additive change” here refers to adding something completely new (that nothing existing could possibly already depend on); this is in contrast to the addition of text which could modify the semantics of existing functions or types, as I believe most of your examples do in some way.
No build system is going to be able to, in general, distinguish those two scenarios. Consider that your definition of an additive change to foo.h depends not just on the contents of foo.h but the contents of every other header and compilation unit referenced by any compilation unit referencing foo.h, taking into account both macro expansions and compile-time Turing-complete template semantics.
The net effect is that you need to re-compile the entire tree of compilation units that ever reference foo.h just to determine whether or not you need to re-compile them, anyways. Otherwise how do you know whether or not
int bar(short x) {return x}is a purely additive change introducing a completely never before seen functionbar, or some more-specific overload of some other bar defined in one of the compilation units that included foo.h? You can’t a-priori rule that out.I’m almost positive that even adding a simple variable doesn’t meet the “unambiguously additive” definition because you could construct some set of templates such that SFINAE outcomes would change in its presence. Ditto typedefs.
There are C macro constructs that also let you alter what the compiler sees based on whether or not a variable/function/etc exists, so even if you had a database of every single thing a compilation unit referenced on last compile and can rule out that foo.c couldn’t see any
barat last compilation it’s going to be impossible to know whether any given addition of abarto a header would cause something new to be seen by the compiler on a subsequent compilation of foo.c, be it via macro trickery or template metaprogramming.A compiler can distinguish between those scenarios. That was the whole point of the blog.
Your assertion is incorrect, or at least you’re not understanding the semantics of C macros and/or C++ template metaprogramming. Holding the AST of foo.c in memory is not sufficient to determine whether or not any change to foo.h is “additive”, because any change to foo.h can in the extreme lead to a completely different AST being constructed from the textual contents of foo.c on the next compile, in addition to executing an arbitrary amount of Turing-complete c++ compile time template metaprogramming.
You need to reconstruct the AST from foo.c based on the new contents of foo.h to determine whether or not the change was “additive”. That’s a recompilation of foo.c. You save nothing.
Compilers are not magic. They have to process things, compile them, if you will, to know what will happen at the end. Now maybe you can just go to the AST (if your compiler does that) and know from there what changed, but you still need to compile everything to an AST again, and diffing the AST to know what comes next can be complicated and more expensive than just turning the AST into the output. Maybe, just maybe, it only makes sense for massive projects, and not your proposed 10 file project
That’s not what I meant, and I used C instead of C++ for a reason. I meant additive in terms of adding to the API without changing what came before it. I could have been clearer.
Is this true? I take additive change to mean one that simply adds lines, without modifying any existing ones. What if the “additive change” is #defining something that is then checked elsewhere and affects the final build?
Putting
#if 0/#endifaround blocks are additive changes as well :) And you definitely want your source to be recompiled after that “addition”. You can also have portions of code using#ifndefstatement that would be impacted by an addition.I think the author’s point does not stand here.
makeonly cares about timestamp, it doesn’t try to be smart and decide for you wether your changes have an impact or not. Let this job to the compiler, and you’ll get optimizations as a bonus.Agree. The theoretical “smart build system” that can tell if a header change is “additive” or not is going to end up resembling the compiler (in terms of complexity & workload) very closely. Most of the promised savings will probably vanish.
I think ccache is as “clever” as viable solutions for avoiding unnecessary C or C++ recompiles can get, in this regard. (side note: ccache is fantastic!)
yes, this is one the design flaws of the C language. The complexity of makefiles isn’t essential complexity.
I don’t really like the idea that every language should have its own build system though. I would like a single package manager that can handle all languages.
That what Bazel does. But most projects are monolingual so it’s overkill for them.
Why “overkill”? A small Bazel file for a C project is very readable (perhaps more readable than the equivalent Make even).
Additionally, you do not need to rethink your build-system if the project scales up.
Oh, it’s definitely readable. But the fixed overhead of using Bazel doesn’t seem worth it for small projects.
Do you mean the overhead of launching? installing?
As a big fan of Bazel, I am genuinely curious why it is not more popular. It feels like the problem of build-system design has largely been solved.
Launching; it chews up a lot of RAM. I’m a big fan of Bazel’s language too but the implementation feels a bit heavy to me. Fine for a large project but overkill for something small.
You might find Pants interesting. It is a Blaze clone that is currently being rewritten in Rust. https://github.com/pantsbuild/pants
Oh, cool! So it uses the same language for the BUILD files & can be a drop-in replacement for Bazel?
It uses the same language (Starlark), but sadly it is not a drop-in replacement. There are small differences such as
sourcesinstead ofsrcs.Maybe a shim library could be made?
Probably it’s due to default
-Xms,-Xmxand similar keys to start jvm instance. As an experiment, it’s possible to change these parameters. However, these problems will not go away easily soon, JVM is still not a good choice for command-line utilities, not sure about GrallVM. Bazel have to use daemon because of slow startup time.Last time when I tried to use it for small project, I faced that it’s too much opinionated: that dependencies should be copied to your repository, that it should be “monorepo”. Almost no one except Google and Facebook do it that way. Or maybe that was Buck, not Bazel, I tried both at that time and may confuse them. I had to retreat to using CMake, which I hated, but it worked (I had to delete
builddirectory few times per day).But now there’s even experimental build rules for CMake integration, it always had http_archive for fetching external dependencies, but genrule still does not support directory as output. Going to try it again some day.
Dependencies can be Git submodules (Buck, Bazel) or whole Git repos (Bazel) or HTTP archives (Bazel) or fetched via a package manager (various tools).
Buck and Bazel do not require a monorepo. In fact, they give more flexibility in module layouts than CMake due to the extra level of indirection (“cells” for Buck, “workspaces” for Bazel).
Buck supports directory output for genrules.
Bazel genrules can have multiple output files, which is almost the same thing.
Works though, and you don’t have to write anything.
If you go further with the author’s reflexion, the same reflexion could be made for many programs actually. Why doesn’t
viopen the only file in you directory? Wouldn’t it make sense forrmdirto delete the current directory by default? Maybemanshould open theman(1)page by default?Stupid tools are boring, because they will never surprise you, either in a good or bad way. I’m glad that the tool I use are not trying to guess what I want to do, this way they will never fail for another reason than me being bad at expressing what I want. The author push the Pony build system forward as an example solution, and the link provided states this:
This means that the build system isn’t deterministic at all, this sounds even worse to me than failing with a meaningful error!
It means that if I want to fetch a program in pony, and run the equivalent of
make install, I will end up with programs like foo-3.6-258-beta in my path, and trying new versions will make it even worse. This might simplify the programmer’s life, but the cost for the packager is huge… “Sure but you can change the bin name with-b!”… so you put that in a shell script shipped with your code, say
build.sh, so you can be sure that your users all get the same program right?Build systems are important, especially because they’re independant from the compiler. Makefiles are good, simple, and work for ANY programming language. One just has to learn how to write and use it correctly, but that is a totally different topic :)
Something that seems fundamental to this, and that I never seem to see people talking about, is that C and C++ must use an external build system. In languages like Rust, C#, Go, etc there are modules with a particular structure that the compiler can investigate the program and construct dependencies between files without human help, and even find what external libraries to use. You can’t do this in C because there are no modules, just header files. And header files are subject to the whims of macros, ifdefs, and so on, plus everyone organizes them more or less as they feel fit. So there’s no possible way to go from a compiler loading main.c to the compiler knowing what else to build and link.
You could try to dictate a module system via convention, something like “if you include
libfoo.hanywhere the program should get linked withlibfoo.a”. But the time to make that sort of decision was probably 1975 or so; now you couldn’t do it without breaking nearly every existing program.This is basically why single-header libraries exist, why make/ninja is always necessary, and so on. Contrast with Rust, where even big and complex programs like Servo are built basically just with Cargo. There’s nothing special about Rust either, besides that it’s had the chance to start fresh and learn from the mistakes of others.
MSVC has something sorta-kinda like this. You can put
#pragma comment(lib, "foo")in libfoo.h. Any file that includes this header will then tell the linker to all try to pull in the “foo” library. Of course the linker still has to be able to find that library somehow. If that library ships with MSVC then it isn’t too much of a problem. But if it is a third-party library you might need to specify the directory where it is located which means it no longer feels like it just works automatically.I’m not sure why this would break every existing program. I’m sure there would be some problems. For instance where a program includes a header file from a library and only wants to use some macros from that header so the program doesn’t bother linking against the actual library. And maybe the actual library isn’t available in the linker’s search path. In that case then adding a pragma like the above to the header file would mean that the program would now fail to link. But this seems like it would be a rare case.
Don’t forget that when C was invented, the goal was to provide a language more comfortable than assembly. Languages like rust, go, … had more time to look at all the alternatives, think about the problems existing elsewhere and find good improvements over it. During that time, C tools could only be patched and upgraded to improve usability. I think it was done well because tools like
makethat were invented afterward but still integrates well to the building process, and can still compete well with the new tools/features from the new languages.Oh, certainly. In context, C can’t be blamed for this… looking at languages made around similar times, such as Pascal or BCPL, they do basically the same thing. (I thought BLISS was on this list too, but apparently not…) It wasn’t until a decade later that it becomes common that you get things like Ada, Modula-2, Common Lisp, and other languages that make modules an integral part of the language.
https://github.com/apenwarr/redo is so much better than make, or cmake or any other build tool I have tried. It is not much effort to support properly incremental builds in C, and your build scripts can be in any language you like.
On the topic of alternatives to Make, there is also tup, which gains efficiency by mounting a fuse overlay filesystem on top of your source tree so it can update dependencies more efficiently (without scanning the whole source directory on ever build).
I agree we can do better, and a custom language specific solutions will almost always beat out a language agnostic one. However, I didn’t see the author mention things like optimizing techniques like inlining and Link Time Optimizations (LTO), which I believe are key reasons why portions that are seemingly disparate actually end up intertwined and thus require rebuilding.
Good point on LTO and optimisation. The thing is, building an optimised binary something is something I rarely do and don’t particularly care about. It’s all about running the tests for me.
Thta’s kind of what I figured, but I didn’t see it mention only debug builds. For debug builds, I totally agree with you!
If you want to go straight from build sources in a directory (or subdirectories) to build artifacts, Qt’s
qmake‘s project generation can actually do a reasonable job, even if you’re not using Qt. The basic pipeline isNormally, the Qt project file
MyProject.prowould be considered part of the build sources, while the Makefile is (when using qmake) considered a throwaway artefact of the build process. However, you could also consider the generated project file to be disposable in a similar way.So something like the following should get you straight from sources to a binary, for a sufficiently simple project:
(I believe qmake will guess “MyProject” as the output name. I can’t remember whether qmake forces out of tree builds, in which case you’d need to cd to an appropriate build directory between the two qmake calls)
I’ve not tried using this for more than trivial projects, but if you have qmake at hand and just want to compile some sources, not write a Makefile from scratch, it can work.
Ninja seems to do this already. You still have to tell it what to build, but I don’t necessarily agree that having a compiler go off and try to guess what the user wants to build, and it what order, is a good thing.. I may not want an entire feature to get built in a project, for example.
We can try to use xmake to simplify the maintenance and construction of c/c++ projects.
Simple syntax, built-in dependency package management
https://github.com/xmake-io/xmake