1. 13
  1.  

  2. 8

    mk(1) is an interesting simplification over make. Reducing the capabilities of make makes the recipes in makefiles a lot easier to parse both for machines and humans. Check it out:

    https://9fans.github.io/plan9port/man/man1/mk.html

    http://www.cs.tufts.edu/%7Enr/cs257/archive/andrew-hume/mk.pdf

    1. 4

      I’m sure it’s fine for small projects, but I’m more interested in tools that everybody including big projects can use (e.g. the Linux kernel which uses GNU Make, or Clang which uses CMake now).

      Android used to be built with 250K+ lines of GNU make, including the “GNU Make Standard Library”, which is basically a Lisp library in Make – complete with Peano numbers as far as I remember! Peano numbers in a library are a good sign you might need integers in your language!!!

      People say bash is bloated, and it is. But I can see that there was demand for every feature. It’s not like they just added features nobody wanted.

      Actually I think the problem is that it has too few features for modern tasks, like say proper hash tables / maps.

      Same with Make.

      For example, how do you extract dependencies from C files with mk? That is, the gcc -M feature I mentioned in the post. Is it documented?

      Also, I just wrote a Makefile for all the access.log.*.gz files that I sync from my web host. That requires a bit of metaprogramming – dynamically constructing the rules from the filenames. The “pattern rules” of GNU make aren’t enough in that case.

      1. 4

        mk builds an entire operating system, plan9. I personally find cmake to be trash. I’m not an expert on mk, but perhaps someone can fill the blanks, but I’m pretty sure mk supports generating dependency lists during the build.

        1. 4

          cmake the language is definitely trash, but it’s now the “go to” thing because it has the best support for cross-platform builds (Mac and Windows in particular).

          As far as I understand, plan9 mk has an easier job, because everything is in the same tree. The C compilers themselves are much simpler than Clang / GCC (but I don’t see anyone clamoring to use Plan 9 C compilers either).

          Plan 9 is small and coherent and adheres to a strict style itself. That is great, but I am more interested in tools that can handle a lot of heterogeneity.

          Everything is small and simple until you need 100 or 1000 people to build it, and for better or worse need 1000 people to build some things (e.g. Android, a completely new mobile phone OS).

          So I would say I am more interested in supersets of GNU Make, not subsets. As far as I can tell, mk is basically a subset of Make.

          1. 3

            As far as I understand, plan9 mk has an easier job, because everything is in the same tree. The C compilers themselves are much simpler than Clang / GCC (but I don’t see anyone clamoring to use Plan 9 C compilers either).

            It goes beyond even that: in Plan 9, there are no shared libraries, and a special #pragma lib "foo" that is inevitably mentioned in foo.h tells Plan 9’s compiler what static libraries a given header depends on. This combo means that mk files can be simpler due to more intelligence in the C compiler itself. You can read a bit more in the Plan 9 programming tutorial (scroll down to “As aside on linking,” which I unfortunately cannot link to directly, since it lacks an anchor.)

          2. 2

            When you have a large complex problem, you can either come up with a convention to reduce the scope of the problem, or you can make the tools more complex to span the whole problem.

            If you control the system, the former is almost always a better choice. If you do not, welcome to the complexity treadmill. It’s hard to write a good, clean, minimal system without a strong set of conventions on the right way to do something.

            1. 4

              I think this orthogonal to the point I was making in the post. If you control the system, sure, I agree. You want to design it with careful conventions and not make everything a special case, blowing up your code.

              But the requirement to “control” the system limits what you can do. I’m more interested in systems with heterogeneity, because I argue every big system has heterogeneity. For example, the Android ecosystem, or even the Apple ecosystem. Apple purposely tries to makes things homogeneous, with some success, but the sheer size of the business means that they ship a ton of code.

              I think of shell as a language for dealing with heterogeneity gracefully. You’re gluing together parts that weren’t designed to be glued together.

              I didn’t make this point that strongly in the post, but here it is:

              • Shell didn’t solve the build problem, so Make was added on top. Shell and Make now largely do the same thing – they invoke processes (in parallel) and they have crappy ways of manipulating strings.
              • The Make language isn’t enough to solve the build problem, so GNU bolted Guile Scheme on top. And I argue that users needed this – I have needed it even for one person projects. It’s not people adding features “for fun”.

              So now you have three Turing complete languages in one system. But the cause is NOT implementing too MUCH, it’s implementing too LITTLE. If shell expanded its domain just a tiny bit, then you wouldn’t need Make. And if shell had richer data structures, or even just integers, you wouldn’t need Guile Scheme.

              I made a point in this direction last year:

              http://www.oilshell.org/blog/2016/11/14.html

              Also, I’ve only read Plan 9 papers, and not used it, but I think it still has this problem of “implementing too little causes other people to implement new tools, increasing global complexity”.

              A sibling comment posted this code which is fairly similar to GNU Make:

              OBJS=${EXES:%=%.o}
              < ${OBJS:%.o=%.d}
              

              In POSIX sh, and I’m guessing Plan 9’s shell rc (?), there are slightly different ways of doing those two things. The languages overlap, which is not “minimal”.

              1. 1

                So now you have three Turing complete languages in one system. But the cause is NOT implementing too MUCH, it’s implementing too LITTLE. If shell expanded its domain just a tiny bit, then you wouldn’t need Make. And if shell had richer data structures, or even just integers, you wouldn’t need Guile Scheme.

                I made a point in this direction last year:

                http://www.oilshell.org/blog/2016/11/14.html

                I completely don’t understand your argument. Mine is that make should do what it does best, which is resolving dependencies. If shell can do it, make shouldn’t replicate. I guess, in plan 9, what actually happened is that awk can do it, shell shouldn’t replicate (about that integer thing). BTW, bash supports integers well, zsh even handles floating points.

                About the parameter expansion, ${OBJS:%.o=%.d}, in plan 9, rc doesn’t have the equivalent syntax as bash, so mk is handling it. This supports the basic ideology that no two tools should share common capabilities.

                1. 3

                  Two answers:

                  I think the argument that make and shell are separate tools, that should do what they do best, is a fallacy. At the very least, shell programs and make programs all have to solve all the same string manipulation problems. Shelling out to sed isn’t really an answer.

                  mk and rc might be somewhat orthogonal rather than overlapping like bash and GNU make, but I would argue that this is probably because they don’t have many users. For example, if rc doesn’t have a method to replace extensions on filenames, that’s probably because nobody uses it very much. That’s an extremely common operation in shell scripts that people need. You need it in Makefiles too obviously.


                  Your original comment sold mk as a subset of GNU Make. As I said, I’m more interested in supersets of GNU make than subsets.

                  However, I took another look at the mk paper now, and it’s actually not a subset of make as you implied. They have a pretty nice comparison at the end.

                  • The regex pattern rules / metarules are exactly what I was referring to in terms of needing “metaprogramming”, so that’s a +1. I still think you need Turing-complete metaprogramming, but being able to capture multiple parts of a path and use \1 and \2 definitely helps.
                  • It seems they support out-of-date computations other than timestamps, so +1. GNU make is completely timestamp-based.
                  • It seems to have a more sane syntax for things like .PHONY, which is just a big hack in GNU Make to avoid actually parsing anything…
                  • I don’t quite see the details, but the algorithm for metarules might be better than that of GNU Make, which I think is O(n^2) at least. Certainly I got huge speedups in practice by disabling the built-in pattern rule database in GNU Make.
                  • I think the model of building the entire dependency graph first makes sense, although I have to look more carefully. GNU Make does have a pretty confused execution model as I mentioned.

                  However, there is a big red flag to me in the first extended example:

                  Now if we want to profile prog (which means compiling everything with the -p option), we need only change the first line to CFLAGS=-g -p and recompile all the object files. The easiest way to recompile everything is with mk -a which says to always make every target regardless of time stamps.

                  It would be a lot better if the build tool used CFLAGS as a key to the cache, so you wouldn’t have to manually rebuild. This is how https://bazel.io/ works for example.

                  So in short, it does seem like mk is interesting and better than the average build tool I see on Github, despite being from 1987!!! And it does look like an improvement on GNU make in many ways. However I think the problem is that there’s no incentive for anyone to switch to it. Being “better” is not enough.

                  This relates to what I’m doing with Oil, in that I believe you have to implement “all of” bash to displace it. Programming languages have network effects (and mk and Make are programming languages), which makes them very hard to displace. Even though I believe mk is better, I still wouldn’t use it for a new project. I would take inspiration from it though.

                  Thanks for the pointer, but I would have liked to have been pointed to the things that it does do rather than what it doesn’t do!

          3. 1

            I’m not sure what gcc -M feature you are talking about. This does not seem to be related to any specific features of GNU make. With mk, you can just do

            CFLAGS=-MMD
            EXES=t
            $EXES:
            OBJS=${EXES:%=%.o}
            < ${OBJS:%.o=%.d}
            %.o:    %.c
                    cc $CFLAGS -c $stem.c
            %:      %.o
                    cc -o $stem $stem.o
            
            1. 1

              By the way, there is a good thing about giving the control back to shell and existing tools rather than being all capable. The include line (the one starts with <) does not really work for multiple targets. A more robust version would be

              <|cat ${OBJS:%.o=%.d} 2>/dev/null || true
              

              You can also throw in that long sed command here if you need it.

              1. 2

                Yeah there are tons of gotchas like this in GNU Make – hence my point about “the obvious thing is wrong”.

                Make gives you virtually no guidance in writing correct build files (i.e. avoiding underspecifying or overspecifying dependencies.) Your build could work fine serially, but be silently incorrect in parallel – the worst kind of bug.

                To me it looks like mk doesn’t address that, but I could be wrong agian. I read the paper a long time ago but haven’t used it.

                Does mk at least solve the problem of not relying on shell? Does it shell out on every line to rc or whatever?

                Also – do you need the sed command or not? I thought you did, but the point of my post was that people told me you don’t (and I agreed after testing it out). It’s documented one way in the GNU make manual, but people do it the other way. So it would be nice if mk documented the correct way to do it.

              2. 1

                The -MMD is what I was talking about. So it looks like mk supports automatic prerequisites in the same way. I guess the only mechanism you need is to have an “include”, which mk does with < apparently.

                That was just one example, and it looks like mk might be sufficient there. One other major feature I want is expressing build variants (i.e. “metaprogramming”).

                For example, it should be easy to make debug/release builds in different trees, and ASAN builds, and code coverage builds, and PGO builds. This is pretty awkward with GNU Make and most build tools I know of.

                I thought I wrote about this here but apparently not:

                http://www.oilshell.org/blog/2017/05/31.html

                One of my pet peeves is recompiling the whole project if I change CFLAGS. I think it discourages the use of important performance and security tools like the ones mentioned above. The compiler has a whole host of things to help you write code, but I feel like these are bottlenecked behind inexpressive build systems which result from inexpressive build languages.