1. 18

    Many years ago when I worked for (what was then) RIM I was working on a team that was creating a new hardware device (spoiler alert: it was never released). I was the system software person and one of the hardware guys ran into what seemed to be a compiler bug with memcpy.

    We take a look and sure enough, copying this chunk of memory from place A to B resulted in the copy not being the same as the original. A few more tests were run and it certainly seemed to be miscompiled. We look at the assembler output, hoping to be right and… it’s fine! Everything makes sense. But we can see it being wrong. So what’s up?

    After a bit of sheer confusion it’s time to break out the hardware examination kit. Upon close examination with a magnifying glass, we find the culprit: the RAM connection was wired wrong, so bytes are being swapped when they are moved from one place to another.

    Unsurprisingly, that sample run of devices was deemed to be no good.

    1. 3

      Yeah, this is a difference I noticed almost as soon as I appeared here! Where I come from, this [the idea that the best programmers are in their 20s] is completely ridiculous. The most important thing in software design is problem framing and you need a lot of expertise for that. The best software designers in our world are often in their 50s or 60s!

      I enjoyed this article and I can’t help but feel it was written just to make the above point.

      1. 5

        I just spent the last few weeks going through and categiorizing (and possibly fixing) all the test failures in a GCC port I’ve been working on for a while. It was tedious but necessary. The happy result is that nearly all of the failures were minor environment issues or simple, inconsequential errors.

        As a bonus, I now understand much more about the interals of the product because in many cases I had to dig into the actual problem. That grunt work can pay off in much deeper knowledge.

        1. 7

          It’s certainly true that using a simple memcmp that doesn’t take advantage of the architecture can make a big difference.

          This test is good torture test of memcmp. Built with gcc -O2 -DTEST_ALL on my machine, it takes about 68 seconds to run. Forcing it to use this implementation of memcmp (and allowing for inlining) results in the test taking about 89 seconds. If you disallow inlining, it takes about 96 seconds.

          memcmp and friends, when using architecture-specific instructions, can be very fast. It is very much worth your time to profile them for your use case because they may make a simpler algorithm with a worse time complexity faster than a more complicated algorithm with a better time complexity, mainly because “linear” may not be as linear as it looks.

          1. 6

            I’ve spent the last couple of years porting and implementing a backend for GCC and not having useful commit messages is a serious drag. I’ve also found that Changelog messages are essentially useless. When there’s no context or discussion around why a change was made, I’m left to dig through the mailing list in hopes it was discussed there (it usually isn’t). If GCC committers followed the advise here (the formatting stuff is less important), I’d probably have fewer problems. (*)

            At work, each commit is supposed to be tied to a ticket/issue number in the issue tracking software. This seems like excessive ceremony, but it pays off in the long run because you can actually track down reasoning behind a decision months later when something goes wrong related to what you are investigating. I think having an issue number is about as good or better than a descriptive commit message.

            (*) More recent GCC commits are better about writing useful commit messages, I must admit.

            1. 6

              I strongly recommend the Clang Undefined Behavior Sanitizer — it adds runtime checks to your code that detect and flag (among many other things) integer overflows. I always enable this in my dev/debug builds.

              Some of the weirder rules only apply to long-obsolete hardware. I think you hav to go back to the 1960s to find CPUs with 36- or 18-bit words, or that don’t use 2s-complement arithmetic, and to the 1970s to find ones whose memory isn’t byte-addressable (i.e. “char” is bigger than 1 byte.) And EBCDIC never made it out of IBM mainframes.

              Bu I guess if you ignore those factors you’ll get outraged bug reports from the retro-computing folks complaining that your code breaks on OS\360, TENEX or TOPS-20…

              1. 6

                Some of the weirder rules only apply to long-obsolete hardware. I think you hav to go back to the 1960s to find CPUs with 36- or 18-bit words, or that don’t use 2s-complement arithmetic, and to the 1970s to find ones whose memory isn’t byte-addressable (i.e. “char” is bigger than 1 byte.) And EBCDIC never made it out of IBM mainframes.

                You need to look no further than the SHARC, still one of the most popular lines of DSP lines at the moment, to find an architecture where sizeof(char) = 4.

                (Edit: FWIW, I think a better approach to these would be to make the standard stricter, even if that means more non-compliant implementations for “special” architectures like the SHARC. I’m pretty sure AD’s C compiler isn’t (or wasn’t, thankfully for my mental sanity I haven’t touched it in like 5 years) standards-compliant anyway because sizeof(int) is also 4. We’re at a stage where you expect inconsistencies and bugs in vendor-supplied compilers anyway. There’s no need for anyone else to suffer just so a bunch of vendors with really particular requirements to be able to claim compliance, when most of them don’t care about it anyway.)

                1. 5

                  You need to look no further than the SHARC, still one of the most popular lines of DSP lines at the moment, to find an architecture where sizeof(char) = 4.

                  Indeed. It’s the little, low-level, specialized devices where all the weirdness shows up. Basically, if it’s not a 32-bit device there is probably something the violates your expectations. Don’t assume the world is made up of the computers you’re used to using.

                  1. 4

                    Ok, mind blown! I did not know that. But I can see how a DSP platform wouldn’t see byte-addressibility as necessary.

                    1. 1

                      Wait, isn’t sizeof(char) = 1 by definition? I suspect that what you meant to say is that for the C implementation that runs on Analog Devices’ SHARC DSP, char is 32 bits wide, int is also 32 bits wide, and actually sizeof(int) = 1.

                      1. 1

                        You may be right, I don’t have the compiler at hand anymore. I’m sure (it caused me a lot of headaches when porting some code from elsewhere) that sizeof(char) and sizeof(int) were equal but I really don’t remember which of these two puzzling results it yielded.

                    2. 3

                      And EBCDIC never made it out of IBM mainframes.

                      This is true, but there’s still a lot of code running (and being maintained!) on mainframes, so it can’t be ignored.

                      1. 1

                        Didn’t ICL mainframes also use EBCDIC?

                    1. 1

                      …when you need to register to GCC passes (we will see what they are), you have to take care of passing -fno-rtti option. I don’t remember even how I find it… for sure caused me a severe headache.

                      Most (all?) of the information needed to deal with plugins are found in the internals documentation. In the subsection on building plugins is describes precisely how to build them, including the -fno-rtti option (which you need because the compiler is built without RTTI info).

                      I noticed that the article doesn’t cite the internals docs in its list of resources. Maybe they will have a better time after reading through that part.

                      1. 12

                        Using XML at all leads inevitably to centralization.

                        This is a logical leap of epic proportions. I fail to see how the argument put forth backs it up.

                        HTTP is also bloated and over-complex in the same way. And don’t get me started on certificates and certificate chains. Or fucking DNS.

                        It doesn’t take a genius to come up with better formats than these for the same applications. These aren’t inherently-complex problems.

                        Hint: the formats are not the compilicated (or even interesting) part. They could all be XML right now and it wouldn’t change much of anything.

                        1. 3

                          If you use XML, you are either writing your own XML parser or using someone else’s.

                          Writing your own is hard – much harder than the project you’re planning to use XML for, probably.

                          Using someone else’s is centralization: there aren’t many good XML parsers available, so you’re placing more power and trust in the hands of Jackson or libxml (which already underpin practically everything).

                          A non-centralizing format is one where rolling your own implementation is not an obviously terrible idea.

                          A format where any developer on the team can implement it more or less correctly in an afternoon is also a format that isn’t going to become a problem later - one where developers can reason about the behavior of parsers and generators, where different implementations can be expected to generally agree, and where corner cases that might be handled differently between implementations can be identified and avoided. It’s a format where auditing third party implementations is also potentially straightforward (and you can reject difficult-to-audit ones out of hand). So there are more reasons to prefer them than just avoiding power consolidation.

                          (Maybe I should rewrite this essay, seeing as how I need to rephrase its core thesis in response to practically every response.)

                          1. 6

                            Using someone else’s is centralization:…

                            A non-centralizing format is one where rolling your own implementation is not an obviously terrible idea.

                            By this logic, code reuse is “centralization”. Is this actually what you’re arguing? Because it sounds pretty ridiculous.

                            I, for one, would prefer to use a well-tested, auditable implementation than roll my own in an afternoon.

                            1. 3

                              As a practical example of the difference between XML and JSON when using popular libraries isn’t possible, let’s look at Monte’s JSON support. Monte doesn’t have any XML parsers. It does have a JSON parser, which started out as a handwritten lexer and parser. Later, we were able to write basic parser combinators, and that led to the current combinator-driven parser, which is smaller and faster. JSON is quicker than XML to support, even considering the effort required to write a parser library.

                              Parsing even a comfortable and reasonable subset of XML is hard enough that I’ve simply not parsed XML in-process; instead, I’ll look for external tools like relpipe (XML example) which will do it for me, handling escaping and CDATA and entities and all of that fun stuff. Calling subprocesses is safer than XML to support, even considering the security risks.

                              I could imagine JSON-specific arguments against this, so as a tiebreaker, we could consider Monte’s support for Capn Proto. We implement the recommended convention, and maintain a tool which compiles Capn Proto schemata into Monte ASTs. This requires a lot of binary parsing, recursive algorithms for nested data structures, deferred/lazy parsing, and other hallmarks of XML parsers. Capn Proto is much easier than XML to support, even considering the code generation that is required.

                              To exemplify a point from the author’s sibling response:

                              If correct implementation is such a problem that you would rather have a third party do it for you, then you are probably not qualified to independently audit or test a third party implementation! Sometimes (as with encryption) this is unavoidable: the problem domain is so complex that you need to trust expert opinion, and rolling your own is never going to be a good idea.

                              Monte is a language where we generally don’t have FFI; we require binding libsodium and are prepared to pay the price for vulnerabilities in that particular C library, but we’re not willing to make that trade for a JSON or XML parser, and we won’t let users try to make that choice for themselves, either.

                              1. 3

                                This matches my experience.

                                I’m not coming to this from a place of ignorance of XML (though I had a fair amount of ignorance of RSS while implementing it); I’m coming from a place of having spent a whole lot of time dealing with Jackson, libxml, and various implementations of xpath and xslt, in multiple languages, for work and often discovering that because the other end actually uses a tiny fraction of the expressive domain of XML, the easiest and most reliable way to get the data I needed was to use sed. At the same time, hand-processing subsets of json (or even all of msgpack) was a lot easier than doing fairly simple things with existing xml parsers.

                              2. 2

                                If you’re reusing your own code? I wouldn’t call that centralization of power (although sometimes poor factoring will lead to some of the same kinds of concerns – ex., if you have a module intended to be general purpose but the needs of the functions that use it cause it to become too tightly coupled, the behavior that must be implemented in order to make this ‘general purpose module’ work for one function may break the other in subtle ways). So, shared code is a shared point of failure.

                                But dependencies, on top of being a shared point of failure, also need to be trusted or audited. Core complexity is both a major factor in implementation difficulty and a major factor in auditing & testing difficulty. Doing a complete audit (where you have absolute confidence that the behavior is correct) can sometimes be more difficult than a full reimplementation, since you need to have a complete model of the correct behavior (and if you are using the right tools, having a complete and correct model of the problem is the most difficult part of implementation); testing can help, at the cost of adding complexity and corner cases (ex., where is it safe and appropriate to add mocks – and must you modify the natural structure of the code to do so?).

                                If correct implementation is such a problem that you would rather have a third party do it for you, then you are probably not qualified to independently audit or test a third party implementation! Sometimes (as wtih encryption) this is unavoidable: the problem domain is so complex that you need to trust expert opinion, and rolling your own is never going to be a good idea. But if you are, for instance, looking to serialize a key-value table or some kind of nested tree structure, there are no ‘essential’ gotchas (anybody with a CS degree knows everything you need to watch out for in both these cases to guarantee correctness) – so why deal with complexity introduced by the format? And on the other hand, why put trust into unknown third parties when the only thing you need to do to make checking correctness easy is to roll your own (simpler) format?

                                Depending on your needs, it is often going to be easier to roll your own format than to use an existing parser implementation for a more complex one (especially with XML, where the format is awkward and all of the parsers are awkward too, simply due to the way the spec is defined); if you are using XML, odds are there’s an existing simpler format (with easier-to-audit implementations) that cleanly handles whatever you need to handle, and furthermore, a subset of that format that supports all your data without ugly corner cases.

                                The correct format to use is the simplest format that is rich enough to express everything you need to express. Don’t use XML when you can use JSON and don’t use JSON when you can use TSV, and don’t use TSV with quoting if you can use TSV without quoting – and if you can use TSV without quoting, implement it with string split.

                                If you use a third party dependency, every bug in that dependency is a bug you are responsible for but can’t promise to resolve – something you’re not in the right position to identify, and don’t have permission to fix. That’s a really ethically dicey proposition. Your users have no choice but to trust you to some degree (the best they can do is trust you or the most trustworthy of your peers – and often, that is no choice at all), and you have chosen to expose them to a risk that you know that you may be unable to protect them against. Most of the time, you’ve done this without it being truly necessary: you could take full responsibility for that functionality, but it’s boring, or you’re on a deadline that makes it impossible (in which case your project manager is morally culpable).

                                Often, all that it takes to move from “I can’t possibly take responsibility for a hundred thousand lines of complex logic in a third party library” to “I can keep an accurate model of every operation this code performs in my head” is replacing large third party dependencies whose features you don’t need with a simpler hand-rolled implementation that has none of those features.

                                1. 4

                                  I find it amusing that you claim that RSS is “webtech mentality” and yet unironically advocate the use of JSON, which is just as much, if not more, “webtech mentality” than RSS. And JSON probably has more corner cases than RSS.

                                  I’m not sure what you mean by “TSV with quoting”—as long as the values don’t contain the tab character, there’s not quoting required, unlike with CSV. I do wish the ASCII separator characters were used, but as Loup-Vailant said:

                                  it was mostly short term convenience: since basically forever, we had this thing we call a text editor, that displays ASCII text. So if your format is based on that, it’s easy to debug input and output by just displaying the text in an editor, or even modifying it manually. It is then very tempting to optimise the text for human readability over a standard text editor… next thing you know, you’re using text for everything.

                                  1. 3

                                    I find it amusing that you claim that RSS is “webtech mentality” and yet unironically advocate the use of JSON, which is just as much, if not more, “webtech mentality” than RSS. And JSON probably has more corner cases than RSS.

                                    I use the term “webtech mentality” to mean a really specific thing here: the idea (produced in part by postel’s law) that interoperability can be guaranteed without simple and consistent design, because the onus of properly interpreting even impossibly-complex or internally-inconsistent specs is upon the individual implementor. This is like the tech equivalent of neoliberalism, and has very similar results.

                                    I advocate JSON over XML as the container language for something like RSS because it has fewer practically-meaningful corner cases for this purpose. I’d advocate for MSGPACK over JSON in basically any case where JSON is appropriate. In many cases where JSON is used, it would make more sense to have a line-based or tabular format (though if you are working inside a web browser or are interacting with code running in a web browser, your choices are seriously limited and it’s not really possible to do anything in a completely sensible way).

                                    I’m not sure what you mean by “TSV with quoting”—as long as the values don’t contain the tab character, there’s not quoting required

                                    You just answered your own question :)

                          1. 3

                            This is a good description of the bare basics of a build system. Where things get messy, though — even in simple projects — is when the source files have dependencies on each other, which are described within those files. In C terms, when a .c or .h file #includes .h files. Then changing a .h file requires recompiling all the .c files transitively dependent upon it.

                            No problem, make can do that! Except (unless make has changed a lot since I last used it) those dependencies have to be described explicitly in the makefile. Now you’ve got a very nasty case of repeating yourself: it’s so easy and common to add an #include to a source file during development. But if you ever forget to add the equivalent dependency to the makefile, you’ve broken your build. And it can break in really nefarious ways that only manifest as runtime errors or crashes that are extremely hard to debug. This in turn leads to voodoo behaviors like “I dunno why it crashed, lets delete all the .o files and build from scratch and hope it goes away.”

                            So now you need a tool that scans your source files and discovers dependencies and updates your makefile. This is why CMake exists, basically. But it add more complexity. This is a big part of why C/C++ are such a mess.

                            (Or you could just use an IDE, of course. Frankly the nay reason I have to deal with crap like makefiles is because not everyone who uses my code has Xcode…)

                            1. 6

                              None of this is necessary. It’s perfectly normal in make-based C/C++ projects to have a build rule which uses the compiler to generate the dependencies during the first build & then include those build dependencies into the Makefile for subsequent incremental builds.

                              There’s no need to keep track of the dependencies for C/C++ files by hand.

                              (For reasons which are not entirely clear to me, Google’s Bazel does not appear to do this. Meson does though, if you want a nice modern build tool.)

                              1. 2

                                Maybe recursive make is where it breaks down. I have fond memories of hardcoding dependencies between libraries in the top level makefile – an activity reserved for special occations when someone had tracked down an obscure stale rebuild issue.

                                (I think recursive make, at least done the obvious top-down way, is flawed.)

                                1. 1

                                  Yeah, you never want to be calling make from within make.

                                2. 2

                                  I imagine the reason is that Bazel requires a static dependency graph, including for all autogenerated intermediate files. I’m not sure why the graph is encoded directly in files instead of maintained in a parallel index though.

                                  There’s internal tooling at Google to automatically update dependencies in BUILD files from source files, but it’s apparently not open sourced.

                                3. 4

                                  You can’t add dependencies on the fly in Make, unfortunately. You can get a list of dependencies of a file in Makefile format in with gcc using -MD and -MF, but that complicates things a lot. Ninja on the other hand has native support for these rules, but from what I’ve heard Ninja is mostly made to be used by higher-level build tools rather than directly. (I mean you can manually write your ninja file and use ninja just like that, but it’s not as pleasant to write and read as Makefiles.)

                                  1. 5

                                    from what I’ve heard Ninja is mostly made to be used by higher-level build tools rather than directly. (I mean you can manually write your ninja file and use ninja just like that, but it’s not as pleasant to write and read as Makefiles.)

                                    That’s an explicit design goal of Ninja. Make is not a good language to write by hand, but it’s just good enough that people do it. Ninja follows the UNIX philosophy. It does one thing: it checks dependencies and runs commands very, very quickly. It is intended to be the target for higher-level languages and by removing the requirement from the high-level languages that they have to be able to run the build quickly, you can more easily optimise them for usability.

                                    Unfortunately, the best tool for generating Ninja files is CMake, whose main selling point is that it’s not as bad as autoconf. It’s still a string-based macro processor pretending to be a programming language though. I keep wishing someone would pick up Jon Anderson’s Fabriquer (a strongly typed language where actions, files and lists are first-class types, with a module system for composition, intended for generating Ninja files) and finish it.

                                    1. 1

                                      CMake, whose main selling point is that it’s not as bad as autoconf. It’s still a string-based macro processor pretending to be a programming language though.

                                      It’s kind of amazing how wretched a programming language someone can create, when they don’t realize ahead of time that they’re creating a programming language. “It’s just a {configuration file / build system / Personal Home Page templater}” … and then a few revisions later it’s metastasized into a Turing-complete Frankenstein. Sigh. CMake would be so much better if it were, say, a Python package instead of a language.

                                      I recall Ierusalemchy saying that Lua was created in part to counter this, with a syntax simple enough to use for a static config file, but able to contain logic using a syntax that was well thought-out in advance.

                                      1. 1

                                        The best tool for generating Ninja files is Meson :P

                                        Admittedly not the most flexible one, if you have very fancy auto-generators and other very unusual parts of the build you might struggle to integrate them, but for any typical unixy project Meson is an absolute no-brainer. It’s the new de-facto standard among all the unix desktop infrastructure at least.

                                        1. 1

                                          I’ve not used Meson, but it appears to have a dependency on Python, which is a deal breaker for me in a build system.

                                      2. 5

                                        You can’t add dependencies on the fly in Make, unfortunately.

                                        The usual way to handle this is to write the Makefile to -include $DEPFILES or something similar, and generate all of the dependency make fragments (stored in DEPFILES, of course) with the -MMD/-MF commands on the initial compile.

                                        1. 2

                                          You can definitely do this, here’s an excerpt from one of my makefiles:

                                          build/%.o: src/%.c | $(BUILDDIR)
                                          	$(CC) -o "$@" -c "$<" $(CFLAGS) -MMD -MP -MF $(@:%.o=%.d)
                                          
                                          -include $(OBJFILES:%.o=%.d)
                                          

                                          Not the most optimal solution, but it definitely works! Just need to ensure you output to the right file, wouldn’t call it particularly complicated, it’s a two line change.

                                          1. 2

                                            You didn’t mention the reason this truly works, which is that if there is a rule for a file the Makefile includes, Make is clever enough to check the dependencies for that rule and rebuild the file as needed before including it! That means your dynamically generated dependencies are always up to date – you don’t have a two-step process of running Make to update the generated dependencies and then re-running it to build the project, you can just run it to build the project and Make will perform both steps if both are needed.

                                      1. 10

                                        I’ve never used Fossil, though it seems like an interesting project. But, serious question, isn’t there a risk in providing “all the things” given that not all of those things are likely to be the best available? I’ve seen this mentioned as something SourceForge did wrong by providing a huge assortment of (mediocre) project hosting tools.

                                        1. 15

                                          Fossil is primarily developed for SQLite, so they add whatever they need. In that sense, it doesn’t worry too much about adding too much or too little.

                                          One difference between Fossil and Sourceforge will be that the former is a tool and the latter a service. I am more independent Fossil, not to break the UI, to insert ads, or to be unavaliable when I have no network connection.

                                          1. 9

                                            Indeed. Also, Fossil doesn’t provide “all the things” so much as building blocks for all the things. Fossil provides defaults, but prescribes nothing. The settings for ticketing system, for example, include the ability to completely rewrite the SQLite schema provided you include the documented fields required for base functionality. The ticket pages themselves are TH1 scripts (a subset of Tcl), which can be completely overwritten by the administrator. Reports are SQL scripts entered directly in the web UI. You can add any kinds of fields you want. You can add JavaScript if you feel like it.

                                            By default, the ticket system is a simple bug tracker. With a little effort you could turn it into a project management system like Jira, or a customer support page like Zendesk. Fossil does all the heavy lifting for you. Of course these systems would be simple, minimal clones of their counterparts—that’s part of the allure. You can build exactly what you need and only what you need with a few hours of work. Who really uses every feature of Jira?

                                            I feel like people underestimate the time investment for setting up and configuring systems like Jira or Zendesk, and overestimate how long it takes to build something bespoke. Especially when using an easily extensible core like Fossil.

                                            1. 3

                                              By default, the ticket system is a simple bug tracker. With a little effort you could turn it into a project management system like Jira, or a customer support page like Zendesk. Fossil does all the heavy lifting for you.

                                              We use Fossil at my place of work for some things and for a while it was also the bug tracking system for a small group of (related) teams. It still has some information in it for some bugs, but at this point most everything has been migrated to Jira. This is probably because Fossil’s bug tracking is wholly inadequate for large scale projects with lots of repositories and bugs or features that affect multiple repositories. To get that to happen would involve enough customization that using an existing system is almost certainly less effort.

                                              1. 2

                                                For sure. I don’t mean to say that customizing Fossil could replace every use case. I more meant simple and minimal use cases where you don’t really need the full power of bigger tools. Was Fossil inadequate before you grew in scale?

                                          2. 7

                                            I run a local instance of Fossil wiki and store my work notes there. These are usually well formatted notes that I could use to quickly refer something, or some code snippet I could quickly copy and send over to a co-worker, but I mostly use it to jog my memory. After about a year, it’s worked well enough, and I’m happy with it.

                                            So far, I have not found a need for the ticketing system or forum as it would be rather lonely, but after playing around with it a little bit, felt that it could be used to manage projects and communication in a lightweight manner, and I will seriously consider using it, when the need arises.

                                            Say you’re running a community VPS, and hosting some projects, you could use something like Gitea, and that has an issue tracking system, but at some point you’ll need a wiki, a forum for which you’ll need to setup and configure other pieces of software. Running Fossil as an integrated code hosting service, wiki, forum, and now Chat might fit the bill for some.

                                            The only downside I see is that you’ll need to learn how to use Fossil-the-VCS, which is kinda disappointing because I’d imagine many of us have invested a rather large amount of time learning to use Git (tig, magit etc.) well, but if one can get over that, I’d say it’s worth a shot!

                                            1. 3

                                              That’s a good point. The major difference, I suppose, is self-hosting. Another reply mentioned this as well. Certainly, in that light, I can see how all-inclusivity is desirable.

                                            2. 5

                                              Mediocre is relative. For a while Sourceforge was best in class for project hosting. I don’t think the assortment of features was their problem. I think it was more due to falling asleep at the wheel when first Google’s Project Hosting and then Github happened.

                                              Add to that the fact that there was a brief stint where later owners were trying to extract money by dubious means and sourceforge is a case study in plain old bad management.

                                              1. 3

                                                They were far and away the best zero-cost hosting, for a very long time. They had custom sites with custom domains, mailing lists, shell access, very robust hosting of downloads on a network of mirrors (I don’t think any competing forge matches that today, honestly), web based support forums, issue tracking and a small pile of ancillary services. Paying someone to host what they did for a medium sized project would’ve been beyond my means in the early 2000s.

                                                At Sourceforge, the process of applying for a project used to be a bit heavy. You had to spend a paragraph or two describing your plans and explaining why your project was a good use of their resources. Then it took a few days (and sometimes a bit of email back and forth) to get it approved. When Google Code, Code Plex, GitHub and BitBucket came along and the process of starting a project on one of those was just a few clicks -> instant repository, Sourceforge didn’t catch up to that for a long time, and that kind of explains why they were on the decline before the unfortunate series of management changes.

                                                Then one of the many “new management” groups started allowing really shitty ads and even “monetizing” installers for popular tools. They’ve stopped that, and I think they’re generally OK now, but nothing there today makes me feel like that’s the place to start something new.

                                              2. 1

                                                Generally one of fossil’s main benefits is that it’s “all inclusive”. I’m a bit worried about ephemeral chat though - like email, it’s hard to imagine a team without access to some form of non-fossil chat - so I’m not sure I see the value.

                                              1. 9

                                                While I can’t tell you precisely why this happens, I can tell you when. (Assume t.c contains the code in your post.)

                                                arm-none-eabi-gcc -Os -mcpu=cortex-m4 -mfpu=fpv4-sp-d16 \
                                                   -mfloat-abi=hard -mthumb \
                                                   -fdump-tree-copyprop2 -fdump-tree-isolate-paths \
                                                   -S -o t.s t.c
                                                

                                                You should get two files of extra output: t.c.121t.copyprop2 and t.c.122t.isolate-paths. The relevant content is the main function.

                                                t.c.121t.copyprop2

                                                main ()
                                                {
                                                  uint32_t c;
                                                
                                                  <bb 2> [local count: 1073741824]:
                                                  c_3 = 0 / 0;
                                                  printf ("c: %u\n", c_3);
                                                  __builtin_puts (&"hello world"[0]);
                                                  return 0;
                                                }
                                                

                                                t.c.122t.isolate-paths

                                                main ()
                                                {
                                                  uint32_t c;
                                                
                                                  <bb 2> [local count: 1073741824]:
                                                  __builtin_trap ();
                                                }
                                                

                                                So this is happening in the isolate-paths pass. The comment at the start of this pass provides some relevant information.

                                                /* Search the function for statements which, if executed, would cause
                                                   the program to fault such as a dereference of a NULL pointer.
                                                
                                                   Such a program can't be valid if such a statement was to execute
                                                   according to ISO standards.
                                                
                                                   We detect explicit NULL pointer dereferences as well as those implied
                                                   by a PHI argument having a NULL value which unconditionally flows into
                                                   a dereference in the same block as the PHI.
                                                
                                                   In the former case we replace the offending statement with an
                                                   unconditional trap and eliminate the outgoing edges from the statement's
                                                   basic block.  This may expose secondary optimization opportunities.
                                                
                                                   In the latter case, we isolate the path(s) with the NULL PHI
                                                   feeding the dereference.  We can then replace the offending statement
                                                   and eliminate the outgoing edges in the duplicate.  Again, this may
                                                   expose secondary optimization opportunities.
                                                
                                                   A warning for both cases may be advisable as well.
                                                
                                                   Other statically detectable violations of the ISO standard could be
                                                   handled in a similar way, such as out-of-bounds array indexing.  */
                                                

                                                There is acknowledgement that a warning may be a good idea, but there isn’t one, for whatever reason that is unclear.

                                                1. 4

                                                  What may be the most interesting part of this post is the fact that the author modified GCC to run with only a simple register allocator and combiner. Using these two backend passes it was able to achieve between 70 and 80 percent the performance of GCC at -O2, which runs many more passes.

                                                  1. 25

                                                    I’m glad someone is taking a look at linkers, since compilers get most of the attention. Linking is a huge bottleneck, rarely parallelized, and massively complicated.

                                                    I sometimes wonder is linking itself is even something worthwhile to have anymore, and that systems could switch to something like either merging compiling and linking into a single step, or having directly runnable objects that can be trivially fixed up.

                                                    1. 33

                                                      LLD is getting a lot of attention. It replaced the multi-pass algorithm that traditional UNIX linkers use with a single-pass one that is O(n) in terms of the number of symbols, rather than O(n*m) in terms of the number of symbols and number of object files. In exchange for this, it uses more memory, but the memory for the sum of all of the symbol tables in all object files in a large project is still pretty small compared to a modern machine’s RAM. It’s also aggressively parallelised. As the linked repo says, LLD takes 12 seconds to link Chromium, gold (which was a rewrite of BFD ld focused on speed) takes 50.

                                                      It seems that the main speed win here is actually an accounting trick: it starts the linking process before compilation starts and so only counts the time between the last compile step finishing and the linker finishing. Because most of what the linker is doing does not depend on every object file, you can get most of the way through the link before you need all of the object files. This probably doesn’t get as much speedup with things like --gc-sections, which require having a complete symbol table so that you can do reachability analysis before you can do the final layout, but even then you can start copying things that are definitely reachable early, you just might end up with something in the final .o that requires you to pull in a load of things from other objects.

                                                      Note that the author of this repo is one of the lead developers on LLD. Rui is awesome, after a 5 minute conversation with him I completely reevaluated how I thought about linkers. I hope that this is an experiment that he’s doing to prototype things that he’ll eventually add to LLD.

                                                      As you your other suggestions:

                                                      merging compiling and linking into a single step

                                                      That’s what languages that do whole-program analysis (Go, Rust) do, and is also almost what LTO does. There’s a trade-off here in how much you can parallelise. If you completely combine final code-generation and linking then the compiler needs to know the locations of everything else in the object file. That’s not feasible because of circular dependencies and so you end up needing something like relocations and a final fixup step, which is doing part of what a linker does.

                                                      The Go (Plan9) toolchain actually does this slightly differently to other things. It combines the assembler and linker so the compiler emits pseudo instructions for indirect references and the assembler / linker expands them into the shortest instruction sequence that can express the displacement that’s actually needed. This is a big win on a lot of architectures: if you can materialise a 16-bit constant in one instruction and a 32-bit one in two, your linker / assembler can do some constraint solving, try to place functions that call each other close together, and insert the single instruction in the common case and the two-instruction sequence for the places where it’s needed. Without this merging, you end up doing one of two things:

                                                      • Use the short form and require the linker to insert thunks: jump with a short displacement and if the target is too far away, insert a trampoline somewhere close that has a three-instruction sequence, burning some branch predictor state and adding overhead when you need to do this.
                                                      • Always emit the long sequence and often have a no-op instruction that writes 0 into a register that’s already 0.

                                                      C++ places a lot more demands on the linker. The modern C++ compilation model[1] generates every used template instantiation in every compilation unit that uses it, puts them in COMDAT sections, and relies on the linker throwing them away. This means that the generated instantiations are available for analysis in every module, but it also means that the compiler does a lot of redundant work. Modules and ThinLTO are intended to address this: most template instantiations will have a single home but can be pulled into other compilation units for inlining if it would help.

                                                      having directly runnable objects that can be trivially fixed up.

                                                      We call this ‘dynamic linking’. It is slower, because there’s still a linker, it just happens every time you run the program. Back when I used BFD ld, I always build LLVM as shared libraries for my iterative compile cycle, because a debug build of clang took 5 minutes to link. I’d always do a statically linked build before I ran the test suite though, because the linking time (even with BFD ld) was less than the difference in time running the test suite (which invokes clang, opt, llc, and so on hundreds of times).

                                                      There’s always a tradeoff here. Code that is faster to link is slower to run. You can make code fast to link by adding a small indirection layer, putting everything in a single big lookup table, not doing any inline fixups and making any access to a global symbol go via that table. You can make it fast to run by aggressively generating the shortest instruction sequence that materialises the exact address. Dynamic linking generally doesn’t want to modify executable code for two reasons: it’s bad for security to have W&X code and it prevents sharing[2]. This means that you end up with indirection layers. Look up ‘copy relocations’ some time for the extreme case of weirdness ere.

                                                      [1] The original 4Front C++ compiler parsed linker error messages for missing symbols and generated the template instantiations once, on demand, in an iterative cycle.

                                                      [2] 32-bit Windows DLLs actually did something differently here: DLLs were statically relocated and expected to run at a fixed address. There was a fun constraint solver that tried to find a place in the address space for all DLLs that works with all installed EXEs. If it failed, the DLL would be statically relocated on load and would not be shared.

                                                      1. 6

                                                        but the memory for the sum of all of the symbol tables in all object files in a large project is still pretty small compared to a modern machine’s RAM.

                                                        … no? I frequently experience OOMs when linking big C++ projects. With medium-scale programs, linking has to run with just one process at a time to not get OOM killed (with my 16GB of RAM). With truly huge projects, such as Chromium (especially with debug symbols), I have to add immense amounts of SWAP space and just let the linker process thrash for a long time.

                                                        In my experience, linker memory usage is the main limiting factor with big C++ projects, and it’s one of the main reasons I’m moving to at least 32GB of RAM the next time I upgrade anything.

                                                        1. 1

                                                          What linker are you using? I haven’t had that since the bad old days of BFD LD. A debug build of LLVM generates about 40GiB of object files, but symbol tables are a tiny fraction of that. The vast majority of it is… object code. That doesn’t have to be resident for lld to work. Last time I checked, lld would mmap all of the object files, but that doesn’t consume any RAM unless you have spare RAM: if RAM is constrained then the OS can just evict the object files from memory.

                                                      2. 8

                                                        having directly runnable objects that can be trivially fixed up

                                                        How trivial can it get?

                                                        The fundamental problem that linkers solve is that if you do incremental compilation, disparate modules need to be able to call into each other. ‘Linking’ just means resolving symbols into addresses; you need to do that anyway at some point. So I don’t disagree that it can get faster, but the essential complexity is still there.


                                                        The zig compiler has been making some cool strides there, incidentally. It sidesteps the issue entirely by doing binary patching. Common lisp does something similar, with its ability to redefine symbols in a running image.

                                                        1. 5

                                                          …that systems could switch to something like either merging compiling and linking into a single step…

                                                          That would likely require all the source and objects to be known at compile time, which probably means getting rid of dynamic libraries. A lot of the complications of linking comes from dynamic libs.

                                                          1. 4

                                                            The abstraction layer is already broken by LTO. Now the linker is an all-in-one compiler, too.

                                                            1. 14

                                                              In GCC, LTO is a “sort of” compiler. What happens is that GCC calls collect2 (its linker wrapper) that then calls ld with the plugin options to have it run lto-wrapper. lto-wrapper has all the objects taking part in the link. It then calls the compiler driver with special options to make it do whole program analysis. This phase then partitions the objects and runs GCC again on all those partitions, possibly using Make. Eventually all the objects that get created from this go back to the linker, which then links them.

                                                              It’s the Unix philosophy at work. And it’s murder to debug.

                                                              1. 6

                                                                The unix philosophy is about composable, reusable tools. Without reuse in different contexts, it’s just an inconvenient split of functionality.

                                                                How would you compose these tools differently to accomplish a different goal with them?

                                                              2. 5

                                                                This isn’t targeted at release builds, though, it’s targeted at development builds.

                                                                I don’t think anyone is using lto for development builds, even if they use regular optimizations.

                                                            1. 9

                                                              Linkers are long overdue for an upgrade. Not just speed leaves a lot to be desired, but the whole compiler-linker interaction is messy. Library include paths are disconnected from source includes and can get out of sync. Flags specified in a wrong order cause weird failures. The linker generally has no idea what the compiler was trying to do.

                                                              When I have an error in the source code, compiler will bend backwards to help and suggest a solution and point it with squiggly underlines. Meanwhile linker will just print “missing _Zzxc329z9xc in asdf.o, bye”.

                                                              1. 8

                                                                Library include paths are disconnected from source includes and can get out of sync

                                                                MSVC and clang have pragmas that you can put in your header to control what is linked.

                                                                Meanwhile linker will just print “missing _Zzxc329z9xc in asdf.o, bye”.

                                                                Every linker I’ve used in the last decade will print an error with a demangled C++ name and tell me which function referenced the symbol that was not found:

                                                                $ cat link.cc
                                                                int x(int);
                                                                
                                                                int main()
                                                                {
                                                                        x(5);
                                                                }
                                                                $ clang++ link.cc
                                                                /usr/bin/ld: /tmp/link-ab4025.o: in function `main':
                                                                link.cc:(.text+0xa): undefined reference to `x(int)'
                                                                clang: error: linker command failed with exit code 1 (use -v to see invocation)
                                                                
                                                                1. 8

                                                                  It’s perhaps underappreciated how much the compiler driver (not the compiler proper) knows about calling the linker and how much it does for you. If you run gcc -dumpspecs you’ll likely see the *link_command directive at the end – and it’s probably an impenetrable mess. The compiler puts all the startup, system libraries, and finalization objects in the correct order (notwithstanding user libs) and also deals with the rather obscure options that are required depending on how you compiled the code.

                                                                  If you had to link things yourself, you’d quickly find it’s very difficult. It requires a fair bit of knowledge of the C library you’re using and the system itself. Compiler drivers handling this for you is very helpful, with the caveat that there ends up being a lot of abstraction making it difficult to decipher what is going on (cough collect2 cough).

                                                                  1. 7

                                                                    Yea, that part is annoying. When writing either makefiles or build systems, it would’ve been nice if you had one kind of rule to compile C files, one kind of rule to compile C++ files, one kind of rule to compile fortran files, and one kind of rule to link object files into an executable. But due to the immense mess of linker options, you can’t just do that. You want to link with the C compiler if your object files are only from C files, link with the C++ compiler if your object files are from C++ files or a mix of C and C++, link with the fortran compiler if your object files are from fortran, and I don’t even know how you would do it if your project has C, C++ and fortran files.

                                                                    It’s not nice.

                                                                1. 3

                                                                  git doesn’t have actual command to un-stage a file(s), though. To get around this limitation…

                                                                  Limitation, or poor UI decision? I’m guessing the latter.

                                                                  1. 10

                                                                    newer versions of git have git restore so I think that counts

                                                                    1. 5

                                                                      git reset -- filename or git reset HEAD filename do the same, tho, right? And that’s been in git for ages.

                                                                      1. 5

                                                                        I know, just wanted to say there is now an actual command. The article claimed there wasn’t one.

                                                                        1. 1

                                                                          Sometimes. If the file is already in HEAD then this works, but if it’s a newly created file I don’t think this works.

                                                                          1. 2

                                                                            It definitely works with newly created files.

                                                                      2. 4

                                                                        The naming problem. There is, and always have been, git reset that does what OP wanted, however the “feeling” that this one command does “a lot of different things” (reset staging and reset working tree, depending on the flags) is what made people say it doesn’t have such command.

                                                                        1. 3

                                                                          I use tig which makes unstaging single files easy and natural, among other things

                                                                        1. 7

                                                                          I think this is the biggest advancement in Rust so far. I’ve been concerned about laying a finger on Rust for a long time because I have a lot of concern about Mozilla and it’s ethics. After the Brendan Eich ordeal it was difficult to rationalize any kind of investment in a company that behaved that way.

                                                                          Bringing Rust into the realm of the Software Foundation, I can say I’ll be following Rust with a renewed interest. Zig beat them to it, and I’d gladly consider a new project in Zig before I chose Rust, for obvious reasons, as I’m sure we’re all pretty sick of hearing about “what Rust can (kinda) do”, but all the same, these are some big names getting behind the project. That much can’t be ignored.

                                                                          2021 is shaping up to be a fast-moving year in PLs and PLT research as well.

                                                                          1. 15

                                                                            as I’m sure we’re all pretty sick of hearing about “what Rust can (kinda) do”

                                                                            At this point I hear more people saying this than I do people actually evangelizing Rust.

                                                                            1. 23

                                                                              Brendan Eich did something that made it hard to believe he’d be fair and welcoming in a global project that extremely heavily depends on public goodwill and participation of volunteers.

                                                                              (And he continues to make controversial, and frankly dangerous and stupid, public statements today. He denies that face masks work during a global pandemic, and actively discourages people from listening to public health experts, for example.)

                                                                              His job was to be the face of the company. People freely chose not to associate with a company who chose someone controversial as their face. Enough people made this free choice that it seemed wise to pick someone else.

                                                                              I never understood why this was so terrible. What is the alternative? Force people to use products and services they don’t want to use? Forbid people from making purchasing decisions based on how they feel about employees at a company?

                                                                              1. 12

                                                                                Enough people made this free choice that it seemed wise to pick someone else.

                                                                                I never understood why this was so terrible.

                                                                                TBH I assumed the bad behavior referred to was that they kept a shitbag like Eich around as long as they did.

                                                                                1. 6

                                                                                  A diverse opinion being rejected in a group inherently portrays that group as exclusive in nature. Bubbling themselves in has alienated a lot of possibilities. Look at recent cuts Mozilla has to make, look at FF’s market share in the browser realm. I see W3 reporting FF as lower than 10% these days.

                                                                                  I don’t know about his opinions on these things, I’m not really trying to open a discussion about Eich, I’m not his follower, I am just presenting the novel idea that booting people for their personal opinions leads to failure and echo chambers and whatever else.

                                                                                  His JOB was co-founder. He CO-founded Mozilla. That’s different than being hired as CEO “here, go be the public figure, soak up those headline bullets and shut up on socials”.

                                                                                  Anyhow, I’m not a Mozilla history expert. I don’t think it’ll be relevant in 20 years, afaict it’s already dead.

                                                                                  Rust however, need not die, insofar as enough resources are dedicated to its longevity factor. Rust needs major reworking to be able to integrate new compiler core team, there’s major work needed to improve syntax issues, there’s HUGE work needed to do something about the compile times. I’ve seen users describe it as “not a better C, but a better C++” and I think that’s a decent designation for it for the time being. Still, without major pivoting and better resource and labor allocation, the project is in big trouble. I see people working on Rust who tweet 50-70+ a day. How productive can they really be???

                                                                                  It’s whatever. I really like the idea of a software foundation. It’s definitely going to be helpful to have diverse minds in diverse fields bouncing ideas around. It’s great.

                                                                                  1. 22

                                                                                    May I refer you to the Paradox of tolerance? Groups that want to maximize diversity must exclude those who are against diversity.

                                                                                    Eich gave public material support to Prop 8. He could have pretended he doesn’t support it, he could have “shut up on socials”, but he chose not to.

                                                                                    1. 16

                                                                                      I remember when he was debating this on Twitter. His response was that people not wanting to work with him because of his support of Prop 8 (which would make same-sex marriage illegal) was “fascism”.

                                                                                      Of course, he said this to people who were freely choosing to not associate with him based on their own opinions and his public statements…while he himself was supporting expanding government power to limit the kinds of associations consenting adults could participate in.

                                                                                      One of those is was way more “fascist” than the other.

                                                                                    2. 14

                                                                                      A diverse opinion

                                                                                      This characterization is both insufficient and inaccurate.

                                                                                      1. 12

                                                                                        His JOB was co-founder. He CO-founded Mozilla. That’s different than being hired as CEO “here, go be the public figure, soak up those headline bullets and shut up on socials”.

                                                                                        No one complained about him until they hired him to be the CEO. I didn’t even know his name before that and I bet a lot of other people are in the same boat. You seem really offended by something but you don’t seem to know what it is…

                                                                                        1. 7

                                                                                          Still, without major pivoting and better resource and labor allocation, the project is in big trouble

                                                                                          You would know better than I would, but this is honestly the first time I’ve ever heard anything other than “Rust is the language of the future and there’s no need to learn anything else ever.” I’m being only slightly facetious.

                                                                                          Seriously, though, from a mostly-outsider’s perspective, it seems like Rust is going nowhere but up and seems to be poised to take over the world. I suppose there’s a difference between Rust-the-language and Rust-the-project, but they’re pretty much identical to me.

                                                                                          1. 6

                                                                                            I see people working on Rust who tweet 50-70+ a day. How productive can they really be???

                                                                                            This is patently ridiculous as an argument.

                                                                                        2. 12

                                                                                          Mozilla did not own or control Rust at any point. The Rust project started out managed by Graydon Hoare in 2006, and Mozilla began financially supporting it in 2009 (link to the history). Mozilla did own the trademarks for the Rust and Cargo names and logos, which were controlled and licensed in an open manner, and protected only to the degree necessary to avoid implied official status or endorsement by the Rust project (link to the policy). Mozilla also paid for the salaries of developers who worked on Servo, for some period one of the two largest Rust projects (the other being the Rust compiler itself), as well as the salaries of some folks who worked on the Rust compiler. However, Mozilla did not exercise or influence the direction of the Rust language, and from an early period a majority of Rust’s contributors, including members of the Core Team and other Rust Teams, did not work for Mozilla.

                                                                                          1. 4

                                                                                            what Rust can (kinda) do

                                                                                            I’m curious what this bit refers to

                                                                                            1. 1

                                                                                              This could refer to many different parts of a immature ecosystem like GUI programming

                                                                                          1. 20

                                                                                            Most candidates cannot solve this interview problem:

                                                                                            Input: “aaaabbbcca”

                                                                                            Output: [(“a”, 4), (“b”, 3), (“c”, 2), (“a”, 1)]

                                                                                            Write a function that converts the input to the output I ask it in the screening interview and give it 25 minutes.

                                                                                            How would you solve it?

                                                                                            I still think this is the best answer to the question.

                                                                                            def func(x):
                                                                                                return [("a", 4), ("b", 3), ("c", 2), ("a", 1)]
                                                                                            
                                                                                            1. 2

                                                                                              Without a close examination, I’d wager most compiler improvements in the past ten years are about keeping up with language standards, adding more support features, and fixing bugs. Probably the most noteworthy changes have been the various sanitizers and better warning/error messages, neither of which are very simple things to do.

                                                                                              1. 6

                                                                                                To generate a GUID, you can use the Online GUID Generator website

                                                                                                Perhaps using the New-Guid cmdlet in Powershell would be simpler.

                                                                                                1. 1

                                                                                                  Oof. You saved me from reading this through to the end…