1. 2

    This seems like spam to me?

    They’ve copy pasted the advisory and put a very stupid title on it. “Ugly, perfect ten-rated bug”, “patch before they’re utterly p0wned”, “btw it’s a denial of service attack”.

    No thanks.

    1. 2

      It’s The Register; the self-admitted British tabloid (like the Daily Mail) of the IT world. Sometimes they can produce a good article, othertimes it’s clickbait where you’re also expecting page 3 to be a naked woman.

      1. 1

        This vulnerability can “allow the attacker to execute arbitrary code and obtain full control of the system,” it’s not just a DoS.

        https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180129-asa1

      1. 4

        On my PC at least, the process name has parens around it, so you can parse it with the Lua pattern %b(). Fishing out the parent PID would be like %S+ %b() %S+ (%S+).

        I don’t think the problem here is that /proc/pid/stat is broken, rather the string parsing tools in libc are not good enough and shouldn’t be used.

        The OpenBSD guys already ripped the patterns code out of Lua for their web server (patterns.c/.h), so it’s simple to add it to your own code base.

        1. 9

          Assuming the rules for process names is the same as the rules for file names, a process name can contain unbalanced parens. What if a process is named for example hello) world? The string in /proc//stat would be (hello) world), and %b() would just find (hello). This doesn’t seem like something which could be fixed by smarter parsing, other than by reading from the end.

          1. 2

            Oh good point, %((.*)%) works, assuming they don’t add parens to any other fields.

          2. 2

            As noted in the original post, even using the parentheses does not guard against this. The only thing I can think of to safely use the current format is scanning the whole file into a buffer and searching for the last closing parenthesis, then taking everything from the first opening parenthesis to the last as executable name.

            This is also not specific to C or libc, this format is bad to parse (and the mistake easy to make) with any language.

          1. 3

            This looks racy to me, can someone explain where I’m going wrong?

            Thread A is the first to acquire the benephore, checks and increments the atomic variable, finds it is zero and proceeds without acquiring the semaphore.

            While Thread A is in the critical section Thread B arrives and acquires the benephore, finds the atomic variable to be non-zero so acquires the semaphore. Because it has the semaphore it proceeds into the critical section. Now there are two threads concurrently in the critical section.

            What prevents this scenario?

            1. 3

              I think you’re right, unless I’m missing something obvious.

              Worse still, if T1 enters the critical section and is followed by T2, if T1 now makes progress it will find benaphore_atom > 1 and call release_sem() on a semaphore it doesn’t hold. Which is probably either a crash or UB, who knows.

              I was missing something obvious.

              The semaphore, is initialized into an ‘unavailable’ state.

              When Thread B attempts to acquire the newly initialized semaphore, it blocks as the semaphore is in its ‘unavailable’ state. Thread A later finishes up in its Critical Section, and seeing that benaphore_atom > 1 it increments the semaphore, allowing Thread B to make progress.

              At the end of this execution, T2 sees !(benaphore_atom > 1) and continues without marking the semaphore as available.

              1. 1

                Semaphores don’t provide mutual exclusion.

                You use them to e.g. count the number of items in a queue and wake up enough threads to process them. Then those threads use a separate mutex to synchronise any operations on the queue.

              1. 5

                If you’re benchmarking a really small piece of code in a loop, you also need to be aware that

                1. you can spend 50% of the time per iteration on loop overhead if you don’t unroll and
                2. CPUs execute multiple instructions in parallel, so if you aren’t careful measuring the time to do 1000 iterations and dividing by 1000 will underestimate the time to run one lone iteration (throughput vs latency)
                1. 6

                  I’m starting work on proper FPS networking for my game. A lot of the concepts have clicked in my head lately so I feel ready to work on it, but it’s so crazy hard. There’s tons of tricky stuff I need to implement and when I sit down to start on it I just have no idea what to type. It feels like I’m programming for the first time again.

                  So for the moment I’m doing mostly random exploration and hoping I find something reasonable. Wish me luck!

                  1. 30

                    All of them:

                    The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                    All of them:

                    The fact that they waste so much effort on incremental builds when the compilers should really be so fast that you don’t need them. You should never have to make clean because it miscompiled, and the easiest way to achieve that is to build everything every time. But our compilers are way too slow for that.

                    Virtually all of them:

                    The build systems that do incremental builds almost universally get them wrong.

                    If I start on branch A, check out branch B, then switch back to branch A, none of my files have changed, so none of them should be rebuilt. Most build systems look at file modified times and rebuild half the codebase at this point.

                    Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                    Virtually all of them:

                    Making me write a build spec in something that isn’t a normal good programming language. The build logic for my game looks like this:

                    if we're on Windows, build the server and all the libraries it needs
                    if we're on OpenBSD, don't build anything else
                    build the game and all the libraries it needs
                    if this is a release build, exit
                    build experimental binaries and the asset compiler
                    if this PC has the release signing key, build the sign tool
                    

                    with debug/asan/optdebug/release builds all going in separate folders. Most build systems need insane contortions to express something like that, if they can do it at all,

                    My build system is a Lua script that outputs a Makefile (and could easily output a ninja/vcxproj/etc). The control flow looks exactly like what I just described.

                    1. 15

                      The fact that they exist at all. The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                      I disagree. Making the build system part of the language takes away too much flexibility. Consider the build systems in XCode, plain Makefiles, CMake, MSVC++, etc. Which one is the correct one to standardize on? None of them because they’re all targeting different use cases.

                      Keeping the build system separate also decouples it from the language, and allows projects using multiple languages to be built with a single build system. It also allows the build system to be swapped out for a better one.

                      Codebases easily fit in RAM …

                      Yours might, but many don’t and even if most do now, there’s a very good chance they didn’t when the projects started years and years ago.

                      Making me write a build spec in something that isn’t a normal good programming language.

                      It depends on what you mean by “normal good programming language”. Scons uses Python, and there’s nothing stopping you from using it. I personally don’t mind the syntax of Makefiles, but it really boils down to personal preference.

                      1. 2

                        Minor comment is that the codebase doesn’t need to fit into ram for you to hash it. You only need to store the current state of the hash function and can handle files X bytes at a time.

                      2. 14

                        When I looked at this thread, I promised myself “don’t talk about Nix” but here I am, talking about Nix.

                        Nix puts no effort in to incremental builds. In fact, it doesn’t support them at all! Nix uses the hashing mechanism you described and a not terrible language to describe build steps.

                        1. 11

                          The build spec should be part of the language, so you get a real programming language and anyone with a compiler can build any library.

                          I’m not sure if I would agree with this. Wouldn’t it just make compilers more complex, bigger and error prone (“anti-unix”, if one may)? I mean, in some cases I do appriciate it, like with go’s model of go build, go get, go fmt, … but I wouldn’t mind if I had to use a build system either. My main issue is the apparent nonstandard-ness between for example go’s build system and rust’s via cargo (it might be similar, I haven’t really ever used rust). I would want to be able to expect similar, if not the same structure, for the same commands, but this isn’t necessarily given if every compiler reimplements the same stuff all over again.

                          Who knows, maybe you’re right and the actual goal should be create a common compiler system, that interfaces to particular language definitions (isn’t LLVM something like this?), so that one can type compile prog.go, compile prog.c and compile prog.rs and know to expect the same structure. Would certainly make it easier to create new languages…

                          1. 2

                            I can’t say what the parent meant, but my thought is that a blessed way to lay things out and build should ship with the primary tooling for the language, but should be implemented and designed with extensibility/reusability in mind, so that you can build new tools on top of it.

                            The idea that compilation shouldn’t be a special snowflake process for each language is also good. It’s a big problem space, and there may well not be one solution that works for every language (compare javascript to just about anything else out there), but the amount of duplication is staggering.

                            1. 1

                              Considering how big compilers/stdlibs are already, adding a build system on top would not make that much of a difference.

                              The big win is that you can download any piece of software and build it, or download a library and just add it to your codebase. Compare with C/C++ where adding a library is often more difficult than writing the code yourself, because you have to figure out their (often insane) build system and integrate it with your own, or figure it out then ditch it and replace it with yours

                            2. 8

                              +1 to all of these, but especially the point about the annoyance of having to learn and use another, usually ad-hoc programming language, to define the build system. That’s the thing I dislike the most about things like CMake: anything even mildly complex ends up becoming a disaster of having to deal with the messy, poorly-documented CMake language.

                              1. 3

                                Incremental build support goes hand in hand with things like caching type information, extremely useful for IDE support.

                                I still think we can get way better at speeding up compilation times (even if there’s always the edge cases), but incremental builds are a decent target to making compilation a bit more durable in my opinion.

                                Function hashing is also just part of the story, since you have things like inlining in C and languages like Python allow for order-dependent behavior that goes beyond code equality. Though I really think we can do way better on this point.

                                A bit ironically, a sort of unified incremental build protocol would let compilers avoid incremental builds and allow for build systems to handle it instead.

                                1. 2

                                  I have been compiling Chromium a lot lately. That’s 77000 mostly C++ (and a few C) files. I can’t imagine going through all those files and hashing them would be fast. Recompiling everything any time anything changes would probably also be way too slow, even if Clang was fast and didn’t compile three files per second average.

                                  1. 4

                                    Hashing file contents should be disk-io-bound; a couple of seconds, at most.

                                    1. 3

                                      You could always do a hybrid approach: do the hash check only for files that have a more-recent modified timestamp.

                                    2. 1

                                      Do you use xmake or something else? It definitely has a lot of these if cascades.

                                      1. 1

                                        It’s a plain Lua script that does host detection and converts lines like bin( "asdf", { "obj1", "obj2", ... }, { "lib1", "lib2", ... } ) into make rules.

                                      2. 1

                                        Codebases easily fit in RAM and we have hash functions that can saturate memory bandwidth, just hash everything and use that figure out what needs rebuilding. Hash all the headers and source files, all the command line arguments, compiler binaries, everything. It takes less than 1 second.

                                        Unless your build system is a daemon, it’d have to traverse the entire tree and hash every relevant file on every build. Coming back to a non-trivial codebase after the kernel stopped caching files in your codebase will waste a lot of file reads, which are typically slow on an HDD. Assuming everything is on an SSD is questionable.

                                      1. 11

                                        This is quite disappointing, his solution for each point is basically “just get it right!”

                                        Here are some actual solutions for the problems he brings up:

                                        Not freeing memory after allocation:

                                        Try to avoid the “free everything individually” pattern where possible because it’s the easiest to get wrong. Use RAII if you like. Try to centralise your resource management higher up the call stack so leaf code does not call malloc/free, people coming from other languages tend to do that because it’s fine if you have a GC, but you can make big messes for yourself in C. Use easier allocation strategies like “free everything allocated after this point when this statement falls out of scope”, which works very well for temp scratch space allocations.

                                        All of your allocations should go through memory managers that check for leaks. The least intrusive way to do it is by keeping a hashtable that maps from the pointer to some info struct, which at least contains file and line of the allocation.

                                        Freeing already freed memory (double-freeing):

                                        Largely mitigated by what I said before. I have no specific suggestions for this because in practice I never have problems with it.

                                        Invalid memory access, either reading or writing:

                                        I don’t have any recommendations for NULL pointers, but stuffing asserts/NULL checks all over the place is a signal you have no idea what your code is doing. TBH it’s not really been a problem for me.

                                        For uninitialised data, making all your allocators call memset( 0 ) is a good start (try to make your classes have some valid default state when they’re all 0), beyond that it’s not really a problem because it’s pretty easy to catch. Use RAII if you like.

                                        Also lobby the C++ committee for something like #pragma primitive_types_initialise_to_zero_unless_you_tell_them_not_to.

                                        Buffer-overflow, either reading or writing:

                                        Implement sane primitives, such as:

                                        template< typename T >
                                        struct array {
                                            T * elems;
                                            size_t n;
                                            T & operator[]( size_t i ) { ASSERT( i < n ); return elems[ i ]; }
                                        };
                                        

                                        Replace any usage of pointer + size with that class. You will probably want StaticArray< size_t, T >, DynamicArray, array2d, etc too.

                                        You’ll also want a sane string class, a sane string parsing library (Lua’s patterns library is very very good), and a sane stream writer/parser class. Use these classes instead of C strings/pointers + sizes.

                                        1. 2

                                          As for your first solution, I would first check to see if the runtime can do the checks for you. I know that the GNU C library can do more checks if you set certain environment variables, and using valgrind is wonderful if you have access to it. Check first before writing your own memory wrapper (that is what lead to Heartbleed, by the way).

                                          Second, pick a better value to initialize memory with than just 0. If you really want to help with debugging on the x86 platform, I suggest using the value 0xCC. As a signed quantity, it’s a large negative value that is easy to spot. As an unsigned quantity, it’s a huge number. As a character, it’s invalid ASCII (but it is a valid UTF-8 initial byte character, but two 0xCC’s in a row is invalid UTF-8). As a pointer, it’s probably an invalid pointer (so you’ll get a SIGSEGV most likely) and if it’s accidentally executed, it’s the INT3 debugging instruction.

                                          I don’t really have any good solutions to string handling in C other than “don’t do that!” Which is why I use Lua if possible.

                                        1. 4

                                          Great writeup!

                                          What things didn’t work out? Any great screenshots of it just failing miserably? Any issues with normals in areas where you swap levels of detail?

                                          1. 4

                                            The main thing that didn’t work was trying to draw just the square tiles and nothing else. The failures from that aren’t very exciting, you end up with overlap on one side and a gap on the other as the inner clipmap levels move around.

                                            The funniest bugs I ran in to were all to do with the snapping. If you don’t snap the levels at all it looks like this, at one point I had a bug where I was snapping the texture coordinates but not the actual world position of the mesh so the world would tremble as the camera moved around, and my best bug had the entire terrain flapping up and down like mad. (don’t remember how I did that one sadly)

                                            Also lots of mundane garbage with mesh generation, like random triangles stretching across the whole world and holes in the ground.

                                            Normals are mostly fine across transitions, but if there’s high frequency detail in the terrain there can be quite a big pop.

                                          1. 16

                                            I really can’t feel bad for companies on this. They’ve demonstrated over and over again that they can’t be trusted to do the right thing on their own.

                                            I wish the United States could enact a law like this, but who am I kidding?

                                            1. 0

                                              I really can’t feel bad for companies on this. They’ve demonstrated over and over again that they can’t be trusted to do the right thing on their own.

                                              Even the one-man companies currently just starting out that are under threat of 20M EUR fines for not complying with rules that are practically impossible to comply with?

                                              You’re not seeing the big picture here. The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can.. It’s fucking ridiculous.

                                              1. 18

                                                Even the one-man companies currently just starting out that are under threat of 20M EUR fines for not complying with rules that are practically impossible to comply with?

                                                Especially those ones because otherwise they have no checks and balances whatsoever and the single person in charge will do whatever they feel like without telling anybody.

                                                We don’t let small restaurants ignore food safety, or small construction companies ignore building codes, why would we let small internet companies ignore privacy regulations?

                                                You’re not seeing the big picture here. The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can.. It’s fucking ridiculous.

                                                Just because the government is spying on us doesn’t mean we should allow corporations to do it too. We don’t have to solve both problems at the same time.

                                                1. -5

                                                  Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?

                                                  I mean.. seriously? I hope you’re trolling, for your sake.

                                                  Just because the government is spying on us doesn’t mean we should allow corporations to do it too.

                                                  Who’s “we”? -The faceless bureaucracy of the EU that’s hoisting this pile of garbage on productive people?

                                                  1. 7

                                                    I’ve downvoted you as troll because we don’t have anything like “unnecessarily rude”.

                                                    Please try to be less abrasive with your posts.

                                                    1. 5

                                                      Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?

                                                      Another way of putting this is “do you sincerely think that small companies need to be threatened with being put out of business entirely if they disregard their customers’ safety?” and my answer is yes, absolutely.

                                                      Restaurants operate perfectly well under there threat of “if you give a noticeable quantity of customers food poisoning even once, the FSA will permanently shut you down”.

                                                      It’s perfectly sensible to me that any other business capable of ruining a whole bunch of peoples’ lives should be held to the same standard.

                                                      1. 0

                                                        Another way of putting this is “do you sincerely think that small companies need to be threatened with being put out of business entirely if they disregard their customers’ safety?” and my answer is yes, absolutely.

                                                        You know you’re comparing apples to oranges, right? Someone’s e-mail address staying in a database somewhere “against his will” is not comparable to feeding people rotten/contaminated food or something.

                                                        Regardless, neither warrants a 20M EUR penalty, and in the first case, it’s just utterly insane. You also understand that just fine, so why are you arguing against me?

                                                        It’s as if EU regulators were completely oblivious to the existence of small businesses, which they’re actually not, of course, because they’re not that fucking retarded/insane.

                                                        So they’ve set MegaCorp level fines for everyone for some reason other than insanity/stupidity. But it’s not for no reason, and it’s not because it’s reasonable either.

                                                        And obvious truth is met with hostility, as always.

                                                        1. 2

                                                          you’re comparing apples to oranges

                                                          Privacy violations don’t happen at a rate of 1 or 2 per incident, they hit thousands of people at a time.

                                                          neither warrants a 20M EUR penalty

                                                          You won’t be paying 20M EUR. Your company will be accruing a 20M EUR debt that it will immediately fold under. You founded a limited liability company for this reason. The number could be 13.37B EUR and that outcome would be the same for small companies.

                                                          Folding your company is a perfectly fair outcome for you flagrantly violating other peoples’ privacy rights.

                                                          met with hostility

                                                          I didn’t say anything rude to you at all. You asked a question, “Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?” and I gave a completely polite answer. You just read it as hostile because you don’t like that answer edit: because people downvoted you a lot, which, to be fair, probably feels hostile.

                                                      2. 2

                                                        Do you sincerely think that small companies need to be threatened with 20M EUR fines to keep them in check?

                                                        I do sincerely think companies who can’t respect people’s privacy and don’t take the issue seriously should go out of business. The size of the company has nothing to do with it. The potential damage done once the info escapes into the wild is the same either way.

                                                        It’s unfortunate that it takes the threat of a 20M EUR fine and possibly going out of business to drive the point home, but asking nicely and hoping companies do the right thing hasn’t worked.

                                                        Who’s “we”? -The faceless bureaucracy of the EU that’s hoisting this pile of garbage on productive people?

                                                        If they can’t harness some of that productivity to protect the private data they collect, then good riddance.

                                                    2. 3

                                                      The EU says it wants to “protect” people with GDPR, while all governments are spying on people as much as they can

                                                      States (for better or worse) need a monopoly on coercion. Some of them have realized that the breakdown of privacy is eroding that monopoly, and they’re reacting.

                                                      1. -4

                                                        That doesn’t make sense, but you’re probably just trolling again, so whatever.

                                                        1. 2

                                                          What doesn’t make sense about it?

                                                          I’m not making a value judgement about whether it’s a good thing or not; I’m giving a reason why a self-interested state would choose this course of action.

                                                          Separating value judgement from behavioral reasoning is the only way to make sense of others behavior when they don’t share your values.

                                                      2. 3

                                                        Why do you think they are impossible to comply with? Germany has had laws like this for years and it works just fine.

                                                    1. 2

                                                      On the one hand this sounds really cool. On the OTHER hand - exploits anyone? :)

                                                      1. 2

                                                        Lua is much easier to sandbox than any other language I know of; basically it’s a one-liner to take a piece of code you’ve loaded and ensure it has no access to any functions outside the whitelist you provide. I would be terrified of such a feature if it used Javascript or something, but doing this securely in Lua is not that difficult.

                                                        1. 2

                                                          Both Redis and Company of Heroes 2 have been exploited through LUA sandbox escapes. JavaScript has similar properties as LUA in that it generally doesn’t assume to have any connection to the outside world and comes with no garbage preloaded. On top of that, JS interpreters are used in browsers and hardened for that use.

                                                          I’d say that if Javascript were a fundamental issue, systems like CouchDB which expose Spidermonkey to the real world would have exploits found there. I can’t find one at quick glance.

                                                          http://benmmurphy.github.io/blog/2015/06/04/redis-eval-lua-sandbox-escape/

                                                          https://gist.github.com/corsix/6575486

                                                          https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-19046/Apache-Couchdb.html

                                                          1. 3

                                                            5.1 was easy to attack because the Lua equivalents of eval would happily execute compiled bytecode. You can run arbitrary scripts safely in a sandbox but not arbitrary bytecode.

                                                            From 5.2 onwards you can pass an argument so the eval like functions ignore bytecode.

                                                            1. 3

                                                              People have found sandbox escapes in Lua 5.2.

                                                              I’m not saying issues cannot be fixed as they come. I’m saying that I don’t buy the argument that Lua is that much of a safer option then stock V8 or Spidermonkey. The latter are exposed to attempts quite regularly.

                                                              https://apocrypha.numin.it/talks/lua_bytecode_exploitation.pdf

                                                              1. 2

                                                                V8 didn’t even exist at the time Lua 5.1 came out. Even with 5.2, the escapes all come from loading bytecode; they don’t seem relevant to the topic of loading source.

                                                                V8 has so many critical security vulnerabilities that Debian doesn’t even attempt to keep up with security updates in the stable release; they recommend that it “should not currently be used with untrusted content, such as unsanitized data from the Internet. “

                                                                https://www.debian.org/releases/stretch/amd64/release-notes/ch-information.en.html#libv8

                                                            2. 1

                                                              Eh… I don’t think that’s a fair comparison since Redis is written in C and CouchDB is written in Erlang; obviously the latter will have a much better security track record.

                                                              1. 2

                                                                Remote code execution through calling loaded C functions or reading data from memory in a database server doesn’t require any support of the host language.

                                                            3. 1

                                                              So, I totally appreciate where you’re coming from here, and appreciate the background info around Lua, security, and embeddability.

                                                              I guess for me things like DNS belong in a ring of services where the server software for them is mega-hardened as much as is humanly possible. I’ve known about PowerDNS for years and worked at companies where it’s been implemented. It has a well earned reputation for solidity. Hopefully this is a configurable you can compile out for those who want to be ultra-paranoid.

                                                          1. 3

                                                            I’ve yet to read the paper in full but the abstract is promising.

                                                            1. 17

                                                              The abstract is promising but unfortunately:

                                                              That is, the only thing we need to do is to execute the model for every key and remember the worst over- and under-prediction of a position.

                                                              and

                                                              For example, a model without hidden layers can be trained on over 200M records in just few seconds if implemented in C++. However, for more complex models, we opted to use Tensorflow, which, because of its overhead, took significantly longer. Yet, we are confident that we can grid-search over the hyperparameter space relatively quickly, likely on the order of minutes for simple models.

                                                              It’s unusable for anything but static data sets, and if you have a static data set presumably you can do better than b-trees. They do also compare with a bad hashtable at the end and only barely beat it.

                                                              1. 4

                                                                I have yet to read the abstract but your comment is promising. :)

                                                                1. 2

                                                                  I have yet to read their comment but your comment is promising. :) (quiz: what is the shape of this infinite linked list?)

                                                                  1. 2

                                                                    data [contentOfUnknownValue] = [] | contentOfUnknownValue : [contentOfUnknownValue]

                                                                    1. 1

                                                                      Got close but not correct. Hint, the shape is similar to something you fry eggs in. Hint 2, start from root comment and follow down.

                                                                      1. 3

                                                                        I was going to give you an answer in pseudocode but never mind. I made about three or four circular lists [1] in formal specs and code trying. Having forgotten coding, I forgot just how messy and fun those suckers were to try to describe in a clean, compact, and low-level way. I did two of three with most of best one being clean in node descriptions but still looked a little messy on one spot. I’m going to save that one as an exercise for near future. :)

                                                                        [1] If it wasn’t what you had in mind, it was still fun to toy with anyway.

                                                                        1. 2

                                                                          Haha nice. Im glad you had some fun. I love programming, it is dangerous. Can be very addicting.

                                                              1. 32

                                                                I wasn’t implying. I was stating a fact.

                                                                And he’s wrong about that.

                                                                https://github.com/uutils/coreutils is a rewrite of a large chunk of coreutils in Rust. POSIX-compatible.

                                                                1. 12

                                                                  So on OpenBSD amd64 (the only arch rust runs on… there are at least 9 others, 8 7 or 6 of which rust doesn’t even support! )… this fails to build:

                                                                  error: aborting due to 19 previous errors
                                                                  
                                                                  error: Could not compile `nix`.
                                                                  warning: build failed, waiting for other jobs to finish...
                                                                  error: build failed
                                                                  
                                                                  1. 8

                                                                    Yep. The nix crate only supports FreeBSD currently.

                                                                    https://github.com/nix-rust/nix#supported-platforms

                                                                  2. 8

                                                                    The openbsd guys are stubborn of course, though they might have a point. tbh somebody could just fork a BSD OS to make this happen. rutsybsd or whatever you want to call it.

                                                                    edit: just tried to build what you linked, does cargo pin versions and verify the downloads? fetching so many dependencies at build time makes me super nervous. Are all those dependencies BSD licensed? It didn’t even compile on my machine, maybe the nixos version of rust is too old - i don’t know if the rust ecosystem is stable enough to base an OS on yet without constantly fixing broken builds.

                                                                    1. 10

                                                                      just tried to build what you linked, does cargo pin versions and verify the downloads?

                                                                      Cargo pins versions in Cargo.lock, and coreutils has one https://github.com/uutils/coreutils/blob/master/Cargo.lock.

                                                                      Cargo checks download integrity against the registry.

                                                                      For offline builds, you can vendor the dependencies: https://github.com/alexcrichton/cargo-vendor, downloading them all and working from them.

                                                                      Are all those dependencies BSD licensed?

                                                                      Yes. Using: https://github.com/onur/cargo-license

                                                                      Apache-2.0/MIT (50): bit-set, bit-vec, bitflags, bitflags, block-buffer, byte-tools, cc, cfg-if, chrono, cmake, digest, either, fake-simd, filetime, fnv, getopts, glob, half, itertools, lazy_static, libc, md5, nodrop, num, num-integer, num-iter, num-traits, num_cpus, pkg-config, quick-error, rand, regex, regex-syntax, remove_dir_all, semver, semver-parser, sha2, sha3, tempdir, tempfile, thread_local, time, typenum, unicode-width, unindent, unix_socket, unreachable, vec_map, walker, xattr

                                                                      BSD-3-Clause (3): fuchsia-zircon, fuchsia-zircon-sys, sha1

                                                                      MIT (21): advapi32-sys, ansi_term, atty, clap, data-encoding, generic-array, kernel32-sys, nix, onig, onig_sys, pretty-bytes, redox_syscall, redox_termios, strsim, term_grid, termion, termsize, textwrap, void, winapi, winapi-build

                                                                      MIT OR Apache-2.0 (2): hex, ioctl-sys

                                                                      MIT/Unlicense (7): aho-corasick, byteorder, memchr, same-file, utf8-ranges, walkdir, walkdir

                                                                      It didn’t even compile on my machine, maybe the nixos version of rust is too old - i don’t know if the rust ecosystem is stable enough to base an OS on yet without constantly fixing broken builds.

                                                                      This is one of my frequent outstanding annoyances with Rust currently: I don’t have a problem with people using the newest version of the language as long as their software is not being shipped on something with constraints, but at least they should document and test the minimum version of rustc they use.

                                                                      coreutils just checks against “stable”, which moves every 6 weeks: https://github.com/uutils/coreutils/blob/master/.travis.yml

                                                                      Can you give me rustc --version?

                                                                      Still, “commitment to stability” is a function of adoption. If, say, Ubuntu start shipping a Rust version in an LTS release, more and more people will try to stay backward compatible to that.

                                                                      1. 2

                                                                        rustc 1.17.0 cargo 0.18.0

                                                                        1. 11

                                                                          You’re probably hitting https://github.com/uutils/coreutils/issues/1064 then.

                                                                          Also, looking at it, it is indeed that they use combinatior functionality that became available in Rust 1.19.0. std::cmp::Reverse can be easily dropped and replaced by other code if 1.17.0-support would be needed.

                                                                          Thanks, I filed https://github.com/uutils/coreutils/issues/1100, asking for better docs.

                                                                          1. 1

                                                                            thanks for doing that, great community outreach :P

                                                                      2. 5

                                                                        Rust is “stable” in the sense that it is backwards compatible. However it is evolving rapidly so new crates or updates to crates may require the latest compiler. This won’t mean you’ll have to constantly fix broken builds; just that pulling in new crates may require you to update to the latest compiler.

                                                                        1. 4

                                                                          Yes, Cargo writes a Cargo.lock file with versions and hashes. Application developers are encouraged to commit it into version control.

                                                                          Dependencies are mostly MIT/Apache in the Rust world. You can use cargo-license to quickly look at the licenses of all dependencies.

                                                                          Redox OS is fully based on Rust :)

                                                                        2. 4

                                                                          Although you’re right to point out that project, one of Theo’s arguments had to do with compilation speeds:

                                                                          By the way, this is how long it takes to compile our grep:

                                                                          0m00.62s real 0m00.63s user 0m00.53s system

                                                                          … which is currently quite undoable for any Rust project, I believe. Cannot say if he’s exaggerating how important this is, though.

                                                                          1. 10

                                                                            Now, at least for GNU coreutils, ./configure runs a good chunk of what rust coreutils needs to compile. (2mins for a full release build, vs 1m20.399 just for configure). Also, the build is faster (coreutils takes a minute).

                                                                            Sure, this is comparing apples and oranges a little. Different software, different development states, different support. The rust compiler uses 4 cores during all that (especially due to cargo running parallel builds), GNU coreutils doesn’t do that by default (-j4 only takes 17s). On the other hand: all the crates that cargo builds can be shared. That means, on a build farm, you have nice small pieces that you know you can cache - obviously just once per rustc/crate pairing.

                                                                            Also, obviously, build farms will pull all kinds of stunts to accelerate things and the Rust community still has to grow a lot of that tooling, but I don’t perceive the problem as fundamental.

                                                                            EDIT: heh, forgot --release. And that for me. Adjusted the wording and the times.

                                                                            1. 5

                                                                              OpenBSD doesn’t use GNU coreutils, either; they have their own implementation of the base utils in their tree (here’s the implementation of ls, for example). As I understand it, there’s lots of reasons they don’t use GNU coreutils, but complexity (of the code, the tooling, and the utils themselves) is near the top of the list.

                                                                              1. 6

                                                                                Probably because most(all?) the openBSD versions of the coreutils existed before GNU did, let alone GNU coreutils. OpenBSD is a direct descendant of Berkeley’s BSD. Not to mention the licensing problem. GNU is all about the GPL. OpenBSD is all about the BSD(and it’s friends) license. Not that your reason isn’t also probably true.

                                                                              2. 2

                                                                                That means, on a build farm, you have nice small pieces that you know you can cache - obviously just once per rustc/crate pairing.

                                                                                FWIW sccache does this I think

                                                                              3. 7

                                                                                I think it would be more fair to look at how long it takes the average developer to knock out code-level safety issues + compiles on a modern machine. I think Rust might be faster per module of code. From there, incremental builds and caching will help a lot. This is another strawman excuse, though, since the Wirth-like languages could’ve been easily modified to output C, input C, turn safety off when needed, and so on. They compile faster than C on about any CPU. They’re safe-by-default. The runtime code is acceptable with it improving even better if outputting C to leverage their compilers.

                                                                                Many defenses of not using safe languages is that easy to discount. And OpenBSD is special because someone will point out that porting a Wirth-like compiler is a bit of work. It’s not even a fraction of the work and expertise required for their C-based mitigations. Even those might have been easier to do in a less-messy language. They’re motivated more by their culture and preferences than any technical argument about a language.

                                                                                1. 3

                                                                                  It’s a show stopper.

                                                                                  Slow compile times are a massive problem for C++, honestly I would say it’s one of the biggest problems with the language, and rustc is 1-2 orders of magnitude slower still.

                                                                                  1. 12

                                                                                    It’s a show stopper.

                                                                                    Hm, yet, last time I checked, C++ was relatively popular, Java (also not the fastest in compilation) is doing fine and scalac is still around. There’s people working on alternatives, but show stopper?

                                                                                    Sure, it’s an huge annoyance for “build-the-world”-approaches, but well…

                                                                                    Slow compile times are a massive problem for C++, honestly I would say it’s one of the biggest problems with the language, and rustc is 1-2 orders of magnitude slower still.

                                                                                    This heavily depends on the workload. rustc is quite fast when talking about rather non-generic code. The advantage of Rust over C++ is that coding in mostly non-generic Rust is a viable C alternative (and the language is built with that in mind), while a lot of C++ just isn’t very useful over C if you don’t rely on templates very much.

                                                                                    Also, rustc stable is a little over 2 years old vs. C/C++ compilers had ample headstart there.

                                                                                    I’m not saying the problem isn’t there, it has to be seen in context.

                                                                                    1. 9

                                                                                      C++ was relatively popular, Java (also not the fastest in compilation) is doing fine and scalac is still around.

                                                                                      Indeed, outside of gamedev most people place zero value in fast iteration times. (which unfortunately also implies they place zero value in product quality)

                                                                                      rustc is quite fast when talking about rather non-generic code.

                                                                                      That’s not even remotely true.

                                                                                      I don’t have specific benchmarks because I haven’t used rust for years, but see this post from 6 months ago that says it takes 15 seconds to build 8k lines of code. The sqlite amalgamated build is 200k lines of code and has to compile on a single core because it’s one compilation unit, and still only takes a few seconds. My C++ game engine is something like 80k if you include all the libraries and builds in like 4 seconds with almost no effort spent making it compile fast.

                                                                                      edit: from your coreutils example above, rustc takes 2 minutes to build 43k LOC, gcc takes 17 seconds to build 270k, which makes rustc 44x slower…

                                                                                      The last company I worked at had C++ builds that took many hours and to my knowledge that’s pretty standard. Even if you (very) conservatively say rustc is only 10x slower, they would be looking at compile times measured in days.

                                                                                      while a lot of C++ just isn’t very useful over C if you don’t rely on templates very much.

                                                                                      That’s also not true at all. Only small parts of a C++ codebase need templates, and you can easily make those templates simple enough that it has little to no effect on compile times.

                                                                                      Also, rustc stable is a little over 2 years old vs. C/C++ compilers had ample headstart there.

                                                                                      gcc has gotten slower over the years…

                                                                                      1. 6

                                                                                        Even if you (very) conservatively say rustc is only 10x slower,

                                                                                        Rustc isn’t slower to compile than C++. Depends on the amount of generics you use, but the same argument goes for C++ and templates. Rust does lend itself to more usage of generics which leads to more compact but slower-compiling code, which does mean that your time-per-LOC is higher for Rust, but that’s not a very useful metric. Dividing LOCs is not going to get you a useful measure of how fast the compiler is. I say this as someone who has worked on both a huge Rust and a huge C++ codebase and know what the compile times are like. Perhaps slightly worse for Rust but not like a 2x+ factor.

                                                                                        The main compilation speed problem of Rust vs C++ is that it’s harder to parallelize Rust compilations (large compilation units) which kind of leads to bottleneck crates. Incremental compilation helps here, and codegen-units already works.

                                                                                        Rust vs C is a whole other ball game though. The same ball game as C++ vs C.

                                                                                        1. 2

                                                                                          That post, this post, my experience, lines, seconds… very scientific :) Hardware can be wildly different, lines of code can be wildly different (especially in the amount of generics used), and the amount of lines necessary to do something can be a lot smaller in Rust, especially vs. plain C.

                                                                                          To add another unscientific comparison :) Servo release build from scratch on my machine (Ryzen 7 1700 @ 3.9GHz, SATA SSD) takes about 30 minutes. Firefox release build takes a bit more. Chromium… even more, closer to an hour. These are all different codebases, but they all implement a web browser, and the compile times are all in the same ballpark. So rustc is certainly not that much slower than clang++.

                                                                                          Only small parts of a C++ codebase need templates

                                                                                          Maybe you write templates rarely, but typical modern C++ uses them all over the place. As in, every STL container/smart pointer/algorithm/whatever is a template.

                                                                                          1. 2

                                                                                            To add another unscientific comparison :) Servo release build from scratch on my machine (Ryzen 7 1700 @ 3.9GHz, SATA SSD) takes about 30 minutes. Firefox release build takes a bit more. Chromium… even more, closer to an hour. These are all different codebases, but they all implement a web browser, and the compile times are all in the same ballpark. So rustc is certainly not that much slower than clang++.

                                                                                            • Firefox 35.9M lines of code
                                                                                            • Chromium 18.1M lines of code
                                                                                            • Servo 2.25M lines of code

                                                                                            You’re saying compiling 2.25M lines of code for a not feature complete browser that takes 30 minutes is comparable to compiling 18-35M lines of code in ‘a bit more’?

                                                                                            1. 4

                                                                                              Line counters like this one are entirely wrong.

                                                                                              This thing only counted https://github.com/servo/servo. Servo code is actually split among many many repositories.

                                                                                              HTML parser, CSS parser, URL parser, WebRender, animation, font sanitizer, IPC, sandbox, SpiderMonkey JS engine (C++), Firefox’s media playback (C++), Firefox’s canvas thingy with Skia (C++), HarfBuzz text shaping (C++) and more other stuff — all of this is included in the 30 minutes!

                                                                                              plus,

                                                                                              the amount of lines necessary to do something can be a lot smaller in Rust

                                                                                              1. 2

                                                                                                Agreed, it grossly underestimates how much code Chromium contains. You are aware of the horrible depot_tools and the amount of stuff they pull in?

                                                                                                My point was, you are comparing a feature incomplete browser that is a smaller code base at least in one order of magnitude but takes 30 minutes compared to “closer to an hour” of Chromium. If think your argument doesn’t hold - you are free to provide data to prove me wrong.

                                                                                              2. 3

                                                                                                Servo’s not a monolithic codebase. Firefox is monolithic. It’s a bad comparison.

                                                                                                Chromium is also mostly monolithic IIRC.

                                                                                      2. 2

                                                                                        Free- and OpenBSD can compile userland from source:

                                                                                        So decent compile times are of essence, especially if you are targeting multiple architectures.

                                                                                      3. 6

                                                                                        Well, ls is listed as only semi done, so he’s only semi wrong. :)

                                                                                        1. 11

                                                                                          The magic words being “There has been no attempt”. With that, especially by saying “attempt”, he’s completely wrong. There have been attempts. At everything he lists. (he lists more here: https://www.youtube.com/watch?v=fYgG0ds2_UQ&feature=youtu.be&t=2112 all of what Theo mentions has been written, in Rust, some even have multiple projects, and very serious ones at that)

                                                                                          For a more direct approach at BSD utils, there’s the redox core utils, which are BSD-util based. https://github.com/redox-os/coreutils

                                                                                          1. 2

                                                                                            Other magic words are “POSIX compatible”. Neither redox-os nor the uutils linked by @Manishearth seem to care particularly about this. I haven’t looked all that closely, but picking some random utils shows that none of them is fully compliant. It’s not even close, so surely they can’t be considered valid replacements of the C originals.

                                                                                            For example (assuming that I read the source code correctly) both implementations of cat lack the only POSIX-required option -u and the implementations of pwd lack both -L and -P. These are very simple tools and are considered done at least by uutils…

                                                                                            So, Theo may be wrong by saying that no attempts have been made, but I believe a whole lot of rather hard work still needs to be done before he will acknowledge serious efforts.

                                                                                            1. 5

                                                                                              This rapidly will devolve into a no true scotsman argument.

                                                                                              https://github.com/uutils/coreutils#run-busybox-tests

                                                                                              uutils is running the busybox tests. Which admittedly test for something other than POSIX compliance, but neither the GNU or BSD coreutils are POSIX-compliant anyway.

                                                                                              uutils is based on the GNU coreutils, redox’s ones are based on the BSD ones, which is certainly a step in the right direction and can certainly be counted as an attempt.

                                                                                              For example (assuming that I read the source code correctly) both implementations of cat lack the only POSIX-required option -u and the implementations of pwd lack both -L and -P.

                                                                                              Nobody said they were complete.

                                                                                              All we’re talking about is Theo’s rather strong point that “there has been no attempt”. There has.

                                                                                        2. 1

                                                                                          I’m curious about this statement in TdR in the linked email

                                                                                          For instance, rust cannot even compile itself on i386 at present time because it exhausts the address space.

                                                                                          Is this true?

                                                                                          1. 15

                                                                                            As always with these complaints, I can’t find any reference to exact issues. What’s true is that LLVM uses quite a bit of memory to compile and rustc builds tend not to be the smallest themselves. But not that big. Also, recent improvements have definitely worked here

                                                                                            I do regularly build the full chain on a ACER c720p, with FreeBSD, which has a celeron and 2 GB of RAM, I have to shut down the X server and everything before, but it works.

                                                                                            As usual, this is probably an issue of the kind “please report actual problems, and we work fixing that”. “We want to provide a build environment for OpenBSD and X Y Z is missing” is something we’d be happy support, some fuzzy notion of “this doesn’t fulfill our (somewhat fuzzy) criteria” isn’t actionable.

                                                                                            Rust for Haiku does ship Rust with i386 binaries and bootstrapping compilers (stage0): http://rust-on-haiku.com/downloads

                                                                                            1. 10

                                                                                              As always with these complaints, I can’t find any reference to exact issues.

                                                                                              Only because it’s a thread on the OpenBSD mailing lists, people reading that list have the full context of the recent issues with Firefox and Rust.

                                                                                              I’ll assume you just don’t follow the list so here is the relevant thread lang/rust: update to 1.22.1

                                                                                              • For this release, I had lot of problem for updating i386 to 1.22.1 (too much memory pressure when compiling 1.22 with 1.21 version). So the bootstrap was initially regenerated by crosscompiling it from amd64, and next I regenerate a proper 1.22 bootstrap from i386. Build 1.22 with 1.22 seems to fit in memory.

                                                                                              As I do all this work with a dedicated host, it is possible that ENOMEM will come back in bulk.

                                                                                              And if the required memory still grows, rustc will be marked BROKEN on i386 (and firefox will not be available anymore on i386)

                                                                                              1. 7

                                                                                                Only because it’s a thread on the OpenBSD mailing lists, people reading that list have the full context of the recent issues with Firefox and Rust.

                                                                                                Sure, but has this:

                                                                                                And if the required memory still grows, rustc will be marked BROKEN on i386 (and firefox will not be available anymore on i386).

                                                                                                Reached the Rust maintainers? (thread on the internals mailing list, issue on rust-lang/rust?)

                                                                                                I’m happy to be corrected.

                                                                                                1. 7

                                                                                                  Reached the Rust maintainers? (thread on the internals mailing list, issue on rust-lang/rust?)

                                                                                                  I don’t know. I don’t follow rust development, however the author of that email is a rust contributor like I mentioned to you in the past so I assume that it’s known to people working on the project. Perhaps you should check on that internals mailing list, I checked rust-lang/rust on github but didn’t find anything relevant :)

                                                                                                  1. 7

                                                                                                    I checked IRLO (https://internals.rust-lang.org/) and also nothing. (“internals” by the way referring to the “compiler internals”, we have no closed mailing list). The problem on projects of that scale seems to be that information travel is a huge issue and that leads to aggrevation. The reason I’m asking is not that I want to disprove you, I just want to ensure that I don’t open a discussion that’s already happening somewhere just because something is going through social media currently.

                                                                                                    Thanks for pointing that out, I will ensure there’s some discussion.

                                                                                                    Reading the linked post, it seems to mostly be a regression when doing the jump between 1.21 to 1.22, so that should probably be a thing to keep an eye out for.

                                                                                                  2. 2

                                                                                                    Here’s a current Rust bug that makes life hard for people trying to work on newer platforms.

                                                                                              2. 2

                                                                                                I’m skeptical; this has certainly worked for me in the past.

                                                                                                I used 32 bit lab machines as a place to delegate builds to back when I was a student.

                                                                                                1. 4

                                                                                                  Note that different operating systems will have different address space layout policies and limits. Your effective space can vary from possibly more than 3GB to possibly less than 2GB.

                                                                                            1. 6

                                                                                              The horror, I would try to statically link EVERYTHING. At least the syscalls are stable.

                                                                                              1. 6

                                                                                                Yep, I just use musl libc and static link everything into the binary. Simpler than all the effort here to use a libc designed to be static linked.. Just need to make sure the kernels you run on for threaded stuff is recent enough, iirc 2.6.16-19ish is when those stabilized. But for most stuff its unlikely to matter.

                                                                                                Irony is the binaries are smaller than the glibc binaries linked against libc. glibc is not my favorite libc.

                                                                                                1. 3

                                                                                                  Unfortunately that doesn’t work for games, you can’t statically link things like X/libGL/ALSA.

                                                                                                  There’s a blogpost about building bins that only use the system dlopen/dlsym but that seems like a hassle. If you use GLFW and want to try it out you’ll need to modify it to use dlopened X (GLFW did change to do this for many parts of X quite recently), not sure about SDL. On the bright side there are already free tools to generate the dlopen/dlsym mess for OpenGL (glad is the best one), and the code to dlopen ALSA is pretty short.

                                                                                                  I assume the glibc dynamic linker is too massive and annoying to statically link, and the musl one can’t load glibc libraries so is useless in practice, so for now you can’t do anything about the libdl dependency. (Now I have the go ahead from my boss to sell my hobby projects, I wonder if people would pay for a solution to this?)

                                                                                                  1. 4

                                                                                                    The author of this blog post is a gamedev working on dhewm3 (Doom 3 port) and yquake2 (Quake 2 port). The post itself touches on SDL, OpenGL and X/Wayland. I might be not seeing something but what makes you believe the article can’t be applied to games (I believe the author did this work to build one of the doom/quake ports that way - but I’m only guessing here).

                                                                                                    1. 5

                                                                                                      I’m responding to ac, not the article!

                                                                                                      My point is that you can’t statically link everything, you have to dynamically load certain things, and to do that you have to dynamically load the dynamic loader.

                                                                                                      1. 3

                                                                                                        For dhewm3 and yq2 this is no problem: It’s open source and we don’t provide Linux binaries ourselves (and if we did an old compiler should work just fine).

                                                                                                        I’ve needed this for a commercial closed source game that uses some C++11 features and can’t be built with GCC 4.7

                                                                                                      2. 4

                                                                                                        (I’m the author of the blog post)
                                                                                                        You’re right, statically linking is no option for games (and probably not for anything else that needs to interact with X11 or Wayland).
                                                                                                        Also note that glibc is not the problem here - usually linking against an old version of it works just fine, and if you want to optionally use a more recent feature, you can still dlsym() it. (It’s annoying that it’s not easy to just point the compiler/linker to a glibc version to use, but that it uses the system version instead - however using a chroot for building release binaries is workable).
                                                                                                        The problems are mostly libstdc++ and libgcc (though maybe that one can actually be statically linked, not sure). This is because some systemlibs you need to interact with (at least libGL) use libstdc++, and if you statically link a different version than the one on the users system, things explode. Same problem if you just bundle and unconditionally use (e.g. via rpath $ORIGIN) a version of libstdc++ and the libGL (or whatever other lib) needs a newer version than the one you bundled.
                                                                                                        Of course all this is detailed in the article.. :-P

                                                                                                        By the way, SDL2 indeed uses dlopen() and dlsym() to interact with most system libs (especially the display and audio specific ones, probably not glibc..). This is pretty awesome: The same libSDL2-2.0.so.0 can support X11 and wayland, and so can your game executable (unless you do X11 specific stuff in addition to what SDL2 offers).

                                                                                                      3. 2

                                                                                                        That’s what I do using MUSL. Seems to work great.

                                                                                                      1. 5

                                                                                                        Home:

                                                                                                        Turns out some of the stuff in the clipmaps paper I thought was unnecessary is actually necessary. Specifically, the 2 square wide tiles are needed to prevent big tiles from overlapping (they can be 1 square wide but it makes trim placement ever so slightly more complicated), and the L shaped trim tiles are needed because if you try to do stretchy seams you end up with missing triangles in the corners.

                                                                                                        I’m quite annoyed that I had to find out the hard way so I’ll probably end up rage blogging a clipmaps overview/tutorial when I’m done.

                                                                                                        I still need to fix a few bugs and fill in the seams, but the seams should really be easy now the trim is in place. It’s looking a lot better now the holes are mostly gone: https://i.imgur.com/ssyXO94.jpg

                                                                                                        Work:

                                                                                                        I got the go ahead to sell the compression algorithm I’ve been working on in the hackathons at work. The product is good and valuable but I really have no idea how to do the business side of things. Do I need to start a company to be able to sell it? Would it be better to start the company in the UK or here in Finland?

                                                                                                        Time to start reading barnacl.es I guess.

                                                                                                        1. 4

                                                                                                          Home: fixing the long tail of bugs introduced by compressing my terrain as BC5. I had a typo in my BC5 decoder -> you fall through the world randomly. I was generating the collision quadtree with the original terrain and not the BC5 terrain (BC5 is lossy!) -> you fall through the world randomly. Trees were placed on the original terrain and I also missed a .0f in my code so they were often intersecting with the ground. Once that was all done I pushed out a release, which was immediately followed by a second release with fixes for AMD GPUs and Windows UAC.

                                                                                                          I think the wireframe view looks quite cool now. You can see the finely tessellated terrain below the camera and the coarse terrain in the distance, you can the engine switch to 2 triangle tiles when they don’t overlap the terrain at all, and you can see the skirt extending to the horizon.

                                                                                                          The only thing missing now is the seams. I keep trying to convince myself they’ll be easy and then when I start working on them it’s just so obviously not true. In particular the worst case looks like this. Maybe the 2 triangle tiles are stupid? It does only save like 150k triangles at best. Maybe they should have full res borders?

                                                                                                          Work: I asked if I can take my hackathon project and try to sell it outside the company. I have no idea if it’s something people would actually spend money on (a faster LZ4) but I guess it would be nice to go through the whole marketing/sales pipe by myself, even for just a single sale.

                                                                                                          1. 2

                                                                                                            Home: more clipmap polishing. Seams are not as easy as I thought and I have overlapping geometry on the other side.

                                                                                                            Collision detection is an endless source of misery. The fork of squish I’m using has a BC5 encoder but not a decoder, so my triangles were all in the wrong place. The intersections looked correct since I use the quadtree to narrow it down to 8 triangles which limits how wrong they can be, but when I drew the triangles they were obviously not right. There are still some bugs that I am yet to figure out but at this point I’m quite tempted to switch to something like bullet.

                                                                                                            Work: We had a hackathon Thursday and Friday. I did some baking at the office and kept working on my LZ4 decoder. Last hackathon I was almost able to match the performance of the reference implementation, this time I decided to deviate from the spec and found some nice perf wins so now I’m 25%-33% faster to decode.

                                                                                                            The big wins were explicitly using AVX loads/stores in the copy loops, and not encoding overlap matches with an offset < 32 (so you don’t have to special case them in the decoder, the AVX copy will just work). I also started working on vectorising the control byte decoding, but ran out of time. I suspect that will be a nice speed boost too but we will have to see.

                                                                                                            1. 3

                                                                                                              Work: the fixed size profiler turned out to be pretty bad, so I’m writing an MPSC queue so we can use code outside the signal handlers to record samples. CDSChecker and Relacy are giving me very confusing error traces so this is turning out to be more work than I expected.

                                                                                                              Home programming: more clipmaps work. I’m now using BC5 for the heightmap with h % 256 in one channel and h / 256 in the other which seems to be working well.

                                                                                                              It turns out that seams should actually be pretty easy, the seam geometry is the same for every transition so you can upload a single mesh and reuse that (like you do for the tiles themselves). There will be some stretching but I’m hoping it won’t be noticeable.

                                                                                                              I also did some misc cleanup that’s been on my todo list for way too long. Replacing GL_FRAMEBUFFER_SRGB with explicit conversions, fixing my triangle winding orders so I cull backfaces instead of frontfaces, less annoying debug primitive renderer API, make the game actually work with UAC on Windows, etc.

                                                                                                              Home not programming: insurance company has sent a debt collector after me without ever sending an invoice. I can either spend ages figuring out wtf happened, which will cost far more of my time than the value of the bill, or I can just pay it. Extremely annoying.

                                                                                                              1. 3

                                                                                                                Work: we launched so the integration work has finally come to end. Going to be moving on to optimising things. One of the problems we have now is our computations use tons of memory, and AWS only sells VMs with normal CPU/memory ratios so the CPUs are underutilised. We are going to be working on eventually reducing memory usage, and in the short term I’m going to be working on making individual computations multithreaded.

                                                                                                                The first step though is writing a profiler that we can keep enabled all the time. Perf generates way too much data to run constantly, so I’m writing a sampling profiler that works in a constant amount of space. Basically it’s a fixed size hashtable, and when you can’t find an empty bucket you overwrite the bucket with the lowest hitcount from the set you looked at while reprobing. It’s probably not the best strategy but it was easy and in quick tests it seems to work ok.

                                                                                                                Home: slow game progress, haven’t been programming much. Geometry clipmaps apparently are not a silver bullet. If I draw tiles I have to deal with seams, if I draw a massive mesh I lose frustum culling. Also struggling to get my head around the details of geomorphing.

                                                                                                                1. 1

                                                                                                                  std::function produces extremely bad code, like pages of assembly for something trivial, and should not be used.

                                                                                                                  You can do it like this instead:

                                                                                                                  #define CONCAT_HELPER( a, b ) a##b
                                                                                                                  #define CONCAT( a, b ) CONCAT_HELPER( a, b )
                                                                                                                  #define COUNTER_NAME( x ) CONCAT( x, __COUNTER__ )
                                                                                                                  
                                                                                                                  template< typename F >
                                                                                                                  struct ScopeExit {
                                                                                                                          ScopeExit( F f_ ) : f( f_ ) { }
                                                                                                                          ~ScopeExit() { f(); }
                                                                                                                          F f;
                                                                                                                  };
                                                                                                                  
                                                                                                                  template< typename F >
                                                                                                                  inline ScopeExit< F > MakeScopeExit( F f ) {
                                                                                                                          return ScopeExit< F >( f );
                                                                                                                  };
                                                                                                                  #define SCOPE_EXIT( code ) auto COUNTER_NAME( SCOPE_EXIT_ ) = MakeScopeExit( [&]() { code; } )
                                                                                                                  

                                                                                                                  (from http://the-witness.net/news/2012/11/scopeexit-in-c11/)

                                                                                                                  1. 4

                                                                                                                    So I’ve only spent like two minutes looking, but I think you can also do pattern matching with switch(variant.index()) if you like.

                                                                                                                    1. 6

                                                                                                                      That would be non-exhaustive at compile-time

                                                                                                                      1. 3

                                                                                                                        What the author’s going for is selecting an alternative and getting the value out should be the same operation, so you can’t hit a runtime type error between checking the alternative and unboxing it.

                                                                                                                        1. 1

                                                                                                                          A guarantee that the type won’t change in a concurrent environment is going to require a lot more machinery I think. There’s accessors to get pointers to the value. And in other cases, the value itself may be a pointer. If the type can change at any time, those pointers will invalidated. If you want the borrow checker, it’s over there I guess, but I didn’t really get the sense that the author wants atomic selectors.

                                                                                                                          1. 3

                                                                                                                            I was unclear there. By “the same operation,” I just mean that selecting and unboxing shouldn’t be separate points in the code. I’m not about to argue that variant needs to be concurrent.

                                                                                                                            What I mean is just that in

                                                                                                                            select (boink.index()) {
                                                                                                                            case 0: int val = get<0>(boink); 
                                                                                                                                ...
                                                                                                                            }
                                                                                                                            

                                                                                                                            you’re mentioning the index twice, and if you mistype one or the other you’ll only find out at runtime.

                                                                                                                            1. 1

                                                                                                                              Oh, I see. Yeah, you could do switch ((idx = boink.index())) (always another workaround) but it is backtreading.

                                                                                                                        2. 2

                                                                                                                          Then you’re writing exactly the same code as you would with a tagged union, except it’s actually worse because you don’t get -Wswitch. (warning: enumeration value ‘ASDF’ not handled in switch [-Wswitch])

                                                                                                                          1. 2

                                                                                                                            Well, it’s solve the problem that the tag and the value diverge because you forget to update one. Guess you get to choose which is worse. It seems the return type of index could be an enum (though obviously they didn’t do that). Not sure how much more difficult that would have been.