1. 21

    Compiling Firefox on 8 core ryzen targetting the host arch takes between 10 and 15 minutes.

    1. 10

      Wow that is fast, takes ~2h on a build server that takes 7 hours for chromium. All the recent rust stuff really slowed it down.

      1. 6

        All the recent rust stuff really slowed it down.

        Oof, yeah, I bet. Rust is notoriously slow to compile. Luckily, incremental compilation is in the nightly compiler right now. I’ve been using it wherever I can and it really does make a difference. But I suppose it wouldn’t do much for an initial compilation of a project. :(

        1. 4

          In this case a large chunk of this is just bindgen; we need to generate bindings so we throw a libclang-based-bindings generator at all of the header files. Twice (complicated reasons for this).

          It’s also pretty single threaded (codegen units will help with this, but currently isn’t, and I have to investigate why).

          Incremental compilation and working codegen units and cleaning up the bindgen situation will help a lot. Going to take a while, though.

          1. 3

            But I suppose it wouldn’t do much for an initial compilation of a project.

            Also not going to help packagers who use only temporary compilation environments which are discarded after a package is built.

            1. 6

              Package managers (and regular builds also) should not be starting from scratch every time. Even if we insist on doing all source editing in ASCII, we need to be delivering modules as fully-typed, parsed ASTs.

              This insistence on going back to plain source code every chance we get and starting over is easily wasting more energy than the bitcoin bubble.

              1. 9

                Package managers (and regular builds also) should not be starting from scratch every time.

                They should if you want reproducible builds.

                1. 3

                  These are completely orthogonal. There’s nothing stopping reproduceable builds where you run the entire pipeline if you insist on comparing the hash of the source and the hash of the output. And you would still get the same benefit by comparing source<->ast and ast<->binary

                  1. 1

                    Yeah, ideally a compiler would take a difference in source to a difference in output. Compiler differentiation?

                  2. 2

                    I believe you can do a decent amount of caching in this space? Like MSFT and co. have compile serverse that will store incrementally compiled stuff for a lot of projects so you’re only compiling changes

              2. 7

                My non-beefy lenovo x series laptop takes ~45 minutes for a complete recompile, ~12min for average changes w/ ccache etc. and ~min for JS-only changes (which you can do as artifact builds, so they’re always 3min unless you need to build C++/Rust components

              1. 25

                Mercurial, made by another (in my opinion) much more well-spoken kernel hacker, is what really introduced me to the concept that you do not break interface for downstream users, no matter how wrongly you may think they are using the interface.

                It’s an attitude that is difficult to convey because software developers always want to have the freedom to “improve” their own software, even at the possible cost of breaking something for some users (and worse, even telling users that they shouldn’t have been doing that in the first place).

                I keep going back to this blog post which I wish more people agreed with (Steve Losh is another person influenced by Mercurial):

                http://stevelosh.com/blog/2012/04/volatile-software/

                1. 4

                  Good write-up. And, yet, backward compatibility is the reason for most woes of IBM/COBOL and Wintel. Both improved their stacks a lot by creating incompatible additions. On IBM’s side, they added stuff from UNIX ecosystem. On Microsoft’s side, they broke the driver model and permission model after switching to managed code for lots of apps. The author you linked to could’ve written a similar piece on Vista as almost everyone did. Although they did botch execution of it, the key, painful changes that were polished up by Windows 7 were great for the long term in both reliability, security, and (.NET) maintainability getting off C++. Just the driver architecture and verifier alone eliminated most blue screens.

                  Note that what I described doesn’t mean changing things randomly and unnecessarily which cause a lot of what author describes. The companies doing high-availability software often create deltas in between that support old functionality/configurations and new ones. Optionally tools to convert between them manually or automatically. Then, the upgrade doesn’t have unplanned downtime or headaches. Minimal at least. We don’t see most proprietary or FOSS software doing that. Instead, it’s “Surprise! Your stuff is now broken!”

                  The other thing to address is the author writes as if developers owe the users something. There’s some moral imperative. There are FOSS developers out there who are fairly selfless in that they’re all about the experience of their users. Many aren’t, though.They might be working for corporations such as Red Hat or IBM contributing to Linux. They might be building something mainly for themselves or a small group of contributors that they share with the world. They might even be building a product with a somewhat-neglected, FOSS version with less features. In any case, most of the users will be freeloaders who the developers are not working for or value very little. If those people are having problems, the developers with such motivations should ignore them.

                  So, I’d ask whether the Skype developers were trying to create a fantastic experience for Steve Losh on his new box or were doing what their managers wanted for the product for whatever reasons the business had. I’m leaning toward the latter which reframes his gripe about them. Maybe similar with the others. They also maybe well-intentioned but sloppy as he says. Who knows. Just not as simple as all developers having a moral imperative to create a specific experience for specific or all users.

                  1. 3

                    “And, yet, backward compatibility is the reason for most woes of IBM/COBOL and Wintel.”

                    I’m no kernel expert, but I think what Linus means about no regressions/don’t change the interface is only referring to minor versions, major versions of the kernel are allowed to change the API?

                    1. 7

                      I’m no kernel expert, but I think what Linus means about no regressions/don’t change the interface is only referring to minor versions, major versions of the kernel are allowed to change the API?

                      No. The kernel’s public API is supposed to be backwards compatible even between major versions. From https://github.com/torvalds/linux/tree/master/Documentation/ABI :

                      Most interfaces (like syscalls) are expected to never change and always be available.

                      1. 1

                        How did you get “Active user with invites disabled” on your profile?

                        1. 1

                          How did you get “Active user with invites disabled” on your profile?

                          By inviting each and every user that asked for an invite through the website form. This must have upset the gatekeepers who decided that since some of those were spammers (probably the self-promotion type) I need to have my inviting rights revoked.

                      2. 4

                        Linus has made it clear that version numbers of the kernel mean nothing and are entirely arbitrary. In his last interview(https://www.youtube.com/watch?v=NLQZzEvavGs&feature=share) He makes it VERY plain this is the case. He said basically that after the minor numbers get into the double digits he starts to lose track and bumps the major number. So we should hit 5.0 around next summer, but he made zero promises of this.

                        1. 4

                          See this other thread. You need to click “Show 89 previous comments” because Google+ seems to be unable to link to specific comments.

                          Alan Cox (a kernel hacker) writes “my 3.6rc kernel will still run a Rogue binary built in 1992. X is back compatible to apps far older than Linux.” and I would assume it still runs today.

                          And below Alan is Linus ranting about Gnome breaking stuff all the time.

                          1. 4

                            To add to what @stefantalpalaru said, actually in Linux version numbers mean not much. They’re added to distinguish between versions, and the major version number changes every time Linus thinks the minor version numbers are getting too big.

                            1. 1

                              I was replying to the linked article.

                        1. 6

                          This is incredible! Is the code open source?

                          1. 13

                            Not yet, but it will be soon!

                            1. 3

                              It doesn’t look like it but there is a draft of the paper available: https://arxiv.org/pdf/1707.07397.pdf

                              1. 2

                                They have a «code» link that says «coming soon» — so, not yet, but hopefully it will be in the future.

                              1. 5

                                I’m confused. Are we just talking about writing tests before your features or not writing tests at all? This post has examples of both.

                                In general, I never want to work for a company again that doesn’t have unit tests on all their major services. Testing in general is crucial and vital to be able to refactor things quickly and move platforms and technologies quickly. There is no substitute for good, automated regression tests. If your tests miss something, you can add a test. If you get a false negative, you can fix/improve the tests.

                                I prefer writing tests first, but it depends on the problem. If it’s a totally new subsystem, you may need to write a few prototypes and figure out how you want to architect it first. Then write your tests. If it’s building into an existing system, it’s usually easier to write the tests first and red/green them. It depends on the problem. But you should always have tests.

                                The post mentions the Linux kernel. It’s more difficult when you need to test with different hardware, but a quick search led to me several independent projects by other companies to at least build some basic automated tests around the Linux kernel. So they do exist, even if not ever kernel developer uses them.

                                1. 2

                                  Linux doesn’t have continuous integration so the few tests that it has are not very useful.

                                1. 4

                                  Heck, I might even start using an RSS reader again. Is Google’s still the best one out there?

                                  Too soon, man :(

                                  1. 21

                                    I promise I’m not that negative of a person, it’s just that only rant inducing stories inspire me to comment. That entire issue is a dumpster fire.

                                    Problem 1: Node has a weird sized integer type smaller than most commonly used integers, causing data loss when using numbers to identify inodes.

                                    The proposed solutions are all kinds of sad. Someone wants to turn the integer into a string when it gets big enough, because I’m sure having a function randomly change it’s return type won’t cause more bugs. Nothing like doubling down on type un-safety when presented with a dataloss bug.

                                    Someone proposes breaking up the 64bit int into a series of ints representable in node. Which makes sense if node is a 8-bit microcontroller, it’s a little disappointing otherwise. This is probably the most sensible backwards-incompatible solution.

                                    There’s lots of handwavy “semantically it’s not an integer”. Whatever. I’m not sure I trust the commenters on that ticket to talk knowledgeably about filesystem semantics.

                                    The ugliest, but maybe most sensible solution for now is someone who proposes a 64->53 bit hashing op. Seems like they might minimize the practical impact of the bug quite a bit that way. This would of course break any code that passed the ino value back into c-land.

                                    Man life would be easier if node had 64bit ints.

                                    1. 17

                                      The irony here is that windows itself returns inode numbers (file ID) as two 32 bit values, high and low.

                                      https://msdn.microsoft.com/en-us/library/aa363788(v=vs.85).aspx

                                      1. 14

                                        There’s lots of handwavy “semantically it’s not an integer”. Whatever. I’m not sure I trust the commenters on that ticket to talk knowledgeably about filesystem semantics.

                                        Man life would be easier if node had 64bit ints.

                                        The question is more, “why should this be an integer?” - what integer operations do we want to perform on an inode? I can’t think of any - an inode is just an identifier, right?

                                        1. 4

                                          That’s a valid question, but I personally think it’s the wrong question. What’s wrong with an integer identifier? We use them in filesystems and sports jerseys. They have one really important operation predefined: increment. An API user doesn’t need to ++, but something generating unique IDs might want to. If inode number was a string and node truncated it, nobody would argue that was a bug. Why should a 64bit integer be different?

                                          I answer the question of “why should this be an integer” by looking at the spec which defines them as integer types: http://pubs.opengroup.org/onlinepubs/009696699/basedefs/sys/types.h.html

                                          It’s a very interesting language design question: how important is it to make sure the apis of the operating system and internet can be faithfully represented in your language?

                                          1. 2

                                            how important is it to make sure…

                                            Not very, if your abstractions and FFI are good enough. Languages, in order to gain traction, have to be somewhat portable. This can be done in many different ways, but most commonly it seems that API designers define a minimal set of functionality needed, and leave the rest as an exercise to the programmer (likely via FFI).

                                            1. 5

                                              In Rust land, we try to expose platform specific functionality via conditional compilation. This means that while you need to be aware of platform specific behavior, you do get standard conveniences once you opt into it. Our first attempt at it was the std::os module, which basically exposes a bunch of platform specific traits that you can bring into scope, which in turn add additional platform specific methods on platform independent types.

                                              For various reasons outlined in this RFC, we’ve been wanting to head more towards the “define the platform specific methods directly on types” approach. But to do that, we want a better linting system to help prevent conditional compilation errors.

                                              Just recently, we did a review of the memmap crate which mostly exposes platform independent behavior. But it sounds like it’s headed in a similar direction as std, where platform dependent functionality will also be exposed.

                                              There’s still a long road to go of course, but it’s a useful counterpoint to “just use FFI” IMO. :-)

                                            2. 2

                                              It is not at all obvious that integers are unique. And they have far more structure that you can’t actually use. For an opaque unique ID something like a UUID would be much better (I believe that’s what ZFS uses, at least internally?)

                                              1. 3

                                                There’s also the nuance that since inode numbers can be reused, increment doesn’t just solve the problem of uniqueness. But, a UUID is just a really big integer, too…

                                                1. 1

                                                  A UUID is implemented as an integer, but semantically it declares a clear intent not to be used as one; most UUID libraries do not expose addition or multiplication operations on them.

                                                  1. 2

                                                    Right. But, of course an ino_t and a UUID are isomorphic. We just don’t have any special representation of ino_t as we do for UUIDs.

                                            3. 3

                                              It’s a collection of flags and some very small integer values.

                                              Lua (up to 5.2; Lua 5.3 has 64-bit integers) has/had this problem, and I solved it by breaking the individual bits out as flags. Yes, not that efficient, but it makes it much easier to use in Lua.

                                              1. 8

                                                What? That’s not right at all, an inode is a unique identifier for a file on the filesystem! And since it’s meaning is opaque other than being unique, puffnfresh is 100% correct.

                                                Perhaps you are thinking of mode? Regardless, an inode is definitely not a collection of flags.

                                                1. 3

                                                  You are right, I got it confused with the mode bits. Sigh.

                                            4. 9

                                              The ugliest, but maybe most sensible solution for now is someone who proposes a 64->53 bit hashing op.

                                              This is a terrible idea, it just hides bugs without fixing the problem at all. I would say it’s the least sensible solution by a huge margin.

                                              1. 1

                                                It is indeed a terrible idea, but even that would be better than sitting around. This issue is still open 2 months later. New thread: https://lobste.rs/s/ycvjzp/node_occasionally_gives_multiple_files

                                                As far as hiding bugs without fixing the problem, you can’t polish a turd (node.js) :P

                                              2. 1

                                                It’s not all that strange considering both Facebook and Twitter encode their IDs as strings in JSON reponses. Twitter adds a *_str field next to the ‘legacy’ fields, which is another solution.

                                                Perhaps it should’ve been foreseen, though. But I think the filesystem interfaces are also some of the oldest?

                                                1. 4

                                                  It’s not strange because twitter solved it that way? Twitter is a web service that provides an abstraction, via their API, to their machines. At such a high level, this seems perfectly reasonable because the ID is completely opaque to users. It’s an implementation detail.

                                                  But in a language that is being used to write local systems, where the inode number actually has a standard implementation of unsigned int, which is required by local APIs and system calls? A bare string of “this is an inode” is meaningless, but a valid sting, and therefore a valid candidate for an inode. Make a new type. Support JS numbers or this new type in all APIs and call it a day.

                                              1. 3

                                                This will be used in a ton of scams and make social engineering a walk in the park.

                                                1. 3

                                                  Did you actually listen to the demos?

                                                  They are somewhat recognizable as the person but a long way from sounding natural and making scams a “walk in the park”.

                                                  1. 2

                                                    Their samples say they are not cherry-picked, so I’d believe it could be more convincing with the current technology. And in security, we say “attacks only get better”.

                                                    1. 1

                                                      They’re not perfect, but I’d imagine they’d sound a lot more realistic if you had to listen to them over the phone.

                                                    2. 1

                                                      Good. Maybe as this technology becomes more accessible, we’ll move on to crypto authentication.

                                                    1. 9

                                                      I submit as a counterpoint that, in yosefk’s experience, low-level is easy.

                                                      1. 4

                                                        Yeah I like this. One observation I’ve had: writing C or C++ is relatively easy, but the corresponding Makefiles and autotools are harder.

                                                        Does anyone else feel that way? Once you are situated and understand pointers and all that, it feels pretty easy to write the actual C or C++ code. Particularly if you are only using libc and libstdc++ and not huge abstractions like APR or OpenGL or something (low-level is easy).

                                                        But the hard parts seem like getting the compilers and build system to do what you want… debugging preprocessor directives can be harder than debugging the program. Getting the debugger to run against a binary with the right build flags (for an arbitrary open source project) is harder than debugging it.

                                                        For one, a Unix-style build system for a mature project requires understanding 3 or 4 different languages: shell, make, sometimes awk, {m4/autotools, cmake, Ninja} etc. Whereas C and C++ is 1 or 2 languages.

                                                        1. 3

                                                          Learning how to use all tools required to be a truly effective C programmer may indeed be approximately as hard as learning basic C.

                                                          1. 3

                                                            Where I work, embedded code that runs on our cameras is generally easy to understand, and reads like a captivating yet approachable exploration in low-level systems programming. Our marginally technical product managers can understand this code, and some indeed do send in PRs against it.

                                                            On the other hand, our externally facing APIs and various data pipelines are a morass of abstractions and concessions to past scalability problems that make even the gnarliest embedded code look like child’s play. There’s a joke around the office that goes something like “embedded tends to suck in everyone from time to time”, but I suspect that people go there to be productive in a simpler, more straightforward world.

                                                            1. 2

                                                              One observation I’ve had: writing C or C++ is relatively easy, but the corresponding Makefiles and autotools are harder.

                                                              Particularly if you are only using libc and libstdc++ and not huge abstractions like APR or OpenGL or something (low-level is easy).

                                                              Those two statements are almost contradictory.

                                                              If your project only depends on libc and libstdc++ and its not using any third party libraries then it’s nobody’s fault but your own if your Makefile is very complicated. Using autotools in that situation buys you nothing but extra complexity and a fancy Makefile.

                                                              IME it’s much easier to create a Makefile template with the targets you want, and then reuse it for small projects. The Makefile syntax itself can be complicated, but 99.99% of the time it’s best to keep it simple.

                                                              1. 2

                                                                Point taken, but if the thesis is “low level is easy” then you can make the comparison between plain C code with no deps, and a huge generated makefile for something with deps.

                                                                In both cases, the higher level layers are what make things hard. I concede that using and understanding the big platform layers in C are certainly more difficult than a makefile.

                                                                I guess the overall point is that on a typical big C project you have two sources of complexity and dofficulty – build time and runtime. Runtime is to some extent essential, but the mess at build time can be fixed.

                                                                1. 2

                                                                  Even with simple code and no dependencies, you need a huge messy Makefile if you want to support debug/asan/release builds on win32/win64/Linux/OSX. (doing make clean and changing CFLAGS whenever you want to switch builds is not good enough)

                                                                2. 2

                                                                  writing C or C++ is relatively easy

                                                                  I don’t feel this way. Writing C that never trips undefined behaviour does not feel particularly easy to me. Also I have to churn out a lot of C code to get it to do fairly trivial things, and then I have to find all the stupid mistakes I left in.

                                                                  I don’t think the actual writing of low-level software is itself easier. You’re implementing algorithms on your hands and knees all the time. You so much as look at cc funny and you get undefined behaviour you might not discover for years. The debugging tools are often broken.

                                                                  I think yosefk is arguing that being a software developer working on low-level software is overall easier, because the part where you actually write the software is somewhat harder, but everything else involved in the job is easier. You have far less difficulties caused by, for example, people handing you ill-defined requirements to implement.

                                                                  I think the important bit of the article I linked is this:

                                                                  As a low-level programmer, you have to convince people not to be afraid when you give them something. As a high-level programmer, you have to convince them that you can’t just give them more and more and MORE.

                                                                  1. 2

                                                                    Well, that’s on you for using autotools and their ilk :)

                                                                    Projects under 50,000 lines or so only need a build script like this:

                                                                    cd build && clang -Wall -flto ../*.c ../*.cpp
                                                                    

                                                                    For larger projects, modern build systems that output ninja are relatively painless.

                                                                    1. 3

                                                                      It’s relatively painless, but not even close to actually painless.

                                                                      You shouldn’t need a several hundred line script to output a several hundred line build config that a ten thousand line build tool parses to run your compiler. But, C doesn’t include any way to specify how your code should be built and our compilers are crap so incremental and parallel compilation are must haves.

                                                                      Unity builds solve the problem of needing massive build tools, but your code has to be written to work as a unity build from the start and using 3rd party libraries becomes a big pain. It also doesn’t really solve the performance issues but it does help a bit.

                                                                      Compiling your code is one of the biggest pain points with C, and the Jai language is aiming to resolve this by making the build specification part of the language and having a compiler that can build hundreds of thousands of lines per second.

                                                                  2. 2

                                                                    Do you think systems programming is low-level? This is an honest question – whenever I read something about OS or systems development, like distributed platform development, I see fairly high-level things. Hardware development is low-level though… but what are systems if not hardware abstractions, so by definition, a higher level thing?

                                                                    1. 3

                                                                      Yes.

                                                                      Specifically about this James Mickens' essay here: the essay is about the implementations of kernels, virtual machines, device drivers, databases and the like. The pain this essay discusses is specifically the pain of low-level programming, such as how bugs in the thing you’re implementing can also break the tools that you’d like to use to debug it, such as a stray pointer breaking your logging subsystem and now, in Mickens' words, “I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS.”

                                                                      By definition: I think systems programming is “writing stuff intended for applications to be built on top of” so it’s more low-level than applications programming, which consumes the software that systems programmers produce.

                                                                      1. 3

                                                                        An example I have seen is someone fixing a localization bug, and added logging to debug/verify that it’s working. However, logging actually called into his localization code which infinitely indirectly recursed, causing the system to hang on boot. This ain’t quite an example of low level debugging but it was low enough to break tools your tools rely on.

                                                                      2. 3

                                                                        In yosefk’s sense, low-dependency code is easy. This describes quite a lot of, but not nearly all, systems code. (E.g. a special case in the PostgreSQL query planner probably isn’t low-dependency.)

                                                                    1. 11

                                                                      Please stop adding author and summary to titles.

                                                                      1. 5

                                                                        May I ask why? Personally this seems to be useful information to have in a title, especially when the actual title is as vague as this one.

                                                                        1. 5

                                                                          Generally speaking changing titles is not great–it confuses search, it adds opportunities for editorializing, etc.

                                                                          If the title is vague, filling in the story description is usually a better option. That also lets you do things like link out to relevant background material.

                                                                          1. 2

                                                                            If the title is vague, filling in the story description is usually a better option.

                                                                            The real takeaway from the comment. For any new people reading, Lobsters' homepage has a symbol to the right of the title that looks like this…

                                                                            http://graphemica.com/%E2%98%B6

                                                                            …that links to a description giving extra info on the article. Result looks just like the page you’re seeing but with the text right under (example) “via ThisIs_MyName.” That comes from the text field of Submit Story. Most regulars seem to prefer you use it for extra details that might clutter up a title on front page.

                                                                        2. 2

                                                                          Will do.

                                                                          1. 5

                                                                            Please keep adding author and synopsis! I get a lot of value from being able to follow an author’s work and decide to read or not.

                                                                            1. 2

                                                                              It’s fine to add them–just in the story description and not the title. :)

                                                                        1. 5

                                                                          Title should just be “The Night Watch”, and probably have the satire tag.

                                                                          1. 4

                                                                            I don’t think ‘satire’ is a good description of this.

                                                                            1. 3

                                                                              It just happens to be the closest tag which lobsters currently has to fit “funny writing”. There’s also a “rants” tag but Mickens' style is IMO much more audience amusement than catharsis.

                                                                            1. 3

                                                                              See the duplicate thread from 18 days ago: https://lobste.rs/s/rhz2eb/lets_encrypt_has_issued_988_certificates

                                                                              My reply:

                                                                              Thanks to certificate transparency, that is unnecessary. Paypal can watch the CT log and “take legal action or quash the domain registration” whenever they feel like it :)

                                                                            1. 11

                                                                              TL;DR: The argument boils down to the fact that the larger pointer size for pointer-heavy code that tends to be in compilers eliminates the advantages of more registers, because fewer pointers can fit in a cache (and the corresponding instruction stream is also larger and less dense, with similar cache effects).

                                                                              Sounds like Windows needs an x32 ABI.

                                                                              1. 7

                                                                                I speak for all Windows devs when I tell you that the absolute last thing we need right now is another ABI. We already had three on x86 (now four) and are up to two on AMD64, and that assumes you don’t count COM and WinRT as distinct ABIs—but I personally would, bringing the totals to six and four, respectively.

                                                                                1. 2

                                                                                  Could you expand on the different APIs and on how they differ? No criticism, just genuine interest.

                                                                                  1. 6

                                                                                    ABIs, not APIs.

                                                                                    Win32 traditionally had three heavily used ABIs: cdecl, which passes arguments on the stack and assumes caller cleanup; stdcall, which is identical but says the callee cleans up; and fastcall, which is similar to stdcall, but passes the first two arguments (that fit) into IIRC ECX and EDX. As a sweeping and unfair generalization, everything uses stdcall, except when it doesn’t, which you’ll find out when your app compiles fine but behaves bizarrely because you missed importing a header and the function got defined implicitly, because C.

                                                                                    On AMD64, Microsoft merged all three of these into a single calling convention, which I believe they called just “the calling convention,” and which is similar to fastcall, but uses RCX, RDX, R8, and R9 for the first four arguments that fit, and also reserves XMM0 through XMM3 for any floating-point arguments. They also require 32 bytes of stack space to be cleared by the caller. I forget why Microsoft thought this was a good idea (mandatory reserved spill area for the four integer registers?), but I frequently appreciate it when I’m trying to figure the hell happened to the stack in a memory viewer, because it gives me at least some idea where the stack frames are.

                                                                                    Meanwhile, Microsoft knew that XMM was cool, which is why it’s part of the x64 convention, but then Intel introduced some new hotness, which made the vector units 256-bits wide. Oops. So they added vectorcall on both x86 and AMD64, which is identical to either fastcall or “the x64 convention”, respectively, but it uses the XMM and YMM registers for argument passing. (I am waiting for the inevitable VMX2 version of this ABI that uses the ZMM registers. Maybe it was already released. If so, I hope it’s called evenmorefasterercall. Or maybe crosses the streams and they call it rastercall. I’m flexible.)

                                                                                    So that gets to the original “four” and “two” I mentioned for x86 and AMD64, respectively. But wait, there’s more!

                                                                                    Everything I’ve just listed are C calling conventions, but Windows has had COM for forever. COM took advantage of the fact that all the C++ compilers on Windows used exactly the same vtable layout, so it provides a way to call C++ objects as long as they’re sitting behind a vtable (plus provides some standard ways to acquire the proper vtable). Skipping past some details, this is usually called a thiscall, and looks like stdcall (or “that thing we use on x64”) with the exception that the this pointer is passed in ECX/RCX. Whether this counts as a separate ABI is left to the reader; I’d be inclined to say yes.

                                                                                    And finally that leaves us with WinRT. WinRT is like COM, but now with full-blown objects, complete with subclasses and properties and exceptions and all kinds of stuff. If you squint, WinRT’s ABI looks identical to COM, but the thing is that it also standardizes arguments and returns to support things like throwing C++-esque exceptions across ABI boundaries—something COM definitely cannot do. Thus, while you could (correctly) say that the arguments are mechanically thiscall, and therefore stdcall/“the x64 thing”, the fact that these functions require special setup and teardown makes me mentally classify them as a different ABI.

                                                                                    So there you’ve got it. Six calling conventions on x86, four on AMD64.

                                                                                    1. 1

                                                                                      ABIs, not APIs.

                                                                                      Oooops, sorry!

                                                                                      So there you’ve got it. Six calling conventions on x86, four on AMD64.

                                                                                      Wow, thanks for your explanation! Very interesting!

                                                                                2. 4

                                                                                  I’m not aware of anyone actually using the x32 ABI in practice or any real support for it? Last time I went looking it seemed that documentation and distro support/prebuilt binaries were really thin on the ground.

                                                                                  1. 2

                                                                                    Gentoo supports the x32 ABI. Just select the x32 profile and it works out of the box :)

                                                                                    https://wiki.gentoo.org/wiki/Project:Multilib/Concepts

                                                                                    https://wiki.gentoo.org/wiki/Profile_(Portage)

                                                                                    1. 1

                                                                                      There were some rumblings that Arch might adopt it, but I don’t know if anything ever came of it.

                                                                                      It was mostly just a jocular reference, though, in that the problems described in TFA are exactly the problems x32 was designed to solve.

                                                                                      As for TFA itself, I’m kinda surprised that it makes a difference. If your project is big enough that cache pressure on a 64-bit arch becomes a problem, I’d think you’d also be running up to the 4GB limit, meaning a 32-bit application isn’t going to work at all.

                                                                                      1. 3

                                                                                        It was mostly just a jocular reference

                                                                                        I was actually hoping you might contradict me with information about support for it that I had just missed. Alas.

                                                                                        cache pressure on a 64-bit arch becomes a problem

                                                                                        Always a problem. On current desktop CPUs, L1 is about 64kB, L2 about 256kB and L3 about 8MB, and there are big differences in latency between them. You can write a program which randomly reads and writes to a 16MB array, thrashes every cache level and spends almost all its time in ~100ns memory stalls on roughly every other read. The smaller your data, the fewer and less severe memory stalls you’ll have operating on it.

                                                                                        1. 2

                                                                                          If your project is big enough that cache pressure on a 64-bit arch becomes a problem, I’d think you’d also be running up to the 4GB limit

                                                                                          Eh? There’s a few orders of magnitude between cache size (maybe low-double-digit megabytes of L3) and 4GB…

                                                                                          1. 2

                                                                                            I was thinking being able to have all of your source/project files/whatever in memory at once, but giving it a moment’s more thought it really doesn’t matter, since the OS’s buffer cache can be larger than 4GB and mapped in and out as needed. You’d still need to syscall out to (re)open the files, but that’s relatively quick and if they’re already in the buffer cache it’s only a tiny bit of overhead.

                                                                                            Maybe the linker for huge binaries with lots of objects or something? Probably a rare enough case to not matter.

                                                                                            So yeah, nevermind.

                                                                                      2. 3

                                                                                        Naively I’d think that access to the vectorization instructions would overwhelm the increased cache miss rate but I might be misunderstanding visual studio’s workload.

                                                                                        1. 7

                                                                                          Vectorization instructions are mainly useful in array operations, and Visual Studio doesn’t have a lot of that. The larger register set would make more of a difference, but even then it’s not much. In general for pointer-heavy cache-un-local code (VS, and most code people use regularly), optimizations like vectorization and better register utilization can’t help much; performance wins generally come from application-level improvements such as data layout optimization. By sticking to x86 rather than x64 you get a big data layout optimization “for free,” modulo unhappy extension writers. Source: I work on the MSVC optimizer, so “common code we can’t optimize” is a fairly constant subject.

                                                                                      1. 2

                                                                                        Very interesting data.

                                                                                        I don’t agree with the author’s comment about “we’re grateful to everyone who contributes to the site” … It certainly doesn’t feel like that

                                                                                        1. 1

                                                                                          Yeah SO is pretty shitty to contributors.

                                                                                          But hey, what’s the alternative? ExpertSexChange?

                                                                                        1. 4

                                                                                          Sometimes I’ll even deliberately push a commit and immediately revert it because I want a record of one possible experiment was.

                                                                                          Isn’t that part of the point of a branch?

                                                                                          The funny thing is that if Github allowed typical FTP then I suspect Zach Holman might just use that instead. I fail to see the point of a VCS if the commit history is completely mired in useless commit titles.

                                                                                          My first question for Zach would be if his problem isn’t with the Git process as an idea, but just the fact that committing and branching takes time? I suspect he enjoys reverting commits just because it’s a faster process (especially if you use a GUI app like Github’s Mac client).

                                                                                          1. 3

                                                                                            Why would someone use VCS to store deliberately reverted commits? Because it’s more convenient than a zip file of patches? History is not the only feature provided by VCS.

                                                                                            1. 2

                                                                                              A zip file of patches still hangs on the idea of a history. “finalFinal2.zip” and such like that. I’m talking about pushing and live-updating a dev server through FTP. It’s the closest thing to madness, but for the use-cases that Zach has; it might fit much better for him.

                                                                                              There are a lot of wonderful things about VCS and Git, but it seems Zach uses all of these as minimally as possible.

                                                                                            2. 1

                                                                                              but just the fact that committing and branching takes time

                                                                                              Bingo.

                                                                                              I have a script that creates a new randomly-named branch if I’m on master, checks out that brach, stages all my changes, commits my changes, and runs some fast tests.
                                                                                              As I work, I might think of a good name for the feature branch and rename it. When I’m done, I’ll squash commits (as needed) and merge.

                                                                                              Without the little script, this would be 5 commands per iteration plus the overhead of thinking of a good branch name. That’s painful.

                                                                                            1. 4

                                                                                              I think it’s unlikely that is the fastest way to test for primes in java.

                                                                                              1. 1

                                                                                                It also has the amusing characteristic of crashing if your number is larger than the largest contiguous span of free memory in the JVM heap.

                                                                                                1. 1

                                                                                                  The JVM heap can keep growing as long as the OS allocator never says no.

                                                                                                  Virtual memory compression at the OS level should be able to compress strings of char[n] to char[lg(n)] with run-length encoding.

                                                                                                  (Well, in theory. I never tried running it :P)

                                                                                                  1. 3

                                                                                                    You’ll run into a much tighter, much stupider limit much sooner than that becomes relevant: the longest possible Java array only has 2 billion-ish elements (2^31-1, to be precise, because Java arrays are indexed by Java ints, which are 32 bit and signed). I can comfortably fit the largest possible char array—4GB—uncompressed in memory on every computer I use regularly.

                                                                                                    1. 1

                                                                                                      No, it can’t. The JVM has a maximum heap size set at start up, which defaults to 1GB or less. And even if the OS compresses that memory, the JVM allocator still counts the uncompressed size.

                                                                                                      1. 1

                                                                                                        Ah ok. I guess -Xmx999999g is in order.

                                                                                                  2. 1

                                                                                                    Sounds like the regex compiler has its work cut out.

                                                                                                    Naw, this is just for fun :)

                                                                                                  1. 1

                                                                                                    Besides the “this is the browser’s job to display the top-level domain differently” or whatever comments, how difficult would it be for Let’s Encrypt to be given a list of commonly-phished websites and delay issuance (and notify the real paypal so they can take legal action or quash the domain registration)?

                                                                                                    1. 5

                                                                                                      It’s a one line perl script to see if a domain name matches a ban list, but who decides what’s in the ban list? (Who decides who decides what’s in the ban list?)

                                                                                                      1. 1

                                                                                                        LE already uses the google safe browsing list, so using the inverse of it (the legitimate websites that are imitated by the entries on the safe browsing list) isn’t really superbly controversial.

                                                                                                        Creating an audit log of all certificates that have been delayed/quashed due to this procedure (along with the legal entity responsible) seems also completely doable.

                                                                                                      2. 3

                                                                                                        Thanks to certificate transparency, that is unnecessary. Paypal can watch the CT log and “take legal action or quash the domain registration” whenever they feel like it :)

                                                                                                      1. 4

                                                                                                        I mean, this is pretty much all bad and written as if by a marketing communications manager, but this bit right here is impressive:

                                                                                                        And my personal favorite: you could even copy-and-paste part of your code from security handbooks! If for example, you realize you use weak cryptography, you could quickly solve the problem by taking free, plug-and-play cryptographic functions implementations that are available online: the well-known OpenSSL could be such an example. Begone, countless hours of development!

                                                                                                        1. 1

                                                                                                          That’s hilarious.

                                                                                                        1. 3

                                                                                                          I had an instinctual negative reaction to this, and I’ve had to think about why. I think part of it comes from $WORK, where we have a service (for one of the Monopolistic Phone Companies) where due to service level agreements, we have redundant servers in redundant geographical locations. So at the very least, you are talking about at least four different IP addresses (more actually) required to run the program.

                                                                                                          Second, having a development system on a production server would probably not pass security—anything to make it harder to further exploit a break-in. Even including unnecessary code (such as in a shared library that includes all functions whether used or not) makes me uneasy.

                                                                                                          Third, having to muck with the source code for configuration complicates version control. I view config files as the variants to an invariant program, like an IP address.

                                                                                                          1. 1

                                                                                                            Editing the source code of the program really shouldn’t be harder than editing a config file. In my workflow, either change requires building a self-contained image, testing it on canaries, and then deploying that image.

                                                                                                            1. 2

                                                                                                              At $WORK, we have a repository for source code and another one for configuration. Configuration can change (IP address, hostname, location of other services) without having to update the already tested code, and the code can change (bug fixes, new features, etc) without having the configuration change (assuming the code changes don’t require an update to the config, which at $WORK, is most often actually).

                                                                                                              With your method, does that mean your canary boxes are identical to production? How is that possible?

                                                                                                              1. 1

                                                                                                                Canary boxes are practically identical to production because the same requests going to production boxes are mirrored to canary. Responses from the canary can be thrown away in most cases.

                                                                                                                That said, they don’t have to be identical for my argument to work: Config and code changes can both cause failures so they should be tested the same way.

                                                                                                          1. 3

                                                                                                            I have the hifive1 Arduino sized RISC-V hardware, it’s very powerful for embedded size hardware, I hope they make desktop sized systems soon.

                                                                                                            Also, why not purchase a secure boot signing key from Microsoft and set up a web service to sign anything? Is that possible?

                                                                                                            1. 2

                                                                                                              You’d probably have to sign a contract with MS saying “I won’t sign anything that I haven’t manually reviewed” or something even more restrictive. If you break the contract, you’ll have to pay damages.

                                                                                                              They’ll also add your promiscuous signing key to the blacklist so that up-to-date machines won’t accept it.

                                                                                                              1. 2

                                                                                                                I think Red Hat has a key that works: https://mjg59.dreamwidth.org/12368.html