1. 76

    Imagine you lived in a country that has a strange tradition where children, once a year, go to stranger’s homes and are given candy. Then, someone, in order to study the safety of this tradition, decide to give out candies laced with a mild and provably non-lethal toxin to children. This someone has a fool-proof plan to inform the children’s parents before anyone gets hurt. Not all parents tests candies for toxins, but enough do – since things like this can happen and parents of this country takes safety reasonably seriously. One parent detected this toxin in the children’s candies. All the parents are informed and said candies were thrown out. No harm, no foul?

    Imagine you lived in a country where no neighbors can be trusted. Imagine you worked in a low trust environment. Imagine stopping the OSS model because none of the contributors can be trusted.

    That’s not the kind of world we want to operate in.

    1. 32

      I think this sums up why I felt a bit sick about that whole story. It undermines the community and is essentially antisocial behaviour disguised as research. Surely they could have found a way to prove their point in a more considerate way.

      1. 8

        Surely they could have found a way to prove their point in a more considerate way.

        Could you propose some alternative approaches? As the saying goes, POC || GTFO, so I suppose the best way to prove something’s vulnerability is a harmless attack against it.

        The kernel community appears to assume good faith in every patch they receive from random people across the Internet, and this time they get mad when the researchers from UMN prove this wishful assumption to be false. On the other hand, cURL goes to great lengths to prevent the injection of backdoors. The kernel is clearly more fundamental than any userland utilities, so either the cURL developers are unnecessarily cautious against supply chain attacks, or the kernel hackers are overly credulous.

        1. 16

          Another possible approach is to ask the lead maintainers if you can perform such an experiment. Linux has a large hierarchy and I think the top level maintainers pull huge patch sets as a bundle.

          If they had permission to use an unrelated e-mail address then it could be pretty much as good. Honestly I would think a umn.edu address would give more credence to a patch, since it seems like its from someone a reputable institution.

          Of course they might not agree, in which case you don’t have consent to do the research.

          1. 18

            This. You ask for permission. Talk to the kernel maintainers, explain your research and your methods, and ask if they want to participate. You can do things like promise a maximum number of bogus patches and a timeframe where they may occur, so people know they won’t get deluged with crap for the rest of time. You could even make a list of email addresses the patches will come from ahead of time and hand it to someone trustworthy involved in the kernel project who won’t be reviewing those patches directly, so once the experiment is over they can easily revert all the bad patches even if the researcher is hit by a bus in the mean time. It’s not that hard to conduct this sort of research ethically, these researchers just didn’t do it.

            1. 6

              That’s a fair point, but I want to point out that the non-lead reviewers are still unknowingly participate in the research, so that’s still not super ethical to them. Doing so merely shifts the moral pressure to the lead maintainers, who need to decide whether or not to “deceive” the rest of the community.

              But yeah, only lead reviewers can revert commits and have enough influence in the tech world, so getting their permission is probably good enough.

              1. 6

                A top comment in a cousin thread on HN suggests, that with proper procedure, AFAIU actually all reviewers could be informed. The trick seems to be to then wait some long enough time (e.g. weeks or more) and send the patches from diverse emails (collaborating with some submitters outside your university). There should be also some agreed upon way of retracting the patches. The comment claims that this is how it’s done in the industry, for pen testing or some other “wargames”.

            2. 5

              In the subsystems that I’ve contributed to, I imagine that it would be possible to ask a maintainer for code review on a patchset, with phrasing like, “I am not suggesting that this be merged, but I will probably ask you to consider merging it in the future.” After the code review is given, then the deception can be revealed, along with a reiterated request to not merge the patches.

              This is still rude, though. I don’t know whether it’s possible to single-blind this sort of study against the software maintainers without being rudely deceptive.

              1. 2

                I think you could ask them if you can anonymously submit some patches sometime over the next few months and detail how some of them will contain errors that you will reveal before merging.

                They might say no, but if they say yes it’s a reasonably blind test, because the maintainer still won’t know which patches are part of the experiment and which are not.

                Another way to do it would be to present the study as something misleading but also do it in private and with compensation so that participants are not harmed. Say you just want to record day-in-the-life stuff or whatever and present them with some patches.

                Finally, you could look at patches historically and re-review them. Some existing patches will have been malicious or buggy and you can see if a more detailed review catches things that were missed.

          2. 17

            This research was clearly unethical, but it did make it plain that the OSS development model is vulnerable to bad-faith commits. I no longer feel what was probably a false sense of security, running Linux. It now seems likely that Linux has some devastating back doors, inserted by people with more on their minds than their publication records.

            1. 15

              This is something every engineer, and every human needs to be aware at some point. Of course, given enough effort, you can fool another human into doing something wrong. You can send anthrax spores via mail, you can fool drivers to drive off a cliff by carefully planted road signs, you can fool a maintainer into accepting a patch with a backdoor. The reason it doesn’t happen all the time is that most people are not in fact dangerous sociopath having no problem causing real harm just to prove their point (whatever that is).

              The only societal mechanism we have for rare incidents such as this one is that they usually get eventually uncovered either by overzealous reviewers or even by having caused some amount of harm. That we’re even reading about patches being reverted is the sign that this imperfect mechanism has in fact worked in this case.

            2. 2

              This country’s tradition is insanely dangerous. The very fact that some parents already tested candy is the evidence that there was some attempts to poison children in the past — and we don’t know how many of these attempts actually succeeded.

              So, if we assumed that the public outcry from this event lead to all parents testing all the candy, or changing tradition altogether, then doing something like this would result in more overall good than evil.

              1. 10

                Meanwhile in real life, poisoned Hallowe’en candy is merely an urban legend: According to Snopes, “Police have never documented actual cases of people randomly distributing poisoned goodies to children on Halloween.”

                The very fact that some parents already tested candy is the evidence that there was some attempts to poison children in the past

                Not really. Again in the real world, hospitals run candy testing services in response to people’s fears, not actual risks. From the same Snopes article: “Of several contacted, only Maryland Hospital Center reported discovering what seemed to be a real threat — a needle detected by X-ray in a candy bar in 1988. … In the ten years the National Confectioners Association has run its Halloween Hot Line, the group has yet to verify an instance of tampering”.

            1. 1

              And he didn’t even mention Hasura once. It made development process so much easier — there’s no need for any server code for read-only queries that go straight to db. Just update schema, click on a couple of buttons in Hasura admin panel, and new data is exposed to the client right away.

              1. 4

                No static typing means caring about type validation

                I’m not sure how you can call “static typing” something that is being done at run time. In my opinion it’s still (value) type validation, only someone else is doing it instead of you.

                Pagination and filtering is not so simple

                I agree, but standardizing on limit, offset pagination instead of keyset is dumb for APIs. Limit/offset assumes that the client needs to manage the cursor state locally, (what does offset 100 mean with regards to the elements we have already seen?)

                1. 1

                  I’m not sure how you can call “static typing” something that is being done at run time.

                  Both my client (React) and server (Node) Typescript code check against types that are generated from GraphQL schema at compile time.

                  1. 1

                    My impression from reading TFA was that he was worried about mangled payloads that the API receives. Somehow this would be “static type checked” for GraphQL. I don’t see it.

                    I don’t have enough experience with the typescript ecosystem to have a meaningful opinion, but from the sides, calling typescript -> javascript “compilation” is not something I can take seriously. Is there anything that sits in the final javascript that ensures received data types match expectations? That would seem like a pretty large overhead.

                1. 13

                  My first two contributions to programs anyone has heard of were to the linux kernel, for which I implemented two things. The second of them runs to this day and processes outbound TCP/UDP packets on countless hardware, and I’m very proud of how it’s largely unchanged after decades. The first got linux banned from some networks by causing ARP storms.

                  There’s a lesson to be learned from this. Two lessons maybe, because Linus never said a harsh word after the first.

                  1. 2

                    If I understand his logic correctly, he would have harsh words for maintainers who approved your change, but not for you.

                    1. 11

                      Uhm, maintainers… this happened back when the linux world was a small clique and Linus helped people install linux. He helped me install linux, in fact ;)

                      As long as I followed kernel development, he didn’t swear at anyone for any bug. I’d have gotten an earful if I had defended my code and suggested that other OSes involved in the ARP fiasco were to blame and ought to change.

                      1. 1

                        [Linus] helped me install linux, in fact

                        Which year was this? How far from 1991? :-)

                        1. 2

                          ≤2 years after that July 1991 posting. I installed in July 1992 and deleted my MSDOS partition early in August. Can’t say precisely when I wrote the ARP code, since I had a rather unfortunate file system event a little later.

                          1. 1

                            “Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it ;)”

                            1. 1

                              Only fools ask root for a favour, if that favour involves doing file system operations manually (instead of waiting for the chore to be done later by a crontab that would remember to enable tape backups).

                  1. 28

                    I love Postgresql, and I’m really grateful for posts like this to balance my opinion and present a well-argumented counter-argument. However, when I read posts like this I mention a pattern: all these downsides seem to be relevant to really huge, perfomance-sensitive projects with enormous loads. The kind of projects where you likely have over a hundred of developers, and probably should move away from one master relational database to micro-services and message queues as a backbone of your architecture.

                    What I’m saying, this kind of criticism just highlights the fact that Postgresql is probably the best choice if you’re smaller than that, both in terms of load and team size.

                    1. 18

                      I almost hit the XID wraparound (I was 2-3 days from an outage; it was bad enough that AWS emailed me) when my company only had 4-5 devs and 100 customers. And I’ve hit connections-related performance issues at least four or five times through the last five years, at relatively modest scale (<100 application servers, in an overwhelmingly read-heavy workload). This affected us as recently as yesterday, as we are bringing a Timescale DB (which is a Postgres extension) into production and we are faced with the prospect of tiered pgBouncers or breaking the Timescale access into its own microservice.

                      I love Postgres but these are real and they can hit earlier than you expect.

                      1. 4

                        I love Postgres, and I was strongly considering moving to it…but for our use case, it simply requires too much care and feeding. We ship databases to customers in appliance form, meaning all the maintenance has to be invisible and automatic and Postgres simply isn’t there.

                        1. 6

                          Having worked for a company that did that and been responsible for Postgres tuning, I say it can be done. In nearly 15 years of shipping out a postgresql db as part of an appliance, I have not seen any of these problems.

                          Edit: Except for long upgrade times. That one is a PITA.

                          JADP.

                          1. 4

                            I’d love to hear about your experience if you have time.

                            Also, I’m drawing a blank on “JADP”…

                            1. 3

                              Just Another Data Point?

                              1. 1

                                Just A Data Point. A weird acronym I picked up from old-timey online fora like USENET, and The Well.

                                I probably can’t say too much more about my experience postgres tuning as.

                                1. It was for a company, and might be considered propietary information.
                                2. It was about 5 years ago and I really don’t recall all that well what I did.

                                sorry, just know that these are really rare problems if you’re dealing with the limited scale inherent in incorporating postgresql as part of an appliance. Most of them deal with syndication, or ginormous tables. They’re web-scale problems, not appliance scale problems

                            2. 2

                              what do you use instead?

                              1. 1

                                What are you planning on using instead?

                                1. 1

                                  I do this as well. There’s definitely a discovery period but I’ve reached the point that I almost never have to check anything on the database side for roughly 200 customers, running varying versions from 9.6 to 11.9.

                                2. 4

                                  Definitely echo that these problems (and others) can hit you way before you get to 100 devs. We were running into the pains mentioned in this article (which admittedly is a more general crritique of SQL databases, and lands on MySQL over Postgres) at more like 10 developers.

                                  It absolutely isn’t that hard to run into the pitfalls of SQL databases at relatively small scale, especially if you’re using them for OLTP workloads in services where uptime/response times matter.

                                3. 5

                                  all these downsides seem to be relevant to really huge, perfomance-sensitive projects with enormous loads. The kind of projects where you likely have over a hundred of developers

                                  A number of these issues affect “normal” users of PostgreSQL as well:

                                  • Replication is something you may want even on smaller use cases.

                                  • “Wasted” space from the “update is really delete + insert”-paradigm can be a problem even on fairly small use cases (i.e. tens of millions of rows). It can make some simple operations rather painful.

                                  • Lack of query hints is pretty annoying, especially for smaller users who don’t have a dedicated DBA with a Ph.D. in the PostgreSQL query planner. It’s also a massive help in development; want to try a new index? Now it’s a drop index, create index, wait, analyse`, wait some more, run some queries, discover that didn’t do what you expected, drop the index, create a different one, wait, etc. It’s very time-consuming and much of the time is spent waiting.

                                  1. 5

                                    Nice example: “complaining” that it is hard to tune it for a million concurrent connections.

                                    Haven’t read it to the end yet, almost hoping to see an ending like “of course I’m happy to have a free DB that gets me in trouble for a million concurrent connections instead of breaking my bank at 1000 connections or when somebody touches advanced debugging like Oracle or

                                    1. 5

                                      FYI you should have read it through to the end, as the article does end on that note.

                                  1. 1

                                    I’ll go just with those that I’m carrying around in my bag at the moment.

                                    The Back Channel by William Burns. I would give it 6/10, as it’s written by a career beurocrat, but it’s a good alternative narrative to how a lot of events have been covered in the last 30 years by the anti-american propaganda, and since he’s not really trying to reveal any secrets or create any sensations, all of his views are pretty well backed up by public sources, which just have fallen out of the spotlight.

                                    I’m also switching to How to win on the battlefield by Rob Johnson, Michael Whitby and John France from time to time. It’s a strange compilation of combat tactic and strategies from all historic eras, and I’m still not sure if it’s really a history book or a army professional manual. But for a person who never had any military training, it still reads very easily, and even gives an impression that you understand something about commanding armies in battle — although as a strategy game enthusiast, all my attempts to put these new ideas to work in games like Steel Division, Wargame and Unity of Command have fallen flat. May be it’s because strategy games don’t actually test strategy skills and instead are more like puzzles or click-speed tests, or it just may be the case that I’m very bad at it. Still, highly recommend if you’re into this kind of thing, 8/10.

                                    1. 1

                                      Gorgeous. Can’t believe it predates vaporwave aesthetic by 20 years.

                                      1. 1

                                        Rewriting the system that I spent the last 3 months designing and building. I didn’t have written down requirements, so I have carefully researched and written them down myself, so we would all be on the same page — but turns out that PM didn’t actually read them, and we discovered that system does not behave as he wants it to only on Thursday. We need to start testing it tomorrow and release to public in 2020.

                                        Fun times.

                                        1. 2

                                          A bit off-topic, but are there videos of this course anywhere online? It looks extremely interesting.

                                          1. 1

                                            Since the recordings of the course contain student information (e.g., faces, names) we cannot release them per federal guidelines (FERPA).

                                          1. 4

                                            How many other languages have this kind of foot shotguns with default compares and parsing being locale aware? To me this seems like it is really an API design problem and something that is not the programmers fault.

                                            1. 5

                                              Why do you describe it as a footgun? C# was created first and foremost for windows desktop application development. Windows has users all around the world, expecting dates, currencies and numbers in their own format. So, for the main use-case of C#, this behaviour was a huge help, not a problem.

                                              I mean, when you work on your desktop that has local settings for your culture, why wouldn’t you expect any program that you run to parse dates as you have said operating system to do it?

                                              1. 1

                                                I would expect a things like DateTime.Parse and double.Parse to not be locale-aware by default. When I say double.Parse I already have a mental model of how the data looks like and changing an external variable should not change that. Changing an operating systems language should not change how to parse numbers by default.

                                                Because of locale aware parsing I never know if I should use 1.9 or 1,9 in Finland, and this problem just starts from here, for a real problem with Java see the following issue.

                                                For ex. For the fi_FI locale , the minus sign is Unicode Character (U+2212) in CLDR , whereas it is Unicode Character (U+002D) in JRE(COMPAT). Hence, there is a parsing exception while parsing the hyphen-minus(U+002D) in JDK 9 while this is successful in JDK 8.

                                                Nobody actually uses U+2212 (the minus sign) for negative numbers in Finland, because (almost) nobody knows that such even exists.

                                                There is a place for locale dependent parsing, but it just should not be the default.

                                                1. 2

                                                  To be honest, I think that strings that are machine or config-readable should be a completely different type than user-readable and user-writable strings, without any implicit conversions, and with machine-readable strings limited to ASCII.

                                              2. 4

                                                C does — some of the standard library parsing/printing functions implicitly use the current locale. For example, the (macOS) man pages for printf and strtod say “The decimal point character is defined in the program’s locale (category LC_NUMERIC).”

                                                IIRC, in C there is no standard cross-platform way to parse and print numbers without locale dependency, or using a fixed locale. BSD and Apple platforms have some extensions but they aren’t available on Linux nor Windows, and probably vice versa.

                                                1. 2

                                                  Of course C does that… I was just thinking that C (and C++) are probably on that list. Maybe, hopefully, the world will become a saner over time.

                                                  1. 4

                                                    It’s not entirely clear to me what the defaults should be. POSIX2008 provides three variants:

                                                    • Use the global (per-process) locale.
                                                    • Use the almost-global (per-thread) locale.
                                                    • Use the locale specified in an argument.

                                                    Of these, the second seems completely useless (it’s useful historically for implementing the third mode), but why would a thread have a locale different from the rest of the program? The only reason I can think of is in web applications where you’d want to set the locale to the client’s locale and have a pool of threads to handle requests and are too lazy to pass a locale_t around in the request object.

                                                    The two locales that make sense to me are:

                                                    • The user’s current locale.
                                                    • A locale specified by the caller that’s context dependent (for example, encoded when parsing some specific file or protocol format).

                                                    C is in a slightly odd situation of predating anyone thinking about locales so printf and friends were originally in what is now called the C or POSIX locale (basically, American using 7-bit ASCII). The idea of a global locale was introduced without any way of specifying a non-default locale. This was really problematic in multithreaded code because it made functions like printf really problematic (especially given that some parts of the locale state, such as multibyte conversion, are stateful and that state is now shared across threads). Eventually POSIX2008 (via an Apple extension) added explicit locale objects but at this point C is a mess.

                                                    For anything without that legacy it seems that defaulting to the current user’s locale is probably the least bad option. Any software that’s parsing a specific format date, currency, or whatever should handle it explicitly. OmniOutliner is my favourite example of someone failing to do this: they provide a single currency type that maps to the current user’s locale. If I safe a file with a column of £s and send it to someone in the US, they’ll load it as a column of $s. If you’re defining a file format that contains dates, currencies, or numbers in string form and not thinking about localisation then you’re probably doing something wrong.

                                                  2. 1

                                                    In fact, I found this link via a reply to a stackoverflow (or unix.stackexchange, can’t find it anymore) answer to the question “how to display each character of a variable to a new line in shell?”. echo "$VAR" | fold -w1 and echo "$VAR" | grep -o . were both suggested, but it was told that fold fails the test (implying it behaves differently with different locales) with for example Arabic:

                                                    echo "مرحبا" | fold -w1
                                                    echo "مرحبا" | grep -o .
                                                    

                                                    As another example there’s also this colourful and detailed commit in mpv that rants about locale issues in C.

                                                  3. 2

                                                    To me this seems like it is really an API design problem

                                                    At least in the case of regex, it very much isn’t – any language that is properly aware of Unicode should also default to using Unicode’s rules here. That does mean you get expanded definitions of certain character classes like “digit”, but the solution is not to make everyone else in the world have to jump through hoops to get support for their “special characters”, it’s to be explicit about what you really wanted in your regex. If you really want to match only [0-9], you can still do that and should have done that to start with.

                                                  1. 9

                                                    It seems that author assumed that PG’s political points are mistaken the moment he realised that PG was “reactionary” — which is very close to the very same logical mistake he’s accusing PG of making.

                                                    1. 25

                                                      I bought one last week and have used it for 7 days now. I was in an initial hype phase as well, but I am more critical now and doubting whether I should return it.

                                                      Performance of native apps is as great as everyone claims. But I think it is a bit overhyped, recent AMD APUs come close in multi-core performance. Of course, that the Air works with passive cooling is a nice bonus.

                                                      Rosetta works great with native x86_64 applications, but performance is abysmal with JIT-ing runtimes like the JVM. E.g. JetBrains currently do not have a native version of their IDEs (JVM, but I think they also use some other non-Java code) and their IDEs are barely usable due to slowness. If you rely on JetBrains IDEs, wait until they have an Apple Silicon version.

                                                      Also, performance of anything that relies on SIMD instructions (AVX, AVX2) is terrible under Rosetta. So, if you are doing data science or machine learning with heavier loads, you may want to wait. Some libraries can be compiled natively of course, but the problem is that there is no functioning Fortran compiler supported on Apple Silicon (outside an experimental gcc branch) and many packages in that ecosystem rely on having a Fortran compiler.

                                                      Another issue with Rosetta vs. native in development is that it is very easy to get environments where native and x86_64 binaries/libraries are mixed (e.g. when doing x86_64 development and CMake building ARM64 objects unless you set CMAKE_OSX_ARCHITECTURES=x86_64), and things do not build.

                                                      Then Big Sur on Apple Silicon is also somewhat beta. Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well). When working longer, this issue disappears, but it’s annoying nonetheless.

                                                      If you haven’t ordered one, it’s best to wait a while until all issues are ironed out. There is currently a lot of (justified hype) around Apple Silicon, but that doesn’t mean that the ecosystem is ready yet. Unless all you do is web browsing, e-mailing, and an occasional app from the App Store.

                                                      Aside from this, I think there are some ethical (sorry for the lack of a better term) issues with newer Apple models. For example, Apple excluding their own services from third-party firewalls/VPNs, no extensibility (reducing the lifespan of hardware), and their slow march to a more and more closed system.

                                                      Edit: returned and ordered a ThinkPad.

                                                      1. 9

                                                        it’s best to wait a while

                                                        If you need a macbook now , for whatever reason, buying one with an Arm chip does sound the most future-proof option. The Intel ones will be the “old” ones soon, and will then be 2nd rate. It’s what happened with the PowerPC transition as well.

                                                        1. 2

                                                          If only there would be the Macs with 32GB RAM I would buy one as I was in need. However due to that, I bought 32GB 13” MacBook Pro instead. I will wait for polishing out the ARMs before next upgrade.

                                                          1. 1

                                                            From what I read, you get way more bang for your RAM in Apple processors. It’s all integrated on the same chip so they can do a lot of black magic fuckery there.

                                                            1. 1

                                                              In native applications - I am pretty sure that this works well, however as an Erlang/Elixir developer I use 3rd party GCed languages and DBs that can use more RAM anyway. However the fact that it is possible to run native apps from iOS and iPad could save some RAM on Slack and Spotify for sure.

                                                              1. 2

                                                                What I mean is, they probably swap to NAND or something, which could very likely be similar performance-wise to RAM you’d find on a x64 laptop (since they have a proprietary connection there instead of NVMe/M.2/SATA). Plus I imagine the “RAM” on the SoC is as fast as a x64 CPU cache. So essentially you’d have “infinite” RAM, with 16gb of it being stupid fast.

                                                                This is just me speculating btw, I might be totally wrong.

                                                                Edit: https://daringfireball.net/2020/11/the_m1_macs CTRL+F “swap”

                                                                1. 1

                                                                  Just wondering if you had any take on this, idk if I’m off base here

                                                          2. 4

                                                            Lots of valuable insights here and I’m interested in discussing.

                                                            Performance of native apps is as great as everyone claims. But I think it is a bit overhyped, recent AMD APUs come close in multi-core performance. Of course, that the Air works with passive cooling is a nice bonus.

                                                            Sure, but the thing is that the AMD 4800U, their high-end laptop chip, runs at 45W pretty much sustained, whereas the M1 caps out at 15W. This is a very significant battery life and heat/sustained non-throttled performance difference. Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                            Rosetta works great with native x86_64 applications, but performance is abysmal with JIT-ing runtimes like the JVM. E.g. JetBrains currently do not have a native version of their IDEs (JVM, but I think they also use some other non-Java code) and their IDEs are barely usable due to slowness. If you rely on JetBrains IDEs, wait until they have an Apple Silicon version.

                                                            Yeah, I didn’t test anything Java. You might be right. You also mention Fortran though and I’m not sure how that matters in 2020?

                                                            Another issue with Rosetta vs. native in development is that it is very easy to get environments where native and x86_64 binaries/libraries are mixed (e.g. when doing x86_64 development and CMake building ARM64 objects unless you set CMAKE_OSX_ARCHITECTURES=x86_64), and things do not build.

                                                            This isn’t as big of a problem as it might seem based on my experience. You pass the right build flags and you’re done. It’ll vanish in time as the ecosystem adapts.

                                                            Then Big Sur on Apple Silicon is also somewhat beta. Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well). When working longer, this issue disappears, but it’s annoying nonetheless.

                                                            Big Sur has been more stable for me on Apple Silicon than on Intel. 🤷

                                                            If you haven’t ordered one, it’s best to wait a while until all issues are ironed out. There is currently a lot of (justified hype) around Apple Silicon, but that doesn’t mean that the ecosystem is ready yet. Unless all you do is web browsing, e-mailing, and an occasional app from the App Store.

                                                            I strongly disagree with this. I mean, the M1 MacBook Air is beating the 16” MacBook Pro in Final Cut Pro rendering times. Xcode compilation times are twice as fast across the board. This is not at all a machine just for browsing and emailing. I think that’s flat-out wrong. It’s got performance for developers and creatives that beats machines twice as expensive and billed as made for those types of professionals.

                                                            Aside from this, I think there are some ethical (sorry for the lack of a better term) issues with newer Apple models. For example, Apple excluding their own services from third-party firewalls/VPNs, no extensibility (reducing the lifespan of hardware), and their slow march to a more and more closed system.

                                                            Totally with you on this. Don’t forget also Apple’s apparent lobbying against a bill to punish forced labor in China.

                                                            1. 19

                                                              You also mention Fortran though and I’m not sure how that matters in 2020?

                                                              There’s really rather a lot of software written in Fortran. If you’re doing certain kinds of mathematics or engineering work, it’s likely some of the best (or, even, only) code readily available for certain work. I’m not sure it will be going away over the lifetime of one of these ARM-based notebooks.

                                                              1. 4

                                                                I’m not sure it will be going away over the lifetime of one of these ARM-based notebooks.

                                                                There will be gfortran for Apple Silicon. I compiled the gcc11 branch with support and it works, but possibly still has serious bugs. I read somewhere that the problem is that gcc 11 will be released in December, so Apple Silicon support will miss that deadline and will have to wait until the next major release.

                                                                1. 2

                                                                  Isn’t Numpy even written in FORTRAN? That means almost all science or computational anything done with Python relies on it.

                                                                  1. 6

                                                                    No, Numpy is written in C with Python wrappers. It can call out to a Fortran BLAS/LAPACK implementation but that doesn’t necessarily need to be Fortran, although the popular ones are. SciPy does have a decent amount of Fortran code.

                                                                  2. 1

                                                                    Wow, who knew.

                                                                    1. 23

                                                                      Almost anyone who does any sort of scientific or engineering [in the structural/aero/whatever sense] computing! Almost all the ‘modern’ scientific computing environments (e.g. in python) are just wrappers around long-extant c and fortran libraries. We are among the ones that get a bit upset when people treat ‘tech’ as synonymous with internet services and ignore (or are ignorant of) the other 90% of the iceberg. But that’s not meant as a personal attack, by this point it’s a bit like sailors complaining about the sea.

                                                                      Julia is exciting as it offers the potential to change things in this regard, but there is an absolute Himalaya’s worth of existing scientific computing code that is still building the modern physical world that it would have to replace.

                                                                  3. 5

                                                                    This is a very significant battery life and heat/sustained non-throttled performance difference.

                                                                    I agree.

                                                                    Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                                    I am not sure what you mean. Modern Intel/AMD CPUs have AES instructions. AMD GPUs (including those in APUs) have acceleration for H.264/H.265 encoding/decoding. AFAIR also VP9. Neural depends a bit on what is expected, but you can do acceleration of neural network training, if AMD actually bothered to support Navi GPUs and made ROCm less buggy.

                                                                    That said, for machine learning, you’ll want to get an discrete NVIDIA GPU with Tensor cores anyway. It blows anything else that is purchasable out of the water.

                                                                    You also mention Fortran though and I’m not sure how that matters in 2020?

                                                                    A lot of the data science and machine learning infrastructure relies on Fortran directly or indirectly, such as e.g. numpy.

                                                                    I strongly disagree with this. I mean, the M1 MacBook Air is beating the 16” MacBook Pro in Final Cut Pro rendering times. Xcode compilation times are twice as fast across the board. This is not at all a machine just for browsing and emailing. I think that’s flat-out wrong.

                                                                    Sorry, I didn’t mean that it is not fit for development. I meant that if you are doing development (unless it’s constrained to Xcode and Apple Frameworks), it is better to wait until the dust settles in the ecosystem. I think for most developers that would be when a substantial portion of Homebrew formulae can be built and they have pre-compiled bottles for them.

                                                                    1. 1

                                                                      Sorry, I didn’t mean that it is not fit for development. I meant that if you are doing development (unless it’s constrained to Xcode and Apple Frameworks), it is better to wait until the dust settles in the ecosystem. I think for most developers that would be when a substantial portion of Homebrew formulae can be built and they have pre-compiled bottles for them.

                                                                      My instinct here goes in the opposite direction. If we know Apple Silicon has tons of untapped potential, we should be getting more developers jumping on that wagon especially when the Homebrew etc. toolchain aren’t ready yet, so that there’s acceleration towards readying all the toolchains quickly! That’s the only way we’ll get anywhere.

                                                                      1. 16

                                                                        Well, I need my machine for work. So, these issues just distract. If I am going to spend a significant chunk of time. I’d rather spend it on an open ecosystem rather than doing free work for Apple ;).

                                                                    2. 5

                                                                      Sure, but the thing is that the AMD 4800U, their high-end laptop chip, runs at 45W pretty much sustained, whereas the M1 caps out at 15W. This is a very significant battery life and heat/sustained non-throttled performance difference. Also these chips don’t have GPUs or the plethora of hardware acceleration for video/media/cryptography/neural/etc. that the M1 has.

                                                                      Like all modern laptop chips, you can set the thermal envelope for your AMD 4800U in the firmware of your design. The 4800U is designed to target 15W by default - 45W is the max boost, foot to the floor & damn the horses power draw. Also, the 4800U has a GPU…an 8 core Vega design IIRC.

                                                                      Apple is doing exactly the same with their chips - the accounts I’ve read suggest that the power cost required to extract more performance out of them is steep & since the performance is completely acceptable at 15W Apple limits the clocks to match that power draw.

                                                                      The M1 is faster than the 4800U at 15W of course, but the 4800U is a Zen2 based CPU - I’d imagine that the Zen3 based laptop APUs from AMD will be out very soon & I would expect those to be performance competitive with Apple’s silicon. (I’d expect to see those officially launched at CES in January in fact, but we’ll have to wait and see when you can actually buy a device off the shelf.)

                                                                    3. 1

                                                                      Edit: returned and ordered a ThinkPad.

                                                                      That made me chuckle. Good choice!

                                                                      1. 1

                                                                        You say that you returned and ordered a ThinkPad, how has that decision turned out? Which ThinkPad did you purchase? How is the experience comparatively?

                                                                        1. 2

                                                                          I bought a Thinkpad T14 AMD. So far, the experience is pretty good.

                                                                          Pros:

                                                                          • I really like the keyboard much more than that of the MacBook (butterfly or post-butterfly scissors).
                                                                          • It’s nice to have a many more ports than 2 or 4 USB-C + stereo jack. I can go places without carrying a bunch of adapters.
                                                                          • I like the trackpoint, it’s nice for keeping your fingers on the home row and doing some quick pointing between typing.
                                                                          • Even though it’s not aluminum, I do like the build.
                                                                          • On Windows, battery time is great, somewhere 10-12 hours in light use. I didn’t test/optimize Linux extensively, but it seems to be ~8 hours in light use.
                                                                          • Performance is good. Single core performance is of course worse than the M1, but having 8 high performance cores plus hyperthreading compensates a lot, especially for development.
                                                                          • Even though it has fans, they are not very loud, even when running at full speed.
                                                                          • The GPU is powerful enough for lightweight gaming. E.g., I played some New Super Lucky’s tale with our daughter and it works without a hitch.

                                                                          Cons:

                                                                          • The speakers are definitely worse than any modern MacBook.
                                                                          • Suspend/resume continues to have issues on Linux:
                                                                            • Sometimes, the screen does not wake up. Especially after plugging or unplugging a DisplayPort alt-mode USB-C cable. Usually moving the TrackPoint fixes this.
                                                                            • Every few resumes, the TrackPad and the left button of the TrackPoints do not work anymore. It seems that (didn’t investigate further) libinput believes that a button is constantly held, because it is not possible to click windows anymore to activate them. So far, I have only been able to reset this state by switching off the machine (sometimes rebooting does not bring bak the TrackPoing).
                                                                            • So far no problems at all with suspend/resume on Windows.
                                                                          • The 1080p screen works best with 125 or 150% scaling (100% is fairly small). Enabling fractional scaling in GNOME 3 works. However, many X11/XWayland applications react badly to fractional scaling, becoming very blurry. Even on a 200% scaled external screen. Also in this department there are no problems with Windows, fractional scaling works fine there.
                                                                          • The finger print scanner works in Linux, but it results in many more false negatives than Windows.

                                                                          tl;dr: a great experience on Windows, acceptable on Linux if you are willing to reboot every few resumes and can put up with the issues around fractional scaling.

                                                                          I have decided to run Windows 10 on it for now and use WSL with Nix + home-manager. (I always have my Ryzen NixOS workstation for heavy lifting.)

                                                                          Background: I have used Linux since 1994, macOS from 2007 until 2020, and only Windows 3.1 and briefly NT 4.0 and Windows 2000.

                                                                        2. 1

                                                                          Everytime I wake up my Mac, after a couple of minutes, it switches to sleep again 1-3 times (shutting of the external screen as well).

                                                                          Sleep seems to be broken on the latest MacOS versions: every third time I close the lid of my 2019 mac, I’m opening it later only to see that it has restarted because of an error.

                                                                          1. 1

                                                                            Maybe wipe your disk and try a clean reinstall?

                                                                        1. 5

                                                                          As for the slow web, simply install a “disable javascript” extension and only enable it on tabs that need it. You’ll be surprised how many news sites and such will be drastically improve by this.

                                                                          1. 15

                                                                            This hasn’t been true for ages. Most sites won’t render a god-damned thing without JavaScript.

                                                                          1. 5

                                                                            Too bad that this blog post doesn’t go into any detail about it. What did they have trouble with, exactly?

                                                                            1. 1

                                                                              Is this related to the drifting of magnetic poles?

                                                                              1. 1

                                                                                Maybe? As far as I know, the actual mechanism for the Earth’s magnetic field is not well known.

                                                                              1. 5

                                                                                If one wishes to participate in a project, really, truly, wishing to participate, the buttons will not be a barrier.

                                                                                When I just want to submit a minor patch for a problem I discovered, the whole process with a modern npm package hosted on Github takes 20 minutes from encountering a problem to submitting the pull request. I don’t “relly, trully” wish to participate in your project, but I can do a small enhancement if you let me. But if you prefer to gatekeep based on messengers I use, I’d really rather spend those 20 minutes elsewhere.

                                                                                1. 4

                                                                                  Personally, I’d rather quickly email a minor patch rather than have to sign up to a Microsoft product in order to do so.

                                                                                  1. 8

                                                                                    I first began contributing to open-source projects in the early 2000s. It was never as simple as “quickly email a minor patch”. Lots of projects even then were on Sourceforge, or had Bugzilla instances or whatever and required you to sign up for an account, figure out the workflow for filing a new bug or feature request, how to generate and correctly format and attach the patch to meet the project’s standards, etc. etc.

                                                                                    And you’d have to endlessly follow up to make sure someone actually looked at it, find out the status, respond to any comments they made… I had a minor patch (literally just tweaking a dialog) that I submitted to GNOME in, I think, 2005 that bounced around multiple different bug trackers and required me to keep creating new accounts and figuring out the new place to follow it for years afterward.

                                                                                    Like it or not, GitHub and its competitors in that space today are a breath of fresh air compared to how things used to be, and that’s why they’ve become so popular and centralized so much of the open-source world onto their platforms.

                                                                                    1. 3

                                                                                      Will this email go through the CI pipeline in the next minute? Will you have a ready email template that maintainer created to help you through the bug report and PR process?

                                                                                      Or you will have to dig into their specific workflow, register to a couple of mailing list and go through the whole initiation process as if you’re really going to become one of the maintainers of this project?

                                                                                  1. 19

                                                                                    There is little of needless code in standard library. And what there is, believe me, over the decades was really polished.

                                                                                    realloc, all of locale.h, strtok, strcpy, strcat, gets… I like C a lot, but I wouldn’t call the stdlib polished. It’s a product of its time and has various footguns and bad APIs.

                                                                                    1. 12

                                                                                      Yeah, and all of this (and POSIX) has been designed before threads were a concern. For example, getenv is theoretically impossible to use in a multi-threaded environment, which makes time.h dangerous, because it may read the TZ env var.

                                                                                      1. 7

                                                                                        The poor and confusing stdlib is one of the least friendly bits of C. Nothing is ever dropped from the stdlib and only rarely dropped from the OS APIs, so all the bad stuff just hands around forever to confuse new programmers.

                                                                                        1. 4

                                                                                          You haven’t lived until you’ve called gets.

                                                                                          1. 3

                                                                                            Strings.

                                                                                            1. 1

                                                                                              Why don’t you like realloc?

                                                                                              1. 2

                                                                                                There are two problems with realloc - scope creep and its API.

                                                                                                Scope creep - realloc attempts to resize a buffer, but will also act as malloc if you pass it a special value (NULL ptr) and as free if you pass it a special value (size 0). This makes it difficult to tell what any given realloc call does when reading code and since it’s rare to actually want that functionality (because malloc/free exist) it tends to paper over bugs when accidentally triggered. It’s likely that the implementation drove the design; it would have been better to disallow resizing a null allocation and disallow resizing an allocation to 0, but it probably ‘fell out’ of how the initial implementation was written so became part of the functionality instead of keeping it tight and focused.

                                                                                                API - the intuitive way to use realloc is incorrect.

                                                                                                void *buf = ; // some existing allocation
                                                                                                buf = realloc(buf, new_size);
                                                                                                

                                                                                                That will leak memory if the resize fails - null is returned and it overwrites the original buf pointer.

                                                                                                The correct way to use it requires this song and dance, which is not a common or intuitive pattern.

                                                                                                void *buf = ; // some existing allocation
                                                                                                void *tmpbuf = realloc(buf, new_size);
                                                                                                if (tmpbuf != null) 
                                                                                                    buf = tmpbuf;
                                                                                                

                                                                                                realloc often gets a wrapper to prevent misuse because this is such a common issue. FreeBSD has reallocf that will automatically free the passed in buf pointer on allocation failure.

                                                                                                It would have been better if realloc had separated the error signaling from the buffer parameter instead of trying to collapse both into a single return value. Example alternate definition

                                                                                                int realloc(void **buf, size);
                                                                                                void *buf = ; // some existing allocation
                                                                                                if (!realloc(&buf, size)) {
                                                                                                    free(buf); // resize failed
                                                                                                }
                                                                                                
                                                                                                1. 4

                                                                                                  Realloc has other problems. Its efficiency depends heavily on the design of the allocator. Early malloc implementations stored the size in the header word and so could easily either expand the size if the memory after the allocation was free or contract it by carving the space into two runs. This family of allocators is generally slow and difficult to make efficient in a multithreaded setting. Modern allocators tend to be sizeclass allocators, which round the requested size up to some fixed-size bucket and then use a slab allocator of that size for the allocation. These are much faster and scalable but typically end up transforming realloc calls into malloc, memcpy, free sequences, syntactically hiding the fact that realloc is now a very slow operation.

                                                                                                  The realloc function needs to be modelled as a free in user code: it invalidates all pointers to existing allocations. A shocking amount of software suffers from memory-safety violations when realloc actually reallocates. If you have any aliasing of the object, code that calls realloc must go and rewrite all of the pointers after the call. That’s easy if there is one pointer (but the C type system doesn’t give you any help ensuring that this is the case) and very hard in the general case.

                                                                                                  To fix this problem, a lot of code depends on undefined behaviour, using something like:

                                                                                                  void *old = somePtr;
                                                                                                  void *new = realloc(old, newSize);
                                                                                                  if (new != old)
                                                                                                  {
                                                                                                     // Go and rewrite pointers that are equal to old.
                                                                                                  }
                                                                                                  

                                                                                                  The problem here is that it is UB in C to compare anything to a freed pointer. If new and old are equal, this is well-defined behaviour. If the realloc call freed the old object, then this is UB, so it’s completely fine for a compiler to assume that the body of this function is never reached (reaching it is only possible if your program contains UB and programs that contain UB are in an undefined state and anything is allowed).

                                                                                            1. 1

                                                                                              $120,000 - $190,000 for a remote position? Are you accepting international candidates? I’m not trying to apply, just curious about this wonderful post-covid engineering labour market.

                                                                                              1. 3

                                                                                                Yes, we have folks in the US, Canada, and Germany so far. And we’ve been remote since the beginning (hey, we make a realtime video meeting platform after all!), but COVID has certainly changed things for a lot of companies.

                                                                                                1. 3

                                                                                                  Well, kudos for offering SF salaries to international remote workers! This is 3+ times what similar local positions typically offer. I’m not looking for a position right now, but sent the link to some folks that might be interested, and will check back to see if you have any backend Node openings.

                                                                                              1. 2

                                                                                                This is a good approach for UI and some certain types of game logic. In particular, the types of game logic that is easily and painlessly separable from visual Unity components: request-response multiplayer games, turn-based games, and other stuff like this.

                                                                                                For games that have game logic that is tightly linked to visual stuff, though, such attempts at decoupling end up in disaster. Simplest example is: if you have hitboxes that move every visual frame together with animations created by your artists, this will only bring pain. And most importantly, it will bring pain not only to developers, but to game designers, level designers and artists, the very people who you want to make most productive with your game code.

                                                                                                Separation of different layers in software is a great tool when these layers really have different meanings that are not connected to one another. When you’re writing a UI to display a counter, number in the counter doesn’t have anything in common with the rendered shapes. But when you’re writing an action game, the logic of shooting at an animated model has everything in common with the animation you use to update the rendered mesh. Don’t try to separate the two.

                                                                                                1. 3

                                                                                                  I’ve had a similar problem with git: for some projects, zsh glob pattern matching would take up to 10 seconds, but the same pattern matching in git is almost instant. So, now I just have a habit of always writing glob patterns quoted.