Threads for stilbruch

  1. 5

    This is inexcusable.

    What is? That alpine contributors disagree with you?

    1. 6

      I don’t think that’s the most charitable interpretation of what the quoted bit means. Later in the paragraph you quoted, the author says:

      Accusing a contributor of ignorance is inexcusable. Comments like this have no place in open source contributions:

      SysTemD is the STD of operating systems. There is no “one little poke”, you can’t be a little bit pregnant.

      That said, the repo’s owner already told the person who made the STD comment that they were out of line, and a lot of the other reactions were politely negative.

      More broadly, I agree with several other people here that it’s not necessarily politics or unfair for developers to reject something someone wants.

    1. 11

      Situation: There are 15 competing standards.

      1. 31

        that’s a lazy response - the problem is there are 15 competing implementations and zero standards. i really hope this blocks endeavour succeeds.

        1. 2

          Though there are also standards?

          1. 2

            Are there?

            1. 6

              From Block Protocol Spec:

              Blocks are web components.

              So, a block is literally using an existing standard.

              There are standards for other things as well. Packages? For web things? That can include type information, data structures, code? Reminds you of one or several things? Protocols? ActivityPub comes to mind, but there are others as well. Likewise for structured data. Essentially, pulling together existing standards to unite them.

              I’ve commented elsewhere, to me personally, this looks a lot like a specific thing aimed at CMS-like things, the article mentions Wordpress, note taking apps, kanban boards etc. Sure, it would be nice if something like this worked okay, but what would happen is an app would support this, but they’d also want to support old-school Wordpress plugins so now I have to support both, right? Which is what the original XKCD is about, isn’t it?

        2. 6

          I haven’t seen a remotely similar attempt at standardize something like this outside of platform specific stuff. Curious if anybody has examples.

          1. 7

            Well, OpenDoc. It ran on Mac, Windows and OS/2. But it wasn’t related to the Web at all.

        1. 4

          It looks cool… and I really like the old school design rather than the „modern“ flat one… but do you expect this project being more successful than e.g. Window Maker?

          1. 9

            Seems like the author is working on the project out of enjoyment, not to “be successful”

          1. 2

            Modern hard drives have more than enough memory to handle your big fat binary and network is fast

            I’d say the cache still matters quite a bit, and “the network” is not that fast for most people

            1. 5

              TLDR: Doing undefined behavior can cause unexpected (undefined?) results. Therefore: “don’t write C code”.

              1. 8

                It’s more: there is no way of expressing an API in C that does not involve undefined behaviour if you use it wrong, therefore use a language that can express this kind of API in such a way that incorrect use is a static type-checking error.

                It’s worth noting that C++11 is the first version of C++ in which it was possible to implement a type-safe printf.

              1. 2

                I would like a Xorg that is easier to build and doesn’t do so many things I don’t need. I understand why some of those things are there historically, but it’s one of the last remaining resource hogs on my system and I am sure there is some way to slim it down.

                1. 16

                  The most interesting part is this tool:

                  https://github.com/GoogleContainerTools/distroless

                  Pretty amazing small images.

                  1. 20

                    The devops people are slowly rediscovering static linking

                    1. 10

                      The main advantage of distroless seems to be dynamically linking glibc properly in a minimal container.

                      1. 16

                        what if the container was a file, and the file was an executable. that’d be pretty minimal. they’ll get there some day…

                        1. 14

                          And then the cycle will begin again. “Hey I have this file but it doesn’t run because it needs to be in the presence of this other file”.

                      2. 4

                        I am not sure what you mean. Static linking was always an option at different levels. Single JAR file is a statically linked package for Java apps for example. You can do the same thing with AWS Lambda packages (single zip).

                        1. 11

                          At least last I checked, a JAR file is a self-contained package but not actually statically linked. All the class path lookup stuff is still done, dynamically, when the program is run.

                    1. 5

                      well this is what proprietary software is like, what did you expect?

                      1. 5

                        I think non-whitespace characters would be a better metric. Those java lines are very dense. Even so, comparing one project by one author doesn’t give a very useful statistic.

                        1. 4

                          (`source code here`).replace(/[\s\n]*/g, “”).length

                          Python - 4542

                          Typescript - 4558 - +0.3%

                          Go - 4736 - +4.2%

                          C# - 4992 - +9.9%

                          Java - 5426 - +19.4%

                          1. 1

                            I agree on the density. The python lines seem like they are half as wide as the java lines. Plus python has two lines between functions whereas the java code has one.

                            One minor nitpick on the python code: it looks like you could use data classes. Would remove a couple of LOC.

                          1. 1

                            Neat! Objective-C also got its start as a preprocessor (also using the @ token.) And so did C++, come to think of it.

                            Why did you decide to use the existing “auto” for cleanup, instead of a new word like the de-facto standard “defer”? Just to avoid adding any reserved words to the language?

                            1. 1

                              :-) My purpose is explicitly not to build this up into a new language, because if you can use a new compiler there are better options out there already.

                              As for using auto, you are exactly right: it was to avoid adding keywords.

                              1. 2

                                Why use keywords in general? For a preprocessor or wouldn’t it make more sense to add a #defer?

                                1. 2

                                  There is the issue that existing editor modes are likely to mis-indent such lines. It would be a deal breaker for me to require custom editor settings.

                            1. 15

                              Please consider signing the open letter against these changes: https://appleprivacyletter.com/

                              1. 10

                                Are you going to post an open letter for Microsoft, Google, DropBox, Facebook, Twitter, and all the other companies who have used the exact same database for this exact purpose for the last decade?

                                1. 8

                                  Which provider has previously used this list against images that aren’t stored on their infrastructure?

                                  1. 4

                                    Images sent via iMessage are stored on Apple’s infrastructure.

                                    1. 1

                                      I think the question had implied “stored in plain text”. iMessage doesn’t do that.

                                      1. 6

                                        Right. So, every other provider has direct access to your photos, and scans for CSAM with their direct access. Apple, rather than give up their E2E messaging, has devised a privacy-preserving scheme to perform these scans directly on client devices.

                                        I really don’t understand how Apple is the bad guy here.

                                        1. 4

                                          Other providers that scan cleartext images are off the hook, because they’ve never had E2E privacy guarantee.

                                          [smart guy meme]: You can’t have encryption backdoor if you don’t have encryption.

                                          Apple’s E2E used to be a strong guarantee, but this scanning is a hole in it. Countries that have secret courts, gag orders, and national security letters can easily demand that Apple slip in a few more hashes. It’s not possible for anyone else to verify what these hashes actually match and where they came from. This is effectively an encryption backdoor.

                                    2. 3

                                      If I understood what I read, although the private set intersection is done on device, it’s only done for photos that are synced with iCloud Photo Library.

                                      1. 2

                                        Apologies to all in this thread. Like many I originally misunderstood what Apple was doing. This post was based on that misunderstanding, and now I’m not sure what to do about it. Disowning feels like the opposite of acknowledging my mistake, but now I have 8 voted based on being a dumbass 🙁

                                        1. 2

                                          iCloud Photos are stored on Apple infrastructure.

                                      2. 4

                                        This page gets the scope of scanning wrong in the second paragraph, so I’m not sure it’s well researched.

                                        1. 3

                                          how so? can you explain?

                                          “Apple’s proposed technology works by continuously monitoring all photos stored or shared on a user’s iPhone, iPad or Mac, and notifying the authorities if a certain number of objectionable photos is detected.”

                                          seems like an appropriate high-level description of what is being done, how is it wrong?

                                          1. 7

                                            I may be wrong but, from what I understood, a team of reviewers is notified to check manually the photos once a certain number of objectionable photos is detected, not the authorities… If (and only if) the team of reviewers agrees with the hashes matches, they notify the authorities.

                                            This is a detail but this introduces a manual verification before notifying the authorities, which is important.

                                            From MacRumors:

                                            Apple’s method works by identifying a known CSAM photo on device and then flagging it when it’s uploaded to ‌iCloud Photos‌ with an attached voucher. After a certain number of vouchers (aka flagged photos) have been uploaded to ‌iCloud Photos‌, Apple can interpret the vouchers and does a manual review. If CSAM content is found, the user account is disabled and the National Center for Missing and Exploited Children is notified.

                                            Link to the resource: https://www.macrumors.com/2021/08/05/apple-csam-detection-disabled-icloud-photos/

                                            1. 1

                                              Second paragraph of the AP article

                                              The tool designed to detected known images of child sexual abuse, called “neuralMatch,” will scan images before they are uploaded to iCloud

                                              This resource from Apple also states that only images uploaded to iCloud are scanned.

                                              1. 2

                                                This quote you cite figures nowhere within the page.

                                              2. 1

                                                Apple’s proposed technology works by continuously monitoring photos saved or shared on the user’s iPhone, iPad, or Mac.

                                                Only photos uploaded to iCloud Photos are matched against known hashes.

                                            2. 4

                                              Or just don’t buy an Apple device. Do you really think a trillion dollar company cares about digital signatures?

                                              1. 6

                                                I think this is a good statement of intent though.

                                                I just bought an iPhone 12 and would be otherwise unlikely to be noticed as a lost sale until the iPhone 14~ since most people don’t upgrade a single minor version.

                                                Giving them warning that they have lost me as a customer because of this is a good signal for them. If they choose not to listen then that’s fine, they made a choice.

                                                Also the more noise we make as a community; the more this topic gains attention from those not in the industry.

                                                1. 4

                                                  I didn’t mean to make some sort of “statement” to Apple. I find that idea laughable. What I meant is that if you are really concerned about your privacy to the point where scanning for illegal images is “threaten[ing] to undermine fundamental privacy protections” (which I think is reasonable), then why buy Apple in the first place? This isn’t the first time they have violated their users’ privacy, and it certainly wont be the last.

                                                  1. 6

                                                    What’s your proposed alternative?

                                                    I think Apple making a stance on privacy, often posturing about it a lot, does cause a lot of good will and generally those who prefer to maintain privacy have been buying their products. (myself included). You can argue that it’s folly but the alternatives are akin to growing your own vegetables on a plot of land in the middle of nowhere connected to no grid (a-la rooted android phones with f-droid) or google owned devices which have a significantly worse privacy track record.

                                                    1. 3

                                                      You oughta update your intel about the “alternative” smartphone space. Things have come a long way from “growing your own vegetables on a plot of land in the middle of nowhere connected to no grid.” The big two user-friendly options are CalyxOS and LineageOS with microG. If you don’t feel like installing an OS yourself, the Calyx Institute, the 501(c)(3) nonprofit which develops CalyxOS, even offers the Pixel 4a with CalyxOS preinstalled for about $600.

                                                      I’m running LineageOS on a OnePlus 6T, and everything works, even banking apps. The experience is somewhere between “nearly identical” and “somewhat improved” relative to that of the operating system which came with the phone. I think the local optimum between privacy-friendliness and user-friendliness in the smartphone world is more obvious than ever, and iOS sure ain’t it these days.

                                                    2. 2

                                                      It does seem folly to make a statement by not buying something, but consider this: When you vote, there are myriad ways that politicians have to dilute your impact (not going to enumerate them here but it’s easy to do). By comparison, when you make an economic choice, ever dollar is counted in full, one way or another. So if you vote, and you should, then there’s every reason to vote with your pocketbook as well.

                                              1. 31

                                                I prefer to see this type of project that builds upon what it considers the good parts of systemd, instead of systemic refusal and dismissal that I’ve seen mostly.

                                                1. 15

                                                  Same. Too often I see “critiques” of systemd that essentially boil down to personal antipathy against its creator.

                                                  1. 22

                                                    I think it makes sense to take in to account how a project is maintained. It’s not too dissimilar to how one might judge a company by the quality of their support department: will they really try to help you out if you have a problem, or will they just apathetically shrug it off and do nothing?

                                                    In the case of systemd, real problems have been caused by the way it’s maintained. It’s not very good IMO. Of course, some people go (way) to far in this with an almost visceral hate, but you can say that about anything: there are always some nutjobs that go way too far.

                                                    1. 3

                                                      Disclaimer: I have not paid close attention to how systemd has been run and what kind of communication has happened around it.

                                                      But based on observing software projects both open and closed, I’m willing to give the authors of any project (including systemd) the benefit of the doubt. It’s very probable that any offensive behaviour they might have is merely a reaction to suffering way too many hours of abuse from the users. Some people have an uncanny ability to crawl under the skin of other people just by writing things.

                                                      1. 6

                                                        There’s absolutely a feedback loop going on which doesn’t serve anyone’s interests. I don’t know “who started it” – I don’t think it’s a very interesting question at this point – but that doesn’t really change the outcome at the end of the day, nor does it really explain things like the casual dismissal of reasonable bug reports after incompatible changes and the like.

                                                        1. 4

                                                          I think that statements like “casual dismissal” and “reasonable bug reports” require some kind of example.

                                                        2. 3

                                                          tbf, Lennart Poettering, the person people are talking about here is a very controversial personality. He can come across as an absolutely terrible know-it-all. I don’t know if he is like this in private, but I have seen him hijacking a conference talk by someone else. He was in the audience and basically got himself a mic and challenged anything that was said. The person giving the talk did not back down, but it was really quite something to see. This was either at Fosdem or at a CCC event, I can’t remember. I think it was the latter. It was really intense and over the top to see. There are many articles and controversies around him, so I think it is fair that people take that into account, when they look at systemd.

                                                          People are also salty because he basically broke their sound on linux so many years ago, when he made pulseaudio. ;-) Yes, that guy.

                                                          Personally I think systemd is fine, what I don’t like about it is the eternal growth of it. I use unit files all the time, but I really don’t need a new dhcp client or ntp client or resolv.conf handler or whatever else they came up with.

                                                          1. 4

                                                            tbf, Lennart Poettering, the person people are talking about here is a very controversial personality.

                                                            In my experience, most people who hate systemd also lionize and excuse “difficult” personalities like RMS, Linus pre-intervention, and Theo de Raadt.

                                                            I think it’s fine to call out abrasive personalities. I also appreciate consistency in criticism.

                                                    2. 4

                                                      Why?

                                                      1. 7

                                                        At least because it’s statistically improbable that there are no good ideas in systemd.

                                                        1. 1

                                                          Seems illogical to say projects that use parts of systemd are categorically better than those that don’t, considering that there are plenty of bad ideas in systemd, and they wouldn’t be there unless some people thought they were good.

                                                          1. 2

                                                            Seems illogical to say projects that use parts of systemd are categorically better than those that don’t

                                                            Where did I say that though?

                                                            1. 2

                                                              I prefer to see this type of project that builds upon what it considers the good parts of systemd

                                                              Obviously any project that builds on a part of system will consider that part to be good. So I read this as a categorical preference for projects that use parts of systemd.

                                                      2. 2

                                                        There have been other attempts at this. uselessd (which is now abandoned) and s6 (which still seems to be maintained)

                                                        1. 4

                                                          I believe s6 is more styled after daemontools rather than systemd. I never looked at it too deeply, but that’s the impression I have from a quick overview, and also what the homepage says: “s6 is a process supervision suite, like its ancestor daemontools and its close cousin runit.”

                                                          A number of key concepts are shared, but it’s not like systemd invented those.

                                                          1. 1

                                                            s6 I saw bunch of folks using s6 in docker, but afaik that’s one of most not user friendly software i’ve been used.

                                                      1. 13

                                                        Congratulations to the author on winning an imaginary argument

                                                        1. 18

                                                          Immediatly after the Secret Service confiscates his devices on Apple’s request:

                                                          I would need to buy a decent replacement. I decided to go to the Apple Store and see what was in stock

                                                          1. 16

                                                            I think we need a word stronger than naive here.

                                                          1. 8

                                                            Happy to see that kernel developers are staunchly defending their email-based workflow, and there is no real threat that GitHub will become the core development infrastructure.

                                                            It’s strange to me that the article frames GitHub as an alternative to email that is better suited to “one-off” contributions. One of the problems with GitHub development is that you can’t contribute without a GitHub account, whereas email based workflows generally don’t require any subscription or account with the project to contribute. In that way email is actually better for one-off contributions.

                                                            If new contributors have difficulty getting “set up” to submit patches in the right format, maybe it would be good to have a bot that monitors a newbie contributors mailing list, identifying problems and suggesting fixes for emails that don’t meet the project standards. Keeping it email-based seems like a much more realistic way to help newbies learn about the kernel development process, rather than giving them the false impression that Linux is a GitHub project.

                                                            1. 6

                                                              Show of hands: who doesn’t have a GitHub account for reasons other than ideological? Even if I refused to use GitHub for my own projects, I’d end up with one purely for contributing to projects. (And it’s still a better experience than email.) It’s not like making an account is a huge barrier either.

                                                              That, and it’s not like GitHub is doing anything dastardly with your account either.

                                                              1. 18

                                                                Anyone who Github has banned or who they are legally forbidden from working with as an American company.

                                                                1. 9

                                                                  I do have a GitHub account, but I use it to open issues.

                                                                  My problem with these changes is not that GitHub is good or bad, but rather an external company becomes a dependency.

                                                                  If gcc does something bad, no worries, we can fork it, but if GitHub does something bad, we cannot.

                                                                  If it matters that much, I would suggest to run a GitLab/gitea server.

                                                                  EDIT: P.S., I have friends who don’t have GitHub accounts, not beacuse of ideology, but beacuse GitHub does not allow them, they live in places like Iran. Also: sometimes Mother Russia blocks websites like GitHub :)

                                                                  1. 4

                                                                    That, and it’s not like GitHub is doing anything dastardly with your account either.

                                                                    Of course, Microsoft would never!

                                                                    1. 3

                                                                      I’ve been waiting for this supposed Microsoft intervention and haven’t seen it. Everything GH is doing now is what they’d have done when they were independent, good or bad.

                                                                      1. 3

                                                                        doing anything dastardly with your account either.

                                                                        Idc about the quality of the service GitHub provides. You said they aren’t doing anything bad with your account (which to me means personal data)

                                                                        This is right up Microsoft’s (or any big tech company’s) alley. You can claim they aren’t doing anything, but because it isn’t open source you are at their mercy.

                                                                        1. 1

                                                                          Yeah most of the things that are bad about Github are equally bad regardless of whether or not they are a separate firm or owned by Microsoft.

                                                                      2. 2

                                                                        The dastardly thing that GitHub is doing is GitHub.

                                                                        1. 1

                                                                          I can’t tell what you’re arguing. That GitHub actually is better than email for one-off contributions because it’s a better experience? Your other points seem to rest on that premise, which is clearly not agreed upon.

                                                                        2. 5

                                                                          It’s strange to me that the article frames GitHub as an alternative to email that is better suited to “one-off” contributions. One of the problems with GitHub development is that you can’t contribute without a GitHub account, whereas email based workflows generally don’t require any subscription or account with the project to contribute. In that way email is actually better for one-off contributions.

                                                                          Exactly. This is why the Sane software manifesto requires that it should be possible to contribute without having an account at particular service and without having signed a contract with particular third-party.

                                                                          Dependency on centralized corporations like GitHub, Google etc. is Evil. Free software and internet needs rather more decentralization.

                                                                          1. 4

                                                                            I definitely also prefer mailing-list development. It makes much more sense when discussing changes iteratively, and the larger the change the more useful it becomes.

                                                                            Technically you still need to subscribe to the mailing list, unless you’re one of the fellows that still knows about gmane. And it does require some decent email client, to both handle the traffic and to make patch submission/retrieval convenient. Since many devs nowadays just use web-based clients, they see mailing-list based development as an annoyance, and that’s why it gets such a bad rep. This is not exclusive to kernel development. @calvin is also right in saying that almost everyone now has a github account for either work of issue submission, you actually have less friction using github than to subscribe to a high-traffic mailing list.

                                                                            It’s also true IMHO that the simplicity of github makes it easy to submit one-off patches, which is both a pro and a con. That’s fine for a typo fix, but it’s also too easy to see PRs with new functionalities, or bug fixes but with terrible code quality which the author has no intention to support beyond the initial submission: these are just a PITA for large projects, and raising the barrier to contribution does help weed them off.

                                                                            Kernel dev is not supposed to be easy, so having a non-zero barrier to entry is beneficial IMHO.

                                                                          1. 29

                                                                            For a month, we looked closely at the parts of the browser that were “sparking joy” for people, and the parts that weren’t

                                                                            “sparking joy” is not very high on my list of things I want a web browser to do.

                                                                            1. 17

                                                                              It’s a reference to Marie Kondo who found popularity with her methods of tidying up. She wrote a book and had (one? more?) popular reality TV show(s). She advocates for going through each of your possessions, asking if it “sparks joy” and to get rid of it if it doesn’t.

                                                                              While I got the reference, I’m a bit apprehensive of a new Firefox re-design. I feel like I just got used to the last one. Admittedly that one was sorely needed. This time around I feel like it’s coming too soon and will waste my time unnecessarily. Hopefully it will still be as customizable as before though, so I can put it all back the way I like it.

                                                                            1. 4

                                                                              At the end it says

                                                                              Meanwhile the average user still doesn’t know what a cookie is, and blindly clicks on the Accept button.

                                                                              This isn’t true. The beginning of the article even explains that it’s against the rules to emphasize accept over reject for cookies, which means the average user is going to click no. And why wouldn’t they?

                                                                              1. 7

                                                                                If that’s in the rules, it sure as hell isn’t followed. A lot of sites have a nice shiny “accept” button and a dimmer “manage my cookies” button.

                                                                              1. 5

                                                                                Meanwhile, PyPy is around 4x faster than CPython.

                                                                                1. 6

                                                                                  Annecdote ain’t data, but I’ve never been successful at getting PyPy to provide improved performance. My use cases have been things like running tooling (Pylint is extremely slow under PyPy, much moreso than CPython), just running web apps, and a lot of other things that aren’t benchmarks.

                                                                                  I don’t want to be too critical of PyPy, I imagine it gets a lot of what a lot of people want. But I don’t know what real workloads end up benefiting from it.

                                                                                  1. 4

                                                                                    PyPy upstream generally treats slowness as a bug and is willing to expend resources to fix it, if you’re willing to file issues with minimal test cases. (Here is a recent example bug about slowness.)

                                                                                    Anecdotes aren’t data, but about a decade ago, I ported a Minecraft server from Numpy and CPython to array.array and PyPy, and at the time, I recorded a 60x speedup on a microbenchmark, and around a 20x speedup for typical gameplay interactions, resulting in a backend that spent most of its time sleeping and waiting for I/O.

                                                                                    As long as we’re on the topic, it’s worth knowing that PyPy comes with a toolkit, RPython, which allows folks to generate their own JITs from Python. So, if one wanted more speed than was available with Python’s language design, then PyPy provides a route for forking the interpreter and standard library, and making arbitrarily distant departures from Python while still having high performance. For example, if we can agree that Dolphin implements “real workloads”, then PyGirl (code, paper) probably does as well.

                                                                                    1. 3

                                                                                      Yeah to me it helps to think of workloads in these categories (even if there are obviously way more than this, and way more dimensions)

                                                                                      1. String / hash / object workloads (similar to web apps. Similar to a linter, and similar to Oil’s parser)
                                                                                      2. Numeric workloads (what people write Cython extensions for; note that NumPy is written largely in Cython.)

                                                                                      JITs are a lot better at the second type of workload than the first. My experience matches yours – when I tried running Oil with PyPy, it was slower and used more memory, not faster.

                                                                                      Also, I think that workload 1 is the more important one for Python. If I want to write fast numeric code, it’s not painful to do in C++. On the other hand, doing string/hash/object graph workloads in C++ is very painful. It’s also less-than-great in Rust, particularly graphs.

                                                                                      So while I think PyPy is an astonishing project (and that impression grows after learning more about how it works), I also think it doesn’t speed up the most important workloads in Python. Not that I think any other effort will do so – the problems are pretty fundamental and there have been a couple decades of attempts.

                                                                                      (In contrast I got much better performance results adding static types manually, and semi-automatically translating Oil to C++. This is not a general solution as its labor intensive and restricts the language, although there are some other benefits to that.)

                                                                                      1. 1

                                                                                        I see the outline of your point, but I’m not sure on the specifics. In particular, a mechanism is missing: What makes strings, dictionaries, and user-customized classes inherently hard to JIT, particularly with a PyPy-style tracing metainterpreter?

                                                                                        Edit: Discussion in #pypy on Freenode yielded the insight that CPUs have trouble with anything which is not in their own list of primitive types, requiring composite operations for composite types. Since JITs compile to CPU instructions, they must struggle with instruction selection for composite types. A lesson for language designers is to look for opportunities to provide new primitive object implementations, using the CPU’s existing types in novel ways.

                                                                                        Our experience in the Monte world is that our RPython-generated JIT successfully speeds up workloads like parsing and compiling Monte modules to bytecode, a task which is string- and map-heavy. Our string and map objects are immutable, and this helps the JIT remove work.

                                                                                        1. 1

                                                                                          Yes the JITs do a lot better on integers and floats because they’re machine types.

                                                                                          The performance of strings and hash tables is sort of “one level up”, and the JITs don’t seem to help much at that level (and for some reason lots of people seem to misunderstand this.)

                                                                                          As an anecdote, when Go was released, there were some benchmarks where it was slower than Python, just because Python’s hash tables were more optimized. And obviously Go is compiled and Python is interpreted, but that was still true. So that is a similar issue.

                                                                                          So there are many dimensions to performance, and many workloads. Saying “4x faster” is doing violence to reality. In some cases it’s the difference between being able to use PyPy and not being able to use it.

                                                                                        2. 1

                                                                                          SciPy has some cython code along with a bunch of fortran code but NumPy is all C.

                                                                                          1. 1

                                                                                            Ah sorry you are right, I think I was remembering Pandas, which has a lot of Cython in its core:

                                                                                            https://github.com/pandas-dev/pandas/tree/master/pandas/_libs

                                                                                          2. 1

                                                                                            cython is also a translator to C. why didn’t you use cython for oil?

                                                                                            1. 1

                                                                                              It generates code that depends on the Python runtime, and Cython is a different language than statically-typed Python. I don’t want to be locked into the former, and translating the code is probably even more labor intensive than what I’m doing (I leveraged MyPy team work on automatic type annotation etc.). It also wouldn’t be fast enough as far as I can tell.

                                                                                          3. 3

                                                                                            pypy is 4x faster…. for long-running tasks that allow the jit to warm up. Lots of python workloads (e.g. pylint) run the interpreter as a one-off so pypy won’t help there. Interpreter startup speed is also critical for one-off workflows and pypy isn’t optimized for that either.

                                                                                            1. 3

                                                                                              I think it’s more like 10x-100x faster OR 10% slower for different workloads – “4x” doesn’t really capture it. See my sibling comment about string/hash/object vs. numeric workloads.

                                                                                            2. 2

                                                                                              I used PyPy recently, for the first time and I had a nice experience. I am experimenting with SQLite and trying to figure out the fast ways to insert 1B rows. My CPython version was able to insert 100M rows in 500 is seconds, same in PyPy took 150 seconds.

                                                                                              The best part was, I did not have to change anything in my original code. It was just drop in, as advertised. Ran it with PyPy and got the speed bumps.

                                                                                            3. 2

                                                                                              Specifically, we want to achieve these performance goals with CPython to benefit all users of Python including those unable to use PyPy or other alternative virtual machines.

                                                                                              1. 1

                                                                                                Apparently the goal is a 2x speed up by 3.11 and a 5x speed up in 4 years.

                                                                                                1. 4

                                                                                                  Yes. Assuming that those numbers are not exaggerated, I expect that PyPy will still be faster than CPython year after year. The reasoning is due to the underlying principle that most improvements to CPython can be ported to PyPy since they have similar internal structure.

                                                                                                  In GvR’s slides, they say that they “can’t change base layout, object layout”. This is the only part of PyPy’s interpreter which is structurally different from CPython. The same slide lists components which PyPy derived directly from CPython: the bytecode, the stack frames, the bytecode compiler, and bytecode interpreter.

                                                                                                  Specializing bytecode has been tried for Python before; I recall a paper which monomorphized integers and other common builtin types. These approaches tend to fail unless they can remove some interpretative overhead. I expect that a more useful product of this effort will be a better memory model and simpler bytecodes, rather than Shannon’s grand explosion of possible bytecode arrangements.

                                                                                                  1. 1

                                                                                                    I’m curious about mypyc personally. Seems to me like (c)python is just hard to optimize and depends too much on implementation details (the C API) to be changed; to get a significant leap in performance it seems like using a statically typed, less dynamic subset, would give significantly higher speedups. Of course the downside is that it doesn’t work for old code (unless it happens to be in this fragment).

                                                                                                    1. 1

                                                                                                      Monomorphizing code does not always speed it up. There are times when tags/types can be checked for free, thanks to the dominating effects of cache thrashing, and so the cost of dynamically-typed and statically-typed traversals ends up being similar.

                                                                                                      It’s not an accident that some half-dozen attempts to monomorphize CPython internals have failed, while PyPy’s tracing JIT is generally effective. Monomorphization can remove inner-interpreter work, but not interpretative overhead.

                                                                                                      1. 2

                                                                                                        Well by “less dynamic” I also mean not having a dictionary per class and this kind of stuff :-). I should have been clearer. tag checks is one thing, but performing dictionary lookups all the time to resolve identifiers or fields is also very heavy. The statically typed aspect, I have no idea if it’s truly necessary, but it’d make it easier to implement, right?

                                                                                              1. 2

                                                                                                I appreciate his point about “keep the web open”, but I think it’s inherently misguided.

                                                                                                You wouldn’t argue for ads in every home, in exchange for a free house, so that everyone can have a house. Because the result would be terrible housing for everyone.

                                                                                                While free markets are also imperfect, they are a much better mechanism for improving the overall quality of life for everyone than government-funded solutions. Or in this case, solutions funded by our Ad-overlords.

                                                                                                If someone wants to charge 100$ a year for their website, and some people can’t afford it, that’s fine. Someone else will find a cheaper way to provide the same service, and capitalize on the unattended market.

                                                                                                Ads are ruining the internet. Give us a real alternative, and let’s see what the people choose.

                                                                                                By real alternative, I mean convenient micropayments, or something of the sort.

                                                                                                1. 2

                                                                                                  While free markets are also imperfect, they are a much better mechanism for improving the overall quality of life for everyone than government-funded solutions. Or in this case, solutions funded by our Ad-overlords.

                                                                                                  Um the “Ad overlords” are some of the largest corporations on planet earth. They are the very definition of “the market”. Seems like you’ve already gotten what you want.

                                                                                                  1. 2

                                                                                                    No, imho a healthy market is one where monopolies are prevented, competition is possible, and new & upcoming players have a chance to grow. It’s not the case right now. Don’t mistake me for a libertarian just because I speak about free markets, which are an amazing invention that we should take more advantage of, but in a controlled manner.

                                                                                                    1. 2

                                                                                                      As the term “Free Markets” is widely misunderstood, I feel it’s appropriate to correct the record: it was coined to describe markets which were free of rentiers preventing new entrants from competing.

                                                                                                1. 2

                                                                                                  Working on a Minecraft bot in python. After a lot of work I’ve finally got it mining at a cobble generator (sort of).

                                                                                                  1. 2

                                                                                                    Working on a Minecraft bot in python

                                                                                                    Very cool! Do you recommend any resources for this?

                                                                                                    1. 2

                                                                                                      Well if you are interested in writing it in python, I have been using pyCraft as a library/reference. It does not implement the entire Minecraft protocol though, so you’ll have to add to it/write your own.

                                                                                                      More generally, https://wiki.vg is the place for Minecraft protocol information. There is no better resource besides working at Mojang AFAIK.