1. 6

    I haven’t owned my own laptop for years, since my employer’s been supplying me MacBook Pros as work machines. But my current one is the 2018 model with the awful keyboard, and lately the IT department has been making us install spyware device management tools and warning us not to install unauthorized software nor use the laptop for personal projects … so I’ve been lusting after a full-size ARM MBP of my very own.

    The good thing is that most of the CPU-transition glitches seem to have been worked out by now, like problems with HomeBrew. I wonder how long it’ll take to get Linux running on the new hardware?

    1. 8

      Asahi seems to be doing pretty well per their progress reports. They’ve got most of the hardware situated, just stuff like sound, 3D acceleration, etc. to write, and polishing up their drivers for i.e. display to upstream.

      1. 2

        Yeah, same here, but I’m sticking with “my” (work) MBP 2015 top of the line configuration, so far so good.

        1. 3

          Just bought a 2015 13” MBP and 15” MBP also 2015… great year for these machines. I may buy yet another 2015 13”, the form factor is so great and they’re pretty cheap on craigslist

        2. 1

          I mean, you are in dangerous IP waters in some jurisdictions if you use company-provided hardware for personal projects. I don’t like the MDM crap any more than anyone else, but the solution here is to use your own machine, really.

        1. 3

          This idea could be extended by using an actual cryptocurrency PoW (Or mining pool PoW) and use it as a captcha AND an income revenue for your user. You could provide easier challenges to solve it in a few seconds and every once in a while you might be able to find a solution to a harder challenge that yields actual currencies.

          1. 2

            I thought about this, and I decided I wanted the opposite. My reasons were:

            • complexity. I want the captcha to be as simple as possible, both to set up and to use.
            • ease-of-abuse. If I use a common scheme that has real hash power behind it, the “captcha solving botnet” that other folks posted about could probably be replaced by a single retired ASIC.

            So I specifically chose Scrypt parameters that are very different from those supported by Scrypt ASICs designed to mine Litecoin.

            I have heard that Monero mining can’t really be optimized much, is that true? I don’t know much about it. I suppose if there is a mining scheme out there that truely resisted being GPU’d or ASIC’d this could be possible. I wonder if the app would eventually get flagged as malicious by Google Safe Browsing because its running a known monero miner script on the user’s browser XD

            1. 3

              I feel that if the goal is to keep the bot out, you are kind of out of luck because the computing power of anyone running bot will overwhelmingly be much greater than any of your human user. Captcha is only good to send away the automated, generic bot or tools. Anyone that really wants to collect your site won’t be stopped by any captcha. So if we agree on that, the algorithm used should not really matter, even if specific hardware already exists for the PoW.

              As for Safe Browsing, I don’t think that would be an issue since you are mining from the website, not an extension or ads. Safe Browsing should only be flagging website that distributes malware/unwanted software executable and phishing.

              1. 1

                Anyone that really wants to […] won’t be stopped

                Exactly, this is drive-by-bot / scattershot deterrent. Agreed that when facing a targeted attack against a specific site, different strategy is needed.

                the algorithm used should not really matter, even if specific hardware already exists for the PoW.

                For SHA256, I can buy a USB ASIC for $200 that can hash faster than 2 million CPUs. I think that’s a meaningful difference. Much more meaningful than the difference between one user and a botnet, probably even more meaningful than the difference between a user’s patience and a bot’s patience.

                AFAIK, Litecoin uses a slightly modified version of Scrypt, and its “CPU and Memory Cost” (N) / “Block Size” (r) parameters are set quite low. This means that Scrypt ASICs designed for Litecoin can’t execute the hash with larger N and r parameters like the ones which would be used for key-derivation (or in my case, anti-spam).

                According to this 2014 paper the hash rates of GPU Scrypt implementations fall off quickly as N and r are increased. In fact, for the exact parameters I use, N = 4096 and r = 8, they cite the 2014 GPU performing a measly 3x faster than the CPU in my 2019 laptop (See section 5.2.4). So for a modern expensive GPU, that might be something like 100-300x faster? I’m not sure, but its certainly different from 2 million times faster. I believe this was actually a design goal of Scrypt from the beginning: it’s intentionally hard to accelerate it to insane hash rates.

                As an aside, I have a friend who took an alternate route, purely based on security through obscurity. They put a “write me a haiku” field on the registration page of their web app. It completely defeated every single bot. I opted for PoW instead of a pure “obscurity-based” approach because I wanted to show/argue that we can come up with bot deterrent strategies which truly scale, even if there are millions of sites using the same deterrent software, It should still be effective against folks who want to hit all 1 million of them. While I doubt my project will ever grow to that scale, I thought it was fun to try to design it to have that potential.

                1. 1

                  Exactly, this is drive-by-bot / scattershot deterrent.

                  Then why does it matter if using a known PoW allows some attacker to be 2 million faster? You can expect that any targeted attack will be a few thousand times better than your average user, even with custom Scrypt parameters. So does it really matter that an attacker is a few thousand or a million times faster? He’s probably done scraping or spamming your site by the end of the day either way. At least with the known PoW you might have made a few bucks.

                  1. 2

                    It’s because like I said, I primarily care about dragnet / “spam all forms on the internet” type stuff.

                    The former is a privacy concern, the latter represents all spam I’ve ever had to deal with in my life.. no one has ever “targeted” me or my site, its just stupid crawlers that post viagra ads on every single un-secured form they can find. I think that being targeted is actually very rare and it happens for political/human social reasons, not as a way to make a profit.

                    The weight of the PoW (from the crawler’s perspective) matters because if its extremely light (sha256) they can simply accept it and hash it at scale for cheap. If its heavy (scrypt with fat memory cost parameter) They literally can’t. It would cost an insane amount to solve gazzillions of these per year. Even if they invest in the GPU farm, it will only make it hundreds of times faster, not millions. And if you have a GPU farm, can’t you make more money renting it to corporations for machine learning model development anyways??

                    Like others have mentioned, that cost can be driven down by botnets. But like I have argued in return, IMO that level of investment is unlikely to happen, and if it does, I’ll be pleasantly surprised.

          1. 3

            Facepalming hard at these articles (posted a few hours ago) https://globalnews.ca/news/8145997/quebec-covid-passports-hack-police/

            It goes to show the public was not well informed enough about what the QR code contains (practically nothing identifying without additional documentation).

            1. 3

              I found most news coverage about the QR Code misleading and reporting on “cybersecurity experts” that keep claiming they “hacked” the QR Code by decoding it. And then this other privacy group claimed they were able to hack many politicians QR Code when in fact they simply went through the portal, provided the information required to get your QR Code (Name, Date of birth, Vaccination dates and health insurance number, which is the previous information + 2 random number). You will guess that for high profile most of these data points are known and the QR Code you get gives you pretty much the exact same data. Lots of FUD being thrown around by privacy groups looking to get media coverage. I’m all for having a debate about privacy, but these people just keep throwing terms they know are going to be misunderstood by the lay persons and this is plain irresponsible. There have been a few actual bug found in the app that was able to validate a crafted QR Code and another one that allowed a government employee to download thousands of QR Code. But these researchers went through the proper channel and didn’t spread fear around these.

              1. 1

                You must be speaking specifically about the iOS app? I’m still waiting for the Android one to get the public keys.

                1. 1

                  The key is not iOS or Android specific. They are the same key pair. You can find it in all the many third party apps that have been developed over the last months.

                  1. 1

                    Oh that I know. What I don’t know is how to download and reverse an iOS application, but I do an Android application.

                    How have the other applications gotten the keys?

                    1. 3

                      There’s this minified and ugly async compiled into sync state machine javascript that has been shared with me (Seems to be using this SDK ) that has been extracted from the iOS app. As for the public key, don’t quote me on this, but I think someone just asked for it to the right person. Now you can find it in the minified code:

                          s.exports = {
                            alg: "ES256",
                            kty: "EC",
                            crv: "P-256",
                            use: "sig",
                            kid: "fFyWQ6CvV9Me_FkwWAL_DwxI_VQROw8tyzSp5_zI8_4",
                            x: "XSxuwW_VI_s6lAw6LAlL8N7REGzQd_zXeIVDHP_j_Do",
                            y: "88-aI4WAEl4YmUpew40a9vq_w5OcFvsuaKMxJRLRLL0",
                          };
                      
                      1. 2

                        I’ve written a decoder, verifier, and encoder. Refresh the document to see it at the bottom.

                        Also I found out the QR code can hold about 250 doses before it starts becoming unscannable. X)

            1. 4

              Very beautiful.

              I wonder what such a clock would be like here in the north, where the day portion of a nychthemeron is very long in the summer and very short in the winter. :)

              1. 2

                Nevertheless, the solution I used was to mark the beginning and end of the day at the point where the Sun is closest to the horizon. This makes the solution continuous, in the sense that a person observing the clock while moving north into the Arctic circle at certain times of the year would see the day progress bar advance normally, while the sunset indicator would move toward either 0 or 100 before vanishing altogether.

                The sunset indicator is moving around, but the progress bar should still take 24 hours to go around.

              1. 4

                Very cool, both at a technical and aesthetic level! One question: you said that the zero-point is sunrise, but if the sunset is marked by the blue dot, what does the 100 indicate? Midnight?

                1. 13

                  100 is tomorrow’s sunrise, and when the progress bar will roll over. Other interesting points in time: solar noon is halfway between zero and the blue dot, solar midnight is halfway between 100 and the blue dot.

                  1. 2

                    I suppose that would be the next sunrise?

                  1. 3

                    [my problem is not primarily with the content of this article, but the context]

                    “Whatever got you here, your infrastructure person has left, maybe suddenly. You have been moved into the role with almost no time to prepare. “

                    This is a terrible, terrible situation.

                    The point of devops is not “devs can do ops!”. The point of devops is that devs and ops should work together, using the tools of modern software development to implement repeatable, debuggable operational infrastructure. This is not a new idea; cfengine was written in 1993, and that wasn’t the first run at devops methods, just the first one that gained traction. Ops have always written software tools to help out.

                    If you have no ops experience on your team, you cannot “do devops.” At best, you can start learning operations skills. Think of this as a software development project where you do not have a subject-matter expert at hand and you probably don’t have a requirements document other than “the spice must flow!”.

                    If your company had a single operations person and suddenly they don’t, you have three problems:

                    1. the bus factor has already kicked over to zero.
                    2. management did not have the resources or forethought to plan for this, so the management is either incompetent or the company is having money trouble
                    3. you now have responsibility without the experience to fix it, and you will need to be experimenting in ways which will cost money or time or both that you probably don’t have

                    If this is the situation you find yourself in, it is probably past time to find another company.

                    If you are the founder/owner: how did you not see this coming? There were no warning signs? Consider pausing all activity in order to rethink the company.

                    1. 2

                      A company with a single devops person is probably a company with limited resources, not enough work for two devops person, and has probably many more risk than losing a devops person. While this is not a nice situation to be in, there are many more terrible situation to be in. It doesn’t make any sense to overcommit scarce resources in one business area, especially if one possible fix is to take an existing developer, hope he keeps the fire down for a while, giving you time to find someone else to keep thing running smoothly. What else would you suggest?

                      1. 9

                        You say “devops person” as though it were a separate job title.

                        In that small company, everyone in development and ops and network engineering and security is on the devnetopsec team. Cross-train. Document. Or accept the risk that one person leaving will sink the company.

                    1. 43

                      Tell me your job is operating kubernetes and you want job security without telling me that your job is operating kubernetes and you want job security.

                      1. 8

                        I find it disappointing that the top comment, with ~30 upvotes and unchallenged for several hours, is cynically questioning the author’s motives. Isn’t there already enough cynicism in the world? We should be better than that.

                        1. 6

                          It’s meant to be taken humorously. The author’s main argument is an appeal to expertise with the statement that he mixed the kool-aid. The rest of the article is based on personal opinion so there isn’t much else to say. If you have a similar experience to the author then you will agree, otherwise not.

                          1. 2

                            I don’t know, every article about kubernetes is followed by some comments about how there are some conspiracies, and how anyone pro-Kubernetes must be some shill or insecure software engineer looking to hype the industry so they can have jobs. To me this sounds more like low quality troll comment than humor. There’s nothing technical or insightful in @cadey comment.

                            1. 10

                              My comments were meant to be taken humorously. This exact comment is in the vein of a rich kind of twitter shitposting of the model “tell me x without telling me x” as a way to point out ironic or otherwise veiled points under the surface of the original poster’s arguments. I am not trying to say anything about the author as a person (English is bad at encoding this kind of intent tersely) or anything about their skill level. I guess the true insight here is something along the lines of this Upton Sinclair quote:

                              It is difficult to get a man to understand something, when his salary depends on his not understanding it.

                              I also burned out on kubernetes clusterfucks so hard I almost quit tech, so there is a level of “oh god please do not do this to yourself” chorded into my sarcastic take.

                        2. 2

                          fwiw - I am currently unemployed and working on a startup in an unrelated space. I haven’t worked on Kubernetes in 2 years.

                        1. 2

                          It seems to me that the cipher functions are vulnerable to reused key such that: E(c1) ^ E(c2) == c1 ^ c2. Is that right?

                          Maybe a solution to not reuse any part of of the cipher stream would be to use 52 unique nonce and try them all until a valid card is decrypted.

                          eg.

                          e := func(src []byte, nonce: int) []byte {...}
                          d := func(src []byte) []byte {
                             ...
                             for nonce in range(52) {
                          		dcipher, _ := chacha20.NewUnauthenticatedCipher(key[:32], nonce)
                          		dcipher.XORKeyStream(res, src)
                          		if isValidCard(res) { return res }
                          	}
                          }
                          
                          for i, card := range deck {
                          	encryptedDeck = append(encryptedDeck, Eb(card.name(), i))
                          }
                          

                          That said, isn’t something that asymetric crypto could solve. My crypto is rusty, but if I remember well, there were some construct where you could encrypt with different keys and decrypt in different order.

                          1. 1
                            1. 1

                              I also just noticed this issue while I was wondering about whether the encryption function is deterministic. With deterministic encryption, wouldn’t the encrypted deck always look the same, and you could recognize encrypted cards from previous rounds?

                              Your solution would also help with that, if you re-generate the nonces each round, I think.

                              1. 1

                                Encryption relies on the key (passphrase). With this game you should never reuse the same passphrase, both because it would lead to reused key attack, but especially because you give it away at the end of each game.

                                1. 1

                                  Right, I misunderstood what a “game” is here. Thanks!

                            1. 29

                              Well written, this were exactly my thoughs when i read this. We don’t need faster programmers. We need more thorough programmers.

                              Software could be so much better (and faster) if the market would value quality software higher than “more features”

                              1. 9

                                We don’t need faster programmers. We need more thorough programmers.

                                That’s just a “kids these days…” complaint. Programmers have always been fast and sloppy and bugs get ironed out over time. We don’t need more thorough programmers, like we don’t need more sturdy furniture. Having IKEA furniture is amazing.

                                1. 12

                                  Source code is a blueprint. IKEA spends a lot of time getting their blueprints right. Imagine if every IKEA furniture set had several blueprint bugs in it that you had to work around.

                                  1. 5

                                    We’re already close though. We have mature operating systems, language runtimes, and frameworks. Going forward I see the same thing happening to programming that happens to carpentry or cars now. A small set of engineers develop a design (blueprint) and come up with lists of materials. From there, technicians guide the creation of the actual design. Repairs are performed by contractors or other field workers. Likewise, a select few will work on the design for frameworks, operating systems, security, IPC, language runtimes, important libraries, and other core aspects of software. From there we’ll have implementors gluing libraries together for common tasks. Then we’ll have sysadmins or field programmers that actually take these solutions and customize/maintain them for use.

                                    1. 7

                                      I think we’re already completely there in some cases. You don’t need to hire any technical people at all if you want to set up a fully functioning online store for your small business. Back in the day, you would have needed a dev team and your own sysadmins, no other options.

                                      1. 1

                                        I see the same thing happening to programming that happens to carpentry or cars now. […] From there we’ll have implementors gluing libraries together for common tasks.

                                        Wasn’t this the spiel from the 4GL advocates in the 80s?

                                        1. 2

                                          Wasn’t this the spiel from the 4GL advocates in the 80s?

                                          No, it was the spiel of OOP/OOAD advocates in the 80s. Think “software IC’.

                                    2. 1

                                      Maybe, maybe not. I just figured that if i work more thoroughly, i get to my goals quicker, as i have less work to do and rewrite my code less often. Skipping error handling might seem appealing at frist, as i reach my goal earlier, but the price for this is that either me or someone else has to fix that sooner or later.

                                      Also mistakes or just imperformance in software nowadays have huge impact due to being so widespread.

                                      One nice example i like to make:

                                      Wikimedia foundation got 21.035.450.914 page views last month [0]. So if we optimize that web server by a single instruction per page view, assuming the CPU runs at 4 GHz, with a perfect optimized code of 1.2 instructions per cycle, we can shave off 4.382 seconds per month. Assuming wikipedia runs average servers [1], this means we shave of 1.034 watt hour of energy per month. With a energy price of 13.24 euro cent [2], this means a single cycle costs us roughly 0.013 euro cent.

                                      Now imagine you can make the software run 1% faster, which are 48.000.000 instructions, this is suddenly 6240€ per month savings. For 1% overall speedup!

                                      High-quality software is not only pleasant for the user. It also saves the planet by wasting less energy and goes easy on your wallet.

                                      So maybe

                                      Programmers have always been fast and sloppy and bugs get ironed out over time. We don’t need more thorough programmers,

                                      this should change. For the greater good of everyone

                                      [0] https://stats.wikimedia.org/#/all-projects/reading/total-page-views/normal|table|2-year|~total|monthly
                                      [1] https://www.zdnet.com/article/toolkit-calculate-datacenter-server-power-usage/
                                      [2] https://www.statista.com/statistics/1046605/industry-electricity-prices-european-union-country/

                                    3. 9

                                      Software could be so much better (and faster) if the market would value quality software higher than “more features”

                                      The problem is there just aren’t enough people for that. That’s basically been the problem for the last 30+ years. It’s actually better than it used to be; there was a time not so long ago where everyone who could sum up numbers in Excel was a programmer and anyone who knew how to defrag their C:\ drive was a sysadmin.

                                      Yesterday I wanted to generate a random string in JavaScript; I knew Math.random() isn’t truly random and wanted to know if there’s something better out there. The Stack Overflow question is dominated by Math.random() in more variations that you’d think possible (not all equally good I might add). This makes sense because for a long time this was the only way to get any kind of randomness in client-side JS. It also mentions the newer window.crypto API in some answers which is what I ended up using.

                                      I can make that judgment call, but I’m not an ML algorithm. And while on Stack Overflow I can add context, caveats, involved trade-offs, offer different solutions, etc. with an “autocomplete code snippet” that’s a lot more limited. And especially for novice less experienced programmer you wouldn’t necessarily know a good snippet from a bad one: “it seems to work”, and without the context a Stack Overflow answer has you just don’t know. Stack Overflow (and related sites) are more than just “gimme teh codez”; they’re also teaching moments.

                                      Ideally, there would be some senior programmer to correct them. In reality, due the limited number of people, this often doesn’t happen.

                                      We’ll have to wait and see how well it turns out in practice, but I’m worried for an even greater proliferation of programmers who can’t really program but instead just manage to clobber something together by trail-and-error. Guess we’ll have to suffer through even more ridiculous interviews to separate the wheat from the chaff in the future…

                                      1. 2

                                        We’ll have to wait and see how well it turns out in practice, but I’m worried for an even greater proliferation of programmers who can’t really program

                                        I don’t see this as a problem. More mediocre programmers available doesn’t lower the bar for places that need skilled programmers. Lobste.rs commenters often talk of the death of the open web for example. If this makes programming more accessible, isn’t that better for the open web?

                                      2. 6

                                        We don’t need faster programmers. We need more thorough programmers.

                                        Maybe we need more than programmers and should aim to deserve the title of software engineers. Writing code should be the equivalent of nailing wood, whether you use a hammer or AI assisted nailgun shouldn’t matter much if you are building a structure that can’t hold the weight it is designed for or can’t deal with a single plank that is going to break or rot.

                                        1. 6

                                          We don’t need faster programmers. We need more thorough programmers.

                                          Not for everything, but given we spend so much time debugging and fixing things, thoroughness is usually faster.

                                          1. 6

                                            Slow is smooth and smooth is fast.

                                        1. 5

                                          The fatal Boeing 737 MAX8 crash involving Ethiopian Airlines in 2019 was the result of AI gone wrong.

                                          This seems incorrect. From the linked article:

                                          Though the Boeing 737 MAX 8 does not include elements that might be considered in the AI bailiwick per se

                                          1. 1

                                            Thanks. I changed the link to the wikipedia page describing the MCAS system. While the author of the article does not consider it AI, I do.

                                            1. 4

                                              Not sure why Wikipedia makes it any more correct. There is no reference whatsoever to AI in the page. Saying that 737 Max 8 accident is caused by AI, followed by mention about black box and learning systems in the context of an article about some Deep Learning tech is misleading. This could be interpreted as if a trained model was part of the Max 8 crashes which is not the case.

                                              But in any case, the 737 Max 8 could have been a trained AI and it wouldn’t matter. Saying the plane crash because of this black box is a very shallow analysis of the issue. Reading Max 8 analysis shows that the issue were much more systemic, lacking in the process, training and safety. A similar conclusion can be found from the analysis of another deadly software issue with the Therac 25. If anything, this should somehow be a counter-argument to your point. Software are going to have bugs, wether written by a human or AI, but this is not an excuse for having critical failure.

                                              1. 3

                                                That’s a fair criticism about linking 737 Max 8. But I still consider it AI. The fact that GPT-3 uses a deep learning system isn’t material. Both GPT-3 and the MSCAS system seem to be black boxes as far as the users are concerned.

                                                I don’t believe it’s a counter-argument to my point because my point is that Copilot is a systemic risk. It also doesn’t require any sort of training process, safety checking, etc.

                                                My point was that people are bad at interacting with black boxes. If the black boxes are good, then people will be less careful with them and therefore when mistakes are made, they will be bad. Copilot being good is worse than if it required the programmer to make changes to it’s suggestions every time. Because complacent programmers will let bugs slip in. With copy-pasting code from stack overflow, you usually have to make changes to the software and therefore read it more carefully. If copilot is really good and compiles more often then not, then programmers will be less compelled to read the code.

                                                In fact, Copilot requires a skill that isn’t taught at all, reading code! Reading code is the most underdeveloped skill programmers have.

                                                1. 5

                                                  You’re right about interaction with black boxes, but you can’t call every black box an A.I. MCAS is presumably implemented as a fuzzy logic, and it couldn’t have been a black box to its authors. The fact that it wasn’t well documented for users doesn’t make it A.I.

                                                  A.I. is a loose term, but I think most people will agree it takes more than a bunch of if statements, and it doesn’t mean just any decision taken by a computer.

                                                  1. 2

                                                    The term A.I is loosely thrown around these days. The artificial in Artificial Intelligence is about constructing an artifice that acts intelligently. It’s about making an artificial process (man made) that responds in a good way. MCAS fits the bill. It is meant to react intelligently when sensors tell the system the plane is going outside the norm. Of course you might not be impressed with an A.I unless it performs as good as or better than a human. But there is only one artificial mechanism that currently performs better than a human, and it’s called science. Science is the best A.I. we have. As Alan Kay says, “Science is a better scientist than any scientist”.

                                                    1. 4

                                                      So do you consider PID controllers AI? Where does the line get drawn?

                                                      1. 4

                                                        You’re just losing your audience by using your definition of A.I. stretched beyond usefulness. A thermostat with if temperature < desired { heat() } behaves “intelligently”. If that’s an A.I., then the term has no useful meaning distinguishing it from just basic code. AFAIK MCAS wasn’t much smarter than a thermostat, except it used angle of attack as input, and nose down as output.

                                                        1. 2

                                                          That’s the issue with the definition of AI. The AI effect keeps moving the goalpost. Certainly someone 300 years ago would consider a thermostat intelligent, if not magical. I joke that if we ever get an AI as smart as people, we will discount it and say “yeah, well people aren’t really that smart anyway!”

                                                          I’m just trying to counter the AI effect ;-)

                                                          1. 3

                                                            There’s a fundamental difference between software that follows comprehensible rules divinable from its source code and software which does what it does based on an opaque neural net trained on so much data that its trainers (and even the people who designed the way to train it) have literally no idea why it did what it did. There are serious ethical, political and societal questions raised by the application of this latter type which will have drastic effects on real people’s lives. Deliberately blurring that distinction seems dangerous to me.

                                                            1. 2

                                                              We are building systems that interact dynamically with the real world. Whether we are building systems knowingly with if/else or whether the if/else is encoded in a neural net, the ethical outcome is the same. We have a responsibility in making systems that interact well with people. Neural nets make that harder.

                                                              Example, it would be unethical for me to put in a line of code “if woman then lower credit score”. It would also be unethical for me to release a neural net which does the same thing. What’s worse about the second one is simply me, the engineer, not knowing what it’s doing.

                                                              But from the person interacting with the system, they don’t know how I built it. They just know it’s acting in a really bad way. To the person interacting with the system, if it’s a neural net or an if/else, it’s a black box either way (unless they have the source code, which is why free software is so important).

                                                              What a system built with neural nets does it increase the unpredictability of the system. It’s a more fragile system. But our ethical responsibilities don’t change using neural nets. They just make building good systems harder.

                                                              1. 1

                                                                OK, sure, of course the ethical responsibilities are the same, and of course using software whose decision tree is illegible to humans makes shouldering them harder, if not impossible. To me that means not only “the risk we take by having neural nets make these decisions is greater than with software based on explicable rules”, but also “so don’t blur the two!” - but for some reason that I can’t fathom, you seem to be trying to blur them on purpose, and I honestly can’t make out what point you’re trying to support by doing so. Thing is that’s probably because it’s all got out of context now, lots of people chipping in on the same thread with different points, so maybe let’s just let it pass. Wish you well :-)

                                                                1. 2

                                                                  My whole original post was how dangerous Copilot is and that it will help propagate bugs into new software. I’m not trying to blur the two in that way because that distinction wasn’t even made. Everyone else is making some distinction between AI and not AI.

                                                                  But if you read my post carefully, it isn’t about neural nets or machines that think like people at all, but computer-human interaction. People here aren’t satisfied with my definition of A.I. So we are just having a discussion about that.

                                                                  Even with a thermostat, if it doesn’t let you know the target temperature, it could cook your cat alive while you are away. That point of interaction is critical for the safety of the system. But everyone keeps focusing on AI.

                                                                  I mean, it’s still an interesting discussion to talk about “What is AI” I guess.

                                                            2. 2

                                                              I know the common A.I. definition keeps shifting towards things we can’t do well yet, which is even more problematic, because widens the gap between what your readers understand as A.I., and what you wanted A.I. to mean. By “countering” it all the way back to ELIZA, you’ve just failed to communicate and distracted from the article’s subject.

                                                              1. 1

                                                                This would be true if I was just talking to a community of regular folks. But this is a community of professionals who should understand this stuff. I think the issue today is human-machine interaction is downplayed. Maybe people here forgot they are building machines that interact with people, animals, plants, and the real world? All programming is a form of making an AI if you really think about it. A user should expect software to respond intelligently to them and as far as I know, all software is artificial.

                                                          2. 1

                                                            MCAS is just an extension of control systems found on airplanes, cars, and many other vehicles today. Most aircraft, even general aviation aircraft, use fly-by-wire to move their control surfaces. Is that AI? Is an older aircraft’s trim wheel an AI? Where do we draw the line? Is a thermostat’s PID an AI? An automatic transmission car’s torque converter? An induction stove’s voltage controller?

                                                            1. 2

                                                              Ants have something as simple as a PID controller for finding food, maybe even simpler. Ants walk around randomly and when they find food, they return to their colony laying down a chemical trail. When another ant happens upon a chemical trail, they will follow it and if they find food, they will reinforce the trail.

                                                              It’s clear each ant has a form of intelligent behavior (though simple) and the collection of ants produce what’s called an ambient network of intelligent objects. It’s a natural intelligence. Ant’s adapt to a dynamic environment. That’s what intelligence is! (Fukada). In fact, an artificial version of this intelligence is used in optimization problems that are intractable otherwise called ant-colony optimization.

                                                              A simple thing like a PID controller is a form of artificial intelligence. A collection of PID controllers is even more intelligent. PID controllers themselves require tuning. Certainly people consider a perceptron in the category of AI. A PID controller can be implemented as a multi layer network (most functions can be). So if that’s the case, why isn’t a PID controller, which is tunable (has constants that need to be tuned) like weights in a network not AI? It’s even possible our own brain neurons have PID like mechanisms because our brain anticipates (predicts) values and corrects it’s model (learns) in a feedback loop.

                                                              The problem is, people have this idea that intelligence is what humans do. If simple ants are intelligent, and collection of ants is even more intelligent, then certainly man made processes that are similar are a form of artificial intelligence.

                                                              AI does not require neural nets and deep learning folks.

                                                              1. 2

                                                                A simple thing like a PID controller is a form of artificial intelligence. A collection of PID controllers is even more intelligent. PID controllers themselves require tuning. Certainly people consider a perceptron in the category of AI. A PID controller can be implemented as a multi layer network (most functions can be). So if that’s the case, why isn’t a PID controller, which is tunable (has constants that need to be tuned) like weights in a network not AI?

                                                                I understand this. I’m well aware of how all of these systems work; they’re all quite deeply intertwined mathematically, and yes the definition of “AI” as used in popular imagination is fuzzy.

                                                                AI does not require neural nets and deep learning folks.

                                                                I’m having trouble understanding the argument here and this feels a little like moving the goalposts. Is your opposition to Copilot that it’s using “intelligence”, like the PID you described? Is it about being an unexplainable black box, also like a PID, but unlike many other forms of “intelligence”, say a decision tree? Or is your opposition about neural nets and deep learning, specific forms of AI?

                                                                More importantly, why is this the line at which there is opposition? What makes the line you’re discussing here the important line to stop intelligence at and not earlier or later on this intelligence curve?

                                                                1. 2

                                                                  My argument is less about AI and more about human-machine interaction where the machines are really good. My opposition is not AI. It’s not “AI is bad, we should not use AI”. My opposition is that Copilot is too good, and therefore will create a lot of complacency. And because Copilot learned not only the good stuff (algos, models, semantics) but also the bad stuff (bugs) which it will happily include in your auto completed code.

                                                                  My argument is that people are really lazy and bad when it comes to monitoring (in this case, code reviewing) intelligent systems.

                                                                  And with programming it’s even worse, because the monitoring is basically a code review. And programmers are not taught to read code well. It’s not a skill that is rigorously taught. Many even hate it.

                                                1. 13

                                                  One non-technical aspect of the post I found interesting is the discussion of the difference in introduction styles between the US and Australia. As a non-US person I always feel rather uncomfortable with some US expectations about how we present people (at some point you are asked for a “short bio” and you are supposed to write in the third person that you are an award-winning this or that). I’m not sure I buy the idea that a more forceful introduction would have made a conversation about Sun stealing someone’s copyright much easier, but it’s interesting to see a positive aspect of “boastful” introductions highlighted here.

                                                  1. 19

                                                    As a US person, I feel uncomfortable with the whole “I have to make myself sound like the most important person in the room” vibe. Updating my resume to add accomplishments is pure horror for me.

                                                    1. 4

                                                      Thanks! Your experience reminds us that even in the US there are people who are not comfortable with this norm. (Maybe even most people? But it’s the norm so people strive to adapt themselves to the norm.) It’s also interesting to be reminded than other places have a different norm, and that even though many people feel more comfortable with those different norms they also have occasional downsides.

                                                      1. 3

                                                        The pessimist’s answer is that this advice feels like it’s for people in like, sales positions without morals. You know, sociopaths. (As someone who does talk to people in a sales capacity at times, I try to be honest with who I am and never try to “peacock”.)

                                                        1. 2

                                                          Or, to take a more useful framing, subcultures generally require less boasting when skills are easy to examine. Engineers can quickly assess each other’s skill levels (or, at least, they believe they can), so it makes sense for engineers to let their own abilities speak for them. What makes someone successful in sales and marketing is far more ineffable and not easily demonstrated at will. The culture therefore depends on people presenting their credentials directly in order for the group to negotiate consensus on who actually knows their stuff.

                                                          1. 4

                                                            Engineers can quickly assess each other’s skill levels (or, at least, they believe they can)

                                                            Do they though? I seem to see at least a popular post every week about how software engineering interviewing is broken and then most people agree this is a hard problem. And this isn’t even getting into sourcing and evaluating quickly if a profile is even worth interviewing.

                                                            1. 2

                                                              Yes, thus my parenthetical.

                                                    2. 4

                                                      Writing about myself in the third person is torturous. For some reason it feels really dishonest to bring up your own skills and accomplishments, even when it is the truth. If you didn’t have impostor syndrome before, you will now!

                                                      1. 2

                                                        What I think at such a time is: “What even are the skills and accomplishments that I could mention that aren’t just boring run-of-the mill things many people mention? I mean, I’ve done some things I thought were nice and that not everyone could do, but I also think many people can do, or have done, similar things. Judging otherwise would require knowing otherwise and I just don’t. I may believe I’m in the top few % of software developers, but that’s still a huge number of people and I don’t believe I’m extraordinary at anything to mention it.”

                                                        1. 2

                                                          If you’ve done something genuinely useful it doesn’t matter if other people have done similar things. You’re not looking to show that you’re better than them, you’re looking to show that you’re among them, that you’ve made some contribution. And… “there are many like it, but this one is mine.”

                                                          Some of the things I could mention include having a patch in a certain well-known open-source project, being a guest on a particular podcast, or being a technical reviewer for a book. None of those are earth-shattering, they’re all things that lots of people have done before me, but by mentioning them, I show that I’m engaged, and by saying which project, which book, which podcast, I tell people something about me and where my interests lie. Pretty simple, really.

                                                          And if you don’t have anything you want to highlight, you can just go with “Confusion has been writing software since $YEAR and currently works on $THING for $EMPLOYER” and leave it at that.

                                                      2. 2

                                                        I think of this more as a difference between business/sales culture and engineering culture. Business/sales is about hyping yourself up front, and engineering is about setting realistic low expectations and showing your skills through example. I think this is even true (especially true?) here in the US. I know I take it as a very strong warning sign if anyone touts their own accomplishments before I really know them.

                                                      1. 5

                                                        Someone I know has been playing around as well: https://github.com/fproulx/shc-covid19-decoder . He also has a piece of code to validate the digital signature with the government public key: https://github.com/fproulx/shc-covid19-decoder/blob/main/src/shc.js#L21

                                                        Also another blog post from someone else I saw shared around: https://marcan2020.medium.com/reversing-smart-health-cards-e765157fae9 (There’s an interesting notebook referenced at the end with steps to parse and create Smart Healthcard).

                                                        1. 9

                                                          Can someone sum up for me why one might like QUIC?

                                                          1. 13

                                                            Imagine you are visiting website and you try to fetch files:

                                                            • example.com/foo
                                                            • example.com/bar
                                                            • example.com/baz

                                                            In HTTP 1.1 you needed to have separate TCP connection for each of them to be able to fetch them in parallel. IIRC it was about 4 in most cases, which meant that if you tried to fetch example.com/qux and example.com/quux in addition to above, then one of the resources would wait. It doesn’t matter that the rest 4 could take a lot of time and could block the pipe, so it would do nothing until resource was fully fetched. So if by chance your slow resource was requested before fast resources, then it could slow whole page.

                                                            HTTP 2 fixed that by allowing multiplexing, fetching several files using the same pipe. That meant that you do no longer need to have multiple pipelines. However there is still problem. As TCP is stream of data, that mean that it need all packets before current to be received before processing given frame. That mean that single missing packet can slow down processing resources that are already received due to fact that we need to wait for marauder that can be retired over and over again.

                                                            HTTP 3 (aka HTTP over QUIC with few bells and whistles) is based on UDP and the streaming is build on top of that. That mean that each “logical” stream within single “connection” can be processed independently. It also adds few different things like:

                                                            • always encrypted communication
                                                            • multi homing (which is useful for example for mobile devices which can “reuse” connection when switching between carriers, for example switching from WiFi to cellular)
                                                            • reduced handshake for encryption
                                                            1. 9

                                                              Afaik, multihoming is proposed but not yet standardized. I know of no implementation which that supports it.

                                                              QUIC does have some other nice features though

                                                              • QUIC connections are independent of IP addresses. I.e. they survive IP address changes
                                                              • Fully encrypted headers: Added privacy and also flexibility. Makes it easier to experiment in the Internet without middleboxes interfereing
                                                              • Loss recovery is better than TCP’s
                                                              1. 4

                                                                Afaik, multihoming is proposed but not yet standardized

                                                                That is true, however it should be clarified that only applies to using multiple network paths simultaneously. As you mentioned, QUIC does fix the layering violation of TCP connections being identified partially by their IP address. So what OP described (reusing connections when switching from WiFi to cellular) already works. What doesn’t work yet is having both WiFi and cellular on at the same time.

                                                                1. 3

                                                                  Fully encrypted headers

                                                                  Aren’t headers already encrypted in HTTPS?

                                                                  1. 8

                                                                    HTTP headers, yes. TCP packet headers, no. HTTPS is HTTP over TLS over TCP. Anything at the TCP layer is unencrypted. In some scenarios, you start with HTTP before you redirect to HTTPS, so the initial HTTP request is unencrypted.

                                                                    1. 1

                                                                      They are. If they weren’t, it’d be substantially less useful considering that’s where cookies are sent.

                                                                      e: Though I think QUIC encrypts some stuff that HTTPS doesn’t.

                                                                  2. 3

                                                                    Why is this better than just making multiple tcp connections?

                                                                    1. 5

                                                                      TCP connection are not free, they required handshakes both at TCP level and SSL. They also consume resource at the OS level which can be significant for servers.

                                                                      1. 4

                                                                        Significantly more resources than managing quic connections?

                                                                        1. 4

                                                                          Yes, QUIC use UDP “under the table” so creation of new stream within existing connection is 100% free, as all you need is just to generate new stream ID (no need for communication between participants when creating new stream). So from the network stack viewpoint it is “free”.

                                                                          1. 3

                                                                            Note that this is true for current userspace implementations, but may not be true in the long term. For example, on FreeBSD you can do sendfile over a TLS connection and avoid a copy to userspace. With a userspace QUIC connection, that’s not possible. It’s going to end up needing at least some of the state to be moved into the kernel.

                                                                      2. 5

                                                                        There are also some headaches it causes around network congestion negotiation.

                                                                        Say I have 4 HTTP/1.1 connections instead of 1 HTTP/2 or HTTP/3 connection.

                                                                        Stateful firewalls use 4 entries instead of 1.

                                                                        All 4 connections independently ramp their speed up and down as their independent estimates of available throughput change. I suppose in theory a TCP stack could use congestion information from one to inform behaviour on the other 3, but in practice I believe they don’t.

                                                                        HTTP/1.1 requires single-duplex transfer on each connection (don’t send second request until entirety of first reponse arrives, can’t start sending second response before entirety of second request arrives). This makes it hard for individual requests to get up to max throughput, except when the bodies are very large, because the data flows in each direction keep slamming shut then opening all the way back up.

                                                                        AIUI having 4 times as many connections is a bit like executing a tiny Sybil attack, in the context of multiple applications competing for bandwidth over a contended link. You show up acting like 4 people who are bad at using TCP instead of 1 person who is good at using TCP. ;)

                                                                        On Windows the number of TCP connections you can open at once by default is surprisingly low for some reason. ;p

                                                                        HTTP/2 and so on are really not meant to make an individual server be able to serve more clients. They deliberately spend more server CPU on each client in order to give each client a better experience.

                                                                      3. 2

                                                                        In theory, HTTP 1.1 allowed pipelining requests: https://en.wikipedia.org/wiki/HTTP_pipelining which allowed multiple, simulteneous fetches over a single TCP connection.

                                                                        I’m not sure how broadly it was used.

                                                                        1. 4

                                                                          Pipeline still require each document to be sent in order. A single slow request clog the pipeline. Also, from Wikipedia, it appears to not be broadly used due to buggy implementation and limited proxy support.

                                                                          1. 3

                                                                            QUIC avoids head-of-line blocking. You do one handshake to get an encrypted connection but after that the packet delivery for each stream is independent. If one packet is dropped then it delays the remaining packets in that stream but not others. This significantly improves latency compared to HTTP pipelining.

                                                                        2. 5

                                                                          A non-HTTP-oriented answer: It gives you multiple independent data streams over a single connection using a single port, without needing to write your own framing/multiplexing protocol. Streams are lightweight, so you can basically create as many of them as you desire and they will all be multiplexed over the same port. Whether or not streams are ordered or send in unordered chunks is up to you. You can also choose to transmit data unreliably; this appears to be a slightly secondary functionality, but at least the implementation I looked at (quinn) provides operations to you like “find my maximum MTU size” and “estimate RTT” that you will need anyway if you want to use UDP for low-latency unreliable stuff such sending as media or game data.

                                                                        1. 5

                                                                          So what’s the backend database? Is it an “real” graph database (like Neo4J)?

                                                                          1. 26

                                                                            We actually tried Neo4J in an early version of Notion (before my time) but it was very slow for the kinds of access Notion does.

                                                                            We try to use very boring technology. Our source-of-truth data store is Postgres, with a Memcached cache on top. Most of our queries are “pointer chasing” - we follow a reference from one record in memory to fetch another record from the data store. To optimize recursive pointer-chasing queries, we cache the set of visited pointers in Memcached.

                                                                            1. 5

                                                                              If could view your data structures as trees, rather than graphs, Postgres has ltree module, I use it for taxonomies, works very well (our performance loads are not very big, so cannot comment on very large deployments experience). Some times ago, I traced inner joins operations of regular tables of JSOB fields with our taxonomy trees, and works well. Postgres looks at indices as appropriate.

                                                                              1. 7

                                                                                I looked at ltree a few years ago when I joined Notion. When I first saw the DB access pattern I asked, “O(n) queries for a page?!? What! This should be O(1)!”. But I believe for ltree, to move a parent node, we’d need to rewrite the path on all descendants, so we get O(1) queries for page reads, but moving a block becomes an O(n) rows updated. That kind of write amplification fanout isn’t worth the more efficient read path.

                                                                                I think this kind of shock is common for relational database users coming from small data apps to much bigger data apps — when your data is big enough, you also have to give up JOIN and FK constraints. It’s just not worth the database resources at a certain scale.

                                                                                1. 1

                                                                                  Yes, makes sense. In my case taxonomy trees are static data (changed rarely), so there are no writes into them. And they are negligible in size, compared to operational data that uses them.

                                                                                  I can see that your system uses trees or graphs for operational data, so writing performance is a key criteria.

                                                                              2. 3

                                                                                Heh, can’t fault you for choosing Postgres! I assume you have many graphs, that aren’t particularly deep? From my understanding of what blocks are used for, it seems like you’d have graphs that are wider than they are deep, and … hundreds of nodes in total? A few thousand? Any idea what the average and largest number of nodes in a graph is?

                                                                                (I am assuming there’s one graph per document?)

                                                                                1. 5

                                                                                  It’s more like a tree descending from the root of a workspace, and a workspace can have 1000+ users in it collaborating on shared pages or working in private. But pages are just a type of block, and like other blocks, can be nested infinitely. To render a page, we crawl recursively down from the “page” block, and we stop at page boundaries, which are just other page blocks below in the tree. (There is a cursor system for doing this incrementally and prioritizing the blocks at the “beginning” of a page that the user needs to see first.).

                                                                                  So, spaces are quite a broad and deep graph, but the scope is about what you estimate for an individual page. I don’t have estimate numbers on hand for depth or block count within a space.

                                                                                  1. 1

                                                                                    Thanks, it’s been really interesting to learn about this :)

                                                                                2. 1

                                                                                  Did you experiment with key value store like lmdb, LevelDB or Rocksdb (Or higher level dbms such as Riak or Myrocks)? I was actually looking to experiment with graphs in the database and was wondering how using these more specialized engine would perform. I suppose from an operational point of view, Postgresql is very nice when you already use it.

                                                                                1. 29

                                                                                  Weird, I definitely disagree on expanding this. It just wastes horizontal space and keystrokes for no benefit.

                                                                                  Maybe I32 and Str instead of i32 and str, but abbreviations for commonly used things are good. You’re not even getting rid of the abbreviation, Int is after all short for Integer.

                                                                                  1. 9

                                                                                    I agree with this (I think lowercase would be fine, too, though).

                                                                                    I think that Rust overdoes it a little bit on the terseness.

                                                                                    I understand that Rust is a systems language and that Unix greybeards love only typing two or three characters per thing, but there’s something to be said for being descriptive.

                                                                                    Examples of very terse things that might be confusing to a non-expert programmer:

                                                                                    • Vec
                                                                                    • fn
                                                                                    • i32, u32, etc
                                                                                    • str
                                                                                    • foo.len() for length/size
                                                                                    • mod for module
                                                                                    • mut - this keyword is wrong anyway
                                                                                    • impl

                                                                                    None of the above bothered me when I learned Rust, but I already had lots of experience with C++ and other languages, so I knew that Vec was short for “vector” immediately. But what if I had come from a language with “lists” rather than “vectors”? It might be a bit confusing.

                                                                                    And I’m not saying I would change all/most of the above, either. But maybe we could tolerate a few of them being a little more descriptive. I’d say i32 -> int32, Vec -> Vector, len() -> count() or length() or size(), and mut -> uniq or something.

                                                                                    1. 11

                                                                                      mut -> uniq

                                                                                      Definitely this!

                                                                                      For the context of those who aren’t familiar, &mut pointers are really more about guaranteeing uniqueness than mutability. The property &mut pointers guarantee is that there is only one pointing at a given object at a time, and that nothing access that object except through them while they exist.

                                                                                      Mut isn’t really correct because you can have mutability through a & pointer using Cell types. You can have nearly no mutability through a &mut pointer by just not implementing any mutable methods on the type (though you can’t stop people from doing *mut_ptr = new_value()).

                                                                                      The decision to call this mut was to be similar to let mut x = 3… I’m still unconvinced by that argument.

                                                                                      1. 4

                                                                                        Not to mention the holy war over whether let mut x = 3 should even exist, or if every binding is inherently a mutable binding since you aren’t actually prevented from turning a non-mutable binding into a mutable one:

                                                                                        let x = 3;
                                                                                        let mut x = x;
                                                                                        // mutate the ever living crap out of x
                                                                                        
                                                                                        1. 4

                                                                                          My favorite is {x} allowing mutability, because now you’re not accessing x, but a temporary value returned by {}.

                                                                                          1. 2

                                                                                            I never knew about that one! Cute.

                                                                                      2. 11

                                                                                        For an example, check out some Swift code. Swift more or less took Rust’s syntax and made it a little more verbose. fn became func, the main integer type is Int, sequence length is .count, function arguments idiomatically have labels most of the time, and so on. The emphasis is on clarity, particularly clarity at the point of use of a symbol — a function should make sense where you find a call to it, not just at its own declaration. Conciseness is desirable, but after clarity.

                                                                                        1. 2

                                                                                          Yep. I also work with Swift and I do like some of those choices. I still think the function param labels are weird, though. But that’s another topic. :)

                                                                                        2. 4

                                                                                          I think this mostly doesn’t matter - I doubt anyone would first-try Rust, given its complexity, so it’s not really that much of an issue. Keywords are all sort of arbitrary anyway, and you’re just gonna have to learn them. Who’d think go would spawn a thread?

                                                                                          I, for one, think these are pretty nice - many people will learn Python so they expect len and str, and fn and mod are OK abbreviations. I think the terseness makes Rust code look nice (I sorta like looking at Rust code).

                                                                                          Though I’d agree on mut (quite misleading) and impl(implement what?).

                                                                                        3. 2

                                                                                          Oh, true.

                                                                                          I don’t care about the exact naming conventions, as long as it is consistent. (This is in fact exactly how I named types in my project though, what a coincidence. :-D)

                                                                                          In general the random abbreviations of everything, everywhere are pretty annoying.

                                                                                          1. 3

                                                                                            It’s the consistency, yes. Why should some types be written with minuscles?

                                                                                            1. 4

                                                                                              Lowercase types are primitive types while camelcase are library types. One has special support from the compiler and usually map to the machine instructions set while the other could be implemented as a 3rd party library.

                                                                                              1. 4

                                                                                                Because they are stack-allocated primitive types that implement Copy, unlike the other types which are not guaranteed to be stack-allocated and are definitely not primitive types.

                                                                                                1. 1

                                                                                                  And how does the lower-case letter convey this fact?

                                                                                                  1. 4

                                                                                                    How does anything convey anything? It’s a visual signal that the type is a primitive, stack-allocated type with copy semantics. As much as I hate to defend Java, it’s similar to the int/Integer dichotomy. If they were Int and Integer, it wouldn’t be quite so clear that one is a primitive type and the other is a class.

                                                                                                    1. 3

                                                                                                      I just searched for “Rust stack allocation lower-case” but couldn’t find anything. Do you have a link that explains the connection?

                                                                                                2. 2

                                                                                                  Because they are used more than anything else :)

                                                                                                  (and really it should be s32 to match u32)

                                                                                                  1. 1

                                                                                                    Not a good reason. Code is read way more often than it is written.

                                                                                                    1. 4

                                                                                                      I don’t see how i32 is less readable. It makes the code overall more readable by making lines shorter and looks better.

                                                                                            1. 2

                                                                                              One issue not addressed here is that plain text protocols are typically hand implemented (a recursive descent parser), where binary protocols are often machine generated (protobuf, grpc, etc.). HTTP is in theory parse-able by a number of parser generators, but in practice it and other text based protocols are hand implemented, leading to bugs, security vulns, and other problems.

                                                                                              The ideal seems to be something like HTTP, where a client/server can fall back to 1.1 if 2.0 isn’t jointly supported, and is general enough to support most anything you’d want to do with a protocol. Similarly, most machine generated formats like protobuf have a text based format as fallback.

                                                                                              1. 7

                                                                                                And this, in turn, leads to security problems. Parsing untrusted data is one of the biggest sources of security vulnerabilities. The easier a protocol is to parse, the easier it is to secure. Most binary protocols require trivial binary pattern matching to parse. About the only check that you ever need to do is whether an offset is contained within a packet and even in an unsafe language it’s pretty easy to abstract that away into a single place. Binary protocols can often just use the header information as-is and parse lazily. Flat Buffers, for example, just uses the wire-protocol message in a buffer and provides accessors that do any offset calculation or endian conversion necessary. The cost of parsing is so low that you can re-parse a field every time you access it.

                                                                                                1. 1

                                                                                                  I feel like if the issue is that text protocal are using handcrafted and unsecure parser, the ideal would then be to have client/server that only use 2.0. Allowing fallback is status-quo and doesn’t fix any security hole.

                                                                                                  The way I see it, the ideal would be to have some non-default dev-mode where text protocol are enabled, and/or those text protocol only support the simplest subset of robust features.

                                                                                                  1. 1

                                                                                                    Grpc is huge. It depends on a large amount of hand written parsing code. Using it is unlikely to reduce the amount of hand written parsing in your system.

                                                                                                    I don’t mind binary protocols, especially if I can handle them with something like Python’s struct module, but grpc is just a bad example. It’s amazing how little functionality they packed into such a huge amount of code.

                                                                                                  1. 17

                                                                                                    Where do we go from here? What can a freedom-minded person do to avoid censorship by tech oligarchs?

                                                                                                    Nearly stopped reading here, but I’ll happily post it again and again: It’s not censorship if one company chooses to not do business with you.

                                                                                                    1. 14

                                                                                                      That’s actually still censorship:

                                                                                                      tr.v. cen·sored, cen·sor·ing, cen·sors To examine and expurgate.

                                                                                                      There’s a subtler meaning in there around “but it isn’t the government doing it”, but given the size of Google, Cloudflare, Facebook, and others who have successfully walled-in the public square, it is pretty disingenuous to pretend like there isn’t at least something going on there.

                                                                                                      For the folks going “hah, so what if it happens to people I don’t like?”–remember that time Tumblr’s ban hurt LGBTQ+ folks? Remember the various pro-BLM folks Twitter banned? Pepperidge Farm remembers.

                                                                                                      It’s completely reasonable for people to be concerned and want to learn how to host their own services, and mocking them for attempted independence seems to me to be both short-sighted and a defection against the hacker spirit.

                                                                                                      1. 2

                                                                                                        a defection against the hacker spirit

                                                                                                        Also on that topic, I’m reminded of the part in chapter 6 of Hackers by Steven Levy, where the MIT AI Lab hackers hated Multics in part because of its fine-grained usage accounting. Kind of like AWS and similar services, no?

                                                                                                        (So yes, the fact that my current project is all-in on AWS causes me cognitive dissonance. Not sure how to resolve it though. Multi-AZ deployments with automated recovery from instance failures are certainly good for peace of mind.)

                                                                                                        1. 2

                                                                                                          Think about what infrastructure changes you would need to make in order not to be fully dependent on AWS, and then make them. Even if you don’t switch away from AWS immediately, being prepared to do so will make it easier on you if they do decide for whatever reason to deplatformed you, or if a competing cloud provider starts offering a better deal.

                                                                                                      2. 9

                                                                                                        maybe it is if the handful of companies that are powerful than most countries decide not to do business with you.

                                                                                                        Regardless of that highly charged political question, I think fighting oligarchy is worthwhile in itself.

                                                                                                        1. 7

                                                                                                          What is it when all the infrastructure providers, payment processors, banks, and social media platforms all decide to stop doing business with you?

                                                                                                          So glad my beliefs are currently in vogue with whatever you call that collective, whatever their non-censorship is, I’m glad I’m not being subjected to it.

                                                                                                          1. 3

                                                                                                            I followed this saga at the time, and my impression was that AWS bent over backwards to accommodate this service. It was only after those responsible failed to moderate the statements made by their users that violated the ToS they had willingly agreed to that service was suspended - not terminated.

                                                                                                            The site is back online. The service is not, which really makes one wonder how much this is a genuine wish to offer a free-speech platform and how much it was an attempt to soak a well-healed backer for a lot of money.

                                                                                                            1. 4

                                                                                                              In what way did they bend over backwards? Was there even a court order telling them to take the site down?

                                                                                                              1. 3

                                                                                                                AWS gave them multiple chances to implement effective moderation.

                                                                                                                There was no court order, the issue was a breach of contract (the ToS).

                                                                                                                I can recommend Techdirt’s coverage of the issue, with this opinion piece as a good starting point https://www.techdirt.com/articles/20210115/00240746061/few-more-thoughts-total-deplatforming-parler-infrastructure-content-moderation.shtml.

                                                                                                                https://www.techdirt.com/search-g.php?q=parler

                                                                                                                1. 4

                                                                                                                  That writer seems to think any action is justified as long as it’s done by private companies in a competitive free market. Under that assumption, of course a breach of contract is more than enough reason to suspend service.

                                                                                                                  But if we care about who actually has power in society and how communication is shaped by different actors, this cold comfort. If AWS “bends over backwards” to offer a service they are being paid for, until the risk of a PR crisis makes it not worth it for them, they are still wielding unaccountable power to limit who gets to speak.

                                                                                                                  1. 2

                                                                                                                    Masnick’s position is more nuanced than you summarize it, but the idea of private parties competing in a free market is one that has served the US economy well for a long time.

                                                                                                                    As for AWS “wielding unaccountable power”, they’re far from a monopoly. Oracle is gunning aggressively for their business, as mentioned in other threads on this very page, and in the article I linked.

                                                                                                                    And whose speech are we talking about? The users of the site are free to create accounts elsewhere, and many have surely done so. What’s left is the limited speech rights of the service to make money hosting these users. This right has to be weighed against AWS’ rights to make money providing cloud computing to many other customers, all of whom AWS is aware can change providers if AWS allows toxic actors on its service.

                                                                                                                    If the service’s business model was to make money hosting speech that was banned elsewhere, it bordered on criminal negligence not to take the risk of being suspended into account, and making plans to shift hosting providers accordingly. Again, this points to this being a grift rather than a sound business idea.

                                                                                                                    1. 2

                                                                                                                      I think it is perverse to give any weight to a large company’s “right” to make money in a certain way, when weighed against issues that affect the mass of society. But I suspect you and I disagree fundamentally about this, so there’s probably no use trying to find agreement.

                                                                                                                      Another thing we will probably disagree about: the potential for competition among a small group of companies does not amount to accountability. People have very little say over what these companies do.

                                                                                                          2. 9

                                                                                                            Agreed. i appreciate the meat of the article but I can definitely do without the edgy quote at the beginning.

                                                                                                            1. 5

                                                                                                              Ditto with the mentions to Parler (including in the title)

                                                                                                            2. 7

                                                                                                              haha agreed, this the second time I’ve seen someone cite the “First they came for” poem to defend groups of people actively “coming for” me and my loved ones.

                                                                                                              1. 8

                                                                                                                Always feel weird when this kind of content use proto-fascism and right-wing extremists as an example for why we need to fight oligarch censorship. Of all the victims of censorship, those are the one I could not care less.

                                                                                                                1. 6

                                                                                                                  Of all the victims of censorship, those are the one I could not care less.

                                                                                                                  I think that’s rather the point of the poem, isn’t it?

                                                                                                                  1. 4

                                                                                                                    No. Not when those being censored are the very same people that would want to be the “They” in the poem. The poem is that we shouldn’t stay idle while some group is trying to take advantage over other groups. I feel like what happened right now is that someone actually did speak up…

                                                                                                                    1. 2

                                                                                                                      Not when those being censored are the very same people that would want to be the “They” in the poem.

                                                                                                                      Plenty of people in Weimar Germany felt that way about the communists.

                                                                                                                    2. 2

                                                                                                                      Not really, at least in my reading. It’s not about the dangers of a “couldn’t care less” mindset but rather one of cowardice.

                                                                                                                    3. 2

                                                                                                                      HUAC was first used to jail Nazi sympathizers

                                                                                                                      1. 2

                                                                                                                        And good for them, let them rot in jail for all I care. I don’t mean that we should stand idle while states and corporations consolidate their power. Let’s speak about how HUAC or oligarchs can use their power monopoly and use it against the rest of the population. Let’s discuss about how we can fight against these crackdown in the point of view of freedom and privacy, not about how a right-wing extremist community should have done better.

                                                                                                                        1. 1

                                                                                                                          Who is discussing how a right-wing extremist community “should have done better”? You lost me there.

                                                                                                                          I guess you don’t value the legal and social norm of free speech as such, and take no issue if that norm is violated to target racist groups. You either don’t think that makes it easier to target non-racist groups, or you don’t care.

                                                                                                                          1. 3

                                                                                                                            Who is discussing how a right-wing extremist community “should have done better”?

                                                                                                                            Taken as-is from the article:

                                                                                                                            Parler’s epic fail: A crash course on running your own servers with a shoestring budget

                                                                                                                            I argue that your chances of survival are much better this way, and Parler is foolish for not going this route. We can do better.

                                                                                                                            Parler was cut off by their cloud hosting provider, Amazon. Where do we go from here? What can a freedom-minded person do to avoid censorship by tech oligarchs?

                                                                                                                            1. 1

                                                                                                                              ah

                                                                                                                    4. 2

                                                                                                                      whos coming for you

                                                                                                                    5. 3

                                                                                                                      Hosting your own content is exactly the method by which you work around private platform companies refusing to do business with you for political reasons.

                                                                                                                    1. 25

                                                                                                                      It’s not that big of a deal. Just do as documented in Whatever language. If one is better than the other, it won’t be more than a small detail advantage. Let it go.

                                                                                                                      I can totally see how non tech people find amusing how nerds care about such small silly things.

                                                                                                                      1. 17

                                                                                                                        We should have a tag for bikeshedding. I think most fields, tech or non-tech have similar silly arguments. Get into a climbing group and ask them about the best cams out there and why they are using hexes or tricams, you can be sure to have them argue for a while. Meanwhile this leaves the whole crag for yourself to climb! Caring about pointless details is always easier than caring about getting actual things done and I have my fair share of guilt at that!