1. 5

    Honestly, I feel the ACM prices are fair.

    Membership with library access is only $198 a year, and realistically that’s not a burden for most people who would want access.

    Furthermore, as the author points out, “The ACM exists to serve the field of computing and society itself,” and the main way they fund that work is through membership fees and charging for library access. The money has to come from somewhere.

    1. 6

      If journal access is the only value that the ACM can provide for their membership fees, then they don’t have much of a legitimate reason to exist. Real professional societies do much more, both for their members and for society at large. I can appreciate that the ACM doesn’t have anything close to the kind of leverage that they would need to really fulfill their mission, but they’ll never get there with that kind of petty gatekeeping mindset and revenue model. It’s simply off-putting.

      Here’s what Scott Aaronson had to say about the practice of charging for access to academic research. Please do follow the links to statements from John Baez and Donald Knuth, too.

      1. 3

        Journal access isn’t the only value they provide, it’s the only thing they provide that gets them net revenue. They also do things like run conferences, running the library, and “magazines”. You can read the full breakdown here.

        1. 2

          Membership is what brings them revenue. Tying that to journal access is just a marketing strategy.

      2. 4

        I looked into getting ACM membership, but unfortunately I did not agree with all of their principles and other things as stated, and did not wish to compromise my own integrity or insult their stance by agreeing to a membership just so I could download articles.

        1. 7

          Huh, I admit I didn’t read the principles very carefully, but after skimming through them these all seem pretty reasonable to me. Could you say which ones you don’t agree with?

          1. 8

            For what it’s worth, while I don’t like the ACM’s code of ethics I do support something in the form of the ASME code of ethics or the AlChE code of ethics.

            A big difference I see between the ACM and those other forms is that there is a focus not on some lofty idealism but instead on the realpolitick of a guild looking to protect its interest. An extremely cynical reading of those classical engineering ethics suggests an impetus of “Hey, look, people don’t understand what we do and they’re gonna blame us if we fuck up. so, we need to explicitly always be trying to serve the public good so that they trust us, ensure we don’t engage in buddyfucking that would harm our guild, and act in a generally moral way that means our mandate to charge what we do to do the work we do to the standards we want is not revoked.”

            I’ll point at a few to give the flavor, though I think @asthasr has a good explanation of overall what bothers me.

            In section 1.2, “harm” is vague and made worse by the frequent qualifier of “unjustified”. If somebody thinks something is “justified”, poof, there goes the protections.

            The requirement to use “generally accepted best practices” is so at odds with how software is actually developed today that it shakes my belief in the empirical grounding of any of what follows. Ditto the absurd idea that “consequences of data aggregation and emergent properties of systems should be carefully analyzed”–not that those sentiments are wrong, but they’re so clearly not how we do software “engineering” outside of a few niche applications.

            The entire bit about “capricious or misguided reporting of risks” is basically useless–practitioners must report risks, expect when there’s a chance the reporting might introduce “harm”. So, your obligation is to report issues except when reporting it might have further issues. The ASME criteria (criteria section, 1.3.1) is a lot more straightforward:

            Whenever the Engineers’ professional judgments are over ruled under circumstances where the safety, health, and welfare of the public are endangered, the Engineers shall inform their clients and/or employers of the possible consequences.

            You report the risks, always. If they go ahead with it and you believe there is imminent danger to the public, you go to the authorities. You don’t have this “a computing professional should carefully assess relevant aspects of the situation” nonsense–if you see something, you say something, and if it’s bad enough to endanger the public you fucking report it.

            The problem with the ACM stuff here is that, frankly, it’s very rare that any individual software system is going to pose such an obvious threat to life and limb (compared with, say, a bridge) that a practitioner is going to risk souring their relationship with their work over it. Only recently have we seen any approximation of this behavior with the walkouts over ICE contracts or what have you, but that’s been quite limited.

            In section 1.4, the wording of discrimination includes the phrase “inappropriate factor”. This implies that there are in fact appropriate factors for discrimination, and thus some subset of people that should be discriminated against. This goes directly against (in my reading) the immediately preceding claim of that we “should foster fair participation of all people, including those of underrepresented groups.”. Well, which is it?

            It goes on to talk about “computing professionals should take action to avoid creating systems or technologies that disenfranchise or oppress people.” What exactly does this mean? If I work on software for ultrasound used in abortions, am I not aiding in disenfranchising or oppressing future people? If I work on door lock software for prisons where we put violent white nationalists, am I not doing the same? What about banking software that by necessity only serves those with some pre-existing form of wealth?

            Section 1.5 is about supporting patents. In the first paragraph, we are told to “respect copyrights, patents, trade secrets, license agreements, and other methods of protecting authors’ works.” In the very next paragraph, “Computing professionals should not unduly oppose reasonable uses of their intellectual works”. Again, which is it? The entire bloody point of patents is to prevent any unauthorized use of intellectual work (for a good cause or no!) and grant a limited monopoly so as to encourage further innovation. The invocation of “protecting of authors’ works” is supporting a fictional narrative where we don’t live in a world where every salaried or engaged software engineer signs away their rights as parts of Assignments of Invention.

            It goes on and on, and I won’t take up more space here.

            The real failure of the ACM document is that it’s so goddamn long. Any sort of reasonable ethical boundaries need to be succinct, trivially seen to be representative of common practice, and ideally not containing text that directly contradicts itself. The ACM Code fails on all three of those counts for me.

            1. 2

              The problem with these sorts of lists is that they are meaningless without an ethics. What does it mean to “do no harm?” What is the “public good?” What is the justifying authority behind the “compelling belief” that accessing someone else’s system would help the public good?

              Without an ethical basis this is worse than useless: it’s gormless sloganeering.

              1. 2

                i don’t think you have to provide a clear plan before saying you want to do no harm

                1. 2

                  So what if I think it’s very harmful not to send your data to my religious organization for inspection to make sure you’re not sinning? That’s clearly “justified” disclosure of information.

            2. 6

              Woof. I see what you mean.

              They’re really walking themselves out onto an ethical cliff, aren’t they?

              I’ve never gone for membership simply because the price/value equation never made sense to me, but having read the above now I have two reasons to avoid it :)

              I own a hardbound set of volumes 1-3 of their History of Programming Languages series though, and treasure them :)

              1. 6

                Any chance you can point out the problematic principle?

                1. 5

                  It’s less about one single principle, and more about the fact that many of these are an INCREDIBLY slippery slope.

                  Let me give you an example:

                  3.1 Ensure that the public good is the central concern during all professional computing work.

                  Step back and think about that for a moment.

                  Is anything any of us ever does in this industry short of those of us doing charity work REALLY in the public good?

                  Lots of folks think writing open source software is in the public good. Is it? It encourages the use of computers, which have some VERY serious environmental impacts both in terms of their manufacture and their use at scale.

                  It’s a very slippery slope. I appreciate the fact that they are striving to live and work by a set of ideals, but my point is that trying to reduce those ideals down to an actionable set of guidelines for our daily work is an incredibly daunting task, and possibly impossible.

                  1. 2

                    Is anything any of us ever does in this industry short of those of us doing charity work REALLY in the public good?

                    Yes! (After a long career of “not really”).

                    World would be a markedly worse place for humans without the stuff I get to work on (a widely used study aid for medicine), including a few people who are alive today because their doctors were able to look things up (in much of the world the specialist textbooks you would otherwise need are prohibitively expensive).

                    1. 2

                      As usual I phrased that poorly.

                      I wasn’t saying that such jobs don’t exist, because I’ve worked at jobs like that as well.

                      What I was ham fisted-ly trying to communicate is that not everyone is so lucky, and so putting that in their list of tenets is creating a moral quandary where there doesn’t need to be one.

                      Do no harm is enough for me.

              2. 4

                I’m sure you’re not the only one. For as toothless and out-of-touch as they are, they sure do a lot of purity testing and virtue signalling. It just serves to keep them irrelevant outside academia… which in turn sure doesn’t enhance the relevance of academia to industrial practice. The whole thing is a sad mess, IMHO.

                1. 3

                  Seems like a pretty reasonable list, and the baseline of ethical, professional responsibility for somebody working in the field of computing. What in particular do you have an issue with?

                  1. 1

                    Interesting. You may not want to explain on a public forum but that’s some really mild stuff. If you dropped the “computing” phrases it wouldn’t be out of line in a list of Abrahamic religion’s sermon topics (and maybe other faiths, I don’t have first hand experience in those).

                1. 28

                  I’m not meaning to dissuade the SourceHut developer(s) from using GraphQL, but I am still on the fence about GraphQL, having used it at $employer for, oh, 4-ish months now. GraphQL is a shift in thinking, and there are sometimes several ways to provide data for a given business need.

                  In any system with models and DB tables, some tables/classes/types have more than one link to them (in an ERD). For example, an Article could have an Author, but it could also be in a MagazineIssue. When you have multiple pathways to reach a given entity, you (as the developer of a given use case, web page, whatever) have to decide which path you’re going to code in your query. Carrying on with the example schema: It could be currentUser -> articlesWritten; or it could be currentUser -> magazinesSubscribedTo -> issues -> articles. This kind of multiplicity of paths increases the complexity of the system, and makes the system more challenging to work with.

                  Authorization with GraphQL is quite non-trivial. Suppose you had: Authors organize with Folders, in which they put Articles; and then Readers read Articles. A simple schema would be to have this heirarchy: authors -> folders -> articles. Folders would have-many articles, and articles would belong-to folders. But if you have a query that needs to get articles for a reader to read, if you stick with this simple schema, then you have to include folders in the query. But readers should not have access to or knowledge about the folder the author has put the article in. Or maybe the reader doesn’t care about who wrote what, and wants to get articles on a given topic or tag, like “politics”. Worse yet: a reader [UI] should not be able to query “upstream”, and get at article -> folder -> author -> author.email. “So don’t put upstream association links in your schema” you might say. But is it so simple? What if a pair of entities are not in a simple, obvious hierarchy where one is “higher” than the other. Say, a Person model with a friendsWith association.

                  I grant that there may be some correct way(s) to go about things to deal with issues like the above, but GraphQL doesn’t come with guardrails to prevent problematic schemas, and I think it’s not only possible, but somewhat likely for a team new to GraphQL to plant footguns and landmines in parts of their schema.

                  So, yeah. I’m still on the fence about this GraphQL thing.

                  1. 24

                    Indeed. At GitHub we had a VP who was All About GraphQL – we GraphQL’d everything, over a period of a couple years. By the time I left, there were RFCs circulating about how we’d undo all the damage done :|

                    1. 23

                      Could you describe some of the damage which had been done? What did the push for GraphQL break?

                      1. 6

                        I’m curious about this as well.

                      2. 18

                        Thanks for these comments. I would appreciate more specific feedback, if you have the time to provide some. You can use our equivalent of the GraphQL playground here:

                        https://git.sr.ht/graphql

                        Expand the text at the bottom to see the whole schema for git.sr.ht. This went through many design revisions, and I’m personally pretty satisfied with the results.

                        Also, specifically on the subject of authorization, the library we’re using provides good primitives for locking down access to specific paths under specific conditions, and combined with a re-application of some techniques we developed in Python, this is basically a non-problem for us. I’m still working on an improved approach to authentication, but none of the open questions have anything to do with GraphQL.

                        1. 2

                          Could I ask about this part?

                          GraphQL does not solve many of the problems I would have hoped it would solve. It does not represent, in my opinion, the ultimate answer to the question of how we build a good web API.

                          Which additional problems would you like to see solved? In which direction might the ultimate answer lie?

                          1. 9

                            I don’t think it solves pagination well. It should have first-class primitives for this. That’s the main issue.

                        2. 9

                          Same here, except we’ve had lackluster success with GraphQL for 18-ish months instead of 4.

                          I could see it being a big step up for data models that are oriented primarily around a graph, but I’m having a really hard time seeing how sourcehut’s would work this way, outside of the git commits themselves. I’d be interested in reading more about the dissatisfaction with REST, because everything we’ve done so far with GraphQL could have been done better with REST along with something like JSON-schema.

                          1. 2

                            For small projects, I’ve started just using json-rpc (what’s old is new again!) with a well defined schema (with validation). Nice side effect is the schema gets used as part of the documentation generation.
                            Saves lots of time noodling around with REST. I probably wouldn’t recommend it for large projects though, as it is a bit harder to version apis (a bit more “client fragile”).

                            I’ve used GraphQL on a few projects, and I thought it was ok. It certainly has its own set of problems.

                          2. 4

                            Caching and pagination are also issues I’ve seen with graphql, as well as many many failures around n+1 queries.

                            1. 1

                              About the n+1 problem:

                              Disclaimer: I have only dabbled with graphql and never used it professionally.

                              Let’s compare this to a REST API (or what would be a better reference point?). In a REST API, you also have the n+1 problem if your entities refer to other entities.

                              It is just more common to solve this on the client rather than on the server – which is worse due to higher roundtrips. Alternatively, you provide a denormalized view in your API that provides all data for, let’s say a screen or page, on the server. That means that you have to change your server in lockstep with the client – and also worry about versioning. In the server code you can either use a generalized solution (much like with GraphQL) or a specialized version because it only has to work with this one query. In my opinion, a generalized version is also usually the preferred option but it is nice to have the option for hand-optimization.

                              Therefore I think that solving the n+1 problem in GraphQL requires some thought - but also in a REST API. There are good enough solutions. On one hand, since you have more information in your schema, you actually have some more options than in REST. On the other hand, REST is more constrained so you can work with more specialized solutions.

                              Am I missing something important?

                              1. 1

                                Pagination is not really a problem, you just have to make a type specifically for pagination.

                                So for an Article you would have an ArticleConnection containing… yeah just use REST it’s way more sane.

                                1. 4

                                  One thing to consider is that “just have to make a type” relies on the fact that GraphQL fields can have parameters. This introduces a challenge for caching, since those parameters become part of your cache key. If you’re trying to take advantage of a normalizing graph cache on your client, the cache engine needs to be aware of the pagination primitives.

                                  Not advocating for REST, since you have a similar problem there. However, REST makes some static decisions about what fields are included and so the cache key is reasonably a URL instead of a GraphQL AST + variables map. If you decide you don’t want a normalizing cache, but do want a naive document cache ah la REST, managed by the browser, you can’t get that unless you have named queries stored on the server. Then you can query for /query/someName?x=1&y=2 sort of thing.

                                  I guess my point is that nothing with GraphQL is “just” anything. You’re discarding a lot of builtin stuff from the browser and have to recreate it yourself through cooperation of client JavaScript and specialized server code. Whether or not that’s worthwhile really depends on what you’re building.

                              2. 4

                                When you have multiple pathways to reach a given entity, you (as the developer of a given use case, web page, whatever) have to decide which path you’re going to code in your query

                                Why is this a bad thing? Presumably you always fetch articles using the same ArticleResolver, this doesn’t take any extra effort to support on the backend. And the multiplicity of paths is the nature of graphs! I don’t see why it makes the front-end more complicated.

                                Authorization with GraphQL is quite non-trivial

                                The best advice I’ve seen for this is: don’t do authorization in GraphQL. You have some system which accepts and resolves GraphQL queries, it fetches data from other systems, those other systems should be doing the authorization. I haven’t had a chance to apply this advice in practice, I think that for some use-cases postgres row-level security would do wonders here, but it seems reasonable to me and it would solve most of your complaints. If your GraphQL resolve can’t even see author.email, then it’s no problem at all to include that link in your schema, it’ll only resolve if the user is allowed to see that email.

                                1. 3

                                  The best advice I’ve seen for this is: don’t do authorization in GraphQL. You have some system which accepts and resolves GraphQL queries, it fetches data from other systems, those other systems should be doing the authorization. I haven’t had a chance to apply this advice in practice, I think that for some use-cases postgres row-level security would do wonders here, but it seems reasonable to me and it would solve most of your complaints. If your GraphQL resolve can’t even see author.email, then it’s no problem at all to include that link in your schema, it’ll only resolve if the user is allowed to see that email.

                                  “just not resolve”: But how does that actually play out in practice?

                                  Your query which requested 40 fields in a hierarchical structure gets 34 values back – and just has to deal with it. Your frontend might have Typescript types corresponding to some models in your system. So then some places, which query under certain contexts, get 38 values back, other places get 34, and so on. So does your Typescript now just have to have everything marked with a ? suffix (not required, null is permissible)? Suppose you have an Angular component which takes an Article type prop, or gets it from NgRX, or wherever. Now, your component cannot just populate its HTML template with all known fields of the Article type, like author name, article title, date published, etc. Sometimes that data will just not be there, “because auth”. So will it have to have *ngIf all over the place? Or just have {{ someField }} interpolations peppered throughout that just render as a blank space – sometimes? Should you have a different Angular component for each context that this type appears?

                                  And this is all just talking about a simple single defined GraphQL type and its fields. Things get even more fun when you have 3+ levels of hierarchy, and arrays of results. What if the model hierarchy is A -> B -> C, and your currentUser has access to the queried A, and the queried C, but not the queried B? Or, because of lack of authorization, it only has access to a subset of the array of values of a given field. So, your frontend now displays a partial set of results – without writing any error in your error log. Or maybe an auth error on an array field makes it return no records at all. Or maybe a null instead of []. So Engineer Alice is expecting to see 10 records, just like she saw on Engineer Bob’s screen when they were pairing yesterday. But today, when she goes to work in her branch, she only sees 7 – and cannot easily see why. “Well, your devs are supposed to know your (whole) schema”, we might say. That’s ideal, but in a team beyond a certain size, that can’t be assured (and, beyond a certain size after that, shouldn’t be expected).

                                  “Okay, so don’t make auth problems silent, make them noisy” we might say. Yes, I agree. So how noisy should these be? Should a single auth problem, in one single node in a 7-level hierarchy of 82 fields in a GraphQL query cause the entire query to return non-200? Teams have to decide about that, and some teams won’t agree to that approach. For example, at $current_employer, our (non-unanimous) agreement at the moment is to write to error logs, dispatch to $third_party_error_service, but return null and return success (200) for the query. But, in so doing, we are still having some engineers come into the Slack channel(s) and reach out for help because they don’t understand why their GraphQL isn’t working any more, or why such and such page used to return whatever yesterday, but is returning something else today.

                                  I’m just trying to point out that GraphQL isn’t problem-free. There are warts and challenges that need to be acknowledged and addressed before success can be had.

                                  1. 6

                                    I put my comment out there knowing it was wrong and hoping someone would come along to tell me what I was wrong about, but I don’t think this particular criticism holds water.

                                    You’re pointing out that if the frontend tries to access data it may or not be authorized to see it… may or not see it. The ? is essential complexity that is no fault of GraphQL, REST will have the same problem.

                                    I think you’re arguing that:

                                    • In GraphQL all of your data fetches happen at once, so it’s not possible to 403 just the missing pieces, or feasible to 403 the whole request. We agree here!
                                    • It’s not feasible to expect every engineer to know how the schema works. I agree, and think that’s exactly why types were invented. If User.email is nullable then engineers using User should expect that there will sometimes not be an email. Going further, you might want to make some types self-documenting. If it’s expected they’ll often fail, due to lack of authorization, you might return a union type: ViewableUser | CloakedUser.
                                    • Silently not returning data you can’t see is a recipe for confusion. That’s sometimes true, I agree! However, say you’re writing a task manager and the query asks for a list of all tasks. There, it’s pretty intuitive that the query would only return the tasks you’re allowed to see. I would expect that in most situations there’s an intuitive solution, authorization is a concept that we’ve all spent a lot of time getting used to.

                                    There’s another technique you might use, which is to return the errors inline with the query. Return both the data and also the errors. If some field fails to resolve you might leave it null and also bundle an error in with the response explaining why the field is null.

                                    Which of these techniques you go for will depend on the situation, and probably there are some situations where none of them are satisfying, but I don’t think it’s reasonable to expect any technology to be problem-free. You’re holding GraphQL up to a very high standard here!

                                    1. 4

                                      Context: After dablling, I think that GraphQL is usually better than a REST API – and I want to find out if I am wrong. I am fine if it doesn’t solve all problems but it shouldn’t make things worse.

                                      Back to your comment: I think, it is normal that your data schema has to take authorization constraints into consideration. Agreed, that does not make it always easy.

                                      If you cannot always return data, you have to make it optional. Most likely, you don’t have to do this field by field but put them into their own sub object.

                                      This is also what you would need to do if you used a REST API with a schema. Or not?

                                      Just an idea: If you want to make authentication restrictions more obvious, maybe you can use union types instead of optional fields? Similar to Either/Try in functional languages or Result Rust, you could either provide the result or the error that prevented it. Or instead of an error, a restricted view… Then clients having trouble see the reason for the trouble directly in their result but the schema gets more complicated.

                                      1. 1

                                        If you cannot always return data, you have to make it optional. Most likely, you don’t have to do this field by field but put them into their own sub object.

                                        This is also what you would need to do if you used a REST API with a schema. Or not?

                                        Just an idea: If you want to make authentication restrictions more obvious, maybe you can use union types instead of optional fields? Similar to Either/Try in functional languages or Result Rust, you could either provide the result or the error that prevented it. Or instead of an error, a restricted view… Then clients having trouble see the reason for the trouble directly in their result but the schema gets more complicated.

                                        Frankly, I would have rathered that auth errors were noisy and fatal, making developers immediately aware of issues before things get to production. The counterargument from my team, though, was that they did not want to provide information to malicious actors about unauthorized things. So the decision was to make auth errors return 200s, and poke null holes in the payload fields, and/or snip subtrees out of the payload.

                                        Context: After dablling, I think that GraphQL is usually better than a REST API – and I want to find out if I am wrong. I am fine if it doesn’t solve all problems but it shouldn’t make things worse.

                                        Indeed, it’s much the same for me. I don’t have any big personal problem with GraphQL. I just try to assess things objectively, and avoid getting pulled along with hype waves without reason. I just haven’t had an overall positive experience yet with GraphQL. It’s been plus-and-minus.

                                        1. 1

                                          Thank you for your thoughtful replies :)

                                          The counterargument from my team, though, was that they did not want to provide information to malicious actors about unauthorized things.

                                          It’s is hard to judge without your schema but that sounds slightly weird. An attacker gets potentially more information by what fields are missing than a generic auth denied. But even if that’s not true, sounds like security by obscurity to me. [But yes, I am frequently wrong, and maybe there is a good reason for this in your case ;)]

                                          1. 1

                                            The counterargument from my team, though, was that they did not want to provide information to malicious actors about unauthorized things.

                                            It’s is hard to judge without your schema but that sounds slightly weird. An attacker gets potentially more information by what fields are missing than a generic auth denied. But even if that’s not true, sounds like security by obscurity to me. [But yes, I am frequently wrong, and maybe there is a good reason for this in your case ;)]

                                            Your point stands, but the nuance here is that it’s more like: If someone doesn’t have access to X, if you tell them it’s missing, they can’t infer existence by trial-and-erroring and seeing what gives error vs. what gives absence.

                                  2. 1

                                    Can you elaborate how you would solve the “graph” thing better with other API designs? E.g. in a REST design you would have the same potential path ways that you could use for querying? (Just in separate requests?)

                                    1. 2

                                      I don’t know if it’s better, but what I do is this: I actually am pretty far from a REST purist, insofar as I do not strictly adhere to a 1:1 mapping between the object hierarchy and the REST endpoints. Instead, I prefer steering towards having endpoints 1:1 with frontend pages (for GETs, at least). So, where there are multiple pathways, my endpoints give only the information needed for the page. For the previous example, a Reader needing Articles would only get an array of Articles back, and would not have any Folders or Authors come along for the ride in the returned payload. If the page needs to display, say, an Author name, then it would come in the [same] payload, too, except as an extra field right on the Articles in the payload.

                                      I like the simplicity and straightforwardness of “1:1 with frontend pages”, even though it causes criss-crossing and mixing of models/types in a given payload. I leave the entity relationship strictness to the model/DB level (has-many, belongs-to, etc.).

                                      1. 1

                                        Thank you for elaborating. I have used that pattern and it worked fairly well for me, too.

                                        You have to keep client/server in sync, though. I really like the flexibility of graphql here but I haven’t used it on a real life project.

                                  1. 2

                                    How would humor be different in this language? You can’t hide the punchline while you’re setting it up. Are there jokes that are just as funny read backwards as forwards?

                                    1. 2

                                      I guess it would be somewhat similar as looking at a single frame cartoon: https://www.google.com/search?q=single+frame+cartoon&tbm=isch

                                      I am sure humor would work differently in this language, but it might introduce alternative ways to express it?

                                      1. 2

                                        I’d also add that, while Sapir-Whorf is in general…well, um, provably wrong, to be blunt…there are certain things, especially around humour, that are language-dependent. Puns are only available in some languages, for example; German can’t really do them. Or they’re available spoken, but not written (for example, Japanese, which has many cognates, but uses a Chinese writing system in part to avoid ambiguity in written forms).

                                        In this case, I can imagine an almost reverse of the Japanese situation: some of these sentences would be recognizable as drawings, and could have a unique form of written-only pun depending on how that worked out.

                                        1. 4

                                          German can do puns. Maybe Germans can’t, though. ;)

                                          1. 2

                                            I speak German okay, not great, so I confess I’m forwarding what I was told. Can you give me a German pun? I love them in English.

                                            1. 5

                                              Germans can definitely do puns, and puns are in fact pretty common in German (though generally considered to be “lame” jokes). The German name is Kalauer. A classic one: “Es wird nie wieder ein Kalauer über meine Lippen kommen, und wenn du lauerst, bist du kahl wirst.” (Of course it doesn’t work in English, but the translation would be “Never again shall a pun cross my lips, even if you lurk till you turn bald”).

                                              1. 1

                                                Ooh, nice! Thanks! I’ll need to ask my buddy what he actually meant.

                                              2. 4

                                                Most famous German pun (at least among English speakers): https://genius.com/Rammstein-du-hast-lyrics

                                                The phrases ‘du hast’ and ‘du hast mich’ when spoken can mean either ‘you have’ or ‘you hate’ and ‘you have … me’ and ‘you hate me’ respectively. When written hate is spelled differently, i.e. hast -> hasst.

                                                In effect the song tricks the hearer into believing that the singer is accusing them of being hateful towards him. Only when the complete sentence is sung is it clear that the much tamer meaning ‘You asked me’ is meant the whole time.

                                                I am not a huge fan but its such a famous example I thought it worth bringing up.

                                                1. 1

                                                  Google for “Wortspiele”.

                                                  Wikipedia has a few: https://de.m.wikipedia.org/wiki/Wortspiel

                                            2. 1

                                              That’s a good comparison! I think Scalar families is in part what prompted this thought. You could make the glyph for “big” be comically big, like 10x bigger than the rest of the sentence.

                                              Maybe you could show irony by making a big glyph that says “small”, or vice versa. Like the trope of a big guy named “Tiny”.

                                            3. 1

                                              An ironic situation is ironic no matter which order you learn about it.

                                            1. 1

                                              Holy shit. What is the Pythonic way?

                                              1. 7

                                                They’re all different use cases. There isn’t “one way to do it” since there are multiple “it”s.

                                                1. 4

                                                  asyncio is an embarrassment and though it is the officially blessed library the truly Pythonic way is to ignore it and use trio

                                                  Here’s a post on some of the issues with asyncio: https://web.archive.org/web/20171206104600/https://veriny.tf/asyncio-a-dumpster-fire-of-bad-design/

                                                  And another one (by the eventual author of trio): https://vorpus.org/blog/some-thoughts-on-asynchronous-api-design-in-a-post-asyncawait-world/

                                                  I used to have some links to what made Trio great but seem to have lost them. The documentation is pretty good: https://trio.readthedocs.io/en/stable/design.html And in practice, I’ve spent unknown amounts of time debugging crazy asyncio issues because some task threw a StopIteration wen it shouldn’t have and caused some other coro to fail silently or whatever the problem of the day is. I’ve used Trio less but these kinds of problems just don’t seem to happen.

                                                  1. 4

                                                    I like Trio but the ecosystem around it seems in it’s infancy. I made a library for the Quart web framework that supports both the asyncio and Trio eventloop. This library communicates with Redis so I had to get an adapter for both.

                                                    For asyncio, aioredis is an easy pick, however for Trio it was hard to find something. I ended up copying some unmaintained toy project into my project as-is since I could not find a better alternative (which is working fine so far, but def. not an optimal situation). This effectively splits the community in half.

                                                    Using Trio for my professional work is a hard sell because of that reason, even though I reckon Trio is probably the better implementation.

                                                    1. 1

                                                      Yes, most libraries are written for asyncio, which is kind of a shame.

                                                      However, trio-asyncio lets you use them from inside your trio app: https://trio-asyncio.readthedocs.io/en/latest/

                                                      It’s kind of a nightmare to port a large existing app from asyncio to trio, and probably isn’t worth it, but I think the existence of trio-asyncio means there are few reasons to start a new asyncio app.

                                                    2. 1

                                                      asyncio is a worse-is-better API. It’s good enough to be productive despite its warts and that’s going to require any replacement to keep a substantial compatibility layer or else stay in relative obscurity as most other better projects suffer when a worse-is-better API or project becomes popular.

                                                      1. 6

                                                        asyncio is a worse-is-better API. It’s good enough to be productive despite its warts

                                                        I’m going to try not to sound too bitter, but asyncio is not even worse-is-better

                                                        It’s not simple, for either implementers or users. It’s not correct, given how it’s essentially impossible to cleanly handle cancellations and exceptions. And it’s not consistent, it’s a collection of layers that have slowly accreted, occasionally you have to drop down and real with the raw generators.

                                                        that’s going to require any replacement to keep a substantial compatibility layer or else stay in relative obscurity

                                                        I would have agreed with you a few years ago, the things you’re saying are generically correct, but now we have Trio. Trio is simpler both in implementation and also in programming model, it just makes sense.

                                                        It also has a lovely compatibility layer: https://trio-asyncio.readthedocs.io/en/latest/

                                                        1. 2

                                                          I’m going to try not to sound too bitter

                                                          It’s okay. I avoid asyncio Protocols and transports like the plague - they’re just too fragile for me to handle.

                                                          Thank you for the trio asyncio adapter. I wonder if I can integrate it into my sanic + uvloop + aioredis + asyncpg project

                                                    3. 1

                                                      Awaiting bare coroutines in the simplest procedural case, using asyncio.get_running_loop() to get the event loop reference and loop.create_task(coroutine) to make Tasks to schedule in parallel for fan-out operations. At a previous job, I actually plugged an ProcessPoolExecutor in the loop.run_in_executor as the default one and used it to do truly parallel fan-outs for a project that would index TomTom product data with an API key, determine the new releases and download in parallel the 7z archives for map shapes, unpack and repack into an S3 backed tar of shapefile tar gzs and index the offsets into a database so it was trivial to fetch exactly what country/state/city polygon on the fly for use in our data science and staged polygon updating. It was pretty nifty actually, considering that archive packing was cpu bound which necessitated process concurrency and my hack allowed me to develop single process first (using a ThreadPoolExecutor descendant class which just scheduled coroutines in parallel) and then go full multiprocess for the things that ate CPU.

                                                    1. 2

                                                      Building something like this has been on my todo someday list for years now. I’m excited to give it a try!

                                                      1. 5

                                                        My first attempt at recording myself coding is using ffmpeg -video_size 3840x2160 -framerate 2 -f x11grab -i :0.0 output.mp4 to record two frames per second of my native screen resolution. Any improvements or other ideas?

                                                        1. 7

                                                          I’ve found OBS Studio painless for desktop recordings, but it sounds like you’re set already.

                                                          1. 3

                                                            If most of your work happens in the terminal you might use script or asciinema. They result in smaller recording sizes.

                                                            1. 2

                                                              If it works for you, it’s probably not worth changing software.

                                                            1. 10

                                                              I struggle with the same predicament: I run my own email server and have seen emails received from my email server fluctuate over the years between the Inbox and Spam folder in my recipient’s email, irrespective of what rules I follow:

                                                              • SPF, DMARC, and other greater than zero length letter email-related acronyms
                                                              • asking people to mark my emails as “Not Spam”
                                                              • registering my IP and domain with Google Webmasters
                                                              • registering my IP and domain with DNSWL.org

                                                              I have gone to lengths trying to solve this problem but gave up a few years back. But let’s take this issue and internalize it within our community:

                                                              What would you do if you were tasked with handling the spam problem at GMail?

                                                              1. 3

                                                                What would you do if you were tasked with handling the spam problem at GMail?

                                                                I’ve been following this debate for a long time, and suffered myself from the problem (albeit it’s outlook.com that’s blocking me usually, not gmail, and outlook.com has quite some marketshare in Germany). Personally, to circumvent it, I pay for an SMTP relay. This means that my mail server directs all outgoing mail to that SMTP relay server, which then does the real delivery. Since many people (mostly businesses) use that SMTP relay server, it has a high volume and mail usually goes directly to INBOX.

                                                                I acknowledge this is a workaround. The other workaround is to pay Google and have Google’s staff fix the problem. But I think there’s a possibility for a real solution (apart from the best solution, which is to fix the SMTP protocol itself). What I see is that it’s almost always low-volume senders that are self-hosting who have the delivery problems. Based on that, I suggest the following:

                                                                People who self-host mail should form groups and create an SMTP relay server with a clean IP reputation. Every member of that group then has his/her mail server direct all the outgoing mail to the group’s SMTP relay, which does the real delivery. As a consequence, the group SMTP relay has the cumulated volume of all group members and has a much better stand in the delivery. The bigger the group, the better.

                                                                It’s required that someone administers that mail relay, most notably to prevent any member from sending spam (e.g. compromised account). Maybe try rotating the administrative duty annually or so. Finally, from a German perspective, an association (Verein) would make a candidate for the legal framework of such a collective SMTP relay. It’d be the legal entity to address in case of problems. It’d also allow some kind of charity “business” model: assocation members may use the SMTP relay for free, but non-members have to pay. In any case, only individuals are accepted as members. German Ehrenamt at work.

                                                                The individual’s fight against GMail and similar companies is not winnable. My suggestion can give the self-hosters a better stand, and still allow real self-hosting: it’s your SMTP server that accepts all the incoming mail, does the spam filtering, the sieve scripts, and whatever you wish. It’s also your server that sends out the outgoing mail, with just one little twist: all outgoing mail is submitted to the group’s SMTP relay server, thereby joining the group SMTP server’s volume and reputation.

                                                                1. 1

                                                                  People who self-host mail should form groups and create an SMTP relay server with a clean IP reputation. Every member of that group then has his/her mail server direct all the outgoing mail to the group’s SMTP relay, which does the real delivery. As a consequence, the group SMTP relay has the cumulated volume of all group members and has a much better stand in the delivery. The bigger the group, the better.

                                                                  Sorry, but this is just really silly. I don’t want someone else to be running my mail server. If you do, you can already use one of the vapourware products for sending bulk email or whatnot (be careful, though, because a lot of it doesn’t actually implement SMTP correctly, and cannot do greylisting, for example, so, you’d probably only want to use it to send mail to Gmail and Outlook, still using your own IP address to send mail to anyone else).

                                                                  OTOH, I certainly wouldn’t mind to contribute to a legal fund to sue Google and Google Mail, due to them having a monopoly position on the corporate email front, and refusing to accept my mail under false pretences of low domain reputation, which was acquired merely because I’m myself a Gmail user, and have been sending mail to my own account that they misclassified as Spam, and extended such misclassification to the rest of their userbase, too.

                                                                2. 1

                                                                  What would you do if you were tasked with handling the spam problem at GMail?

                                                                  Maybe this, right?

                                                                  I mean, the current situation is kind of the worst of both worlds. Not only do we have an ossified protocol nobody can change, but it isn’t even giving us interoperability!

                                                                  However, I can only assume there’s been a slowly-escalating cat-and-mouse game, and as more and more people use email powered by one of the big providers it’s at least plausible to me that more spam than legitimate mail would be coming from self-hosted mail. Without working there I’m not sure how I could estimate what kind of spam they’re getting, and without knowing what kind of spam they’re getting I don’t know that they’re actually doing the wrong thing. Maybe the proper application of bayes rule really should lead you to conclude that self-hosted mail is probably spam.

                                                                  I would hate it if that were true, but I don’t know how I would be able to tell if it wasn’t true.

                                                                  1. 1

                                                                    The biggest problem is that the whole Google Mail thing is a black box, that there’s no way to know why exactly your mail is being rejected (e.g., it tells me that my domain has a low reputation, but doesn’t tell me why), and that there’s no recourse on fixing the situation (e.g., there’s no way for appealing the misclassification of those cron emails that I sent from my own domain to my own Gmail account that Google, apparently, deems to contribute to the low reputation).

                                                                    Google Search already has a solution for picking up new web properties very fast, whereas the rest of the players (e.g., Yandex) take ages to move a new site from a Spam to a non-Spam status and to include it in their main index — there’s little reason this couldn’t be done for email. The fact that it’s the very small players that are most affected leads me to believe that it’s merely a CoI at play — they don’t want you or me to run our own mail servers — they want all of us to simply use Gmail and G Suite instead.

                                                                1. 6

                                                                  If powershell seems interesting but you’re put off it nushell is a less-mature implementation of some of the core concepts, it looks promising.

                                                                  If I remember correctly Oil shell is another attempt to make shells easier to use, it’s not as obviously inspired by powershell, but includes the ability to pass around structured data (among many other improvements).

                                                                  1. 45

                                                                    Things like this are why I don’t trust Martin’s opinions. He didn’t say a single bad thing about Clojure, he didn’t have any nuance, he doesn’t respect the other viewpoints. He does the same thing with TDD and clean code, where it’s impossible for them to be the wrong tool. He’s been programming for five decades and still thinks in silver bullets.

                                                                    For the record, his Clojure example is even shorter in J. It’s just *: i. 25.

                                                                    1. 10

                                                                      His example is shorter even in Clojure:

                                                                      (map sqr (range 25))

                                                                      but that misses the point. Both Clojure examples are easy to explain and understand, where in J it is not obvious what *: and . stand for, and how these should be changed should we wanted to compute something different. But even that is not the point.

                                                                      The point is that Uncle Bob is writing about his own experience with the language that he finds fascinating. He writes about his experience and he gets to choose how he writes it. If anyone disagrees (plenty people do, I suppose) they are very well entitled to write about their experience themselves.

                                                                      1. 19

                                                                        I don’t want to sound like an asshole, but what exactly is his experience besides teaching and writing books ? Cause we see so many people advocating for specific language/technology without any substantial real world experience.

                                                                        1. 2

                                                                          As professional advocates go, he’s well known and (at least be me) well regarded.

                                                                          A professional advocate advocating for something is a signal too… and a lot of the things he was advocating 25 years ago are still relevant today.

                                                                          http://web.archive.org/web/20000310234010/http://objectmentor.com/base.asp?id=42

                                                                          1. 10

                                                                            A professional advocate advocating for something is a signal too

                                                                            Yes, it’s called Appeal to Authority.

                                                                            I’m also not convinced he’s much of an authority. I’d say he’s a zealot. His tirades against types are tired. His odes to discipline are masturbatory. His analogies… well… This is the same guy who said C++ is a “man’s language” and that you need big balls to write it.

                                                                            1. 4

                                                                              His analogies… well… This is the same guy who said C++ is a “man’s language” and that you need big balls to write it.

                                                                              This is called an ad hominem. If you’re going to be a stickler about logical fallacies I’m surprised that you can’t even make it a few sentences without contradicting yourself. Are they important or not?

                                                                              A professional advocate advocating for something is a signal too

                                                                              This is called inductive reasoning. Given some evidence, such as a well-regarded professional advocating for some tool, we can try to generalize that evidence, and decide the tool has a good chance of being useful. You’ve surely heard of Bayesian probability; signals exist and they’re noisy and often incorrect but minding them is necessary if you want to make any sense of the world around you.

                                                                              Yes, it’s called Appeal to Authority.

                                                                              Logical fallacies only really apply when you’re working in the world of what’s called deductive reasoning. Starting from some premises which are assumed to be true, and moving forward using only techniques which are known to be sound, we can reach conclusions which are definitely true (again, assuming the premises). In this context, the one of deductive reasoning, appeal to authority is distinctly unsound and yet quit common, so it’s been given a nice name and we try to avoid it.

                                                                              Tying it all together, the parent is saying something like “here’s some evidence”, and you’re interjecting with “evidence isn’t proof”. Great, everybody already knew that wasn’t proof, all that we’ve really learned from your comment is that you’re kind of rude.

                                                                              1. 4

                                                                                Fallacies can apply to inductive arguments too, but you are right in that there’s an important distinction between the two types and how they differ. I would say that the comment you’re replying to is referring to the idea of informal fallacies in the more non-academic context. The Stanford encyclopedia has a good in-depth page about the term.

                                                                                Also, not all fallacies are equal, appeal to authority may be seen as worse than ad hominem these days.

                                                                                1. 5

                                                                                  This thread started with, “Things like this are why I don’t trust Martin’s opinions.” Uncle Bob’s star power (or noteriety) and whether that qualifies as social proof or condemnation, is the point of the discussion, not a distraction.

                                                                        2. 11

                                                                          The point is that Uncle Bob is writing about his own experience with the language that he finds fascinating. He writes about his experience and he gets to choose how he writes it.

                                                                          I wouldn’t be complaining if he was just sharing a language he liked. The problem is he’s pushing clojure as the best language for (almost) everything. Every language has tradeoffs. We need to know those to make an informed decision. Not only is he not telling us the tradeoffs, he’s saying there aren’t any! He’s either naïve or disingenuous, so why should we trust his pitch?

                                                                          1. 1

                                                                            The problem is he’s pushing clojure as the best language for (almost) everything.

                                                                            That’s not what he said though. The closest he came to that is:

                                                                            Building large systems is Clojure is just simpler and easier than in any other language I’ve used.

                                                                            Note the qualification: ‘… than any other language I’ve used’. This implies there may well be languages which are easier for building large systems. He just hasn’t used them.

                                                                            Not only is he not telling us the tradeoffs, he’s saying there aren’t any!

                                                                            He repeated, three times for emphasis, that it doesn’t have static typing. And that it doesn’t give you C-level performance.

                                                                            1. 7

                                                                              Note the qualification: ‘… than any other language I’ve used’. This implies there may well be languages which are easier for building large systems. He just hasn’t used them.

                                                                              We need to consider the connotations and broader context here. He frames the post with

                                                                              I’ve programmed systems in many different languages; from assembler to Java. I’ve written programs in binary machine language. I’ve written applications in Fortran, COBOL, PL/1, C, Pascal, C++, Java, Lua, Smalltalk, Logo, and dozens of other languages. […] Over the last 5 decades, I’ve used a LOT of different languages.

                                                                              He doesn’t directly say it, but he’s really strongly implying that he’s seen enough languages to make a universal judgement. So “than anything other language I used” has to be seen in that context.

                                                                              Nor does he allow special cases. Things like

                                                                              But what about Javascript? ClojureScript compiles right down to Javascript and runs in the browser just fine.

                                                                              Strongly connotating that “I’m writing frontend code for the web” is not a good enough reason to use Clojure, and he brushes off the lack of “C-level performance” with

                                                                              But isn’t it slow? … 99.9% of the software we write nowadays has no need of nanosecond performance.

                                                                              If Clojure is not the best choice for only 0.1% of software, or even 5% of software, that’s pretty darn close to “best language for (almost) everything.”

                                                                              He repeated, three times for emphasis, that it doesn’t have static typing.

                                                                              He repeats it as if the reader is hung up on that objection, and not listening to him in dismissing it. Note the increasing number of exclamations he uses each time. And he ends with

                                                                              OK, I get it. You like static typing. Fine. You use a nice statically typed language, and I’ll use Clojure. And I’ll be in Scotland before ye.

                                                                              Combined with his other posts (see “The Dark Path”), he doesn’t see static typing as a drawback. We can infer it as a drawback, but he thinks we’d be totally wrong in doing so.

                                                                          2. 3

                                                                            You have to explain both examples for them to make sense. What does map do? How do you change sqr out for a different function? If you learn the purpose of the snippet, or the semantics of each of the individual elements, you can understand either the J or Clojure example just as well as the other (if your understanding of both languages is equal).

                                                                            Also the meat of the article is trying to convince the reader to use Clojure (by explaining the syntax and semantics, comparing its syntax to two of the big 5 languages, and rebutting a bunch of strawman arguments - nothing particularly in-depth). I don’t see a balance of pros and cons that would be in a true account of an experience learning and using the language, including more than just a bullet point on the ecosystem, tooling, optimisation, community, etc.

                                                                            1. 3

                                                                              I am sure that any programmer that has any experience in any language would guess that you change sqr out for a different function by typing the name of that other function. For example, you compute exp instead of sqr by, well, typing “exp” instead of “sqr”.

                                                                              The same with map. Of course that someone has to know what particular function does to be able to use it effectively. The thing with Clojure (and other Lisps) is that it is enough to know that. You don’t need special case syntax rules. Any expression that has pretty much complex semantics is easy to write following a few basic rules.

                                                                              1. 5

                                                                                I understand the benefits of the uniformity of Lisp, but my point was just that you can’t really say that (map sqr (range 25)) is any more or less understandable than *: i. 25 if you know the purpose of the expressions and the semantics of their constituent parts. And given that knowledge, you can reasonably make substitutions like exp for sqr or ^: for *: (though I would end up consulting a manual for the exact spelling).

                                                                                Further experimentation would require more knowledge of either language. For instance, why if isn’t a function in Clojure, or why lists don’t have delimiters in J. It’s all apples and oranges at this superficial level.

                                                                            2. 1

                                                                              My version of Clojure doesn’t define sqr—is that built in?

                                                                              That aside, I don’t find either version very easy to explain to someone who isn’t already experienced with functional programming. What does “map” mean? How does it make sense that it takes a function as an argument? These seem obvious once you’ve internalized them, but aren’t easy to understand from scratch at all.

                                                                              If I were reviewing this code, I would suggest they write (for [x (range 25)] (* x x))

                                                                              1. 4

                                                                                Of course that one has to understand the semantics of what they’re doing. But, in Clojure, and Lisps it is enough to understand the semantics, while in most other languages, one has to additionally master many syntax rules for special cases.

                                                                                1. 5

                                                                                  Closure has quite a lot of special syntax compared to many Lisps. for example, data type literals and other reader macros like literal lambdas, def forms, let forms, if forms and other syntax macros like -> are all built in. Each of these has their own special rules for syntax and semantics.

                                                                                  1. 2

                                                                                    We’re on the same page I think, except that I think knowledge of semantics should be enough to understand any language. If you see a verb and a noun in close proximity, you’d be able to make a good guess as to what’s happening regardless of the glyphs representing their relationship on screen.

                                                                                    1. 5

                                                                                      If you want a language that emphases semantics over syntax, then APL is the language for you! There are just a few things to understand about syntax, in order of importance.

                                                                                      1. Array literals
                                                                                        • Numeric arrays are just numbers separated by spaces. Negative numbers are prefixed with ¯. Some dialects have special-case syntax for complex or rational numbers: 42 3.14 1J¯4
                                                                                        • Character arrays are just text delimited by '' quotes. Doubling the quote inside an array escapes it: 'is' or 'isn''t'
                                                                                      2. Array indexing with [] braces: 'cafe'[3 2 1 4] ←→ 'face' (Many APLers have a disdain for this form because it has some inconsistency with the rest of the language.)
                                                                                      3. Function definitions
                                                                                        • Inline anonymous “direct” functions delimited by {} braces.
                                                                                        • Traditional named functions defined by the ∇ form.
                                                                                      4. Statement sequencing with (Mainly useful for jamming more code into a single line)

                                                                                      From there, the grammatical rules are simple and natural in the form of verb noun or noun verb noun or verb adverb noun etc. Probably the most difficult thing to learn and remember is that there is no operator precedence and evaluation reduces from right-to-left.

                                                                                      When I’m programming in APL, I rarely think about the syntax. When I’m programming in Clojure, syntax is often a concern. Should I use map or for? Should I nest these function calls or use ->?

                                                                                      1. 2

                                                                                        When I’m programming in Clojure, syntax is often a concern. Should I use map or for? Should I nest these function calls or use ->?

                                                                                        None of those are syntax. map is a function and the rest are macros. They’re all inside the existing Clojure syntax.

                                                                                        1. 1

                                                                                          Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages. [my emphasis]

                                                                                          https://clojure.org/reference/macros

                                                                                          1. 2

                                                                                            True enough. However, at least in Clojure, macros are pretty deliberately limited so as not to allow drastically changing the look-and-feel of the language. So I’m pretty sure every macro you’ll come across (except I guess reader macros) will have the same base syntax, (a b ...).

                                                                            1. 2

                                                                              I struggled to find the right tags for this. There’s no “prolog’ or “logic”. “math” is pretty close, the link does get into some interesting math! But I figure the kinds of people who follow the “math” tag might disagree with me :) “database” is also nearly correct, datalog is intimately related to relational algebra, but again this isn’t the content someone following the “database” tag would expect to see.

                                                                              1. 3

                                                                                Datalog is a strange beast (it’s a mixture of logic, database and classical AI search strategies) so I don’t think we have the right tag.

                                                                                But I agree with you we need a prolog/logic tag!

                                                                              1. 3

                                                                                I wonder if this could have been patched by changing the programming model to be something more declarative. It’s not the CSS which is insecure, the JS which comes afterwards messes everything up!

                                                                                A big change might be to put the DOM behind a curtain. The page can emit a DOM (which might include minimal support for scripting, definitely no network access) but can never read that DOM back out. In response to events it can emit a new DOM which is diffed and applied by the browser itself.

                                                                                I think this would also allow relaxing the same-origin restriction? You still don’t want to allow buttons which hit unrelated domains and send the author money on venmo, but what’s the problem in letting a page pull in the list of my venmo payments, as long as the JS for that page can’t see it?

                                                                                It’s a shame there’s no way to AJAX without javascript, surely there’s a purely-declarative model which could have been used to pull data from external sources.

                                                                                1. 5

                                                                                  CSS is also insecure here.

                                                                                  By changing the size of a link inside a fixed-size container, I can alter how much space is left for an <img srcset and then record which file gets downloaded.

                                                                                  1. 1

                                                                                    Well damn, good point. Let’s just burn it all down.

                                                                                    This is insecure because all of the following are true:

                                                                                    • Your browser has a secret which shouldn’t be made public
                                                                                    • Your browser uses that secret when rendering a page
                                                                                    • After that secret has entered the equation your browser fetches more resources
                                                                                    • Those fetches can be used to infer the secret

                                                                                    It sounds like breaking any of the above would fix the problem, but I’m not sure where the break belongs:

                                                                                    • You probably want your browser to remember your history
                                                                                    • You probably want to style :visited links differently
                                                                                    • It’s hard to imagine an alternative to CSS which first made requests and only then injected secrets.
                                                                                    • How can you request resources without letting the world know what you’re requesting?

                                                                                    Maybe your browser should fetch every alternative? Or, requesting all content by hash (like ipfs) would mean the server which gets the request and can infer your secret is unlikely to be the server trying to track you?

                                                                                  2. 2

                                                                                    Putting sensitive data and untrusted code in a sandbox and hoping nothing leaks is a difficult problem.

                                                                                    1. 1

                                                                                      The idea is more, let’s not put untrusted code in there!

                                                                                      And since removing JS entirely is a lost cause, maybe we can get most of the benefit by adding a boundary beyond which it can’t reach.

                                                                                  1. 3

                                                                                    I came to this article wanting to agree but found its arguments dissatisfying.

                                                                                    For one, it makes the baseless claim that Google “seek[s] to neutralize ad-blocking”. Google wanted to speed up page loads and proposed a number of changes to the way extensions work, one of which was incredibly reasonable (it would in fact speed up page loads!) but incidentally made the lives of ad-blocking extension authors more difficult. Google walked back the change to this unreleased api once they were made aware of their mistake. This is exactly how things should work! The article later implies that the Chrome team doesn’t care about user experience… but this case is an example of how they do care! Google is still, even after reaching a quasi-monopoly position, working very hard on making your browsing faster. Where do you think QUIC and SPDY came from?

                                                                                    It also claims that Chrome tracks your browsing history. So does… every other browser? Chrome does not upload your history and give it to Google unless you have that feature enabled.

                                                                                    There’s no mention of Google’s pretty cool usage of differential-privacy: https://www.chromium.org/developers/design-documents/rappor i.e. your browsing history and a lot of other metrics are uploaded but in a form which can’t be used to learn much about you. RAPPOR is not the invention of a company which rapaciously takes anything you have to give it.

                                                                                    Brushing all that aside, I wonder if the real problem here is a governance issue. A monoculture doesn’t sound so bad if the single implementation is controlled by the users and definitely working in our interests. It is frustrating that Google doesn’t seem totally aligned. Faster page loads mean more page views mean more ad impressions so you can trust them to keep pushing that post forward but a short-sighted Google would prefer to slurp up your entire browsing history.

                                                                                    An implementation controlled by the users sounds kind of like Firefox. But Mozilla can’t do as well as Google can, for whatever reason. Servo took how many years and still isn’t done? I don’t mean to be flippant, they’re working on a very hard problem and it would take me a while too. Mozilla does seem to have lost fair and square though.

                                                                                    Maybe the problem is that users don’t know what they want. We really wanted all the old Firefox extensions to stop working, because that gives Mozilla some extra agility to add support for things like multiprocessing. Users said they wanted the add-ons to keep working though, and that really slowed things down.

                                                                                    Or maybe the problem is that Google has an infinity of resources, not just money but also brilliant engineers, and we’ve made a faustian bargain where they make a great browser for us in exchange for our needing to monitoring how well they respect our privacy.

                                                                                    Maybe the problem is that we’re thinking about browsers all wrong. Do they have to be so integrated? What if users could pick their rendering engine?

                                                                                    Maybe there’s no problem at all. Google does seem to mostly respect your privacy. They are plenty of Chromium-based browsers you can use if you don’t like Chrome.

                                                                                    1. 4

                                                                                      Maybe the problem is that users don’t know what they want. We really wanted all the old Firefox extensions to stop working, because that gives Mozilla some extra agility to add support for things like multiprocessing. Users said they wanted the add-ons to keep working though, and that really slowed things down.

                                                                                      Users legitimately stopped using Firefox because of broken extensions. They also stopped using Firefox because Chrome was performing better. The only case I know of here where users’ revealed preference disagrees with their stated preference is security: users continue to use browsers that regularly have CVE’s, and they usually advise users to not visit web pages that they don’t trust (which is probably part of how we ended up with the Facebook monoculture…) If Firefox had wanted to keep users above all else, they could’ve just allowed extensions to continue to do whatever they want even after the browser went multiprocess; the only reason why they didn’t do that is because it would’ve allowed extensions to have data races in them.

                                                                                      1. 1

                                                                                        users continue to use browsers that regularly have CVE’s, and they usually advise users to not visit web pages that they don’t trust

                                                                                        I’ll add there was a significant chunk of people using IE and Firefox on Windows with sandboxing software that neutralized many CVE’s. They sometimes had combos of them like DefenseWall, SandboxIE (not just IE), maybe NoScript/Flashblock/Adblock for Firefox, and/or some security suite. Then, there were people running browser VM’s for extra-untrustworthy stuff or LiveCD’s for online banking (esp with Brian Krebs advice).

                                                                                        I was one of the ultra-sandboxed people on Windows at one point. It was security by malware economics where the kit developers weren’t targeting a setup like that given they were casting a wide net to maximize the number of Windows machines in botnets with minimal effort. So, it was fairly safe outside targeted attacks. Some of us were even training privacy-conscious laypeople to use NoScript with rules of thumb for what to allow or disallow. Many picked it up without knowing tech. I’ve been on Linux for a while now, though. I can’t tell you current state of things or even how to configure a Windows box. My vague memories stop with Windows 7: the best Windows. :)

                                                                                        Edit: To confirm on switching, I did have to ditch Firefox for Chrome on a bartered backup I used (Presario w/ Celeron). The newer Firefox’s were running at a crawl speed that didn’t match the pace I needed for research. The Core Duo 2 I had performed fine but Celeron with Firefox was apparently the bound. Chrome ran fast on it. Firefox actually got slower on that machine with the Quantum updates that were supposed to make it faster. That was around when I switched off it.

                                                                                    1. 17

                                                                                      Somebody needs to solve the mismatch between the value generated by free software and the inability to get paid. Programmers shouldn’t have to take a huge pay cut to work on libre software.

                                                                                      Having to ask for ‘donation’ is an insult to the dignity of a competent programmer who can otherwise get a very lucrative offer for his/her skills.

                                                                                      1. 11

                                                                                        I honestly think to a large degree this has been solved if we follow the example of SQLite. Rather than trying to reach out all all possibly users of SQLite, trying to get a monthly donation of like $1/$2/$5 from each user, they focus on the corporate users of the software and ask for significant investment for things that corporate users specifically care about and aren’t “donations”:

                                                                                        • Email Support: $1500 a year.
                                                                                        • Phone Support: $8000 a year base, for SLA response time goes up to $50,000.
                                                                                        • Consortium Membership: $85,000+
                                                                                        • Title/License: $6000
                                                                                        • AES Custom Version: $2000
                                                                                        • CEROD Custom Version: $2000
                                                                                        • ZIPVFS Custom Version: $4000
                                                                                        1. 3

                                                                                          Note that this doesn’t work if the software isn’t going to be used by companies. For instance I have a hard time picturing a company pay for sway or aerc.

                                                                                          1. 1

                                                                                            Absolutely, stuff that is of no use to a corporation is harder to deal with this way. I would argue that at certain levels of corporate dependency, even niche products like text editors and diff tools can get widespread financial backing. I have seen both (text editors and diff tools) get major contributions in terms of cash from corporations.

                                                                                        2. 9

                                                                                          Donations are difficult to justify by companies both legally and in terms of business. They also cannot guarantee any continuity to the recipient. Moreover, donations are inherently unfair to donors VS non-donors.

                                                                                          Public funding has been invented exactly for this.

                                                                                          1. 2

                                                                                            Moreover, donations are inherently unfair to donors VS non-donors.

                                                                                            Could you elaborate “fair” a little?

                                                                                            I cannot settle on a definition of fairness around donations (esp. to develop open source software) that I, myself, would use in this situation, and so I would surely fail at assuming the definition intended in your comment.

                                                                                            1. 4

                                                                                              Forgive me for the platitude: If a company donates a lot and a competitor does not (while still benefiting from the same shared public good), the latter has an advantage. This little prisoner dilemma around donations encourage greed over cooperation. That’s why taxes are mandatory.

                                                                                              1. 2

                                                                                                If a company donates a lot and a competitor does not (while still benefiting from the same shared public good), the latter has an advantage.

                                                                                                That sounds right but might not be. IBM backed Linux when Microsoft and SCO were going to patent sue everyone trying to use it. IBM both was donating labor to Linux and would counter-sue to keep it open. The result over time was IBM got massive benefit from Linux while proprietary offerings folded or went into legacy mode over time.

                                                                                                I mean, IBM and Linux are both kind of unique. That success might not extrapolate. It might for companies who can stand to gain a lot from a community around their product. The community might be bigger if the company significantly invests in open source and community activities.

                                                                                              2. 3

                                                                                                I assume the rational is that open source code is a public good in the same way that clean water or science is. If you spend a lot of money making sure that your local river has clean water, or a lot of money to study physics then the benefits are shared by everybody but the costs were incurred by just you.

                                                                                                “Fairness” in the context of shared resources generally means that the costs of providing the resource are shared by the users of the resource in proportion to the benefit those users each receive.

                                                                                              3. 2

                                                                                                I agree that public funding was meant to solve problems much like this, but that doesn’t make it an easy solution.

                                                                                                There are thousands of new libraries created every day, which ones will you fund? How much money should you give Pixie Lang?

                                                                                                The NSF gives grants to researchers who are motivated to publish papers, which journals will only accept if the papers reveal something new and important. If you give money to open source developers do they have any external motivation to produce useful things? What is preventing them from adding a million new features to OpenSSL rather than carefully auditing the code and fixing tricky bugs?

                                                                                                If ruby is given enough public funding to hire 10 developers, won’t that make the developers who weren’t chosen feel like they’re not as important? Would they continue contributing as much as they have when they know somebody else is getting paid?

                                                                                                Many open source projects have contributors from many different nations. Is the agency doing public funding okay with giving money to non-nationals?

                                                                                                1. 2

                                                                                                  public funding was meant to solve problems much like this, but that doesn’t make it an easy solution

                                                                                                  It worked better than other alternatives during the last 100 years to develop phones, early computers, semiconductors, satellites, a lot of medicine, aeronautics, chemistry… Anything that does not have a short or medium-term return.

                                                                                                  Is the agency doing public funding okay with giving money to non-nationals?

                                                                                                  A lot of long-term scientific research is funded through global nongovernmental organizations.

                                                                                              4. 6

                                                                                                Not a great comfort to a libertarian, I’m sure - but for those who believe in government intervention, taxpayer-funded work on core infrastructure is an obvious way to share the load (since broadly speaking, society at large benefits from access to improved technology).

                                                                                                IIRC at least one of the OpenSSL core team was funded - for years - off the German social security pension. RMS’s work at MIT was largely funded through the US government. DARPA paid for a ton of computing infrastructure.

                                                                                                1. 4

                                                                                                  Who is this somebody who needs that?

                                                                                                  Describing your own desires as someone else’s needs is a cop-out.

                                                                                                  1. 1

                                                                                                    I discuss this a lot. It usually breaks the moment I bring in the notion that this somebody should probably be paid, at around, say, 10-20% if what the developers get.

                                                                                                  2. 1

                                                                                                    If the software is valuable, you can license it such that you can sell it for money.

                                                                                                    1. 6

                                                                                                      This is a pretty often mentioned, but not every FOSS software has a straight forward business model attached. For example, programming languages are far too remote from an actual product for people to actually invest in them on large scale. Yet, they certainly have huge value! If you see the struggle to get a widely used project as MRI funded…

                                                                                                      Sure, I could get my money by consulting in that programming language and being an expert in it, but there, the incentive is actually again to have other people developing it and just run around using their stuff.

                                                                                                      Also, not every programmer wants to become a salesperson or build end-user products.

                                                                                                      1. 3

                                                                                                        You can also license it freely and sell it for money. There’s no inherent contradiction in “commercial free software”. Indeed, sr.ht seems like it fits this category.

                                                                                                        1. 1

                                                                                                          Great example (and congrats again) :)

                                                                                                          In my experience, most such software is very hard to deploy for yourself (since the primary maintainer has no real reason to work on that aspect and nobody else tends to step up).

                                                                                                          This is in no way a jab at your fantastic work - merely an observation of how this, like every funding structure, exerts a pull on the project funded.

                                                                                                          1. 1

                                                                                                            Congrats? For what? I’m not Drew.

                                                                                                            1. 1

                                                                                                              Huh, somehow I got my wires crossed, sorry.

                                                                                                        2. 1

                                                                                                          I wonder if that’s true, and if not, why.

                                                                                                          You’ve done it. And perhaps I have too (although one might tell my own story in different ways). But the people who manage to create functioning open source software from scratch have failed to earn real money from it with such frequency that I wonder whether there’s some blocker. That some personality trait that’s necessary to create that software also blocks generating income from the software.

                                                                                                          1. 1

                                                                                                            I absolutely believe this is the case, personality traits that draw people to open source software tend to push them away from the obvious avenues of income. I think they also fear angering their communities if they start to value corporate users over regular users. I think this fear is misguided, if regular users get much a much better product because of that corporate support, I believe they will be very understanding / support (ala sqlite).

                                                                                                            1. 1

                                                                                                              That some personality trait that’s necessary to create that software also blocks generating income from the software.

                                                                                                              I don’t believe this is the case. FOSS comes out of a culture where many people could make their ends meet. Either by being employed by MIT or by having a paid day job.

                                                                                                              It’s something our community - as a whole - could easily ignore and not build structure around. That’s falling on our feet now. Those structures will take years to build.

                                                                                                        1. 4

                                                                                                          Finished this book yesterday, it’s pretty good! However, I feel Dependent Types weren’t a good fit for this series’ format. This series never talks about theory except obliquely. For example, page 175 talks about the Curry-Howard correspondence and the rest of the book elaborates on it by writing proofs of various things but it’s never mentioned by name. A lot of other interesting facts were mentioned but I have no way of looking them up! This is a fine format if you’re learning Scheme, there’s not much theory there, but I feel like dependent types should have been treated differently. I’ve now been shown some examples of what you can do with dependent types but I’m still very unclear as to what they actually are, the book never explained!

                                                                                                          1. 1

                                                                                                            Something I don’t understand is that this isn’t how capitalism is supposed to work. If there exist enough women who are willing to pay for pants with larger pockets then they’ll be made, won’t they?

                                                                                                            I’ve also heard many of my female friends complain about their tiny pockets, maybe I’m in a bubble and most women don’t mind? Maybe my friends keep trying pants with larger pockets and not buying then because they don’t look stylish enough? Maybe there’s a global conspiracy? I’m having a hard time deciding which of these options is most likely to be true.

                                                                                                            1. 7

                                                                                                              It’s probably a disconnect between what people complain about and what they buy. I heard fake pockets make pants look slimmer so when a designer adds large pockets they sell less of that design even though thats what people say they want.

                                                                                                              1. 1

                                                                                                                Presumably the women in question care more about looking good than carrying things.

                                                                                                                The women I know who have different priorities wear different clothes–as do the men I know.

                                                                                                                1. 2

                                                                                                                  Then thats just the market working as normal. There will be no shift to more useful clothing if people don’t buy that even if they complain to others after buying it.

                                                                                                                  1. 2

                                                                                                                    I speculate that people who care about the appearance (rather than functionality) of clothes buy more clothes.

                                                                                                                    If that effect exists, how strong is it? Do 10% of people buy 90% of clothes?

                                                                                                                    I heard it is like that for booze: 10% buy 90%. And it ain’t top shelf…

                                                                                                              2. 2

                                                                                                                I found this article really interesting since I had no idea the problem existed. I think maybe the suppliers dont either. The stuff they make that’s slimmer or lighter in pockets sells the most. When Im in Walmart or Target, the heavier stuff isn’t what people are trying on. They might have noticed this eventually modifying supply practices to sell attributes majority were buying.

                                                                                                                Another thing thing comes to mind is that each piece of clothing has multiple attributes. Women probably buy something that looks and feels nice despite smaller pockets they don’t like. However, a simplistic analysis of purchasing data might count those sales as a yes to all three. They mighy think small pockets are in demand even though they didnt matter. Then, start putting them on more clothing.

                                                                                                                In any case, there’s certainly an unserved need in the market. Smaller suppliers with a proven design/style should consider making a version of each with bigger pockets. A big uptick in sales of those might cause larger suppliers to follow suit.

                                                                                                              1. 11

                                                                                                                Recall what happened: we decided to represent characters of text, including punctuation and whitespace, using fixed numbers (a practical, though rather dubious, decision; again, a product of early computing)

                                                                                                                Around here is where he lost me, how else are we supposed to do it? As he admits, computers just manipulate bits, if you want to represent anything else you’ll need a mapping from those bits to your character set.

                                                                                                                1. 2

                                                                                                                  There’s no term for it, but the alternative is structured data: a “binary” AST with data at the leafs, and if that data is text it has to be with explicit mentioning of the character table.

                                                                                                                  1. 1

                                                                                                                    Everything just becomes a single node containing a blob of text again.

                                                                                                                    Messaging actually shows promise, as does networking, but I think the former is more likely than the latter (despite much more energy being put into the latter by researchers)

                                                                                                                    1. 2

                                                                                                                      Why would everything become a single node with a blob of text? This is evidently true when it would have to work with systems that were primarily made to deal with unstructured text files, but that’s the circular argument addressed in the article.

                                                                                                                      Messaging and networking are different solutions for problems not mentioned in the article. Not sure how it would help with the “build and destroy the world” issue.

                                                                                                                      1. 6

                                                                                                                        We get blobs of text because of serialisation, something we need to do to stream out to spinning platters made of rust, or to beam waves over a wire.

                                                                                                                        We often prefer to choose the serialisation format because it’s always faster than a general purpose serialisation format one might try for baking down that “AST”: Remember, even if we have no other opinion we can choose from S-expressions and XML and JSON and ASN.1 and PkZip and so on. Each with different disadvantages.

                                                                                                                        And once you serialise, you might as well freeze them someplace. Maybe a hierarchy. This thing is called a file system, and those frozen blobs are called files.

                                                                                                                        Messaging and networking are a way to build a platform that doesn’t have a filesystem of files. They aren’t mentioned in the article, but then: no solutions are really offered by the article.

                                                                                                                        1. 1

                                                                                                                          Messaging and networking are a way to build a platform that doesn’t have a filesystem of files. They aren’t mentioned in the article, but then: no solutions are really offered by the article.

                                                                                                                          This is interesting, what do you mean by using messaging as a substitute for a filesystem, what would that look like?

                                                                                                                          1. 2

                                                                                                                            iOS does something like this (awkwardly; through a blessed but ad hoc mechanism). You send a message to another app- and ask it send a message back to you.

                                                                                                                            One obvious use is storing things that we used to store in files, like photos and preference and music, but we can also use it to authenticate (who are you), to authorise (do you allow this), to purchase, and perhaps for other things.

                                                                                                                            Urbit is exploring some of these themes in a much more grand scale, but is so much less a “complete” operating environment at this point to teach us what computing will be like in this way.

                                                                                                                            HDCP/HDMI has another (limited) use of this where you can play a (protected) video at an x/y/w/h without revealing the bits.

                                                                                                                            The Mother of All Demos hinted at some of this with their collaborative single user super computer.

                                                                                                                            And so on.

                                                                                                                1. 23

                                                                                                                  Kinda late on UNIX bashing bandwagon :)

                                                                                                                  Also, Windows owes more of it’s legacy to VMS.

                                                                                                                  1. 10

                                                                                                                    It does, but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.

                                                                                                                    Programmers rarely even use the VMSy goodness, especially if they also want their stuff to work on Mac. They treat Windows as a kind of retarded UNIX cousin (which is a shame because the API is better; IOCP et al)

                                                                                                                    Sysadmins often struggle with Windows because of all the things underneath that aren’t files.

                                                                                                                    Message/Object operating systems are interesting, but for the most part (OS/2, BeOS, QNX) they, for the most part, degraded into this “everything is a file” nonsense…

                                                                                                                    Until they got rid of the shared filesystem: iOS finally required messaging for applications to communicate on their own, and while it’s been rocky, it’s starting to paint a picture to the next generation who will finally make an operating system without files.

                                                                                                                    1. 10

                                                                                                                      If we talk user experiences, it’s more a CP/M clone than anything. Generations later, Windows still smells COMMAND.COM.

                                                                                                                      1. 6

                                                                                                                        yes, the bowels are VMS, the visible stuff going out is CP/M

                                                                                                                        1. 4

                                                                                                                          Bowels is a good metaphor. There’s good stuff in Windows, but you’ve got to put on a shoulder length glove and grab a vat of crisco before you can find any of it.

                                                                                                                      2. 10

                                                                                                                        I think you’re being a little bit harsh. End-users definitely don’t grok the VMSy goodness; I agree. And maybe the majority of developers don’t, either (though I doubt the majority of Linux devs grok journald v. syslogs, really understand how to use /proc, grok Linux namespaces, etc.). But I’ve worked with enough Windows shops to promise you that a reasonable number of Windows developers do get the difference.

                                                                                                                        That said, I have a half-finished book from a couple years ago, tentatively called Windows Is Not Linux, which dove into a lot of the, “okay, I know you want to do $x because that’s how you did it on Linux, and doing $x on Windows stinks, so you think Windows stinks, but let me walk you through $y and explain to you why it’s at least as good as the Linux way even though it’s different,” specifically because I got fed up with devs saying Windows was awful when they didn’t get how to use it. Things in that bucket included not remoting in to do syswork (use WMI/WinRM), not doing raw text munging unless you actually have to (COM from VBScript/PowerShell are your friends), adapting to the UAC model v. the sudo model, etc. The Windows way can actually be very nice, but untraining habits is indeed hard.

                                                                                                                        1. 6

                                                                                                                          I don’t disagree with any of that (except maybe that I’m being harsh), but if you parse what I’m saying as “Windows is awful” then it’s because my indelicate tone has been read into instead of my words.

                                                                                                                          The point of the article is that those differences are superficial, and mean so very little to the mental model of use and implementation as to make no difference: IOCP is just threads and epoll, and epoll is just IOCP and fifos. Yes, IOCP is better, but I desperately want to see something new in how I use an operating system.

                                                                                                                          I’ve been doing things roughly the same way for nearly four decades, despite the fact that I’ve done Microsoft/IBM for a decade, Linux since Slackware 1.1 (Unix since tapes of SCO), Common Lisp (of all things) for a decade, and OSX for nearly that long. They’re all the same, and that point is painfully clear to anyone who has actually used these things at a high level: I edit files, I copy files, I run programs. Huzzah.

                                                                                                                          But: It’s also obvious to me who has gone into the bowels of these systems as well: I wrote winback which was for a long time the only tools for doing online Windows backups of standalone exchange servers and domain controllers; I’m the author of (perhaps) the fastest Linux webserver; I wrote ml a Linux emulator for OSX; I worked on ECL adding principally CL exceptions to streams and the Slime implementation. And so on.

                                                                                                                          So: I understand what you mean when you say Windows is not Linux, but I also understand what the author means when he says they’re the same.

                                                                                                                          1. 2

                                                                                                                            That actually makes a ton of sense. Can I ask what would qualify as meaningfully different for you? Oberon, maybe? Or a version of Windows where WinRT was front-and-center from the kernel level upwards?

                                                                                                                            1. 2

                                                                                                                              I didn’t use the term “meaningfully different”, so I might be interpreting your question you too broadly.

                                                                                                                              When I used VMS, I never “made a backup” before I changed a file. That’s really quite powerful.

                                                                                                                              The Canon Cat had “pages” you would scroll through. Like other forth environments, if you named any of your blocks/documents it was so you could search [leap] for them, not because you had hierarchy.

                                                                                                                              I also think containers are very interesting. The encapsulation of the application seems to massively change the way we use them. Like the iOS example, they don’t seem to need “files” since the files live inside the container/app. This poses some risk for data portability. There are other problems.

                                                                                                                              I never used Oberon or WinRT enough to feel as comfortable commenting about them as I do about some of these other examples.

                                                                                                                          2. 2

                                                                                                                            If it’s any motivation I would love to read this book.

                                                                                                                            Do you know of any books or posts I could read in the meantime? I’m very open to the idea that Windows is nice if you know which tools and mental models to use, but kind of by definition I’m not sure what to Google to find them :)

                                                                                                                            1. 4

                                                                                                                              I’ve just been hesitant because I worked in management for two years after I started the book (meaning my information atrophied), and now I don’t work with Windows very much. So, unfortunately, I don’t immediately have a great suggestion for you. Yeah, you could read Windows Internals 6, which is what I did when I was working on the book, but that’s 2000+ pages, and most of it honestly isn’t relevant for a normal developer.

                                                                                                                              That said, if you’ve got specific questions, I’d love to hear them. Maybe there’s a tl;dr blog post hiding in them, where I could salvage some of my work without completing the entire book.

                                                                                                                            2. 1

                                                                                                                              I, for one, would pay for your “Windows is not Linux” book. I’ve been developing for Windows for about 15 years, but I’m sure there are still things I could learn from such a book.

                                                                                                                            3. 7

                                                                                                                              but users don’t use any of the VMSy goodness in Windows: To them it’s just another shitty UNIX clone, with everything being a file or a program (which is also a file). I think that’s the point.

                                                                                                                              Most users don’t know anything about UNIX and can’t use it. On the UI side, pre-NT Windows was a Mac knockoff mixed with MSDOS which was based on a DOS they got from a third party. Microsoft even developed software for Apple in that time. Microsoft’s own users had previously learned MSDOS menu and some commands. Then, they had a nifty UI like Apple’s running on MSDOS. Then, Microsoft worked with IBM to make a new OS/2 with its philosophy. Then, Microsoft acquired OpenVMS team, made new kernel, and a new GUI w/ wizard-based configuration of services vs command line, text, and pipes like in UNIX.

                                                                                                                              So, historically, internally, layperson-facing, and administration, Windows is a totally different thing than UNIX. Hence, the difficulty moving Windows users to UNIX when it’s a terminal OS with X Windows vs some Windows-style stuff like Gnome or KDE.

                                                                                                                              You’re also overstating the everything is a file by conflating OS’s that store programs or something in files vs those like UNIX or Plan 9 that use file metaphor for about everything. It’s a false equivalence: from what I remember, you don’t get your running processes in Windows by reading the filesystem since they don’t use that metaphor or API. It’s object based with API calls specific to different categories. Different philosophy.

                                                                                                                              1. 3

                                                                                                                                Bitsavers has some internal emails from DEC at the time of David Cutler’s departure.

                                                                                                                                I have linked to a few of them.

                                                                                                                                David Cutler’s team at DECwest was working on Mica (an operating system) for PRISM (a RISC CPU architecture). PRISM was canceled in June of 1988. Cutler resigned in August of 1988 and 8 other DECwest alumni followed him at Microsoft.

                                                                                                                            4. 5

                                                                                                                              I have my paper copy of The Unix Hater’s Handbook always close at hand (although I’m missing the barf bag, sad to say).

                                                                                                                              1. 5

                                                                                                                                I always wanted to ask the author of The Unix Hater’s Handbook if he’s using Mac OS X

                                                                                                                                8~)

                                                                                                                                1. 6

                                                                                                                                  It was edited by Simson Garfinkel, who co-wrote Building Cocoa Applications: a step-by-step guide. Which was sort of a “port” of Nextstep Programming Step One: object-oriented applications

                                                                                                                                  Or, in other words, “yes” :)

                                                                                                                                  1. 2

                                                                                                                                    Add me to the list curious about what they ended up using. The hoaxers behind UNIX admitted they’ve been coding in Pascal on Macs. Maybe it’s what the rest were using if not Common LISP on Macs.

                                                                                                                                2. 7

                                                                                                                                  Beat me to it. Author is full of it right when saying Windows is built on UNIX. Microsoft stealing, cloning, and improving OpenVMS into Windows NT is described here. This makes the Linux zealots’ parodies about a VMS desktop funnier given one destroyed Linux in desktop market. So, we have VMS and UNIX family trees going in parallel with the UNIX tree having more branches.

                                                                                                                                  1. 4

                                                                                                                                    The author doesn’t say Windows is built on Unix.

                                                                                                                                    1. 5

                                                                                                                                      “we are forced to choose from: Windows, Apple, Other (which I shall refer to as “Linux” despite it technically being more specific). All of these are built around the same foundational concepts, those of Unix.”

                                                                                                                                      Says it’s built on the foundational concepts of UNIX. It’s built on a combo of DOS, OS/2, OpenVMS, and Microsoft concepts they called the NT kernel. The only thing UNIX-like was the networking stack they got from Spider Systems. They’ve since rewritten their networking stack from what I heard.

                                                                                                                                      1. 4

                                                                                                                                        Says it’s built on the foundational concepts of UNIX.

                                                                                                                                        I don’t see any reason to disagree with that.

                                                                                                                                        The only thing UNIX-like …

                                                                                                                                        I don’t think that’s a helpful definition of “unix-like”.

                                                                                                                                        It’s got files. Everything is a file. Windows might even be a better UNIX than Linux (since UNC)

                                                                                                                                        Cutler might not have liked UNIX very much, but Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.

                                                                                                                                        1. 10

                                                                                                                                          It’s got files. Everything is a file.

                                                                                                                                          Windows is object-based. It does have files which are another object. The files come from MULTICS which UNIX also copied in some ways. Even the name was a play on it: UNICS. I think Titan invented the access permissions. The internal model with its subsystems were more like microkernel designs running OS emulators as processes. They did their own thing for most of the rest with the Win32 API and registry. Again, not quite how a UNIX programming guide teaches you to do things. They got clustering later, too, with them and Oracle using the distributed, lock approach from OpenVMS.

                                                                                                                                          Windows and UNIX are very different in approach to architecture. They’re different in how developer is expected to build individual apps and compose them. It wasn’t even developed on UNIX: they used OS/2 workstations for that. There’s no reason to say Windows is ground in the UNIX philosophy. It’s a lie.

                                                                                                                                          “Windows NT ended up UNIX anyway because none of that VMS-goodness (Versions, types, streams, clusters) ended up in the hands of Users.”

                                                                                                                                          I don’t know what you’re saying here. Neither VMS nor Windows teams intended to do anything for UNIX users. They took their own path except for networking for obvious reasons. UNIX users actively resisted Microsoft tech, too. Especially BSD and Linux users that often hated them. They’d reflexively do the opposite of Microsoft except when making knockoffs of their key products like Office to get desktop users.

                                                                                                                                          1. 3

                                                                                                                                            Windows is object-based.

                                                                                                                                            Consider what methods of that “object” a program like Microsoft Word must be calling besides “ReadFile” and “WriteFile”.

                                                                                                                                            That the kernel supports more methods is completely pointless. Users don’t interact with it. Programmers avoid it. Sysadmins don’t understand it and get it wrong.

                                                                                                                                            I don’t know what you’re saying here.

                                                                                                                                            That is clear, and yet you’re insisting I’m wrong.

                                                                                                                                            1. 3

                                                                                                                                              Except, that’s completely wrong.

                                                                                                                                              I just started Word and dumped a summary of its open handles by object type:

                                                                                                                                              C:\WINDOWS\system32>handle -s -p WinWord.exe
                                                                                                                                              
                                                                                                                                              Nthandle v4.11 - Handle viewer
                                                                                                                                              Copyright (C) 1997-2017 Mark Russinovich
                                                                                                                                              Sysinternals - www.sysinternals.com
                                                                                                                                              
                                                                                                                                              Handle type summary:
                                                                                                                                                ALPC Port       : 33
                                                                                                                                                Desktop         : 1
                                                                                                                                                Directory       : 3
                                                                                                                                                DxgkSharedResource: 2
                                                                                                                                                DxgkSharedSyncObject: 1
                                                                                                                                                EtwRegistration : 324
                                                                                                                                                Event           : 431
                                                                                                                                                File            : 75
                                                                                                                                                IoCompletion    : 66
                                                                                                                                                IoCompletionReserve: 1
                                                                                                                                                IRTimer         : 8
                                                                                                                                                Key             : 171
                                                                                                                                                KeyedEvent      : 24
                                                                                                                                                Mutant          : 32
                                                                                                                                                Process         : 2
                                                                                                                                                Section         : 67
                                                                                                                                                Semaphore       : 108
                                                                                                                                                Thread          : 138
                                                                                                                                                Timer           : 7
                                                                                                                                                Token           : 3
                                                                                                                                                TpWorkerFactory : 4
                                                                                                                                                WaitCompletionPacket: 36
                                                                                                                                                WindowStation   : 2
                                                                                                                                              Total handles: 1539
                                                                                                                                              

                                                                                                                                              Each of these types is a distinct kernel object with its own characteristics and semantics. And yes, you do create and interact with them from user-space. Some of those will be abstracted by lower-level APIs, but many are directly created and managed by the application. You’ll note the number of open “files” is a very small minority of the total number of open handles.

                                                                                                                                              Simple examples of non-file object types commonly manipulated from user-land include Mutants (CreateMutex) and Semaphores (CreateSemaphore). Perhaps the most prominent example is manipulating the Windows Registry; this entails opening “Key” objects, which per above are entirely distinct from regular files. See the MSDN Registry Functions reference.

                                                                                                                                              1. 0

                                                                                                                                                None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.

                                                                                                                                                Did you even read the article? Or are you trolling?

                                                                                                                                                1. 3

                                                                                                                                                  None of these objects can exist on a disk; they cannot persist beyond shutdown, and do not have any representation beyond their instantaneous in-memory instance. When someone wants an “EtwRegistration” they’re creating it again and again.

                                                                                                                                                  Key objects do typically exist on disk. Albeit, the underlying datastore for the Registry is a series of files, but you never directly manipulate those files. In the same sense you may ask for C:\whatever.txt, you may ask for HKLM:\whatever. We need to somehow isolate the different persisted data streams, and that isolation mechanism is a file. That doesn’t mean you have to directly manipulate those files if the operating system provides higher-level abstractions. What exactly are you after?

                                                                                                                                                  From the article:

                                                                                                                                                  But in Unix land, this is a taboo. Binary files are opaque, say the Unix ideologues. They are hard to read and write. Instead, we use Text Files, for it is surely the path of true righteousness we have taken.

                                                                                                                                                  The Windows Registry, which is a core part of the operating system, is completely counter to this. It’s a bunch of large binary files, precisely because Microsoft recognised storing all that configuration data in plain text files would be completely impractical. So you don’t open a text file and write to it, you open a Registry key, and store data in it using one of many predefined data types (REG_DWORD, etc…).

                                                                                                                                                  Did you even read the article? Or are you trolling?

                                                                                                                                                  It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.

                                                                                                                                                  1. -3

                                                                                                                                                    What exactly are you after?

                                                                                                                                                    Just go read the article.

                                                                                                                                                    It’s about whether basing our entire interactions with a computer on a specific reduction of verbs (read and write) is really exploring what the operating system can do for us.

                                                                                                                                                    That is a very interesting subject to me.

                                                                                                                                                    Some idiot took party to the idea that Windows basically “built on Unix” then back-pedalled it to be about whether it was based on the same “foundational” concepts, then chooses to narrowly and uniquely interpret “foundational” in a very different way than the article.

                                                                                                                                                    Yes, windows has domains and registries and lots of directory services, but they all have the exact same “file” semantics.

                                                                                                                                                    But now you’re responding to this strange interpretation of “foundational” because you didn’t read the article either. Or you’re a troll. I’m not sure which yet.

                                                                                                                                                    Read the article. It’s not well written but it’s a very interesting idea.

                                                                                                                                                    Each of these types is a distinct kernel object with its own characteristics and semantics

                                                                                                                                                    Why do you bring this up in response to whether Windows is basically the same as Unix? Unix has lots of different kernel “types” all backed by “handles”. Some operations and semantics are shared by handles of different types, but some are distinct.

                                                                                                                                                    I don’t understand why you think this is important at all.

                                                                                                                                                    It sounds like you’re not interested in a constructive and respectful dialogue. If you are, you should work on your approach.

                                                                                                                                                    Do you often jump into the middle of a conversation with “Except, that’s completely wrong?”

                                                                                                                                                    Or are you only an asshole on the Internet?

                                                                                                                                                    1. 4

                                                                                                                                                      Or are you only an asshole on the Internet?

                                                                                                                                                      I’m not in the habit of calling people “asshole” anywhere, Internet or otherwise. You’d honestly be more persuasive if you just made your points without the nasty attacks. I’ll leave it at that.

                                                                                                                                            2. 2

                                                                                                                                              networking for obvious reasons

                                                                                                                                              Them being what? Is the BSD socket API really the ultimate networking abstraction?

                                                                                                                                              1. 7

                                                                                                                                                The TCP/IP protocols were part of a UNIX. AT&T gave UNIX away for free. They spread together with early applications being built on UNIX. Anyone reusing the protocols or code will inherit some of what UNIX folks were doing. They were also the most mature networking stacks for that reason. It’s why re-using BSD stacks was popular among proprietary vendors. On top of the licensing.

                                                                                                                                                Edit: Tried to Google you a source talking about this. I found one that mentions it.

                                                                                                                                  1. 1

                                                                                                                                    This is a neat trick!

                                                                                                                                    1. 2

                                                                                                                                      I read the whitepaper last night, it’s worrying that SipHash is being used in this way. A second-preimage attack against SipHash could help speed up the low-memory algorithm and SipHash wasn’t explicitly designed to be secure against those when the secret key is known.

                                                                                                                                      Zooko says it better than I can:

                                                                                                                                      One detail that bothers me is that SipHash is being used in Cuckoo PoW in a way that the attacker gets to control all the inputs to SipHash, and that is not what SipHash was designed to resist. SipHash was designed to resist an attacker who doesn’t control — and actually doesn’t even know — the key. There’s a possibility (although it seems unlikely to me) that an attacker could exploit something about the way Cuckoo uses SipHash to find Cuckoo solutions faster than by treating SipHash as a random oracle.

                                                                                                                                      1. 3

                                                                                                                                        I love the idea of giving out bounties to anyone who can speed up the miners.

                                                                                                                                        If you’ve found a speed-up it makes far more sense to claim the bounty now than to hope nobody else notices before a public chain launches and becomes valuable.