1. 4

    I would never have imagined that the term ‘air-gapped’ even includes systems that have had a usb device plugged into them that was ever plugged into a non-air-gapped system. If someone sent me confidential documents on a usb drive my first thought would be ‘damn there is probably all sorts of malware on this’. Also people forgetting to disable autorun on a windows machine with sensitive material on it… When I saw the headline I was expecting something much more sophisticated.

    1. 7

      All the air-gapped systems and networks I’ve had the pleasure of working on all used CDs and DVDs as their method of data transfer. When we would ship air-gapped systems to particular customers, we would fill all USB ports with epoxy resin.

      1. 3

        no data-diodes for inbound? (or if so, what COTS products have you found for that).

        1. 1

          I’ve always been fascinated with air-gapped systems/networks so thanks for the post/background. Is there a reason why y’all don’t just ship systems without USB ports altogether, or is it cheaper to grab an off-the-shelf system and then airgap it rather than build a custom PC for that purpose?

          1. 2

            In a lot of cases, they don’t actually epoxy the USB ports, they fit removable covers on them. It turns out that they’re actually useful for debugging purposes, as long as you can restrict their use to authorised people.

            Back in the early Xen days, a certain three-letter agency looked at using direct device pass through of one NIC to an untrusted VM to avoid the need for separate computers for their secure and insecure networks. The idea was to have a separate NIC for the insecure network and run insecure machine as a VM. Even without VM escapes, they were concerned that you could establish a covert channel based on the interrupt latency of the insecure NIC and exfiltrate data.

            My favourite air-gap side channel attack was a Tor node decloaking attack from Stephen Murdoch and friends. He discovered that the temperature change from a Tor node processing a lot of traffic was detectable from the thermal sensors in other machines in the same rack (and probably from other machines in nearby racks with more samples). If a nation-state adversary can buy space in the same co-located datacenter as your Tor node, then they can start to unpeel the onion.

            1. 2

              Back in the early Xen days, a certain three-letter agency looked at using direct device pass through of one NIC to an untrusted VM to avoid the need for separate computers for their secure and insecure networks.

              That’s really funny to learn. In the early VMware days (late 2000 or so) that same three-letter agency wanted us to partner with HP to deliver “NetTop”. The idea is to run SELinux on the host, and then use its restrictions to run multiple Windows VMs at different classification levels.

              It didn’t pan out, since the NSA security team was able to send a half a bit per second or so of data from a cooperating process in the high security VM to a cooperating process in the unclassified VM by attaching and detaching virtual USB devices, and measuring the timing perturbations in the other VM. No doubt there were many other timing side channels they could have used.

              There were running jokes for years internally along the lines of “Out for vacation all next week. If you need to reach me, simply plug and unplug virtual USB devices in the agreed upon pattern…”.

              1. 1

                Not really at the same level, but when I did my thesis at a large Swedish telecommunications manufacturer in the late 90s I heard of someone bypassing the firewall by accessing a machine that wrote data from the WAN onto a hard drive, which was then read by the LAN interface, and vice-versa. Slow but if you want to access Slashdot, needs must.

                1. 1

                  Yeah. We had some rather special requirements regarding all ports, not just USB. :-D

                2. 2

                  Using COTS is much cheaper and enabled us to more easily customize the system based on customer needs.

                  1. 1

                    Seems to me that it is also expedient to reinstall the OS, preferably with non-windows. I understand it depends what you are doing but if I would think about using raspberry pi or better yet, arduino. Then desolder any communication hardware from the board. You would have to use only self built or self programmed software so if you are doing anything more complicated than reading documents it could be an issue. This was another thing that surprised me about the article. If one is doing something this sensitive why would one trust a vendor installed OS? Or one with as commonly compromised as windows?

              2. 1

                Fortunately for Israel, the folks working at the Iranian nuclear facility did not have the same mindset and plugged in a USB stick containing malware that infected the control systems of their centrifuges and destroyed them.

              1. 19

                While I agree with the reasoning in the article and think the approach suggested makes sense, I also think it misses the wider picture and is only likely to result in small gains. The real issue with modern tech is not technical it is political, and there is not technical solution to this problem.

                The solution proposed here may get adopted and result in marginal gains if and only if some corporate executives decide it could make their company more profitable.

                The real issue with tech as I see it is the participation of for profit corporations. The goals of for profit corporations are not aligned with those of the human race in general, or individual humans in particular, unless those humans are shareholders. They put non-technical people in charge of development projects because those people have the capital everyone needs to get the job done. They prefer opaque proprietary protocols and interfaces because it enables them to entrap their users and gives them a chance to build monopolies. The create broken by design systems like DRM to protect profits. They turn a blind eye to bad behaviour like spam, clickbaiting, trolling and scamming whenever these are sources of profit for them. They hide their source from scrutiny which might otherwise discourage bad behaviour and discover security issues. They work programmers too hard for too long hours, while browbeating them and breaking their spirit until they no longer take pride in their work. Their focus is always short term. Worst of all, they discourage the population at large from understanding how their devices work, preferring instead to dumb all human interfaces down and make people dependant on ‘tech support’. Their interest is not in building a good base of software for enabling human beings to live full and happy lives with everyday technology, their goal is to siphon resources out of society and use it to grow out of control and ultimately take over everything. Like a malignant tumour.

                I know a lot of programmers don’t like talking about this aspect of things but I think it is important. It may seem off topic but if your read the three introductory paragraphs of the article again I think you will see that what I am talking about is probably a major factor in the problems the author is trying to solve.

                1. 23

                  We see the same problems in open source projects, free software, and government tech. I bet if you peeked into Soviet software companies you’d see the same problems, too. Blaming capitalism is the lazy way out.

                  1. 17

                    This is too conflationary of a dismissal. Look again at the list of problems:

                    • Non-technical leaders giving top-down orders
                    • Building and publishing proprietary interfaces
                    • Designing DRM
                    • Giving reputation to bad actors as long as it is profitable
                    • Building private code repositories and refusing to share knowledge
                    • Forcing programmers to work long hours
                    • Blocking right-to-repair and right-of-first-sale

                    Your reply is that this also happens in the open-source world, which is a corporate bastardization of Free Software; and that it happened in the USSR, a time and place where a government pretended to be “communist” but still had shops, jobs, wages, and a wealthy upper class. Yes, I imagine that these capitalist and corporatist patterns are recurring in many contexts. No, it is wrong to lump them all together with Free Software.

                    1. 16

                      Looking at the list of problems:

                      • Not in the article.
                      • Not in the article.
                      • Not in the article.
                      • Not in the article.
                      • Not in the article.
                      • Arguable?
                      • Not in the article.

                      That’s why it’s the lazy way out. It doesn’t engage with the article at all. You just have to say “it’s capitalism’s fault” and watch the upvotes roll in.

                      1. 12

                        Just because anti-capitalist/corporate sentiment is popular doesn’t make it lazy or disingenuous. Putting the argument in a cultural/political context is a perfectly valid way of engaging with it. Pointing out that they come up with their own examples that are not in the article, as if that’s a bad thing, is a weird way to make a counterargument, given that the article is concerned with “ways we keep screwing up tech.” Giving corporations too much control over its direction seems like a pretty big way we’ve screwed up tech so far.

                        1. 5

                          Software development suffers during demand imbalance. In a market situation with fierce life or death competition, people work insane hours and abandon practice just to get things out the door. In markets where companies make money/survive no matter what they do, practice (and code) atrophy. No one cares. The latter case has effects that are remarkably similar among government projects, non-profits, and large institutions like banks and insurance companies with little competition.

                          You can blame Capitalism, but the truth is people and projects need purpose and incentive; just not too much. There’s a sweet spot. It is like having a good exercise routine. Without it you atrophy; with too much, you destroy yourself.

                          1. 2

                            I largely agree with that point. And I’d be naive to argue that capitalism alone is the cause of all problems in the tech industry, which is why I didn’t. At the same time, I think that capitalism, at least as it exists in the United States today, offers pretty bad incentives for teams and individuals most of the time. Professionals in the computing field have it better than most in that they have large monetary incentives. But above about $75K, as Kahneman and Deaton found, more income has little effect on one’s evaluation of own’s own life. Beyond that, you still have the problem that the interests of working people, including programmers, mostly don’t align with oligarchs’ interests. I’m not alone in the experience that even at a substantial salary, it’s pretty hard to feel incentivized to deliver value, ultimately, to those who own vastly more wealth. Not what I would call a sweet spot.

                            1. 3

                              It would probably be good for you to talk to programmers in Sweden (if you haven’t). Less capitalistic, much more socialist. A lot of big employers and relatively fewer startups. They have problems too. And that’s the thing. All systems have problems. It’s just a matter of tradeoffs. It’s nice that there’s enough diversity in the world to be able to see them.

                        2. 8

                          (You should know that I wrote and deleted about seven paragraphs while attempting to engage with the article seriously, including this one.)

                          Let’s take the article’s concluding suggestion seriously; let’s have a top-ten list of ways that “the technology industry” is “screwing up creating tech”. I would put at number one, at the very top: The technology industry’s primary mode of software creation is providing services for profit, and this damages the entire ecosystem. What ten behaviors would you put above this one? The only two contenders I can think of are closely related: Building patent pools and using work-for-hire provisions to take copyrights from employees.

                          It’s not capitalism’s fault that self-described “experts” are having trouble giving useful advice.

                          1. 2

                            Is there an “industry” that doesn’t exist for profit?

                            I think needless churn, overcomplication, harmful treatment of developers, harmful treatment of users, and wasteful use of energy are all ways the industry screws up tech…but then again, so too does libre software!

                            And unlike industry, there isn’t even an evolutionary pressure you can hijack to keep it honest.

                            1. 3

                              Beavers do not terraform for profit. The bulk of free culture is not profit-driven either.

                              Note carefully that you substituted the original list of seven problems which are all created by profit-seeking behavior with your own list of five issues which are not. Yours are also relatively vague to the point of equivocation. I’ll give a couple examples to illustrate what I mean.

                              For example, where you say “harmful treatment of users”, I said “giving reputation to bad actors as long as it is profitable” and the original comment said “spam, clickbaiting, trolling and scamming”; and I do not know of examples in the FLOSS world which are comparable to for-profit spamming or other basic fraud, to say nothing of electoral interference or genocide, although I’m open to learning about atrocities committed in the furtherance of Free Software.

                              For another example, where you say “harmful treatment of developers”, I said “forcing programmers to work long hours” and they said “[for-profit corporations] work programmers too hard for too long hours, while browbeating them and breaking their spirit until they no longer take pride in their work”; you have removed the employer-employee relationship which was the direct cause of the harmful treatment. After all, without such a relationship, there is no force in the Free Software world which compels labor from people. I’ll admit that it’s possible to be yelled at by Linus, but the only time that happened to me was when I was part of a team that was performing a corporation-led restructuring of a kernel subsystem in order to appease a still-unknown client.

                              1. 3

                                You asked for a top-ten list, I gave five; don’t bait-and-switch.

                                Each of those examples I gave can and does happen under both a for-profit model and under a libre-hippy-doing-it-for-free model.

                          2. 7

                            Not in the article.

                            “But once I started working with teams-of-teams, process, and changing large groups of people, things did not work the way I expected.” “I found myself repeating the same things over and over again and getting the same substandard results.” “ came to realize that these were honest efforts to get more traction.” “I got worse at learning new stuff” “tired of watching my friends repeating the same things over and over again, snatching at this or that new shiny in an effort to stay relevant”

                            It is fine that you disagree with my interpretation of how the article describes the problem. Maybe I completely misunderstood what the author was referring to. Perhaps you could enlighten me and provide the correct interpretation. I look forward to your response.

                        3. 9

                          Solely blaming capitalism certainly won’t address all problems with software development, but I do think there’s a case to be made that the profit motive is just as good a tool for producing pathological states as innovation. DRM is a prime example.

                          1. 11

                            The original post had nothing to with DRM, putting nontechnical people in charge, hiding source code, working long hours, or dumbed down human interfaces. Dunkhan didn’t engage at all with the actual article, he just said it was all because of “for profit corporations” and listed unrelated claims.

                            You could argue that profit motive is a good tool for producing pathological states; he didn’t.

                          2. 7

                            I mean, open source in the large is dominated by corporate interests, so this isn’t much of a gotcha. Just ignoring capitalism as a fundamental factor seems like the significantly lazier option.

                            1. 10

                              I never blamed capitalism. Adam Smith, and most respected capitalist economists that followed have all said that monopoly formation is to be avoided at all costs and if not correctly controlled will completely undermine any benefits of a capitalist system. If you go through the behaviour I called out, almost every example is either the result of, or an effort to achieve a monopoly. It would be more accurate to say I am defending capitalism against those corporations who are ruining it. If you want an -ism that I am blaming, I think the closest analogy to what they are doing is some kind of feudalism. Lets call it corporate neofeudalism.

                              The main tool we have for combating this behaviour also comes directly from the same canon of capitalist thinkers. Namely, antitrust laws. All we need to do is expand them, make them harsher, and then ruthlessly enforce them. There is no need for new and creative solutions to this problem, it is an old problem and has been solved already.

                              1. 3

                                IMO when somebody says “the problems with software are that it’s being done for profit” there’s no need to talk with them about software any more. They’re not interested in fixing software, and if that’s all you feel like talking about right here and right now there’s no useful outcome in engaging with them.

                                1. 0

                                  No one said that.

                                  If you want to put words in other people’s mouths and then dismiss their opinions based on those words go right ahead, but it does not reflect well on you. The fact that you see any attack on corporations as an attack on profit making of all kinds also concerns me a great deal. Did you take the Citizens United ruling so seriously that you really believe corporations are people, or are you just getting paid to defend their interests on the web?

                            1. 2

                              The second you ban exceptions, you aren’t coding in C++ anymore.

                              Quoting Stroustrup:

                              The origins of exception handling lie in the problems experienced managing a variety of error-handling approaches, such as C’s errno, error-states, callbacks, return codes, and return objects. In addition, it was observed that there were no really good approaches to reporting errors detected in constructors and in operators. I and others were very aware of the literature on exceptions and error handling stretching back to Goodenough’s pioneering work. Before exceptions were adopted into C++, many standards meetings had sessions devoted to the topic and industry experts from outside the committee were consulted. Implementation alternatives were examined, and ideals were articulated. A relatively brief description can be found in D&E. The integration of exception handling with constructor/destructor resource managements in the form of RAII is one of the major achievements of C++ – still not fully appreciated by people relying on error-code based error handling.

                              The language was constructed in a way that exceptions and ctors/dtors go hand-in-hand. If you remove one, the others stop making sense.

                              1. 1

                                Counter-arguments can be found here.

                                I know little about c++ and have no opinion on this topic, but after reading this article and this comment I wanted to know why companies like google do disable exceptions, and I thought others might want this info too.

                                1. 1

                                  Quoting from your link:

                                  On their face, the benefits of using exceptions outweigh the costs, especially in new projects.

                                  The only reason Google does not have exceptions is legacy, and legacy is never a valid technical argument, it’s at best a social one.

                                  1. 3

                                    Pretty much every large C++ codebase I’ve worked on disables exceptions, for several reasons:

                                    • Exceptions make every call a conditional branch from the perspective of the compiler and so impedes some optimisations.
                                    • Exceptions require RTTI, which adds a lot to the binary size.
                                    • C++ has only unchecked exceptions, which make it impossible to statically verify whether errors are handled. Explicitly returning an option type does not have this limitation. LLVM has some nice templates for this that abort the program if you don’t check the error (which can include propagating the error to the caller via the move constructor).

                                    The last of these is the really important one. Getting error handling right is one of the most important things for reliability and in a C++ codebase with exceptions it is impossible to audit a function to ensure that every error condition is either handled or propagated to a caller that knows that it needs to handle it. If you disable exceptions then you can do this kind of local analysis: the function either checks the return values of things that can fail or returns the option type.

                                    The list of pros in the list are not really compelling to me:

                                    Exceptions allow higher levels of an application to decide how to handle “can’t happen” failures in deeply nested functions, without the obscuring and error-prone bookkeeping of error codes.

                                    Except that C++ has unchecked exceptions and so reasoning about what exceptions are thrown and making sure that they’re caught is very hard. The compiler doesn’t help and you end up needing catch(...) { /* no idea what do do here! */ } everywhere in your code if you want it to actually be exception safe.

                                    Prior to C++17, you could put throws() at the end of your function and the compiler would insert code that would prevent any exceptions propagating out of your function. This was unpopular because it made codegen worse and binaries bigger. In C++17 this was replaced entirely with noexcept that simply says it is UB if you throw an exception out of the function, so in C++17 exceptions introduce a new way of adding UB into your codebase.

                                    Exceptions are used by most other modern languages. Using them in C++ would make it more consistent with Python, Java, and the C++ that others are familiar with.

                                    Other languages typically have checked exceptions. In Java, for example, the exceptions that can be thrown are part of the function signature. If you don’t explicitly catch an exception in a function that you call or advertise it on the list that you throw, then your code doesn’t compile.

                                    C++ has a feature that is somewhat like a feature in other languages but with limitations that mean that it doesn’t have the advantages that it has in other languages is not a great reason.

                                    Some third-party C++ libraries use exceptions, and turning them off internally makes it harder to integrate with those libraries.

                                    Possibly an advantage, though it’s very rare for me to find a library that I want to use that requires exceptions (and when I do, the above problems make me sad and glad that I disable them everywhere else).

                                    Exceptions are the only way for a constructor to fail. We can simulate this with a factory function or an Init() method, but these require heap allocation or a new “invalid” state, respectively.

                                    This is true and is, in my mind, the only benefit of exceptions in C++. I see this more as a limitation of the language though. Allowing constructors to have explicit return types would fix this.

                                    Exceptions are really handy in testing frameworks.

                                    Uh, okay? I’ve never wanted to use exceptions in a testing framework.

                                    Oh, and using factory methods doesn’t preclude on-stack allocation (it does preclude subclassing, though you can also make your constructor protected if you want to allow subclassing). Here’s an example:

                                    class Example
                                            int x;
                                            Example(int x) : x(x) {}
                                            static bool validate_constructor_args(int x)
                                                    return x < 42;
                                            Example(const Example &other) : x(other.x) {}
                                            static std::optional<Example> create(int x)
                                                    if (!validate_constructor_args(x))
                                                            return std::nullopt;
                                                    return Example{x};

                                    You can create instances of this on the stack with:

                                    auto x = Example::create(12);

                                    You now have an on-stack option type that either contains an Example if you passed valid arguments to the constructor, or doesn’t if the arguments were invalid. No heap allocation here at all and it’s trivial for me to audit this use and see if you tested whether the constructor failed.

                              1. 6

                                Please choose option 2. This goes for everyone. I see no other take-away from this article. If the author had done things properly he would have been fired. We have learned nothing new about best practices or how to effectively test. All the techniques and practices that would have prevented this were developed decades ago and are well established. The only solution is to quit your job. I urge everyone that is in a similar situation to do the same. Even if it is risky, it is one of the only means humans have left for taking control.

                                I feel like we all need to consider if what we are doing for work is productive or destructive for society at large. Big companies and corporations are constantly gaining in power at the cost of democratic institutions and the time is drawing ever closer that they wont really need human beings so much anymore (I am talking long term, not in the next few years). If you aren’t sure, it is likely destructive in the long term. I understand a lot of people rely on their job for survival and feel like they have no choice. People will sometimes put their lives at risk to try to disrupt a totalitarian regime that is oppressing them, I don’t see this as much different. Given the continual and steady erosion of human agency and self determination and the fact that these entities are mostly mindless and driven only by hunger (profit), it seems like we should be taking this a bit more seriously. Yes I know in this case only a few thousand people lost their ability to communicate with other humans at a distance for a short time, it could have been much worse, but this is happening everywhere and the consequences in aggregate are massive.

                                1. 3

                                  Doesn’t the callout of the biologist directly contradict the premise of the post? “Calling 4 scikit-learn functions on a tiny dataset” sounds like “boring machine learning” to me, yet he’s being criticized in this post which ostensibly promotes boring machine learning.

                                  Is the problem that that application of ML is too boring for the author? That boring ML must simultaneously be cutting-edge in order to be worthy of praise?

                                  1. 5

                                    The article calls out people that hailed him as a genius-level polymath, not Horvath himself. Having said that I think being able to navigate academia, have an expert understanding of biology, and also have a decent grasp of machine learning is pretty close to being a genius level polymath :-p

                                  1. 4

                                    I think that the human-level intelligence is also partially a way to enable more people to “access” APIs, especially the people with less affinity towards technology.

                                    Just as an example, consider ordering a pizza. If you are able to order a pizza via a website I have never not seen the option to add custom text regarding the order. You simply will have people that do not have their needs met by the ordering mask or do not find the option they really want. In this case it can make sense to have a computer attempt to decipher the customers wish.

                                    This could also be applied to ordering via phone, which is still very common in Germany. This would be another avenue where voice recognition could make sense and improve efficiency. But ultimately, those problems are intricately connected to UX and not necessarily to fancy ML models, too.

                                    In the end these types of ML models reach more people and it’s important that they do not feel left behind, which can cause rejection and opposition to modern technology. That is a very social problem and I am not convinced that there is a technological solution to it.

                                    1. 3

                                      I think I agree with this, it might be that the “left behind” angle is one I don’t consider enough when thinking about this, ultimately the vast majority of people, can’t use google maps, heck, even many professionals like cab drivers which would benefit immensely from it.

                                      I guess I’m not sure how much of this can be fixed by “education” once technology becomes to useful to be ignored, but it’s the above google map example that made me hedgy on this, after all it’s a prime example of a life-transforming technology that many people just ignore.

                                      Maybe in some cases it’s just more fun to live life without technology? I’ve heard this argument made by literate people that preferred using paper maps and compass navigation just because it was exciting. But under that scenario I’m not sure simpler UX would help, or that any help is needed per se

                                      1. 8

                                        I have a personal theory that many people firmly believe, in a deep, maybe even unconscious way, that they are entitled to stop learning new things at some point in their lives. It’s something I’ve been noticing more and more with the pandemic cutting off so many “non-tech” ways of doing things.

                                        1. 3

                                          I have noticed this too. It really pains me. I personally favour the hypothesis that our education systems are to blame. All humans start off with curiosity as a primary motivator, perhaps the primary motivator after survival stuff. It seems there is a strong correlation between attending a few years of school and this no longer being a strong motivator for many people. As far as causation goes there are reasons to favour this hypothesis - the degree to which curiosity is actively suppressed in most education systems, and also the degree to which education is made to be associated with bland, boring and generally unpleasant hard work, rather than discovery. Unfortunately there is not a lot of good data regarding well-educated individuals that have not been subjected to a classical western style education. There are alternative systems like Montessori but there is no clear way to quantify cognitive ability, we use western school performance for that in most instances. The Finnish school system is an interesting case study though.

                                          This may all seem off topic, but to tie it in to the main theme: It is actually trivial to create a human level intelligence. Many people do it by accident when they are drunk. This makes creating them artificially doubly redundant. I think that parallel to improving inhuman AI as the article suggests, we should be looking into improving the human level intelligence we have easy access to, which is mainly a software problem. Most of the software that is widely available for the human brain is decades out of date and designed for applications that are no longer needed.

                                          1. 1

                                            To some extent though this might be a good conservative instinct.

                                            Don’t get me wrong, I like being in some “privileged” 0.01% of the world’s population that knows the technology well enough to make 6 figures writing stuff on a laptop, I’m sure most people here feel the same way.

                                            But if I had to choose between being in the top 20% or in the bottom 20% I’d choose to bottom.

                                            Technology very quickly becomes a tool that most people end up using to fulfil destructive patterns more than anything.

                                            That’s not to say it has no uses, it does, but it’s way too easy to get into pitfalls if you spend too much time on computers and don’t understand how good certain programs are at manipulating your preferences.

                                            1. 1

                                              I’ve noticed it myself with myself. After 30 years in the computer industry, I’m just sick and tired of having to relearn the hot new development methodology. Again. For the umteenth time. Can’t things just settle down for a few years? It seems as if the singularity has hit and no one can keep up.

                                            2. 3

                                              I suppose to some people it is more fun. I write my day-to-day plans still on paper because I prefer it to electronic bookkeeping. The battery of a sheet of paper cannot run out, although my phone hardly does either these days. But there still is something that makes me prefer to write them by hand, it also helps me remember what I actually wrote down. To some extent it is also control over the medium; I know that I am responsible if I lose it or it somehow is destroyed. I guess the aspect of control and being in charge can also translate to navigation or in general other aspects of life (although I have to agree with you, navigating via online maps is much better).

                                              Potentially education could help with furthering usage of technology. But it could also be that most people are just stubborn and want to continue with their way of life indefinitely. Depending on how big that share of people is, it is important to make new non-human intelligence backwards compatible, so to say. Then once most people stop relying on the old technology (die out?) it can be deprecated.

                                          1. -3

                                            The article forget to describe prior art, except Mathematica, Wolfram Alpha, and Google. Further research will yield more tools like conceptnet.io (that scrape wiktionaries), dbpedia (that scrape wikipedia infoboxes), further wikidata side to make wikidata easier to query https://query.wikidata.org/querybuilder/, also there is Marie Destandau work ergonomic SPAQRL federated query builder, and mediawiki $10M project dubbed abstract wikipedia. There is also the work on wikidata Q/A aka. https://hal.archives-ouvertes.fr/hal-01730479/document.

                                            The article did not mention the fuzzy situation regarding the licensing terms wiktionary and wikipedia vs. wikidata and more broadly the fuzzy legal framework.

                                            More fundamental problem that the OP @zverok seems to have no clues about: extracting structured data from unstructured or semi-structured data like html or plain text in the general case is a very hard problem and possibly AI-Complete.

                                            So, yes I agree given wikipedia and wiktionary, and the fact they are mainstream well-established, eventually they became user-friendly, it would have been better for wikidata to bet on extracting structured RDF triples from those, and invest into making automated and semi-automated approach to extract structured data from unstructured data, merely for legal reason, mediawiki C level want wikidata to be CC0.

                                            Also, I want to stress that ML mainstream frenzy shadowed one of Google tool of choice: freebase, and its related tools and practices.

                                            We need to parse our civilization’s knowledge and make it programmatically available to everybody. And then we’ll see what’s next.


                                            On a related note: I stopped giving money to wikimedia.

                                            1. 7

                                              The article forget to describe prior art

                                              That’s not a scientific article, rather a blog post about some problems I am thinking about and working upon. I am aware (at least of some) of the “prior art” (well, since I am playing with the problems at hand for 6 years now, my “links.txt” is a few books worth). I honestly don’t feel obliged to mention everything that is done in the field unless my work is inspired by/related to others’ work. Yes, a lot of people do a lot of stuff, some of it is dead-ends, some of it is fruitful, and I have the highest respect for them, but being just a developer I am, I am just doing what seems interesting and writing about it, no more, no less.

                                              More fundamental problem that the OP @zverok seems to have no clues about: extracting structured data from unstructured or semi-structured data like html or plain text in the general case is a very hard problem and possibly AI-Complete.

                                              What makes you think that person stating that spent many years on a problem “has no clues about” one of the most obvious aspects of it? The problem is obviously unsolvable “generically” (e.g. “fetch any random page from Internetz and say what information it contains”), and that’s the exact reason I am investigating approaches to solve it for some practical purposes—in a somewhat generic way.

                                              1. 5

                                                The problem is obviously unsolvable “generically” (e.g. “fetch any random page from Internetz and say what information it contains”)

                                                People saying a problem is AI-complete is a big red flag for me in terms of taking the rest of what they say seriously. I am reminded of the famous Arthur C. Clarke quote:

                                                If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong.

                                                The issue is not that there aren’t AI-complete problems. It is not even that this problem isn’t AI-complete. The point is that the AI-completeness is just not a relevant criterion for evaluation a project that aims to be used by humans. Language translation is also AI complete but I have heard some people use google translate…

                                                An AI-Complete problem is by definition achievable by human intelligence (if not it is simple called an impossible problem) and therefore tools that help humans think, remember or formulate things can help with all of them.

                                                Also any time you take an AI-complete problem and reduce it to a finite and programmatically defined domain it ceases to become AI-complete. I remember when I used to try to convince people that real language AI should be part of computer games (I have given up now). This was people’s standard reply. But it is simply not true. AI understanding real human language in every possible case in the real world is AI complete. AI understanding and generating basic descriptive sentences in a single non-evolving language, in a world that is small and has well defined boundaries (both in actual size and also in complexity), and where every aspect of the world is programmatically defined and accessible to the AI is not even really AI-hard, it is merely a quite large and complicated software problem that will take careful planning and a lot of work.

                                                There is a long history of people saying “AI will never be able to do X” and being completely wrong. The people working on AI don’t really listen though.

                                              2. 4

                                                More fundamental problem that the OP @zverok seems to have no clues about

                                                That’s quite harsh. Was this really necessary in this context?

                                                1. 1

                                                  I did not mean to be harsh \cc @zverok, sorry! I agree with the other threads, whether it may be AI-Complete or not, I like the project, it is an interesting project, possibly difficult and possibly with low-hanging fruit. The tone of my comment was not the one it meant to have, I was trying to draw a picture of existing, and some time far fetched related projects, so that someone (OP or else) can jump in more easily, in a subject that interests me a lot, and imo that deserves more attention!

                                                  Re @Dunkhan, the whole comment is interesting, I quote one part:

                                                  AI understanding and generating basic descriptive sentences in a single non-evolving language, in a world that is small and has well defined boundaries (both in actual size and also in complexity), and where every aspect of the world is programmatically defined and accessible to the AI is not even really AI-hard, it is merely a quite large and complicated software problem that will take careful planning and a lot of work.

                                                  +1, hence the importance to review prior art, whether one is a science officer or a hobbyist, and put on continuum what-is-done vs. what-is-impossible, and try to find a new solution for “it for some practical purposes—in a somewhat generic way”.

                                                  I am eager to read a follow up article on the subject.

                                                  Again sorry for the tone of my comment. I repeat that, for sure, I am clueless about what OP is clueless about. I only meant to do common good.

                                              1. 4

                                                What are the disadvantages of Wikidata here? I’ve made SPARQL queries against Wikidata and have gotten similar behavior to what the Mathematica example was showing. Wikidata is certainly still incomplete compared to Wikipedia, but I think it’s fairly feasible to programmatically bridge the two.

                                                1. 6

                                                  That’s a good question I answer every time I am talking about this project/set of projects (so maybe the next blog post will be dedicated to it).

                                                  First of all, I am not rejecting the Wikidata (the “reality” project used it, too, alongside Wikipedia and OpenStreetMap) But currently, Wikipedia has two huge advantages: a) just more content (including structured content), and b) the path to content is more discoverable for “casual” developer/data users.

                                                  On “more content”: look for example at Everest’s entities in Wikipedia and Wikidata. The former has many interesting structured and semi-structured tables and lists not represented (yet?) in the latter (like “Selected climbing records” or even “Climate” (neat and regular table present in many geographic entities); in addition to unstructured text data which still can be fruitfully regex-mined.

                                                  Or, look at, linked from the Wikidata item, another item list of 20th-century summiters of Mount Everest… and corresponding Wikipedia article.

                                                  On “discoverability”: if we’ll look at, say randomly, Bjork at Wikipedia and Wikidata, the latter does have albums associated, but not the filmography (movies Bjork starred in). If you’ll start from the movie you can see they are linked via cast member predicate, so “all movies where Bjork is a cast member” can be fetched with SPARQL, but you need to investigate and guess the data is there; in Wikipedia, most of the people-from-movies just have “Filmography” section in their articles.

                                                  That being said, playing with Wikidata to make it more accessible is in the scope of my studies definitely :)

                                                  1. 4

                                                    If you can extract the data from Wikipedia programmatically then it sounds as if it would then be quite easy to programmatically insert it into Wikidata. I’d love to see tooling for this improved. I hope that eventually Wikipedia will become a Wikidata front end so any structured data added there is automatically reflected back.

                                                    1. 3

                                                      Yeah, just posted the same here: https://lobste.rs/s/mbd6le/why_wikipedia_matters_how_make_sense_it#c_kxgtem (down to the front/back wording :))

                                                  2. 2

                                                    Maybe we can kill two birds with one stone here. What is the chance that a project like this could actually be used to assist wikidata contributors? I don’t think it would be great to attempt to just programatically fill out wikidata like this, but having a nice tool where you can produce wikidata content for a specific wikipedia entry and then look over it and manually check it would save a lot of time for contributors.

                                                    I also feel like the wikidata approach is slightly flawed. Wikipedia collects its data by leveraging the combined knowledge of all humans (ideally) in a very accessible way. If we have a common data system that tries to collect the same data, but in a much less accessible way and only from trained data professionals we have lost a huge amount of utility. Especially given that the less accessible approach means way fewer eyeballs looking for errors.

                                                    1. 2

                                                      Maybe we can kill two birds with one stone here. What is the chance that a project like this could actually be used to assist wikidata contributors?

                                                      Bingo! One of my hopes is indeed that the new tool (once matured a bit) can be used fruitfully both on Wikipedia (to analyze articles structure) and Wikidata (to propose missing data on basis on Wikipedia)

                                                      I also feel like the wikidata approach is slightly flawed. Wikipedia collects its data by leveraging the combined knowledge of all humans (ideally) in a very accessible way. If we have a common data system that tries to collect the same data, but in a much less accessible way and only from trained data professionals we have lost a huge amount of utility. Especially given that the less accessible approach means way fewer eyeballs looking for errors.

                                                      That’s totally true. My (distant) hope is that Wikipedia/Wikidata might once become coupled more title: with Wikipedia being the “front-end” (both more friendly for the reader and for the contributor) with Wikidata the “back-end” (ensuring the formal structure of important parts). But there is a long road ahead, even if this direction is what Wikimedia wants.

                                                      1. 1

                                                        This was exactly where I was going with my reply in fact. Cheers.

                                                    1. 2

                                                      If there is someone here who speaks lawyer, what is the situation with distribution of bug fixes and patched binaries? It seems clear that this ruling allows me to fix bug on my software on my machine but can I post the fix online? Can I sell the fix to other license holders?

                                                      1. 8

                                                        I know this is off topic a little, but I always hated the term ‘Social Distance’. It is simply incorrect. What it refers to is physical distance. Social distance is when you stop answering your phone and emails and just ghost everyone. That behaviour is to be avoided during a pandemic not encouraged. Not a criticism of the article though, the common usage is clear.

                                                        1. 2

                                                          True. How we use “social distancing” during the pandemic is odd when you stop and think about it.

                                                          1. 1

                                                            This was realized almost immediately and the WHO IIRC recommended “physical distancing” instead.

                                                            It didn’t catch on.

                                                            Similarly I believe “lockdown” as a term originates from the phenomenon of “active shooter situations”. Orwell would have had a field day with this.

                                                          1. 2

                                                            Couldn’t you just use the existing system but allow operators to share the same precedence numbers so that some combinations are ambiguous? The solution here seems to have a combinatorial explosion of complexity if you increase the number of operators.

                                                            1. 2

                                                              How does that work? If you say a + b - c is ambiguous because + an - share the same precedence people won’t like that.

                                                              1. 1

                                                                How does that work in the current system? If they have different numbers then I would expect a + b - c or a - b + c to have an error message but I don’t see one. The must be some notion of commutativity built in.

                                                                How is this dealt with in the solution proposed by the article? Maybe I am a bit dense but I can’t see any examples there that answer this. Are + and - ambigous or do they have a defined precedence?

                                                                1. 1

                                                                  + and - are left-associative (if we ignore precedence, then the issue here is associativity, not commutativity). Your grammar may have a rule like:

                                                                  additive-expression ::= term | additive-expression [‘+’ | ‘-’] term

                                                                  Given ‘a + b - c’, we have a top-level additive expression comprising the term ‘c’, the operator ‘-’, and another additive expression; the latter comprises the term ‘b’, the operator ‘+’, and another additive expression consisting only of the term ‘a’.

                                                                  In other words, + and - are specified by the same grammar rule, and it is defined to be left-associative, such that a full parenthesization of ‘a + b - c’ is ‘(a + b) - c’.

                                                                  (Floating-point issues aside, addition is associative, such that even if we had redundant syntactical rules, ‘a + b + c’ would be semantically unambiguous—we will get the same answer no matter which parse we choose. The same does not apply to subtraction. So really the same problem applies to ‘a - b - c’; is that ‘(a - b) - c’ or ‘a - (b - c)’?)

                                                                  1. 2

                                                                    Thanks for the clarification. So (bear with my lack of understanding of operator precedence systems) that means that in the current norm of most programming languages, + and - are both left associative and are treated as having the same precedence, so a - b - c is in fact processed as (a - b) - c. Is that correct? I think I also see the flaw in my initial question. If bitwise expressions are ambiguous with multiplicative, and also with additive expressions, then all three would need the same precedence. This would mean that additive and multiplicative are also ambiguous which is obviously wrong.

                                                                    I stand corrected.

                                                                    1. 1

                                                                      Proof by contradiction is a very strange way to convince yourself of something, but if it works for you… :)

                                                            1. 5

                                                              I agree with all but the first. While debugging I have often found myself going through code and moving debug log statements from after to before. The reason is that if there is a catastrophic failure you want to know what was being attempted at the time.

                                                              If you have something like

                                                              initialising content service
                                                              getting image1
                                                              initialising config service
                                                              -- crash -- 

                                                              you know what module to look at, but if it is just

                                                              initialised content service
                                                              got image 1
                                                              -- crash -- 

                                                              then it is difficult to diagnose. Was there an image2? What module is loaded after content service?

                                                              Before people respond with the usual ‘you should have a stack trace’ or ‘if you are logging properly you will know anyway’ the times when I have needed this are when working with large legacy code-bases or trying to extend very large complex software that is opaque and badly documented in ways that were not envisioned by the original developers (read game modding). Yes in a perfect world you wouldn’t ever be left with a log that just ends without warning, or a totally opaque and incomprehensible error message. But if we lived in a perfect world we wouldn’t need logs for debugging because we would have no bugs.

                                                              With regard to the reason given in the article for this rule, “When reading it, you will not know if the REST call was successful or not” - If it is important to know if an action succeeded you should have an explicit succeeded message in the log in addition to the general progress reporting message. In the example from the article, if the REST call fails you will not get any output from that line if an exception is thrown, meaning you not only don’t know if the call succeeded, you don’t even know if the call was attempted. If no exception is thrown you will get “Made request to REST API” which does not tell you it failed.

                                                              1. 1

                                                                The author does mention that they don’t enforce this rule for debug logging.

                                                              1. 7

                                                                The best any static type checking will let you do is array[float]

                                                                Can’t Idiris at least do the “Monotonically Increasing Float” part?

                                                                1. 9

                                                                  It can do the whole thing, because the dependent type system is capable of end-to-end formal verification. Any FM coding language can verify properties statically, even if it’s not in the type system, so you can do this in HOL4, SPARK, Dafny, etc. It’s just balls-hard (because formal verification is balls-hard in principle).

                                                                  1. 12

                                                                    It can do the whole thing, because the dependent type system is capable of end-to-end formal verification.

                                                                    This effectively moves the complexity of expressing your invariants from one place (like tests) to another (a type system extensible by arbitrary code).

                                                                    Type systems only feel magical as long as they stay simple and declarative. Which necessarily keeps them limited.

                                                                    1. 3

                                                                      Type systems only feel magical as long as they stay simple and declarative.

                                                                      I wholeheartedly agree with this statement, but I doubt you’ll find any pair of programmers that agree on what’s simple and declarative. I bet if you asked around, you’d find that any point in the entire spectrum will be picked by someone as the sweet spot.

                                                                    2. 2

                                                                      Last I checked most of these end-to-end FV attempts require a lot of human intervention (i.e. a human has to prove the properties and encode it in the coding language). Has that changed?

                                                                      1. 10

                                                                        Nope, still sucks and I still hate it

                                                                        (It’s gotten better in the past 6 years as people got better at SMTing all the things, but you still need a lot of human intervention to get anything interesting verified)

                                                                        1. 1

                                                                          Lean has some cool machine-assisted tactics like suggest and library_search that will offer help to find a theorem or proposition that will complete a proof, but these are pretty brute and still often require a lot of work. They’re hit-or-miss but when they hit it’s like magic.

                                                                    3. 9

                                                                      Every half-serious statically typed language can create new types kilometers and miles from float. Then you won’t confuse array[kilometers] and array[miles] and possibly prevent $330 million dollars from being almost totally wasted (https://en.wikipedia.org/wiki/Mars_Climate_Orbiter).

                                                                      That’s quite a bit better than array[float]. And you don’t need to use a research language to do it, either.

                                                                      1. 3

                                                                        Mars climate orbiter is a good example of why the phrase ’‘should be caught in testing” is not sufficient justification for not using additional checks and fail-safes. I assume NASA LM’s testing regimes are fairly rigorous (correct me if there is evidence to the contrary) but in some applications you need another layer of checking. Having said that I also assume that the software in question did have a type system, and it simply wasn’t used in the way you describe.

                                                                        1. 14

                                                                          The type system couldn’t help in this case: the data in US units (pound-seconds) which should have been in metric units (Newton-seconds) was read from a file on disk. The software reading the data from that file thought it was metric, but it wasn’t. Perhaps the file format should have included the units? But the file format as specified was only ever supposed to contain Newton-seconds, so why would they include the units?

                                                                          Anyway, it was an interface problem, rather than a type system problem. See Mars Climate Orbiter Mishap Investigation Board Phase I Report

                                                                          1. 3

                                                                            In a way, the interface problem was a type system problem, but perhaps one we aren’t well equipped to solve across systems.

                                                                            One could imagine attaching typed schemas to certain files, and writing language bindings that enforced that stored values must be strongly typed, with explicit units.

                                                                            1. 3

                                                                              Seems like modern type systems don’t solve this problem as c– mentioned. It is certainly true that no error checking method is 100% foolproof. My main point above was merely that testing isn’t either, and having multiple independent error checking systems is a good idea. Type systems are useful but should not be relied upon.

                                                                              1. 1

                                                                                Absolutely. It even goes beyond merely having multiple independent error checking systems: you have to have the processes, or organizational culture, in place to act accordingly. It’s interesting that the Mars Climate Orbiter problem was actually detected multiple times before the disaster occurred, but no-one quite took it seriously enough.

                                                                      2. 3

                                                                        This misses the payoff of things like type-classes, where you can get the language to provide implementations of all sorts of things. A simple example is deriving lexicographic ordering for tuples if the type of each member is ordered. As much as I enjoy editing in a sexp-aware editor, the idea that static types only constrain you is very outdated thinking.

                                                                        1. 2

                                                                          Even dispensing with sophisticated types, having a smart constructor that might fail at runtime but on success returns a wrapper type which guarantees it’s an “array of numbers with a mean value of such and such and a standard deviation of such and such” can help avoid many mistakes.

                                                                        1. 2

                                                                          Maybe it is me missing something in this whole thread (seems the most likely explanation), but why are we talking about randomness and determinism as if they are mutually exclusive, when in fact all random algorithms are deterministic. As long as you apply the randomness correctly, you should be able to save the randomiser seed and then play back the exact sequence of numbers again at any time in the future, enabling each test to be repeated exactly to debug failures, while still having novel, randomised data and properties on each test run.

                                                                          According to the mathematical definitions there are no actual random algorithms in programming, only deterministic algorithms with unpredictable inputs. If you save the input used to generate the random values you can repeat the ‘random’ behaviour every time with never any deviation. Seems like based on this if people object you can simply take your current random algorithm and say ‘don’t worry, this is deterministic’.

                                                                          It may be theoretically possible in modern processors to implement an algorithm that makes use of quantum tunnelling to generate random values, but processors are set up to avoid tunnelling and it would probably be horribly resource intensive and generate garbage values.

                                                                          Seems like next time someone says your randomised tests need to be deterministic you can just say ‘Don’t worry, the algorithm is completely deterministic’

                                                                          I have a sneaking feeling that I am in fact misunderstanding this whole conversation and that the word determinism in this context means ‘always the same’.

                                                                          1. 4

                                                                            why are we talking about randomness and determinism as if they are mutually exclusive, when in fact all random algorithms are deterministic

                                                                            Random and deterministic are mutually exclusive. A random algorithm depends on a source of entropy, which is by definition not deterministic. A pseudorandom algorithm depends on an approximation of entropy that is unpredictable (in the ideal case) without knowing a seed value. Most computers now provide a hardware random number generator that uses some free-running ring oscillators and generates a sequence of bits that are either 1 or 0 depending on some quantum properties at the gate level that are either random or our understanding of physics is more wrong than we think it is. OS random number generators typically use this as one of the inputs of something like Fortuna (which is designed to be able to merge multiple entropy sources without knowing the quality of the entropy and still give cryptographically secure random numbers).

                                                                            The information-theoretic definition of a random sequence is one that cannot be generated by a program that is shorter than the sequence. Most random number generators come very close to this: you cannot predict the sequence without encoding the PRNG algorithm and all of the entropy measurements, so as long as you query less often than you add entropy they meet the definition and if you don’t then the difference between the theory and practice matters only in theory.

                                                                            All of that said, I agree that we need to be careful about terminology, because a random algorithm can be fed a fake entropy source to give deterministic output. This can be incredibly useful for exploring a large state space but still being able to debug test failures.

                                                                            1. 2

                                                                              Most computers now provide a hardware random number generator that uses some free-running ring oscillators and generates a sequence of bits that are either 1 or 0 depending on some quantum properties at the gate level

                                                                              How can I access this as a programmer and how can I find out if a given random function is using this feature? The first example I looked up (https://docs.microsoft.com/en-us/dotnet/api/system.random?view=net-5.0) is not using it. What languages have support for this?

                                                                              Also, I feel like this is only useful in cryptography. Using a non-deterministic true random has the advantage of being opaque and non-reproducible, I see no advantage of being opaque in unit testing. The other use-case for me for random generators is procedural content generation in games and virtual worlds, where opaqueness is also more of a downside than a desired feature.

                                                                              1. 3

                                                                                How can I access this as a programmer and how can I find out if a given random function is using this feature?

                                                                                On x86, the RDRAND instruction reads from this. On most *NIX systems it’s one of the things that seeds a cryptographically secure random number generator exposed via /dev/random or similar. It’s generally recommended to use this rather than using the random number generator directly because there are some theoretical supply chain attacks on the hardware random number generator that are undetectable in practice. Before being exposed to users, the entropy is fed through a ‘whitening’ process that guarantees a uniform distribution. It’s possible to build one of these that uses a secret key that, if known to an attacker, makes it possible to guess the next random number with fairly high probability (on the order of 1/2^16) but which provides a completely uniform output that is unguessable if you don’t know the secret key. Filtering through Yarrow, Fortuna, or similar avoids this.

                                                                                As far as telling whether a given random function uses this, you can do that only by reading the documentation. Detecting whether an entropy source is a true random source is not computable in the general case. You can do some statistical sampling that will tell you if something has a good distribution but you can’t tell if it’s the output of a purely deterministic PRNG as part of black-box testing.

                                                                                What languages have support for this?

                                                                                C? I’d be pretty shocked if any higher-level language didn’t provide an interface to seed a PRNG with an OS-provided entropy source.

                                                                                Also, I feel like this is only useful in cryptography

                                                                                Or anything where your security (including availability) depends on a truly unguessable random number. For example, TCP sequence numbers, UUIDs, and so on.

                                                                                Using a non-deterministic true random has the advantage of being opaque and non-reproducible, I see no advantage of being opaque in unit testing

                                                                                Completely agreed. Reproducibility is the most important constraint for a test suite, random numbers should not be used. Pseudo-random number generators are a great fit for this.

                                                                                The other use-case for me for random generators is procedural content generation in games and virtual worlds, where opaqueness is also more of a downside than a desired feature.

                                                                                Also agreed. You almost always want a PRNG for these. You might want to seed it with a random number, but you often want to record it for later. Elite was the first game I’m aware of to really take advantage of this by generating 255 galaxies procedurally on systems with 32 KiB of RAM (not enough to hold all of the galaxies in memory at the same time), one for each of the possible seeds to their 8-bit PRNG.

                                                                          1. 26

                                                                            There are a lot of extensions that automatically select the ‘reject all’ or walk the list and decline them all. Why push people towards one that makes them agree? The cookie pop-ups are part of wilful misinterpretation of the GDPR: you don’t need consent for cookies, you need consent for tracking and data sharing. If your site doesn’t track users or share data with third parties, you don’t need a pop up. See GitHub for an example of a complex web-app that manages this. Generally, a well-designed site shouldn’t need to keep PII about users unless they register an account, at which point you can ask permission for everything that you need to store and explain why you are storing it.

                                                                            Note also that the GDPR is very specific about requiring informed consent. It is not at all clear to me that most of these pop-ups actually meet this requirement. If a user of your site cannot explain exactly what PII handling they have agreed to then you are not in compliance.

                                                                            1. 4

                                                                              Can’t answer this for other people, but I want tracking cookies.

                                                                              When people try to articulate the harm, it seems to boil down to an intangible “creepy” feeling or a circular “Corporations tracking you is bad because it means corporations are tracking you” argument that begs the question.

                                                                              Tracking improves the quality of ad targeting; that’s the whole point of the exercise. Narrowly-targeted ads are more profitable, and more ad revenue means fewer sites have to support themselves with paywalls. Fewer paywalls mean more sites available to low-income users, especially ones in developing countries where even what seem like cheap microtransactions from a developed-world perspective would be prohibitively expensive.

                                                                              To me, the whole “I don’t care if it means I have to pay, just stop tracking me” argument is dripping with privilege. I think the ad-supported, free-for-all-comers web is possibly second only to universal literacy as the most egalitarian development in the history of information dissemination. Yes, Wikipedia exists and is wonderful and I donate to it annually, but anyone who has run a small online service that asks for donations knows that relying on the charity of random strangers to cover your costs is often not a reliable way to keep the bills paid. Ads are a more predictable revenue stream.

                                                                              Tracking cookies cost me nothing and benefit others. I always click “Agree” and I do it on purpose.

                                                                              1. 3

                                                                                ‘an intangible “creepy” feeling’ is a nice way of describing how it feels to find out that someone committed a serious crime using your identity. There are real serious consequences of unnecessary tracking, and it costs billions and destroys lives.

                                                                                Also I don’t want ads at all, and I have no interest in targeted ads. If I want to buy things I know how to use a search bar, and if I don’t know I need something, do I really need it? If I am on a website where I frequently shop I might even enable tracking cookies but I don’t want blanket enable them on all sites.

                                                                                1. 4

                                                                                  How does it “costs billions and destroys lives”?

                                                                                  1. 2

                                                                                    https://www.ftc.gov/system/files/documents/reports/consumer-sentinel-network-data-book-2020/csn_annual_data_book_2020.pdf see page 8. This is in the US alone and does not take the other 7.7b people in the world into account. I will admit it is not clear what percentage of fraud and identity theft are due to leaked or hacked data from tracking cookies so this data is hardly accurate for the current discussion, but I think it covers the question of ‘how’. If you want more detail just google the individual categories in the report under fraud and identity theft.

                                                                                    Also see this and this

                                                                                    But I covered criminal prosecution in the same sentence you just quoted from my reply above so clearly you meant ‘other than being put in prison’. Also, people sometimes die in prison, and they almost always lose their jobs.

                                                                                    1. 4

                                                                                      The first identity theft story doesn’t really detail what exactly happened surrounding the ID theft, and the second one is about a childhood acquaintance stealing the man’s ID. It doesn’t say how exactly either, and neither does that FTC report as far as I can see: it just lists ID theft as a problem. Well, okay, but colour me skeptical that this is cause by run-of-mill adtech/engagement tracking, which is what we’re talking about here. Not that I think it’s not problematic, but it’s a different thing and I don’t see how they’re strongly connected.

                                                                                      The NSA will do what the NSA will do; if we had no Google then they would just do the same. I also don’t think it’s as problematic as often claimed as agencies such as the NSA also do necessary work. It really depends on the details on who/why/what was done exactly (but the article doesn’t mention that, and it’s probably not public anyway; I’d argue lack of oversight and trust is the biggest issue here, rather than the actions themselves, but this is veering very off-topic).

                                                                                      In short, I feel there’s a sore lack of nuance here and confusion between things that are (mostly) unconnected.

                                                                                      1. 2

                                                                                        Nevertheless all this personal data is being collected, and sometimes it gets out of the data silos. To pretend that it never causes any harm just because some stranger on the internet failed to come up with a completely airtight example case in 5 minutes of web searching is either dishonest or naive. If you really want to know, you can do the research yourself and find real cases. If you would rather just feel comfortable with your choice to allow all tracking cookies that is also totally fine. You asked how, I believe my answer was sufficient and roughly correct. If you feel the need to prove me wrong that is also fine, and I will consider any evidence you present.

                                                                                        1. 2

                                                                                          The type of “personal data” required for identity theft is stuff like social security numbers, passport numbers, and that kind of stuff. That’s quite a different sort of “personal data” than your internet history/behaviour.

                                                                                          To pretend that it never causes any harm just because some stranger on the internet failed to come up with a completely airtight example case in 5 minutes of web searching is either dishonest or naive. If you really want to know, you can do the research yourself and find real cases.

                                                                                          C’mon man, if you’re making such large claims such as “it costs billions and destroys lives” then you should be prepared to back them up. I’m not an expert but spent over ten years paying close attention to these kind of things, and I don’t see how these claims bear out, but I’m always willing to learn something new which is why I asked the question. Coming back with “do your own research” and “prove me wrong then!” is rather unimpressive.

                                                                                          If you would rather just feel comfortable with your choice to allow all tracking cookies that is also totally fine.

                                                                                          I don’t, and I never said anything which implied it.

                                                                                          If you feel the need to prove me wrong that is also fine, and I will consider any evidence you present.

                                                                                          I feel the need to understand reality to the best of my ability.

                                                                                          1. 1

                                                                                            I feel the need to understand reality to the best of my ability.

                                                                                            Sorry I was a bit rude in my wording. There is no call for that. I just felt like I was being asked to do a lot of online research for a discussion I have no real stake in.

                                                                                            GDPR Article 4 Paragraph 1 and GDPR Article 9 Paragraph 1 specify what kind of information they need to ask permission to collect. It is all pretty serious stuff. There is no mention of ‘shopping preferences’. Social security numbers and passport numbers are included, as well as health data, things that are often the cause of discrimination like sexuality/religion/political affiliation. Also included is any data that can be used to uniquely identify you as an individual (without which aggregate data is much harder to abuse) which includes your IP, your real name.

                                                                                            A lot of sites just ask permission to cover their asses and don’t need to. This I agree is annoying. But if a site is giving you a list of cookies to say yes or no to they probably know what they are doing and are collecting the above information about you. If you are a white heterosexual English speaking male then a lot of that information probably seems tame enough too, but for a lot of people having that information collected online is very dangerous in quite real and tangible ways.

                                                                                    2. 3

                                                                                      I am absolutely willing to have my view on this changed. Can you point me to some examples of serious identity theft crimes being committed using tracking cookies?

                                                                                      1. 2

                                                                                        See my reply to the other guy above. The FTC data does not specify where the hackers stole the identity information so it is impossible for me to say what percentage are legitimately caused by tracking cookies. The law that mandates these banners refers to information that can be used to identify individuals. Even if it has never ever happened in history that hacked or leaked cookie data has been used for fraud or identity theft, it is a real danger. I would love to supply concrete examples but I have a full time job and a life and if your claim is “Sure all this personal data is out there on the web, and yes sometimes it gets out of the data silos, but I don’t believe anyone ever used it for a crime” then I feel like its not worth my time spending hours digging out case studies and court records to prove you wrong. Having said that if you do some searching to satisfy your own curiosity and find anything definitive I would love to hear about it.

                                                                                      2. 2

                                                                                        someone committed a serious crime using your identity

                                                                                        because of cookies? that doesn’t follow

                                                                                      3. 1

                                                                                        Well this is weird. I think it’s easy to read that and forget that the industry you’re waxing lyrical about is worth hundreds of billions; it’s not an egalitarian development, it’s an empire. Those small online services that don’t want to rely on asking for donations aren’t billion-dollar companies, get a deal entirely on someone else’s terms, and are almost certainly taken advantage of for the privilege.

                                                                                        It also has its own agenda. The ability to mechanically assess “ad-friendliness” already restricts ad-supported content producers to what corporations are happy to see their name next to. I don’t want to get too speculative on the site, but there’s such a thing as an ad-friendly viewer too, and I expect that concept to become increasingly relevant.

                                                                                        So, tracking cookies. They support an industry I think is a social ill, so I’d be opposed to them on that alone. But I also think it’s extremely… optimistic… to think being spied on will only ever be good for you. Advertisers already leave content providers in the cold when it’s financially indicated—what happens when your tracking profile tells them you’re not worth advertising to?

                                                                                        I claim the cost to the individual is unknowable. The benefit to society is Cambridge Analytica.

                                                                                      4. 2

                                                                                        The cookie law is much older than GDPR. In the EU you do need consent for cookies. It is a dumb law.

                                                                                        1. 11

                                                                                          In the EU you do need consent for cookies. It is a dumb law.

                                                                                          This is not true. In the EU you need consent for tracking, whether or not you do that with cookies. It has to be informed consent, which means that the user must understand what they are agreeing to. As such, a lot of the cookie consent UIs are not GDPR compliant. Max Schrems’ company is filing complaints about non-compliant cookie banners.

                                                                                          If you only use functional cookies, you don’t need to ask for consent.

                                                                                          1. 3

                                                                                            https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:31995L0046 concerns consent of user data processing.

                                                                                            https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32002L0058 from 2002 builds on the 1995 directive, bringing in “cookies” explicitly. Among other things it states “The methods for giving information, offering a right to refuse or requesting consent should be made as user-friendly as possible.”

                                                                                            In 2009 https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32009L0136 updated the 2002 directive, closing a few loop holes.

                                                                                            The Do-Not-Track header should have been enough signal to cut down on cookie banners (and a few websites are sensible enough to interpret it as universal rejection for unnecessary data storage), but apparently that was too easy on users? It went as quickly as it came after Microsoft defused it by enabling it by default and parts of adtech arguing that the header doesn’t signify an informed decision anymore and therefore can be ignored.

                                                                                            If banners are annoying it’s because they’re a deliberate dark pattern, see https://twitter.com/pixelscript/status/1436664488913215490 for a particularly egregious example: A direct breach of the 2002 directive that is typically brought up as “the cookie law” given how it mandates “as user-friendly as possible.”

                                                                                            1. 2

                                                                                              I don’t understand what you’re trying to say. Most cookie banners on EU sites are not at all what I’d call a dark pattern. They’re just trying to follow the law. It is a stupid law which only trained people to click agree on all website warnings, making GDPR less effective. Without the cookie law, dark patterns against GDPR would be less effective.

                                                                                              1. 3

                                                                                                The dark pattern pgeorgi refers to is that on many cookie banners, the “Refuse all” button requires more clicks and/or more careful looking than the “Accept all” button. People who have trained themselves to click “Accept” mostly chose “Accept” because it is easier — one click on a bright button, and done. If “Refuse all” were equally easy to choose, more people would train themselves to always click “Refuse”.

                                                                                                Let’s pretend for a moment the cookie law no longer exists. A website wants to set a tracking cookie. A tracking cookie, by definition, constitutes personally identifiable information (PII) – as long as the cookie is present, you can show an ad to specifically that user. The GDPR recognizes 6 different conditions under which processing PII is lawful.

                                                                                                The only legal ground to set a tracking cookie for advertising purposes is (a) If the data subject has given consent to the processing of his or her personal data. I won’t go over every GDPR ground, but suffice it to say that tracking-for-advertising-purposes is not covered by

                                                                                                • (b) To fulfil contractual obligations with a data subject;
                                                                                                • nor is it covered by (f) For the legitimate interests of a data controller or a third party, unless these interests are overridden by interests of the data subject.

                                                                                                So even if there were no cookie law, GDPR ensures that if you want to set a tracking cookie, you have to ask the user.

                                                                                                Conversely, if you want to show ads without setting tracking cookies, you don’t need to get consent for anything.

                                                                                                1. 2

                                                                                                  I feel the mistake with the whole “cookie law” thing is that it focuses too much on the technology rather than what people/companies are actually doing. That is, there are many innocent non-tracking reasons to store information in a browser that’s not “strictly necessary”, and there are many ways to track people without storing information in the browser.

                                                                                                2. 1

                                                                                                  I’m not saying that dark patterns are employed on the banners. The banners themselves are dark patterns.

                                                                                                  1. 1

                                                                                                    The banners often come from freely available compliance packages… It’s not dark, it’s just lazy and badly thought out, like the law itself.

                                                                                                    1. 1

                                                                                                      What about the law do you think is badly thought out?

                                                                                                      1. 1

                                                                                                        The cookie part of the ePrivacy Directive is too technological. You don’t need consent, but you do have to inform the user of cookie storage (or localstorage etc) no matter what you use it for. It’s unnecessary information, and it doesn’t protect the user. These are the cookie banners that only let you choose “I understand”, cause they only store strictly necessary cookies (or any kind of cookie before GDPR in 2016).

                                                                                                        GDPR is the right way to do it. The cookie part of EPR should have been scrapped with GDPR. That would make banners that do ask for PII storage consent stand out more. You can’t make you GDPR banner look like an EPR information banner if EPR banners aren’t a thing.

                                                                                            2. 2

                                                                                              Usually when I see the cookie consent popup I haven’t shared any personal information yet. There is what the site has from my browser and network connection, but I trust my browser, uBlock origin and DDG privacy tools to block various things and I use a VPN to somewhere random when I don’t want a site to know everything it can about my network location.

                                                                                              If I really do want to share personal info with a site, I’ll go and be very careful what I provide and what I agree too, but also realistic in that I know there are no guarantees.

                                                                                              1. 8

                                                                                                If you’re using a VPN and uBlock origin, then your anonymity set probably doesn’t contain more than a handful of people. Combined with browser fingerprinting, it probably contains just you.

                                                                                                1. 2

                                                                                                  Should I be concerned about that? I’m really not sure I have properly thought through any threats from the unique identification that comes from that. Do you have any pointers to how to figure out what that might lead to?

                                                                                                  1. 9

                                                                                                    The point of things like the GDPR and so on is to prevent people assembling large databases of correlated knowledge that violate individual privacy. For example, if someone tracks which news articles you read, they have a good first approximation of your voting preferences. If they correlate it with your address, they can tell if you’re in a constituency where their candidate may have a chance. If you are, they know the issues that are important to you and so can target adverts towards you (including targeted postal adverts if they’re able to get your address, which they can if they share data with any company that’s shipped anything physical to you) that may influence the election.

                                                                                                    Personally, I consider automated propaganda engines backed by sophisticated psychological models to be an existential threat to a free society that can be addressed only by some quite aggressive regulation. Any unique identifier that allows you to be associated with the kind of profile that these things construct is a problem.

                                                                                                  2. 2

                                                                                                    Do you have a recommendation?

                                                                                                2. 2

                                                                                                  The problem with rejecting all the tracking is that without it most ad networks will serve you the worst/cheapest untargeted adverts which have a high chance of being a vector for malware.

                                                                                                  So if you reject the tracking you pretty much have to also run an ad-blocker to protect yourself. Of course if you are running an ad blocker then the cookies arent going to make much difference either way.

                                                                                                  1. 1

                                                                                                    I don’t believe it makes any difference whether you agree or disagree? the goal is just to make the box go away

                                                                                                    1. 2

                                                                                                      Yes. If I agree and they track me, they are legally covered. If I disagree and they track me then the regulator can impose a fine of up to 5% of their annual turnover. As a second-order effect: if aggregate statistics say 95% of people click ‘agree’ then they have no incentive to reduce their tracking, whereas if aggregate statistics say ‘10% leave the page without clicking either, 50% click disagree’ then they have a strong case that tracking will lose them business and this will impact their financial planning.

                                                                                                  1. 10

                                                                                                    I do care about cookies

                                                                                                    1. 3

                                                                                                      You’re in luck then. Soon every single useless site will have a banner :)

                                                                                                      1. 12

                                                                                                        Except the sites that don’t try to track you :)

                                                                                                        1. 4

                                                                                                          My hope is that sites will see their numbers drop from having the banners and rethink the policy of tracking everyone. You can still have analytics without saving identifying information, you can still show content without tracking the user. There is no need for this behaviour. Having said that I don’t have an issue with people blocking the banners, apparently the majority don’t even block ads so I can’t see this becoming widespread enough to damage the incentives.

                                                                                                          1. 2

                                                                                                            What about default Apache servers or WordPress installations? I was under the impression that GDPR technically requires them to have a banner, but I find that ridiculous.

                                                                                                      1. 16

                                                                                                        I work on a team that primarily writes Haskell. We’re part of a small-ish (~20 person) engineering org where most people come in knowing Go or JavaScript or Ruby. We’ve had good (surprising!) success onboarding new engineers to Haskell. Zero to production-ready code review takes around 6 weeks, which isn’t that different from most languages.

                                                                                                        Because of this, I’ve had quite a bit of experience teaching working programmers what monads are, and seen a ton of different approaches. Pedagogically, I would say that “monads as monoids of endofunctors” is probably one of the least effective approaches we’ve tried. There’s just too many concepts here that you need to introduce at the same time. Monoids make sense to most people (programmers write a lot of monoids day-to-day, so it’s easy to motivate a monoid with concrete examples like strings, lists, numbers, or Builders), but endofunctors and categories usually do not. What is a category? What is an example of a category in code I’ve seen before? These questions tend to trip people up because it’s hard to understand something until you’ve played with it for a bit (alas, You Can’t Tell People Anything).

                                                                                                        The approach I’ve found to be most effective is to start from something a working programmer has seen before, and compare and contrast. Fortunately, most programmers usually do have a good starting point! Depending on their native language, this is usually a Promise or a Future or a continuation or a Stream - usually some kind of object that represents a computation, where these objects can be sequenced to construct larger computations. I’ve had great success working with JavaScript programmers and telling them “monads are a lot like then-ables (Promises), and async/await is a lot like do-syntax, and here’s where they’re similar, and here’s how they differ”.

                                                                                                        I think this approach succeeds because it’s much easier to teach a concept by starting from something that someone has actually used before (and by using their existing vocabulary) than by starting from scratch. If I tell someone “monads are values that represent computations that can be sequenced”, I get a lot of blank stares. If I follow that up with “just like Promises!”, I see a lot of lightbulbs go off.

                                                                                                        (This is why I think it’s actually particularly hard to teach applicative functors. Lots of Haskell pedagogy attempts to go in the order of the typeclass hierarchy, teaching functors then applicatives then monads. I think it’s actually more pedagogically effective to skip around, teaching functors then monads then applicatives, because it’s hard to motivate applicatives as “functors containing functions” and easy to motivate applicatives as “weaker monads when you don’t need the power and you want to constrain what the computation can do”.)

                                                                                                        1. 7

                                                                                                          “Monoid in the category of endofunctors” is true statement, but is an old joke that escaped containment.

                                                                                                          Can you elaborate on your async/await comparison? I haven’t been in the JS world for a long time.

                                                                                                          Re: Applicative: I used to help teach the System-F (nee Data61) FP Course, and we seemed to have reasonable success motivating Applicative by trying to map a binary function a -> b -> c over a functor f a and getting “stuck” with an f (b -> c). This motivates Applicative as the typeclass that provides lift0 and lift[2..\infty] functions, if you think of Functor as the typeclass that provides lift1 (as an alternative perspective on fmap).

                                                                                                          1. 10

                                                                                                            map a binary function a -> b -> c over a functor f a

                                                                                                            The hard part about this example is that the immediate response is “why would I ever do this?”. I don’t think I have a compelling answer for that. I’ve found that helping someone answer “why would I ever use X?” for any concept really helps them understand the concept better.

                                                                                                            Can you elaborate on your async/await comparison?

                                                                                                            Sure! TL;DR: async/await is do-syntax for the Promise monad.

                                                                                                            JavaScript has a concept of “Promises”, which are asynchronous computations that will eventually return a value. Promises are more convenient than using callbacks since you avoid the Pyramid of Doom problem (although they trade this off with some annoyances of their own), and most of the JS ecosystem has settled on Promises as a primitive for asynchronous computation. (Fun fact: Node actually had Promise support really early on, then switched to callbacks for most standard library functions, then switched back to Promises! There’s a really interesting aside about this in Platform as a Reflection of Values, a talk by Bryan Cantrill who was there at Joyent at the time.)

                                                                                                            Promises provide a couple primitives:

                                                                                                            • resolve(value: T): Promise<T>, which takes a regular value and “wraps” it into a Promise that immediately provides the value.
                                                                                                            • then(this: Promise<A>, f: A -> Promise<B>): Promise<B>, that takes a Promise<A> and a function A -> Promise<B> and provides you with a Promise<B>. Mechanically, then waits until this has a value ready, and then runs f and returns the result.

                                                                                                            (There are some other primitives for constructing promises from callbacks, synchronizing multiple promises, doing error handling, etc., but they’re not important for this comparison.)

                                                                                                            Why do you have to use then in order to call a function using the Promise’s value? Because the Promise’s value isn’t guaranteed to be there yet! What you’re really saying is “here’s a computation that I want to run once the value becomes available”. (If you know monads, this should start sounding familiar.)

                                                                                                            So programmers usually wind up with a bunch of code that looks like this:

                                                                                                              .then(a -> {
                                                                                                                // Do a bunch of synchronous stuff,
                                                                                                                // then end up doing something
                                                                                                                // asynchronous, and return a Promise.
                                                                                                              }).then(b -> {
                                                                                                                // etc.
                                                                                                              }).then(c -> {
                                                                                                                // etc.
                                                                                                              // This eventually becomes a long chain.

                                                                                                            Later, JavaScript adopted what’s called “await/async” syntax, which is a syntax sugar that allows you to write code like this instead:

                                                                                                            // Some types:
                                                                                                            // somethingThatReturnsAPromise: () -> Promise<A>
                                                                                                            // someOtherAsyncThing: A -> Promise<B>
                                                                                                            // yetAnotherAsyncThing: B -> Promise<C>
                                                                                                            a = await somethingThatReturnsAPromise()
                                                                                                            // Do stuff...
                                                                                                            b = await someOtherAsyncThing(a)
                                                                                                            // etc.
                                                                                                            result = await yetAnotherAsyncThing(b) // result: Promise<C>

                                                                                                            This is a bit clearer and a bit more convenient to write, and basically desugars down to the same thing.

                                                                                                            Now, how is this all related to monads? Well Promises are actually just an instance of monad! (Not really because of implementation details, but they’re monadic in spirit.)

                                                                                                            Let’s look at how they compare. Monad is a typeclass, which basically means it’s an interface (or really more like a Rust trait). (Again not really, but correct in spirit - most of what I’m saying here is going to be not-super-technically correct.) A type is-a monad when it implements that interface. The monad interface basically looks like:

                                                                                                            interface Monad for type T<A> {
                                                                                                              resolve(value: A): T<A>
                                                                                                              then(this: T<A>, f: A -> T<B>): T<B>

                                                                                                            (Haskell’s actual Monad typeclass names these functions return instead of resolve, and bind instead of then.)

                                                                                                            Hold on! Doesn’t that look familiar? Yes, this is almost exactly what Promise provides! Now let’s look at do-syntax:

                                                                                                            // Some types:
                                                                                                            // somethingThatReturnsT: () -> T<A>
                                                                                                            // someOtherThingThatReturnsT: A -> T<B>
                                                                                                            // yetAnotherThingThatReturnsT: B -> T<C>
                                                                                                            result: T<A> // Where T is a monad
                                                                                                            result = do
                                                                                                              a <- somethingThatReturnsT()
                                                                                                              // Do stuff in let-bindings...
                                                                                                              b <- someOtherThingThatReturnsT(a)
                                                                                                              // etc.

                                                                                                            Huh… isn’t that almost exactly async/await? Right again.

                                                                                                            Why is this useful? Two reasons:

                                                                                                            1. It turns out that there are lots of monads. This idea of “give me some computations, and I’ll sequence them together in a certain way” turns out to be very general. In Promise, the “way I sequence computations” is “wait until the asynchronous value is available”. In Maybe, it’s “don’t run the next computations if previous computations didn’t succeed”. In State, it’s “let future computations ask for a value that previous computations have put”. All of these implementations of Monad differ in how they implement then and resolve.
                                                                                                            2. It turns out that you can write functions that are general across every single kind of monads! For example, you can write when or unless or forever or sequence (see Control.Monad for other examples). This makes it easier to reuse code and concepts across different kinds of monads. If you can also write your business logic to work across different kinds of monads, it also makes testing easier: you can use a monad that provides logging by writing to disk in production, and swap that out for a monad that provides logging by saving log messages in memory in testing, all without changing a single line of code in your business logic. Monads are like type-checked, effect-tracked dependency injection! (This idea is its own whole blog post - TL;DR: monads and DI are both about effects, so they wind up being used for similar purposes.)

                                                                                                            Sorry in advance for typos or incoherence - most of this is off of the top of my head.

                                                                                                            1. 2

                                                                                                              Thanks for the detailed post. Is your type signature for then(this: Promise<A>, f : A -> B) incorrect? It should be a Promise<B>, right?

                                                                                                              The hard part about this example is that the immediate response is “why would I ever do this?”

                                                                                                              We’d usually motivate this with the Maybe applicative: apply f to both arguments if both are Just; return Nothing otherwise. Then consider other applicatives and notice how an embarrassingly large amount of imperative programming is subsumed by things you can build out of applicative operations (traverse and sequence in particular).

                                                                                                              1. 2

                                                                                                                Yes, good catch on the typo!

                                                                                                                Hmm, I haven’t thought about teaching applicatives in the context of traverse. That’s an interesting idea, I’ll have to think about that more.

                                                                                                                1. 1

                                                                                                                  Traversable makes it more powerful, but more complicated. Try specialising to the [] Traversable first, if you end up giving it a go?

                                                                                                                2. 1

                                                                                                                  Then consider other applicatives and notice how an embarrassingly large amount of imperative programming is subsumed by things you can build out of applicative operations (traverse and sequence in particular).

                                                                                                                  Could you elaborate on this?

                                                                                                            2. 3

                                                                                                              I understand your approach, and while I think it’s effective, it does have some well-known weaknesses. The most important thing to emphasize is the algebraic laws, but your teams are working in Haskell, which is oblivious to those laws; it is easy for programmers to learn the entirety of the Monad typeclass in practice but never realize the laws. (I was such a Haskeller!)

                                                                                                              What is a category?

                                                                                                              C’mon, really? I know that you’re just playing, but this question really needs to be given a standard answer. A category is a collection of structures plus a collection of transformations from one structure to another. We usually call the structures “objects” and the transformations “arrows” or “morphisms”, but this is merely a traditional note so that readers can read common category-theoretic documentation. The main point of a category is to consider composition of transformations.

                                                                                                              What is an example of a category in code I’ve seen before?

                                                                                                              The following systems are categories which I’ve used in serious code. Each system has a proper title, but this is again merely mathematical tradition. This should not be a surprising collection; computer scientists should recognize all of these components, even if they’ve never considered how they’re organized.

                                                                                                              • Set, whose objects are sets and arrows are functions
                                                                                                              • Rel, whose objects are sets and arrows are relations
                                                                                                              • P, whose objects are sets and arrows are permutations
                                                                                                              • Circ, whose objects are the natural numbers and arrows are Boolean circuits
                                                                                                              • Mat_R, whose objects are the natural numbers and arrows are matrices over some semiring R
                                                                                                              • Eff, whose objects are sets with computable equality of elements and arrows are computable functions
                                                                                                              • Pos, whose objects are partial orders and arrows are monotone maps

                                                                                                              In addition, every programming language gives a category via the native type theory construction. This is actually misleading for Haskellers, since the category Hask is actually bogus, but there is a non-Hask category which accurately describes how Haskell transforms data.

                                                                                                              1. 5

                                                                                                                The most important thing to emphasize is the algebraic laws

                                                                                                                Yes, agreed. Something I’ve found is useful for teaching is to explicitly map the maximally precise terminology that the Haskell community likes to use back to similar, vaguer terminology that working programmers are familiar with.

                                                                                                                For example, I’ve found more success teaching “algebraic laws” as “coding conventions but motivated because look at how useful these equalities are”. Most programmers are familiar with conventions (e.g. don’t do side effects in your toString implementation), but I’ve found a lot of people get hung up on the terminology of what “algebraic” means and what “laws” are.

                                                                                                                A category is a collection of structures plus a collection of transformations from one structure to another.

                                                                                                                The difficulty I’ve had with a definition like this is answering “why is looking at something as a category useful to me as a programmer?”. I’ve found that providing a satisfying answer to this question for any concept (“why is X useful?”) helps tremendously with making the concept stick.

                                                                                                                There are definitely categories in the list of examples you’ve provided that I’ve worked with (e.g. Set, Rel, P) in other languages, but where I’ve been struggling is providing a satisfying answer to “if I could look at this as a category in a programming language I’m familiar with, what would be easier?”.

                                                                                                                It’s been relatively easier for me to answer this question for ideas like effect tracking (“your logging library will never secretly call the network again”) or algebraic data types (“you’ll never have a struct whose fields are in an inconsistent state”), but I haven’t found a satisfying answer for categories yet. If you’ve heard of one, I’d love to use it!

                                                                                                                1. 4

                                                                                                                  A disconnect between (nominally) mathematical and pragmatic perspectives I’ve noticed that seems important to address: people who aren’t conditioned to think as mathematicians do tend to, understandably, relate to mathematical invariants as inscrutable laws passed down from the universe, when in fact they are just as much determined by a criterion of usefulness as any programming construct is. Take the case of linear transformations as a tensor product between vectors and covectors: considered on their own, the only matrices you get from just that operation are singular. Such a purity of definition results in a universe of discourse (singular matrices) that is, frankly, not very useful, so we go on to consider sums of such products so that we can talk about linear transformations with full rank.

                                                                                                                  It’s certainly the case that what counts as “useful” to a mathematician might very well diverge wildly from what counts as useful to a programmer, but it’s been my experience that conveying the understanding that mathematical constructs are chosen as much on the basis of what they can do as are programming constructs is an essential aspect of answering the question of “why is framing this practical concern in terms of a more abstract mathematical construct useful to me as a programmer?”

                                                                                                                  1. 3

                                                                                                                    All that said, though, I still harbor a suspicion that most invocations of category theory in the context of programming are fetishistic and convey little in the way of enhanced understanding of the shape of the problem at hand. The mere fact of dealing with a problem where one is operating on objects within a category and morphisms between them doesn’t at all imply that conceiving of the problem in category-theoretic terms is useful, even in the case of a lone creator who is free to organize their work according to whichever models they see fit, and much less in the case of someone who must work with others (which is to say, someone whose work could ever matter) and develop a shared mental model of their collective project with unquestionably competent people who nonetheless are in general unlikely to share the category-theoretic lens. It’s pretty silly to me to cite the mere fact of any one category’s objects and morphisms being a frequent domain of consideration as evidence of the notion that thinking of that domain in terms of its categorical representation is inherently useful.

                                                                                                                    1. 1

                                                                                                                      If literally nothing else, category theory is tightly related to programming language theory and design via type theory. I’ve explained before how this works for the MarshallB programming language, but it also works for languages like Joy. Categories also show up in computability theory, due to the fact that Turing categories are restriction categories.

                                                                                                                      1. 1

                                                                                                                        I’m absolutely in agreement with you on that, in that category theory provides an extraordinarily useful set of tools for modeling constructive type theories. Again, though, among all the problems that might ever be solved by a Turing-complete symbol system, how many benefit notably from a category-theoretic framing? To be clear, I’m coming at this from the position that cases like Haskell’s Monad typeclass don’t qualify, since they seem to be constructs that are merely influenced by category theory rather than being faithful representations of some category-theoretical construct: consider that return as implemented isn’t actually a natural transformation η in the theoretical sense, because it doesn’t actually map between functors: the identity functor is merely implied or assumed, even though the input is merely an object of Hask that need not be a functor itself.

                                                                                                              2. 1

                                                                                                                Thanks, the description of explaining things in terms of what the programmer already knows helped me a lot. I have been working in c# for years and I knew that I have been using and understanding the use of monads most of that time through linq. But I was having issues translating that to a more abstract definition and overall concept. After reading your comment I just googled ‘monads in c#’ and got this https://mikhail.io/2018/07/monads-explained-in-csharp-again/ which explained to me how what I already new related to the more general concept.

                                                                                                              1. 3

                                                                                                                Hmm 0 references to procedural generation. I know the article is about design but I think it is still an important part of the discussion. Minecraft wasn’t out at the time this was written but Rogue and many of its successors were. Sometimes the problem with virtual world design is too much design, leading to the player being able to predict the world like a tv show with bad writing. Procedural worlds enable designers to focus on things that matter, and ignore the mundane details like placing specific trees in a forest or deciding what a nobody peasant npc’s hair colour should be.

                                                                                                                1. 3

                                                                                                                  Procgen IIRC had a bad rap then. It’s very easy to make games that promise “infinite gameplay” but are hollow because of the lack of design.

                                                                                                                  1. 1

                                                                                                                    That reputation persists to this day in some circles.

                                                                                                                    1. 1

                                                                                                                      Because it often is used as an excuse. Unless you’re like, Derek Yu, making a game that uses procgen is hard, unless the systems are that interesting to deal with. And you will almost certainly never get a good plot out of procgen.

                                                                                                                      1. 1

                                                                                                                        And you will almost certainly never get a good plot out of procgen.

                                                                                                                        I have been wanting to do some experimental work in that direction for years. I can never find the time or resources though. I have a good mental picture of how it could be possible though. Dwarf Fortress lays some interesting groundwork. True most of the plots are either too random and unlikely, or weirdly generic and mundane, but every now and then it hits that sweet spot and you get something really interesting. My plan one day is to try to expand on that principle with the goal of a) reducing the amount of world complexity required (the hard part), and b) optimising to increase the frequency of good plots. It is a really hard problem, but a fascinating one and one that I think will eventually be cracked by someone. Being able to generate plots would be a big deal not just for games but also for AI prose fiction generation, and probably a bunch of other applications.

                                                                                                                  2. 3

                                                                                                                    Some other early procgen games: Elite (1984) and Starflight (1986).

                                                                                                                    Sorry to keep posting links to the Digital Antiquarian, but I find that blog to be a an undersung yet extremely detailed history of gaming that everyone should at least bookmark or browse around in. The Starflight article is actually preceded (click “back” twice) by an entry-level description of Forth, so the reader can appreciate the significance of writing an entire game in it. :)