1. 4

    Empirical Software Engineering: I followed a course at my university on this. It was an eye opener. I can publish some of my reviews and summaries sometime. Let me already give you some basic ideas:

    1. Use sensible data. Not just SLOC to measure “productivity,” but also budget estimates, man-hours per month, commit logs, bug reports, as much stuff you can find.
    2. Use basic research tools. Check whether your data makes sense and filter outliers. Gather groups of similar projects and compare them.
    3. Use known benchmarks in your advantage. What are known failure rates, how can these be influenced? When is a project even considered a success?
    4. Know about “time compression” and “operational cost tsunamis”: these are phenomena such as an increase in the total cost by “death march” projects, and how operational costs are incurred already during development.
    5. Know about quality of estimates, and estimates of quality and how these can improve over time. Estimates of the kind “this is what my boss wants to hear” are harmful. Being honest about total costs allows you to manage expectations: some ideas (lets build a fault-tolerant and distributed, scalable and adaptable X) are more expensive than others (use PHP and MySQL to build simple prototype for X).
    6. Work together with others in business. Why does another department need X? What is the reason for deadline X? What can we deliver and how to decrease cost, so we can develop oportunity X or research Y instead?
    7. Optimize on the portfolio level. Why does my organization have eight different cloud providers? Why does every department build its own tools to do X? What are strategic decisions and what are operational decisions? How can I convince others of doing X instead? What is organizational knowledge? What are the risks involved when merging projects?

    Finally, I became convinced that for most organizations software development is a huge liability. But, as a famous theoretical computer scientist said back in the days: we have to overcome this huge barrier, because without software some things are simply impossible to do. So keep making the trade off: how much are you willing to lose, with the slight chance of high rewards?

    1. 2

      Any books you want to recommend?

      1. 1

        I’m also interested in that but online articles instead of books. The very nature of empiricism is to keep looking for more resources or angles as new work might catch what others miss. Might as well apply it to itself in terms of what methods/practices to use for empirical investigation. Let get meta with it. :)

    1. 14

      If documents and models are unstructured, then consider using an unstructured model. An untyped language is actually typed with a single all-encompassing type. This can be simulated in any sufficiently powerful type system.

      Consider, for example, Gson: it allows conversion into POJOs, thereby making use of existing data structures. But it is also possible to use Gson’s built-in all-encompassing type, and explore and navigate the unstructured JSON directly.

      I don’t see any problem.

      1. 2

        You can also add a variant class to make typed code somewhat typeless, ie. at my work we have a C++ wrapper around jsonc that puts everything into a tree with a variant class for the values that represents a bool, int, float or string. You can create a variant that represents as many values as you want to supports, with run time type safety. You can also assert or debug if the wrong type coercion is attempted.

        1. 1

          To me, it seems like the type system equivalent of the safe vs unsafe debate in languages. The best practice on the safe side is to make it safe-by-default with an unsafe keyword that gives the developer freedom very selectively (eg per module). It becomes a big warning sign that the component needs strong code/data verification and validation. In this case, the dynamic typing could be the unsafe type of an otherwise statically-typed language that the program is implemented in. Just those things changing are marked with this unconstrained type with anything touching those variables validating the data they contain. Maybe some extra logging, too.

        1. -3

          Proof of Work can be useful as follows. Blockchain systems are a realization of the P=NP problem in theoretical computer science. A problem is in P if it is intuitively easy to solve on a single conventional machine. A problem is in NP if to check an answer to the problem is intuitively easy to do on a single machine.

          Now many problems are in NP, cracking hashes is only one of them. If you have a huge network of cooperating computers, then solving a difficult problem also solves consensus. Compare this with a group of schoolchildren; those who have the answers to the test, everybody likes and listens to. But these answers are very hard to attain.

          Now if P=NP, then we can observe this empirically from the networks’ converging behavior. If otherwise, we won’t observe a convergence, we can’t reject the hypothesis that P!=NP (which is what most scientists believe).

          Blockchain is a massive theoretical experiment, and to pay for it, we have popularized it and attained a critical mass of followers.

          1. 5

            Now if P=NP, then we can observe this empirically from the networks’ converging behavior. If otherwise, we won’t observe a convergence, we can’t reject the hypothesis that P!=NP (which is what most scientists believe).

            what do you mean by “convergence”?

            1. -2

              So, as the popularity of a blockchain increases, the number of participants also increase. To keep solving the consensus problem, the difficulty is increased too. If, however, the difficulty of the PoW improves without an increase of participants, another phenomenon must be at play.

              If the network at some point becomes unstable (viz. unfairness of consensus) because a few nodes have apparently a superior solution generator, we observe a bias towards particular participants. If, however, such solution generator is hard or impossible to attain, we wont observe this bias.

              Convergence is that, ultimately, there are only two stable states: stability under participants/difficulty ratio, or, instability from the incommensurability of participants and difficulty.

              Of course, there are many subtleties in this argument, requiring careful analysis. Consider (false positive): one participant actually owns a mining farm with ASICs, giving it a disproportionate amount of computing power, is that a reasonable explanation of bias? How about scenario’s such as the one of Enigma’s code breaking (false negative), where participants with a superior solution generator won’t tell the others by keeping their advantage to within bounds of reasonable chance, to prevent other parties from inferring the information that the actual problem is already solved. Then can we still observe a bias?

              1. 2

                The hashing problem that Bitcoin is based on is not known to be NP-Complete, so even if a polynomial-time algorithm for that were discovered, it would amount to a security hole in SHA256, not a proof that P = NP.

                On the other hand if someone did have a constructive proof that P = NP, they would be able to use it to break SHA256 (if it were a small enough polynomial with small enough constants). So the Bitcoin network does provide another mechanism of reward for someone who comes up with a constructive proof for P = NP. But the stability of Bitcoin doesn’t add much evidence that P =/= NP, as IMO the other rewards for proving P = NP are more than enough to motivate researchers.

                Granted, the Bitcoin network does provide further security testing for the crytographic primitives it’s based on, as any high-stakes cryptosystem does.

          1. 3

            It might be a naive thought, but “Thinking the Unthinkable,” was not about programming languages themselves but their developing research programmes, and the overarching paradigms in which these are grounded.

            One example of “the unthinkable” would be non-well-founded set theory, as studied by Aczel in 1988. It is a provocative work, and its even stating its intended purpose as breaking fundaments, with the admission of “axioms of non-fundament.” My hypothesis is that it resulted, at least in Europe, in a split between conventional CS and so-called TCS.

            This split seems like a huge distance in retrospect. TCS is how I like to call it: concrete mathematics. Its machinery and tools has even developed past years to automate most of the mundane research tasks of the previous paradigm. In concrete terms, these researchers write compilers for new languages as easily as we do programming. Programming language theory thus has moved to the bin “academically uninteresting,” and many universities renamed their research groups to focus on new topics.

            In my own work I also try to break foundations. My claims, without providing too much details, are: algorithms are not the fundamental mathematical objects we should study, and the Halting problem is not actually a problem but a corollary of Cantor’s diagonalization proof of the existence of real numbers, and termination proofs are as important as productivity proofs. Instead of algorithms, we should study protocols. Protocols compose multiple underlying computers in a cooperative network. By designating some computers as irrational, i.e. we can only observe their behavior but not finitely represent it, we can model the real-world quite well. E.g. a Byzantine failure model becomes an irrational network component. Security and privacy properties are now the fundamental questions, which by sufficient mathematical modelling can be studied and understood.

            1. 3

              Very convincing review and replication study.

              1. 18

                I really love legacy, and have been working on a DOS application that is in use since 1986. I helped to patch the blob to solve clock frequency issues around 2005, and virtualized it completely in 2015 (now allowing the app to reach files and print over the network!)

                I really hate legacy, and have found enormous amounts of garbage and myself struggeling not only with the incomprehensible and untangible structure of bloated software architectures, but also with consequent motivational problems. I even had to disappoint the customer, who invested a lot in me: despite the promising progress, fixing it for real would cost way too much.

                Sometimes I like to tell junior programmers some war stories, especially when they complain when working with the code of others. I romanticize what I call “software archeology,” and declare my love of unraveling the mysteries hidden behind the unknown. This I do for two reasons: I hope to motivate them beyond the point of misery (the trap, in which you believe you can not deal with the problem, and give up) and I hope to give them another perspective, as follows.

                Legacy is something to be proud of. It is the work of a precious generation (be it 30 years or 6 months ago), which dealt with perhaps completely different circumstances. Respect their work, just as you wish others will respect your own. Instilling this picture, that legacy is something great and is what you ultimately hope to produce, might result in work that one can be proud of: work that builds upon the great work of others, and tries to improve upon it!

                1. 4

                  “Your code could be legacy some day!” is a legitimate motivational phrase, in my opinion. There’s often a lot wrong with legacy code, but that’s because you’re often looking at it from a very different perspective. Understanding the original authors’ viewpoint is important. You might call it “code empathy”.

                  1. 1

                    I have made similar experiences with dealing with legacy. It’s easy to complain about certain design decisions, but really, sometimes it just seemed like a good idea at the time. Much can be learned from legacy code, too. Tricks that nobody uses today, space and memory optimization and such.

                    Grab a copy of some 70s or 80s source code and go to town with it sometime. Bring it into the 21st century. Enjoy the journey.

                    1. 1

                      Realizing that legacy code is code that did served well it’s purpose, is part of our path to professionalism.

                      But IMHO, this should not stop us to rethink things.

                      Most of complexities of modern software (and most of it’s flaws) come from legacy decisions that we didn’t challenge.

                      And I’m not only talking about JavaScript in the browsers, but even about things like dynamic linking, root users, using HTTP to deliver computations and so on…

                      All useful things, when they were thought. But today we can do it better.

                      1. 1

                        You know, I did get a pretty uplifted feeling reading that. The respect point is both incredibly spot on, and incredibly not the norm.

                        1. 1

                          Yes! I often really enjoy working in fifteen year old legacy code for exactly that reason. Sure the abstractions may not be great, but it is useful code that has served the company well all that time. My main job when working in legacy code is to not break what it gets right.

                          1. 1

                            This is all fine, but what turned me off a bit wrt consulting is a high frequency of modernizing legacy.

                            The code did not appear in a vacuum and there are always some of the original forces in place; budgets and schedules and such. Tradeoffs have to be made, and often these include not taking the upgrade path all the way to the latest version.

                            This leads to boredom, although the customers are always super and their domains where they work different from each other. It also raises the barrier to invest time in the latest and greatest, since the bulk of it would have to be done out of passion on free-time hobby projects.

                          1. 7

                            First of all, I love love love how vibrant the Lobsters formal methods community is getting. I’m much more likely to find cool FM stuff here than any other aggregator, and it’s awesome.

                            Second, maybe I’ve been spending too much time staring at specifications, but I’m not seeing how level 1 is different from level 2. Is level 1 “this is broken for probable inputs”, while level 2 is “this is broken for some inputs”? Different in degrees of probability.

                            1. 4

                              Level 1 is statements about specific executions; level 2 is statements about the implementation. This is for all statements, not just correctness.

                              1. 1

                                First of all, I love love love how vibrant the Lobsters formal methods community is getting.

                                Me too.

                                I believe crustaceans care more for high-quality code than the average programmer. Maybe pride of the craftsmen? This is a distinction to Hacker News, where monetary considerations get more attention. Formal methods is certainly one of the big topics to improve the quality of code.

                                1. 1

                                  I also love that we get more formal methods discussions taking place here!

                                  I am however not sure how much “depth” is accepted; should I post any paper I find interesting here, with short summaries why and personal reflections?

                                  1. 2

                                    I usually just post the papers. One thing I do, though, is try to make sure they show some practical use along with what they could or couldn’t handle. Especially in abstract so readers can assess with a glance. I’ll sometimes add summaries, reflections, brainstorming, etc if I feel it’s useful. I will warn the acceptance are hit or miss with many of the PDF’s getting 1-3 votes. Then, out of nowhere, they really appreciate one. Also, I only submit anything important on Monday-Friday since many Lobsters seem to be off-site on weekends.

                                    So, that’s been my MO. Hope that helps.

                                1. 21

                                  To be honest, most of my goodwill towards Tim Berners-Lee (which there was a lot of, by the way) went away when he started shilling for web DRM. Requiring w3c compliant browsers to ship closed source BLOBs in order to correctly display w3c compliant web pages is against the very core of the open web; not to mention how the w3c wouldn’t even protect security researchers who want to see if there are security issues with said BLOBs. I know Berners-Lee probably isn’t responsible for every one of those decisions, but he publicly (and probably internally in the w3c) argued for DRM.

                                  For further reading, here’s a great (albeit long) article from the EFF: https://www.eff.org/deeplinks/2017/10/drms-dead-canary-how-we-just-lost-web-what-we-learned-it-and-what-we-need-do-next

                                  1. 7

                                    Computers, the Internet, and the web represent some of the greatest innovations in the history of mankind and the fruition of what could have only been a fantasy for billions of our ancestors for thousands of years. To see it so quickly, in the course of a few decades, and thoroughly corrupted by the interests of corporate profits is profoundly sad. I am severely disappointed to have dedicated my life to the pursuit of mastering these technologies which increasingly exist primarily to exploit users. DRM is a thread in a tragic tapestry.

                                    1. 3

                                      At this point my usual plea is, judge what’s spoken, not by whom it’s spoken. TBL’s authority is one thing, and the merit of what he has to say about that “Solid” thing is quite another. The idea feels very sane to me, although I don’t see a clear path of shoving it past the influence of all the silo-oriented companies like Facebook and Google.

                                      1. 2

                                        “At this point my usual plea is, judge what’s spoken, not by whom it’s spoken.”

                                        This sentiment was drummed into me as a child and ordinarily I would strive to do this to a point, but the topic of putting locks on the open web by way of DRM is to me related to the apparently opposed mission of “solid”.

                                        Arguing for decoupling data from applications provided by corporate giants in the interests of user control seems absurd when he just played a major part in removing transparency and control from a user’s web experience.

                                        I’m not quite sure what to make of this.

                                        1. 2

                                          Did you consider the possibility that DRM could also work in reverse? The Digital Rights Management of individuals. I think that is the underlying motivation for allowing DRM: to protect assets and information. Users can not freely copy media to which they have no right of ownership, and conversely, companies can not freely copy user data to which they should have no right of ownership.

                                    1. 2

                                      Algebraic Graphs with Class by Mokhov. See my blog article which has a reference to this paper. http://www.hansdieterhiep.nl/blog/graphs-in-haskell/

                                      1. 1

                                        Summary: If a grammar specifies a language, and a language is defined as a set of words where a word is a sequence of letters in some alphabet, you can take a random element out of this set, to use it to test whether any word in the language is accepted by your code.

                                        Have you also looked at regular languages, and regular expressions to (completely) describe such languages, and how to generate fuzzers for them? Might be useful to generalize your work, and allow construction of fuzzers given only a regex, e.g: #[a-zA-Z0-9]{3}[a-zA-Z0-9]{3}?

                                        1. 12

                                          I like how the article notes one of the main sources of burnout in senior engineers: continually cleaning up messes caused by other people.

                                          However, I think that it does miss the biggest pathology of engineers–a continual reinvention of shiny and experimentation that I can only describe as neurotic. Watching engineers throw away perfectly good tooling in order to try the framework of the month reminds me of a cockatoo plucking its own feathers out because it isn’t getting to do anything fundamentally interesting in its cage.

                                          Companies have this problem where they refuse to acknowledge the commonalities that their businesses have with every other business on the planet (and hence they won’t accept standardized solutions) and also where they won’t actually pay engineers in such a way as to reward them for delivering value on time and under-budget.

                                          I ask you fellow lobsters–if you were guaranteed 1-5% of growth profits that your company had this year, how much harder would you work? How much less would you invest in new toys?

                                          Similarly, if there is no further growth, maybe it’s time to stop writing software. Maybe the business is completed, and we can all go do something else rewarding with our lives.

                                          1. 5

                                            Similarly, if there is no further growth, maybe it’s time to stop writing software.

                                            I think there’s also an important detail: the job isn’t to write software. It’s not to ship things. It’s not to fix bugs.

                                            The job of everyone is to do the “right things” so the company can make more money/be more sustainable/whatever the final goal of the company.

                                            Sometimes this means not goofing off on rewrites. Sometimes it means killing off a project because this isn’t actually important. Sometimes it’s not even about technology. If you’re sitting around making store page updates but company logistics are causing your company to lose shipments, maybe you need to go help the mailroom.

                                            This is why I love having engineering handle user support. It helps teach you that shipping doesn’t mean anything if your old stuff is breaking. It teaches you that you’re not doing things in a vacuum. And it helps engineers also realize that results matter, and software development is just one piece of that.

                                            A lot of product teams build out these huge product plans, but in the end spend half of their time in “firefighting” mode. Most of those teams should immediately drop everything and…. just fix their stuff. Nothing else matters.

                                            Whenever you end up spending a bunch of time on things but it doesn’t seem to have an effect on the bottom line, it weighs on you. If you can get out of the box of your job title, though, you can go immediately towards fixing the things that need fixing, adding the things that need adding. And if you’re right, you will know, and you will know immediately.

                                            1. 4

                                              “However, I think that it does miss the biggest pathology of engineers–a continual reinvention of shiny and experimentation that I can only describe as neurotic.“

                                              We agree on this idea, that software engineering seems to operate in circles. Fundamentally, there is not much change happening: just new layers or ways of expressing. Each has its positives and negatives with respect to expressivity. For example, where first-class “objects” in object-orientation could help model concurrency and interactivity, now futures in event streams seem to be it all. The underlying model of concurrency has however not fundamentally changed for 40 years, and will unlikely change in another 40.

                                              Sometimes I am concerned, when discussing programming with fellow students who seem to be dissatisfied with extremely simple programs, that we learn to expect complicated solutions everywhere. In reality, only simple solutions are really solutions in the literal sense: a solution dissolves the original problem into understandible and simpler terms. The simpler terms allows one to perform a computation more easily, thereby tackling the original problem. I am not talking about algorithmics here, e.g. divide and conquer, but modeling humane computational problems.

                                              I would be even inclined to believe that too much focus on the programming activity actually distracts from solving any problem. Keep programming activitities at minimum, focus on the humane part of development of a project. Learn as much as is possible, or as much as one wants, from the problem domain. Attack small problems first, built it out into an ecosystem of solutions. Validate the solutions: try to explain peers what is the problem, why it is important and how it can be simplified. These solutions live in the minds of people within an organisation. Develop training programs to effectively learn new hires the legacy of the company comprising all existing ideas and ways of thinking.

                                              Most people equate the role of developer and programmer. That is wrong. Project development is people-first, machine-second. Now, the word “neurotic” perfectly fits this description in my mind: people problems can not be solved by thinking like a machine.

                                              1. 3

                                                The evidence to say that money movitvates (see https://hbr.org/2013/04/does-money-really-affect-motiv) is not strong, so the guaranteed bonus might not have an incentive effect…

                                              1. 4

                                                I really like that OP mentions the importance of contracts. An important use of contracts in programming is in concurrent programming. Sequential consistency and linearizability are often expressed in terms of “adhering to the contract of a (concurrent) object”. It seems very important to understand this notion of contracts before moving on to more advanced topics, e.g. to study the possible interleavings of multiple threads interacting with some shared object. That might be an interesting topic for a next post? And since contracts also intimately relate to model checking, viz. checking properties of semantical models, of programs, and I know OP is very interested in that topic, it might all come together as follows:

                                                1. Start with the notion of “objects” that encapsulate behaviour and data. Clearly, data is the “statica” and behaviour is the “dynamica” of some system. Each could be represented by the other.
                                                2. Introduce contracts as specifications, declaring intent or external expectations. They bridge the gap between the data and the behaviour: what expected performed behaviour can be proven by inspecting the data before/after some moment in time.
                                                3. Allow abstraction to specify classes of objects that share the same contract, arriving at Abstract Data Types.
                                                4. Introduce tools to verify that a given implementation adheres to its contract. Could be QuickCheck/fuzzing/random testing, or more formally by proving (inductively) in some logical system that contract clauses are never breached, or by model checking/finding that could produce a counter-example to disprove adherence to a contract.
                                                5. Explore alternative program semantics: what if exceptions could occur at random places (intentional failure), what if programs can crash at random places (extentional failure), what if programs are executed in parallel but isolated from eachother (intentional concurrency), what if programs could be executed in parallel and interfere (extentional concurrency)? How would that affect the correctness of an implementation with respect to its contract?

                                                One should also immediately be convinced that contracts are not more than a tool, e.g. an error or ommission in a contract could still cause errors or unexpected behaviour in programs. Important questions could be: when is a contract sufficiently detailed while leaving enough room for different implementations? What is refinement of contracts? What is composability of multiple contracts, e.g. could two contracts describe differently the same behaviour of some object or could we show that a contract holds given the assumption another contract holds? What is consistency of a contract? And, finally, is a contract even inhabitable? These questions could lead to a deeper understanding of programming, serve as a nice spring board to type theory, and I would therefore agree that contracts are an important notion for any learner of programming semantics.

                                                Thought provoking…

                                                1. 2

                                                  That ties to one of my pet peeves about the “modern correctness world”. People seem to think that each technique is at odds with every other technique, and you have to pick a side of “unit tests!” or “types!” or “contracts!” Where as you pointed out, they all synergize with each other. ADTs and contracts are related, PBT makes contracts more powerful, types reduce the load on unit tests, etc. Really “contracts can do things that tests can’t” isn’t an invitation to stop using test, it’s an invitation use both.

                                                  BTW, I loved your posts on bisimulation. Are you planning on doing any more?

                                                1. 7

                                                  Kind of meta-observation: it seems that the Web 1.0 “under construction” sign/meme is reused these days for all sorts of libraries, tools and new programming languages. I kind of see a similarity between GitHub and GeoCities. Hopefully, someone is archiving these projects, since it might be of historical value in a decade or three.

                                                  1. 2

                                                    That’s a good observation. It does remind me of GeoCities now that you mention it.

                                                  1. 1

                                                    At one point in the article, it states about software applications that “It’s the primary way we communicate our thoughts and feelings to our friends and family.”

                                                    Am I the only one who finds this deeply shocking? I am sure not alone who refuses to use software (instant messaging, social media, e-mail, phone, long-distance video calling, online gaming) as a primary way of contact. I’d rather meet with my friends and family in person and have a good time, than communicate indirectly. I still use those tools, of couse, for the logistical part of meeting: agreeing on time and location, or letting them know where I am.

                                                    Can people give me examples of why one would use software as a primary means to communicate with ones intimi?

                                                    1. 4

                                                      Distance is one reason; many of my friends and family don’t live anywhere near me, since I’ve moved a few times, and I’d like to talk to them more than once or twice a year. This might be more idiosyncratic, but for me, I also find it easier to have personal conversations via methods that are more mediated, especially text chat and email. I find that with people I mainly hang out with in person, the relationship is more like them being acquaintances, while close friends are people I’ve gotten to know at least in part through a lot of textual communication.

                                                      1. 4

                                                        18 years ago I started writing some software so I could update my wesite easily because I was planning on taking a contracting job for six months about a thousand miles away (In the end I did not get the job) and I felt this would be an easy way to keep in contact with friends and family while I was away. I still have the blog though.

                                                        These days my best friend lives two thousand miles away. Until Niven-style teleportation becomes a reality, how else am I to remain in contact with my best friend (whom I’ve known for almost 40 years now)?

                                                        1. 3

                                                          Because flights are much more expensive than XMPP messages

                                                        1. 11

                                                          This article has a very low quality: it reports on two not interesting facts (retrieving code via http and loading a DLL from current directory). Why is it upvoted so much?

                                                          1. 14

                                                            Two relevant points at least: (1) an unverified binary is downloaded and installed and (2) on Windows outdated and likely exploitable versions of OpenVPN and OpenSSL are installed.

                                                            Then there’s the other more trivial stuff in the article that may or may not be interesting.

                                                            Despite the abrupt nature of the article, it seems worth talking about.

                                                            1. 4

                                                              Yeah, I’m kind of appalled that the author completely glanced over the setup step where you download the script asking you to send your password over an unsecured connection. Forget about a man in the middle attack that switches the script, all you need is to log the HTTP requests and you get the user’s password.

                                                              1. 4

                                                                They seem to stop because “they couldn’t be bothered” right where things could be about to get interesting. Which is fine ofc, that’s their prerogative, but it leads to a largely uninformative article.

                                                                1. 1

                                                                  Yeah, something has got to be wrong here. How can an article have 49 upvotes but not a single comment in its first 13 hours?

                                                                  1. 8

                                                                    It’s an interesting find but not super engaging. It’s gross incompetence with very much not industry best practices. There isn’t much to learn from this other than “don’t write sketchy code”.

                                                                1. 9

                                                                  Hah, I was actually curious whether AST will make a move. Good to see he did.

                                                                  Still, it’s sad that he doesn’t seem to care about ME.

                                                                  1. 7

                                                                    Whether he cares about ME is irrelevant here. By releasing the software under most (all?) free software and open source licenses, you forfeit the right to object even if the code is being used to trigger a WMD - with non-copyleft licenses you agree not to even see the changes to the code. That’s the beauty of liberal software licenses :^)

                                                                    All that he had asked for is a bit of courtesy.

                                                                    1. 4

                                                                      AFAIK, this courtesy is actually required by BSD license, so it’s even worse, as Intel loses here on legal ground as well.

                                                                      1. 5

                                                                        No, it is not - hence the open letter. You are most likely confused by the original BSD License which contained the so called, advertising clause.

                                                                        1. 5

                                                                          Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

                                                                          http://git.minix3.org/index.cgi?p=minix.git;a=blob;f=LICENSE;h=a119efa5f44dc93086bc34e7c95f10ed55b6401f;hb=HEAD

                                                                          1. 9

                                                                            Correct. The license requires Intel to reproduce what’s mentioned in the parent comment. The distribution of Minix as part of the IME is a “redistribution in binary form” (i.e., compiled code). Intel could have placed the parts mentioned in the license into those small paper booklets that usually accompany hardware, but as far as I can see, they haven’t done so. That is, Intel is breaching the BSD license Minix is distributed under.

                                                                            There’s no clause in the BSD license to inform Mr. Tanenbaum about the use of the software, though. That’s something he may complain about as lack of courtesy, but it’s not a legal requirement.

                                                                            What’s the consequence of the license breach? I can only speak for German law, but the BSD license does not include an auto-termination clause like the GPL does, so the license grant remains in place for the moment. The copyright holder (according to the link above, this is Vrije Universiteit, Amsterdam) may demand compensation or acknowledgment (i.e. fulfillment of the contract). Given the scale of the breach (it’s used in countless units of Intel’s hardware, distributed all over the globe by now), he might even be able to revoke the license grant, effectively stopping Intel from selling any processor containing the then unlicensed Minix. So, if you ever felt like the IME should be removed from this world, talk to the Amsterdam University and convince them to sue Intel over BSD license breach.

                                                                            That’s just my understanding of the things, but I’m pretty confident it’s correct (I’m a law student).

                                                                            1. 3

                                                                              It takes special skill to break a BSD license, congrats Intel.

                                                                              1. 5

                                                                                Actually, they may have a secret contract with the University of Amsterdam that has different conditions. But that we don’t know.

                                                                                1. 2

                                                                                  Judging from the text, doesn’t seem AST is aware of it.

                                                                                  1. 2

                                                                                    University of Amsterdam (UvA) is not the Vrije University Amsterdam (VU). AST is a professor at VU.

                                                                              2. 1

                                                                                I’ve read the license - thanks! :^)

                                                                                The software’s on their chip and they distribute the hardware so I’m not sure that actually applies - I’m not a lawyer, though.

                                                                                1. 5

                                                                                  Are you saying that if you ship the product in hardware form, you don’t distribute software that it runs? I wonder why all those PC vendors were paying fees to Microsoft for so long.

                                                                                  1. 2

                                                                                    For the license - not the software

                                                                                    1. 3

                                                                                      Yes, software is licensed. It doesn’t mean that if you sell hardware running software, you can violate that software’s license.

                                                                                  2. 3

                                                                                    So, they distribute a binary form of the OS.

                                                                                    1. 4

                                                                                      This is the “tivoization” situation that the GPLv3 was specifically created to address (and the BSD licence was not specifically updated to address).

                                                                                      1. 2

                                                                                        No, it was created to address not being able to modify the version they ship. Hardware vendors shipping GPLv2 software still have to follow the license terms and release source code. It’s right in the article you linked to.

                                                                                        BSD license says that binary distribution requires mentioning copyright license terms in the documentation, so Intel should follow it.

                                                                                        1. 3

                                                                                          Documentation or other materials. Does including a CREDITS file in the firmware count? (For that matter, Intel only sells the chipset to other vendors, not end users, so maybe it’s in the manufacturer docs? Maybe they’re to blame for not providing notice?)

                                                                                          1. 3

                                                                                            You have a point with the manufacturers being in-between Intel and the end users that I didn’t see in my above comment, but the outcome is similar. Intel redistributes Minix to the manufacturers, which then redistribute it to the end-users. Assuming Intel properly acknowledges things in the manufacturer’s docs, it’d then be the manufacturers that were in breach of the BSD license. Makes suing more work because you need to sue all the manufacturers, but it’s still illegal to not include the acknowledgements the BSD license demands.

                                                                                            Edit:

                                                                                            Does including a CREDITS file in the firmware count?

                                                                                            No. “Acknowledging” is something that needs to be done in a way the person that receives the software can actually take notice of.

                                                                                            1. 2

                                                                                              The minix license doesn’t use the word “acknowledging” so that’s not relevant.

                                                                                              1. 2

                                                                                                You’re correct, my bad. But “reproduce the above copyright notice” etc. aims at the same. Any sensible interpretation of the BSD license’s wording has to come to the result that the receivers of the source code must be able to view those parts of the license text mentioned, because otherwise the clause would be worthless.

                                                                                      2. 1

                                                                                        If they don’t distribute that copyright notice (I can’t remember last seeing any documentation coming directly from Intel as I always buy pre-assembled hardware) and your reasoning is correct, then they ought to fix it and include it somewhere.

                                                                                        However, the sub-thread started by @pkubaj is about being courteous, i.e. informing the original author about the fact that you are using their software - MINIX’s license does not have that requirement.

                                                                            2. 7

                                                                              I think he is just happy he has a large company using minix.

                                                                              1. 5

                                                                                Still, it’s sad that he doesn’t seem to care about ME.

                                                                                Or just refrains from fighting a losing battle? It’s not like governments would give up on spying on and controlling us all.

                                                                                1. 6

                                                                                  Do you have a cohesive argument behind that or are you just being negative?

                                                                                  First off, governments aren’t using IME for dragnet surveillance. They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me. They pose a giant risk to us because they’ll eventually be used in general-purpose malware, but the government wouldn’t actually fight much (or maybe at all, publicly) to keep IME.

                                                                                  Second off, security engineering is a sub-branch of economics. Arguments of the form “the government can hack anyone, just give up” are worthless. Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. We’re not even close to any diminishing returns falloff when it comes to security expenditures. While it’s technically true that the government (or any other well-funded attacker) could probably own any given consumer device that exists right now, it might cost them millions of dollars to do it (and then they have only a few days/weeks to keep using the exploit).

                                                                                  By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources. This is the primary goal for now. An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored.

                                                                                  1. 2

                                                                                    Do you have a cohesive argument behind that or are you just being negative?

                                                                                    Behind what? The idea that governments won’t give up on spying on us? Well, it’s quite simple. Police states have happened all throughout history, governments really really want absolute power over us, and they’re free to work towards it in any way they can.. so they will.

                                                                                    They (almost certainly) have some 0days, but they aren’t going to burn them on low-value targets like you or me.

                                                                                    Sure, but do they even need 0days if they have everyone ME’d?

                                                                                    They pose a giant risk to us because they’ll eventually be used in general-purpose malware

                                                                                    Yeah, that’s a problem too!

                                                                                    Defenders currently have the opportunity to make attacking orders of magnitude more expensive, for very little cost. [..] An important part of “marginally better security” is getting people to stop buying things that are intentionally backdoored

                                                                                    If you mean using completely “libre” hardware and software, that’s just not feasible for anyone who wants to get shit done in the real world. You need the best tools for your job, and you need things to Just Work.

                                                                                    By just getting everyday people do adopt marginally better security practices, we can make dragnet surveillance infeasibly expensive and reduce damage from non-governmental sources.

                                                                                    “Just”? :) I’m not saying we should all give up, but it’s an uphill battle.

                                                                                    For example, the blind masses are eagerly adopting Face ID, and pretty soon you won’t be able to get a high-end mobile phone without something like it.

                                                                                    People are still happily adopting Google Fiber, without thinking about why a company like Google might want to enter the ISP business.

                                                                                    And maybe most disgustingly and bafflingly of all, vast hordes of Useful Idiots are working hard to prevent the truth from spreading - either as a fun little hobby, or a full-time job.

                                                                                  2. 4

                                                                                    It reads to me like he just doesn’t want to admit that he’s wrong about the BSD license “providing the maximum amount of freedom to potential users”. Having a secret un-auditable, un-modifiable OS running at a deeper level than the OS you actually choose to run is the opposite of user freedom; it’s delusional to think this is a good thing from the perspective of the users.

                                                                                    1. 2

                                                                                      And the BSD code supported that by making their secret box more reliable and cheaper to develop.

                                                                                    2. 3

                                                                                      Oh, it’s still not lost. ME_cleaner is getting better, Google is getting into it with NERF, Coreboot works pretty well on many newish boards and on top of that, there’s Talos.

                                                                                    3. 2

                                                                                      He posted an update in which he says he doesn’t like IME.

                                                                                    1. 4

                                                                                      All this meta-talk concerning the deleted post warrants that the post be undeleted: for two reasons.

                                                                                      1. People can not form a proper judgement of the post themselves, if the actual content is not available. It was mentioned in the previous thread that the comment was on Web Archive, but that is a matter of timing and luck.

                                                                                      2. The comment had brought a lot of value, namely by being deleted bringing up these discussions. Hence, it is worth being published, for two reasons: 1) if all such comments would spur this activity, people might become incentivized to write similar comments (whether that is Good or Bad, is not something I judge). To base similarity, easy access to the original content is useful. 2) For people who disagree, it is useful to collect and learn from similar “low value” comments, to decide on a strategy to explain and convince to authors to write something otherwise. Without the data, by censorship, it becomes harder for both sides.

                                                                                      1. 49

                                                                                        Thanks very much for this detailed response.

                                                                                        Free speech and public moderation is a core community value on this site. After all, the site was founded in response to capricious, arbitrary, and unannounced moderation on HN.

                                                                                        That said, I think that jcs might have been a bit too light in his hand in moderation and I approve of pushcx trying to improve the quality of discussion here.

                                                                                        1. 26

                                                                                          I agree, especially with this sentiment:

                                                                                          Communities like Usenet, 4chan, and YouTube with little to no human moderation sink into useless garbage.

                                                                                          As much as technologists want to believe that the right software will enable the virtues of humanity to shine through, it’s ultimate a human problem that can’t be solved with a technological solution. I wish I could find it now, but there was a study on unmoderated vs moderated hate speech in online communities which found that moderating hate speech does, in fact, extinguish it, as opposed to it finding other avenues.

                                                                                          Moderation does matter.

                                                                                          1. 8

                                                                                            At Tumblr I think we solved it (mostly) mechanically. I left in 2015 so some of this may have changed, but we felt very strongly that “sunlight is the best disinfectant” and we did not delete content unless it was child porn, self-harm, or gore. You can still go see terrorist cat blogs posting under #catsofjihad. The community was great about shutting down those who were abusive, and the mechanics of response were important for that. Once you created a post, anyone could reference your original, even if you deleted it. Tumblr threads are not trees, but the particular chain of responses that are favored by whoever decided to respond, giving responders full control over the context, making responses far more clear. You lose the full-picture, but gain clarity of context. This was backed up by a support team that was usually great about taking action when abuse was happening. The clarity of context is what I believe gave the community the tools it needed to effectively self-police. This is something I believe is sorely missed in other platforms.

                                                                                            1. 3

                                                                                              we felt very strongly that “sunlight is the best disinfectant”

                                                                                              This is, unfortunately, naïve, even to the point of being malicious—always in consequence, even if not in intent. And exponentially moreso online, where vectors for abuse are numerous and practically zero-cost.

                                                                                              https://twitter.com/yonatanzunger/status/914605545490857984

                                                                                              1. 6

                                                                                                I don’t think your link has anything to do with the mechanism in my comment. Twitter gives total contextual power to the content creator, not to the responders. A malicious user deletes the tweet and the responses lose their power. This mechanism totally strips the community’s ability to self-police. I’m in agreement with the author of the link you posted, and I think twitter has made terrible trade-offs. There’s a reason tumblr has tended to be the place where many members of marginalized communities find the first group of people that accepts them, and these communities flourished there. That doesn’t happen with G+.

                                                                                                1. 5

                                                                                                  There’s a reason tumblr has tended to be the place where many members of marginalized communities find the first group of people that accepts them, and these communities flourished there.

                                                                                                  To be fair, this only applied if the community was not sufficiently odious that Tumblr nuked them.

                                                                                                  1. 1

                                                                                                    What do you mean? We let terrorists post anything that was legal…

                                                                                                    1. 5

                                                                                                      If memory serves there were a bunch of tags and adult blogs (some of whom were legitimately scummy folks doing commerce) that got rounded up and purged. I think around the time of Gamergate there were also some blogs purged, notably [REDACTED–see edit].

                                                                                                      (Do yourself a favor: don’t search for that last one. There is nothing there that will make you a happier person. It’s some of the most hateful and misogynistic stuff posted online. Again, do not read it. I found it by accident through webcomics circles, and saw it descend pretty immediately into awful shit.)

                                                                                                      EDIT: On second thought, I’m going to actually censor that last one. They’re shitty human beings and I don’t want to drive traffic their way, even for the curious.

                                                                                                      1. 3

                                                                                                        I mind this kind of censorship, be it self-induced or by moderators: I believe it is everyones own judgement to see for themselves and learn about it. Why mention it in description but not allow further research? In my opinion, your post is no longer credible because I can not verify it independently.

                                                                                                        (Edit: Reverted autocorrect. Perhaps add public record of post edits?)

                                                                                                        1. 1

                                                                                                          I remember a few cases where certain tags were blocked on mobile because if one of the app store testers found porn using a test of the app it could lead to them rejecting an update. Once, this unfortunately included tags like bisexual, gay and lesbian. It was not a fun decision to make, because it hurt the communities we tried so hard to protect, but Apple was holding the business hostage with this, and we chose to block lgbt tags in the short term and figure out something else later. There was significant blowback.

                                                                                                          We often mass-blocked clusters of associated spammers, which is uncontroversial as far as I know. Other controversial mass-blocks may have happened, but I don’t remember them.

                                                                                            2. 7

                                                                                              I agree, on both points. Transparency has always been a core goal of the site, and should continue to be. And I also think it makes sense to do at least slightly more moderation than there has historically been.

                                                                                              1. 1

                                                                                                I personally think the quality is/was fine and don’t care if there are flame wars as long as they’re not just 4chan style insults.

                                                                                              1. 3

                                                                                                I agree with the “unneccessary urgency” observation: this is a clever trick. I never knew the actual last time was shown in the tooltip. However, I do like the fact that this urgency allows you to make quick decisions and book sooner: otherwise, you might spend a lot of time pondering your choices.

                                                                                                Perhaps this post should be taken with a grain of salt: the author clearly booked a hotel which was not nice. However some guests prefer to stay in these kinds of “party-goer” hotels and might give a very high rating. So it seems fair that Booking shows these positive reviews:

                                                                                                9.6: “Great location for canal street night out” 10: “… Had an unforgettable after party” 10: “Great place to stay, especially to party”

                                                                                                It seems that guests that go to sleep at only 4 o’clock don’t mind the bar below. The blog author finally reconciles and takes the blame: “To be fair, this is more of the hotel’s fault than Booking’s. Also, I should have read the fine print.”

                                                                                                1. 7

                                                                                                  I think the point of that anecdote was that the categories skew ratings very high. It’s almost impossible for a hotel to get less than 2-3 stars as they’ll all have at least some redeeming qualities. With a traditional 5 star rating system you can bet there would be a lot of 1 star reviews due to the noise, but that’s not the case here because the staff happens to be friendly. It also seems like booking.com only displays the most positive reviews (which is common practice on many sites.. but still).

                                                                                                  1. 3

                                                                                                    They don’t (we use them constantly). You can also filter reviews by guest type like family, business traveler etc. to find those that are more likely from those with similar expectations. It is always a good idea to ignore best reviews and check those with lower scores (can filter on that too).

                                                                                                    Reviews are not completely reliable (are they anywhere?) and booking certainly uses a lot of dark patterns, but I don’t think it is that difficult to have a generally positive experience. Unless my wife and I are so much luckier or more capable than others, which I doubt.

                                                                                                    1. 3

                                                                                                      I’ve had much worse experience with TripAdvisor than Booking as far as being able to narrow down reviews to “people like me”.

                                                                                                1. 5

                                                                                                  No mention of LyX: it is a What You See Is What You Mean text editor for latex documents. It allows me to focus on content only. You can only apply semantic operations to your content, e.g. no bold, only strong formatting of characters. It generates Latex under the hood. It also allows you to fall back to Latex commands if needed. It also features a highly desirable math expression editor. I would recommend this tool to any Latex newcomer.

                                                                                                  1. 1

                                                                                                    I would also mention gummi as a visual editor for LaTex.