1. 13

    Rich has been railing on types for the last few keynotes, but it looks to me like he’s only tried Haskell and Kotlin and that he hasn’t used them a whole lot, because some of his complaints look like complete strawmen if you have a good understanding and experience with a type system as sophisticated as Haskell’s, and others are better addressed in languages with different type systems than Haskell, such as TypeScript.

    I think he makes lots of good points, I’m just puzzled as to why he’s seemingly ignoring a lot of research in type theory while designing his own type system (clojure.spec), and if he’s not, why he thinks other models don’t work either.

    1. 14

      One nit: spec is a contract system, not a type system. The former is often used to patch up a lack of the latter, but it’s a distinct concept you can do very different things with.

      EDIT: to see how they can diverge, you’re probably better off looking at what Racket does than what Clojure does. Racket is the main “research language” for contracts and does some pretty fun stuff with them.

      1. 4

        It’s all fuzzy to me. They’re both formal specifications. They get overlapped in a lot of ways. Many types people are describing could be pre/post conditions and invariants in contract form for specific data or functions on them. Then, a contract system extended to handle all kinds of things past Boolean will use enough logic to be able to do what advanced type systems do.

        Past Pierce or someone formally defining it, I don’t know as a formal, methods non-expert that contract and type systems in general form are fundamentally that different since they’re used the same in a lot of ways. Interchangeably, it would appear, if each uses equally powerful and/or automated logics.

        1. 13

          It’s fuzzy but there are differences in practice. I’m going to assume we’re using non-FM-level type systems, so no refinement types or dependent types for full proofs, because once you get there all of our intuition about types and contracts breaks down. Also, I’m coming from a contract background, not a type background. So take everything I say about type systems with a grain of salt.

          In general, static types verify a program’s structure, while contracts verify its properties. Like, super roughly, static types are whether a program is sense or nonsense, while contracts are whether its correct or incorrect. Consider how we normally think of tail in Haskell vs, like, Dafny:

          tail :: [a] -> [a]
          
          method tail(s: seq<T>) returns (o: seq<T>)
          requires s.Len > 0
          ensures s[0] + o = s
          

          The tradeoff is that verifying structure automatically is a lot easier than verifying semantics. That’s why historically static typing has been compile-time while contracts have been runtime. Often advances in typechecking subsumed use cases for contracts. See, for example, how Eiffel used contracts to ensure “void-free programming” (no nulls), which is subsumed by optionals. However, there are still a lot of places where they don’t overlap, such as in loop invariants, separation logic, (possibly existential contracts?), arguably smart-fuzzing, etc.

          Another overlap is refinement types, but I’d argue that refinement types are “types that act like contracts” versus contracts being “runtime refinement types”, as most successful uses of refinement types came out of research in contracts (like SPARK) and/or are more ‘contracty’ in their formulations.

          1. 3

            Is there anything contacts do that dependent types cannot?

            1. 2

              Fundamentally? Not really, nor vice versa. Both let you say arbitrary things about a function.

              In practice contracts are more popular for industrial work because they so far seem to map better to imperative languages than dependent types do.

              1. 1

                That makes sense, thanks! I’ve never heard of them. I mean I’ve probably seen people throw the concept around but I never took it for an actual thing

        2. 1

          I see the distinction when we talk about pure values, sum and product types. I wonder if the IO monad for example isn’t kind of more on the contract side of things. Sure it works as a type, type inference algorithms work with it, but the sife-effect thing makes it seem more like a pattern.

        3. 17

          I’m just puzzled as to why he’s seemingly ignoring a lot of research in type theory

          Isn’t that his thing? He’s made proud statements about his disinterest in theory. And it shows. His jubilation about transducers overlooked that they are just a less generic form of ad-hoc polymorphism, invented to abstract over operations on collections.

          1. 1

            wow, thanks for that, never really saw it that way but it totally makes sense. not a regular clojure user, but love lisp, and love the ML family of languages.

            1. 1

              So? Theory is useless without usable and easy implementation

            2. 6

              seemingly ignoring a lot of research in type theory

              I’ve come to translate this utterance as “it’s not Haskell”. Are there languages that have been hurt by “ignoring type theory research”? Some (Go, for instance) have clearly benefited from it.

              1. 12

                I don’t think rich is nearly as ignorant of Haskell’s type system as everyone seems to think. You can understand this stuff and not find it valuable and it seems pretty clear to me that this is the case. He’s obviously a skilled programmer who’s perspective warrants real consideration, people who are enamored with type systems shouldnt be quick to write him off even if they disagree.

                I don’t like dynamic languages fwiw.

                1. 3

                  I dont think we can assume anything about what he knows. Even Haskellers here are always learning about its type system or new uses. He spends most of his time in a LISP. It’s safe to assume he knows more LISP benefits than Haskell benefits until we see otherwise in examples he gives.

                  Best thing tl do is probably come up with lot of examples to run by him at various times/places. See what says for/against them.

                  1. 9

                    I guess I would want hear what people think he’s ignorant of because he clearly knows the basics of the type system, sum types, typeclasses, etc. The clojure reducers docs mention requiring associative monoids. I would be extremely surprised if he didn’t know what monads were. I don’t know how far he has to go for people to believe he really doesn’t think it’s worthwhile. I heard edward kmett say he didn’t think dependent types were worth the overhead, saying that the power to weight ratio simply wasn’t there. I believe the same about haskell as a whole. I don’t think it’s insane to believe that about most type systems and I don’t think hickey’s position stems from ignorance.

                    1. 2

                      Good examples supporting he might know the stuff. Now, we just need more detail to further test the claims on each aspect of languge design.

                      1. 2

                        From the discussions I see, it’s pretty clear to me that Rich has a better understanding of static typing and its trade offs than most Haskell fans.

                2. 10

                  I’d love to hear in a detailed fashion how Go has clearly benefited from “ignoring type theory research”.

                  1. 5

                    Rust dropped GC by following that research. Several languages had race freedom with theirs. A few had contracts or type systems with similar benefits. Go’s developers ignored that to do a simpler, Oberon-2- and C-like language.

                    There were two reasons. dmpk2k already said first, which Rob Pike said, that it was designed for anyone from any background to pick up easily right after Google hired them. Also, simplicity and consistency making it easy for them to immediately go to work on codebases they’ve never seen. This fits both Google’s needs and companies that want developers to be replaceable cogs.

                    The other is that the three developers had to agree on every feature. One came from C. One liked stuff like Oberon-2. I dont recall the other. Their consensus is unlikely to be an Ocaml, Haskell, Rust, Pony, and so on. It was something closer to what they liked and understood well.

                    If anything, I thought at the time they shouldve done something like Julia with a mix of productivity features, high C/Python integration, a usable subset people stick to, and macros for just when needed. Much better. I think a Noogler could probably handle a slighty-more-advanced language than Go. That team wanted otherwise…

                    1. 2

                      I have a hard time with a number of these statements:

                      “Rust dropped GC by following that research”? So did C++ also follow research to “drop GC”? What about “C”? I’ve been plenty of type system conversation related to Rust but nothing that I would attribute directly to “dropping GC”. That seems like a bit of a simplification.

                      Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared? I’ve seen Rob Pike talk about wanting to appeal to C and C++ programmers but nothing about ignorning type research. I’d be interested in hearing about that being done and what they thought the benefits were.

                      It sounds like you are saying that the benefit is something familiar and approachable. Is that a benefit to the users of a language or to the language itself? Actually I guess that is more like, is the benefit that it made Go approachable and familiar to a broad swath of programmers and that allowed it to gain broad adoption?

                      If yes, is there anything other than anecdotes (which I would tend to believe) to support that assertion?

                      1. 9

                        “That seems like a bit of a simplification.”

                        It was. Topic is enormously complex. Gets worse when you consider I barely knew C++ before I lost my memory. I did learn about memory pools and reference counting from game developers who used C++. I know it keeps getting updated in ways that improve its safety. The folks that understand C++ and Rust keep arguing about how safe C++ is with hardly any argument over Rust since its safety model is baked thoroughly into the language rather than an option in a sea of options. You could say I’m talking about Rust’s ability to be as safe as a GC in most of an apps code without runtime checks on memory accesses.

                        “Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared?”

                        Like with the Rich Hickey replies, this burden of proof is backwards asking us to prove a negative. If assessing what people knew or did, we should assume nothing until we see evidence in their actions and/or informed opinions that they did these things. Only then do we believe they did. I start by comparing what I’ve read of Go to Common LISP, ML’s, Haskell, Ada/SPARK, Racket/Ometa/Rascal on metaprogramming side, Rust, Julia, Nim, and so on. Go has almost nothing in it compared to these. Looks like a mix of C, Wirth’s stuff, CSP like old stuff in 1970’s-1980’s, and maybe some other things. Not much past the 1980’s. I wasn’t the first to notice either. Article gets point across despite its problems the author apologized for.

                        Now, that’s the hypothesis from observation of Go’s features vs other languages. Lets test it on intent first. What was the goal? Rob Pike tells us here with Moray Taylor having a nicer interpretation. The quote:

                        The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                        It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.

                        So, they’re intentionally dumbing the language down as much as they can while making it practically useful. They’re doing this so smart people from many backgrounds can pick it up easily and go right to being productive for their new employer. It’s also gotta be C-like for the same reason.

                        Now, let’s look at its prior inspirations. In the FAQ, they tell you the ancestors: “Go is mostly in the C family (basic syntax), with significant input from the Pascal/Modula/Oberon family (declarations, packages), plus some ideas from languages inspired by Tony Hoare’s CSP, such as Newsqueak and Limbo (concurrency).” They then make an unsubstantiated claim, in that section at least, that it’s a new language across the board to make programming better and more fun. In reality, it seems really close to a C-like version of the Oberon-2 experience one developer (can’t recall) wanted to recreate with concurrency and tooling for aiding large projects. I covered the concurrency angle in other comment. You don’t see a lot of advanced or far out stuff here: decades old tech that’s behind current capabilities. LISP’ers, metaprogrammers and REBOL’s might say behind old tech, too. ;)

                        Now, let’s look at execution of these C, Wirth-like, and specific concurrency ideas into practice. I actually can’t find this part. I did stumble upon its in-depth history of design decisions. The thing I’m missing, if it was correct, is a reference to the claim that the three developers had to agree on each feature. If that’s true, it automatically would hold the language back from advanced stuff.

                        In summary, we have a language designed by people who mostly didn’t use cutting-edge work in type systems, employed nothing of the sort, looked like languages from the 1970’s-1980’s, considered them ancestors, is admittedly dumbed-down as much as possible so anyone from any background can use it, and maybe involved consensus from people who didn’t use cutting-edge stuff (or even much cutting-edge at 90’s onward). They actually appear to be detractors to a lot of that stuff if we consider the languages they pushed as reflecting their views on what people should use. Meanwhile, the languages I mentioned above used stuff from 1990’s-2000’s giving them capabilities Go doesn’t have. I think the evidence weighs strongly in favor of that being because designers didn’t look at it, were opposed to it for technical and/or industrial reasons, couldn’t reach a consensus, or some combo.

                        That’s what I think of Go’s history for now. People more knowledgeable feel free to throw any resources I might be missing. It just looks to be a highly-practical, learn/use-quickly, C/Oberon-like language made to improve onboarding and productivity of random developers coming into big companies like Google. Rob Pike even says that was the goal. Seems open and shut to me. I thank the developers of languages like Julia and Nim believing we were smart enough to learn a more modern language, even if we have to subset them for inexperienced people.

                    2. 4

                      It’s easy for non-LtU programmers to pick up, which happens to be the vast majority.

                      1. 3

                        Sorry, that isn’t detailed. Is there evidence that its easy for these programmers to pick up? What does “easy to pick up” mean? To get something to compile? To create error-free programs? “Clearly benefited” is a really loaded term that can mean pretty much anything to anyone. I’m looking for what the stated benefits are for Go. Is the benefit to go that it is “approachable” and “familiar”?

                        There seems to be an idea in your statement then that using any sort of type theory research will inherintly make something hard to pick up. I have a hard time accepting that. I would, without evidence, be willing to accept that many type system ideas (like a number of them in Pony) are hard to pick up, but the idea that you have to ignore type theory research to be easy to pick up is hard for me to accept.

                        Could I create a language that ignores type system theory but using a non-familiar syntax and not be easy to pick up?

                        1. 5

                          I already gave you the quote from Pike saying it was specifically designed for this. Far as the how, I think one of its designers explains it well in those slides. The Guiding Principles section puts simplicity above everything else. Next, a slide says Pascal was a minimalist language designed for teaching non-programmers to code. Oberon was similarly simple. Oberon-2 added methods on records (think simpler OOP). The designer shows Oberon-2 and Go code saying it’s C’s syntax with Oberon-2’s structure. I’ll add benefits like automatic, memory management.

                          Then, the design link said they chose CSP because (a) they understood it enough to implement and (b) it was the easiest thing to implement throughout the language. Like Go itself, it was the simplest option rather than the best along many attributes. There were lots of people who picked up SCOOP (super-easy but with overhead) with probably even more picking up Rust’s method grounded in affine types. Pony is itself doing clever stuff using advances in language. Go language would ignore those since (a) Go designers didn’t know them well from way back when and (b) would’ve been more work than their intent/budget could take.

                          They’re at least consistent about simplicity for easy implementation and learning. I’ll give them that.

                      2. 3

                        It seems to me that Go was clearly designed to have a well-known, well-understood set of primitives, and that design angle translated into not incorporating anything fundamentally new or adventurous (unlike Pony and it’s impressive use of object capabilities). It looked already old at birth, but it feels impressively smooth, in the beginning at least.

                        1. 3

                          I find it hard to believe that CSP and Goroutines were “well-understood set of primitives”. Given the lack of usage of CSP as a mainstream concurrency mechanism, I think that saying that Go incorporates nothing fundamentally new or adventurous is selling it short.

                          1. 5

                            CSP is one of oldest ways people modeled concurrency. I think it was built on Hoare’s monitor concept from years before which Per Brinch Hansen turned into Concurrent Pascal. Built Solo OS with mix of it and regular Pascal. It was also typical in high-assurance to use something like Z or VDM for specifying main system with concurrency done in CSP and/or some temporal logic. Then, SPIN became dominant way to analyze CSP-like stuff automatically with a lot of industrial use for a formal method. Lots of other tools and formalisms existed, though, under banner of process algebras.

                            Outside of verification, the introductory text that taught me about high-performance, parallel computing mentioned CSP as one of basic models of parallel programming. I was experimenting with it in maybe 2000-2001 based on what those HPC/supercomputing texts taught me. It also tied into Agent-Oriented Programming I was looking into then given they were also concurrent, sequential processes distributed across machines and networks. A quick DuckDuckGo shows a summary article on Wikipedia mentions it, too.

                            There were so many courses teaching and folks using it that experts in language design and/or concurrency should’ve noticed it a long time ago trying to improve on it for their languages. Many did, some doing better. Eiffel SCOOP, ML variants like Concurrent ML, Chapel, Clay with Wittie’s extensions, Rust, and Pony are examples. Then you have Go doing something CSP-like (circa 1970’s) in the 2000’s still getting race conditions and stuff. What did they learn? (shrugs) I don’t know…

                            1. 10

                              Nick,

                              I’m going to take the 3 different threads of conversation we have going and try to pull them all together in this one reply. I want to thank you for the time you put into each answer. So much of what appears on Reddit, HN, and elsewhere is throw away short things that often feel lazy or like communication wasn’t really the goal. For a long time, I have appreciated your contributions to lobste.rs because there is a thoughtfulness to them and an attempt to convey information and thinking that is often absent in this medium. Your replies earlier today are no exception.


                              Language is funny.

                              You have a very different interpretation of the words “well-understood primitives” than I do. Perhaps it has something to do with anchoring when I was writing my response. I would rephrase my statement this way (and I would still be imprecise):

                              While CSP has been around for a long time, I don’t that prior to Go, that is was a well known or familiar concurrency model for most programmers. From that, I would say it isn’t “well-understood”. But I’m reading quite a bit, based on context into what “well-understood” means here. I’m taking it to me, “widely understood by a large body of programmers”.

                              And I think that your response Nick, I think it actually makes me believe that more. The languages you mention aren’t ones that I would consider familiar or mainstream to most programmers.

                              Language is fun like that. I could be anchoring myself again. I rarely ask questions on lobste.rs or comment. I decided to on this occasion because I was really curious about a number of things from an earlier statement:

                              “Go has clearly benefited from “ignoring type theory research”.

                              Some things that came to mind when I read that and I wondered “what does this mean?”

                              “clearly benefited”

                              Hmmm, what does benefit mean? Especially in reference to a language. My reading of benefit is that “doing X helped the language designers achieve one or more goals in a way that had acceptable tradeoffs”. However, it was far from clear to me, that is what people meant.

                              “ignoring type theory research”

                              ignoring is an interesting term. This could mean many things and I think it has profound implications for the statement. Does ignoring mean ignorance? Does it mean willfully not caring? Or does it mean considered but decided not to use?

                              I’m familiar with some of the Rob Pike and Go early history comments that you referenced in the other threads. In particular related to the goal of Go being designed for:

                              The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                              It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.

                              I haven’t found anything though that shows there was a willful disregard of type theory. I wasn’t attempting to get you to prove a negative, more I’m curious. Has the Go team ever said something that would fall under the heading of “type system theory, bah we don’t need it”. Perhaps they have. And if they have, is there anything that shows a benefit from that.

                              There’s so much that is loaded into those questions though. So, I’m going to make some statements that are possibly open to being misconstrued about what from your responses, I’m hearing.

                              “Benefit” here means “helped make popular” because Go on its surface, presents a number of familiar concepts for the programmer to work with. There’s no individual primitive that feels novel or new to most programmers except perhaps the concurrency model. However, upon the first approach that concurrency model is fairly straightforward in what it asks the programmer to grasp when first encountering it. Given Go’s stated goals from the quote above. It allows the programmers to feel productive and “build good software”.

                              Even as I’m writing that though, I start to take issue with a number of the assumptions that are built into the Pike quote. But that is fine. I think most of it comes down to for me what “good software” is and what “simple” is. And those are further loaded words that can radically change the meaning of a comment based on the reader.

                              So let me try again:

                              When people say “Go has clearly benefited from “ignoring type theory research” what they are saying is:

                              Go’s level of popularity is based, in part, on it providing a set of ideas that should be mostly familiar to programmers who have some experience with the Algol family of languages such as C, C++, Python, Ruby etc. We can further refine that to say that from the Algol family of languages that we are really talking about ones that have type systems that make few if any guarantees (like C). That Go put this familiarity as its primary goal and because of that, is popular.

                              Would you say that is a reasonable summation?


                              When I asked:

                              “Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared?”

                              I wasn’t asking you for to prove a negative. I was very curious if any such statements existed. I’ve never seen any. I’ve drawn a number of conclusions about Go based mostly on the Rob Pike quote you provided earlier. I was really looking for “has everyone else as well” or do they know things that I don’t know.

                              It sounds like we are both mostly operating on the same set of information. That’s fine. We can draw conclusions from that. But I feel at least good in now saying that both you and I are inferring things based on what appears to be mostly shared set of knowledge here and not that I am ignorant of statements made by Go team members.

                              I wasn’t looking for proof. I was looking for information that might help clear up my ignorance in the area. Related to my ignorance.

                              1. 2

                                I appreciate that you saw I was trying to put effort into it being productive and civil. Those posts took a while. I appreciate your introspective and kind reply, too. Now, let’s see where we’re at with this.

                                Yeah, it looks like we were using words with a different meaning. I was focused on well-understood by PLT types that design languages and folks studying parallelism. Rob Pike at the least should be in both categories following that research. Most programmers don’t know about it. You’re right that Go could’ve been first time it went mainstream.

                                You also made a good point that it’s probably overstating it to say they never considered. I have good evidence they avoided almost all of it. Other designers didn’t. Yet, they may have considered it (how much we don’t know), assessed it against their objectives, and decided against all of it. The simplest approach would be to just ask them in a non-confrontational way. The other possibility is to look at each’s work to see if it showed any indication they were considering or using such techniques in other work. If they were absent, saying they didn’t consider it in their next work would be reasonable. Another angle would be to look at, like with C’s developers, whether they had a personal preference for simpler or barely any typing consistently avoiding developments in type systems. Since that’s lots of work, I’ll leave it at “Unknown” for now.

                                Regarding its popularity, I’ll start by saying I agree its simple design reusing existing concepts was a huge element of that. It was Wirth’s philosophy to do same thing for educating programmers. Go adopted that philosophy to modern situation. Smart move. I think you shouldn’t underestimate the fact that Google backed it, though.

                                There were a lot of interesting languages over the decades with all kinds of good tradeoffs. The ones with major, corporate backing and/or on top of advantageous foundations/ecosystems (eg hardware or OS’s) usually became big in a lasting way. That included COBOL on mainframes, C on cheap hardware spreading with UNIX, Java getting there almost entirely through marketing given its technical failures, .NET/C# forced by Microsoft on its huge ecosystem, Apple pushing Swift, and some smaller ones. Notice the language design is all across the board here in complexity, often more complex than existing languages. The ecosystem drivers, esp marketing or dominant companies, are the consistent thread driving at least these languages’ mass adoption.

                                Now, mighty Google claims they’re backing for their massive ecosystem a new language. It’s also designed by celebrity researchers/programmers, including one many in C community respect. It might also be a factor in whether developers get a six digit job. These are two, major pulls plus a minor one that each in isolation can draw in developers. Two, esp employment, will automatically make a large number of users if they think Google is serious. Both also have ripple effects where other companies will copy what big company is doing to not get left behind. Makes the pull larger.

                                So, as I think of your question, I have that in the back of my mind. I mean, those effects pull so hard that Google’s language could be a total piece of garbage and still have 50,000-100,000 developers just going for a gold rush. I think that they simplified the design to make it super-easy to learn and maintain existing code just turbocharges that effect. Yes, I think the design and its designers could lead to significant community without Google. I’m just leaning toward it being a major employer with celebrity designers and fanfare causing most of it.

                                And then those other languages start getting uptake despite advanced features or learning troubles (esp Rust). Shows they Go team could’ve done better on typing using such techniques if they wanted to and/or knew about those techniques. I said that’s unknown. Go might be the best they could do in their background, constraints, goals, or whatever. Good that at least four, different groups made languages to push programming further into the 90’s and 2000’s instead of just 70’s to early 80’s. There’s at least three creating languages closer to C generating a lot of excitement. C++ is also getting updates making it more like Ada. Non-mainstream languages like Ada/SPARK and Pony are still getting uptake even though smaller.

                                If anything, the choices of systems-type languages is exploding right now with something for everyone. The decisions of Go’s language authors aren’t even worth worrying about since that time can be put into more appropriate tools. I’m still going to point out that Rob Pike quote to people to show they had very, very specific goals which made a language design that may or may not be ideal for a given task. It’s good for perspective. I don’t know designers’ studies, their tradeoffs, and (given alternatives) they barely matter past personal curiosity and PLT history. That also means I’ll remain too willfully ignorant about it to clear up anyone’s ignorance. At least till I see some submissions with them talking about it. :)

                                1. 2

                                  Thanks for the time you put into this @nickpsecurity.

                                  1. 1

                                    Sure thing. I appreciate you patiently putting time into helping me be more accurate and fair describing Go designers’ work.

                                    1. 2

                                      And thank you, I have a different perspective on Go now than I did before. Or rather, I have a better understanding of other perspectives.

                    3. 6

                      I don’t see anything of substance in this comment other than “Haskell has a great type system”.

                      I just watched the talk. Rich took a lot of time to explain his thoughts carefully, and I’m convinced by many of his points. I’m not convinced by anything in this comment because there’s barely anything there. What are you referring to specifically?

                      edit: See my perspective here: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey#c_povjwe

                      1. 3

                        That wasn’t my point at all. I agree with what Rich says about Maybes in this talk, but it’s obvious from his bad Haskell examples that he hasn’t spent enough time with the language to justify criticizing its type system so harshly.

                        Also, what he said about representing the idea of a car with information that might or might not be there in different parts of a program might be correct in Haskell’s type system, but in languages with structural subtyping (like TypeScript) or row polymorphism (like Ur/Web) you can easily have a function that takes a car record which may be missing some fields, fills some of them out and returns an object which has a bit more fields than the other one, like Rich described at some point in the talk.

                        I’m interested to see where he’s gonna end up with this, but I don’t think he’s doing himself any favors by ignoring existing research in the same fields he’s thinking about.

                        1. 5

                          But if you say that you need to go to TypeScript to express something, that doesn’t help me as a Haskell user. I don’t start writing a program in a language with one type system and then switch into a language with a different one.

                          Anyway, my point is not to have a debate on types. My point is that I would rather read or watch an opinion backed up by real-world experience.

                          I don’t like the phrase “ignoring existing research”. It sounds too much like “somebody told me this type system was good and I’m repeating it”. Just because someone published a paper on it, doesn’t mean it’s good. Plenty of researchers disagree on types, and admit that there are open problems.

                          There was just one here the other day!

                          https://lobste.rs/s/dldtqq/ast_typing_problem

                          I’ve found that the applicability of types is quite domain-specific. Rich Hickey is very clear about what domains he’s talking about. If someone makes general statements about type systems without qualifying what they’re talking about, then I won’t take them very seriously.

                      2. 4

                        I don’t have a good understanding of type systems. What is it that Rich misses about Haskells Maybe? Does changing the return type of a function from Maybe T to T not mean that you have to change code which uses the return value of that function?

                        1. 23

                          Does changing the return type of a function from Maybe T to T not mean that you have to change code which uses the return value of that function?

                          It does in a way, but I think people sometimes over-estimate the amount of changes that are required. It depends on whether or not really really care about the returned value. Let’s look at a couple of examples:

                          First, let’s look at an example. Let’s say that we had a function that was going to get the first element out of a list, so we start out with something like:

                          getFirstElem :: [a] -> a
                          getFirstElem = head
                          

                          Now, we’ll write a couple of functions that make use of this function. Afterwards, I’ll change my getFirstElem function to return a Maybe a so you can see when, why, and how these specific functions need to change.

                          First, let’s imagine that I have some list of lists, and I’d like to just return a single list that has the first element; for example I might have something like ["foo","bar","baz"] and I want to get back "fbb". I can do this by calling map over my list of lists with my getFirstElem function:

                          getFirsts :: [[a]] -> [a]
                          getFirsts = map getFirstElem
                          

                          Next, say we wanted to get an idea of how many elements we were removing from our list of lists. For example, in our case of ["foo","bar","baz"] -> "fbb", we’re going from a total of 9 elements down to 3, so we’ve eliminated 6 elements. We can write a function to help us figure out how many elements we’ve dropped pretty easily by looking at the sum of the lengths of the lists in the input lists, and the overall length of the output list.

                          countDropped :: [[a]] -> [b] -> Int
                          countDropped a b =
                            let a' = sum $ map length a
                                b' = length b
                            in a' - b'
                          

                          Finally, we probably want to print out our string, so we’ll use print:

                          printFirsts =
                            let l = ["foo","bar","baz"]
                                r = getFirsts l
                                d = countDropped l r
                            in print l >> print r >> print d
                          

                          Later, if we decide that we want to change our program to look at ["foo","","bar","","baz"]. We’ll see our program crashes! Oh no! the problem is that head doesn’t work with an empty list, so we better go and update it. We’ll have it return a Maybe a so that we can capture the case where we actually got an empty list.

                          getFirstElem :: [a] -> Maybe a
                          getFirstElem = listToMaybe
                          

                          Now we’ve changed our program so that the type system will explicitly tell us whether we tried to take the head of an empty list or not- and it won’t crash if we pass one in. So what refactoring do we have to do to our program?

                          Let’s walk back through our functions one-by-one. Our getFirsts function had the type [[a]] -> [a] and we’ll need to change that to [[a]] -> [Maybe a] now. What about the code?

                          If we look at the type of map we’ll see that it has the type: map :: (c -> d) -> [c] -> [d]. Since both [[a]] -> [a] and [[a]] -> [Maybe a] satisfy the constraint [a] -> [b], (in both cases, c ~ [a], in the first case, d ~ a and in the second d ~ Maybe a). In short, we had to fix our type signature, but nothing in our code has to change at all.

                          What about countDropped? Even though our types changed, we don’t have to change anything in countDropped at all! Why? Because countDropped is never looking at any values inside of the list- it only cares about the structure of the lists (in this case, how many elements they have).

                          Finally, we’ll need to update printFirsts. The type signature here doesn’t need to change, but we might want to change the way that we’re printing out our values. Technically we can print a Maybe value, but we’d end up with something like: [Maybe 'f',Nothing,Maybe 'b',Nothing,Maybe 'b'], which isn’t particularly readable. Let’s update it to replace Nothing values with spaces:

                          printFirsts :: IO ()
                          printFirsts =
                            let l = ["foo","","bar","","baz"]
                                r = map (fromMaybe ' ') $ getFirsts' l
                                d = countDropped l r
                            in print l >> print r >> print d
                          

                          In short, from this example, you can see that we can refactor our code to change the type, and in most cases the only code that needs to change is code that cares about the value that we’ve changed. In an untyped language you’d expect to still have to change the code that cares about the values you’re passing around, so the only additional changes that we’ve had to do here was a very small update to the type signature (but not the implementation) of one function. In fact, if I’d let the type be inferred (or written a much more general function) I wouldn’t have had to even do that.

                          There’s an impression that the types in Haskell require you to do a lot of extra work when refactoring, but in practice the changes you are making aren’t materially more or different than the ones you’d make in an untyped language- it’s just that the compiler will tell you about the changes you need to make, so you don’t need to find them through unit tests or program crashes.

                          1. 3

                            countDropped should be changed. To what will depend on your specification but as a simple inspection, countDropped ["", "", "", ""] [None, None, None, None] will return -4, which isn’t likely to be what you want.

                            1. 2

                              That’s correct in a manner of speaking, since we’re essentially computing the difference between the number of characters in all of the substrings minutes the length of the printed items. Since [""] = [[]], but is printed " ", we print one extra character (the space) compared to the total length of the string, so a negative “dropped” value is sensible.

                              Of course the entire thing was a completely contrived example I came up with while I was sitting at work trying to get through my morning coffee, and really only served to show “sometimes we don’t need to change the types at all”, so I’m not terribly worried about the semantics of the specification. You’re welcome to propose any other more sensible alternative you’d like.

                              1. -3

                                That’s correct in a manner of speaking, since …

                                This is an impressive contortion, on par with corporate legalese, but your post-hoc justification is undermined by the fact that you didn’t know this was the behavior of your function until I pointed it out.

                                Of course the entire thing was a completely contrived example …

                                On this, we can agree. You created a function whose definition would still typecheck after the change, without addressing the changed behavior, nor refuting that in the general case, Maybe T is not a supertype of T.

                                You’re welcome to propose any other more sensible alternative you’d like.

                                Alternative to what, Maybe? The hour long talk linked here is pretty good. Nullable types are more advantageous, too, like C#’s int?. The point is that if you have a function and call it as f(0) when the function requires its first argument, but later, the requirement is “relaxed”, all the places where you wrote f(0) will still work and behave in exactly the same way.

                                Getting back to the original question, which was (1) “what is it that Rich Hickey doesn’t understand about types?” and, (2) “does changing the return type from Maybe T to T cause calling code to break?”. The answer to (2) is yes. The answer to (1), given (2), is nothing.

                                1. 9

                                  I was actually perfectly aware of the behavior, and I didn’t care because it was just a small toy example. I was just trying to show some examples of when and how you need to change code and/or type signatures, not write some amazing production quality code to drop some things from a list. No idea why you’re trying to be such an ass about it.

                                  1. 3

                                    She did not address question (1) at all. You are reading her response to question (2) as implying something about (1) that makes your response needlessly adverse.

                              2. 1

                                This is a great example. To further reinforce your point, I feel like the one place Haskell really shows it’s strength in these refactors. It’s often a pain to figure out what the correct types should be parts of your programs, but when you know this and make a change, the Haskell compiler becomes this real guiding light when working through a re-factor.

                              3. 10

                                He explicitly makes the point that “strengthening a promise”, that is from “I might give you a T” to “I’ll definitely give you a T” shouldn’t necessarily be a breaking change, but is in the absence of union types.

                                1. 2

                                  Half baked thought here that I’m just airing to ask for an opinion on:

                                  Say as an alternative, the producer produces Either (forall a. a) T instead of Maybe T, and the consumer consumes Either x T. Then the producer’s author changes it to make a stronger promise by changing it to produce Either Void T instead.

                                  I think this does what I would want? This change hasn’t broken the consumer because x would match either alternative. The producer has strengthened the promise it makes because now it promises not to produce a Left constructor.

                                  1. 4

                                    When the problem is “I can’t change my mind after I had insufficient forethought”, requiring additional forethought is not a solution.

                                    1. 2

                                      So we’d need a way to automatically rewrite Maybe t to Either (forall a. a) t everywhere - after the fact. ;)

                              4. 2

                                Likewise, I wonder what he thinks about Rust’s type system to ensure temporal safety without a GC. Is safe, no-GC operation in general or for performance-critical modules desirable for Clojure practitioners? Would they like a compile to native option that integrates that safe, optimized code with the rest of their app? And if not affine types, what’s his solution that doesn’t involve runtime checks that degrade performance?

                                1. 7

                                  I’d argue that GC is a perfectly fine solution in vast majority of cases. The overhead from advanced GC systems like the one on the JVM is becoming incredibly small. So, the scenarios where you can’t afford GC are niche in my opinion. If you are in such a situation, then types do seem like a reasonable way to approach the problem.

                                  1. 3

                                    I have worked professionally in Clojure but I have never had to make a performance critical application with it. The high performance code I have written has been in C and CUDA. I have been learning Rust in my spare time.

                                    I argue that both Clojure and Rust both have thread safe memory abstractions, but Clojure’s solution has more (theoretical) overhead. This is because while Rust uses ownership and affine types, Clojure uses immutable data structures.

                                    In particular, get/insert/remove for a Rust HashMap is O(1) amortized while Clojure’s corresponding hash-map’s complexity is O(log_32(n)) for those operations.

                                    I haven’t made careful benchmarks to see how this scaling difference plays out in the real world, however.

                                    1. 4

                                      Having used clojure’s various “thread safe memory abstractions” I would say that the overhead is actual not theoretical.

                                2. 2

                                  Disclaimer: I <3 types a lot, Purescript is lovely and whatnot

                                  I dunno, I kinda disagree about this. Even in the research languages, people are opting for nominal ADTs. Typescript is the exception, not the rule.

                                  His wants in this space almost require “everything is a dictionary/hashmap”, and I don’t think the research in type theory is tackling his complaints (the whole “place-oriented programming” stuff and positional argument difficulties ring extremely true). M…aybe row types, but row types are not easy to use compared to the transparent and simple Typescript model in my opinion.

                                  Row types help o solve issues generated in the ADT universe, but you still have the nominal typing problem which is his other thing.

                                  His last keynote was very agressive and I think people wrote it off because it felt almost ignorant, but I think this keynote is extremely on point once he gets beyond the maybe railing in the intro

                                1. 3

                                  “When pursuing a vertical scaling strategy, you will eventually run up against limits. You won’t be able to add more memory, add more disk, add more “something.” When that day comes, you’ll need to find a way to scale your application horizontally to address the problem.”

                                  I should note the limit last I checked was SGI UV’s for data-intensive apps (eg SAP Hana) having 64 sockets with multicore Xeons, 64TB RAM, and 500ns max latency all-to-all communication. I swore one had 256 sockets but maybe misremembering. Most NUMA’s also have high-availability features (eg “RAS”). So, if it’s one application per server (i.e. just DB), many businesses might never run into a limit on these things. The main limit I saw studying NUMA machines was price: scaling up cost a fortune compared to scaling out. One can get stuff in low-to-mid, five digits now that previously cost six to seven. Future-proofing scale up by starting with SGI-style servers has to be more expensive, though, than scale-out start even if scale-out starts on a beefy machine.

                                  You really should modify the article to bring pricing up. The high price of NUMA machines was literally the reason for inventing Beowful clusters which pushed a lot of the “spread it out on many machines” philosophy towards mainstream. The early companies selling them always showed the price of eg a 64-256 core machine by Sun/SGI/Cray vs cluster of 2-4 core boxes. First was price of mini-mansion (or castle if clustered NUMA’s) with second ranging from new car to middle-class house. HA clustering goes back further with VMS, NonStop, and mainframe stuff. I’m not sure if cost pushed horizontal scaling for fault-tolerance to get away from them or if folks were just building on popular ecosystems. Probably a mix but I got no data.

                                  “The number of replicas, also known as “the replication factor,” allows us to survive the loss of some members of the system (usually referred to as a “cluster”). “

                                  I’ll add that each could experience the same failure, esp if we’re talking attacks. Happened to me in a triple, modular redundancy setup with a single component faulty. On top of replication, I push hardware/software diversity much as one’s resources allow. CPU’s built on different tools/nodes. Different mobo’s and UPS’s. Maybe optical connections if worried about electrical stuff. Different OS’s. Different libraries if they perform identical function. Different compilers. And so on. The thing that’s the same on each node is one app you’re wanting to work. Even it might be several written by different people with cluster having a mix of them. The one thing that has to be shared is the protocol for starting it all up, syncing the state, and recovering from problems. Critical layers like that should get the strongest verification the team can afford with SQLite and FoundationDB being the exemplars in that area.

                                  Then, it’s really replicated in a fault-isolating way. It’s also got a lot of extra failure modes one has to test for. Good news is several companies and/or volunteers can chip in each working on one of the 3+ hardware/software systems. Split the cost up. :)

                                  1. 2

                                    Those are planned subjects for future posts. I wanted to keep this one fairly simplistic for folks who aren’t experts in the area.

                                    1. 1

                                      That makes sense. Thanks for clarifying.

                                  1. 5

                                    I rarely get hyped by a talk, but this one definitely spoke to me. This is awesome work!

                                    1. 3

                                      I spent the day beforehand trying to get Edwin to change his talk to be about the rules of Cricket. Probably best that he ignored me.

                                      1. 1

                                        As an american that has looked glancingly at cricket rules, I am happy this was ignored. >.<

                                    1. 7

                                      One thing I would love to read more about is how to determine the cutoff between scaling horizontally and investing time/money in optimizing your software.

                                      1. 2

                                        Oooo, that’s a good one. I have a couple hand-wavey heuristics. I’ll think more about trying to turn that into something more real.

                                        I have the next 2-3 posts vaguely planned out. But, I’ll definitely be thinking about your idea @hwayne.

                                        1. 1

                                          Doesn’t it boil down to, basically, “it’s a knack, spend 10 years doing it and you’ll get good at judgment”?

                                          1. 2

                                            I’d definitely take a more “business” driven approach to that problem. Optimizing your software for cost should only be done if the money it saves is superior to what it costs to optimize it.

                                            You also have to take into account indirect costs like when using weird optimization tricks can make code less readable sometimes and also has a cost for future development.

                                            1. 1

                                              On the other hand, scaling horizontally adds costs of coordinating multiple servers, which includes load balancing, orchestrating deployments, distributed system problems, etc.

                                              1. 1

                                                The business-driven approach is not always the best for society as a whole. It doesn’t take into account negative externalities like the environmental cost of running inefficient programs and runtimes.

                                              2. 2

                                                I had a few that helped a lot:

                                                1. Use the fastest components. Ex: Youtube used lighttpd over Apache.

                                                2. If caching can help, use it. Try different caching strategies.

                                                3. If it’s managed, use a system language and/or alternative GC.

                                                4. If fast path is small, write the data-heavy part in an optimizable language using best algorithms for that. Make sure it’s cache and HD-layout friendly. Recent D submission is good example.

                                                5. If it’s parallelizable, rewrite the fast path in a parallel, programming language or using such a library. Previously, there was PVM, MPI, Cilk, and Chapel. The last one is designed for scale-up, scale-out, and easy expression simultaneously. Also, always use a simpler, lighter solution like that instead of something like Hadoop or SPARK if possible.

                                                6. Whole-program, optimizing compilers (esp profile-guided) if possible. I used SGI’s for this at one point. I’m not sure if LLVM beats all of them or even has a profile-guided mode itself. Haven’t looked into that stuff in a while.

                                                Notice that most of this doesn’t take much brains. A few take little to no time either. They usually work, too, giving anything from a small to vast improvement. So, they’re some of my generic options. Via metaprogramming or just good compiler, I can also envision them all integrated into one language with compiler switches toggling the behavior. Well, except choosing fastest component or algorithm. Just the incidental stuff.

                                          1. 4

                                            I worked on parts of this release and there’s some neat things, including a streamlined Python API and the ability to run Connectors in Python 3. If you have any questions about this release or you want to talk about stream processing in Python please post something here.

                                            1. 1

                                              Not about this, but more generally: what’s the best introduction to why I should wallaroo rather than any other distributed processing doobry?

                                              Also why isn’t there a pony api?

                                              1. 3

                                                Ugh, I typed in a nice long answer and lost it when I hit the back button. Blarg. This version will be shorter.

                                                There is a Pony API but its “unsupported” meaning we don’t document it. Documenting it would mean time spent on upkeep. If people start requesting the Pony API, that could change.

                                                Re: why use Wallaroo vs something else. We don’t have anything specifically comparing Wallaroo to other systems. There’s a lot of good Wallaroo related material on the blog: https://blog.wallaroolabs.com/

                                                The number #1 reason that people are interested in Wallaroo’s Python API is they are interested in doing streaming with Python, don’t want to be tied to a specific ingest source, and aren’t interested in the overhead that comes from using a system that is written in Java and involves shuffling data back and forth between Python and Java processes.

                                                However, there are other reasons that people have as well. If you have specific use cases for streaming that you are interested in, I’d be happy to discuss the pros and cons of Wallaroo for those use-cases.

                                            1. 15

                                              To me, this is more of a grab bag of random things rather than a well thought out argument for uniqueness. I still enjoyed it.

                                              And it did make me think that I’d love to read an article that is about what made BeOS unique at the time (from one of us who was part of the community then) and also what makes Haiku unique now (from someone who is part of the community now). I’d love to read that article (or two articles if the authors worked together to make them a nice pairing).

                                              The what made BeOS unique might be harder because you’d need to give a lot of background around why some things were revolutionary then for not just the tech but the effect that are fairly commonplace feeling now.

                                              Anyway, enough babbling.

                                              1. 5

                                                I was part of the BeOS community almost 2 decades ago, pretty much from the time they ported it to x86.

                                                A few things that made it unique at the time: 64 bit filesystem that could be queried like a database; incredible support for multiprocessing and low-latency that allowed for professional quality media editing apps; an API that made programmers want to write software for it even when there weren’t that many users.

                                              1. 2

                                                I’m curious. Why?

                                                I wouldn’t think to compare D and Nim much less put in the effort to maintain a comparison repo.

                                                Color me curious. Anyone have any insight?

                                                1. 4

                                                  They’re both C++ replacements that claim to be better. They have some feature overlap. A comparison makes sense given either could become someone’s general-purpose language for new projects.

                                                  There’s fans of each on the site. So, I figured I throw it on here in case they wanted to play with a new language this weekend.

                                                  1. 3

                                                    Interesting. I’ve done some Nim in the past. Never considered it as a C++ replacement.

                                                    1. 3

                                                      They told me it had ability to do unsafe things, integrate well with C, and metaprogramming for customization. Anything with these can potentially be a C++ replacement. If I was misinformed about unsafe support, then it woudnt be as suited to this task. Unlike C++, it also spits out C.

                                                1. 3

                                                  This looks really interesting but whooo I bet confluent is not happy about this

                                                  1. 4

                                                    They’re apparently publicly upset about this.

                                                    “They’re profiting from our use of resources,” Kreps said. “We’re charging our customers for the service, paying for compute and network resources that we use.”

                                                    https://twitter.com/kmassada/status/1068553502455037952

                                                    Unfortunately I can’t find the original source.

                                                  1. 2

                                                    I didn’t watch to the end, but is there any movement on adding the pthread/thread local optimization to speed things up?

                                                    1. 1

                                                      Not that I am aware of

                                                    1. 5

                                                      There is an informative series or articles from one of the Midori authors which includes object-capabilities as provided/defined/enforced by the kernel, see Midori: Objects as Secure Capabilities.

                                                      1. 2

                                                        Joe’s series is awesome. Highly recommended.

                                                      1. 9

                                                        cf Ponylang for an implementation of this model: https://tutorial.ponylang.io/capabilities/

                                                        1. 7

                                                          Pony also has capabilities on external resources. For example, you need a capability to open a socket.

                                                          1. 3

                                                            This seems very similar to Rust’s borrowing model, and both of these are excellent!

                                                            However, capabilities with regards to variables is only one part of the picture.

                                                            I very much want my programs to be capability safe in terms of external resources too: the filesystem, exec, network access, etc. Functions and modules which require access to these would need to be provided them in order to utilize it.

                                                            Monte implements this and has interesting demonstrations like https://monte.readthedocs.io/en/latest/taste.html#cooperation-without-vulerability which can safely eval() code from the internet. It doesn’t prevent an abusive user from using all the RAM or all the CPU, but it is impossible for the abusive user to read RAM, write to the FS, send spam emails, etc.

                                                            EDIT 1 Additionally, capability safety doesn’t even have to apply to a single program! TahoeLAFS is a capability-safe networked filesystem.

                                                          1. 4

                                                            Besides a couple that are already mentioned (like Alan Kay and Rands)…

                                                            • Hernán Wilkinson. He radically changed my view on what my job (and now my passion) is. He also lead me to the discovery of other great role models (like Alonso Church, the father of lambda calculus). Hernán and his pals have and promote my favorite way to organize a company. If you have some time to spare, you should check how 10Pines works.
                                                            • Robert Virding. I had the amazing luck to work with him, teaching one of the languages he has created (Erlang). He is such a smart but down-to-earth fellow, he transpiles that inspiring feeling of I’m no special, you could totally do what I did. Just put yourself into it.
                                                            • Jordan Mechner. I could never understood (even after reading the books) how two floppy disks could contain such an inmersive experience as Prince of Persia. I’m not a game developer nor a UI expert by any means, but I always admired the talent of people that can do so much with so limited resources.
                                                            1. 3

                                                              I was fortunate enough to be Hernán’s boss for a couple years and we’ve remained friends.

                                                            1. 6

                                                              I’m giving a Pony tutorial at Code Sync London on Wednesday

                                                              1. 1

                                                                This is one of my favorite papers. I’ve done a talk built around the ideas in it and have referenced it in multiple other talks.

                                                                1. 1

                                                                  It looks pretty awesome. Learned about “data lineage” as a term to use searching. If you or @hwayne look for others, use the terms “provenance” and “taint tracking.” Most stuff like in this paper I saw in past used one of those in conjunction with “information flow” analysis or control.

                                                                  Ill do some searches on data lineage later on to see if anything interestimg turns up.

                                                                1. 31

                                                                  All this talk about ethics, open, and free brings another angle to mind: people pushing that with no-cost licenses are themselves misrepresenting what they are achieving in at least U.S.. I used to push for OSS/FOSS in the past. Now I’m switching to hybrids. The reason is that encouraging people to play “give it all away” or “use low-revenue models” in a capitalist country where opponents of freedom make billions of dollars for their software shifted all the money (and therefore power) to the latter. They then paid off politicians and used pricey lawyers to win more power against OSS/FOSS in ways they couldn’t fight against without piles of money. This includes ability to patent/copyright troll users of open/free software and especially Oracle’s API ruling which jeopardizes OSS/FOSS, backwards-compatible implementations of anything that had a proprietary API.

                                                                  From what I see, OSS/FOSS have done great things but are a fundamentally-flawed model in a capitalist country where money wins. As many as possible need to be charging by default both to support contributors and send money/power the other way. They and FOSS-using companies that don’t depend on patent/copyright money need to pool money together to fight legal advances of patent/copyright-trolling companies that want lock-in. Otherwise, in a game where only one side is playing for keeps, the OSS/FOSS groups keep losing by default software freedoms and ability to enforce their licenses while preaching that they’re maintaining them. Seems dishonest. Also, strange I almost never read about these issues in FOSS writers articles about business model and licensing recommendations.

                                                                  Far as hybrids, I can’t give you the answer yet since it’s too soon. For FOSS, I’m looking at Open Core and Dual-Licensing with strongest copyleft available. For non-FOSS, Source-available from public-benefit companies and nonprofits chartered to implement most software freedoms for customers on top of free for non-commercial or under certain use. These freedoms and justifications would also be in licenses and contracts with huge penalties for non-compliance for extra layers of defense. Maybe expire into FOSS after certain time passes or revenue threshold. We need more experimentation that lets companies currently supplying or later releasing as FOSS to get millions to hundreds of millions in revenue collectively to fight this battle. Again, it’s not optional: the other side is actively fighting to remove software freedom inch by inch. And mostly winning despite FOSS organizations’ little victories.

                                                                  1. 5

                                                                    Apologies if this is a threadjack, but I’m wrestling with these kinds of questions. I’ve been doing open source more or less my whole career, and usually in some form of hybrid.

                                                                    Now I’m going out on my own and am searching for a model that makes sense. I like the collaboration of open source, but I also very much want to make money, and don’t want to do that with hallucinogenic business models.

                                                                    I’m building a game that has a music synthesizer in it as a core mechanic. What I’m building has basically 3 layers - infrastructure for building such things in Rust, the synthesizer itself (with GUI), and the game logic on top. What I’m converging on is doing the first two layers as very much community open source with permissive licenses, and the third layer as just straight up proprietary software, no pretending to be anything else. There’s stuff to fine-tune around the edges, for example somebody brought up a delayed open release of the game source, after the monetization has run its course, but I don’t want to commit to that right now because it might constrain working with a commercial publisher. If I end up self-publishing, I’ll strongly consider that though, especially if people tell me it helps motivate their contribution.

                                                                    1. 3

                                                                      I think you should flip your plans on business models around this code - the infrastructure and synthesizer are the things that could have enough value to a business that you could do well selling them. The median game does not make a profit and, as entertainment, games are a hit-driven (usually-)one-time purchase not bought based on predictable need.

                                                                      1. 4

                                                                        I’ve certainly thought about it. But here’s my thinking. First, there’s currently no business for Rust infrastructure, the community is very much organized around permissive licenses. Second, the market for synthesizers and music plugins is pretty crowded, while games in this particular genre are, as far as I can tell, underserved. Third, I think the Switch is a promising platform, and it doesn’t really do free games. Fourth, if the game is a dud but the free music tools catch on, I can always do a pro version, and I get free marketing and market research. Lastly, the game is definitely riskier, but I’m at a point where I’m ok with that; if this stuff doesn’t monetize, I just go back to a corporate job.

                                                                      2. 1

                                                                        I think it’s pretty obvious that the ‘correct’ way of doing free software games without violating the freedom of users by making anything proprietary is to make all the code free software but not make the art/music/etc. free. After all, software freedom is about software, not about art or music.

                                                                        People can modify the software, they can use it as they see fit, but they can’t redistribute it along with the art and music. They can either come up with their own art and music or they can redistribute it without the art and music and it’ll be useful to others that already have the game (because they already have the art and music).

                                                                        1. 1

                                                                          Your model (bottom two free, top not) was exactly what first popped into my mind when I read the first couple of sentences of your comment, so you’re not the only one who thinks it makes sense, fwiw :-)

                                                                          By the way (off-topic question), as someone who has recently bought a midi controller (MPK261) and started playing around with some of the synths that I got free (Hybrid Air, Sonivox), and has a decent mathy ability to understand any given synthesis concept, but absolutely no intuition for what changes will sound like… is your game aimed at me? :-)

                                                                          1. 2

                                                                            Yes, it’s made for exactly you :) I’ll put you on my list for beta testing.

                                                                            1. 1

                                                                              OMG that’s fantastic!

                                                                          2. -1

                                                                            why don’t you want to deal LSD

                                                                            1. 1

                                                                              Beg your pardon?

                                                                              1. 3

                                                                                hallucinogenic business models

                                                                                1. 4

                                                                                  Ah, right, right. Basically I want to create value honestly and do a reasonable job of recovering revenue from the value I create, rather than playing these games that seem to increasingly substitute for that these days.

                                                                                  1. 1

                                                                                    what games are you referring to?

                                                                                    1. 2

                                                                                      Financial engineering in general, more specifically the kinds of things that startups do when they’re looking for an exit or when their purpose is to burn through VC money rather than make a business. MoviePass, Juicero, that kind of thing.

                                                                                      1. 1

                                                                                        ah okay. in reading your original comment i thought you were saying it’s hallucinogenic to think you could make a living writing free software.

                                                                          3. -5

                                                                            This would make sense if these ‘attacks on free software’ actually existed, but they don’t. They literally don’t exist.

                                                                            Source available is a violation of user freedom. It’s unacceptable. That’s all there is to it, if you care about user freedom. If you don’t then I feel sorry for you.

                                                                            FLOSS doesn’t need ‘hundreds of millions in revenue’ to fight any battle because there is no battle. I don’t know what it is, but there seems to be a recurring thread I see on forums a lot recently: everything is framed as a battle. For an unrelated example, if a game developer makes an unpopular change to their game? It’s a WAR to get them to fix it. No it isn’t. I see the term ‘culture war’ thrown around too. There’s no such thing. Not everything in life is a war or a battle.

                                                                            People and groups of people that produce non-free software aren’t at war with people that do.

                                                                            1. 7

                                                                              “This would make sense if these ‘attacks on free software’ actually existed, but they don’t. They literally don’t exist.” “FLOSS doesn’t need ‘hundreds of millions in revenue’ to fight any battle because there is no battle.”

                                                                              You missed the war on open source software by Microsoft et al… thwarted largely by IBM saying it will drop a fortune defending Linux which fits my comment… the DMCA attacks, the patent trolling (Android vendors alone pay billions), the copyright ruling from an expensive case that applies to API’s FOSS often depends on, and so on. Hell, a patent defense alone facing one of these big companies gets quoted as about $200,000 on average from expensive, law firms. You bet FOSS folks need a fortune if one of these companies wants to use a legal team to destroy them.

                                                                              You really don’t see such things much, though. You wonder why if there’s a threat as I claim. They mostly ignore FOSS developers since they’re (a) free labor that the big companies are monetizing or (b) penniless opponents with weak or non-existent marketing to big spenders. For (b), the common MO is to hit companies building on the product, FOSS or not, for royalties once they’re financially successful. They parasite off them instead of destroy while using a slice of that money investing in lobbyists and courts to ensure they can continue. Alternatively, they use a combo of the lure of money with the threat of market destruction (patent or otherwise) to pressure them to sell the company. Microsoft and IBM have taken entire markets using their dominant positions, patent portfolios, and large[-for-small-player] offerings to get acquisitions. It’s rare for someone to stare at both that kind of money and corporate threat telling them to get lost. So…

                                                                              “People and groups of people that produce non-free software aren’t at war with people that do.”

                                                                              …the big players wanting to dominate markets, maximize profits from locked-in customers, and eliminate disruptive competition are always at war with folks producing anything that threatens that. Always. It’s never changed since they have people on top whose bonuses depend on this shit. They’ll also remember companies that were sent to non-existence or limbo from new stuff they didn’t stamp out when they had a chance. Just because you or many FOSS folks aren’t playing doesn’t mean they aren’t. They certainly are. They even have people working 24/7 on Capitol Hill to screw people over. Not just here in states: they’re represented in international, treaty negotiations as well under the I.P. agreements. And if you think copyright enforcement isn’t war or this is just U.S., just look at the nice, unarmed, law-abiding people that came after Kim Dotcom over U.S.-generated complaints. Those situations illustrates the heights things can go to when it matters to those in power. Better to be the ones with power.

                                                                              1. -5

                                                                                You missed the war on open source software by Microsoft et al…

                                                                                Yes, we get it. This happened. A long time ago, now, but it happened. Okay, move on. It’s no longer relevant. The corporate world has long since embraced free software.

                                                                                the DMCA attacks, the patent trolling (Android vendors alone pay billions), the copyright ruling from an expensive case that applies to API’s FOSS often depends on, and so on.

                                                                                Patent trolling is something very specific. All companies that own patents and enforce them are not automatically patent trolls.

                                                                                Also patent trolls target companies that produce proprietary software just as much as they target companies that produce free software. It has nothing to do with free or proprietary software. They target both, because they target software in general.

                                                                                …the big players wanting to dominate markets, maximize profits from locked-in customers, and eliminate disruptive competition are always at war with folks producing anything that threatens that.

                                                                                No they aren’t.

                                                                                just look at the nice, unarmed, law-abiding people that came after Kim Dotcom

                                                                                If you think Kim Dotcom was law-abiding you have another thing coming mate.

                                                                                1. 7

                                                                                  You are very wrong.

                                                                                  I am trying to bring some more floss to the public sector and the resistance from the entrenched vendors is a thing.

                                                                                  Straight up bribes are manageable, but lobbying and regulatory capture is the true evil. We actually need floss lobbyists of our own.

                                                                                  1. 6

                                                                                    You are twisting @nickpsecurity’s words and I do not understand why you are doing this.

                                                                                    1. 1

                                                                                      No I am not.

                                                                                2. 3

                                                                                  You’re right that there isn’t a war as such, but there is definitely a certain kind of dynamic. I think @nickpsecurity is also pointing out (quite correctly) the wider implications of the wealth concentration resulting from the free work that goes into producing free software.

                                                                                  One of the issues is that the companies that use free software don’t always comply with the terms of the licence. There are many examples:

                                                                                  The problems are obvious unless you stick to a very narrow and dogmatic view. Consequences matter. Just like the ill-considered, short-sighted, damn-the-consequences technological “disruption”, free software is an idea that produced a lot of unintended side effects (and in fact, contributed to the aforementioned “disruption”).

                                                                              1. 10

                                                                                The post by a16z explains it well: https://a16z.com/2014/02/14/why-there-will-never-be-another-redhat-the-economics-of-open-source/

                                                                                Elastic vs Algolia is a nice case study here as well. Let’s see which one ends up doing better long term.

                                                                                Problem with open source is that it’s hard to monetize. In most cases if something is hard to monetize it results in poor product quality. Of course you can point to large open source projects which are excellent, but that’s really the exception not the rule. Relying on people contributing free work to a project just doesn’t work in most cases.

                                                                                It also doesn’t help that the open source community has some negative feelings towards anything commercial. That culture is also pushing a lot of software from open source to closed solutions. This post is a great example calling Redis Labs shady for wanting to have a way to make money. Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                The only thing this post achieves is convincing more companies to stop with the whole open source thing.

                                                                                1. 15

                                                                                  This post is a great example calling Redis Labs shady for wanting to have a way to make money.

                                                                                  Isn’t he just calling Redis Labs shady for calling a non-free license free? He even specifically states “You have every right to license your work in any way you choose.”. That doesn’t smell rabidly anti-commercial to me.

                                                                                  Monetizing open-source is not an easy task, though, that’s true. But the antidote to difficulty should not be dishonesty. Perhaps the honest thing would be to just stop trying to monetize it, and generate the income from some other thing?

                                                                                  1. 20

                                                                                    This post is a great example calling Redis Labs shady for wanting to have a way to make money. Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                    You’re being dishonest. I’m calling them shady for calling their software open source when it’s not. I think that engineers deserve to be paid for their work but that doesn’t make it okay to lie about the licensing of your software.

                                                                                    1. 8

                                                                                      Standardized open-source tools/infrastructure are what make modern software products (open and closed) possible. The “exceptions” you mention represent a huge share of critical business systems (Linux, many widely deployed databases, web and other network servers, TLS/security infrastructure, web frameworks, most widely used language implementations, etc.).

                                                                                      Problem with open source is that it’s hard to monetize. In most cases if something is hard to monetize it results in poor product quality.

                                                                                      If you mean that “the standard of quality for software is low”, that seems easy to agree on. Asserting that closed-source software is somehow higher quality doesn’t hold water - see the list above and contrast with any of their closed-source competitors from 10-20 years ago.

                                                                                      Making money is hard, whether you’re closed source or open source. TFA is just asserting that being deliberately misleading is bad.

                                                                                      1. 4

                                                                                        Problem with open source is that it’s hard to monetize.

                                                                                        Not really. The problem with commons is that it’s hard to sell permits when everyone can just help themselves to come in. The mitigation has been known for a long time. Fund it from the taxes. You are represented, so the commons you care about should be maintained.

                                                                                        In other words, want to research? Get a grant. Still not good enough? Maybe it’s time to acknowledge that floss movement should be represented better and start pushing.

                                                                                        Here we have a Pirate party. I personally know an elected representative. He shares my opinions on floss.

                                                                                        Have you talked to your ER?

                                                                                        1. 1

                                                                                          In other words, want to research? Get a grant.

                                                                                          I thought about that. Unfortunately, the funding authorities for academic research are about quantity of papers instead of quality or code attached. Many will actually either give no credit for implementations or criticize the people doing them since it takes away from papers that bring in more money. An example was Antti Kantee of Rump Kernels telling me how they didn’t care that he actually built one to go with the paper. They just wanted the paper.

                                                                                          Now, there’s institutions that do build software to go along with their papers. Usually just prototypes they throw together but it’s something at least. Some build software outside of their papers. Some of those like Racket are pretty polished. Getting into one of those institutions might allow a person to use tax dollars to write open-source software. Much smaller number of positions than academia as a whole. Some percentage might also be doing it in their spare time on top of a full day’s work as an academic, though.

                                                                                          1. 2

                                                                                            As a library, we get small grants for improvements as well as software, but the situation is far from ideal. We can also get our hands on EU grants, but those do require some serious administration.

                                                                                            I have met the Racket team during their stay in Prague and I am still amazed at how good people they actually manage to fund to work on the PLT projects.

                                                                                            I think that it should be much easier to get funding for open R&D. EU still measures improvements using the number of patents granted, though.

                                                                                        2. 1

                                                                                          Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                          I would like to upvote this more. Sadly, I can’t.

                                                                                          1. 1

                                                                                            I appreciate the domain name.

                                                                                          1. 1

                                                                                            Gmail everywhere

                                                                                            1. 3

                                                                                              Kick ass @hwayne

                                                                                              1. 7

                                                                                                I still owe folks a blog post about total vs partial math in Pony and how that relates to division by 0. Sadly I’ve had zero time for that as I’ve been spending all my time on Wallaroo Labs work.

                                                                                                Part of that post was going to be “and as a pre-1.0 language all this is going to change as we will be introducing partial integer math operators in the future”. Well, those operators are here. All the division by zero kerfuffle got someone inspired to implement the RFC that has been open and waiting to be.

                                                                                                Still, I owe folks a post on partial vs total integer math and eventually that will come. Maybe when I’m on vacation in November, although honestly, that sounds like an awful vacation.

                                                                                                1. 4

                                                                                                  Incidentally, have you watched Evan Czaplicki’s recent talk, ‘The Hard Parts of Open Source’? It sounds like you’re in the same situation (and I’ve probably contributed to that, sorry!).

                                                                                                  1. 2

                                                                                                    I just finished watching the talk. I enjoyed it. Thank you for the recommendation.

                                                                                                    1. 1

                                                                                                      I haven’t watched it. Evan was incredibly thoughtful and nice when I met him a few years ago at ICFP and hung out. From that interaction and the title, I imagine its something I would enjoy.

                                                                                                      Care to summarize it?

                                                                                                      1. 4

                                                                                                        I think you would enjoy it. He examines patterns of behaviour in open source communities that seem hurtful, like ‘Why don’t you just do it like this?’, ‘What gives you the right to do this?’, and so on, and traces them back to the birth of hacker culture and other very interesting historical context that directly influence today’s online communities.

                                                                                                        1. 18

                                                                                                          That does sound interesting. It certainly expands past open source communities. Programmers in general are quite happy to critique the product of a series of tradeoffs without context.

                                                                                                          We do this when we look at other people’s systems and pick out one thing to critique outside of the other feature. The Pony divide by zero kerfuffle was an example of that. Many people who knew nothing about Pony critiqued that single “feature” on the basis of the impact within system they know rather than as a “feature” within Pony as a whole (which in the end is what my blog post needs to be about). In that Pony case, a series of decisions that were made to make Pony safer moved it towards being part programming language and part theorem prover. It’s in an interesting place right now where we have downsides from both that lead to “interesting” things like divide by zero being zero because of a series of other choices. All in all, Pony is safer than many languages but, we can to find that there were a number of features needed to address issues like divide by zero. For example, dependent types, partial integer math as an option.

                                                                                                          I think this happens in every system. You make a number of well intentioned decisions where they are the right decision but inevitabily, they are going to lead to “wat” and “ugh” moments as they come together. I’ve never seen a language that doesn’t have those and if you spend the time to understand the language and its choices, you can see how when favoring certain values, you would end up there. No tool will ever be perfect.

                                                                                                          There’s a Bryan Cantrill talk on this that is really good: “Platform as a refection of value”.

                                                                                                          Often times, we also see the results of constraints on the code. For example, perhaps there was an artificial but reasonable time limit. “This needs to be fixed but we only have a couple weeks to do it, what is the best we can make this in two weeks because other things are more pressing”.

                                                                                                          I had real problems with this earlier in my career. I was incredibly judgemental. Sean in his 20s would have been all over Pony for the “stupid divide by zero”, Why? Well, I wouldn’t have taken time to understand the problem. Everything I knew had divide by zero as an error so that would “obviously be the right thing to do”. And in general, I lacked empathy. I had no ability to try and understand why someone would do something that I could see a reason to do. Worse, I didn’t care to understand. I just loved to go “wat” and laugh at things. I was awful towards PHP for example. Now, I recognize that PHP is an awesome tool for some tasks. I don’t really ever take on those tasks but that doesn’t make PHP any less valuable for those tasks.

                                                                                                          I had to work incredibly hard on empathy. Its not something we do in my family. My mother, to this day, is still incredibly selfish and as one of her children, I picked that up. My stepfather hurt his back a couple years ago. My mother was somewhat concerned with his injury but mostly was annoyed with how it impacted on her life and the extra work she had to do because he wasn’t capable.

                                                                                                          It wasn’t until I worked for an asshole CEO and was a team leader and had to try and hold my team together that I really started to get good at empathy. I realized that in order to deal with said asshole, I needed to try and understand why they did what they did. Without that understanding, there was no way I could get what my team needed from said CEO. There was no way, I could put together an argument that would speak to his needs, desires, and concerns. I developed this empathy skill for purely selfish purposes but, its turned out to be incredibly helpful in general. I have a much better appreciation and understanding of other people’s software. Where before I would judgementally dismissing things as crap, I have now often taken the time to understand why the software was the way it is, and, I’ve learned a ton in the process.

                                                                                                          Anyway, I could write 5,000 more words on this topic and things tangential to it. Given the context, that seems like rambling to the extreme, so I’m going to stop now. Thanks for the talk recommendation. I’ll definitely check it out.

                                                                                                          I’d really advise anyone who read this and found that it reasonated at all the check out that Cantrill talk. It’s really really good. And also, if you don’t think empathy and understanding can be valuable as an engineer, I’d pass along the advice to give it a serious try for a couple years. If you are like me, you will be amazed and delighted with the results.

                                                                                                          1. 3

                                                                                                            Thanks a lot for your very open and honest text, it was really moving and I’m glad you’ve made empathy a priority, and that it has worked out for you.

                                                                                                            1. 2

                                                                                                              Thanks for the context. I too struggle with being empathetic. We’re all such a deep well of emotions and desires, that sometimes I feel like if I try to open that door of trying to understand people on a deeper level, I’ll spend all my emotional budget on it. But even on a superficial level–what I try to do nowadays (not always successfully!) is realize that people probably do things that make sense to them and it’s OK for it not to make sense to me, because it doesn’t affect my life.

                                                                                                              1. 2

                                                                                                                Empathy and doing stuff with other humans is about being at the point where it does affect your life and still being able to deal with other people as fully formed, totally broken but in a different way to you, feeling, incomprehensible beings.

                                                                                                                1. 2

                                                                                                                  That sounds about right. I’m still learning, I guess!

                                                                                                              2. 2

                                                                                                                “All in all, Pony is safer than many languages but, we can to find that there were a number of features needed to address issues like divide by zero. For example, dependent types, partial integer math as an option.”

                                                                                                                I suggest a translation to WhyML in Why3 platform that feeds verification conditions to automated provers. Why3 is the middle-end that Frama-C, SPARK, and the Java one all use. They prove absence of things like you describe. The automation in SPARK is often over 90%. The backends keep improving.

                                                                                                                So, my default recommendation for any type/verification of things like number ranges is either static analyzer that’s extensible or a langusge-specific front-end for Why3. A side benefit is SPARK-style annotations are easy for programmers to learn. And you can do property-based, test generation if proof is too hard.

                                                                                                                1. 1

                                                                                                                  We do this when we look at other people’s systems and pick out one thing to critique outside of the other feature. The Pony divide by zero kerfuffle was an example of that. Many people who knew nothing about Pony critiqued that single “feature” on the basis of the impact within system they know rather than as a “feature” within Pony as a whole (which in the end is what my blog post needs to be about). In that Pony case, a series of decisions that were made to make Pony safer moved it towards being part programming language and part theorem prover. It’s in an interesting place right now where we have downsides from both that lead to “interesting” things like divide by zero being zero because of a series of other choices.

                                                                                                                  The main criticism I saw was criticism of it being presented as somehow more ‘mathematically pure’, at least around here, not criticism of Pony. The blog post claiming that it 1/0 = 0 was actually consistent with mathematics was nonsense.

                                                                                                                  1. 1

                                                                                                                    We never presented it as more mathematically pure. The langauge we used was that it was an unfortunate compromise. If you want to argue with @hwayne about his blog post, go for it.

                                                                                                                    1. 0

                                                                                                                      I never said you did