1. 7

    The irony is that he’s now trying to build better tools that use embedded DSLs instead of YAML files but the market is so saturated with YAML that I don’t think the new tools he’s working on have a chance of gaining traction and that seems to be the major part of angst in that thread.

    One of the analogies I like about the software ecosystem is yeast drowning in the byproducts of their own metabolic processes after converting sugar into alcohol. Computation is a magical substrate but we keep squandering the magic. The irony is that Michael initially sqauandered the magic and in the new and less magical regime his new tools don’t have a home. He contributed to the code-less mess he’s decrying because Ansible is one of the buggiest and slowest pieces of infrastructure management tools I’ve ever used.

    I suspect like all hype cycles people will figure it out eventually because ant used to be a thing and now it is mostly accepted that XML for a build system is a bad idea. Maybe eventually people will figure out that infrastructure as YAML is not sustainable.

    1. 1

      What alternative would you propose to DSLs or YAML?

      1. 4

        There are plenty of alternatives. Pulumi is my current favorite.

        1. 3

          Thanks for bringing Pulumi to my radar, I hadn’t heard of it earlier. It seems quite close to what I’m currently trying to master, Terraform. So I ended up here: https://pulumi.io/reference/vs/terraform.html – where they say

          Terraform, by default, requires that you manage concurrency and state manually, by way of its “state files.” Pulumi, in contrast, uses the free app.pulumi.com service to eliminate these concerns. This makes getting started with Pulumi, and operationalizing it in a team setting, much easier.

          Which to me seemed rather dishonest. Terraform’s way seems much more flexible and doesn’t tie me to Hashicorp if I don’t want that. Pulumi seems like a modern SAAS money vacuum: https://www.pulumi.com/pricing/

          The positive side, of course, is that doing many programmatic-like things in Terraform HCL is quite painful, like all non-turing programming tends to be when you stray from the path the language designers built for you … Pulumi handles that obviously much better.

          1. 3

            I work at Pulumi. To be 100% clear, you can absolutely manage a state file locally in the same way you can with TF.

            The service does have free tier though, and if you can use it, I think you should, as it is vastly more convenient.

            1. 3

              You’re welcome to use a local state file the same way as in Terraform.

            2. 1

              +100000

        1. 10

          In a burndown chart, I want to be able to run simulations as well. “What happens to this project if X work falls behind. What happens if Bob gets sick?”

          Is anything in software development predictable enough to make this kind of analysis useful?

          1. 8

            No, this is basically a management wish-fulfillment fantasy.

            Seeing that we thought something would take 8 hours of work but we spent 24 is incredibly valuable. We can revisit our assumptions we made when estimating see where we got it wrong. Then, we can try and account for it next time. Yes, estimating is hard but its also a skill you can get better at if you work at it and have support for proper tools.

            Likewise, I have heard this asserted by every manager I’ve ever worked with and for. No evidence has ever been presented, nor have estimates actually improved over time. (The usual anti-pattern is: estimates get worse over time because every time an estimate turns out to be low, a fix is proposed–“what if we do all-team estimates? what if we update estimates more frequently?”–which inevitably means spending more time on estimates, which means spending less time on development, which means we get slower and slower.)

            1. 7

              I personally used to be quite bad at estimating. I’ve worked at it, I’ve gotten much better about estimating. There are things you can do to get much better at it. None of the things you’ve mentioned are ones I think would help. I plan on writing a post about the things I’ve learned and taught others that have helped make estimates more accurate.

              1. 3

                That would make great reading.

                Are there any existing accounts of effective software schedule estimation you’d recommend?

                1. 1

                  Two things I would recommend (and will be primary topics of said blog post).

                  Estimates slipping is usually about not accurately accounting for risk. Waltzing with Bears is a great book on dealing with risk management. The ideas in it might be overkill for many folks but the process of thinking about how you should account for risk and establishing your own practices is invaluable. The book is great even if you only use it as “that seems overblown, what if I…”.

                  The second is to record your estimates and why you made them. What did you know at the time that you made your estimate. Then, when your estimate is wrong, examine why. What didn’t you account for? When I first started doing this, I realized that most of my estimates were wrong because I didn’t bother to explore the problem enough and that I was being tripped up by not accounting for known knowns. Eventually I got better at that and started getting tripped up by known unknowns (that’s risk). I’ve since adopted some techniques for fleshing our risks when I am estimating and then front loading working on risks. If you think something might take a week or it might take a month, work on that first. Dig in to understand the problem so you can better estimate it. Maybe the problem is way harder than you imagine and you shouldn’t be doing the project at all. This isn’t a new concept but its one that is rarely used. In agile methodologies, its usually called a “spike”.

                  I’ve worked on projects that spanned over the course of months and we spent a couple weeks on estimation and planning. A big part of that time digging in, understanding the problem better, discussing it and coming up with what we needed to explore more to really understand the project so we could course correct as we went along.

                  1. 1

                    Ooh, a DeMarco book! Will definitely check it out. Thanks!

                2. 1

                  Please do write this, I need to improve in this area.

                3. 1

                  The wish-fulfilment does not exist in a vacuum.

                  Your customers might not be happy with your team constantly running late. Your pre-revenue startup might have a hard time raising investment. Whatever. There are external reasons for why a professional developer must be reliable in his or her estimates to actually get things out the door.

                  I’ve been changing my opinion on this back and forth. Especially in a pïss-poor startup, where the the biz guys wanted us to skip unit testing to achieve results faster, refusing to estimate was a fuck you. The code base got convoluted, but also dealing with how they represented things was frustrating.

                  I feel that in those cases the problem runs deeper in how geeks are supposed to be managed. Hell, it could be that estimation starts eating up time because the managers drove the geeks into protesting, which is - of course - as unprofessional as delivering late. Still you need to sort out your org for smooth ops before taking care of estimates.

                  Yet this isn’t car repair for old vehicles, where you find more and more problems as you go along, making estimates tough without thorough diagnostics, but the customer is happy with the substitute car you gave out in the meantime.

                  1. 0

                    The fact that customers want something, or that it is necessary to the business’s success, do not cause it to become possible. I’m not disputing the desirability of accurate estimates. I’m disputing the idea that they are possible. I have not seen any team or technique generate such estimates reliably, over time, in various circumstances. (Of course, like any gambler with a “system,” sometimes people making estimates get lucky and they turn out to be right by coincidence.) I have heard many managers claim to have a system for reliable estimates which worked in some previous job; none was able to replicate that success on the teams I observed directly.

                    (It’s not just software, either. Many people point to the building trades as an example of successful estimation and scheduling. In my experience maintaining and restoring an old house and the experiences of friends and acquaintances who’ve undertaken more ambitious restorations, this is more wishful thinking. It’s common for estimates by restoration contractors on larger jobs to be off by months or years, and vast amounts of money. If so mature an industry can’t manage reliable scheduling, what hope is there for us?)

                    Yet this isn’t car repair for old vehicles, where you find more and more problems as you go along, making estimates tough without thorough diagnostics, but the customer is happy with the substitute car you gave out in the meantime.

                    I’d argue that is exactly what much software development is like (except there is no substitute for the customer).

                    1. 1

                      Maybe I’d like to be more optimistic about learning to estimate better ;) But for sure @SeanTAllen touched on a lot of pertinent points. Is it Alice or Bob who gets the task? How well is the problem space, the code, known? And so on.

                      It’s hard as balls, and you’re not wrong with your gambler analogy, but not all systems for getting things right occasionally are equally unlikely to succeed. Renovators also learn what to look out for and how long those issues tend to take, as well as the interactions. Probabilities are usually ok by customers if that’s all you got.

                      In my car analogy, the point kinda was that we’re screwed because we can’t give out substitutes. We can deliver temporary hacks, although nothing is as permanent as a temporary hack.

                4. 1

                  A lot of activities in software development try to improve predictability. For example: following style guidelines, continuous integration, unit testing, etc. All of these have a cost and slow developers down. The upside of course is to reduce bugs which will slow you down much more later. Or maybe not. The risk is generally too high, so we generally prefer the predictability of a slow incremental process.

                  I have a feeling that Demings thought about that when he talked about “variation”, but I need to read more from the father of Lean and grandfather of Agile to understand that. Currently, I believe that I don’t assign quite the correct meaning to the words I read from him.

                1. 3

                  I am powerfully tempted to repartition one of my drives and give this a shot.

                  1. 5

                    Do it! :)

                    1. 1

                      I think you should

                    1. 13

                      Rich has been railing on types for the last few keynotes, but it looks to me like he’s only tried Haskell and Kotlin and that he hasn’t used them a whole lot, because some of his complaints look like complete strawmen if you have a good understanding and experience with a type system as sophisticated as Haskell’s, and others are better addressed in languages with different type systems than Haskell, such as TypeScript.

                      I think he makes lots of good points, I’m just puzzled as to why he’s seemingly ignoring a lot of research in type theory while designing his own type system (clojure.spec), and if he’s not, why he thinks other models don’t work either.

                      1. 14

                        One nit: spec is a contract system, not a type system. The former is often used to patch up a lack of the latter, but it’s a distinct concept you can do very different things with.

                        EDIT: to see how they can diverge, you’re probably better off looking at what Racket does than what Clojure does. Racket is the main “research language” for contracts and does some pretty fun stuff with them.

                        1. 4

                          It’s all fuzzy to me. They’re both formal specifications. They get overlapped in a lot of ways. Many types people are describing could be pre/post conditions and invariants in contract form for specific data or functions on them. Then, a contract system extended to handle all kinds of things past Boolean will use enough logic to be able to do what advanced type systems do.

                          Past Pierce or someone formally defining it, I don’t know as a formal, methods non-expert that contract and type systems in general form are fundamentally that different since they’re used the same in a lot of ways. Interchangeably, it would appear, if each uses equally powerful and/or automated logics.

                          1. 13

                            It’s fuzzy but there are differences in practice. I’m going to assume we’re using non-FM-level type systems, so no refinement types or dependent types for full proofs, because once you get there all of our intuition about types and contracts breaks down. Also, I’m coming from a contract background, not a type background. So take everything I say about type systems with a grain of salt.

                            In general, static types verify a program’s structure, while contracts verify its properties. Like, super roughly, static types are whether a program is sense or nonsense, while contracts are whether its correct or incorrect. Consider how we normally think of tail in Haskell vs, like, Dafny:

                            tail :: [a] -> [a]
                            
                            method tail(s: seq<T>) returns (o: seq<T>)
                            requires s.Len > 0
                            ensures s[0] + o = s
                            

                            The tradeoff is that verifying structure automatically is a lot easier than verifying semantics. That’s why historically static typing has been compile-time while contracts have been runtime. Often advances in typechecking subsumed use cases for contracts. See, for example, how Eiffel used contracts to ensure “void-free programming” (no nulls), which is subsumed by optionals. However, there are still a lot of places where they don’t overlap, such as in loop invariants, separation logic, (possibly existential contracts?), arguably smart-fuzzing, etc.

                            Another overlap is refinement types, but I’d argue that refinement types are “types that act like contracts” versus contracts being “runtime refinement types”, as most successful uses of refinement types came out of research in contracts (like SPARK) and/or are more ‘contracty’ in their formulations.

                            1. 3

                              Is there anything contacts do that dependent types cannot?

                              1. 2

                                Fundamentally? Not really, nor vice versa. Both let you say arbitrary things about a function.

                                In practice contracts are more popular for industrial work because they so far seem to map better to imperative languages than dependent types do.

                                1. 1

                                  That makes sense, thanks! I’ve never heard of them. I mean I’ve probably seen people throw the concept around but I never took it for an actual thing

                          2. 1

                            I see the distinction when we talk about pure values, sum and product types. I wonder if the IO monad for example isn’t kind of more on the contract side of things. Sure it works as a type, type inference algorithms work with it, but the sife-effect thing makes it seem more like a pattern.

                          3. 17

                            I’m just puzzled as to why he’s seemingly ignoring a lot of research in type theory

                            Isn’t that his thing? He’s made proud statements about his disinterest in theory. And it shows. His jubilation about transducers overlooked that they are just a less generic form of ad-hoc polymorphism, invented to abstract over operations on collections.

                            1. 1

                              wow, thanks for that, never really saw it that way but it totally makes sense. not a regular clojure user, but love lisp, and love the ML family of languages.

                              1. 1

                                So? Theory is useless without usable and easy implementation

                              2. 6

                                seemingly ignoring a lot of research in type theory

                                I’ve come to translate this utterance as “it’s not Haskell”. Are there languages that have been hurt by “ignoring type theory research”? Some (Go, for instance) have clearly benefited from it.

                                1. 12

                                  I don’t think rich is nearly as ignorant of Haskell’s type system as everyone seems to think. You can understand this stuff and not find it valuable and it seems pretty clear to me that this is the case. He’s obviously a skilled programmer who’s perspective warrants real consideration, people who are enamored with type systems shouldnt be quick to write him off even if they disagree.

                                  I don’t like dynamic languages fwiw.

                                  1. 3

                                    I dont think we can assume anything about what he knows. Even Haskellers here are always learning about its type system or new uses. He spends most of his time in a LISP. It’s safe to assume he knows more LISP benefits than Haskell benefits until we see otherwise in examples he gives.

                                    Best thing tl do is probably come up with lot of examples to run by him at various times/places. See what says for/against them.

                                    1. 9

                                      I guess I would want hear what people think he’s ignorant of because he clearly knows the basics of the type system, sum types, typeclasses, etc. The clojure reducers docs mention requiring associative monoids. I would be extremely surprised if he didn’t know what monads were. I don’t know how far he has to go for people to believe he really doesn’t think it’s worthwhile. I heard edward kmett say he didn’t think dependent types were worth the overhead, saying that the power to weight ratio simply wasn’t there. I believe the same about haskell as a whole. I don’t think it’s insane to believe that about most type systems and I don’t think hickey’s position stems from ignorance.

                                      1. 2

                                        Good examples supporting he might know the stuff. Now, we just need more detail to further test the claims on each aspect of languge design.

                                        1. 2

                                          From the discussions I see, it’s pretty clear to me that Rich has a better understanding of static typing and its trade offs than most Haskell fans.

                                  2. 10

                                    I’d love to hear in a detailed fashion how Go has clearly benefited from “ignoring type theory research”.

                                    1. 5

                                      Rust dropped GC by following that research. Several languages had race freedom with theirs. A few had contracts or type systems with similar benefits. Go’s developers ignored that to do a simpler, Oberon-2- and C-like language.

                                      There were two reasons. dmpk2k already said first, which Rob Pike said, that it was designed for anyone from any background to pick up easily right after Google hired them. Also, simplicity and consistency making it easy for them to immediately go to work on codebases they’ve never seen. This fits both Google’s needs and companies that want developers to be replaceable cogs.

                                      The other is that the three developers had to agree on every feature. One came from C. One liked stuff like Oberon-2. I dont recall the other. Their consensus is unlikely to be an Ocaml, Haskell, Rust, Pony, and so on. It was something closer to what they liked and understood well.

                                      If anything, I thought at the time they shouldve done something like Julia with a mix of productivity features, high C/Python integration, a usable subset people stick to, and macros for just when needed. Much better. I think a Noogler could probably handle a slighty-more-advanced language than Go. That team wanted otherwise…

                                      1. 2

                                        I have a hard time with a number of these statements:

                                        “Rust dropped GC by following that research”? So did C++ also follow research to “drop GC”? What about “C”? I’ve been plenty of type system conversation related to Rust but nothing that I would attribute directly to “dropping GC”. That seems like a bit of a simplification.

                                        Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared? I’ve seen Rob Pike talk about wanting to appeal to C and C++ programmers but nothing about ignorning type research. I’d be interested in hearing about that being done and what they thought the benefits were.

                                        It sounds like you are saying that the benefit is something familiar and approachable. Is that a benefit to the users of a language or to the language itself? Actually I guess that is more like, is the benefit that it made Go approachable and familiar to a broad swath of programmers and that allowed it to gain broad adoption?

                                        If yes, is there anything other than anecdotes (which I would tend to believe) to support that assertion?

                                        1. 9

                                          “That seems like a bit of a simplification.”

                                          It was. Topic is enormously complex. Gets worse when you consider I barely knew C++ before I lost my memory. I did learn about memory pools and reference counting from game developers who used C++. I know it keeps getting updated in ways that improve its safety. The folks that understand C++ and Rust keep arguing about how safe C++ is with hardly any argument over Rust since its safety model is baked thoroughly into the language rather than an option in a sea of options. You could say I’m talking about Rust’s ability to be as safe as a GC in most of an apps code without runtime checks on memory accesses.

                                          “Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared?”

                                          Like with the Rich Hickey replies, this burden of proof is backwards asking us to prove a negative. If assessing what people knew or did, we should assume nothing until we see evidence in their actions and/or informed opinions that they did these things. Only then do we believe they did. I start by comparing what I’ve read of Go to Common LISP, ML’s, Haskell, Ada/SPARK, Racket/Ometa/Rascal on metaprogramming side, Rust, Julia, Nim, and so on. Go has almost nothing in it compared to these. Looks like a mix of C, Wirth’s stuff, CSP like old stuff in 1970’s-1980’s, and maybe some other things. Not much past the 1980’s. I wasn’t the first to notice either. Article gets point across despite its problems the author apologized for.

                                          Now, that’s the hypothesis from observation of Go’s features vs other languages. Lets test it on intent first. What was the goal? Rob Pike tells us here with Moray Taylor having a nicer interpretation. The quote:

                                          The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                          It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.

                                          So, they’re intentionally dumbing the language down as much as they can while making it practically useful. They’re doing this so smart people from many backgrounds can pick it up easily and go right to being productive for their new employer. It’s also gotta be C-like for the same reason.

                                          Now, let’s look at its prior inspirations. In the FAQ, they tell you the ancestors: “Go is mostly in the C family (basic syntax), with significant input from the Pascal/Modula/Oberon family (declarations, packages), plus some ideas from languages inspired by Tony Hoare’s CSP, such as Newsqueak and Limbo (concurrency).” They then make an unsubstantiated claim, in that section at least, that it’s a new language across the board to make programming better and more fun. In reality, it seems really close to a C-like version of the Oberon-2 experience one developer (can’t recall) wanted to recreate with concurrency and tooling for aiding large projects. I covered the concurrency angle in other comment. You don’t see a lot of advanced or far out stuff here: decades old tech that’s behind current capabilities. LISP’ers, metaprogrammers and REBOL’s might say behind old tech, too. ;)

                                          Now, let’s look at execution of these C, Wirth-like, and specific concurrency ideas into practice. I actually can’t find this part. I did stumble upon its in-depth history of design decisions. The thing I’m missing, if it was correct, is a reference to the claim that the three developers had to agree on each feature. If that’s true, it automatically would hold the language back from advanced stuff.

                                          In summary, we have a language designed by people who mostly didn’t use cutting-edge work in type systems, employed nothing of the sort, looked like languages from the 1970’s-1980’s, considered them ancestors, is admittedly dumbed-down as much as possible so anyone from any background can use it, and maybe involved consensus from people who didn’t use cutting-edge stuff (or even much cutting-edge at 90’s onward). They actually appear to be detractors to a lot of that stuff if we consider the languages they pushed as reflecting their views on what people should use. Meanwhile, the languages I mentioned above used stuff from 1990’s-2000’s giving them capabilities Go doesn’t have. I think the evidence weighs strongly in favor of that being because designers didn’t look at it, were opposed to it for technical and/or industrial reasons, couldn’t reach a consensus, or some combo.

                                          That’s what I think of Go’s history for now. People more knowledgeable feel free to throw any resources I might be missing. It just looks to be a highly-practical, learn/use-quickly, C/Oberon-like language made to improve onboarding and productivity of random developers coming into big companies like Google. Rob Pike even says that was the goal. Seems open and shut to me. I thank the developers of languages like Julia and Nim believing we were smart enough to learn a more modern language, even if we have to subset them for inexperienced people.

                                      2. 4

                                        It’s easy for non-LtU programmers to pick up, which happens to be the vast majority.

                                        1. 3

                                          Sorry, that isn’t detailed. Is there evidence that its easy for these programmers to pick up? What does “easy to pick up” mean? To get something to compile? To create error-free programs? “Clearly benefited” is a really loaded term that can mean pretty much anything to anyone. I’m looking for what the stated benefits are for Go. Is the benefit to go that it is “approachable” and “familiar”?

                                          There seems to be an idea in your statement then that using any sort of type theory research will inherintly make something hard to pick up. I have a hard time accepting that. I would, without evidence, be willing to accept that many type system ideas (like a number of them in Pony) are hard to pick up, but the idea that you have to ignore type theory research to be easy to pick up is hard for me to accept.

                                          Could I create a language that ignores type system theory but using a non-familiar syntax and not be easy to pick up?

                                          1. 5

                                            I already gave you the quote from Pike saying it was specifically designed for this. Far as the how, I think one of its designers explains it well in those slides. The Guiding Principles section puts simplicity above everything else. Next, a slide says Pascal was a minimalist language designed for teaching non-programmers to code. Oberon was similarly simple. Oberon-2 added methods on records (think simpler OOP). The designer shows Oberon-2 and Go code saying it’s C’s syntax with Oberon-2’s structure. I’ll add benefits like automatic, memory management.

                                            Then, the design link said they chose CSP because (a) they understood it enough to implement and (b) it was the easiest thing to implement throughout the language. Like Go itself, it was the simplest option rather than the best along many attributes. There were lots of people who picked up SCOOP (super-easy but with overhead) with probably even more picking up Rust’s method grounded in affine types. Pony is itself doing clever stuff using advances in language. Go language would ignore those since (a) Go designers didn’t know them well from way back when and (b) would’ve been more work than their intent/budget could take.

                                            They’re at least consistent about simplicity for easy implementation and learning. I’ll give them that.

                                        2. 3

                                          It seems to me that Go was clearly designed to have a well-known, well-understood set of primitives, and that design angle translated into not incorporating anything fundamentally new or adventurous (unlike Pony and it’s impressive use of object capabilities). It looked already old at birth, but it feels impressively smooth, in the beginning at least.

                                          1. 3

                                            I find it hard to believe that CSP and Goroutines were “well-understood set of primitives”. Given the lack of usage of CSP as a mainstream concurrency mechanism, I think that saying that Go incorporates nothing fundamentally new or adventurous is selling it short.

                                            1. 5

                                              CSP is one of oldest ways people modeled concurrency. I think it was built on Hoare’s monitor concept from years before which Per Brinch Hansen turned into Concurrent Pascal. Built Solo OS with mix of it and regular Pascal. It was also typical in high-assurance to use something like Z or VDM for specifying main system with concurrency done in CSP and/or some temporal logic. Then, SPIN became dominant way to analyze CSP-like stuff automatically with a lot of industrial use for a formal method. Lots of other tools and formalisms existed, though, under banner of process algebras.

                                              Outside of verification, the introductory text that taught me about high-performance, parallel computing mentioned CSP as one of basic models of parallel programming. I was experimenting with it in maybe 2000-2001 based on what those HPC/supercomputing texts taught me. It also tied into Agent-Oriented Programming I was looking into then given they were also concurrent, sequential processes distributed across machines and networks. A quick DuckDuckGo shows a summary article on Wikipedia mentions it, too.

                                              There were so many courses teaching and folks using it that experts in language design and/or concurrency should’ve noticed it a long time ago trying to improve on it for their languages. Many did, some doing better. Eiffel SCOOP, ML variants like Concurrent ML, Chapel, Clay with Wittie’s extensions, Rust, and Pony are examples. Then you have Go doing something CSP-like (circa 1970’s) in the 2000’s still getting race conditions and stuff. What did they learn? (shrugs) I don’t know…

                                              1. 10

                                                Nick,

                                                I’m going to take the 3 different threads of conversation we have going and try to pull them all together in this one reply. I want to thank you for the time you put into each answer. So much of what appears on Reddit, HN, and elsewhere is throw away short things that often feel lazy or like communication wasn’t really the goal. For a long time, I have appreciated your contributions to lobste.rs because there is a thoughtfulness to them and an attempt to convey information and thinking that is often absent in this medium. Your replies earlier today are no exception.


                                                Language is funny.

                                                You have a very different interpretation of the words “well-understood primitives” than I do. Perhaps it has something to do with anchoring when I was writing my response. I would rephrase my statement this way (and I would still be imprecise):

                                                While CSP has been around for a long time, I don’t that prior to Go, that is was a well known or familiar concurrency model for most programmers. From that, I would say it isn’t “well-understood”. But I’m reading quite a bit, based on context into what “well-understood” means here. I’m taking it to me, “widely understood by a large body of programmers”.

                                                And I think that your response Nick, I think it actually makes me believe that more. The languages you mention aren’t ones that I would consider familiar or mainstream to most programmers.

                                                Language is fun like that. I could be anchoring myself again. I rarely ask questions on lobste.rs or comment. I decided to on this occasion because I was really curious about a number of things from an earlier statement:

                                                “Go has clearly benefited from “ignoring type theory research”.

                                                Some things that came to mind when I read that and I wondered “what does this mean?”

                                                “clearly benefited”

                                                Hmmm, what does benefit mean? Especially in reference to a language. My reading of benefit is that “doing X helped the language designers achieve one or more goals in a way that had acceptable tradeoffs”. However, it was far from clear to me, that is what people meant.

                                                “ignoring type theory research”

                                                ignoring is an interesting term. This could mean many things and I think it has profound implications for the statement. Does ignoring mean ignorance? Does it mean willfully not caring? Or does it mean considered but decided not to use?

                                                I’m familiar with some of the Rob Pike and Go early history comments that you referenced in the other threads. In particular related to the goal of Go being designed for:

                                                The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

                                                It must be familiar, roughly C-like. Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.

                                                I haven’t found anything though that shows there was a willful disregard of type theory. I wasn’t attempting to get you to prove a negative, more I’m curious. Has the Go team ever said something that would fall under the heading of “type system theory, bah we don’t need it”. Perhaps they have. And if they have, is there anything that shows a benefit from that.

                                                There’s so much that is loaded into those questions though. So, I’m going to make some statements that are possibly open to being misconstrued about what from your responses, I’m hearing.

                                                “Benefit” here means “helped make popular” because Go on its surface, presents a number of familiar concepts for the programmer to work with. There’s no individual primitive that feels novel or new to most programmers except perhaps the concurrency model. However, upon the first approach that concurrency model is fairly straightforward in what it asks the programmer to grasp when first encountering it. Given Go’s stated goals from the quote above. It allows the programmers to feel productive and “build good software”.

                                                Even as I’m writing that though, I start to take issue with a number of the assumptions that are built into the Pike quote. But that is fine. I think most of it comes down to for me what “good software” is and what “simple” is. And those are further loaded words that can radically change the meaning of a comment based on the reader.

                                                So let me try again:

                                                When people say “Go has clearly benefited from “ignoring type theory research” what they are saying is:

                                                Go’s level of popularity is based, in part, on it providing a set of ideas that should be mostly familiar to programmers who have some experience with the Algol family of languages such as C, C++, Python, Ruby etc. We can further refine that to say that from the Algol family of languages that we are really talking about ones that have type systems that make few if any guarantees (like C). That Go put this familiarity as its primary goal and because of that, is popular.

                                                Would you say that is a reasonable summation?


                                                When I asked:

                                                “Is there documentation that Go developers ignored type research? Has the Go team stated that? Or that they never cared?”

                                                I wasn’t asking you for to prove a negative. I was very curious if any such statements existed. I’ve never seen any. I’ve drawn a number of conclusions about Go based mostly on the Rob Pike quote you provided earlier. I was really looking for “has everyone else as well” or do they know things that I don’t know.

                                                It sounds like we are both mostly operating on the same set of information. That’s fine. We can draw conclusions from that. But I feel at least good in now saying that both you and I are inferring things based on what appears to be mostly shared set of knowledge here and not that I am ignorant of statements made by Go team members.

                                                I wasn’t looking for proof. I was looking for information that might help clear up my ignorance in the area. Related to my ignorance.

                                                1. 2

                                                  I appreciate that you saw I was trying to put effort into it being productive and civil. Those posts took a while. I appreciate your introspective and kind reply, too. Now, let’s see where we’re at with this.

                                                  Yeah, it looks like we were using words with a different meaning. I was focused on well-understood by PLT types that design languages and folks studying parallelism. Rob Pike at the least should be in both categories following that research. Most programmers don’t know about it. You’re right that Go could’ve been first time it went mainstream.

                                                  You also made a good point that it’s probably overstating it to say they never considered. I have good evidence they avoided almost all of it. Other designers didn’t. Yet, they may have considered it (how much we don’t know), assessed it against their objectives, and decided against all of it. The simplest approach would be to just ask them in a non-confrontational way. The other possibility is to look at each’s work to see if it showed any indication they were considering or using such techniques in other work. If they were absent, saying they didn’t consider it in their next work would be reasonable. Another angle would be to look at, like with C’s developers, whether they had a personal preference for simpler or barely any typing consistently avoiding developments in type systems. Since that’s lots of work, I’ll leave it at “Unknown” for now.

                                                  Regarding its popularity, I’ll start by saying I agree its simple design reusing existing concepts was a huge element of that. It was Wirth’s philosophy to do same thing for educating programmers. Go adopted that philosophy to modern situation. Smart move. I think you shouldn’t underestimate the fact that Google backed it, though.

                                                  There were a lot of interesting languages over the decades with all kinds of good tradeoffs. The ones with major, corporate backing and/or on top of advantageous foundations/ecosystems (eg hardware or OS’s) usually became big in a lasting way. That included COBOL on mainframes, C on cheap hardware spreading with UNIX, Java getting there almost entirely through marketing given its technical failures, .NET/C# forced by Microsoft on its huge ecosystem, Apple pushing Swift, and some smaller ones. Notice the language design is all across the board here in complexity, often more complex than existing languages. The ecosystem drivers, esp marketing or dominant companies, are the consistent thread driving at least these languages’ mass adoption.

                                                  Now, mighty Google claims they’re backing for their massive ecosystem a new language. It’s also designed by celebrity researchers/programmers, including one many in C community respect. It might also be a factor in whether developers get a six digit job. These are two, major pulls plus a minor one that each in isolation can draw in developers. Two, esp employment, will automatically make a large number of users if they think Google is serious. Both also have ripple effects where other companies will copy what big company is doing to not get left behind. Makes the pull larger.

                                                  So, as I think of your question, I have that in the back of my mind. I mean, those effects pull so hard that Google’s language could be a total piece of garbage and still have 50,000-100,000 developers just going for a gold rush. I think that they simplified the design to make it super-easy to learn and maintain existing code just turbocharges that effect. Yes, I think the design and its designers could lead to significant community without Google. I’m just leaning toward it being a major employer with celebrity designers and fanfare causing most of it.

                                                  And then those other languages start getting uptake despite advanced features or learning troubles (esp Rust). Shows they Go team could’ve done better on typing using such techniques if they wanted to and/or knew about those techniques. I said that’s unknown. Go might be the best they could do in their background, constraints, goals, or whatever. Good that at least four, different groups made languages to push programming further into the 90’s and 2000’s instead of just 70’s to early 80’s. There’s at least three creating languages closer to C generating a lot of excitement. C++ is also getting updates making it more like Ada. Non-mainstream languages like Ada/SPARK and Pony are still getting uptake even though smaller.

                                                  If anything, the choices of systems-type languages is exploding right now with something for everyone. The decisions of Go’s language authors aren’t even worth worrying about since that time can be put into more appropriate tools. I’m still going to point out that Rob Pike quote to people to show they had very, very specific goals which made a language design that may or may not be ideal for a given task. It’s good for perspective. I don’t know designers’ studies, their tradeoffs, and (given alternatives) they barely matter past personal curiosity and PLT history. That also means I’ll remain too willfully ignorant about it to clear up anyone’s ignorance. At least till I see some submissions with them talking about it. :)

                                                  1. 2

                                                    Thanks for the time you put into this @nickpsecurity.

                                                    1. 1

                                                      Sure thing. I appreciate you patiently putting time into helping me be more accurate and fair describing Go designers’ work.

                                                      1. 2

                                                        And thank you, I have a different perspective on Go now than I did before. Or rather, I have a better understanding of other perspectives.

                                      3. 6

                                        I don’t see anything of substance in this comment other than “Haskell has a great type system”.

                                        I just watched the talk. Rich took a lot of time to explain his thoughts carefully, and I’m convinced by many of his points. I’m not convinced by anything in this comment because there’s barely anything there. What are you referring to specifically?

                                        edit: See my perspective here: https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey#c_povjwe

                                        1. 3

                                          That wasn’t my point at all. I agree with what Rich says about Maybes in this talk, but it’s obvious from his bad Haskell examples that he hasn’t spent enough time with the language to justify criticizing its type system so harshly.

                                          Also, what he said about representing the idea of a car with information that might or might not be there in different parts of a program might be correct in Haskell’s type system, but in languages with structural subtyping (like TypeScript) or row polymorphism (like Ur/Web) you can easily have a function that takes a car record which may be missing some fields, fills some of them out and returns an object which has a bit more fields than the other one, like Rich described at some point in the talk.

                                          I’m interested to see where he’s gonna end up with this, but I don’t think he’s doing himself any favors by ignoring existing research in the same fields he’s thinking about.

                                          1. 5

                                            But if you say that you need to go to TypeScript to express something, that doesn’t help me as a Haskell user. I don’t start writing a program in a language with one type system and then switch into a language with a different one.

                                            Anyway, my point is not to have a debate on types. My point is that I would rather read or watch an opinion backed up by real-world experience.

                                            I don’t like the phrase “ignoring existing research”. It sounds too much like “somebody told me this type system was good and I’m repeating it”. Just because someone published a paper on it, doesn’t mean it’s good. Plenty of researchers disagree on types, and admit that there are open problems.

                                            There was just one here the other day!

                                            https://lobste.rs/s/dldtqq/ast_typing_problem

                                            I’ve found that the applicability of types is quite domain-specific. Rich Hickey is very clear about what domains he’s talking about. If someone makes general statements about type systems without qualifying what they’re talking about, then I won’t take them very seriously.

                                        2. 4

                                          I don’t have a good understanding of type systems. What is it that Rich misses about Haskells Maybe? Does changing the return type of a function from Maybe T to T not mean that you have to change code which uses the return value of that function?

                                          1. 23

                                            Does changing the return type of a function from Maybe T to T not mean that you have to change code which uses the return value of that function?

                                            It does in a way, but I think people sometimes over-estimate the amount of changes that are required. It depends on whether or not really really care about the returned value. Let’s look at a couple of examples:

                                            First, let’s look at an example. Let’s say that we had a function that was going to get the first element out of a list, so we start out with something like:

                                            getFirstElem :: [a] -> a
                                            getFirstElem = head
                                            

                                            Now, we’ll write a couple of functions that make use of this function. Afterwards, I’ll change my getFirstElem function to return a Maybe a so you can see when, why, and how these specific functions need to change.

                                            First, let’s imagine that I have some list of lists, and I’d like to just return a single list that has the first element; for example I might have something like ["foo","bar","baz"] and I want to get back "fbb". I can do this by calling map over my list of lists with my getFirstElem function:

                                            getFirsts :: [[a]] -> [a]
                                            getFirsts = map getFirstElem
                                            

                                            Next, say we wanted to get an idea of how many elements we were removing from our list of lists. For example, in our case of ["foo","bar","baz"] -> "fbb", we’re going from a total of 9 elements down to 3, so we’ve eliminated 6 elements. We can write a function to help us figure out how many elements we’ve dropped pretty easily by looking at the sum of the lengths of the lists in the input lists, and the overall length of the output list.

                                            countDropped :: [[a]] -> [b] -> Int
                                            countDropped a b =
                                              let a' = sum $ map length a
                                                  b' = length b
                                              in a' - b'
                                            

                                            Finally, we probably want to print out our string, so we’ll use print:

                                            printFirsts =
                                              let l = ["foo","bar","baz"]
                                                  r = getFirsts l
                                                  d = countDropped l r
                                              in print l >> print r >> print d
                                            

                                            Later, if we decide that we want to change our program to look at ["foo","","bar","","baz"]. We’ll see our program crashes! Oh no! the problem is that head doesn’t work with an empty list, so we better go and update it. We’ll have it return a Maybe a so that we can capture the case where we actually got an empty list.

                                            getFirstElem :: [a] -> Maybe a
                                            getFirstElem = listToMaybe
                                            

                                            Now we’ve changed our program so that the type system will explicitly tell us whether we tried to take the head of an empty list or not- and it won’t crash if we pass one in. So what refactoring do we have to do to our program?

                                            Let’s walk back through our functions one-by-one. Our getFirsts function had the type [[a]] -> [a] and we’ll need to change that to [[a]] -> [Maybe a] now. What about the code?

                                            If we look at the type of map we’ll see that it has the type: map :: (c -> d) -> [c] -> [d]. Since both [[a]] -> [a] and [[a]] -> [Maybe a] satisfy the constraint [a] -> [b], (in both cases, c ~ [a], in the first case, d ~ a and in the second d ~ Maybe a). In short, we had to fix our type signature, but nothing in our code has to change at all.

                                            What about countDropped? Even though our types changed, we don’t have to change anything in countDropped at all! Why? Because countDropped is never looking at any values inside of the list- it only cares about the structure of the lists (in this case, how many elements they have).

                                            Finally, we’ll need to update printFirsts. The type signature here doesn’t need to change, but we might want to change the way that we’re printing out our values. Technically we can print a Maybe value, but we’d end up with something like: [Maybe 'f',Nothing,Maybe 'b',Nothing,Maybe 'b'], which isn’t particularly readable. Let’s update it to replace Nothing values with spaces:

                                            printFirsts :: IO ()
                                            printFirsts =
                                              let l = ["foo","","bar","","baz"]
                                                  r = map (fromMaybe ' ') $ getFirsts' l
                                                  d = countDropped l r
                                              in print l >> print r >> print d
                                            

                                            In short, from this example, you can see that we can refactor our code to change the type, and in most cases the only code that needs to change is code that cares about the value that we’ve changed. In an untyped language you’d expect to still have to change the code that cares about the values you’re passing around, so the only additional changes that we’ve had to do here was a very small update to the type signature (but not the implementation) of one function. In fact, if I’d let the type be inferred (or written a much more general function) I wouldn’t have had to even do that.

                                            There’s an impression that the types in Haskell require you to do a lot of extra work when refactoring, but in practice the changes you are making aren’t materially more or different than the ones you’d make in an untyped language- it’s just that the compiler will tell you about the changes you need to make, so you don’t need to find them through unit tests or program crashes.

                                            1. 3

                                              countDropped should be changed. To what will depend on your specification but as a simple inspection, countDropped ["", "", "", ""] [None, None, None, None] will return -4, which isn’t likely to be what you want.

                                              1. 2

                                                That’s correct in a manner of speaking, since we’re essentially computing the difference between the number of characters in all of the substrings minutes the length of the printed items. Since [""] = [[]], but is printed " ", we print one extra character (the space) compared to the total length of the string, so a negative “dropped” value is sensible.

                                                Of course the entire thing was a completely contrived example I came up with while I was sitting at work trying to get through my morning coffee, and really only served to show “sometimes we don’t need to change the types at all”, so I’m not terribly worried about the semantics of the specification. You’re welcome to propose any other more sensible alternative you’d like.

                                                1. -3

                                                  That’s correct in a manner of speaking, since …

                                                  This is an impressive contortion, on par with corporate legalese, but your post-hoc justification is undermined by the fact that you didn’t know this was the behavior of your function until I pointed it out.

                                                  Of course the entire thing was a completely contrived example …

                                                  On this, we can agree. You created a function whose definition would still typecheck after the change, without addressing the changed behavior, nor refuting that in the general case, Maybe T is not a supertype of T.

                                                  You’re welcome to propose any other more sensible alternative you’d like.

                                                  Alternative to what, Maybe? The hour long talk linked here is pretty good. Nullable types are more advantageous, too, like C#’s int?. The point is that if you have a function and call it as f(0) when the function requires its first argument, but later, the requirement is “relaxed”, all the places where you wrote f(0) will still work and behave in exactly the same way.

                                                  Getting back to the original question, which was (1) “what is it that Rich Hickey doesn’t understand about types?” and, (2) “does changing the return type from Maybe T to T cause calling code to break?”. The answer to (2) is yes. The answer to (1), given (2), is nothing.

                                                  1. 9

                                                    I was actually perfectly aware of the behavior, and I didn’t care because it was just a small toy example. I was just trying to show some examples of when and how you need to change code and/or type signatures, not write some amazing production quality code to drop some things from a list. No idea why you’re trying to be such an ass about it.

                                                    1. 3

                                                      She did not address question (1) at all. You are reading her response to question (2) as implying something about (1) that makes your response needlessly adverse.

                                                2. 1

                                                  This is a great example. To further reinforce your point, I feel like the one place Haskell really shows it’s strength in these refactors. It’s often a pain to figure out what the correct types should be parts of your programs, but when you know this and make a change, the Haskell compiler becomes this real guiding light when working through a re-factor.

                                                3. 10

                                                  He explicitly makes the point that “strengthening a promise”, that is from “I might give you a T” to “I’ll definitely give you a T” shouldn’t necessarily be a breaking change, but is in the absence of union types.

                                                  1. 2

                                                    Half baked thought here that I’m just airing to ask for an opinion on:

                                                    Say as an alternative, the producer produces Either (forall a. a) T instead of Maybe T, and the consumer consumes Either x T. Then the producer’s author changes it to make a stronger promise by changing it to produce Either Void T instead.

                                                    I think this does what I would want? This change hasn’t broken the consumer because x would match either alternative. The producer has strengthened the promise it makes because now it promises not to produce a Left constructor.

                                                    1. 4

                                                      When the problem is “I can’t change my mind after I had insufficient forethought”, requiring additional forethought is not a solution.

                                                      1. 2

                                                        So we’d need a way to automatically rewrite Maybe t to Either (forall a. a) t everywhere - after the fact. ;)

                                                4. 2

                                                  Likewise, I wonder what he thinks about Rust’s type system to ensure temporal safety without a GC. Is safe, no-GC operation in general or for performance-critical modules desirable for Clojure practitioners? Would they like a compile to native option that integrates that safe, optimized code with the rest of their app? And if not affine types, what’s his solution that doesn’t involve runtime checks that degrade performance?

                                                  1. 7

                                                    I’d argue that GC is a perfectly fine solution in vast majority of cases. The overhead from advanced GC systems like the one on the JVM is becoming incredibly small. So, the scenarios where you can’t afford GC are niche in my opinion. If you are in such a situation, then types do seem like a reasonable way to approach the problem.

                                                    1. 3

                                                      I have worked professionally in Clojure but I have never had to make a performance critical application with it. The high performance code I have written has been in C and CUDA. I have been learning Rust in my spare time.

                                                      I argue that both Clojure and Rust both have thread safe memory abstractions, but Clojure’s solution has more (theoretical) overhead. This is because while Rust uses ownership and affine types, Clojure uses immutable data structures.

                                                      In particular, get/insert/remove for a Rust HashMap is O(1) amortized while Clojure’s corresponding hash-map’s complexity is O(log_32(n)) for those operations.

                                                      I haven’t made careful benchmarks to see how this scaling difference plays out in the real world, however.

                                                      1. 4

                                                        Having used clojure’s various “thread safe memory abstractions” I would say that the overhead is actual not theoretical.

                                                  2. 2

                                                    Disclaimer: I <3 types a lot, Purescript is lovely and whatnot

                                                    I dunno, I kinda disagree about this. Even in the research languages, people are opting for nominal ADTs. Typescript is the exception, not the rule.

                                                    His wants in this space almost require “everything is a dictionary/hashmap”, and I don’t think the research in type theory is tackling his complaints (the whole “place-oriented programming” stuff and positional argument difficulties ring extremely true). M…aybe row types, but row types are not easy to use compared to the transparent and simple Typescript model in my opinion.

                                                    Row types help o solve issues generated in the ADT universe, but you still have the nominal typing problem which is his other thing.

                                                    His last keynote was very agressive and I think people wrote it off because it felt almost ignorant, but I think this keynote is extremely on point once he gets beyond the maybe railing in the intro

                                                  1. 3

                                                    “When pursuing a vertical scaling strategy, you will eventually run up against limits. You won’t be able to add more memory, add more disk, add more “something.” When that day comes, you’ll need to find a way to scale your application horizontally to address the problem.”

                                                    I should note the limit last I checked was SGI UV’s for data-intensive apps (eg SAP Hana) having 64 sockets with multicore Xeons, 64TB RAM, and 500ns max latency all-to-all communication. I swore one had 256 sockets but maybe misremembering. Most NUMA’s also have high-availability features (eg “RAS”). So, if it’s one application per server (i.e. just DB), many businesses might never run into a limit on these things. The main limit I saw studying NUMA machines was price: scaling up cost a fortune compared to scaling out. One can get stuff in low-to-mid, five digits now that previously cost six to seven. Future-proofing scale up by starting with SGI-style servers has to be more expensive, though, than scale-out start even if scale-out starts on a beefy machine.

                                                    You really should modify the article to bring pricing up. The high price of NUMA machines was literally the reason for inventing Beowful clusters which pushed a lot of the “spread it out on many machines” philosophy towards mainstream. The early companies selling them always showed the price of eg a 64-256 core machine by Sun/SGI/Cray vs cluster of 2-4 core boxes. First was price of mini-mansion (or castle if clustered NUMA’s) with second ranging from new car to middle-class house. HA clustering goes back further with VMS, NonStop, and mainframe stuff. I’m not sure if cost pushed horizontal scaling for fault-tolerance to get away from them or if folks were just building on popular ecosystems. Probably a mix but I got no data.

                                                    “The number of replicas, also known as “the replication factor,” allows us to survive the loss of some members of the system (usually referred to as a “cluster”). “

                                                    I’ll add that each could experience the same failure, esp if we’re talking attacks. Happened to me in a triple, modular redundancy setup with a single component faulty. On top of replication, I push hardware/software diversity much as one’s resources allow. CPU’s built on different tools/nodes. Different mobo’s and UPS’s. Maybe optical connections if worried about electrical stuff. Different OS’s. Different libraries if they perform identical function. Different compilers. And so on. The thing that’s the same on each node is one app you’re wanting to work. Even it might be several written by different people with cluster having a mix of them. The one thing that has to be shared is the protocol for starting it all up, syncing the state, and recovering from problems. Critical layers like that should get the strongest verification the team can afford with SQLite and FoundationDB being the exemplars in that area.

                                                    Then, it’s really replicated in a fault-isolating way. It’s also got a lot of extra failure modes one has to test for. Good news is several companies and/or volunteers can chip in each working on one of the 3+ hardware/software systems. Split the cost up. :)

                                                    1. 2

                                                      Those are planned subjects for future posts. I wanted to keep this one fairly simplistic for folks who aren’t experts in the area.

                                                      1. 1

                                                        That makes sense. Thanks for clarifying.

                                                    1. 5

                                                      I rarely get hyped by a talk, but this one definitely spoke to me. This is awesome work!

                                                      1. 3

                                                        I spent the day beforehand trying to get Edwin to change his talk to be about the rules of Cricket. Probably best that he ignored me.

                                                        1. 1

                                                          As an american that has looked glancingly at cricket rules, I am happy this was ignored. >.<

                                                      1. 7

                                                        One thing I would love to read more about is how to determine the cutoff between scaling horizontally and investing time/money in optimizing your software.

                                                        1. 2

                                                          Oooo, that’s a good one. I have a couple hand-wavey heuristics. I’ll think more about trying to turn that into something more real.

                                                          I have the next 2-3 posts vaguely planned out. But, I’ll definitely be thinking about your idea @hwayne.

                                                          1. 1

                                                            Doesn’t it boil down to, basically, “it’s a knack, spend 10 years doing it and you’ll get good at judgment”?

                                                            1. 2

                                                              I’d definitely take a more “business” driven approach to that problem. Optimizing your software for cost should only be done if the money it saves is superior to what it costs to optimize it.

                                                              You also have to take into account indirect costs like when using weird optimization tricks can make code less readable sometimes and also has a cost for future development.

                                                              1. 1

                                                                On the other hand, scaling horizontally adds costs of coordinating multiple servers, which includes load balancing, orchestrating deployments, distributed system problems, etc.

                                                                1. 1

                                                                  The business-driven approach is not always the best for society as a whole. It doesn’t take into account negative externalities like the environmental cost of running inefficient programs and runtimes.

                                                                2. 2

                                                                  I had a few that helped a lot:

                                                                  1. Use the fastest components. Ex: Youtube used lighttpd over Apache.

                                                                  2. If caching can help, use it. Try different caching strategies.

                                                                  3. If it’s managed, use a system language and/or alternative GC.

                                                                  4. If fast path is small, write the data-heavy part in an optimizable language using best algorithms for that. Make sure it’s cache and HD-layout friendly. Recent D submission is good example.

                                                                  5. If it’s parallelizable, rewrite the fast path in a parallel, programming language or using such a library. Previously, there was PVM, MPI, Cilk, and Chapel. The last one is designed for scale-up, scale-out, and easy expression simultaneously. Also, always use a simpler, lighter solution like that instead of something like Hadoop or SPARK if possible.

                                                                  6. Whole-program, optimizing compilers (esp profile-guided) if possible. I used SGI’s for this at one point. I’m not sure if LLVM beats all of them or even has a profile-guided mode itself. Haven’t looked into that stuff in a while.

                                                                  Notice that most of this doesn’t take much brains. A few take little to no time either. They usually work, too, giving anything from a small to vast improvement. So, they’re some of my generic options. Via metaprogramming or just good compiler, I can also envision them all integrated into one language with compiler switches toggling the behavior. Well, except choosing fastest component or algorithm. Just the incidental stuff.

                                                            1. 4

                                                              I worked on parts of this release and there’s some neat things, including a streamlined Python API and the ability to run Connectors in Python 3. If you have any questions about this release or you want to talk about stream processing in Python please post something here.

                                                              1. 1

                                                                Not about this, but more generally: what’s the best introduction to why I should wallaroo rather than any other distributed processing doobry?

                                                                Also why isn’t there a pony api?

                                                                1. 3

                                                                  Ugh, I typed in a nice long answer and lost it when I hit the back button. Blarg. This version will be shorter.

                                                                  There is a Pony API but its “unsupported” meaning we don’t document it. Documenting it would mean time spent on upkeep. If people start requesting the Pony API, that could change.

                                                                  Re: why use Wallaroo vs something else. We don’t have anything specifically comparing Wallaroo to other systems. There’s a lot of good Wallaroo related material on the blog: https://blog.wallaroolabs.com/

                                                                  The number #1 reason that people are interested in Wallaroo’s Python API is they are interested in doing streaming with Python, don’t want to be tied to a specific ingest source, and aren’t interested in the overhead that comes from using a system that is written in Java and involves shuffling data back and forth between Python and Java processes.

                                                                  However, there are other reasons that people have as well. If you have specific use cases for streaming that you are interested in, I’d be happy to discuss the pros and cons of Wallaroo for those use-cases.

                                                              1. 15

                                                                To me, this is more of a grab bag of random things rather than a well thought out argument for uniqueness. I still enjoyed it.

                                                                And it did make me think that I’d love to read an article that is about what made BeOS unique at the time (from one of us who was part of the community then) and also what makes Haiku unique now (from someone who is part of the community now). I’d love to read that article (or two articles if the authors worked together to make them a nice pairing).

                                                                The what made BeOS unique might be harder because you’d need to give a lot of background around why some things were revolutionary then for not just the tech but the effect that are fairly commonplace feeling now.

                                                                Anyway, enough babbling.

                                                                1. 5

                                                                  I was part of the BeOS community almost 2 decades ago, pretty much from the time they ported it to x86.

                                                                  A few things that made it unique at the time: 64 bit filesystem that could be queried like a database; incredible support for multiprocessing and low-latency that allowed for professional quality media editing apps; an API that made programmers want to write software for it even when there weren’t that many users.

                                                                1. 2

                                                                  I’m curious. Why?

                                                                  I wouldn’t think to compare D and Nim much less put in the effort to maintain a comparison repo.

                                                                  Color me curious. Anyone have any insight?

                                                                  1. 4

                                                                    They’re both C++ replacements that claim to be better. They have some feature overlap. A comparison makes sense given either could become someone’s general-purpose language for new projects.

                                                                    There’s fans of each on the site. So, I figured I throw it on here in case they wanted to play with a new language this weekend.

                                                                    1. 3

                                                                      Interesting. I’ve done some Nim in the past. Never considered it as a C++ replacement.

                                                                      1. 3

                                                                        They told me it had ability to do unsafe things, integrate well with C, and metaprogramming for customization. Anything with these can potentially be a C++ replacement. If I was misinformed about unsafe support, then it woudnt be as suited to this task. Unlike C++, it also spits out C.

                                                                  1. 3

                                                                    This looks really interesting but whooo I bet confluent is not happy about this

                                                                    1. 4

                                                                      They’re apparently publicly upset about this.

                                                                      “They’re profiting from our use of resources,” Kreps said. “We’re charging our customers for the service, paying for compute and network resources that we use.”

                                                                      https://twitter.com/kmassada/status/1068553502455037952

                                                                      Unfortunately I can’t find the original source.

                                                                    1. 2

                                                                      I didn’t watch to the end, but is there any movement on adding the pthread/thread local optimization to speed things up?

                                                                      1. 1

                                                                        Not that I am aware of

                                                                      1. 5

                                                                        There is an informative series or articles from one of the Midori authors which includes object-capabilities as provided/defined/enforced by the kernel, see Midori: Objects as Secure Capabilities.

                                                                        1. 2

                                                                          Joe’s series is awesome. Highly recommended.

                                                                        1. 9

                                                                          cf Ponylang for an implementation of this model: https://tutorial.ponylang.io/capabilities/

                                                                          1. 7

                                                                            Pony also has capabilities on external resources. For example, you need a capability to open a socket.

                                                                            1. 3

                                                                              This seems very similar to Rust’s borrowing model, and both of these are excellent!

                                                                              However, capabilities with regards to variables is only one part of the picture.

                                                                              I very much want my programs to be capability safe in terms of external resources too: the filesystem, exec, network access, etc. Functions and modules which require access to these would need to be provided them in order to utilize it.

                                                                              Monte implements this and has interesting demonstrations like https://monte.readthedocs.io/en/latest/taste.html#cooperation-without-vulerability which can safely eval() code from the internet. It doesn’t prevent an abusive user from using all the RAM or all the CPU, but it is impossible for the abusive user to read RAM, write to the FS, send spam emails, etc.

                                                                              EDIT 1 Additionally, capability safety doesn’t even have to apply to a single program! TahoeLAFS is a capability-safe networked filesystem.

                                                                            1. 4

                                                                              Besides a couple that are already mentioned (like Alan Kay and Rands)…

                                                                              • Hernán Wilkinson. He radically changed my view on what my job (and now my passion) is. He also lead me to the discovery of other great role models (like Alonso Church, the father of lambda calculus). Hernán and his pals have and promote my favorite way to organize a company. If you have some time to spare, you should check how 10Pines works.
                                                                              • Robert Virding. I had the amazing luck to work with him, teaching one of the languages he has created (Erlang). He is such a smart but down-to-earth fellow, he transpiles that inspiring feeling of I’m no special, you could totally do what I did. Just put yourself into it.
                                                                              • Jordan Mechner. I could never understood (even after reading the books) how two floppy disks could contain such an inmersive experience as Prince of Persia. I’m not a game developer nor a UI expert by any means, but I always admired the talent of people that can do so much with so limited resources.
                                                                              1. 3

                                                                                I was fortunate enough to be Hernán’s boss for a couple years and we’ve remained friends.

                                                                              1. 6

                                                                                I’m giving a Pony tutorial at Code Sync London on Wednesday

                                                                                1. 1

                                                                                  This is one of my favorite papers. I’ve done a talk built around the ideas in it and have referenced it in multiple other talks.

                                                                                  1. 1

                                                                                    It looks pretty awesome. Learned about “data lineage” as a term to use searching. If you or @hwayne look for others, use the terms “provenance” and “taint tracking.” Most stuff like in this paper I saw in past used one of those in conjunction with “information flow” analysis or control.

                                                                                    Ill do some searches on data lineage later on to see if anything interestimg turns up.

                                                                                  1. 33

                                                                                    All this talk about ethics, open, and free brings another angle to mind: people pushing that with no-cost licenses are themselves misrepresenting what they are achieving in at least U.S.. I used to push for OSS/FOSS in the past. Now I’m switching to hybrids. The reason is that encouraging people to play “give it all away” or “use low-revenue models” in a capitalist country where opponents of freedom make billions of dollars for their software shifted all the money (and therefore power) to the latter. They then paid off politicians and used pricey lawyers to win more power against OSS/FOSS in ways they couldn’t fight against without piles of money. This includes ability to patent/copyright troll users of open/free software and especially Oracle’s API ruling which jeopardizes OSS/FOSS, backwards-compatible implementations of anything that had a proprietary API.

                                                                                    From what I see, OSS/FOSS have done great things but are a fundamentally-flawed model in a capitalist country where money wins. As many as possible need to be charging by default both to support contributors and send money/power the other way. They and FOSS-using companies that don’t depend on patent/copyright money need to pool money together to fight legal advances of patent/copyright-trolling companies that want lock-in. Otherwise, in a game where only one side is playing for keeps, the OSS/FOSS groups keep losing by default software freedoms and ability to enforce their licenses while preaching that they’re maintaining them. Seems dishonest. Also, strange I almost never read about these issues in FOSS writers articles about business model and licensing recommendations.

                                                                                    Far as hybrids, I can’t give you the answer yet since it’s too soon. For FOSS, I’m looking at Open Core and Dual-Licensing with strongest copyleft available. For non-FOSS, Source-available from public-benefit companies and nonprofits chartered to implement most software freedoms for customers on top of free for non-commercial or under certain use. These freedoms and justifications would also be in licenses and contracts with huge penalties for non-compliance for extra layers of defense. Maybe expire into FOSS after certain time passes or revenue threshold. We need more experimentation that lets companies currently supplying or later releasing as FOSS to get millions to hundreds of millions in revenue collectively to fight this battle. Again, it’s not optional: the other side is actively fighting to remove software freedom inch by inch. And mostly winning despite FOSS organizations’ little victories.

                                                                                    1. 5

                                                                                      Apologies if this is a threadjack, but I’m wrestling with these kinds of questions. I’ve been doing open source more or less my whole career, and usually in some form of hybrid.

                                                                                      Now I’m going out on my own and am searching for a model that makes sense. I like the collaboration of open source, but I also very much want to make money, and don’t want to do that with hallucinogenic business models.

                                                                                      I’m building a game that has a music synthesizer in it as a core mechanic. What I’m building has basically 3 layers - infrastructure for building such things in Rust, the synthesizer itself (with GUI), and the game logic on top. What I’m converging on is doing the first two layers as very much community open source with permissive licenses, and the third layer as just straight up proprietary software, no pretending to be anything else. There’s stuff to fine-tune around the edges, for example somebody brought up a delayed open release of the game source, after the monetization has run its course, but I don’t want to commit to that right now because it might constrain working with a commercial publisher. If I end up self-publishing, I’ll strongly consider that though, especially if people tell me it helps motivate their contribution.

                                                                                      1. 3

                                                                                        I think you should flip your plans on business models around this code - the infrastructure and synthesizer are the things that could have enough value to a business that you could do well selling them. The median game does not make a profit and, as entertainment, games are a hit-driven (usually-)one-time purchase not bought based on predictable need.

                                                                                        1. 4

                                                                                          I’ve certainly thought about it. But here’s my thinking. First, there’s currently no business for Rust infrastructure, the community is very much organized around permissive licenses. Second, the market for synthesizers and music plugins is pretty crowded, while games in this particular genre are, as far as I can tell, underserved. Third, I think the Switch is a promising platform, and it doesn’t really do free games. Fourth, if the game is a dud but the free music tools catch on, I can always do a pro version, and I get free marketing and market research. Lastly, the game is definitely riskier, but I’m at a point where I’m ok with that; if this stuff doesn’t monetize, I just go back to a corporate job.

                                                                                        2. 1

                                                                                          I think it’s pretty obvious that the ‘correct’ way of doing free software games without violating the freedom of users by making anything proprietary is to make all the code free software but not make the art/music/etc. free. After all, software freedom is about software, not about art or music.

                                                                                          People can modify the software, they can use it as they see fit, but they can’t redistribute it along with the art and music. They can either come up with their own art and music or they can redistribute it without the art and music and it’ll be useful to others that already have the game (because they already have the art and music).

                                                                                          1. 1

                                                                                            Your model (bottom two free, top not) was exactly what first popped into my mind when I read the first couple of sentences of your comment, so you’re not the only one who thinks it makes sense, fwiw :-)

                                                                                            By the way (off-topic question), as someone who has recently bought a midi controller (MPK261) and started playing around with some of the synths that I got free (Hybrid Air, Sonivox), and has a decent mathy ability to understand any given synthesis concept, but absolutely no intuition for what changes will sound like… is your game aimed at me? :-)

                                                                                            1. 2

                                                                                              Yes, it’s made for exactly you :) I’ll put you on my list for beta testing.

                                                                                              1. 1

                                                                                                OMG that’s fantastic!

                                                                                            2. -1

                                                                                              why don’t you want to deal LSD

                                                                                              1. 1

                                                                                                Beg your pardon?

                                                                                                1. 3

                                                                                                  hallucinogenic business models

                                                                                                  1. 4

                                                                                                    Ah, right, right. Basically I want to create value honestly and do a reasonable job of recovering revenue from the value I create, rather than playing these games that seem to increasingly substitute for that these days.

                                                                                                    1. 1

                                                                                                      what games are you referring to?

                                                                                                      1. 2

                                                                                                        Financial engineering in general, more specifically the kinds of things that startups do when they’re looking for an exit or when their purpose is to burn through VC money rather than make a business. MoviePass, Juicero, that kind of thing.

                                                                                                        1. 1

                                                                                                          ah okay. in reading your original comment i thought you were saying it’s hallucinogenic to think you could make a living writing free software.

                                                                                            3. -5

                                                                                              This would make sense if these ‘attacks on free software’ actually existed, but they don’t. They literally don’t exist.

                                                                                              Source available is a violation of user freedom. It’s unacceptable. That’s all there is to it, if you care about user freedom. If you don’t then I feel sorry for you.

                                                                                              FLOSS doesn’t need ‘hundreds of millions in revenue’ to fight any battle because there is no battle. I don’t know what it is, but there seems to be a recurring thread I see on forums a lot recently: everything is framed as a battle. For an unrelated example, if a game developer makes an unpopular change to their game? It’s a WAR to get them to fix it. No it isn’t. I see the term ‘culture war’ thrown around too. There’s no such thing. Not everything in life is a war or a battle.

                                                                                              People and groups of people that produce non-free software aren’t at war with people that do.

                                                                                              1. 7

                                                                                                “This would make sense if these ‘attacks on free software’ actually existed, but they don’t. They literally don’t exist.” “FLOSS doesn’t need ‘hundreds of millions in revenue’ to fight any battle because there is no battle.”

                                                                                                You missed the war on open source software by Microsoft et al… thwarted largely by IBM saying it will drop a fortune defending Linux which fits my comment… the DMCA attacks, the patent trolling (Android vendors alone pay billions), the copyright ruling from an expensive case that applies to API’s FOSS often depends on, and so on. Hell, a patent defense alone facing one of these big companies gets quoted as about $200,000 on average from expensive, law firms. You bet FOSS folks need a fortune if one of these companies wants to use a legal team to destroy them.

                                                                                                You really don’t see such things much, though. You wonder why if there’s a threat as I claim. They mostly ignore FOSS developers since they’re (a) free labor that the big companies are monetizing or (b) penniless opponents with weak or non-existent marketing to big spenders. For (b), the common MO is to hit companies building on the product, FOSS or not, for royalties once they’re financially successful. They parasite off them instead of destroy while using a slice of that money investing in lobbyists and courts to ensure they can continue. Alternatively, they use a combo of the lure of money with the threat of market destruction (patent or otherwise) to pressure them to sell the company. Microsoft and IBM have taken entire markets using their dominant positions, patent portfolios, and large[-for-small-player] offerings to get acquisitions. It’s rare for someone to stare at both that kind of money and corporate threat telling them to get lost. So…

                                                                                                “People and groups of people that produce non-free software aren’t at war with people that do.”

                                                                                                …the big players wanting to dominate markets, maximize profits from locked-in customers, and eliminate disruptive competition are always at war with folks producing anything that threatens that. Always. It’s never changed since they have people on top whose bonuses depend on this shit. They’ll also remember companies that were sent to non-existence or limbo from new stuff they didn’t stamp out when they had a chance. Just because you or many FOSS folks aren’t playing doesn’t mean they aren’t. They certainly are. They even have people working 24/7 on Capitol Hill to screw people over. Not just here in states: they’re represented in international, treaty negotiations as well under the I.P. agreements. And if you think copyright enforcement isn’t war or this is just U.S., just look at the nice, unarmed, law-abiding people that came after Kim Dotcom over U.S.-generated complaints. Those situations illustrates the heights things can go to when it matters to those in power. Better to be the ones with power.

                                                                                                1. -5

                                                                                                  You missed the war on open source software by Microsoft et al…

                                                                                                  Yes, we get it. This happened. A long time ago, now, but it happened. Okay, move on. It’s no longer relevant. The corporate world has long since embraced free software.

                                                                                                  the DMCA attacks, the patent trolling (Android vendors alone pay billions), the copyright ruling from an expensive case that applies to API’s FOSS often depends on, and so on.

                                                                                                  Patent trolling is something very specific. All companies that own patents and enforce them are not automatically patent trolls.

                                                                                                  Also patent trolls target companies that produce proprietary software just as much as they target companies that produce free software. It has nothing to do with free or proprietary software. They target both, because they target software in general.

                                                                                                  …the big players wanting to dominate markets, maximize profits from locked-in customers, and eliminate disruptive competition are always at war with folks producing anything that threatens that.

                                                                                                  No they aren’t.

                                                                                                  just look at the nice, unarmed, law-abiding people that came after Kim Dotcom

                                                                                                  If you think Kim Dotcom was law-abiding you have another thing coming mate.

                                                                                                  1. 7

                                                                                                    You are very wrong.

                                                                                                    I am trying to bring some more floss to the public sector and the resistance from the entrenched vendors is a thing.

                                                                                                    Straight up bribes are manageable, but lobbying and regulatory capture is the true evil. We actually need floss lobbyists of our own.

                                                                                                    1. 6

                                                                                                      You are twisting @nickpsecurity’s words and I do not understand why you are doing this.

                                                                                                      1. 1

                                                                                                        No I am not.

                                                                                                  2. 3

                                                                                                    You’re right that there isn’t a war as such, but there is definitely a certain kind of dynamic. I think @nickpsecurity is also pointing out (quite correctly) the wider implications of the wealth concentration resulting from the free work that goes into producing free software.

                                                                                                    One of the issues is that the companies that use free software don’t always comply with the terms of the licence. There are many examples:

                                                                                                    The problems are obvious unless you stick to a very narrow and dogmatic view. Consequences matter. Just like the ill-considered, short-sighted, damn-the-consequences technological “disruption”, free software is an idea that produced a lot of unintended side effects (and in fact, contributed to the aforementioned “disruption”).

                                                                                                1. 10

                                                                                                  The post by a16z explains it well: https://a16z.com/2014/02/14/why-there-will-never-be-another-redhat-the-economics-of-open-source/

                                                                                                  Elastic vs Algolia is a nice case study here as well. Let’s see which one ends up doing better long term.

                                                                                                  Problem with open source is that it’s hard to monetize. In most cases if something is hard to monetize it results in poor product quality. Of course you can point to large open source projects which are excellent, but that’s really the exception not the rule. Relying on people contributing free work to a project just doesn’t work in most cases.

                                                                                                  It also doesn’t help that the open source community has some negative feelings towards anything commercial. That culture is also pushing a lot of software from open source to closed solutions. This post is a great example calling Redis Labs shady for wanting to have a way to make money. Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                                  The only thing this post achieves is convincing more companies to stop with the whole open source thing.

                                                                                                  1. 15

                                                                                                    This post is a great example calling Redis Labs shady for wanting to have a way to make money.

                                                                                                    Isn’t he just calling Redis Labs shady for calling a non-free license free? He even specifically states “You have every right to license your work in any way you choose.”. That doesn’t smell rabidly anti-commercial to me.

                                                                                                    Monetizing open-source is not an easy task, though, that’s true. But the antidote to difficulty should not be dishonesty. Perhaps the honest thing would be to just stop trying to monetize it, and generate the income from some other thing?

                                                                                                    1. 20

                                                                                                      This post is a great example calling Redis Labs shady for wanting to have a way to make money. Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                                      You’re being dishonest. I’m calling them shady for calling their software open source when it’s not. I think that engineers deserve to be paid for their work but that doesn’t make it okay to lie about the licensing of your software.

                                                                                                      1. 8

                                                                                                        Standardized open-source tools/infrastructure are what make modern software products (open and closed) possible. The “exceptions” you mention represent a huge share of critical business systems (Linux, many widely deployed databases, web and other network servers, TLS/security infrastructure, web frameworks, most widely used language implementations, etc.).

                                                                                                        Problem with open source is that it’s hard to monetize. In most cases if something is hard to monetize it results in poor product quality.

                                                                                                        If you mean that “the standard of quality for software is low”, that seems easy to agree on. Asserting that closed-source software is somehow higher quality doesn’t hold water - see the list above and contrast with any of their closed-source competitors from 10-20 years ago.

                                                                                                        Making money is hard, whether you’re closed source or open source. TFA is just asserting that being deliberately misleading is bad.

                                                                                                        1. 4

                                                                                                          Problem with open source is that it’s hard to monetize.

                                                                                                          Not really. The problem with commons is that it’s hard to sell permits when everyone can just help themselves to come in. The mitigation has been known for a long time. Fund it from the taxes. You are represented, so the commons you care about should be maintained.

                                                                                                          In other words, want to research? Get a grant. Still not good enough? Maybe it’s time to acknowledge that floss movement should be represented better and start pushing.

                                                                                                          Here we have a Pirate party. I personally know an elected representative. He shares my opinions on floss.

                                                                                                          Have you talked to your ER?

                                                                                                          1. 1

                                                                                                            In other words, want to research? Get a grant.

                                                                                                            I thought about that. Unfortunately, the funding authorities for academic research are about quantity of papers instead of quality or code attached. Many will actually either give no credit for implementations or criticize the people doing them since it takes away from papers that bring in more money. An example was Antti Kantee of Rump Kernels telling me how they didn’t care that he actually built one to go with the paper. They just wanted the paper.

                                                                                                            Now, there’s institutions that do build software to go along with their papers. Usually just prototypes they throw together but it’s something at least. Some build software outside of their papers. Some of those like Racket are pretty polished. Getting into one of those institutions might allow a person to use tax dollars to write open-source software. Much smaller number of positions than academia as a whole. Some percentage might also be doing it in their spare time on top of a full day’s work as an academic, though.

                                                                                                            1. 2

                                                                                                              As a library, we get small grants for improvements as well as software, but the situation is far from ideal. We can also get our hands on EU grants, but those do require some serious administration.

                                                                                                              I have met the Racket team during their stay in Prague and I am still amazed at how good people they actually manage to fund to work on the PLT projects.

                                                                                                              I think that it should be much easier to get funding for open R&D. EU still measures improvements using the number of patents granted, though.

                                                                                                          2. 1

                                                                                                            Funny enough the post is written by an engineer who also thinks he should get paid for his work.

                                                                                                            I would like to upvote this more. Sadly, I can’t.

                                                                                                            1. 1

                                                                                                              I appreciate the domain name.