1. 4

    Previous crustacean post on the sorry story. Baishakhi Ray is now an assistant professor at Columbia

    1. 1

      A comment from the post links to Intro to Empirical Software Engineering. Insane talk!

    1. 17

      Replacing statistics by clueless button pushing, aka machine learning, would be a disaster.

      I understand that statistics is unpopular: it requires people to think and carefully collect lots of data. But it can be used to provide insight about how things work.

      Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.

      Machine learning in SE research is a train wreck

      Software effort estimation is mostly fake research

      1. 4

        I understand that statistics is unpopular: it requires people to think and carefully collect lots of data. But it can be used to provide insight about how things work.

        That’s not why statistics are different from ML methods. The term you’re looking for is “explainability”, the ability for an inferred model to actually explain the situation being observed to the practitioner. You’re strawmanning statistics and ML while trying to make your point.

        Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.

        Could you please actually respond to the article instead of grinding your axe? Or, just don’t respond at all. I think axe grinding just decreases the quality of commentary on the forum. Not answering and not voting is a way to deal with topics that you don’t have a substantive critique for but you still dislike.

        1. 3
          1. The links you provided are non-specific whining related to… ahm, some talks about ML at UCL and software effort estimation research is wrong. I’m unsure how this is at all related to the OP.

          Replacing statistics by clueless button pushing, aka machine learning, would be a disaster. There is no hard line between the two, but IF you grant that people are clueless about stat models, which they are, we might as well switch to a paradigm that generates a more powerful model that doesn’t require people to understand the model to begin with in order to ensure valid results.

          Machine learning is a black box. Chuck in lots of data, push some buttons and sprout clueless techno-babble.

          So are statistical models, unless you understand them, in which case they are obviously outright wrong in a lot of cases, and thus they are worst than a black box, they are a fake model of reality (see, for example: assuming everything fits a Gaussian or making binaries/multiclass issues out of continuous problems). So if people are already taking a black-box approach, we might as well useless biased black boxes that generate more powerful models with validation techniques that don’t require strong assumptions.

          On the whole, I’d love to actually understand your issues with the content, other than the title, because based on the posts you linked I don’t think our views of “proper research” are actually very different.

          1. 3

            Using statistics does not guarantee that the model is correct. No technique guarantees correctness.

            The appropriate adage is: “All models are wrong, but some are useful”.

            1. 3

              Exactly, which is why any modelling effort requires:

              1. An explicit target (What does “useful” mean?)
              2. The best model possible to achieve that target

              The statistical approaches a lot of fields currently use don’t do either very well, hence, the article.

        1. 3

          This article leaves me uneasy between the broad brush paints against not-sure-what and pushing from one fallacy (scientists don’t know enough stats, the system is rigged with stupid indices to evaluate research (p-value)) to another one (highly complex models without any control or understand but that may or not predict something based another set of indices). I find it dishonest to shout out only social sciences where medecine and biomedical sciences are at the heart of the replication crisis. The first list about the differences between “classical statistics” and ML seems false based on how I learned. Cross-validation was always part of the models evaluation, [experimental design] is utterly essential knowledge to assure the analysis of “Aan experiment (in generally : How, why and by whom your data are created is essential knowledge to identify biais at the origin of the dataset).

          One really good book to better understand statistics for anyone is Statistics done wrong by Alex Reinhart.

          First quote learned about models and the only to always keep in mind “All models are wrong, but some are useful” attributed to G. Box.

          1. 1

            There is a replication crisis in all areas of research. It just so happens that Medicine and biomedical researchers are the first to make a major fuss about the issue, and start doing something about it.

            The replication crisis is motivated by the low status of performing replications.

            Software engineering is a long way away from a replication crisis, because so few experiments are done in the first place.

          1. 11

            GPL’ed header files are now holed below the waterline.

            1. 8

              So is windows.h. That’s probably a good thing, over all.

              1. 1

                windows.h has been independently re-derived a number of times, making it a nonissue regardless.

                1. 3

                  How are those cases different from what Google did with Java?

                  1. 5

                    Copyright protects your work from being copied by others. If I look at the windows.h that microsoft distributes and use that as a basis for my own windows.h, it’s possible that what I produce constitutes a derivative work that thus infringes on microsoft’s copy rights. (This is what google v oracle was about.)

                    However, if I derive the windows api in the context of a clean room, the result is not a derivative work and thus not infringing. This, for instance, is what the wine and reactos projects have done, and wine expressly prohibits people who might have had contact with windows source code from contributing for this reason.

                    1. 1

                      What if you clean room derive a 100% identical header?

                      (Not impossible if the original, say, didn’t have comments and was formatted with a tool like clang-format which you’ve also decided to use…)

                      1. 3

                        clean room derive a 100% identical header?

                        Then that’s fine, as long as you didn’t copy it from the original.

                        1. 1

                          Or better put, where do you draw a line between clean room based development output vs an actual system you are converting?

                      2. 2

                        The work that Oracle claims is copyrighted in this case is the “Structure Sequence and Organization” of the APIs, which is a term they created that somehow includes things like names.

                        I presume Moonchild’s point is that people have reversed the windows ABI into an API many times, and the ABI doesn’t include the requisite elements to be copyrightable even if Oracle is right that their API is copyrightable. I’m not sure I agree with that argument, but it is an interesting one.

                  2. 2

                    Shit, hadn’t thought of that. Good catch.

                    1. 2

                      Even if they’re C++ libraries where everything is in the header file?

                      (You just know some court’s going to be unable or unwilling to distinguish between declarations and code.)

                      1. 1

                        I don’t know, this is a massive fair use question! You would have to show that there’s broad knowledge of the header file contents in particular.

                        Java being Java is where the fair use conclusion came in. Millions of people knowing The Java API definitions! I think it’s non controversial to state that Java APIs are more universally known than almost anything under the sun

                        And this case in the outset is like “look this is complicated and fast moving, we’re going to state stuff here but don’t consider it broad in scope”. Pretty tough fight to win IMO

                      1. 16

                        scc is a pretty neat tool to count the lines of code; it also prints the COMOCO estimation by default.

                        Here’s a small side-project project I worked on for perhaps two weeks (full time) at the most (probably shorter):

                        Estimated Cost to Develop $1,282,396
                        Estimated Schedule Effort 15.116892 months
                        Estimated People Required 7.536604
                        

                        $1.2 million!

                        Now, this is cheating a lot since this includes the Unicode dataset as a Go file which gets counted. But even if I exclude that:

                        Estimated Cost to Develop $64,479
                        Estimated Schedule Effort 4.852825 months
                        Estimated People Required 1.180440
                        

                        5 months and $65k for 2 weeks? Not bad. Either this tool is wildly inaccurate or I must be one of those 10x developers I heard so much about! I’d prefer it if it was the latter, but it’s probably the former.

                        The main project I’ve been working on for the last year or so:

                        Estimated Cost to Develop $978,057
                        Estimated Schedule Effort 13.638056 months
                        Estimated People Required 6.371295
                        

                        This is also wildly inaccurate; the time estimate is roughly accurate, but 6.3 people? lolno. Just me. And this doesn’t even include all the parts I released as separate libraries (16 of them) which aren’t counted in here.

                        COCOMO is a joke, and even the most casual of glances will show that. A bloody horoscope would probably be more accurate most of the time.

                        1. 6

                          tbf if 10x programmers exist you’re probably one of them

                          EDIT: or more precisely, if someone asked me “are there any 10x programmers on lobsters”, I’d think of you first

                          1. 3

                            I am not sure about 10x programmers, but I have seen 0.1x programmers several times. Like that time they have submitted an app that was totally broken and printed “dick” on the stdout. Then they asked us to pay.

                            1. 3

                              If you have seen 0.1x programmers, then you have seen 10x ones - as I understand it, the “10x” term was coined to describe the difference between the worst and the best performers, not between the average and the best ones.

                              1. 1

                                It’s a different way of looking at it though; instead of saying “this is normal and these are exceptionally good”, it’s “this is normal, and these are exceptionally bad”.

                                Given the supply/demand of programmers, these people keep landing jobs. But in a healthier work force a lot of them probably wouldn’t. I’ve worked with some people that were quite frankly just hopeless, but not quite hopeless enough to not get nothing done at all; just at a 0.1× rate (or even lower).

                            2. 2

                              I wish. Until somebody comes up with an effective technique for measuring programmer performance, we can all live in hope.

                              My experience is that there is a step change between those who can and those who cannot. Measuring within those who can is difficult because of all the interrelated factors involved.

                              1. 1

                                You don’t need to have an exact measurement to give a roughly accurate estimation.

                              2. 1

                                Ow, you’re making me blush 😅 I’m really not that exceptionally smart or good; probably about average. There’s probably something in attitude that makes a bit of a difference somehow, but I’m not entirely sure what.

                              3. 3

                                Fitting the data shows that one of the published models is wrong, and there are relatively large error bounds on the other models (which is to be expected for a model built from 63 data points)..

                              1. 3

                                Excuse the plug. I offer a free analysis of software estimation data, provided I can make an anonymized version of the data public.

                                Here is one I did earlier.

                                1. 2

                                  There are small number of historians of computing, and they can only write about so many topics.

                                  There is no reason that over time more historians will write the kind of History that Knuth finds interesting. But at the moment this is not the fashion .

                                  The other issue is the availability of data, and the expertise needed to evaluate it.

                                  At the moment the histories are rather bland and essentially a list of events tied together. More pointed views are rarely heard.

                                  1. 1

                                    The Language Construction Kit and The Color of Magic

                                    1. 1

                                      The Language Construction Kit is a great overview of how languages are put together, and shows science fiction authors having a better grasp of the subject that many of those ‘designing’ their own language.

                                    1. 2

                                      An article from 2009? Here is an update from 2013.

                                      I’m sure there are more recent articles.

                                      1. 3

                                        I don’t have any data to hand, but until a bit more than a year ago I worked at a smaller cloud provider and I recall the DDR4 failure rates were even higher than DDR3. We had ECC, so we would at least find out – machines would conk out across the fleet due to uncorrectable failures way more than was acceptable. Nothing about the trajectory of DRAM reliability inspires any confidence at all.

                                        But what is clear is one ought to ignore anybody pushing the idea that ECC isn’t the absolute minimum level of protection needed. Run fast and far away.

                                      1. 5

                                        Kate is the one editor that I always thought could-have-been. It is the most elegant and modern looking of all editors I used back between 2010-2015. But it neither had the hackability of Emacs (my daily go-to) or the plugin ecosystem of atom/sublime/vscode that came later. Even now, it ticks most of the boxes - looks great, runs natively on Windows, Mac and Linux. But tthe community behind it just does not exist

                                        1. 2

                                          and it’s close enough to vi[m] to be useful, but sufficiently different to be annoying.

                                        1. 2

                                          diff a column at a time and merge results to give a row by row diff?

                                          1. 5

                                            I took a handful of extremely good classes in grad school that were life-changing for me … specifically an Advanced Computer Security seminar where I met people and did projects that helped me get where I am today. However, 80% of it was a waste, just classes I didn’t really care about that were required for the program. I’d find those couple of specific classes that are really important to you, excel in them, and don’t worry about a whole degree.

                                            (I went during the last recession, when the best jobs I could get paid surprisingly poorly, so the opportunity cost was relatively low. Today things are different!)

                                            Sadly, there’s good evidence that companies negatively select against MS degree holders: https://blog.alinelerner.com/how-different-is-a-b-s-in-computer-science-from-a-m-s-in-computer-science-when-it-comes-to-recruiting .

                                            1. 3

                                              I have felt the same about my own university experience and have heard others say similar things too. Effectively to the tune of “one or a handful of classes were life changing but the rest wasn’t really useful”. The only issue is, how would you have taken that class if it weren’t for the required curriculum of the degree program? I keep coming back to the question of “How would we discover such life changing courses on their own and outside of academia?”

                                              1. 7

                                                The world is full of nonsense research and papers; developing effective techniques for finding the gold nuggets is a requirement for success in any field.

                                            1. 1

                                              Until somebody invents a time machine it will never be a solved problem.

                                              1. 16

                                                The paper obviously isn’t using “solved” to mean “will predict correctly 100% of the time”. The paper points out that even given the constraint that we don’t have a time machine, there are things which could be done to improve accuracy, and lays out some possible new approaches to research.

                                                You presumably know this though, which leaves me wondering why you left the comment you did.

                                                1. 2

                                                  I guess people like being bitter for no real reason.

                                                  Paper looks interesting.

                                                  1. 1

                                                    How do you know they were being bitter?

                                                    And, if they were being bitter, how do you know they didn’t have a reason?

                                                    And, if they were being bitter for no reason, how do you know they liked it?

                                                    1. 0

                                                      You sound pretty bitter; do you like it?

                                                      1. 1

                                                        Thanks for checking in :) ‘Bitter’ definitely doesn’t fit where I was at when I wrote that comment, or how I’m feeling now. I’ve been feeling optimistic and curious while posting in this thread.

                                                        The questions I asked aren’t rhetorical; I’d actually like readers to try answering them.

                                              1. 1

                                                Suggest add release tag, remove science tag. Possibly add practices tag.

                                                Congratulations on your hard work!!!

                                                1. 6

                                                  Why remove the science tag? Seems like this book is precisely about applying scientific methodologies to the question of how to best write good code; I.e. it is itself an attempt to do science.

                                                  1. 4

                                                    The material is science oriented, given that no engineering tag is available this looked the closest fit.

                                                    The intent is to create future practices.

                                                    1. 2

                                                      Also do you have a way of receiving cash money for doing this? I’d like to reward it even though I don’t have a lot to spare at the moment.

                                                      1. 4

                                                        Thanks for your offer. I am lucky enough not to be in need of money (unless you’re offering $$$ which would allow me to run some experiments).

                                                        What you can do for me is to keep your eyes open for human software engineering data, and let me know if you find anything that looks interesting. There is lots of source available on github and people are busy measuring it, but data on the human side of things is hard to find.

                                                        1. 1

                                                          What do you mean by human side? This one maybe:

                                                          E. Kalliamvakou, C. Bird, T. Zimmermann, A. Begel, R. DeLine and D. M. German, “What Makes a Great Manager of Software Engineers?,” in IEEE Transactions on Software Engineering, vol. 45, no. 1, pp. 87-106, 1 Jan. 2019, doi: 10.1109/TSE.2017.2768368.

                                                          Microsoft has published quite a few studies. For example, look through Nachi Nagappans publications.

                                                          1. 3

                                                            I cite six of Nagappan’s papers.

                                                            Evidence-based papers (or empirical research as it is known in academia) will contain plots and tables (if you are lucky there will be an appendix listing the data). A surprising number of papers talk-the-talk but don’t actually contain any data.

                                                            Difficult to obtain human data includes:

                                                            • effort estimation, one I did earlier. A companies anonymised Jira data would be great (I do offer a free data analysis service if companies will let me make the data public).

                                                            • usage data. How much time do people spend using software. This kind of data is ultra-rare.

                                                            • experimental data using professional developers. This tends to get published and so I am likely to know about this data. But if you find an obscure report someplace, let me know.

                                                            Not, I repeat NOT surveys of developers. The vast majority are fake research. There are a handful of good ones out there, but I don’t want to waste other people’s time looking at survey data.

                                                          2. 1

                                                            $$$

                                                            OK so I can’t buy your book.

                                                            Can I get this book as a book? Failing that, can I get this book as an ebook? (not a PDF)

                                                            May I pay you for either of those goods / services?

                                                            1. 2

                                                              I’m investigating a dead tree version. Color and paper size limit the possibilities, along with me perhaps taking too much advantage of margin whitespace.

                                                              Generating an ebook would be a lot of work.

                                                              1. 1

                                                                ebook

                                                                FWIW, I came to ask about an e-book too.

                                                                Regardless, thank you for publishing the book freely!

                                                                1. 1

                                                                  Checkout the experimental mobile friendly pdf.

                                                                  Does this make life easier?

                                                                  1. 1

                                                                    I’ve just tried that on my android phone using xodo and it is certainly easier to read than a full-width format PDF. (I like xodo because it can auto-crop margins).

                                                                    I had been hoping to load it onto my kindle and have the text reflow, but the narrow PDF is a good compromise. Thank you.

                                                                    1. 1

                                                                      I tried it on my iPhone and the page size / font size was far too small to be read comfortably. Like @edd said, my biggest hope is for reflow.

                                                                      I’ll give it a go on my laptop!

                                                                      1. 2

                                                                        This topic is being discussed in issue 16 of the book’s Github page, which is probably more appropriate than this comments page.

                                                                        1. 1

                                                                          Cheers! I’ll follow the issue.

                                                                  2. 1

                                                                    I’d like a dead tree version too. Since I like to have a copy on my bookshelf, I don’t mind paying $50-100 if it looks nice enough and has quality paper that will last (although less is always nice).

                                                                    I’m happy to help you investigate and plan as I’m interested in a general way to print these electronic versions that isn’t like the poor quality I’ve seen in Amazon’s print to order fiction books.

                                                                    As a quality reference, I’m thinking of those Barnes and noble classics [0] or the stripe press hardcovers [1].

                                                                    [0] https://www.google.com/search?q=barnes+and+noble+hardcover+classics&client=safari&hl=en-us&prmd=sinv&sxsrf=ALeKk03PrUxhtQlAaCyMdmA8mFaldP_uPg:1605276708018&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjr6cnf2f_sAhUHKK0KHV21BQ4Q_AUoAnoECAMQAg&biw=375&bih=553&dpr=2

                                                                    [1] https://www.google.com/search?q=stripe+press+books&tbm=isch&ved=2ahUKEwjv3N3g2f_sAhXJi60KHRbZAZ8Q2-cCegQIABAC&oq=stripe+press&gs_lcp=ChJtb2JpbGUtZ3dzLXdpei1pbWcQARgBMgIIADIECAAQGDIECAAQGDIECAAQGDIECAAQGDoFCAAQzQI6BAgAEEM6BQgAELEDOgYIABAFEB46BggAEAgQHlDrlgJYk6YCYJOxAmgCcAB4AIABdIgBzQqSAQM5LjWYAQCgAQGwAQDAAQE&sclient=mobile-gws-wiz-img&ei=JpSuX--xGsmXtgWWsof4CQ&bih=553&biw=375&client=safari&prmd=sinv&hl=en-us

                                                                    1. 2

                                                                      The more people checking out the option the sooner a solution is found :-)

                                                                      There was a lobste.rs discussion on this last year.

                                                                      The issues are color, paper size and number of pages.

                                                                      IngramSpark look like they can handle the job. Lulu cannot handle it, neither can Amazon. Have not checked any other printers recently.

                                                                      I’m going to order a single copy from them, to see what the quality is like. First I have to get the pdf into a form they can process, i.e., pdf/a format

                                                                  3. 1

                                                                    Would a pdf version with shorter lines help? I could probably generate one of those without too much effort.

                                                                    Are there guidelines for pdf friendly formats for mobiles?

                                                            1. 2

                                                              Great reference! Thank you very much!

                                                              How long does it take to compile the book? R plots can surely be parallelized but then it is one huge TeX document?

                                                              1. 4

                                                                The book’s workflow is rather involved. I have just timed the LaTeX part and was surprised to see it took 15 seconds (it felt longer). Everything else is much longer. Generating figures from R takes less than an hour (never timed it), but there are 600+ of them (one takes several hours and I rarely build it).

                                                                The real time sink is checking layout. LaTeX users will be familiar with the way that figure ‘float’ around.

                                                              1. 3

                                                                I’ve been waiting for this for a while. I’m already reading it and it’s full of quotables (at least the beginning). Great job.

                                                                I’m also considering to set up a study group about it if it lives up to the hype.

                                                                1. 5

                                                                  Hype? Whoever my invisible (to me) marketeer is, keep up the good work!

                                                                  1. 2

                                                                    I guess I’ve heard about the project years ago and was waiting for it to come out. It might also be a fake memory. The introduction reinforced the hype because it’s a topic I’m very keen on. Not much disproving the myths, but the role of rituals, myths and social structures in shaping both the process and the result of technological production. Clearly your book opens a can of worms that was there waiting for a work like this.

                                                                1. 12

                                                                  If I may be so superficial for a moment, I love the cover! It feels like a combination of medieval manuscripts, 1970’s American cartoons, and Godel Escher Bach. I never would have expected to see that combination of things together. Delightful!

                                                                  1. 7

                                                                    Bosch’s The Garden of Earthly Delights was a suggestion included in the original list of ideas I sent the artist

                                                                  1. 1

                                                                    Designing languages is a vanity activity, no research involved (although there lots of claims about power, usability get made, supported by ego and bluster).

                                                                    1. 2

                                                                      You’re right, of course. At least, more than half right in my personal estimation, and I’ve spent some time wandering in the PL research mines. The places where you’re wrong are the really interesting places, but they are small islands in a blustery sea of dubious, overstated, generally uncontested claims. Bad money drives out good, as they say. Or, bathwater drives out babies? Something like that.

                                                                      But your definition of “research” is so narrow that it’s at odds with standard usage, sorry to say. Science is just very political and ego driven, and “computer science” is barely even science. Industry has good reasons to mostly ignore it.

                                                                      1. 1

                                                                        I think that slightly less than all would be more accurate.

                                                                        I’m interested to know what you think are the islands. Always on the look out for real research.

                                                                        1. 1

                                                                          Ah. Well, I can’t really promise that my own idea of “interesting” PL research is at all equivalent to your idea of “real” PL research, but here’s a wee handful of influential ideas, off the top of my head:

                                                                          • structured and procedural programming (FORTRAN, ALGOL, Pascal… etc)
                                                                          • “objects” (Simula, Smalltalk, a few commercially successful languages you may have heard of)
                                                                          • actor semantics for parallel computation (Act 1, CSP, Pi-calculus, Erlang…)
                                                                          • logic programming
                                                                          • algebraic data types
                                                                          • Hindley-Milner type checking

                                                                          … I think I’d better stop before I get carried away. All began as academic projects, just like basically all the underlying ideas in our immature little field that didn’t get inherited from our neglectful parents, electrical engineering and mathematical logic. Academia may innovate very little relative to all the hot air it produces, but industry simply does not innovate, as a rule. (It sometimes refines, though, and that’s valuable. Also valuable as a great proving ground, although there are some deep problems with that too; for example, like Kuhn observed, epistemic change happens on generational time scales.)

                                                                          1. 1

                                                                            These are interesting ways to think about programming.

                                                                            Is interesting their only benefit? If half of them never happened, how much difference would it make in practice?

                                                                      2. 1

                                                                        Much has been gained by the study of PL design, the most impactful one is the notion of Safety and Soundness. There is so much more, but I won’t enumerate them all.

                                                                        Are you saying that all the languages that need to be, already exist? Would you want to dissuade someone from designing a new language?

                                                                        1. 1

                                                                          There has been very little meaningful study of PL design, just lots of proof by ego and bluster.

                                                                          Yes, it is possible to prove stuff about suitably restrictive languages. The only impact of this research appears to have been advancing the careers of those involved.

                                                                          If somebody wants to invent a new language, that is their business. But let’s not pretend it’s anything other than a vanity activity, unless they do experimental research to back up any claims of usability, readability, maintainability, etc.

                                                                      1. 1

                                                                        The paper does not come with any data to download :-(

                                                                        “However, beyond the observation that the three variables are positively associated, the strength of the associations and their precise relationships are a matter of open debate and controversy in the research community.”

                                                                        One solution is for the test community to learn how to do statistical analysis that is more powerful that the t-test. Don’t mean to pick on you, this statement can be said of virtually all researchers in software engineering.

                                                                        1. 1

                                                                          We will have the data available soon. Will keep you updated.

                                                                          1. 1

                                                                            Actual link: http://www.coding-guidelines.com/c89.tgz

                                                                            Also:

                                                                            Unless you happen to have an AT&T 3b2 and know which options to give nroff, you are very unlikely to be able to generate something that looks like C89.