1. 8

    Single use, tiny languages are my catnip. I actually have to keep the temptation to learn them in check.

    1. 3

      If you check out the documentation section of this site there is a DSL for rolling different dice and visualizing the distributions. It’s the most niche DSL I think I’ve encountered.

      https://anydice.com/

      1. 3

        That’s pretty interesting! Some of the basic features aren’t surprising: being able to combine dice, sequences of rolls, etc., is what I’d expect from someone making a dice-rolling DSL. But this has a whole programming language pretty clearly included too, complete with looping and function calls. It’s even seemingly DIY syntax, not exactly lifted from an existing language. I wonder what the history is. Did these features accrete over time due to being needed in modeling some specific dice-roll situations? Was it designed up front as such a full-featured DSL?

        1. 3

          I don’t know what the history is, but I thought it was fascinating when I stumbled upon it. I was trying to simulate dice rolls for D&D. (What’s the distribution of 1d20s vs 2d10s? How would you do advantage/disadvantage for 2d10s? Etc.) It was very much an “everything you try to do someone else has done 100x better” situation. I was happy with my ASCII command line charts and this guy over here made an entire programming language dedicated to this one problem.

    1. 19

      I always find it kind of funny that Google didn’t want generics in Go, but hacked it into one of their projects: https://github.com/google/gvisor/blob/master/tools/go_generics/generics.go

      1. 26

        There are many people in google, surely not all teams agree on this issue.

        1. 16

          There’s also a bit of a difference between an implementation that works for one project and The Solution built into the language, which is supposed to (ideally) be a choice in the design space that’s fairly good across a broad range of uses. There are a bunch of solutions that work for various definitions of work, but the ones I’ve read all still have weird warts and edge cases (but different ones). Besides lower-level questions like how they interact with other parts of Go syntax and semantics, there’s also still quite a bit of disagreement, I think, on which of the basic tradeoffs in generics design are right for Go. Some people want Go generics more along the lines of ML-style parametric polymorphism or Java-style generics, while others want something more like C++ or D-style templates that instantiate specialized versions per-type. (The implementation linked above looks like the 2nd type.) Probably only one of these mechanisms will end up in the core language, not both.

          Edit: This “problem overview” from August 2018 has more details on the tradeoffs and current draft proposal.

          1. -2

            And yet apparently the language was shitty enough to force them to do this?

        1. 8

          This is an interesting article, but surprised they didn’t dig into the other failure modes shown in the US DoE chart (59% for non-LED electronics).

          I very much suspect the electrolytic capacitors on the driver inputs - 25,000 hours is a long time for a can full of electrolyte to be dealing with AC ripple and heat cycles.

          Also, I wonder if anyone has researched this aspect:

          A quick calculation says that the old bulbs paid for themselves more than three times over in electricity savings relative to the incandescents they replaced, and put that much less carbon into the atmosphere. They may well continue to burn for another 15,000 hours, but after weighing the degraded output and the cost to replace them with brighter, more efficient versions, I’m headed back to the store.

          … from the perspective of embodied energy (& related carbon emissions). Incandescents use more power when plugged in but have a lot less stuff in them, so I would guess they use less total energy to make. Although they’re maybe less recyclable as well…?

          1. 4

            Although they’re maybe less recyclable as well…?

            I believe current de-facto practice is that LEDs and incadescent bulbs are both mostly treated as general waste. LEDs are tricky to recycle in a cost-effective manner, though as they become a bigger part of the waste stream and more standardized, something might be solved there.

            1. 2

              That’s depressing. Although the “less than 2% of the waste stream” number in the linked article seems encouraging for longevity so far…

          1. 15

            This post is entitled “Why Don’t People Use Formal Methods?”, but I feel it’s maybe underselling isself. It does explain why people don’t use formal methods, in the author’s opinion. But it also explains why you might use formal methods in the first place — better than most other sources I’ve seen.

            1. 2

              GNU tar has an incremental format also, which should be long-term supported. It does things on a per-file basis though, rather than diffing files, so wouldn’t be suitable if you regularly have large binary files that undergo small changes.

              1. 2

                Yeah I looked at that. Avoiding it for two reasons:

                • I don’t want to keep around the “listed-incremental” file
                • It’s only supported by GNU tar
              1. 15

                In addition to all the resume reviews, I’ll add that your location strategy might be working against you as well, namely living in the SF Bay Area but applying to Seattle

                In Feb 2017, I was living in Portland, decided to move back to SF, and started my job search only for SF. I was interviewing for 3 months finishing the move in May 2017. I had a few onsites which involved flying but the response rate always felt low for the initial application

                After moving back, it felt like the floodgates opened back up. I had the same amount of onsites in those 3 months in 1 month

                The message here is:

                • Recruiters might be filtering against people who aren’t local (reasoning might include interview process taking longer, no desire to pay relocation, might assume you improperly applied) so conditions might improve once you move
                • Applying to Seattle instead of SF Bay Area might be working against you
                1. 5

                  Some companies assume relocation assistance would need to be factored in for non-local candidates (if the position isn’t remote friendly), and thus higher acquisition costs.

                  My guess is that you are right and this would indeed be a strong filtering criteria.

                  1. 1

                    I’m surprised that would figure strongly, given what tech salaries are, unless people are being offered much larger relocation packages than I have been. For a non-international move I’ve been offered $5k usually, maybe negotiable up to $10k, which is basically in the salary-negotiation noise, a single-digit percentage of the first year’s pay.

                    1. 2

                      10k is single-digit %? I’d love to talk to your agent!

                      But yeah, otherwise this accords with my experience.

                      1. 1

                        “9” is a single digit.

                        1. 1

                          That it is. I may ought to have had more coffee before posting.

                  2. 2

                    I don’t know how much of the following applies in the US…

                    If the problem is that they won’t consider you because you’re not in the area, then if you have friends or family in the area, you might be able to use their address e.g. on your CV to get through the initial “not in the area” filter.

                    The company I work for is always a little suspicious of applicants who aren’t local, mainly because:

                    • There’s a good chance they’re either just doing a countrywide search, or a recruiter is on their behalf. If they don’t really have any intention of moving, then it’s a waste of time interviewing them.
                    • Especially for young graduates, there’s a fairly high chance that they’ll spend a year or two here, then decide that the area isn’t for them and move back home / to the cities. (There’s nothing wrong with doing that, but if a company thinks you’re going to leave, they’re less likely to invest time and resources in you than in an identical candidate who’s likely to stay for longer.)

                    The way to get past these blocks is to tell a convincing story about why you want to move to the area. If you have family in the area, that will look promising. If you’ve lived in the area before, that’s also worth mentioning.

                  1. 2

                    Packing up my apartment to move to the USA next week (from the UK). My furniture isn’t really fancy enough to be worth shipping transatlantically so this involves a bunch of selling stuff on Gumtree and Facebook Marketplace. Mildly stressful, but not too bad so far.

                    1. 2

                      Out of the frying pan into the fire :) Good luck! Where in the US will you land?

                      1. 3

                        Washington, DC, for a job as assistant professor in CS at American University.

                        And thanks! Will definitely be a change of scenery, pros and cons but hopefully good overall. Right now I live in a 20,000-person seaside town that’s 5 hours from London, while in a few weeks I’ll be living right in the Imperial Capital!

                        1. 2

                          Congratulations! DC is a fun town! If you drink beer the bar scene is rather lively (I’m a beer fan and so am rather fond of the U Street Saloon . Lots of great culture happening there.

                    1. 4

                      At what stage in the interview process do you have this in mind? If it’s late-ish, seems plausible. If it’s early-ish, seems like a lot to ask up front from a candidate to spend a day grokking a codebase when they’re still at the point where they might be summarily rejected after 10 minutes’ review. You do mention that some people might not have the time, but even for those who do, is it a good use of their time?

                      The academic-job version of this is a university wanting your initial application for a faculty position to come with a custom-made syllabus for one of their new courses, or a review of their degree program with suggested revisions, or something of that kind. (Distinct from asking you to send in an example syllabus of a course you might teach or have taught in the past, which isn’t custom, employer-specific work.) I usually pass on applying to those. I am happy to prep custom material if I made it to the shortlist and the potential employer shows they’re serious enough about me as a candidate to fly me out for an on-site interview, though. Then it seems fair and more likely to be time not wasted.

                      1. 4

                        I figured this would replace the in person technical interviews. So for a company, the recruiting flow might be

                        1. Initial Phone Interview
                        2. Maybe a fast phone technical filter interview
                        3. Give them the project, tell them to make a change (and that they’ll be reviewing a couple of PRs for the onsite)
                        4. On site: culture fit, discuss their change to the codebase, have them do the code review
                        5. Hire/no hire decision.
                        1. 1

                          For me, this would be a red flag - 2/3 interviews (including the absolute worst kind, the phone interview) and a day long coding test? Maybe if I was applying to be CTO of a Fortune 500 company. For a lowly developer, this is too much.

                          1. 4

                            You’ve definitely been luckier with interviews than I have. Every company I’ve ever interviewed with had at least three rounds!

                            1. 1

                              My last two interviews were just one short onsite session each. Both led to an offer. I turned down companies with more involved & time consuming process.

                      1. 3

                        I’m @mjn@icosahedron.website. I post some mix of tech/research stuff and miscellaneous my-daily-life sorts of stuff.

                        1. 7

                          Are FreeBSD jails remotely as usable as Docker for Linux? Last time I checked they seemed rather unusable.

                          1. 3

                            In technical terms they’re just fine, in my semi-professional experience. What they lack is the ergonomics of Docker.

                            1. 5

                              I’m not very impressed with the ergonomics of docker, and it’s definitely not obvious to me that BSD jails are an inferior solution to it.

                              1. 5

                                Ok, so I’m a big fan of BSDs, so I’d be very interested if there’d be a nice (not necessarily identical, but similar) way to do the roughly the following things with jails:

                                vi Dockerfile # implement your container based on another containers
                                docker build . # build it
                                docker push https://<internal_storage>/money-maker:0.9 # push it to internal repo
                                ssh test_machine
                                docker run https://<internal_storage_server>/money-maker:0.9 # run the container on the test machine
                                
                                1. 5

                                  The obvious equivalent I can think of is:

                                  • Create a jail
                                  • Set it up (whether manually or via a Dockerfile-equivalant shell script)
                                  • Store a tar of its filesystem to https://<internal_storage>/money-maker:0.9
                                  • Create a jail on the destination machine
                                  • Untar the stored filesystem
                                  • Start the jail

                                  These steps aren’t integrated nicely the way they are with docker, but they are made of small, otherwise-useful parts which compose easily.

                                  1. 4

                                    Sure. How much work do you think needs to be done to get the benefits of Docker’s layer-based approach to containers? If your containers are based on each other, you get significant space savings that way.

                                    1. 0

                                      ZFS deduplicates stored blocks, so you would still get the space savings. You would still have to get it over the network, though.

                                      1. 6

                                        ZFS does not dedup by default, and deduping requires a lot of ram to the point that I’d not turn it on for performance reasons. I tried a 20TiB pool with/without, the speed was about 300k/s versus something closer to the underlying ssd’s performance. It was that bad, even after trying to tune the piss out of it.

                                        Hardlinks would be faster at that point.

                                        1. 3

                                          No no no, ZFS dedup wastes some ridiculous amount of RAM. Do you use it IRL or are you just quoting a feature list?

                                          1. 1

                                            I use it, bit not on anything big, just my home BAS.

                                    2. 2

                                      One option is to use a jail-management system built on top of the raw OS functionality. They tend to take an opinionated approach to how management/launching/etc. should work, and enforce a more fleshed-out container model. As a result they’re more ergonomic if what you want to do fits with their opinions. CBSD is probably the most full-featured one, and is actively maintained, but there are a bunch of others too. Some of them (like CBSD) do additional things like providing a unified interface for launching a container as either a jail or a bhyve VM.

                              1. 21

                                I used to work in academia, and this is an argument that I had many times. “Teaching programming” is really about teaching symbolic logic and algorithmic thinking, and any number of languages can do that without the baggage and complexity of C++. I think, if I was in a similar position again, I’d probably argue for Scheme and use The Little Schemer as the class text.

                                1. 10

                                  This is called computational thinking. I’ve found the topic to be contentious in universities, where many people are exposed to programming for the first time. Idealists will want to focus on intangible, fundamental skills with languages that have a simple core, like scheme, while pragmatists will want to give students more marketable skills (e.g. python/java/matlab modeling). Students also get frustrated (understandably) at learning “some niche language” instead of the languages requested on job postings.

                                  Regardless, I think we can all agree C++ is indeed a terrible first language to learn.

                                  1. 9

                                    Ironically, if you’d asked me ten years ago I would’ve said Python. I suppose I’ve become more idealist over time: I think those intangible, fundamental skills are the necessary ingredients for a successful programmer. I’ve worked with a lot of people who “knew Python” but couldn’t think their way through a problem at all; I’ve had to whiteboard for someone why their contradictory boolean condition would never work. Logic and algorithms matter a lot.

                                    1. 9

                                      I think python is a nice compromise. The syntax and semantics are simple enough that you can focus on the fundamentals, and at the same time it gives a base for students to explore more practical aspects of they want.

                                    2. 7

                                      Students also get frustrated (understandably) at learning “some niche language” instead of the languages requested on job postings.

                                      Yeah, I feel like universities could do a better job at setting the stage for this stuff. They should explain why the “niche language” is being used, and help the students understand that this will give them a long term competitive advantage over people who have just been chasing the latest fads based on the whims of industry.

                                      Then there is also the additional problem of industry pressuring universities into becoming job training institutions, rather than places for fostering far-looking, independent thinkers, with a deep understanding of theory and history. :/

                                      1. 3

                                        I’ve been thinking about this a bit lately, because I’m teaching an intro programming languages course in Spring ‘19 (not intro to programming, but a 2nd year course that’s supposed to survey programming paradigms and fundamental concepts). I have some scope to revise the curriculum, and want to balance giving a survey of what I think of as fundamentals with picking specific languages to do assignments in that students will perceive as relevant, and ideally can even put on their resumes as something they have intro-level experience in.

                                        I think it might be getting easier than it has been in a while to square this circle though. For some language families at least, you can find a flavor that has some kind of modern relevance that students & employers will respect. Clojure is more mainstream than any Lisp has been in decades, for example. I may personally prefer CL or Scheme, but most of what I’d teach in those I can teach in Clojure. Or another one: I took a course that used SML in the early 2000s, and liked it, but it was very much not an “industry” language at the time. Nowadays ReasonML is from Facebook, so is hard to dismiss as purely ivory tower, and OCaml on a resume is something that increasingly gets respect. Even for things that haven’t quite been picked up in industry, there are modernish communities around some, e.g. Factor is an up-to-date take on stack languages.

                                        1. 3

                                          I one way you can look at it is: understanding how to analyse the syntax and semantics of programming languages can help you a great deal when learning new languages, and even in learning new frameworks (Rails, RSpec, Ember, React, NumPy, Regex, Query builders, etc. could all be seen as domain specific PLs embedded in a host language). Often they have weird behaviours, but it really helps to have a mental framework to quickly understand new language concepts.

                                          Note that I wouldn’t recommend this as a beginner programming language course - indeed I’d probably go with TypeScript, because if all else fails they’ll have learned something that can work in many places, and sets them on the path of using types early on. From the teaching languages Pyret looks good too, but you’d have to prevent it from being rejected. But as soon as possible I think it’s important to get them onto something like Coursera’s Programming Languages course (which goes from SML -> Racket -> Ruby, and shows them how to pick up new languages quickly).

                                      2. 7

                                        I started college in 1998, and our intro CS class was in Scheme. At the time, I already had done BASIC, Pascal, and C++, and was (over)confident in all of them, and I hated doing Scheme. It was different, it was impractical, I saw no use in learning it. By my sophomore year I was telling everyone who would listen that we should just do intro in Perl, because you can do useful things in it!

                                        Boy howdy, was I wrong, and not just about Perl. I didn’t appreciate it at the time, and I didn’t actually appreciate it until years later. It just sorta percolated up as, “Holy crap, this stuff is in my brain and it’s useful.”

                                        1. 3

                                          I hear this reasoning, about teaching tangible skills, but even one two or three quarters for Python is not enough for a job, at least it shouldn’t be. If it is, then employers are totally ok with extremely shallow knowledge.

                                          1. 1

                                            I didn’t even realize I had read this a month ago, nevermind I had commented on it, before I wrote my own post on the topic. Subconscious motivations at its finest.

                                        1. 5

                                          These have been floating around FOR-EVER but I’m glad they keep cropping up. I see evidence of these constantly in just about every technical community I inhabit.

                                          They were an eye opener for me at the time. Particularly #2 (accept me as I am) and #4 (transitive).

                                          Grokking the fundamental falsehood of some of these deeply was definitely a step towards finally growing up in certain ways that I REALLY needed to (and had for a long time).

                                          I also credit having successfully jettisoned #2 with being why at age 35 I finally started dating and met my wife :)

                                          1. 5

                                            I recognize some of these patterns, but I don’t think I associate them with technical communities. Where I’ve run into them is in “cultural geek” communities, those organized around things like fandoms. This could be idiosyncratic based on which specific kinds of both communities I’ve run into though.

                                            1. 2

                                              I’ll take your word for it. In my case, the degree ov overlap between technical communities and various fandoms is extremely high.

                                              1. 1

                                                That’s interesting and believable too, which is why I added the caveat that it could well be idiosyncratic. I’ve definitely read about this kind of thing in my area, artificial intelligence, e.g. the old MIT hacker culture. I just haven’t encountered it in person, and it always felt like something that existed way before my time. Out of curiosity, what kinds of technical communities have you encountered where the overlap is high?

                                                The AI conferences I personally go to do have a handful of geeky people, but way more business/startup/government/military/professor types. A bunch of these factors pretty clearly don’t apply as far as I can tell, for better or worse. For example, socially awkward and/or unhygienic people are pretty much jetissoned without a second thought if someone thinks they might interfere with funding.

                                                1. 2

                                                  So, I want to be sure to constrain this properly.

                                                  I came into the Boston technical scene in the early 1990s. At that time, the overlap with the Boston science fiction fandom community was HUGE as it was for the Polyamory and BDSM communities (of which I’ve never been a part. Vanilla and proud yo :)

                                                  In fact, I pretty much got my start in the Boston tech scene by showing up at a science fiction fandom oriented group house in the middle of a blizzard and passing out my resume to anyone who’d talk to me :) I ended up living in that group house for a time.

                                                  I’m fairly sure this isn’t representative of the here and now. Our industry has become a very VERY different place several times over since then (mostly for the better) and I suspect that younger folks are being drawn from a much more diverse background of interests.

                                                  1. 1

                                                    Hah interesting, I know some people who I think had a similar kind of experience in the SF Bay Area in the ’90s, living in group houses to get broadband internet and such. I got into tech in the late ‘90s in suburban Houston, which might have had a geek scene, but if so I didn’t know about it. The tech scene I was exposed to was much more “professional engineering” oriented, anchored by people who worked at NASA or NASA contractors (plus some people doing tech at oil companies).

                                              2. 1

                                                I’m not found that to be the case, even here in the Lobsters community in its forum and chat forms.

                                              3. 2

                                                I’m curious how #2 motivated you to start dating. Were you just generally more receptive of criticism from friends, and if so, how does that translate to wanting to start dating?

                                                1. 4

                                                  Not so much about wanting to start dating, but being willing to make the changes necessary to be perceived as attractive.

                                                  “Friends accept me as I am”.

                                                  Who cares if I have a giant sloppy beard, dress in sweat pants and faded T-shirts all the time, and generally take PRIDE in not giving two shits about my personal appearance? My TRUE friends will accept me for who I am and see past all that.

                                                  Except that this is the real world. How you look DOES matter. Shave the beard, lose a few pounds, buy some decent clothing and it’s a whole different ballgame.

                                                  1. 1

                                                    I definitely agree with what you’re saying, but it reminds me of some definitions from Douglas Coupland’s novel Generation X :

                                                    anti-victim device (AVD) - a small fashion accessory worn on an otherwise conservative outfit which announces to the world that one still has a spark of individuality burning inside: 1940s retro ties and earrings (on men), feminist buttons, noserings (on women), and the now almost completely extinct teeny weeny “rattail” haircut (both sexes).

                                                    … and:

                                                    personality tithe - a price paid for becoming a couple; previously amusing human beings become boring: “Thanks for inviting us, but Noreen and I are going to look at flatware catalogs tonight. Afterward we’re going to watch the shopping channel.”

                                                    https://en.wikiquote.org/wiki/Generation_X:_Tales_for_an_Accelerated_Culture

                                                    Some parts of a given personality are stupid and need to be shorn so the person can have a more interesting life. It’s easy to lionize the idea that someone can be Good Enough, or, in somewhat different subcultures, Cool Enough, that you never have to compromise on those things, but even if you luck into a job where that works, it doesn’t and can never work in a healthy personal relationship.

                                                    1. 4

                                                      Sounds like I need to read that book!

                                                      I don’t personally see it as compromise.

                                                      The truth is that my self confidence was in the shitter at that time. My personal appearance was just one outward manifestation of that.

                                                      Recognizing that I needed to make changes if I wanted to meet someone and be less lonely was a first step towards letting go of some of that baggage.

                                              1. 3

                                                I wish the article were a bit more substantive.

                                                They touch on type inference. Does anyone have some refs for how to do this in Lisp?

                                                1. 6

                                                  They touch on type inference. Does anyone have some refs for how to do this in Lisp?

                                                  Henry Baker has a paper about it: The Nimble Type Inferencer for Common Lisp-84

                                                  http://home.pipeline.com/~hbaker1/TInference.html

                                                  1. 4

                                                    You don’t do this, the compiler does it for you. Inferring it from the code, with or without the aid of type declarations by programmer.

                                                    The article has the example of declaring the types, which allows compiler to infer tight, specialized code.

                                                    1. 2

                                                      Sure, agreed. But in my case, I’m implementing the compiler. So I was hoping for some refs on how to do type inference.

                                                      1. 12

                                                        SBCL implements it as type propagation using dataflow analysis. Types in some places are known from explicit declarations, constant literals, known functions, etc., and this information is propagated in a pass named “constraint propagation”, to infer the types of other expressions when possible.

                                                        I’m not sure whether there’s proper documentation anywhere, but this post on some of the type propagator’s weaknesses starts with a summary of how it works [1].

                                                        “Gradual typing” has become a trend outside the Lisp world lately, and I believe those systems do something vaguely similar to propagate types. That might be another place to find implementation hints.

                                                        [1] Note when reading this post that “Python” is the name of SBCL’s compiler; the name predates Guido van Rossum’s language. The reason the compiler has a separate name in the first place is that SBCL’s predecessor, CMUCL, had an interpreter, a bytecode compiler, and a native-code compiler, and Python was the name of the native-code compiler.

                                                        1. 2

                                                          SBCL implements it as type propagation using dataflow analysis. Types in some places are known from explicit declarations, constant literals, known functions, etc., and this information is propagated in a pass named “constraint propagation”, to infer the types of other expressions when possible.

                                                          Ah! Interesting! This is what I was planning to do with my toy lisp :) It might be worth for scholary searches, I heard it first when using Erlang under the name of Succession Typing. I could be completely remembering, but I think Dialyzer uses this method of program typing.

                                                          1. 2

                                                            I don’t think this is quite the same thing as Dialyzer’s Success Typing, though there might be some overlap. Dialyzer’s goals are very different in that it’s only concerned with correctness and not performance. But it might use some similar methods.

                                                        2. 2

                                                          This may help, it’s an overview of FB’s Flow system for JS https://www.youtube.com/watch?v=VEaDsKyDxkY

                                                    1. 5

                                                      Only tangentially on topic: I hadn’t noticed there’s a Talos II in a more hobbyist affordable price range now. Last time I looked the complete-system prices were $7k or so, and that’s still what the flagship one goes for, but now they have a $2k “Entry-Level Developer System” with one 4-core/16-thread CPU. For $2.5k you can bump it to 16G RAM and 8-core/32-thread. Still not cheap for those specs compared to x86 unless you have pretty specific POWER9-optimized stuff, but it’s at least not an absurd amount of money.

                                                      1. 5

                                                        Rumour is that there’ll be a sub-$1k option along the lines of a Mac Mini at some point, too.

                                                      1. 9

                                                        This strikes me as a step-by-step rediscovery of “Checked Exceptions”, but with an even more awkward encoding.

                                                        1. 10

                                                          Checked exceptions are underrated.

                                                          1. 5

                                                            There’s a lobste.rs discussion from 6 months ago that’s somewhat relevant on that (but with Rust rather than Haskell).

                                                          1. 14

                                                            There are kind of two things here. The high-level APIs for machine learning are typically in Python or R. The actual implementations of the data structures and algorithms mostly are in C, C++, or Fortran, though, with Python/R bindings. The usual reasons given for this split approach are that pure Python/R libraries are too slow, but using C/C++/Fortran libraries directly is too tedious and low-level.

                                                            1. 4

                                                              Judging from what I’ve seen in other domains, that depends. There’s no doubt that in a proper ML library, most of the fundamental parts are heavily optimized by, e.g., implementing them in a lower-level language and using FFI bindings. However, anything built on top of them is not. And that can include additional performance-sensitive components, written outside the performance haven of C/C++/Fortran. If these are useful enough for other devs to make use of them, inefficiency can spread through the ecosystem.

                                                              No matter how much performance boilerplate you handle in a library, developers will always have their own data structures in mind. But perhaps in ML, people use mostly pre-made components, to a degree such that most of the performance-sensitive logic really does happen in more optimized code. Do you find this to be the case?

                                                              1. 3

                                                                people use mostly pre-made components

                                                                Varies by area, but I think this is true in many of them. This is probably best established in the R world, since it’s older (going back decades to its predecessor, Bell Labs S). Far more people use the system as data analysts than implement new statistical methods. In fact many S/R users traditionally didn’t see themselves as programming, but as using an interactive console, which is why it has features like saving/reloading the current workspace supporting a purely interactive usage without ever explicitly writing a program/script in a text editor. (It’s gotten a little more common to write explicit scripts lately though, with the reproducible-science push.)

                                                                Python/ML people are more likely to see Python as a programming language rather than just a data-analysis console, but if you look at what most people use it for, it’s the same kind of workflow of loading and cleaning data, choosing a model, setting model parameters, exporting or plotting results. The heavy lifting is done by something like TensorFlow which your average ML practitioner is not going to be contributing code to (even most ML researchers won’t be).

                                                            1. 3

                                                              I found this very interesting. I’ve noticed that in the media “A.I.” seems to often refer to just deep learning, so its failures will definitely affect the public perception of the whole field.

                                                              1. 2

                                                                That’s one of the sad (or funny) parts of the “AI winter” phenomenon: each cycle, there’s a different technology called “AI”. But let’s be honest, public perception owes more to Hollywood movies than scientific consensus. The interesting question is who’s handing out money, and what do they think will succeed or fail. Recently it’s been difficult to get funding for anything but deep neural nets.

                                                                1. 1

                                                                  Recently it’s been difficult to get funding for anything but deep neural nets.

                                                                  For venture-capital funding, yes, but public funding bodies aren’t putting all their eggs in the deep-learning basket. The NSF, at least, continues to fund a pretty wide range of methods, including logic-based AI, symbolic planning, Bayesian methods, etc.

                                                              1. 2

                                                                How did it come to be like this? I don’t imagine this has anything to do with efficiency, judging by the amount of labour (on the employer’s end) exerted to make candidates jump through hoops.

                                                                1. 8

                                                                  Nobody wants to take a risk and get blamed for a bad hire, so they set up more and more process. It’s like sifting for gold, except you have like twenty pans and you throw away everything that doesn’t make it through any of the sifters without looking.

                                                                  1. 3

                                                                    That explanation seems plausible, but then I wonder, why is the process so much more heavyweight in tech than just about any other field, including other STEM fields? In sheer number of hours of interviewing that it takes to get a job, counting all the phone screens, take-home assignments, in-person interviews, etc., tech is way out of sync with norms elsewhere. A typical hiring process is any other STEM field is a resume screen, followed by one phone screen (typically an hour), followed by an on-site interview that can last somewhere between a few hours and a full day.

                                                                    1. 8

                                                                      Survivorship bias could be why. The ones perpetuating this broken process are those who sailed through it.

                                                                      There’s also a lot of talent floating around, and the average company won’t be screwed by an average hire. So even if you miss out on that quirky dev with no social skills but the ability to conjure up a regex interpreter solely from memory, it doesn’t really matter to them.

                                                                      It should matter to startups, though, because hiring average devs means you’ll fail.

                                                                      1. 3

                                                                        Depends on the startup; until you have product-market fit, you don’t need amazing engineers so much as you need people who can churn out prototypes fast.

                                                                      2. 4

                                                                        It might be partly due to the volume of applicants. With tech you have:

                                                                        1. Massive concentration of talent (e.g silicon valley)
                                                                        2. Remote work

                                                                        For those reasons you can often get hundreds of applicants to a posting. Other STEM disciplines don’t support working remotely, and in some cases (think civil engineering) need their engineers to be physically on-site. I’d wager they tend to be much more dispersed around the country and companies can only draw from the local talent pool.

                                                                        1. 3

                                                                          I applied to a remote London based, three-person not-a-startup. I did the homework and got among the 50 or so people they interviewed on phone. They told they got over 2000 applications.

                                                                        2. 4

                                                                          Particularly in other STEM fields it’s pretty common to have more rigorous formal education requirements as part of the hiring bar (either explicitly or by convention.) Software development has always been somewhat more open to those from other backgrounds, but the flip side to that is that there seems to be a desire to set a higher performance/skills bar (or at least look like you are) as a result. There are potentially pros and cons to both.

                                                                          I’d also wonder, particularly around the online tests/challenges/screenings/etc…, whether this is a result of tech people trying to come up with a tech solution to scale hiring the same way you’d approach scaling a technological system, and the resulting expansion in complexity.

                                                                      3. 4

                                                                        Hiring is hard, and a lot of work, and not something most engineers will willfully dive into. Therefore, at most companies, as much as possible of the hiring process gets farmed out to HR / management. And they do the best job they can, given their lack of domain knowledge. Unsurprisingly, they also favor potential employees that they think will be “good” based on their ability to sit, stay, heel, and jump through hoops. Fetch. Good boy. Who wants a cookie. ;)

                                                                        Another take: Mistakes really, really, suck. And if you just add more analysis and testing to a hiring process you’re more likely to spot a problem in a candidate.

                                                                        1. 2

                                                                          I think mistakes are a big part of it. Software work is highly leveraged: what you write might run hundreds, thousands or millions of times per day. Being a little off can have big downstream consequences.

                                                                        2. 4

                                                                          I think it’s partly because there’s no training for it in most jobs, it’s very different to expertise in software, it’s very unclear what best practices are (if there are any), and for a lot of people it’s a time suck out of their day, when they’ve already got lots of work to do.

                                                                          So you end up with these completely ad-hoc processes, wildly different from company to company (or differing even person to person during the interview), without anyone necessarily responsible for putting a system in place and getting it right.

                                                                          Not to mention HR incentives may not align (points / money for getting someone hired) with engineering, and then you’ve got engineers who use the interview as a way to show off their own smarts, or who ask irrelevant questions (because though you all do code review, no-one does interview question review), or who got the interview dumped on their plate at the last minute because someone else is putting out a dumpster fire, and they’ve never heard of you or seen your resume before they walk into the room…

                                                                          And decision making is ad-hoc, and the sync-up session after the interview gets put off for a couple of days because there’s a VP who wants to be on the call but they’re tied up in meetings, and in the meantime the candidate has an interview with another company so you’ve just moved forward with booking the onsite anyway…

                                                                          So many reasons :)

                                                                          1. 2

                                                                            It’s all marketing.

                                                                            I don’t think I would have taken any of my jobs if the recruiters were like “we’re not going to bother interviewing you because all we have is monkey work, when can you start?”, even though in hindsight that would have been totally adequate.

                                                                            So companies play hard to get and pretend 99% of their applicants are too bad to do the jobs on offer, when the reality is closer to the opposite.

                                                                            1. 1

                                                                              <[apple] insert [google] company [facebook] here [microsoft]> only hires the best.

                                                                            1. 2

                                                                              Our research group is going “on the road” to work in a public space as scientists-in-residence at the Eden Project this week. They have a new thing where there is an art space and a science lab, with rotating groups there each week. Should be interesting, even though as computing researchers we don’t have the kind of impressive looking “science equipment” to bring along that people like chemists have. We’re bringing along a 3d printer and some small robots though so it isn’t just a few researchers coding on laptops, even though that would be a more authentic representation of what our lab usually looks like.

                                                                              1. 1

                                                                                Those are some beautiful images!

                                                                              1. 1

                                                                                I’m glad to received a link to my own article, but I do disagree somewhat with that is said in this one.

                                                                                The specific example of cron/NFS is in fact a hard dependency: cron runs reboot tasks when it starts, and if they need NFS mounts, then those mounts should be a hard requirement of cron, “ordering” is not sufficient.

                                                                                The implied issue is that cron doesn’t need the NFS mounts once it’s run those tasks, so the dependency “becomes false” at that point. If I understand the argument correctly, it is: seeing as “the system as a whole wants both”, you could use a startup ordering to avoid leaving a lingering dependency once the @reboot jobs have run while still ensuring that NFS mounts are available before cron starts. This is true, but it would be fragile and racy. For instance, there would be nothing to prevent the NFS mounts, even with the co-operation of the service manager, being unmounted just after crond begins execution, but before it has even started (or possibly when it is midway through) running the @reboot tasks.

                                                                                In my eyes there are two ways to solve it properly: separate cron boot tasks from regular cron so that you can run them separately (that would mean changing cron or using some other tool), or having the cron boot tasks work by starting short-running services (which can then list NFS mounts as a dependency). This latter requires non-priviliged users be allowed to start services though, and that’s opening a can of worms. I feel that ultimately the example just illustrates the problems inherent in cron’s @reboot mechanism.

                                                                                (Not to mention that there’s a pre-existing problem: for cron, “reboot” just means “cron started”. If you start and stop cron, those reboot tasks will all run again…)

                                                                                1. 2

                                                                                  Belatedly (as the author of the linked-to article): In our environment, NFS mounts are a hard dependency of those specific @reboot cron jobs, but not of cron in general. In fact we specifically want cron to run even if NFS mounts are not there, because one of the system cron jobs is an NFS mount updater that will, as a side effect, keep retrying NFS mounts that didn’t work the first time. Unfortunately there is no good way to express this in current init systems that I know about and @reboot cron jobs are the best way we could come up with to allow users to start their own services on boot without having to involve sysadmins to add, modify, and remove them.

                                                                                  (With sufficient determination we could write our own service for this which people could register with and modify, and in that service we could get all of the dependencies right. But we’re reluctant to write local software, such a service would clearly be security sensitive, and @reboot works well enough in our environment.)

                                                                                  1. 1

                                                                                    But it’s not a dependency of cron, it’s a dependency of these particular tasks. Cron the package that contains the service definition has no idea about what you put into your crontab.

                                                                                    Yes, it’s a problem in cron. This is why there’s movement towards just dropping cron in favor of integrating task scheduling into service managers. Apple launchd was probably first, systemd of course has timers too, and the “most Unix” solution is in Void Linux (and runit-faster for FreeBSD now): snooze as runit services. In all these cases, each scheduled task can have its own dependencies.

                                                                                    (Of course the boot tasks then are just short-running services as you described.)

                                                                                    1. 1

                                                                                      But it’s not a dependency of cron, it’s a dependency of these particular tasks

                                                                                      Agreed, but if you’re the sysadmin and know that cron jobs are using some service/unit, then you’d better make sure that the cron service is configured with an appropriate dependency. At least, that’s how I view it. Without knowing more about the particular system in question, I’m not sure we can say much more about how it should be configured - I agree that cron isn’t perfect, particularly for “on boot” tasks, but at least it’s a secure way of allowing unprivileged users to set up their own time-based tasks. (I guess it’s an open question whether that should really be allowed anyway).

                                                                                    2. 1

                                                                                      I was also confused by that, but from the discussion in the comments, I think the reason they don’t want it to be a hard dependency is that, in their setup, some machines typically have NFS configured and some don’t. In the case where the machine would start NFS anyway, they want an ordering dependency so it starts before cron. But if NFS wasn’t otherwise configured to start on that machine, then cron should start up without trying to start NFS.

                                                                                      1. 1

                                                                                        Yes, that accords with the comments below the post:

                                                                                        On machines with NFS mounts and Apache running, we want Apache to start after the NFS mounts; however, we don’t want either NFS mounts or Apache to start the other unless they’re explicitly enabled. If we don’t want to have to customize dependencies on a per-machine basis, this must be a before/after relationship because neither service implies the other

                                                                                        The problem is “don’t want to have to customize dependencies” is essentially saying “we are ok with the dependencies being incomplete on some machines if it means we can have the same dependency configurations on all machines”. That seems like the wrong approach to me; you should just bite the bullet and configure your machines correctly; you’ve already got to explicitly enable the services you do want on each machine, anyway.

                                                                                        1. 1

                                                                                          This gets into the philosophy of fleet management. As someone who manages a relatively small fleet by current standards (we only have a hundred machines or so), my view is that the less you have to do and remember for specific combinations of configurations, the better; as much as possible you want to be able to treat individual configuration options as independent and avoid potential combinatorial explosions of combinations. So it’s much better to be able to set a before/after relationship once, globally, than to remember that on a machine that both has NFS mounts and Apache that you need a special additional customization. Pragmatically you’re much more likely to forget that such special cases exist and thus set up machines with things missing (and such missing cases may be hard to debug, since they can work most of the time or have only tiny symptoms when things go wrong).

                                                                                          (One way to think of it is that it is a building blocks approach versus a custom building approach. Building blocks is easier.)