1. 3

    Note that what they are complaining about are not excusable as responses to the XY problem. Even if the XY problem is in play, these responses are inadequate.

    1. 1

      So much anger and sadness over something so relatively unimportant :(

      Don’t stake alliances, friendships, your future or your life on being right, unless you are certain it’s really that important. Your code never is. Nor are most other things.

      1. 12

        If you feel paralyzed when an interviewer asks you how you see yourself in five years, remember she doesn’t want you to read the future, she wants to learn about your motivation and goals.

        And yet, she doesn’t ask you “what’s your motivation and what are some goals?”. Also, she’s looking out for red-flags like “well, my mother-in-law is getting old and we might have to move back across the country next year to look after her, so I’m just looking for something in the interim”. Which is a totally, 100% legit (and compassionate, and honest) thing to tell the interviewer. But, let’s be honest: don’t tell the interviewer that, they all want to hear that you’re looking to make a long term impact working on whatever BS run-of-the-mill project/team/company they’re interviewing you for. Tell them the mother-in-law story and you won’t be making it to the next round.

        Edit: Very little of this, to me, is “how to pretend you have social skills”. It covers some specific interactions with some people you might encounter in a software job, in situations that are highly-specific to software jobs and in particular (I think, from the article) junior developers.

        Edit edit: I feel I have been a bit snarky. I don’t want to knock the post too hard: clearly effort and thought went into it. But I’m not sure what the takeaways are. Some of it’s “don’t be nervous in interviews”, some of it’s “I (the author) am a good interviewer”, some of it’s “understand the dynamics of working with the angry senior tech lead” (which is good advice). But… coulda been bullet points?

        1. 10

          I agree the title is inappropriate, but for different reasons. If you could act as the author proposes in all of the four situations described, you’re not pretending to have social skills: you actually have social skills.

          1. 3

            I’ve always hated the 5 year plan question. In interviews I’ll usually say, “Well I expect to not be answering canned interview questions,” if it’s a job I’m not really interested in. If it’s a job I want, I tend to say, “There’s no way to know what will happen in 5 years. Scientifically, we don’t even understand why we can only move forward in time. In space you can move in any six directions, but with time, we all move forward at a constant rate (unless you’re very rich and have a very fast space ship, and even then you can only slow down your time travel relative to everyone else).

            “So it’s not useful to think about regrets or what you would have done. Our decisions only affect our present and our future.”

            I’ve gotten job offers answering that question both ways.

            In general I agree with you on this. I hate this idea that most engineers don’t have social skills. We certainly do. I’ve had to deal with clients in many different industries. Articles like this are more about teaching people to be less truthful and tell people what they want to hear to get your goals.

            You don’t get great jobs that way. It’s good advice if you just need a job. Once you have a job, it’s bad advice when you’re trying to switch to a job you actually want.

            1. 3

              Without comment in regards to how you approach answering this question, one of the reasons people ask it is that they want to know if you have a future. And further they want to know if they can be a part of it. The thinking goes that if you don’t have a future or your future won’t include them then there is no opportunity for a relationship to develop that will work for everyone. I suppose that can be said, as in the article, as “learn about your motivation and goals.” I certainly consider that framing of the question to be weak and, as you allude to, potentially disingenuous.

              1. 3

                I think asking if you will stay with the company for five years at a job interview is equivalent to asking for marriage plans at a first date.

          1. 13

            How to Solve It - G. Polya.

            1. 6

              The only thing I ever remember about that book is, paraphrased, if there’s a problem you can’t solve, there’s also a smaller problem you can’t solve. Find that smaller problem and solve it first.

              1. 2

                Sounds like a good summary, but for much broader meaning of the word “problem” that would seem from it. The book gives a list of questions designed to explore the space around your problem from many different perspectives, hopefully guiding you towards a different problem that’s easier to solve, and will contribute necessary insight or data into your main one.

                1. 1

                  The idea is called divide and conquer.

                  1. 3

                    Divide and conquer is a specific instance of that tactic. It’s one where solving the smaller problem actually gives you part of the solution of the larger problem. But in mathematics you often solve a truly simpler version of a problem to discover a strategy to attack the larger problem. The solution to the simpler problem itself is not used any more.

              1. 17

                Personal Opinion: No. RFC725 says, HTTP-451 should be used “when a server operator has received a legal demand to deny access to a resource” and “This type of legal demand typically most directly affects the operations of ISPs and search engines.” Sure, there is no explicit mention of the word censorship, but that is what is implied by the RFC and explicitly stated the author of the original proposal.

                The GDPR doesn’t require a website owner to block EU users, just to respect their rights as data subjects. Since they refuse service, a 403 code seems most appropriate (”[403] indicates that the server understood the request but refuses to authorize it.”).

                1. 2

                  I agree. If a website operates in a way that violates the GDPR, serving the content to a EU citizen is accompanied by a violation of the GDPR. It is illegal for the website to serve them the content, not because of the content itself, but because of the technical process used to serve the content. With the enactment of the GDPR, the server operator has effectively received a demand to deny access to all resources of the website to EU citizens, as long as the technical process has not changed. But it’s about the process, not about the content. And 451 is about the content.

                1. 2

                  The comments in the post discuss a 5XX vs a 4XX error and that client-side errors should be fixed by the client. Now I am wondering if the GDPR applies to European citizens or people that are currently in Europe (maybe a day trip or what ever). I usually thought that these GDPR filters are using geoIp. But what if a European citizen is in the US and the other way around? I only checked Wikipedia for this and they say the GDPR applies to EU-citizens. So how to figure out if a web client is a EU-citizen? What am I doing wrong?

                  1. 4

                    The companies are just trying to protect themselves as best they can. Realistically, a European citizen suing a US-only company in a European court over European law is being frivolous and the company will likely not be affected in any way, so the butt-covering of geoip blocking is more a political statement to potential sue-ers than it is actual legal protection.

                    1. 6

                      What is the actual message to European users of such political statement?

                      We don’t want your money? We don’t want your data? You do not deserve our technology? We are the Spiders of the Web and you are just a fly?

                      Btw, as an European I would really appreciate a clear statement on a website saying “we are sorry but we cannot protect your data and respect your rights, please search for one of our competitor that can do it better”.

                      I’m not ironic.
                      GDPR defines several important rights for the data subject that imply certain investments in cybersecurity and a basic quality of service. Being able to say “I cannot do this right, please ask to someone else” is a sign of professionalism.

                    2. 3

                      You figure it out by asking them. There are many sites that don’t serve US citizens for various reasons. When you enter them, they ask you to declare you are not a US citizen. It’s as simple as that. If they lie, it’s on them.

                      Honestly, this GDPR thing has gotten many Americans acting indignated and generally quite irrational over something that hardly changes anything and is not without a slew of precedent. It’s just the first time US companies are visibly seriously affected by law elsewhere. Now you know how it feels. Get over the feeling and deal with it.

                      1. 1

                        Well, in principle, I would guess that European courts might be apprehensive about dictating law globally, which would essentially be the case if it was found that GDPR applies to European citizens wherever they may be, and even if a website operator had taken all reasonable precautions to block European citizens from using their cite.

                        1. 3

                          GDPR apply to data of European citizens worldwide and to data of non European citizens collected while they are in the Union.

                          However, if your registration form have a mandatory checkbox “I’m NOT a European citizen and I’m not going to use your services while in the European Union” AND the such checkbox is uncheked by default AND you block all European IPs, I think no European court will ever annoy you.

                      1. 1

                        This is way too much attention for a teen. I mean what if their idea is wrong in some small way. pop

                        1. 8

                          If someone of the caliber of Scott Aaronson is sufficiently convinced you are right to put his name on the paper, there is no shame in being wrong. No one in the field would hold that against Tang if it turned out to be wrong.

                          I fail to see what being a teen has to do with anything. If he was a year older, or already in grad school, it would have been fine?

                          To me that sounds like an argument from the same visceral response I have to these kinds of stories: jealousy. I really have to suppress the urge to rain on his parade. And doing that via arguments that seem to have someone’s best interest in mind is the socially safest way to do so. It’s a way of stealthily sabotaging someone’s accomplishment.

                          1. 3

                            I think what voronoipotato may have been alluding to was that aside from that age largely being turmoil for emotions as they learn to deal with them. Teenagers aren’t as battle hardened as adults and having so much attention plastered upon them for it to suddenly turn nasty can be a massive blow emotionally and without the right support can end up dissuading them from continuing.

                            It sounds like the teen will be fine, with someone like Scott Aaronson on the same team - even if the idea turns out to be flawed they will have all the support they need to continue.

                            1. 3

                              Exactly, to be precise the problem I had was with the article pushing Tang into the limelight and not the research Tang contributed to. The research is fine.

                            2. 3

                              I’m not jealous at all. I don’t have any desire to get into academia or compete in that way. I stopped with a community college degree. The difference a few years can make at that age in emotional development is pretty big. Also they have other past successes however small to fall back on. Some people have one big (perceived) catastrophic failure at the beginning and they give up, and they never come back. More importantly, notoriety isn’t Tang’s accomplishment. Saying that they shouldn’t be forced into the limelight on their first attempt isn’t saying they should never have had the opportunity to submit or contribute to scientific progress.

                            3. 6

                              Once Tang had completed the algorithm, Aaronson wanted to be sure it was correct before releasing it publicly. “I was still nervous that once Tang put the paper online, if it’s wrong, the first big paper of [Tang’s] career would go splat,” Aaronson said.

                              The article specifically mentions that point.

                              1. 2

                                Yeah I don’t see how that makes it any better.

                            1. 15

                              I’m not particularly thrilled with either the tone or content of this essay.

                              First, there’s some revisionism:

                              The first reason that WebAssembly is different is that it’s succeeded, and those technologies did not.

                              The essay goes on to redefine “success” in some super-niche way and qualify it, but this is rhetorically dishonest. I might as well write in an essay that “The first reason I disagree with them is that @steveklabnik is a fascist” and then go on to explain that by that I mean a fascist who supports government control of resources and careful policing of speech…sure, under that tortured definition I am both consistent and not wrong, but you wouldn’t be faulted for observing that I’m not right either.

                              They were never standardized, and that matters a lot.

                              JVM was never standardized.

                              Flash was not built on ES3/ES4 standards.

                              Unfortunately, just because something is standardized and went through the wringer doesn’t mean it isn’t hot steaming garbage.

                              If you built an applet in one of these technologies, you didn’t really build a web application […] You lost all of the benefits of other web technologies; you lost HTML, you lost CSS, you lost the accessibility built into the web.

                              Most of those features didn’t exist when Java applets first came out in 1995. Most of the useful stuff that Flash was good at didn’t exist in browsers for most of the 00’s. The essay is trying to sell you on a state of history that didn’t exist in any meaningful way.

                              “Don’t break the web” is a very, very important rule for browser vendors.

                              “…unless it gets you marketshare” is the rest of the sentence that the author leaves out. Browser vendors (and hell, let’s be real here: we mean Google, Apple, Microsoft, and that’s basically it these days) break things all the time and don’t care. https://caniuse.com/ is a monument to this fact.

                              Then there’s seeming misunderstanding about how browser development works:

                              These companies have significant motive to make profit, not to make the web better.

                              “Making profit” is how IE managed to deliver the first working version of CSS. Similarly, how they delivered a little thing called XMLHttpRequest, a small api with passing utility later adopted by every other browser manufacturer.

                              Google Chrome delivered lots of neat features ahead of standardization specifically so they could feed the ad machine. And Mozilla happily rode those coattails for a good long time.

                              I think the notion of “let’s make the web better” ultimately–intentionally or not–boils down to “let’s serve ads better”, once you look at things in context.

                              …and how companies work…

                              WebAssembly is a joint venture between Mozilla, Google, Apple, Microsoft, and others. It’s not trying to push anyone’s particular platform, but instead represents the shared interests of a wide number of stakeholders, both corporate and individual.

                              So, two ad-driven companies, a company known specifically for locking-down platforms (and goofing around in standards and industry groups if one’s familiar with OpenGL or the development of the Cell processor), and a company who is switching to moving as much of their stuff into the cloud–where it can be taxed and leased to users without fear of reprisal. I see why they might want to support WASM.

                              …and how maintenance works…

                              It’s hard enough to maintain one runtime, let alone two. And then how do you integrate them?

                              Runtimes don’t all need to be integrated, and we handily managed to keep the runtimes for JVM and Flash maintained for more than a decade, by letting interested parties support them.

                              …and how language proliferation and the Tower of Babel work…

                              Do we really want to christen one language as the next language of the web? We already have JavaScript. Are we someday going to introduce a third language? A fourth? An agnostic approach is significantly better for longevity’s sake.

                              Picking one language is better for longevity’s sake. Standardization–the shibboleth touched on here and there in this essay–would suggest that we take one language and adopt and update it as needed.

                              Wasm, as I rallied about many a time, is likely to make frontend dev skills even less portable than they already are. It’s an opportunity to let language writers and well-meaning developers rediscover every mistake in their language and find some way of introducing it into the browser and then saddling devs with their shortcomings!

                              …and how security works.

                              Wasm is missing some features of regular assembly language that can cause security vulnerabilities, like stack smashing. Wasm is memory-safe, which is huge!

                              And yet it directly enables malware and side-channel attacks, while it’s proponents kinda ignore the issue and forge ahead.

                              ~

                              We’re all going to be stuck with the outcome of this, and folks promoting the technology for their own career advancement or aesthetics are doing so, seemingly to me, without enough care for the ramifications of what they’re pushing.

                              EDIT: Cleanups to soften this up a bit…my annoyance got in the way of my politeness.

                              EDIT: Removed many “actually”’s in various headers, since I read them in my head as “well ak-shu-alee”

                              1. 23

                                I’m going to give you one reply, but that’s it; we have such a divergence of opinion so often that I don’t feel like we’re really going to agree, but I would like to respond to some things.

                                The essay goes on to redefine “success” in some super-niche way and qualify it, but this is rhetorically dishonest.

                                I tried to be super clear here that this is from an implementor’s perspective. Success in this context means “becoming part of the web platform.” I don’t feel that’s dishonest, it’s about being clear about exactly what I’m trying to talk about.

                                JVM was never standardized.

                                That’s a specification, not a standard.

                                Flash was not built on ES3/ES4 standards.

                                “built on” does not mean “conforms with”; ActionScript and ECMAScript are similar, but different.

                                Unfortunately, just because something is standardized and went through the wringer doesn’t mean it isn’t hot steaming garbage.

                                This is true, but that’s why I qualified what this post is about. The user’s perspective is something else entirely.

                                Most of those features didn’t exist when Java applets first came out in 1995.

                                CSS was in development at the time, and shipped in 1996, it’s true. But regardless of the start, that kept being true. They could have added support, but they did not. It’s effectively the same.

                                https://caniuse.com/ is a monument to this fact.

                                CanIUse, to me, is more about when you can adopt new features. They ship in different browsers at different times, but eventually, a lot of stuff is the same. Crossplatform web development has never been easier.

                                Runtimes don’t all need to be integrated, and we handily managed to keep the runtimes for JVM and Flash maintained for more than a decade, by letting interested parties support them.

                                Yet they still were full of vulnerabilities. This was due to the disconnect I talk about in the article. It also doesn’t address the inherent complexity in coordinating two runtimes compared to one.

                                enables malware

                                This was already possible with JS, wasm doesn’t fundamentally change things here

                                and side-channel attacks,

                                This article is clickbait nonsense. It says that this could happen, but ignores that everyone involved with WebAssembly is acutely aware of these issues, and is not moving forward until they’re addressed. Heck, before Meltdown/Spectre was announced, I was in a room with one of the main designers of WebAssembly, and someone brought up SharedArrayBuffer for some reason. He said “yeah so you shouldn’t rely on that and I can’t talk about why”. Then we all found out why.

                                You’re letting your disdain bias you against the facts.

                                while it’s proponents kinda ignore the issue and forge ahead.

                                TC39 is not the body that standardizes WebAssembly.

                                We’re all going to be stuck with the outcome of this, and folks promoting the technology for their own career advancement or aesthetics are doing so, seemingly to me, without enough care for the ramifications of what they’re pushing.

                                This kind of slander is why I rarely post to lobste.rs anymore. I’m out.

                                1. 2

                                  This was already possible with JS, wasm doesn’t fundamentally change things here

                                  It makes things easier, and the general increase in performance allows more interesting and obtrusive categories of malware than we saw with JS.

                                  Heck, before Meltdown/Spectre was announced, I was in a room with one of the main designers of WebAssembly, and someone brought up SharedArrayBuffer for some reason.

                                  There’s no sane way to have multiple threads and shared memory primitives of which I’m aware that don’t enable timing attacks at this time. The option seems to be to remove them entirely, and the Github link shows that at least a few people are hesitant to do that.

                                  TC39 is not the body that standardizes WebAssembly.

                                  Thank you for the correction–I don’t know if there is a lot of bleedover between the groups. If there is, my concern remains.

                                  This kind of slander is why I rarely post to lobste.rs anymore. I’m out.

                                  That’s my honest opinion, slander wasn’t my intent. This pattern is being repeated everywhere in our industry. If you don’t think it applies in your case, please correct me.

                                  1. 12

                                    That’s my honest opinion, slander wasn’t my intent.

                                    Making an unsubstantiated accusation in a public forum is slander, even if you happen to believe the accusation to be true.

                                    And to be clear, you accused promoters of wasm of self-interestedly putting career/aesthetics above the common good. You made no allowance for the idea that they might actually have decent reasons for believing and acting as they do. If they disagree with you on wasm, then they are simply a bad person.

                                    Putting all of that behind a “it seems to me” doesn’t actually change what you are saying. If you meant something else, I strongly suggest rewording. If not, then please don’t post such attacks on Lobsters.

                                    1. 3

                                      Please consider the original phrasing–there’s a reason I picked it:

                                      We’re all going to be stuck with the outcome of this, and folks promoting the technology for their own career advancement or aesthetics are doing so, seemingly to me, without enough care for the ramifications of what they’re pushing.

                                      I state one thing as fact: we’re stuck with the outcome of the wasm debate.

                                      I state the rest as opinion: the set of people promoting the technology, who are doing so for their own career advancement or aesthetics, seem to be doing so without enough care for the ramifications of widespread wasm adoption it seems to me.

                                      There is a much more direct way of slandering people if that’d been my goal.

                                      1. 5

                                        Sidestepping the question of whether it’s slander: it’s non-constructive and counterproductive to speculate about the intentions and motivations of the people you are discussing with. It’s poisoning the well.

                                    2. 6

                                      That’s my honest opinion

                                      It’s an accusation, not a matter of opinion.

                                  2. 6

                                    Browser vendors (and hell, let’s be real here: we mean Google, Apple, Microsoft, and that’s basically it these days)

                                    I’m sure you know you’re writing to a Mozilla employee. Does saying “you’re not even a real browser vendor” really help your argument?

                                    (I’ve worked at Mozilla. Loved it, but competing against some of the largest and most well-funded corporations in the world is hard, and it can be frustrating to see what they get away with. Jabs like this pile up and demoralize you.)

                                    1. 2

                                      In my haste I plain forgot to list Mozilla. One of many glaring oversights I’ve made today.

                                      That said, my point is that there are really only 4 vendors that matter, since the vast, vast majority of browsers are powered using one of 3 or 4 engines.

                                      1. 2

                                        That makes more sense. Thanks for clarifying.

                                  1. 9

                                    You either use an existing framework, or you end up re-inventing one… poorly.

                                    1. 6

                                      I’ve yet to see a web framework I truly enjoy using. Most of them don’t even try to tame the incidental complexity of the web, preferring to heap on even more. I think this is because the type of people that make web frameworks often love the web to the point where they’re blind to the incidental complexity.

                                      These frameworks seem to take special delight in taking over every aspect of your application, because ‘convention.’ Apparently, one of the greatest evils of software development is that there is no standard directory for models unless we institute The Right Way. Meanwhile, massive coupling makes testing difficult, causing years of “Fast tests using this one weird trick” presentations continue to be given.

                                      The best libraries are the ones you can lock away somewhere and forget about.

                                      1. 1

                                        Enjoyability seems to me to be a bad criterion to judge a tool on. I may enjoy one hammer more than another, but I still need to use specific ones for specific tasks and the ones I enjoy less are no less functional and suitable.

                                        I don’t think the remainder of your post depends on your remark about ‘enjoying’ a web framework, which I feel is extra evidence for that not mattering.

                                        1. 3

                                          Enjoyability seems to me to be a bad criterion to judge a tool on.

                                          Also “Enjoyability” is based on what timeline you measure it on. If you measure it based on day 1 enjoyability, day 100 enjoyability or day 1000 enjoyability. Stuff like unit testing and fuzzing might not be very enjoyable on day 1 – might be far more enjoyable on day 100 and put you a state of absolute bliss with same testing and fuzzing on day 1000.

                                        2. 0

                                          Of course web frameworks are not optimal. However, I take the leaky abstraction here and there any time over the mess I have seen with non-framework code. I did Python starting with Python 2.3 which was released in the early 2000s. Back then, I didn’t do Python web development much, yet every now and then I wrote something or looked at options how to do things. This was the time of mod_python and still cgi. Nowadays we have Django, Pyramids and if you feel like having a bit more freedom - Flask. I must say I wouldn’t want to go back.

                                          Potentially, if you have very special requirements, that actively go against the typical patterns, no framework is an option, but other wise it isn’t. at least I wouldn’t like to take over maintenance of such a codebase.

                                          That’s actually quite a nice indicator: “If someone would use that advice, would I like to take over maintenance of that code base?”.

                                      1. 5

                                        Whether it is Material Design, Bootstrap, or some other pseudo-standard there is a lot of advantage in starting from a point of things “working/looking the same” across products. If “it just works” for the end user because they have encountered the same design choices on multiple other apps/sites they use then it is a convenience to the end user.

                                        There are definitely cases where completely unique designs are good, but this argument that it is a political statement to use Material Design is weak, but it went beyond weak to down-right offensive by comparing Material Design to the Nazi swastika.

                                        1. 3

                                          I think you misinterpret that comparison. In my reading it wasn’t comparing the swastika with Material Design. It was explaining how “nothing is neutral”, by noting that although a swastika is in principle just an abstract shape, no one in the Western world can view it like that any more: it has become a very meaningful shape, in whatever form it appears.

                                          I think it’s not a very good explanation, because it takes one of the most strong examples to explain a principle that seems to apply in at most moderate extent to Material Design

                                          1. 2

                                            I picked up this part of the analogy as well, and if the only intent was to show that symbols take on meaning over time then something like a dove (symbol of peace & religion) could be used, or the caduceus (universal symbol for a medical alert), or many many others.

                                            It is my opinion based on the rest of the content in the article that the author chose the swastika intentionally to evoke a strong negative emotion from readers that they would then transfer onto Google and Material Design whether consciously or unconsciously.

                                            1. 2

                                              I didn’t get that impression. I think the author was just looking for a strong example of a symbol evoking an idea due to past context, rather than anything inherent about its shape.

                                        1. 1

                                          The license of this article doesn’t allow derivative works.

                                          It seems in conflict with arguing for right to repair if you don’t want the right to remix..

                                          1. 7

                                            The article is was freely given. I have no doubt you can offer the author something for a license to remix their content. The bike was already paid for.

                                            1. 7

                                              Could you tell me why one would one to remix an article? The current license lets you quote the whole thing or sections of it for use in your own work. I’m not sure what extra would be gained by derivatives but I am happy to change it if there is reason.

                                            1. -1

                                              As one insignificant user of this language, please stop adding these tiny edge case syntax variations and do something about performance. But I am one small insignificant user …

                                              1. 56

                                                This is exactly the attitude that leads to maintainers’ burn outs.

                                                Do realize this:

                                                • Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.
                                                • Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.
                                                • If you want something to happen in an open source project, volunteer to do it.

                                                (None of this is aimed at you personally, I don’t know who you are. I’m dissecting an attitude that you’ve voiced, it’s just all too common.)

                                                1. 23

                                                  Python is not a product, and you’re not a paying customer, you don’t get to say “do this instead of that” because none of the volunteer maintainers owes you to produce a language for you. Just walking by and telling people what to do with their project is at the very least impolite.

                                                  I agree with the general direction of your post, but Python is a product and it is marketed to people, through the foundation and advocacy. It’s not a commercial product (though, given the widespread industry usage, you could argue it somewhat is). It’s reasonable of users to form expectations.

                                                  Where it goes wrong is when individual users claim that this also means that they need to be consulted or their consultation will steer the project to the better. http://www.ftrain.com/wwic.html has an interesting investigation of that.

                                                  1. 2

                                                    Where it goes wrong is when users claim that this also means that they need to be consulted or their consultation will steer the project to the better.

                                                    Wait, who is the product being built for, if not the user? You can say I am not a significant user, so my opinion is not important, as opposed to say Google which drove Python development for a while before they focused on other things, but as a collective, users’ opinions should matter. Otherwise, it’s just a hobby.

                                                    1. 5

                                                      Sorry, I clarified the post: “individual users”. There must be a consultation process and some way of participation. RFCs or PEPs provide that.

                                                      Yet, what we regularly see is people claiming how the product would be a better place if we listened to them (that, one person we never met). Or, alternatively, people that just don’t want to accept a loss in a long-running debate.

                                                      I don’t know if that helps clarifying, it’s a topic for huge articles.

                                                      1. 3

                                                        I often find what people end up focusing on - like this PEP - is bike shedding. It’s what folks can have an opinion on after not enough sleep and a zillion other things to do and not enough in depth knowledge. Heck I could have an opinion on it. As opposed to hard problems like performance where I would not know where to start, much less contribute any code, but which would actually help me and, I suspect, many other folks, who are with some sighing, migrating their code to Julia, or, like me, gnashing their teeth at the ugliness of Cython.

                                                        1. 4

                                                          Yeah, it’s that kind of thing. I take a harsh, but well-structured opinion any time and those people are extremely important. What annoys me is people following a tweet-sized mantra to the end, very much showing along the path that they have not looked at what is all involved or who would benefit or not knowing when to let go off a debate.

                                                  2. 17

                                                    Adding syntax variations is not done at the expense of performance, different volunteers are working on what’s more interesting to them.

                                                    Regrettably, a lot of languages and ecosystems suffer greatly from the incoherence that this sort of permissive attitude creates.

                                                    Software is just as much about what gets left out as what gets put in, and just because Jane Smith and John Doe have a pet feature they are excited about doesn’t mean they should automatically be embraced when there are more important things on fire.

                                                    1. 8

                                                      the incoherence that this sort of permissive attitude creates

                                                      The Haskell community would’ve just thrown PEP 572 behind {-# LANGUAGE Colonoscopy #-} and been done with it.

                                                      Sure, this doesn’t get us out of jail free with regard to incoherence, but it kicks down the problem from the language to the projects that choose to opt-in.

                                                      1. 2

                                                        I find it hard to see this as a good thing. For me, it mostly highlights why Haskell is a one-implementation language… er, 2 ^ 227 languages, if ghc --supported-extensions | wc -l is to be taken literally. Of course, some of those extensions are much more popular than others, but it really slows down someone trying to learn “real world” Haskell by reading library code.

                                                        1. 2

                                                          Of course, some of those extensions are much more popular than others

                                                          Yeah, this is a pretty interesting question! I threw some plots together that might help explore it, but it’s not super conclusive. As with most things here, I think a lot of this boils down to personal preference. Have a look:

                                                          https://gist.github.com/atondwal/ee869b951b5cf9b6653f7deda0b7dbd8

                                                      2. 4

                                                        Yes. Exactly this. One of the things I value about Python is its syntactic clarity. It is the most decidedly un-clever programming language I’ve yet to encounter.

                                                        It is that way at the expense of performance, syntactic compactness, and probably some powerful features that could make me levitate and fly through the air unaided if I learned them, but I build infrastructure and day in, day out, Python gets me there secure in the knowledge that I can pick up anyone’s code and at the VERY LEAST understand what the language is doing 99% of the time.

                                                      3. 4

                                                        I find that “people working on what interests them” as opposed to taking a systematic survey of what use cases are most needed and prioritizing those is a hard problem in software projects, and I find it curious that people think this is not a problem to be solved for open source projects that are not single writer/single user hobby projects.

                                                        Python is interesting because it forms core infrastructure for many companies, so presumably they would be working on issues related to real use cases. Projects like numpy and Cython are examples of how people see an important need (performance) and go outside the official language to get something done.

                                                        “If you want something to happen in an open source project, volunteer to do it.” is also one of those hostile attitudes that I find curious. In a company with a paid product of course that attitude won’t fly, but I suspect that if an open source project had that attitude as a default, it would gradually lose users to a more responsive one.

                                                        As an example, I want to use this response from a library author as an example of a positive response that I value. This is a library I use often for a hobby. I raised an issue and the author put it in the backlog after understanding the use case. They may not get to it immediately. They may not get to it ever based on prioritization, but they listened and put it on the list.

                                                        Oddly enough, I see this kind of decent behavior more in the smaller projects (where I would not expect it) than in the larger ones. I think the larger ones with multiple vendors contributing turn into a “pay to play” situation. I don’t know if this is the ideal of open source, but it is an understandable outcome. I do wish the hostility would decrease though.

                                                        1. 12

                                                          Performance has never been a priority for Python and this probably won’t change, because as you said, there are alternatives if you want Python’s syntax with performance. Also its interoperability with C is okeish and that means that the small niche of Python’s users that use it for performance critical operations that are not already supported by Numpy, Numba and so on, will always be free to go that extra mile to optimize their code without much trouble compared to stuff like JNI.

                                                          If you want raw performance, stick to C/C++ or Rust.

                                                          1. 3

                                                            I also observe the same tendency of smaller projects being more responsive, but I think the issue is scale, not “pay to play”. Big projects get so much more issue reports but their “customer services” are not proportionally large, so I think big projects actually have less resource per issue.

                                                          2. 0

                                                            He did say “please”.

                                                          3. 7

                                                            please stop adding these tiny edge case syntax variations and do something about performance.

                                                            There’s a better forum, and approach, to raise this point.

                                                            1. 2

                                                              I guess you are saying my grass roots campaign to displace “Should Python have :=” with “gradual typing leading to improved performance” as a higher priority in the Python world is failing here. I guess you are right :)

                                                            2. 1

                                                              Have you tried Pypy? Have you tried running your code through Cython?

                                                              Have you read any of the zillion and one articles on improving your Python’s performance?

                                                              If the answer to any of these is “no” then IMO you lose the right to kvetch about Python’s performance.

                                                              And if Python really isn’t performant enough for you, why not use a language that’s closer to the metal like Rust or Go or C/C++?

                                                              1. 6

                                                                Yes to all of the above. But not understanding where all the personal hostility is coming from. Apparently having the opinion that “Should := be part of Python” is much less important than “Let’s put our energies towards getting rid of the GIL and creating a kickass implementation that rivals C++” raises hackles. I am amused, entertained but still puzzled at all the energy.

                                                                1. 5

                                                                  There was annoyance in my tone, and that’s because I’m a Python fan, and listening to people kvetch endlessly about how Python should be something it isn’t gets Ooooold when you’ve been listening to it for year upon year.

                                                                  I’d argue that in order to achieve perf that rivals C++ Python would need to become something it’s not. I’d argue that if you need C++ perf you should use C++ or better Rust. Python operates at a very high level of abstraction which incurs some performance penalties. Full stop.

                                                                  1. 5

                                                                    This is an interesting, and puzzling, attitude.

                                                                    One of the fun things about Cython was watching how the C++ code generated approaches “bare metal” as you add more and more type hints. Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.

                                                                    Haskell is a high level language that compiles down to very efficient code (barring laziness and thunks and so on).

                                                                    Yes, I find this defense of the slowness of Python (not just you but by all commentators here) and the articulation that I, as one simple, humble user, should just shut up and go away kind of interesting.

                                                                    I suspect that it is a biased sample, based on who visits this post after seeing the words “Guido van Rossum”

                                                                    1. 8

                                                                      My hypothesis is that people who want performance are minority among Python users. I contributed to both PyPy and Pyston. Most Python users don’t seem interested about either.

                                                                      1. 3

                                                                        For me that has been the most insightful comment here. I guess the vast majority of users employ it as glue code for fast components, or many other things that don’t need performance. Thanks for working on pypy. Pyston I never checked out.

                                                                      2. 5

                                                                        Not clear at all to me why Python can not become something like Typed Racket, or LISP with types (I forget what that is called) that elegantly sheds dynamism and gets closer to the metal the more type information it gets.

                                                                        Isn’t that what mypy is attempting to do? I’ve not been following Python for years now, so really have no horse in this race. However, I will say that the number of people, and domains represented in the Python community is staggering. Evolving the language, while keeping everyone happy enough to continue investing in it is a pretty amazing endeavor.

                                                                        I’ll also point out that Python has a process for suggesting improvements, and many of the core contributors are approachable. You might be better off expressing your (valid as far as I can see) concerns with them, but you might also approach this (if you care deeply about it) by taking on some of the work to improve performance yourself. There’s no better way to convince people that an idea is good, or valid than to show them results.

                                                                        1. 4

                                                                          Not really. Mypy’s goal is to promote type safety as a way to increase program correctness and reduce complexity in large systems.

                                                                          It doesn’t benefit performance at all near as I can tell, at least not in its current incarnation.

                                                                          Cython DOES in fact do this, but the types you hint with there are C types.

                                                                          1. 2

                                                                            Ah, I thought maybe MyPy actually could do some transformation of the code, based on it’s understanding, but it appears to describe itself as a “linter on steroids,” implying that it only looks at your code in a separate phase before you run it.

                                                                            Typed Racket has some ability to optimize code, but it’s not nearly as sophisticated as other statically typed languages.

                                                                          2. 3

                                                                            Be aware that even Typed Racket still has performance and usability issues in certain use cases. The larger your codebase, the large the chance you will run into them. The ultimate viability of gradual typing is still an open question.

                                                                          3. 3

                                                                            In no way did I imply that you should “shut up and go away”.

                                                                            What I want is for people who make comments about Python’s speed to be aware of the alternatives, understand the trade-offs, and generally be mindful of what they’re asking for.

                                                                            I may have made some false assumptions in your case, and for that I apologize. I should have known that this community generally attracts people who have more going on than is the norm (and the norm is unthinking end users posting WHY MY CODE SO SLOW?

                                                                            1. 2

                                                                              Hey, no problem! I’m just amused at the whole tone of this set of threads set by the original response (not yours) to my comment, lecturing me on a variety of things. I had no idea that (and can’t fathom why) my brief comment regarding prioritization decisions of a project would be taken so personally and raise so much bile. What I’m saying is also not so controversial - big public projects have a tendency to veer into big arguments over little details while huge gaps in use cases remain. I saw this particular PEP argument as a hilarious illustration of this phenomenon in how Python is being run.

                                                                              1. 3

                                                                                Thinking about this a little more - sometimes, when languages ‘evolve’ I feel like they forget themselves. What makes this language compelling for vast numbers of programmers? What’s the appeal?

                                                                                In Python’s case, there are several, but two for sure are a super shallow learning curve, and its tendency towards ‘un-clever’ syntax.

                                                                                I worry that by morphong into something else that’s more to your liking for performance reasons, those first two tenets will get lost in the shuffle, and Python will lose its appeal for the vast majority of us who are just fine with Python’s speed as is.

                                                                                1. 1

                                                                                  Yes, though we must also remember that as users of Python, invested in it as a user interface for our code ideas, we are resistant to any change. Languages may lose themselves, but changes are sometimes hugely for the better. And it can be hard to predict.

                                                                                  In Python’s 2.x period, what we now consider key features of the language, like list comprehensions and generator expressions and generators, were “evolved” over a base language that lacked those features altogether, and conservatives in the community were doubtful they’d get much use or have much positive impact on code. Likewise for the class/type system “unification” before that. Python has had a remarkable evolutionary approach over its long 3-decade life, and will continue to do so even post-GvR. That may be his true legacy.

                                                                      3. 1

                                                                        Heh. I think this is an example of the Lobste.rs rating system working as it should :) I posted an immoderate comment borne of an emotional response to a perfectly reasonable reply, and end up with a +1: +4 -2 troll, -1 incorrrect :)

                                                                    1. 7

                                                                      The viewer doesn’t give a shit about that.

                                                                      Some viewers may not. For others, the story of how the work of art came about is what matters. The artwork is just the final chapter of the story, not interesting in itself.

                                                                      So you may as well argue that most regular art is hurt by a lack of intellectualism.

                                                                      1. 2

                                                                        So you may as well argue that most regular art is hurt by a lack of intellectualism.

                                                                        Are you arguing this? Because there’s a wealth of evidence to the contrary.

                                                                        Just for example, here’s a random pick from the NY Review website:

                                                                        https://www.nybooks.com/articles/2018/07/19/sculpture-bodies-spitting-image/

                                                                        (link slightly NSFW, image of two nude torsos)

                                                                        1. 1

                                                                          No, my point is that someone ‘intellectually inclined’ could argue that in a way consistent with the author’s argument, which I then expect the author, like you, to disagree with. So the argument seems internally inconsistent to me.

                                                                          So to explicate: if the author feels this artform is hurt by too much intellectualism (apperently at the cost of inspiration, because the main complaint in the article pertains the lack thereof; the article may as well have been titled ‘lack of inspiration hurts generative art’) then the ratio of intellectualism to inspiration must be lower than it is for other artforms. But if one appreciates the intellectual component, that means they would probably think the other artforms would be short on intellectualism and they could argue that contrapositive with the same argument.

                                                                          What I think is that intellectualism and inspiration aren’t mutually exclusive and that the author is arguing ‘replacement’ of inspiration by intellectualism to not have to come out and openly say ‘generative artists lack inspiration’.

                                                                      1. 5

                                                                        Just plumbing… The question I always feel compelled to ask: has the author ever actually done any plumbing? Or at least thoroughly thought about it?

                                                                        I have not done much plumbing, but I’ve done my share of home improvement projects and know people with jobs often viewed as ‘boring’, repetitive and non-creative.

                                                                        To continue the plumbing metaphor: if you’re renovating, you run into all kinds of non-standard setups, walls that are already tiled, non-standard existing piping, … But even greenfield plumbing is nontrivial and challenging. You don’t get to design the house, they may be pretty late in bringing you in, there will be constraints by wall and floor materials, supporting walls, slightly non-standard requests, etc.

                                                                        Every project is different and requires creativity. You’re often better off paying a bit more for a smarter plumber.

                                                                        1. 3

                                                                          Decision tables are useful on a whiteboard, but manually checking a decision table for completeness can be surprisingly tricky and error prone. They really benefit from being written down in a (programming) language. Then they are quite easy to check for completeness and also pretty easy to minimize (figure out “don’t cares”, columns subsumes by other columns, etc. As an extension, you can assign (multiple) actions to be taken for a certain valuation of the conditions.

                                                                          The tricky things are supporting a nice way to specify the tables in your language and a sufficiently powerful way of specifying the conditions to evaluate and actions to perform.

                                                                          1. 3

                                                                            No, it’s vector multiplication in disguise as a markov chain.

                                                                            1. 3

                                                                              That seems like a category mistake the me, whereas the title of the article doesn’t.

                                                                              A Markov chain may be a specific pattern of vector multiplications, but that pattern makes all the difference. Markov chains and vector multiplications are on a different level. On the other hand ‘deep learning’ and ‘Markov chain’ are terms for alternative patterns of vector multiplications, one a lot more involved than the other.

                                                                              1. 2

                                                                                There’s a video on YT somewhere of a talk by a physicist (IIRC) on why deep learning is so ridiculously effective - it pretty much boils down to the same reason that mathematics is so unreasonably effective in describing physical systems in general, i.e. (handwaving extremely wildly from memory) that physical systems tend to be simple functions of their inputs (albeit with many, many inputs!) where causality is preserved. This is what makes it possible for RNNs and the like to approximate physical systems in various ways, because the nature of said physical systems is exactly what permits approximations of the information content of the system to be at least partially valid instead of being a total loss.

                                                                                (I tried to find the video, but there are too many terrible ones on the same topic these days. I’ll have another look later.)

                                                                              2. 4

                                                                                No, it’s a monoid in the category of endofunctors.

                                                                              1. 4

                                                                                My experience is that this is just utterly wrong - I’m not even sure how to start to respond. Of course the best way to express a program fragment is a programming language. Of course the best way to think about a program is with a programming language. There is no distinction between programming and mathematics - of course you want to think mathematically about what you’re constructing, but the best languages for that are programming languages. Why would you want two subtly different descriptions of your program that need to be kept in sync when you could have one description of your program? Some programming languages distract from writing a good expression of your construction by making you specify irrelevant execution details, but the appropriate response is to avoid those languages. A good mathematical description of your algorithm is a program that implements your algorithm, given a decent compiler - and thankfully we’re good enough at those these days.

                                                                                1. 9

                                                                                  Lobster’s own Hillel expressed it really well just a few days ago:

                                                                                  So many software development practices - TDD, type-first, design-by-contract, etc - are all reflections of the same core idea:

                                                                                  1. Plan ahead.
                                                                                  2. Sanity-check your plan

                                                                                  It’s reasonable to want that “plan ahead” stuff to be incorporated in the program (design by contract, tdd), but using an external plan can have a large chunk of the benefit.

                                                                                  1. 6

                                                                                    Maybe, but that’s not an argument for using a non-integrated, harder-to-check plan if you have the option of building the “plan” right into the program.

                                                                                    1. 4

                                                                                      Because there’s design tradeoffs in specification. Integration is a pretty big benefit but also a pretty big cost, often reducing your expressiveness (what you can say) and your legibility (what properties you can query). As a couple of examples, you can’t use integrable specifications to assert a global property spanning two independent programs. You also can’t distinguish between what are possible states of the system and what are valid states, or what behavioral properties must be satisfied.

                                                                                  2. 7

                                                                                    Strong disagree here; you get a lot more expressive power when using a specification language. Let me pose a challenge: given a MapReduce algorithm with N workers and 1 reducer, how do you specify the property “if at least one worker doesn’t crash or stall out, eventually the reducer obtains the correct answer”? In TLA+ it’d look something like this:

                                                                                    (\E w \in Workers: WF_vars(Work(w))) /\ WF_vars(Reducer) 
                                                                                      => <>[](reducer.result = ActualResult)
                                                                                    

                                                                                    Why would you want two subtly different descriptions of your program that need to be kept in sync when you could have one description of your program?

                                                                                    I’ve written 100 line TLA+ specs that captured the behavior of 2000+ lines of Ruby. Keeping them in sync is not that hard.

                                                                                    1. 9

                                                                                      I’ve written 100 line TLA+ specs that captured the behavior of 2000+ lines of Ruby. Keeping them in sync is not that hard.

                                                                                      Keeping code in sync with comments that literally live along side them is even more “not that hard”, and yet fails to happen on an incredibly regular basis.

                                                                                      In my experience, in any given system where two programmer artifacts have to be kept in sync manually, they will inevitably fall out of sync, and the resulting conflicting information, and confusion or mistaken assumption of which one is correct, will result in bugs and other programmer errors impacting users. The solution is usually to either generate one artifact from the other, or try to restructure one artifact such that it obviates the need for the other.

                                                                                      1. 4

                                                                                        Keeping code in sync with comments that literally live along side them is even more “not that hard”, and yet fails to happen on an incredibly regular basis.

                                                                                        The difference is that if your code falls out of sync with your comments, your comments are wrong. But if your code falls out of sync with your formal spec, your code probably has a subtle bug. So there’s a lot more instutional pressure to update your spec when you update the code, just to make sure it still satisfies all of your properties.

                                                                                        The solution is usually to either generate one artifact from the other, or try to restructure one artifact such that it obviates the need for the other.

                                                                                        This has been a cultural problem with formal methods for a long time: people don’t value specifications that aren’t directly integrated into code. This has held the field back, because actually getting direct integration is really damn hard. It’s only in the past 15ish years that we’ve accepted that it’s alright to write specs that can’t generate code, and that’s why Alloy and TLA+ are becoming more popular now.

                                                                                        1. 6

                                                                                          The difference is that if your code falls out of sync with your comments, your comments are wrong. But if your code falls out of sync with your formal spec, your code probably has a subtle bug

                                                                                          What justifies that assumption. Some junior will inevitably, in response to some executive running in with their hair on fire over some “emergency”, alter the behaviour of the code to “get it done quick” and defer updating the spec until a “later” that may or may not ever arrive. Coming along and then altering the code to meet the spec then re-introduces the emergency situation.

                                                                                          The fundamental problem here is that you’ve created two sources of truth about what the application should be doing, and you cannot a priori conclude that one or the other is always the correct one.

                                                                                          1. 6

                                                                                            And what happens when that “emergency” fix loses your client data, or breaks your data structure, or ruins your consistency model, or violates your customer requirements, or melts your xbox, or drops completed jobs?

                                                                                            Yes, it’s true that sometimes the spec needs to be changed to match changing circumstances. It’s also seen again and again that specs catch serious bugs and that diverging from them can be seriously dangerous.

                                                                                            1. 2

                                                                                              And what happens when that “emergency” fix loses your client data, or breaks your data structure, or ruins your consistency model, or violates your customer requirements, or melts your xbox, or drops completed jobs?

                                                                                              Nobody’s arguing that the spec is useless, just that the reality is that it does introduce risks that require care and attention and which cannot be handwaved away with “keeping them in sync is not that hard” because sync issues will bite organizations in the ass.

                                                                                          2. 2

                                                                                            It’s only in the past 15ish years that we’ve accepted that it’s alright to write specs that can’t generate code, and that’s why Alloy and TLA+ are becoming more popular now.

                                                                                            It would be very helpful if you could at least generate test cases from those specs though. But then that’s why I work on a model based testing tool ;)

                                                                                            1. 2

                                                                                              Which one?

                                                                                              1. 3

                                                                                                Proprietary of my employer (Axini). Based on symbolic transition systems, a Promela and LOTOS inspired modeling language and the ioco conformance relation. Related open source tools are TorX/JTorX/TorXakis. Our long term goal is model checking, but we believe model based testing is a good (necessary?) intermediate step to convince the industry of the added value by providing a way where formal modeling can directly help them test their software more thoroughly.

                                                                                                1. 2

                                                                                                  Really neat stuff. Thanks. I’ll try to keep Axini in mind if people ask about companies to check out.

                                                                                                  1. 2

                                                                                                    Thanks, I’ve also been regularly forwarding articles and comments by you to colleagues :)

                                                                                                    1. 1

                                                                                                      Cool! Glad thry might be helpig yall out.:)

                                                                                          3. 2

                                                                                            This was a problem in high-assurance systems. All write-ups indicated it takes discipline. That’s no surprise given that’s what good systems take to build regardless of method. Many used tools like Framemaker to keep it all together. That said, about every case study I can remember found errors via the formal specs. Whether it was easy or not, they all thought they were beneficial for software quality. It was formal proof that varied considerably in cost and utility.

                                                                                            In Cleanroom, they use semi-formal specs meant for human eyes that are embedded right into the code as comments. There was tooling from commercial suppliers to make its process easier. Eiffel’s Design-by-Contract kept theirs in the code as well with EiffelStudio layering benefits on top of that like test generation. Same with SPARK. The coder that doesn’t change specs with code or vice versa at that point is likely just being lazy.

                                                                                        2. 3

                                                                                          Why would you want two subtly different descriptions of your program that need to be kept in sync when you could have one description of your program?

                                                                                          For example to increase the number and variety of reviewers and thus reducing bugs.

                                                                                          A good mathematical description of your algorithm is a program that implements your algorithm

                                                                                          Are you thinking of a specific compiler? I agree that programmers think mathematically even when they use programming languages to express their reasoning, but I still feel some “impedence” in every language I use.

                                                                                          1. 3

                                                                                            For example to increase the number and variety of reviewers and thus reducing bugs.

                                                                                            That seems very unlikely though. Even obscure programming languages are better-known than TLA+. More generally I can’t imagine getting valuable input on this kind of subject from anyone who wasn’t capable of understanding a programming language. I find the likes of Cucumber to be worse than useless, and the theoretical rationale for those is stronger since test cases seem further away from the essence of the program than program analysis is.

                                                                                            Are you thinking of a specific compiler? I agree that programmers think mathematically even when they use programming languages to express their reasoning, but I still feel some “impedence” in every language I use.

                                                                                            I mostly work in Scala so I guess that influences my thoughts. There are certainly improvements to the language that I can imagine, but not enough to be worth using something other than the official language when communicating with other people.

                                                                                        1. 1

                                                                                          I think this article makes the case that it is valuable to think of all code as temporary, replaceable or disposable. Don’t be attached to code or a feature just because it already exists. The world changes, your company changes, your customers change and you change and even the best code responsible for your formerly most treasured feature may have to been thrown out at some point.

                                                                                          But I agree with @leeg that that does not make it ‘technical debt’, nor do I think it makes sense to expand the meaning of ‘technical debt’ to include this way of viewing your codebase

                                                                                          1. 12

                                                                                            Explanation of the name for non-German speaking folks: Zwiebel is German for Onion and ‘freunde’ are friends.

                                                                                            1. 1

                                                                                              There seems to be some unspoken assumption here that SQL is simpler/easier/cheaper than ML/AI. That’s not my experience: everything you can do with modern tools is possible in SQL, sure, but the modern tools make it much easier and eliminate a lot of the pitfalls.

                                                                                              1. 5

                                                                                                I don’t follow: what kind of modern ML/AI tools are you thinking of (as simple/easy/cheap as SQL)?

                                                                                                1. 1

                                                                                                  Mainly Spark. Hardware performance/cost is probably worse unless you’re on a really big dataset, but for me it more than made up for it in programmer cost: I found it so much easier to answer questions based on our data when I could use a shell in a normal programming language with access to our data as normal values and just call e.g. .aggregateByKey and pass code to do what I wanted.

                                                                                                  1. 5

                                                                                                    Calling aggregateByKey isn’t using ML, it’s using straightforward querying of a dataset. Spark isn’t an ML solution so much as it’s a non-relational data store that’s queried differently than SQL-based data stores.

                                                                                                    You can build ML solutions on top of Spark, much the same as you can on top of SQL.

                                                                                                    The big difference between the two is instead the declarative vs. imperative nature of querying. With SQL, you describe what you want, whereas with Spark (and a number of other big data and nosql stores), you describe how to get it. The latter is more familiar to many imperative/OO programmers, but the former is generally more approachable to non-programmers, and tends to deal with changes to data more smoothly.

                                                                                                    In fact, the declarative approach is useful enough that Spark SQL exists and is widely used.

                                                                                                    1. 1

                                                                                                      So maybe: “you don’t necessarily need ML/AI: you should consider just using SQL if you already know it.“?

                                                                                                      1. 1

                                                                                                        I still don’t follow: as far as I know, Spark is not a ML/AI tool.