Threads for felixyz

  1. 1

    Totally unrelated but you have come to a very Leibnizian conclusion.

    1. 2

      That’s interesting. How so?

        1. 3

          I know Leibniz would agree with “monads are everywhere”, I was hoping to hear something about Leibniz thinking “Maybe that’s bad”, that would be news to me. :)

          1. 2

            That would definitely be exciting :)

    1. 7

      How does this new relational query language compare to the family of relational query languages specified by TTM / The Third Manifesto / Tutorial D, based on the relational algebra? The sales pitch for PRQL (a small number of powerful, orthogonal primitives that can be freely composed together, more expressive than SQL) sounds like the sales pitch for TTM and the relational algebra, so is there a connection?

      1. 5

        Very good question! It’s unfortunate that virtually all “SQL but better” initiatives don’t bother to look at the solid foundations of relational algebra and relational calculus, and more specifically the third manifesto manifestations. I’m not saying there are no further improvements to be made, but that should be the starting point.

        1. 2

          I’m surprised that you thought about D. To me D is “Relations are everything. You’re going to think and write in terms of types and relations. So much relations that we won’t even list ordering in examples.”

          On the other hand PRQL is “Lines are pretty much joined by and-then. You write imperative-ish code and we’ll do relations in the background.” I see them almost like opposite ideas that compile to the same lower language.

          1. 1

            Love to see Tutorial D mentioned. My favorite implementation is Alf in Ruby since it taught me the most.

          1. 12

            The lesson here sounds more like “bad protocols will make your client/server system slow and clumsy”, not “move all of your system’s code to the server.” The OP even acknowledges that GraphQL would have helped a lot. (Or alternatively something like CouchDB’s map/reduce query API.)

            I don’t really get the desire to avoid doing work on the client side. Your system includes a lot of generally-quite-fast CPUs provided for free by users, and the number of these scales 1::1 with the number of users. Why not offload work onto them from your limited and costly servers? Obviously you’re already using them for rendering, but you can move a lot of app logic there too.

            I’m guessing that the importance of network protocol/API design has been underappreciated by web devs. REST is great architecturally but if you use it as a cookie-cutter approach it’s non-optimal for app use. GraphQL seems a big improvement.

            1. 17

              Your system includes a lot of generally-quite-fast CPUs provided for free by users

              Yes, and if every site I’m visiting assumes that, then pretty quickly, I no longer have quite-fast CPUs to provide for free, as my laptop is slowly turning to slag due to the heat.

              1. 8

                Um, no. How many pages are you rendering simultaneously?

                1. 3

                  I usually have over 100 tabs open at any one time, so a lot.

                  1. 5

                    If your browser actually keeps all those tabs live and running, and those pages are using CPU cycles while idling in the background and the browser doesn’t throttle them, I can’t help you… ¯\_(ツ)_/¯

                    (Me, I use Safari.)

                    1. 3

                      Yes, but assuming three monitors you likely have three, four windows open. That’s four active tabs, Chrome put the rest of them to sleep.

                      And even if you only use apps like the one from the article, and not the well-developed ones like the comment above suggests, it’s maybe five of them at the same time. And you’re probably not clicking frantically all over them at once.

                      1. 2

                        All I know is that when my computer slows to a crawl the fix that usually works is to go through and close a bunch of Firefox tabs and windows.

                        1. 4

                          There is often one specific tab which for some reason is doing background work and ends up eating a lot of resources. When I find that one tab and close it my system goes back to normal. Like @zladuric says, browsers these days don’t let inactive tabs munch resources.

                2. 8

                  I don’t really get the desire to avoid doing work on the client side.

                  My understanding is that it’s the desire to avoid some work entirely. If you chop up the processing so that the client can do part of it, that carries its own overhead. How do you feel about this list?

                  Building a page server-side:

                  • Server: Receive page request
                  • Server: Query db
                  • Server: Render template
                  • Server: Send page
                  • Client: Receive page, render HTML

                  Building a page client-side:

                  • Server: Receive page request
                  • Server: Send page (assuming JS is in-page. If it isn’t, add ‘client requests & server sends the JS’ to this list.)
                  • Client: Receive page, render HTML (skeleton), interpret JS
                  • Client: Request data
                  • Server: Receive data request, query db
                  • Server: Serialize data (usu. to JSON)
                  • Server: Send data
                  • Client: Receive data, deserialize data
                  • Client: Build HTML
                  • Client: Render HTML (content)

                  Compare the paper Scalabiilty! But at what COST!, which found that the overhead of many parallel processing systems gave them a high “Configuration to Outperform Single Thread”.

                  1. 4

                    That’s an accurate list… for the first load! One attraction of doing a lot more client-side is that after the first load, the server had the same list of actions for everything you might want to do, while the client side looks more like:

                    • fetch some data
                    • deserialize it
                    • do an in-place rerender, often much smaller than a full page load

                    (Edit: on rereading your post your summary actually covers all requests, but missed how the request and response and client-side rerender can be much smaller this way. But credit where due!)

                    That’s not even getting at how much easier it is to do slick transitions or to maintain application state correctly across page transitions. Client side JS state management takes a lot of crap and people claim solutions like these are simpler but… in practice many of the sites which use them have very annoying client side state weirdness because it’s actually hard to keep things in sync unless you do the full page reload. (Looking at you, GitHub.)

                    1. 6

                      When I’m browsing on mobile devices I rarely spend enough time on any single site for the performance benefits of a heavy initial load to kick in.

                      Most of my visits are one page long - so I often end up loading heavy SPAs when a lighter, single page optimized to load fast from an un sched blank state would have served me much better.

                      1. 4

                        I would acknowledge that this is possible.

                        But that’s almost exactly what the top comment said. People use framework of the day for a blog. Not flattening it, or remixing it or whatever.

                        SPAs that I use are things like Twitter, the tab is likely always there. (And on desktop i have those CPU cores.)

                        It’s like saying, I only ride on trains to work, and they’re always crowded, so trains are bad. Don’t use trains if your work is 10 minutes away.

                        But as said, I acknowledge that people are building apps where they should be building sites. And we suffer as the result.

                        What still irks me the most are sites with a ton of JavaScript. So it’s server-rendered, it just has a bunch of client-side JavaScript that’s unused, or loading images or ads or something.

                    2. 4

                      You’re ignoring a bunch of constant factors. The amount of rendering to create a small change on the page is vastly smaller than that to render a whole new page. The most optimal approach is to send only the necessary data over the network to create an incremental change. That’s how native client/server apps work.

                      1. 5

                        In theory yes but if in practice if the “most optimal approach” requires megabytes of JS code to be transmitted, parsed, executed through 4 levels of interpreters culminating in JIT compiling the code to native machine code all the while performing millions of checks to make sure that this complex system doesn’t result in a weird machine that can take over your computer then maybe sending a “whole new page” consisting of 200 kb of static HTML upon submitting a form would be more optimal.

                        1. 4

                          In theory yes but if in practice if the “most optimal approach” requires megabytes of JS code to be transmitted, parsed, executed through 4 levels of interpreters culminating in JIT compiling the code to native machine code all the while performing millions of checks to make sure that this complex system doesn’t result in a weird machine that can take over your computer

                          This is hyperbole. Sending a ‘“whole new page” of 200 kb of static HTML’ has your userspace program block on the kernel as bytes are written into some socket buffer, NIC interrupts the OS to grab these bytes, the NIC generates packets containing the data, userspace control is then handed back to the app which waits until the OS notifies it that there’s data to read, and on and on. I can do this for anything on a non-embedded computer made in the last decade.

                          Going into detail for dramatic effect doesn’t engage with the original argument nor does it elucidate the situation. Client-side rendering makes you pay a one-time cost for consuming more CPU time and potentially more network bandwidth for less incremental CPU and bandwidth. That’s all. Making the tradeoff wisely is what matters. If I’m loading a huge Reddit or HN thread for example, it might make more sense to load some JS on the page and have it adaptively load comments as I scroll or request more content. I’ve fetched large threads on these sites from their APIs before and they can get as large as 3-4 MB when rendered as a static HTML page. Grab four of these threads and you’re looking at 12-16 MB. If I can pay a bit more on page load then I can end up transiting a lot less bandwidth through adaptive content fetching.

                          If, on the other hand, I’m viewing a small thread with a few comments, then there’s no point paying that cost. Weighing this tradeoff is key. On a mostly-text blog where you’re generating kB of content, client-side rendering is probably silly and adds more complexity, CPU, and bandwidth for little gain. If I’m viewing a Jupyter-style notebook with many plots, it probably makes more sense for me to be able to choose which pieces of content I fetch to not fetch multiple MB of content. Most cases will probably fit between these two.

                          Exploring the tradeoffs in this space (full React-style SPA, HTMX, full SSR) can help you come to a clean solution for your usecase.

                          1. 1

                            I was talking about the additional overhead required to achieve “sending only the necessary data over the network”.

                    3. 4

                      I don’t really get the desire to avoid doing work on the client side.

                      My impression is that it is largely (1) to avoid JavaScript ecosystem and/or* (2) avoid splitting app logic in half/duplicating app logic. Ultimately, your validation needs to exist on the server too because you can’t trust clients. As a rule of thumb, SSR then makes more sense when you have lower interactivity and not much more logic than validation. CSR makes sense when you have high interactivity and substantial app logic beyond validation.

                      But I’m a thoroughly backend guy so take everything that I say with a grain of salt.


                      Edit: added a /or. Thought about making the change right after I posted the comment, but was lazy.

                      1. 8

                        (2) avoid splitting app logic in half/duplicating app logic.

                        This is a really the core issue.

                        For a small team, a SPA increases the amount of work because you have a backend with whatever models and then the frontend has to connect to that backend and redo the models in a way that makes sense to it. GraphQL is an attempt to cut down on how much work this is, but it’s always going to be some amount of work compared to just creating a context dictionary in your controller that you pass to the HTML renderer.

                        However, for a team that is big enough to have separate frontend and backend teams, using a SPA decreases the amount of communication necessary between the frontend and backend teams (especially if using GraphQL), so even though there’s more work overall, it can be done at a higher throughput since there’s less stalling during cross team communication.

                        There’s a problem with MPAs that they end up duplicating logic if something can be done either on the frontend or the backend (say you’ve got some element that can either be loaded upfront or dynamically, and you need templates to cover both scenarios). If the site is mostly static (a “page”) then the duplication cost might be fairly low, but if the page is mostly dynamic (an “app”), the duplication cost can be huge. The next generation of MPAs try to solve the duplication problem by using websockets to send the rendered partials over the wire as HTML, but this has the problem that you have to talk to the server to do anything, and that round trip isn’t free.

                        The next generation of JS frameworks are trying to reduce the amount of duplication necessary to write code that works on either the backend or the frontend, but I’m not sure they’ve cracked the nut yet.

                        1. 4

                          For a small team, a SPA increases the amount of work because you have a backend with whatever models and then the frontend has to connect to that backend and redo the models in a way that makes sense to it

                          Whether this is true depends on whether the web app is a client for your service or the client for your service. The big advantage of the split architecture is that it gives you a UI-agnostic web service where your web app is a single front end for that service.

                          If you never anticipate needing to provide any non-web clients to your service then this abstraction has a cost but little benefit. If you are a small team with short timelines that doesn’t need other clients for the service yet then it is cost now for benefit later, where the cost may end up being larger than the cost of refactoring to add abstractions later once the design is more stable.

                          1. 1

                            If you have an app and a website as a small team, lol, why do you hate yourself?

                            1. 4

                              The second client might not be an app, it may be some other service that is consuming your API.

                        2. 4

                          (2) avoid splitting app logic in half/duplicating app logic.

                          The other thing is to avoid duplicating application state. I’m also a thoroughly a backend guy, but I’m led to understand that the difficulty of maintaining client-side application state was what led to the huge proliferation of SPA frameworks. But maintaining server-side application state is easy, and if you’re doing a pure server-side app, you expose state to the client through hypertext (HATEOAS). What these low-JS frameworks do is let you keep that principle — that the server state is always delivered to the client as hypertext — while providing more interactivity than a traditional server-side app.

                          (I agree that there are use-cases where a more thoroughly client-side implementation is needed, like games or graphics editors, or what have you.)

                          1. 1

                            Well, there’s a difference between controller-level validation and model-level validation. One is about not fucking up by sending invalid data, the other is about not fucking up by receiving invalid data. Both are important.

                          2. 4

                            Spot on.

                            this turns out to be tens (sometimes hundreds!) of requests because the general API is very normalized (yes we were discussing GraphQL at this point)

                            There’s nothing about REST I’ve ever heard of that says that resources have to be represented as separate, highly normalized SQL records, just as GraphQL is not uniquely qualified to stitch together multiple database records into the same JSON objects. GraphQL is great at other things like allowing clients to cherry-pick a single query that returns a lot of data, but even that requires that the resolver be optimized so that it doesn’t have to join or query tables for data that wasn’t requested.

                            The conclusion, which can be summed up as, “Shell art is over,” is an overgeneralized aesthetic statement that doesn’t follow from the premises. Even if the trade-offs between design choices were weighed fully (which they weren’t), a fundamentally flawed implementation of one makes it a straw man argument.

                            1. 1

                              The Twitter app used to lag like hell on my old Thinkpad T450. At the very least, it’d kick my fan into overdrive.

                              1. 1

                                Yay for badly written apps :-p

                                Safari will notice when a page in the background is hogging the CPU, and either throttle or pause it after a while. It puts up a modal dialog on the tab telling you and letting you resume it. Hopefully it sends an email to the developer too (ha!)

                                1. 1

                                  It wound spin up on load, not in the background, because loading all that JS and initializing the page is what would cause the CPU usage. And then after I closed the page, 20 seconds later someone else would send me another Twitter link and I’d get to hear the jet engine again.

                            1. 4

                              Today I learned about the “if myfunc” pitfall. I’m pretty fond of shell programming in general, in a punk/lomography kind of way, but that one scares me.

                              1. 2

                                Yeah it comes up on the bash mailing list with some frequency. I think it’s been true for 30 years or more, and every so often there is a big thread about it. Hence “groundhog day”.

                                Here’s another one (where I participate but so do many other people):

                                https://news.ycombinator.com/item?id=27163494

                                It’s weird shell folk knowledge.

                                I add this to my scripts to flag it when run under OSH:

                                shopt -s strict:all 2>/dev/null || true 
                                

                                https://github.com/oilshell/oil/blob/master/test/spec.sh#L9

                                (Right now Oil’s own scripts keep are run under bash as well.)

                                1. 1

                                  +1 for

                                  punk/lomography

                                  I’ll henceforth use that to explain the essence of shell programming.

                                1. 3

                                  I’ve been digging into DataScript (and Datomic) recently and there’s some very unique ideas that both have for representing data and relationships. I strongly encourage people to read into these tools, they’re very interesting!

                                  I have a mostly incomplete experimentation of a similar datastore using Python and SQLite, primarily as a means of determining if building a Datomic-clone on top of SQL stores (such as PostgreSQL) is feasible.

                                  1. 3

                                    Datomic itself can use eg Postgres as it’s “data service”. It’s delegating transactions and persistence to existing databases.

                                    https://docs.datomic.com/on-prem/overview/storage.html#sql-database

                                    1. 1

                                      Oh, I had no idea! I was under the impression they built their own storage layer.

                                  1. 22

                                    hi, direnv author here :)

                                    1. 7

                                      Don’t have a question but love the tool! Thank you!

                                      1. 3

                                        thanks! :)

                                      2. 3

                                        Hi, thanks for direnv! It’s been life-changing. Thanks to direnv, my virtualenvs are automatically activated/deactivated per-directory, and project config and secrets are all in environment variables and loaded by direnv, making it easy to have parity between dev and production.

                                        Since you’re here, wanted to ask: you think unload hooks can ever be a thing? There’s a longstanding ticket for the feature: https://github.com/direnv/direnv/issues/129

                                        1. 3

                                          Since you’re here, wanted to ask: you think unload hooks can ever be a thing? There’s a longstanding ticket for the feature: https://github.com/direnv/direnv/issues/129

                                          There are too many edge cases. For example, what happens if the user just closes the shell? direnv is just a hook that is executed before every prompt and wouldn’t know about it. Or what happens if two shells are opened on the same project, how does direnv keep track?

                                          Users that ask for this generally want to start/stop service on demand for a project. I think the best for that is to actually open a second shell and start a process manager by hand. Then you get to see what services are running, you can start/stop them, …

                                          1. 1

                                            “not guaranteed to be called in case of closed terminal” are semantics I would welcome. I just want to hook into where it says “direnv: unloading”. My idea was to style my terminal (eg. terminal tab) by directory, and hooking into direnv’s activated inheritance seemed the perfect place to set/unset it. I can do something on directory enter with a plugin but not on directory leave without unload.

                                        2. 1

                                          I freaking love direnv. Thank you so much!

                                        1. 23

                                          Instead, please put the unit in the function name.

                                          (sleep-seconds 300)

                                          1. 5

                                            Too simple. People need to complicate and add yet another bit of complexity such as purpose created types, nammed parameters and whatnot that ultimately needs to be looked up in the documentation.

                                            Funny that the obvious solution isn’t even mentioned because people don’t even consider a design flaw from their all mighty favorite languages.

                                            1. 2

                                              I hate types

                                              1. 5

                                                And yet types exist whether they are declared or not, even in Lisp-y languages! Types are a characteristic of the universe we inhabit. I wouldn’t take my car to a hair salon to have the oil changed, and I know not to do this without actually trying it because the types don’t work out.

                                                1. 3

                                                  Right. I’m working on a new essay on this but in short, I tried to design a dynamically typed lisp with maximal applicability of procedures—as in, “reverse” should be able to reverse a string, a vector, or a list, as opposed to having “string-reverse” or “vector-reverse” defined separately. I found that in order to implement that, I needed an isa-hierarchy of types (or, rather, derive an isa-graph from a tag cluster, kinda duck typing style). For example, strings, vectors, and lists are all “sequences”. So, in that sense, types are great.

                                                  In order to not have to deal with types (as much) on the app level, I really do need to do them carefully (at the primitives level). So types aren’t useless. I still hate them though and languages that make it a point to do everything through types. Dependent types are the worst.

                                                  I don’t wanna type annotate and I don’t wanna limit procedures to only compile on certain types. I don’t want the compiler to prevent you from taking your car to the hair salon.

                                                  1. 3

                                                    APL achieves something very similar to what you are trying to do. Here’s a talk making the point (I think) that APL is so expressive, types would just get in the way.

                                                    Does APL Need a Type System? by Aaron W Hsu https://www.youtube.com/watch?v=z8MVKianh54

                                                    NB: I’m in general very type-friendly (as is the speaker it seems), but that just makes this perspective all the more interesting to me.

                                                    1. 1

                                                      I love APL ♥

                                                    2. 3

                                                      I don’t want the compiler to prevent you from taking your car to the hair salon.

                                                      But why wouldn’t you want that? The hair salon will do nothing useful with the car, and might even mess it up!

                                                      What if I call reverse on an integer? I’d love to find out that my mental model is wrong right after I type it (or, at worst, when I run the compiler) rather than when I run the code and get some inscrutable error in another part of my program because reverse silently did nothing with the value (or, even worse, treated the int as a bit string).

                                                      The fact that some languages have string-reverse and vector-reverse is more a function of the language and less a function of the types. You can easily define a generic reverse function in many languages and still get static types (and, if the language supports it, type inference so you don’t need to write out the types). There are also languages that support things like extension methods so that you can add interfaces to existing types.

                                                      1. 1

                                                        Sometimes I feel like this is, figuratively, an ergonomics issue. Some programmers feel more comfy with the kind of checks and balances you’re talking about and others (like me) hate them. I’m kinda glad that there are both kinds of languages.

                                                        We are veering in to off-topic because the thread is about sleep time units.

                                                        Let’s say I annotate 10 types. 2 of them find bugs, and then there are three or four bugs that weren’t related to types. Then I have done meaningless work 8 times, and put myself in a workflow or state of mind that make those three or four bugs way harder to find, all for the sake of finding two bugs (the kind of bugs that are often obvious on first run anyway). Instead, if I don’t have types, I check everything at the REPL and make sure it gives sane outputs from inputs.

                                                        Like, say I wanna make the procedure

                                                        (define (frobnicate foo)
                                                          (+ foo (* 6 7)))
                                                        

                                                        If I meant to type (* 6 7) but accidentally type (* 6 9), type annotation wouldn’tve caught that. Only testing can.

                                                        But why wouldn’t you want that? The hair salon will do nothing useful with the car, and might even mess it up!

                                                        Maybe they can drive around in the car and give haircuts all over town.

                                                        A lot of my best decisions as a programmer have been me realizing that the same procedure is much more general than I first thought (and then giving it a new, more general name). I like to type as I think, refactor and mold into the perfect parameter signature. (“Oh! I don’t need to pass in the list here, if I only pass the last pair, this function could share code with foo!”)

                                                        What if I call reverse on an integer?

                                                        My language converts it to decimal and reverses the digits. Useful for calculating probabilities for Unknown Armies.

                                                        I’d love to find out that my mental model is wrong right after I type it (or, at worst, when I run the compiler) rather than when I run the code and get some inscrutable error in another part of my program because reverse silently did nothing with the value (or, even worse, treated the int as a bit string).

                                                        So this is why humanity haven’t found the perfect language yet. Some people like different things. I’m not here to stop the type fest that’s been going on. Cool things might be invented from that camp down the line, it’s good that people are trying different things. If type inference could be made better so we don’t have to annotate…

                                                        1. 1

                                                          Let’s say I annotate 10 types. 2 of them find bugs, and then there are three or four bugs that weren’t related to types. Then I have done meaningless work 8 times, and put myself in a workflow or state of mind that make those three or four bugs way harder to find, all for the sake of finding two bugs (the kind of bugs that are often obvious on first run anyway).

                                                          Can you expand on this more? How do types make it harder to find non-type related bugs? In my experience, by completely eliminating an entire class of bugs (that aren’t always obvious catch-on-the-first run bugs, especially if you have a really nice type system!) it gets easier, not harder, to identify logic errors.

                                                          1. 3

                                                            As an analogy, overly relying on spell checkers can make some people miss things that are still legit spellings but are the wrong words in that particular sentence, like effect/affect.

                                                            But, it’s worse than that since (and I’m complaining about type annotation, not type inference) you need to enter the type info anyway. It’s “bugfinding through redundancy”. Sort of the same philosophy as “write a test for every line of code” but more rigid and limited. Of course reduntantly writing out what you want the function to accept and return is going to catch some bugs.

                                                            If you like this sort of type checking, you’re not alone. A lot of people love them, and ultimately there’s no need to argue. Time will tell if that style of programming does lead to overall fewer bugs, or at least does so for programmers of a certain taste, and if so, that’s fine by me. I’m not gonna take your ML away.

                                                            But as my “42 is odd” example shows, I’m not too happy with the whole “statically typed programs are Provably Correct” hype leading into the same hopeless dead end as Russell and Whitehead did a hundred years earlier.

                                                            Coming from C and Pascal, when I first discovered languages that didn’t have type annotations in the nineties (like awk, perl, and scheme) I felt as if I had found the holy grail of programming. No longer would I have to write out boring and obvious boilerplate. It was a breath of fresh air. Other people obviously feel differently and that’s fine.

                                                            For me, it seems that a lot (not all, but some) of the things a good type annotation system helps you with are things you don’t even need to do with dynamically typed languages. It also feels like with a type annotated language, there’s a catch-22 problem leading you to have to basically write the function before you write it (maybe with pseudocode or flowcharts) just so you can know what type signatures to use.

                                                            I felt that wow, a cons pair of car and cdr can express data in so many ways, I can just immediately write the actual logic of my code. Whereas when I worked as a Java dev (don’t worry, I’ve looked at modern type languages too, like Haskell) we had to slog through writing types (classes and instances), UML diagrams, wall full of post-its, architecture, ConnectionKeepingManagerFrobnicator.new() etc. With Scheme it was just, all that red tape just fell away. No need for pseudocode since I could just send whatever I was thinking into the REPL.

                                                            The type community loves the expression “to reason about the code”. Well, to me it’s a heck of a lot easier to reason about the code when it’s a fifth the size. (Sexps help too since it’s easier for me to grok a code tree than a linear sequence of unparsed tokens of code data.)

                                                            Obviously, type fans have had similar epiphanies but in the other direction, falling in love with static just like I fell in love with dynamic. And that’s cool. Let the deserts bloom. Humanity can’t be betting all of its eggs on my approach. I see the type craze as an experiment. One that might even be right. So please, go ahead.

                                                            I’m just really, really grateful that it’s not me who have to slog through it. I can sit under the cork tree sniffing dynamically typed flowers.

                                                            1. 2

                                                              Uh, wait, why did I get baited into writing all that when I see now that I already answered it in what you snipped out:

                                                              Instead, if I don’t have types, I check everything at the REPL and make sure it gives sane outputs from inputs.

                                                              Like, say I wanna make the procedure

                                                              (define (frobnicate foo) (+ foo (* 6 7)))

                                                              If I meant to type (* 6 7) but accidentally type (* 6 9), type annotation wouldn’tve caught that. Only testing can.

                                                              1. 1

                                                                hm, that doesn’t answer my question at all but it your longer post did, so thanks.

                                                                I think the point about “boilerplate” is pretty tired and not even true any more with how good type inference is nowadays. Yes, Java involved/involves a lot of typing. No, it’s no longer the state of they art.

                                                                It’s true that in the case where you use the wrong term that has the same type as the correct term, the typechecker will not catch this. Not having types is also not going to catch this. I’m going to see the error at the same exact time with both approaches. Having a REPL is orthogonal to having types, so I also often check my Haskell functions at the REPL.

                                                                I see the type craze as an experiment.

                                                                Calling an entire field of active research a craze is a little upsetting.

                                                                1. 1

                                                                  I am complaining about annotated type systems specifically, which I clarified nine times. Inference type systems are fine.

                                                                  Not having types is also not going to catch this.

                                                                  The idea is that checking at the REPL will find it.

                                                                  I’m going to see the error at the same exact time with both approaches. Having a REPL is orthogonal to having types, so I also often check my Haskell functions at the REPL.

                                                                  Me too. Which made me feel like the whole type thing was a little unnecessary since I needed to do just as much checking anyway.

                                                                  (As noted elsewhere in this thread, I’ve changed my tune on types a little bit since I realized I do need an isa-taxonomy for primitives. I.o.w. to get rid of types, I’m gonna have to define types.)

                                                                  Calling an entire field of active research a craze is a little upsetting.

                                                                  It’s more the whole dependent type / provably correct thing that’s a li’l bit of a craze, not the entire field of type research as a whole. As I wrote in my essay, types have a lot of benefits including serious performance gains, and that’s awesome. It’s the whole “it fixes bugs!” that gets a li’l cargo culted and overblown sometimes. Not by you, who do understand the limits of type systems, but by hangers-on and newly converted acolytes of typing.

                                                          2. 1

                                                            Lots of points from your arguments can be achieved by using generic types, and everything would work safely, giving the programmer quick feedback if the types work for the particular combination. No need to guess and check in the runtime.

                                                            My language converts it to decimal and reverses the digits. Useful for calculating probabilities for Unknown Armies.

                                                            So what would be the output of 2.reverse() * 2?

                                                            1. 2

                                                              Four.

                                                              1. 1

                                                                I’m wondering if that would also be the case with "2".reverse() * 2. Because if the output would be 4, then I’d wonder what would be the output of "*".reverse() * 2 would be. I hope it wouldn’t be **.

                                                                No matter what the answers are, I’ve already dedicated a lot of time to decode how some basic operations work. With types, I would have this information immediately, often without needing to dig through the docs.

                                                                1. 1

                                                                  4 and ** respectively.

                                                                  1. 2

                                                                    All kidding aside, the idea isn’t to overload and convert but to have a consistent interface (for example, reversing, or walking, or concatenating strings and lists similarly) and code reuse. I’m not insisting on shoehorning sequence operations to work on non-sequence primitives. Which is why I already said I needed an isa taxonomy of primitive types.

                                                  2. 1

                                                    So sleep(seconds: 1) needs to be looked up in documentation whereas sleep-seconds(1) does not?

                                                    1. 2

                                                      If you language only supports the second, then use the second which is perfectly clear. You would by no means be ina. Situation where lack of language features limit code clarity.

                                                      Notice that while parameters have names in most languages, in many of them you can’t include the name on your code but rather need to pass them in order.

                                                  3. 2

                                                    this way makes the most sense to me, at least for sleep

                                                    1. 1

                                                      It used to be a common sense that sleep uses seconds, until other languages not following that.

                                                      1. 5

                                                        That’s not how common sense works!

                                                    1. 10

                                                      I’m learning Nix lately, and I don’t (yet?) hate the configuration syntax. But I do find the user experience to be on the unfamiliar side, particularly when it comes to terminology. For instance, derivations are Nix’s packages, and everyone calls them packages, but the software doesn’t, so new users can get hung up on that. Better to cut down the learning curve by calling them packages, then teach that Nix packages are different in such and such way.

                                                      Regarding the configuration language, I wonder if there have been any attempts to translate code in another, more palatable language to Nix? I’m not suggesting a new language like TypeScript. I mean a library in a common language for emitting Nix configuration, the way imperative CDK code yields the CloudFormation YAML that’s so clumsy to write by hand.

                                                      1. 9

                                                        If you might like parens more than an MLish DSL (or if you want a full language and not a DSL) come check out Guix :)

                                                          1. 6

                                                            If I had a Nickel for every new config language I came across…

                                                          2. 6

                                                            For instance, derivations are Nix’s packages, and everyone calls them packages, but the software doesn’t, so new users can get hung up on that.

                                                            There’s a subtle difference, in that what people usually call “packages” (see e.g. nixpkgs) are higher level expression which get translated to derivations (by nix-instantiate). So the word package is actually quite ambiguous.

                                                            1. 5

                                                              I think generating Nix is a bad idea, because Nix already generates shell code, and invokes many subprocesses. So then you’re going to have a leaky abstraction on top of a leaky abstraction.

                                                              It would be better just to build something directly on top of shell, which is fundamental since all packages themselves use it .

                                                              And all distros use it, even Nix does. Nix somewhat covers it up, but this long debate shows how central shell is to Nix:

                                                              https://github.com/NixOS/rfcs/pull/99#

                                                              From my understanding every package build basically depends on this, and then in the Nix package defs you’ll also see templated shell to fix up specific issues, just like any other package manager:

                                                              https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh

                                                              1. 6

                                                                So then you’re going to have a leaky abstraction on top of a leaky abstraction.

                                                                I still have scars from this. Nix has some cool functionality that basically reimplements Rust’s Cargo in Nix + shell scripts (buildRustCrate). Since Cargo actually supports features with spaces in them (which I hope no crate will ever use), I once worked on a patch that added support for features with spaces to buildRustCrate. IIIRC I eventually got it working, but it was a terrible quotation hell.

                                                                1. 11

                                                                  Yup exactly, this is the “rewriting upstream” problem, which I mentioned with Bazel here:

                                                                  http://www.oilshell.org/blog/2021/04/build-ci-comments.html#two-problems-with-bazel-and-gg-again

                                                                  Both Bazel and Nix have strong models of the world and assume it’s more homogeneous than it actually is. On the other hand, shell embraces heterogeneity and you can always find a solution without rewriting, and containers add to that flexibility.

                                                                  They should enable correct parallelism, incremental builds, and distribution. (It looks like https://earthly.dev/ is pointing in this direction, though I’ve only read about it and not used it.)

                                                                  Just like Nix prefers rewriting in its expression language; Bazel prefers rewriting with Starklark. You generally throw out all the autoconf and write Starlark instead, which is a lot of work.

                                                                  Many years I reviewed the R build rules for Bazel which my coworker wrote, which is analogous to the Rust problem in Nix (thanks for the example).

                                                                  There is some argument that rewriting is “good”, but simply from spinning our wheels on that work, I no longer have any appetite for it. I’ll concede that upstream doesn’t have all the nice properties of Nix or Bazel, but I also want to spend less time messing with build systems, and I don’t want to clear this hurdle whenever trying out ANY new language (Zig, etc.) I think language-specific package managers are something of a anti-pattern, but that’s the world we live in.

                                                                  We need a “meta-build” system that solves the reproducibility/parallelism/distribution/incrementality problem for everything at once without requiring O(N) rewrites of upstream build systems. I don’t really have any doubt that this will be based on containers.


                                                                  Nix and Bazel have something else in common which I mentioned here – they have a static notion of dependencies that has to be evaluated up front, which can be slow.

                                                                  https://lobste.rs/s/ypwgwp/tvix_we_are_rewriting_nix#c_9wnxyf

                                                              2. 3

                                                                The solution some friends and I were kicking around was similar to this. At this point, we’ve got a relatively strong sense of how we like to build our servers and appliances, and we figured it’d be nice just to focus in on the parts we care about (mainly, the ease of specifying packages and users and keys and whatnot) in a JSON blob and then have a script barf out the relevant Nix files and flakes.

                                                                Sure, we give up like 90% of the bizarre shit Nix gives you, but I honestly don’t need most of that.

                                                              1. 2

                                                                By the author of Chicken Scheme, right? Looks really interesting. Anyone here have any example code written in it I can look at? Curious to see some real code.

                                                                1. 4

                                                                  Some libraries and example programs can be found here: https://gitlab.com/b2495/flenglibs, but note that this is all relatively new and has undergone very little testing.

                                                                  1. 2

                                                                    I was confused looking at the repo full of Haskell files until I realized .ghc isn’t the Haskell file extension and these files contain FLENG code. Just curious, why .ghc instead of .fl(eng)?

                                                                    1. 3

                                                                      So the manual says:

                                                                      If the source file name has an .fl or .fleng extension, then the “fleng” driver assumes it contains FLENG code. In all other cases the file is considered to be an FGHC or Strand source file.

                                                                      and:

                                                                      Additionally a front end for “Flat Guarded Horn Clauses” (FGHC)[2] and “Strand”[5] is provided, which are slightly higher level and more convenient languages.

                                                                      I haven’t taken a deeper look, but I assume FLENG is too minimal for general programming.

                                                                      1. 1

                                                                        Huh, ok. Thanks.

                                                                    2. 1

                                                                      Interesting. It looks quite a bit like simple Haskell code, i.e. pure functional. Is that a reasonable intuition? I’ve never really understood how logic programming can be used for normal application development.

                                                                      1. 3

                                                                        Indeed, logic programming is purely functional (if you disregard assertions to a database in Prolog or non-standard extensions like global variables). Concurrent logic languages usually have a somewhat simpler control flow than Prolog, though - most dialects have no backtracking, e.g. are what is called “committed choice” languages.

                                                                        But there is no reason why logic languages should be less capable of application development than other functional languages, it just depends on the libraries. The only obstacle is IMHO a sparse implementation landscape, lack of easy to use “glue” to use foreign code effectively and the effort needed to learn a paradigm that differs strongly from mainstream languages.

                                                                        1. 1

                                                                          Wouldn’t Prolog et al more commonly (and correctly) be described as relational, rather than functional? (Because a predicate can map an element in its domain to many elements in the co-domain, hence generalizing functions.) In any case, the point about purity is the important one and I think the great similarities between functional and logic programming languages are not often appreciated, maybe because of the differences in syntax or the fact that most logic programming languages are dynamically typed (in contrast to most pure functional languages).

                                                                  1. 5

                                                                    Very much enjoyed the format of this post - lots of detail but with the green and red boxes as guides.

                                                                    1. 2

                                                                      This looks great! I’m all for database-centric architectures. am a big fan of Materialize (right in the process of introducing it in a project I’m working in) and followed Eve closely when it was still alive. (I created the little bouncing ball example that became a benchmark of sorts towards the end.) Will definitely try to participate in this!

                                                                      1. 2

                                                                        What are Materialize and Eve? I’m puzzled by “little bouncing ball” and relationship with databases — that sounds pretty interesting!

                                                                        1. 4

                                                                          Materialize is a streaming database based on differential dataflow, from Frank McSherry and team. McSherry is well-known for his “Cost that outperforms single-threaded” (COST) paper [PDF].

                                                                          1. 3

                                                                            This is one of my favorite papers from Frank McSherry. The other being “A COOL AND PRACTICAL ALTERNATIVE TO TRADITIONAL HASH TABLES” which is a great intro to Cuckoo hashing, and a great analysis: https://www.ru.is/faculty/ulfar/CuckooHash.pdf

                                                                        1. 3

                                                                          I’ve only read the ToC but already love this post - so much value in spreading knowledge about all the capabilities of Postgres. I guess many people think of these as “advanced features”, but that’s just because the expectations of what a RDBMS is have been narrowed down for years.

                                                                          1. 1

                                                                            This is a fantastic resource! While the rough outline of the development of formal logic over the last 200 years (Leibniz is not always remembered) is known among people who have a reason to care, this overview comes from a programmer’s perspective, which is unusual. It also goes straight to the heart of exactly how and why logic, abstract maths (eg category theory), and computing fit together, a topic which is often alluded to but which I feel is usually shrouded in mystery (which is the first point the author makes). This is something I wish I had read ten years ago, but very happy to be reading it now.

                                                                            1. 2
                                                                              1. 1

                                                                                This is a fantastic presentation! I’ve been reading the first part over the weekend. Would you post it as a separate story? If you do, please link from here so I don’t forget to upvote. (And if for some reason you don’t want to, I’ll post it.)

                                                                                  1. 1

                                                                                    Great!

                                                                              1. 4

                                                                                Love to see it. I use Elixir everyday at work, and am a big fan of static types, so in principle I’m the target audience. However, I’m reluctant to give up the Elixir ecosystem I love do much. E.g. I’m really excited LiveView, Livebook, Nx, etc.

                                                                                What are the benefits / necessity of a whole new language as opposed to an improved dialyzer, or maybe a TypeScript style Elixir superset with inline type annotations?

                                                                                1. 11

                                                                                  One issue is that existing Elixir code will be hard to adapt to a sound type system, more specifically in how pattern matching is used. For example, consider this common idiom:

                                                                                  {:ok, any} = my_function()
                                                                                  

                                                                                  (where my_function may return {:ok, any} or {:error, error} depending on whether the function succeeded)

                                                                                  Implicitly, this means “crash, via a badmatch error, if we didn’t get the expected result”. However this is basically incompatible with a sound type system as the left-hand side of the assignment has the type {:ok, T} and the function has the return type {:ok, T} | {:error, Error}.

                                                                                  Of course we could add some kind of annotation that says “I mismatched the types on purpose”, but then we’d have to sprinkle these all over existing code.

                                                                                  This is also the reason why Dialyzer is based on success typing rather than more “traditional” type checking. A consequence of this is that Dialyzer, by design, doesn’t catch all potential type errors; as long as one code path can be shown to be successful Dialyzer is happy, which reflects how Erlang / Elixir code is written.

                                                                                  1. 5

                                                                                    Of course we could add some kind of annotation that says “I mismatched the types on purpose”, but then we’d have to sprinkle these all over existing code.

                                                                                    This is what Gleam does. Pattern matching is to be total unless the assert keyword is used instead of let.

                                                                                    assert Ok(result) = do_something()
                                                                                    

                                                                                    It’s considered best practice to use assert only in tests and in prototypes

                                                                                    1. 1

                                                                                      What do you do when the use case is “no, really, I don’t care, have the supervisor retry because I can’t be bothered to handle the error and selectively reconcile all of this state I’ve built up, I’d rather just refetch it”?

                                                                                      1. 1

                                                                                        Maybe we need another keyword:

                                                                                        assume Ok(result) = do_something()
                                                                                        
                                                                                        1. 1

                                                                                          That is what assert is for.

                                                                                        2. 1

                                                                                          That’s what assert is. If the pattern doesn’t match then it crashes the process.

                                                                                          1. 1

                                                                                            So why not use it in production?

                                                                                            1. 2

                                                                                              You can for sure. I was a bit too simple there. I should have said “It is best to only use assert with expected non-exceptional errors in prototypes`. There’s place for Erlang style supervision in Gleam in production.

                                                                                      2. 3

                                                                                        Great, succinct explanation!

                                                                                        1. 2

                                                                                          This is an interesting example. I’m still not sure I understand how it’s “basically incompatible”, though. Is it not possible to annotate the function with the possibility that it raises the MatchError? It feels kind of like Java’s unchecked exceptions a bit. Java doesn’t have the greatest type system, but it has a type system. I would think you could kind of have a type system here that works with Elixir’s semantics by bubbling certain kinds of errors.

                                                                                          Are you assuming Hindley Milner type inference or something? Like, what if the system were rust-style and required type specifications at the function level. This is how Elixir developers tend to operate already, anyway, with dialyzer.

                                                                                          1. 1

                                                                                            I don’t see how that’s a problem offhand. I’m not sure how gleam does it, but you can show that the pattern accommodates a subtype of the union and fail when it doesn’t match the :ok.

                                                                                            1. 4

                                                                                              The problem is the distinction between failing (type checking) and crashing (at runtime). The erlang pattern described here is designed to crash if it encounters an error, which would require that type checking passes. But type checking would never pass since my_function() has other return cases and the pattern match is (intentionally) not exhaustive.

                                                                                              1. 1

                                                                                                Ah, exhaustive pattern matching makes more sense. But also feels a little odd in Erlang. I’ll have to play with Gleam some and get an understanding of how it works out.

                                                                                          2. 4

                                                                                            One thing is that TypeScript is currently bursting at the seams as developers aspirationally use it as a pure functional statically-typed dependently-typed language. The TypeScript developers are bound by their promise not to change JavaScript semantics, even in seemingly minor ways (and I understand why this is so), but it really holds back TS from becoming what many users hope for it to be. There’s clearly demand for something more, and eventually a language like PureScript / Grain / etc will carve out a sizable niche.

                                                                                            So, I think starting over from scratch with a new language can be advantageous, as long as you have sufficient interoperability with the existing ecosystem.

                                                                                            1. 2

                                                                                              I won’t go too much into Dialyzer as I’ve never found it reliable or fast enough to be useful in development, so I don’t think I’m in a great place to make comparisons. For me a type system is a writing assistant tool first and foremost, so developer UX is the name of the game.

                                                                                              I think the TypeScript question is a really good one! There’s a few aspects to this.

                                                                                              Gradual typing (TypeScript style) offers different guarentees to the HM typing of Gleam. Gleam’s type system is sound by default, while with gradual typing you opt-in to safety by providing annotations which the checker can then verify. In practice this ends up being quite a different developer experience, the gradual typer requires more programmer input and the will to resist temptation not to leave sections of the codebase untyped. The benefit here is that it is easier to apply gradual types to an already existing codebase, but that’s not any advantage to me- I want the fresh developer experience that is more to my tastes and easier for me to work with.

                                                                                              Another aspect is just that it’s incredibly hard to do gradual typing well. TypeScript is a marvel, but I can think of many similar projects that have failed. In the BEAM world alone I can think of 4 attempts to add a type checker to the existing Elixir or Erlang languages, and all have failed. Two of these projects were from Facebook and from the Elixir core team, so it’s not like they were short on expertise either.

                                                                                              Lastly, a new language is an oppotunity to try and improve on Elixir and Erlang. There’s lots of little things in Gleam that I personally am very fond of which are not possible in them.

                                                                                              One silly small example is that we don’t need a special .() to call an anonymous function like Elixir does.

                                                                                              let f = fn() { 1 }
                                                                                              f()
                                                                                              

                                                                                              And we can pipe into any position

                                                                                              1
                                                                                              |> first_position(2)
                                                                                              |> second_position(1, _)
                                                                                              |> curried_last_position
                                                                                              

                                                                                              And we have type safe labelled arguments, without any runtime cost. No keyword lists here

                                                                                              replace(each: ",", with: " ", in: "A,B,C")
                                                                                              

                                                                                              Thanks for the questions

                                                                                              edit: Oh! And RE the existing ecosystem, you can use Gleam and Elixir or Erlang together! That’s certainly something Gleam has been built around.

                                                                                              1. 1

                                                                                                Two of these projects were from Facebook and from the Elixir core team, so it’s not like they were short on expertise either.

                                                                                                Oh, wow, I don’t think I’ve heard of these! Do you have any more info? And why was Facebook writing a typechecker for Elixir? Are you talking about Flow?

                                                                                                1. 1

                                                                                                  Facebook were writing a type checker for Erlang for use in WhatsApp. It’s not flow, but it is inspired by it. I’m afraid I don’t think much info is public about it.

                                                                                            1. 8

                                                                                              The title is really click-bait. Most points are “This sucks, but there is a workaround/solution for it”.

                                                                                              It sounds that the main point of the Author is “but for beginners this can lead to a lot of pain”. But a lot of topics are not beginner-friendly, like replication and performance optimisation.

                                                                                              1. 12

                                                                                                Most points are “This sucks, but there is a workaround/solution for it”.

                                                                                                Experts talking about the flaws in their tools is important.

                                                                                                1. 6

                                                                                                  In fact the ability to readily list a bunch of problems with a thing is a massive tell that someone is a real expert in that thing.

                                                                                                  1. 1

                                                                                                    In interviews, I sometimes ask candidates to name a few pros of waterfall methodology and cons of agile. If you can’t name pros and cons of both, you don’t really know either.

                                                                                                2. 7

                                                                                                  I don’t agree. The Postgres replication story just isn’t there. The “workarounds” described boil down to “implement replication yourself for your use case.”

                                                                                                  Whatever else you may say about MySQL synchronous replication, it does work. And it’s one of the main reasons large deployments like Google and Facebook used MySQL over Postgres. Vitess relies on MySQL row level replication, for example.

                                                                                                  Postgres is better than MySQL in almost every way—XID wrap around and query hints notwithstanding—but synchronous replication is a hard requirement for lots of use cases.

                                                                                                1. 10

                                                                                                  “Scrum is the opposite of Agile”

                                                                                                  I can’t disagree with that. Instead of the word “agile”, many times I use the phrase “dynamic process tailoring”, because that really captures how it works more than anything else. There is no standard for agile. It’s a marketing term. It labels the place in the bookstore you go to find current cool dev practices. Not having a standard is not a bad thing, actually, because there’s no one way to do everything.

                                                                                                  [paraphrasing] “I’ve been in projects where I could see where the design was headed in three months, and instead of taking a week or two and coding that, the team spent three, six months and still never got completely there”

                                                                                                  This is one of the reasons I’m spending so much time on “how to talk about the solution” Good teams have good conversations and Scrum “goes away”. The code “goes away”. Instead they just solve the problem quickly and move on to the next project. Bad or average teams get hung up on the details, whether in the tooling or in the process they’re using, and the more they get hung up the longer everything is going to take.

                                                                                                  While true, my statement above is not descriptive enough for people to grok it who’ve never seen it work in practice. More work needs to be done.

                                                                                                  1. 3

                                                                                                    I’m intrigued but you’re right that “just solve the problem and move on” sounds to me more like a description of what is wrong with many software teams. Getting hung up is of course not good, but constantly keeping one eye on possible improvements in tooling and process is in my experience necessary in order to not get stuck with diminishing returns. Sure, balance is needed.

                                                                                                    1. 2

                                                                                                      Yup, for such a simple thing there’s a lot of depth here that I don’t think the industry has adequately described, either to current practitioners or new folks entering the field. I know some of the advice I got really early on led me in the wrong direction. So many things in tech seem impenetrable/silly, until one day you get it, then it’s so trivial you don’t bother (or can’t) explain it to the next guy. The real meaning of OO was like that for me, or the beauty of well-designed databases. This is another one of those things.

                                                                                                      You have to stay on top of the tooling infrastructure changes or you’d still be coding with stone knives and bearskins (Star Trek joke). While the result is perhaps one we can all agree on, getting there is something else entirely. I wrote a book on this, took a chapter and wrote a blog on it the other day, and plan on following up with another book on infrastructure/architecture. Shameless link: https://danielbmarkham.com/for-the-love-of-all-thats-holy-use-ccl-to-control-complexity-in-your-systems/

                                                                                                  1. 1

                                                                                                    Some really neat ideas, but also, a lot of headshaking around things like the XLA/Tensorflow dependency. The serious numerical story in Elixir doesn’t exist because core doesn’t care and, frankly, most Elixir folks are basically just doing web stuff as Ruby + Rails refugees.

                                                                                                    Nice agitprop though if you want to boost your language’s popularity by riding the AI/ML gravy train.

                                                                                                    1. 17

                                                                                                      The serious numerical story in Elixir doesn’t exist because core doesn’t care and, frankly, most Elixir folks are basically just doing web stuff

                                                                                                      Same applied to Python way back. That the Elixir devs are attempting to broaden their ecosystem is good.

                                                                                                      1. 6

                                                                                                        BEAM would not be my first choice for anything where I had to do some serious number crunching. I wonder who will even use this, and if it’ll die out like the Swift extensions for it.

                                                                                                        I use Elixir because it’s good at networking, concurrency, and parsing, not anything AI.

                                                                                                        1. 3

                                                                                                          I had the same initial reaction and would rather bet on Julia for the future of ML. But at the very least, breaking Python’s monopoly is good and, of course, there is no good reason for Python’s ascent in this field. So, there’s no reason other ecosystems that have certain advantages over Python shouldn’t try to make a dent here.

                                                                                                          A lot of ML projects use Ray (ray.io) for orchestration and workflows (or so I hear), basically implementing the actor model, and Elixir/BEAM is probably a more natural fit for that side of the problem.

                                                                                                          1. 3

                                                                                                            On the other hand, not everybody needs serious number crunching. This could let people in the elixir ecosystem stay within it for longer.

                                                                                                            1. 1

                                                                                                              Seems like they are using multistage programming to make this rather fast - I’m guessing this stuff wouldn’t be running on the BEAM directly? Could be wrong though.

                                                                                                            2. 5

                                                                                                              The serious numerical story in Elixir doesn’t exist because core doesn’t care

                                                                                                              Ehhhh. Clearly you don’t mean “Elixir core”, as Jose is the one writing this. Even if you mean the Erlang core team, they might not care enough to lead implementation but they’re open to additions that support it: https://github.com/erlang/otp/pull/2890

                                                                                                              Anyway, BEAM isn’t the logical execution environment for this stuff, it needs GPU to fly. For this use case, BEAM is a great place to write a compiler, and will function adequately as a test environment.

                                                                                                              1. 1

                                                                                                                ¯\_(ツ)_/¯

                                                                                                              2. 5

                                                                                                                FWIW, the compiler backend of Nx is pluggable, so other compilers aside from XLA can also be integrated with. XLA happened to be the first one they’ve gone with thus far.

                                                                                                                What do you mean with “the core doesn’t care”?

                                                                                                                1. 5

                                                                                                                  Paraphrasing a colleague’s gripes…Elixir has a bunch of math functionality inherited from Erlang, and hasn’t bothered to fix any of the problems around it.

                                                                                                                  In erlang:

                                                                                                                  2> math:exp(344).
                                                                                                                  2.4963287283217065e149
                                                                                                                  3> math:exp(3444).
                                                                                                                  ** exception error: an error occurred when evaluating an arithmetic expression
                                                                                                                       in function  math:exp/1
                                                                                                                          called as math:exp(3444)
                                                                                                                  

                                                                                                                  In Elixir:

                                                                                                                  iex(3)> :math.exp(344) 
                                                                                                                  2.4963287283217065e149
                                                                                                                  iex(3)> :math.exp(3444)  
                                                                                                                  ** (ArithmeticError) bad argument in arithmetic expression
                                                                                                                      (stdlib 3.13.2) :math.exp(3444)
                                                                                                                  

                                                                                                                  Contrast with JS:

                                                                                                                  > Math.exp(344)
                                                                                                                  2.4963287283217065e+149
                                                                                                                  > Math.exp(3444)
                                                                                                                  Infinity
                                                                                                                  

                                                                                                                  What’s going on here is that Erlang uses doubles internally (except for integers, but that’s a different kettle of fish), but does not do IEEE754 support for special values like NaN or Infinite. This is certainly a choice they could make, but it rears its ugly head when you implement something mathematically well-behaved like the sigmoid function:

                                                                                                                  iex(5)> sigmoid = fn (x) -> 1.0 / (1.0 + :math.exp(-x)) end
                                                                                                                  #Function<44.97283095/1 in :erl_eval.expr/5>
                                                                                                                  iex(6)> sigmoid.(-1000)                                    
                                                                                                                  ** (ArithmeticError) bad argument in arithmetic expression
                                                                                                                      (stdlib 3.13.2) :math.exp(1000)
                                                                                                                  iex(6)> sigmoid.(-100) 
                                                                                                                  3.7200759760208356e-44
                                                                                                                  

                                                                                                                  In JS, we’d get a negative in the denominator, it’d go to Inf, and the function would work fine. In Elixir, it explodes.

                                                                                                                  Core has had the opportunity to fix this for a long time, but has been spending cycles elsewhere.

                                                                                                                  1. 5

                                                                                                                    Elixir has a bunch of math functionality inherited from Erlang, and hasn’t bothered to fix any of the problems around it.

                                                                                                                    I found no tickets in the Erlang (where BEAM issues belong) or Elixir bug trackers discussing infinity.

                                                                                                                    1. 2

                                                                                                                      For all the supposed number crunching going on, nobody’s requesting working IEEE doubles?

                                                                                                                      1. 2

                                                                                                                        Not really, as most of “interesting” part of IEEE is already there. What is not supported are qNaN (sNaN is supported as it’s behaviour is exactly the same as what Erlang does - just throw exception) and infinites. All other values are fully supported in Erlang. And while it can be irritating to have exception in infinity, I think that in most situations you do not mind, as that would mean that the result of computation would be probably garbage to you anyway. So throwing up early is probably what will provide clearer information, especially with Erlang error isolation approach.

                                                                                                                2. 2

                                                                                                                  The serious numerical story in Elixir doesn’t exist because core doesn’t care and, frankly, most Elixir folks are basically just doing web stuff

                                                                                                                  That perfectly describes how I got into Elixir. Phoenix got me into the ecosystem, but I branched out pretty quickly once I understood the power of the OTP model. About two years ago I started looking into writing drone flight control software in Elixir and ran into a huge hurdle with the numerical processing portion of it; I was fiddling around with NIFs and things, but it felt like I was just compromising the robustness of the system by having it call out to my own C code.

                                                                                                                  The Nx project has me very very excited that the project I put on the shelf might be viable now!

                                                                                                                1. 4

                                                                                                                  I’m happy the tech stack is working for the author, but in my experience that tech stack has way too many components for my one-man projects. Especially the devops pieces. I’ve given up on using AWS in my own projects, even though I’m experienced with AWS and write CloudFormation configs regularly, it’s just not worth it. So much easier to just deploy to a vanilla Linux server, using a Makefile for automation. I stopped using Docker for similar reasons. I do like GitHub Actions.

                                                                                                                  To each his own. My hypothesis is that the complexity of a lot of tech stacks today hurts more than helps.

                                                                                                                  I’ll complain about Django some other time.

                                                                                                                  1. 1

                                                                                                                    How many pieces would typically go into one of these deployments? Db, web processes, message bus…? What do you use for monitoring and crash recovery and such? (Just curious :) )

                                                                                                                  1. 2

                                                                                                                    Very interesting! Congratulations on getting it this far.

                                                                                                                    I think there is a clear case for light-weight pre-processor-style SQL wrappers, a bit like Sass for CSS or various templating languages for HTML. This is clearly more ambitious - if I were to use Preql, I would likely output SQL files for manual inspection and then install that code as views or functions in the db itself.

                                                                                                                    since the functions (and the rest of the code) are part of the client, and not the server, you can use version control, such as git, to keep history, code review, accept pull-requests, and so on.

                                                                                                                    I think it’s very important to keep SQL code under version control, and I know the (perceived) difficulty of doing so is a common argument against keeping logic in the database - but there is a rather simple and elegant solution, namely that pioneered by subZero/PostgREST: https://docs.subzero.cloud/managing-migrations/

                                                                                                                    I’m very curious about what the description “relational programming language” means to you. That term is often taken to mean languages in the Prolog tradition (ie logic programming). SQL and relational algebra/calculus is similar, but more restricted. IMO a common failure of many attempts to wrap or improve on SQL is that they discourage “relational thinking”, ie in terms of sets and relations between sets. Any thoughts on this and how Preql fits into the picture?

                                                                                                                    1. 1

                                                                                                                      if I were to use Preql, I would likely output SQL files for manual inspection and then install that code as views or functions in the db itself.

                                                                                                                      I understand your reluctance to trust the magic. When I was using C in the early days, I always wanted to see exactly what Assembly code it produced, to make sure that it’s correct and efficient. But as the compilers got better, and I got busier, I stopped that altogether.

                                                                                                                      I’m very curious about what the description “relational programming language” means to you

                                                                                                                      I think the magic of SQL, which makes many people love it despite all of its downsides, is its heavy reliance relational algebra, i.e. working with sets and using joins. I took care to put this kind of algebra front and center, so there is a special operator for selection (i.e. where), projection (i.e. select), for set operations (&, |), and so on. I also put a lot of effort to make joins convenient, and I plan to keep improving them.

                                                                                                                      … Prolog tradition (ie logic programming). SQL and relational algebra/calculus is similar, but more restricted.

                                                                                                                      Do you think it’s technically possible to implement real-world logic programming using SQL? Or is it too restricted for it? (of course, anything can be done with a Turing machine, but I mean natively in the language)

                                                                                                                      1. 1

                                                                                                                        I understand your reluctance to trust the magic.

                                                                                                                        Partly that, but mostly because I prefer to store logic in the db (as I hinted above).

                                                                                                                        I think the magic of SQL, which makes many people love it despite all of its downsides, is its heavy reliance relational algebra,

                                                                                                                        Agree!

                                                                                                                        I took care to put this kind of algebra front and center, so there is a special operator for selection

                                                                                                                        Would you say then the resulting language is mostly a sugared SQL, or a separate language in the relational model tradition, which “compiles down” to SQL? I think both are interesting and worthwhile! But it seems to me that all the quirks of SQL makes the latter approach pretty hard, if we want to make full use of SQL-as-it-exists. And I understand that’s why you provided an escape hatch to write arbitrary SQL - which seems useful but makes me wonder again if we can actually go beyond “more succinct SQL” as long as we’re working with SQL databases.

                                                                                                                        Do you think it’s technically possible to implement real-world logic programming using SQL?

                                                                                                                        With recursive CTEs you can at least theoretically see how it could be done, but a) I don’t think it could be made efficient, b) Prolog is not purely declarative, the procedural interpretation is integral to the language, and that would be very hard to model. Datalog on the other hand is precisely a logic language intended for database programming. I have been looking at the Datalog Educational System as a practical way to compare SQL, relational algebra, relational calculus and Datalog. The pyDatalog project also claims to be able to run Datalog queries on SQL databases, but I haven’t tried it or looked closely at it.

                                                                                                                        There have also been some database languages based on set/list comprehensions as they are known from Haskell and Python (and before that Miranda etc). The most successful one seems to have been CPL (Collection Programming Language; here’s a paper that describes it). I think set building notation might be the most intuitive entry into all these paradigms, and SQL, Prolog etc could be described in relation to this foundation. Just something I’ve been thinking about lately.

                                                                                                                        1. 1

                                                                                                                          Would you say then the resulting language is mostly a sugared SQL,

                                                                                                                          I guess that depends, would you say that C is mostly sugared Assembly? There is nothing that you can do with C, that you can’t do with an assembler. And yet, the comparatively high-level nature of C, and its expressiveness, end up making C programming better in almost every way.

                                                                                                                          For example, here’s code that you can write in Preql:

                                                                                                                          func apply_twice(f, x) = f(f(x))
                                                                                                                          func add1(x) = x + 1
                                                                                                                          print [1, 2, 3]{apply_twice(add1, item)}
                                                                                                                          // result is [3, 4, 5]
                                                                                                                          

                                                                                                                          This type of coding just isn’t possible in SQL, and opens up a lot of options for code-reuse, and having a real standard library, like Python does.

                                                                                                                          all the quirks of SQL makes the latter approach pretty hard

                                                                                                                          They definitely make it harder than it should be, but don’t entirely prevent it.

                                                                                                                          With recursive CTEs you can at least theoretically see how it could be done

                                                                                                                          Recursive CTEs have a very glaring deficiency. They can only UNION on the entire tuple, which means that if you want to keep track of depth in your search (and you usually would), you can’t prevent nodes from being visited twice.

                                                                                                                          I think most major databases offer alternative ways to do graph search, but CTEs have very limited utility.

                                                                                                                          Thanks for the links, and for your thoughtful response!