1. 6

    Hey folks, author here. Happy to answer questions if you have any!

    1. 4

      Nice post! I was confused about Dark at first, but it’s starting to make more sense. Generally I’m skeptical of integrated languages and editors, but I can see where they can get some wins with that approach.

      I’m also skeptical of clean-slate / commercial languages, if only because even a meager standard library takes tremendous effort to build.

      But choosing HTTP backends as the initial use case is smart because you don’t need as much of a standard library. In that sense, it’s like a shell. :) The shell is also a lot of effort to build, but it’s feasible for one person because a lot of the “library” is in external processes.

      (In contrast, my work with the Python intepreter has showed that if you want feature parity for even the basic set of objects, leaving aside the standard lib, it’s a huge effort. Which is why I find Micro Python really impressive.)


      I also liked this part – it seems like a balanced analysis of static vs. dynamic.

      There’s little I find more frustrating than discovering in step 2 that the type changes I diligently propagated in step 1 were wrong and that I wasted the last hour.

      I usually ignore most things on the Internet about static vs. dynamic, because people are usually extrapolating from small projects, or small parts of a big project (e.g. maintaining code they didn’t design). But this post seems well-informed and realistic.

      1. 4

        Thanks!

        But choosing HTTP backends as the initial use case is smart because you don’t need as much of a standard library.

        Yeah, that was kinda one point that I realized that Dark might actually be feasible. Since we just want to write programs that receive data (over HTTP), process it, store it, and send it, then we don’t lots of other stuff that current languages/stdlibs have to deal with: unix, containers, machines, filesystems.

        1. 1

          Does the Dark language currently have concurrency or a design for it? I’d be interested in what you came up with for that problem, which seems very relevant to HTTP backends.

          For example, someone might want to write a backend that scrapes data from several Web APIs and compares them, and handles concurrency / slow APIs / timeouts / failure.

          It seems like you’re influenced by Elm and OCaml. Would concurrency be based on the constructs in either of those languages or something more novel? I know OCaml has async libraries but I haven’t used them, and I’m not sure if people like them.


          As a historical note, Python 3’s asyncio was influenced by Guido’s work on a similar problem, making concurrent database queries in Google App Engine:

          https://pyvideo.org/europython-2012/ndb-the-new-data-store-library-for-google-app-en.html

          Python did have a long history with async, e.g. Twisted, Tornado, and even “Medusa”/asyncore if anybody remembers that. And I remember Guido synthesizing a lot of experience from those projects. But they were always outside the language, and Python 3 changed drastically, with many new language features, after his work on NDB.

          1. 1

            We haven’t really looked at concurrency yet. We have background jobs but that only solves a small fraction of use cases.

        2. 3

          Indeed, HTTP backends are a great choice for initial use-case! You get the UI for free by implementing a simple templating engine, you don’t need to worry too much about hardware compatibility because you’re just listening to standard network traffic & responding to it, etc.

        1. 9

          Statically-typed functional languages are amazing. They have a property that exists almost nowhere else: if your program compiles, it works.

          Ahem…

          1. 4

            It’s true if the author has very distorted notion of what “it works” means.

            Lets say you wanted a program that sorts a list. What’s the type signature? It needs some sorting function, and list to sort as the input. The result would be a sorted list.

            Now if we do Haskell we can’t represent sorted lists (probably some dude comes to say it’s got an extension for this), but hey it’s okay. We can accept that it works if it compiles.

            sort :: (a -> a -> bool) -> [a] -> [a]
            sort lt list = []
            

            Since it compiles, it works. But you didn’t get the list sorted? Well that’s because you were unable to tell the type system that the list must be sorted. But it works since it compiles. The program perfectly matches the definition in the type after all.

            If the user complains too much, you can supply the second program:

            sort lt list = sort lt list
            

            Since it compiles, it works. Perfectly. It’s perfectly correct program because we say so.

            1. 0

              Now if we do Haskell we can’t represent sorted lists

              Are you sure?

              1. 7

                Yes. We’ll consider Haskell 2010. In this example, your referenced type is defined as:

                newtype SortedList a = SortedList [a] deriving (Eq, Ord)
                

                This clearly does not encode any sorted requirement in the type. Only smart constructors and programmer discipline can keep that list sorted.

                Here are some list functions. They all have the same signature, but different behaviors:

                id :: [a] -> [a]
                return . head :: [a] -> [a]
                reverse :: [a] -> [a]
                const [] :: [a] -> [a]
                init :: [a] -> [a]
                (++ []) :: [a] -> [a]
                

                Nothing in the type system can issue salvation here. Nothing surrounding the type system, either; the relevant theorem for free concerns map, which doesn’t appear here. Haskell’s type system doesn’t cover:

                • Partial functions, like head
                • Shape-preserving element-agnostic permutations, like reversing a list without looking inside it
                • Linearity and naturality, ensuring that arguments are used
                • Shape-preserving non-linear transformations, like init
                • Operations which normalize to id but predictably and reliably incur runtime costs, like appending an empty list to the end of a singly-linked list
                1. 0

                  Of course you are totally correct, but I’d like to make the argument in practical terms. If you want to encode a sorted list in the type signature, you can, at least in practice.

                  This is like the inverse of the argument that you can do FP in JavaScript. Theoretically you can, but in practical terms it just doesn’t work.

            2. 2

              This is effectively true for pure functional code if you unit-test it. Once you add side-effects, it’s harder to be sure.

              1. 3

                Unit testing helps, yes, but that’s not what the author is implying.

                1. 2

                  This isn’t true either. Consider a complex simulation encoded as a pure function. A unit test just shows it gives the expected output for a given input, not that it correctly solves the problem for all inputs.

                  This is also true for PBT. Just pick more complex pure functions.

                  1. 3

                    Indeed. That’s why I said it “helps”: you are still up against test coverage.

                    You can unit test and statically analyse all day, but you can always insert bugs that won’t be caught.

                2. 1

                  Author here. Good point. What I was referring to was the observation that many people make when using Haskell/OCaml/Elm, that once they’ve got it compiling it usually (or maybe just often?) works. I definitely overstated that.

                  1. 1

                    Well, you won’t get any type errors ;)

                1. 14

                  I’m skeptical of the grandiose claims this project makes about making programming meaningfully more accessible to the world than it already is; and I dislike the implication that a webapp is the only sort of software that you might want to use a programming language to build.

                  That being said, I think that some of the ideas presented here, like trying to make calls to external HTTP APIs look identical to ordinary function calls, or making things like deploy environment and feature flags and database schemas and indices first-class language constructs, are interesting if they can be made to work. But this post isn’t going into enough detail about those neat-sounding ideas to evaluate them yet.

                  1. 3

                    I dislike the implication that a webapp is the only sort of software that you might want to use a programming language to build

                    Gotta start somewhere, and this is our somewhere :) Ultimately, we’d love to support building all sorts of things, but this is the area we see the biggest painpoint that we think we know how to solve.

                    But this post isn’t going into enough detail about those neat-sounding ideas to evaluate them yet.

                    For sure. The APIs as ordinary fn calls worked ok in a naive form, but it still needs a lot of work (esp around reliability in a distributed system, but also language support for optional params and union types). The deploy thing works really great and people love it when they’re starting their app, but it needs more support for people with stable applications, esp from feature flags. With feature flags, our first two versions weren’t great, and we believe we need more tools around running both paths simultaneously, and integration with our user module, to really make it good. The DB indexing we haven’t got enough load for it to have reached the top of our priority list, but migrations are and we’re pretty confident in our solution there.

                    1. 1

                      You should be skeptical…it’s marketing spam.

                      1. 1

                        That being said, I think that some of the ideas presented here, like trying to make calls to external HTTP APIs look identical to ordinary function calls . . . are interesting if they can be made to work.

                        This was the promise of CORBA, and its abject failure taught us that this goal is neither possible nor desirable.

                        edit: I guess it’s possible to stake out a position of, roughly: we’ve learned a lot since CORBA, and think we can do it right this time. But I’m skeptical, and I’d like to see at least some academic response to e.g. the fallacies of distributed computing in any such claim.

                        1. 2

                          CORBA said “let’s make remote objects and pretend they’re local objects”. We’re saying “there’s all these 3rdparty HTTP APIs, and they’re all vaguely RESTy, can we get a joint abstraction that incorporates retries, rate limiting, auth, network errors, etc”, and add language features that make these really easy to handle.

                          Most importantly, we don’t have any control of the other end of the APIs, so all we can do is make it nicer than the current (awful and disjoint) experience.

                      1. 7

                        Our goal is to make coding 100x easier, which we believe will allow a billion people to build software.

                        Aside from this personally being undesirable, I also think it’s unreasonable. People aren’t not programming because it’s too complex to use Emacs and the shell, or Eclipse and one of it’s many plugins. It’s because they don’t have an interest in building software. Every attempt at making programming accessible to everyone to “inspire” them to code feels misguided, because the people who want to write code will find out how and do so, they’re intrigued by the spirit of the abstract machine and the things it can do. And do 1 billion people on this planet even have the ability to program to a degree where their time wouldn’t be better spent learning a trade or an art? Maybe someone can give me a great justification for this, but Ramanujan didn’t need Mathematica to become a great mathematician, and it makes me think we’re not losing any great programmers to existing user interfaces.

                        Why do we need to know how to use all these tools to write software?

                        Aside from most of these tools not actually being mandatory, the wealth of them allows one to optimise their workflow, increase the robustness of their systems, and simplify some task. And if any of these utilities are deprecated by something better or an alternative approach appears, that link in the toolchain can be replaced without changing the rest of the programmer’s workflow. This is why we have pipes and IPC. The only complexity involved (with the concept at least) is reading the documentation of the simple tools you’re chaining together.

                        1. 7

                          And do 1 billion people on this planet even have the ability to program to a degree where their time wouldn’t be better spent learning a trade or an art?

                          Currently, there are about 750m users of Excel. All these people are doing data editing, manipulation, and visualization, which is close to enough to coding for me. However, they lack the tools to put “code” that they write on a server to make available to others. They also lack the tools to mix the data streams that exist in their lives, whether their fitbit data, their calendars, their emails, their spotify.

                          As an example, think of recruiters. They do a lot of work which is automatable (scheduling, gathering interview feedback, background checks, etc), but they do it by buying one of the thousands of recruiting-automation SaaS products out there. Wouldn’t it be great if they could actually customize their own experience instead of relying on the experience that someone else created? You can extend that to data scientists, analysts, anyone in finance, and really, almost any knowledge worker at all.

                          Maybe we don’t need a billion people to be software engineers, but we (Dark) believe that everyone has things in their work or lives that they could automate and customize and build, if given the tools to do so.

                          1. 2

                            They also lack the tools to mix the data streams that exist in their lives, whether their fitbit data, their calendars, their emails, their spotify.

                            Also I’m curious how Dark compares to ifttt or Zapier? I think they are addressing these kinds of use cases.

                            1. 5

                              Yes, we think of ourselves as a mix of Zapier, Glitch, Lambda, and Parse.

                            2. 1

                              Wow, 750M is a huge number, but it’s not implausible. I noticed the local community college teaching Excel and Word, and I’m sure every community college has similar courses. It’s easy to forget that there’s a very long tail of computer use.

                              Any thoughts on Google Apps Script? It’s basically JavaScript edited on the server, collaboratively, connected to the data in Google Sheets (their Excel clone). It’s a solution for sharing and editing data and code.

                              I’m a shell/Python/C++ person, but I used it because it was able to send mail reliably :-/ Mails sent from my hosting provider via PHP would get rejected by Google (hmmm something is fishy there …).

                              I can see a number of reasons why Google Apps Script isn’t the future of programming, to put it mildly. But if the goal is really to enable 750m users of excel, then starting with simple data entry and then graduating to programming seems like a good idea. And Google Sheets does let you do that.

                              Also I wouldn’t underestimate the power of muscle memory and all those community college courses. Google Sheets and LibreOffice still support the F2 key and simple formulas like =A1+A2 :)

                              1. 1

                                Sure, there’s lots of ways to skin this cat. Our belief is that with the current ascendency of “coding” as a desirable skill, if we make something great for existing coders, the long tail (the next billion coders) will likely follow there. Of course, there are tons of companies making the opposite bet (airtable, google app script, zapier, etc).

                            3. 5

                              Most people aren’t interested. Many people are who will not take up programming because of poor educational resources, its complexity, and/or some of the communities. Addressing those will get more people into programming. A few, popular languages did address one or more of those. So, that’s corroboration.

                              1. 1

                                What kind of programming languages address those sorts of complaints? Just as an example.

                                1. 1

                                  It’s more how you teach them. Most are using a subset of Python today with example problems that increase in usefulness and difficulty. They can script or do web stuff without a ton of training required. For Scheme, Racket with How to Design Programs but that’s heavier. A recent one is Pyret combined with Bootstrap program. They say it’s getting results with middle schoolers. Scratch was another one before it that made programming look like lego blocks with a community effect. Some kids even became entrepreneurs through their Scratch creations.

                                  So, I’m mostly talking about stuff like beginner Python, Pyret w/ Bootstrap, and Scratch. Let’s them get into programming in a fun way with less, incidental complexity. What they create motives them to try the other stuff.

                              2. 4

                                Some choice quotes from a genius that disagrees with you

                                By relieving the brain of all unnecessary work, a good notation sets it free to concentrate on more advanced problems, and in effect increases the mental power of the race.

                                It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments.

                                Both are from Alfred Whitehead. Accessibility is always a good thing and we should strive to make programming as accessible as possible.

                                Your take on the matter though is second rate gatekeeping and downright regressive because you are using an appeal to some internal magical drive and talent to delineate classes of people.

                                1. 2

                                  Gatekeeping isn’t my intention, I’d love to help people get interested in programming. But, I think programming is already as accessible as could really be, sans towards those with certain disabilities. To design a new way that would inspire people in a way the numerous existing approaches haven’t been able to, I don’t see as anything to do with tools, but to do with culture, I guess. (Drive to program isn’t anything magical, some people are just interested in different things.)

                                  1. 3

                                    Then I misinterpreted what you said and the disagreement seems to be around

                                    But, I think programming is already as accessible as could really be…

                                    I don’t think this is true.

                                    1. 3

                                      I explained myself poorly, it’s no surprise you misinterpreted it, my apologies

                                2. 6

                                  I agree about having one billion people programming. If that were true I wouldn’t be able to sleep at night. Re the complexity of the current server-based ecosystem, I’ll just respectfully say that liking it might be a touch of Stockholm Syndrome on our part. If we can reliably get what we need without having to deal with that complexity all the time, we should. It’s the same as the argument for serverless. It may not be for everyone but within certain problem spaces, the reduction in complexity and manual choice are worthwhile.

                                  I don’t know that Dark is the answer, but it’s the only thing I see right now that is at least tackling the right problem.

                                  1. 3

                                    Only thing?!

                                1. 7

                                  Author here, thanks for posting. Happy to answer questions!

                                  1. 6

                                    I can’t help but compare this to VB6. While I wasn’t a fan of some major parts of the VB language (error handling being the largest problem for me) I loved that once you got the IDE installed, lots of things ‘just worked’.

                                    Are you able to draw comparisons and point to differences (in philosophy) for those who are familiar with that platform?

                                    1. 2

                                      Yeah, I think VB6 was one of those huge shifts in computing that opened up development to so many more people, so I think there’s a lot of philosophical similarities. There’s also similarities to the rise of Java (bringing GC to the masses), Heroku (easy CRUD apps). I don’t know enough VB6 to compare effectively, but high level I agree.

                                      1. 6

                                        One of my Unpopular Programming Opinions is that VB6 is still way ahead of its time

                                        1. 1

                                          It absolutely is. This is how apps should be made. It is actually fun to design a UI this way.

                                    2. 1

                                      Has your team done any work yet on making the IDE, particularly the structured editor, accessible to blind users? That’s the thing that always worries me about attempts to do programming without conventional text-based coding.

                                      1. 3

                                        We haven’t yet, as we’re still figuring out what it is. As a company, we believe strongly in inclusivity, so I expect to spend time on making this work. I’ll note that code in dark is still textual, it’s just not “code is a stream of bytes in a file inode on a posix system” textual.

                                      2. 1

                                        Are you looking for any more language hackers right now?

                                        1. 1

                                          Just posted today: https://news.ycombinator.com/item?id=19283993

                                          We’re looking for frontend and infra folks, but we all do everything and involvement in the language design is high. We’re not specifically hiring PL people as that skill is overrepresented on our team right now (and besides, PL is only a fraction of what goes into Dark).

                                      1. 9

                                        First of all, it needs to be noted that it’s an announcement for a yet unreleased project.

                                        That said, IIUC the author created CircleCI (?), which would lend him at least a bit more initial credibility than in case of some Joe Random Hacker in a similar situation. Some history of successful delivery reduces the chances of this being vaporware (though, Duke Nukem Forever was a thing too).

                                        As to the project itself, I think the advertised “save = deploy with feature flag” functionality piques my interest. And I’m especially curious how it’s going to work low-level.

                                        The “editor allows only valid syntax” feature may either be empowering, or crippling, I believe. The idea seems to come back often, but it’s main drawback IIUC is that it makes code drafting hard, interrupting the train of thoughts (at least in typical implementations). So I wonder if this challenge was taken into consideration and resolved somehow.

                                        1. 9

                                          First of all, it needs to be noted that it’s an announcement for a yet unreleased project.

                                          I saw a demo of this in person back in September, and at that point it was pretty far along. At least to my untrained eyes it looked fairly late stage; this is def a real thing.

                                          1. 5

                                            FWIW I’ve been following repl.it, and it seems like their vision is similar, except that they don’t have their own language:

                                            https://repl.it/site/blog/deploy

                                            They basically wrap the real Python interpreter, node interpreter, etc. in a web-based UI, and let people code on the server.

                                            It works surprisingly well, and they have “multiplayer” concurrent editing as well.

                                            Then supposedly you just save and open a port and you have “deployed” your server.

                                            I would say repl.it itself is a crazy hard problem, but having to also build your own programming language and structured editor on top is even harder! That was my initial reading of Dark. But I’ll withhold judgement until I try it :)

                                            1. 2

                                              repl.it is really cool! We feel that it’s actually simpler to build our own language and the exact experience we want than to have to fit within the bounds of existing languages. As an example, we want to meld the DB and the language to have zero impedance mismatch, which I think is something they’re not trying to do, but would also be a much bigger challenge taking their approach.

                                            2. 3

                                              I don’t know anything about this but Paul was in my YC class, built CircleCI, and is a very smart and thoughtful guy. If anyone could do it &c. &c. &c.

                                            1. 4

                                              Sounds very interesting, thanks for sharing!

                                              Can this allow writing CLI apps in Elm, by compiling them with the OCaml compiler after being translated with Phillip2? What are your thoughts on this?

                                              Also, super curious what was your reason to migrate from Elm! Could you share some quick “executive” summary? (I imagine you may be planning to write a separate blogpost with details, but I’m afraid I might overlook it, so would be grateful for any quick notes that you could share now!)

                                              1. 10

                                                Can this allow writing CLI apps in Elm, by compiling them with the OCaml compiler after being translated with Phillip2? What are your thoughts on this?

                                                It’s really intended to be used with Bucklescript, which compiles to JS. Once you have the code in OCaml though, and assuming you don’t use much of the Elm Architecture, it would lovely be pretty straightforward to wire it up. So I’d be optimimistic, if those assumptions hold.

                                                Also, super curious what was your reason to migrate from Elm!

                                                Long story short, the Elm 0.19 decision to remove the ability to call native JS was the breaking point for us. We had a bunch of frustration with different parts of Elm and how it’s developed, but elm 0.19 gave us a point at which we needed to make a decision. Our backend is in OCaml so that was a major factor too.

                                                1. 2

                                                  So I’d be optimistic, if those assumptions hold.

                                                  Isn’t the point of this to aid a one-time migration of an existing Elm codebase to OCaml rather than to be a compiler backend for Elm?

                                                  1. 1

                                                    Oh yeah, good point. I was thinking they’d be doing a once-off conversion to OCaml but that’s not what they’re asking I think.

                                                    You could probably cobble together a janky script to do the whole conversion, if you really wanted a high-maintenance project :)