1. 66
  1. 13

    Thanks @sjl! I keep thinking I should try again at Lisp and this post will make a handy roadmap to refer to.

    But my number 1 question, why aren’t there more significant projects written in lisp?

    I would think that it would become a dependency for some important piece of software but it never is (aside from Emacs).

    Git is so important that it will require perl and bash on my local machine (even on windows!), mercurial will include python, firefox is starting to sneak Rust in piece by piece, but is lisp just so happy being on its own island that it never gets used by any other project? Clojure flipped this by buildling the lisp on top of the other language, which I think contributes to its success.

    Or does it’s flexibility make it difficult to use in large organizations (with varying levels of experience)? Or tsomething else?

    Thanks

    1. 13

      First, there are at least some popular non-trivial projects that use Common Lisp – pgloader is an example. But you’re right that it’s certainly rare.

      The glib answer to “why aren’t there a lot of Common Lisp projects” is”because there aren’t a lot of Common Lisp programmers”. I think there are a couple of reasons for that.

      One reason is that the barrier to entry for Common Lisp is really high. I’ve had people tell me I could learn Go in a month. I don’t know if that’s true (though I’ll be finding out when I start a new job in Go in October) but I can tell you that you definitely cannot really learn Common Lisp in a month. I think even six months of hard work would be pushing it. It’s been three years for me, with two of them being working in CL almost full time (thanks, grad school), and I’m comfortable now but still feel like I have a lot to learn. Most people aren’t lucky enough to have a few years of their life to dedicate to learning a language, so the pool of Common Lisp programmers is always going to be smaller than languages with less of a barrier to entry.

      Another reason is that there are some really common misconceptions about Common Lisp, the primary one being “it’s a functional programming language”. It’s not. It’s a language you can do functional programming in if you want, but is really a procedural language at heart. The misconception is bad for two reasons: people who start learning it because they want a functional language get disillusioned and quit, and people who would otherwise want a procedural Lisp never consider it because of the reputation.

      Your example of Python is a good one because it illustrates another aspect of the chicken/egg problem: Common Lisp isn’t installed by default anywhere. If I write a small script in Python I can be pretty sure I can just run it on a server somewhere. If a newbie wants to get started writing some Python on MacOS or Linux they can just dive in and worry about all the virtualenv shit later. But CL isn’t included by default on any distro that I know of, so that’s another barrier to entry to overcome.

      Those are just a few reasons off the top of my head. To sum it up: I think there’s a bunch of reasons that all feed back on each other and result in a high barrier to entry for Common Lisp. This means there are fewer Common Lisp programmers overall, and that results in fewer Common Lisp projects overall.

      1. 6

        Why is the barrier to entry for Common Lisp so high? Is it just the absurdly high number of things that can be done in it? It kinda sounds like the C++ of parenthesis-languages, if the learning curve is that long.

        1. 10

          A couple of reasons:

          • It’s a big language (though not C++ big, I think). CLtL2 is one book, but it’s a thousand-page book.
          • A lot of things are structured and/or named oddly for historical reasons. You need to unlearn and relearn a bunch of stuff, and just have to memorize other things.
          • There’s more high and low-level power than you might be used to in a single language. If you really want to learn CL you’ll want to eventually get comfortable writing super high-level metaprogramming macros and also reading x86 assembly spit out by DISASSEMBLE.
          • CL tends to err on the side of giving the programmer freedom instead of saying “there is exactly one way to do it” (e.g. the package system’s orthogonality with files).

          In short: the barrier to entry is so high for the same reasons it took me ten thousand words to describe how to learn it :)

          1. 8

            Shorter: Common Lisp is the combined second-system effect of multiple existing Lisp implementations, all implemented by people who worked at places like the MIT AI Lab and Symbolics and who were, therefore, accustomed to complicated and powerful systems. I mean, when your OS’s command line interpreter is a machine code debugger (HACTRN on ITS), you obviously don’t have much respect for an argument that a given feature gives programmers “too much” power.

            1. 4

              That’s exactly what I read happened. Each one was a powerful toolset people were using. Common LISP tried to be the one language to rule them all. It was a huge mess that achieved its goal. A programmer wanting LISP-like power without compatibility with older, merged LISP’s might want something else entirely. Hence, the success of some Scheme’s, Clojure, and even non-LISP’s with macro systems. There’s still people that like what Common LISP has, too.

              If CL is too much, there’s other options that get one many benefits.

            2. 3

              It’s a big language (though not C++ big, I think)

              It looks like CL standard is quite close in size to C++.

          2. 6

            You don’t write about Clojure much lately, is that because of a preference for CL / distaste for Clojure? Would be interested in your current thoughts on Clojure.

            1. 2

              Having learned Go (and written the majority of a decent sized system in production), a solid programmer can be productive in 1 month. I taught it to myself (mostly) with a greenfield project and if you were working in an environment with knowledgable co-workers or some code to reference/build on, that shouldn’t be too much of a stretch.

              1. 3

                You can be a productive programmer in a couple of weeks in CL as well. IIRC Dan Weinreb said they gave new hires PCL and to weeks. But being able to contribute to a code base doesn’t require learning the whole language. For example you can be productive without learning how to customize the reader.

                1. 2

                  Respectfully, I’m going to say Dan Weinreb had a sampling bias. Not all of us live near schools that have students (much less faculty) that are bastions of Lisp (or scheme) programmers. Not to mention that most MIT engineers don’t typically work next to programmers from $SMALL_STATE_SCHOOL.

                  1. 6

                    Lisp, and Common Lisp, are not complicated languages to learn. Quite the opposite in fact and that is the reason I like them.

                    There is a myth that they are complicated. When I tell others Common Lisp is my favorite language one of the first things said (besides the obvious parenthesis jokes) is: “Ooh, that’s such a hard language!” I don’t know where that comes from.

                    C++ is hard, Common Lisp is not.

                    1. 2

                      Not all of us live near schools that have students (much less faculty) that are bastions of Lisp (or scheme) programmers

                      They were talking about people that didn’t knew any Lisp before joining the company. So I’m not sure why would the availability of Lisp programmers would be of any relevance to the question how long does it takes a programmer to go from I don’t know Lisp to I can contribute to a code base.

                2. 2

                  I can tell you that you definitely cannot really learn Common Lisp in a month

                  I think you can be productive in a month or two. (In any language, of course, it takes much longer to become expert.) A lot of people compare their experience learning Common Lisp on their own to learning other languages on the job, or in some other collaborative environment where it’s easy to ask questions of more experienced people, or where you are working on an existing codebase, written by people skilled in the language, with patterns you can emulate. It’s much easier to learn a language in that kind of environment, but there are few opportunities to learn Common Lisp that way. I think it’s these circumstantial factors, not anything intrinsic, that give Lisp its reputation as a hard language to learn;

                  1. 1

                    Is the barrier to Common Lisp really high?

                    I’m not sure I agree. It’s pretty easy (except perhaps on Windows) to just start up SBCL and fiddle around. There’s no huge swathes of things you need to install and IDE’s to tweak to do a simple “Hello, World!” or to do some excercises from a book.

                    Also, it’s a very practical language that does not get in the way. Lisp itself is syntactically very simple.

                    I do admit that for certain projects (f.e. GUIs) things get a little harder and the standard library is huge but that’s something you pick up along the way.

                  2. 10

                    Lisp provokes an instinctive hostility from others, much like Haskell.

                    The Common Lisp community is a crew of mostly loners.

                    Writing Lisp well takes, in my experience, an above average programmer, and in a group, a programmer sensitive to varying capabilities.

                    Many groups reject tools that demand anything above average.

                    GUI integration is, as usual for most open source, terrible.

                    Web service software tends to be messy.

                    No company or Glamorous Person really is championing Common Lisp. Unlike Haskell, the academy has rejected Common Lisp (but tolerates Scheme).

                    Companies absolutely loathe risk and extra investment, so the above keeps Lisp out of the corporate environment.

                    All of the above keep Common Lisp roughly around OCaml in capability and mindshare. It’s a shame.

                    1. 3

                      Writing Lisp well takes, in my experience, an above average programmer, and in a group, a programmer sensitive to varying capabilities.

                      I disagree with this assessment. The main attraction of CL to me is that it gets out of my way when I want to do something. I don’t have to jump through hoops of the arbitrary restrictions of a particular programming language. All programers may benefit from this quality. I would concede that it may take an above average programmer to exploit said freedom.

                      1. 7

                        A lot of programmers are simply uninterested in expressive power. And, a lot of engineering leads prefer to limit expressive power to limit cleverness. Hence Java and Go.

                        I understand the power of Common Lisp - I maintainish two Common Lisp assistance sites and used it for 8 years nearly continuously at home; I still use it myself for my own site. But how typical industrial programming work goes is simply opposed to the Common Lisp ideals, in my experience and discussion with others.

                    2. 3

                      The main reason why you won’t see a Lisp ‘killer app’ as part of the user’s OS is that Lisp’s world view of the world is incompatible with Unix. Unix accommodates to different languages by providing a ‘common interface’ in processes. To some degree, one can draw the analogy with processes being function calls , the standard input being the parameters of the function and the standard output its return value. By contrast, in Lisp (and Smalltalk) you start with a base world (your image) and you add your program into the world little by little, by incremental compilation. Your compiler, debugger, and the rest of the environment is part of your image. To accommodate to the Unix world view would mean to recompile the whole program each time, which destroys the advantages of incremental development offered by Lisp.

                      Or does it’s flexibility make it difficult to use in large organizations (with varying levels of experience)? Or something else?

                      We know its been used in large organization (e.j. ITA now Google Flights) so its not a question of capability.

                      1. 1

                        I would say the mind set of not wanting to deal with the outside world seems problematic. You wouldn’t want to write a program that solves a problem for the rest of the world, because the other problems are solved by something else?

                        I knew of ITA/Google Flights, but didn’t think it qualified as a large organization (though I’ll happily be corrected).
                        I know that the designers of Java and much later Go and even later Dart all specifically tailored aspects of the language towards large groups with varying skill levels, but I’ve never come across anything that mentioned that was a design goal of Common Lisp - again, happy to be corrected.

                        1. 2

                          I would say the mind set of not wanting to deal with the outside world seems problematic.

                          I didn’t say that Lisp doesn’t want to deal with the outside world. I said that its view of the world is different from Unix. In Lisp a ‘program’ is not different than a function. How else are you going to edit your compiler while its running? Your debugger while your debugging?

                          Mind you that you could a Lisp implementation that performs the edit/compile/run cycle from the CLI, mocl does AoT compilation and there is wcl (which is mostly a PoC) but that would destroy any semblance of interactive development which is one of the main differentiating features of Lips. It would be no more interactive than the other batch oriented PLs, say Racket.

                          I knew of ITA/Google Flights, but didn’t think it qualified as a large organization

                          So what number qualifies? 50+ (Or 70 I’m not sure) developers working on a +2M LoC is a large organization in my books.

                          1. 1

                            Thanks, I was unsure of the size of ITA (pre-acquisition) or Google Flights (post-acquisition).

                    3. 6

                      This was great. I particularly appreciated the story about “kludges” 😂

                      Several quotes apply to far more than Lisp:

                      • “The Hamster Wheel of Backwards Incompatibility we deal with every day is a fact of life in most modern languages, though some are certainly better than others.. I try to stick to fewer than ten or so dependencies for my applications and no more than two or three for my libraries (preferably zero, if possible), but I’m probably a bit more conservative than most folks. I really don’t like the Hamster Wheel.”

                      • ” Skimming is a very useful skill to practice as a programmer. I think it’s better for authors to err on the side of explaining too much when writing books and documentation — expert readers should be comfortable skimming if you explain too much, but new users will be stuck wallowing in confusion if you’re too terse. Creating hours of newbie misery and confusion to save a few flicks of an expert’s scroll wheel is a poor tradeoff to make.”

                      • “If your arrow keys and backspace don’t work in the REPL, use rlwrap to fix that. rlwrap is a handy tool to have in your toolbox anyway.”

                      1. 1

                        How is SBCL’s Windows support these days? The SBCL home page says it’s experimental. I ask because the article only recommended ClozureCL for Mac users.

                        Does anyone use CL to develop native mobile apps? mocl seems to be abandoned; the front page hasn’t been updated in years. Is there any practical alternative for iOS and Android?

                        1. 4

                          I think “experimental” is just a standard disclaimer. While SBCL’s Windows support is perhaps not at the Linux level it is pretty good. I’ve used threading and FFI without much issues years ago.

                          Would I use SBCL on Windows in production when money is at risk? I don’t know, I would extensively test it then. (But I wouldn’t use Windows when money is at risk anyway.)

                          1. 1

                            As I understand, ECL works for iOS and Android. Quoting the article:

                            ECL can be embedded in a C program, and can translate Common Lisp code to C code.

                            1. 1

                              I’d be interested in hearing about CL on mobile as well. Kawa Scheme can run on Android/Dalvik, but can ABCL?

                            Stories with similar links:

                            1. A Road to Common Lisp via majkinetor 2 years ago | 59 points | 33 comments