I’ve been using lisp nearly exclusively at work since 2009, and it’s been very successful. I believe the reason you don’t see it is that it requires companies to invest in training their people. You can’t hire many people who have the skills you need, but you can teach them, and we’ve had great success in leveling people up with lisp who have never seen it before in their life. But it requires an investment and many companies simply aren’t willing to invest in people.
The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA’s New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.)
The author also offers this up at the end:
The demise of Lisp at JPL is a tragedy. The language is particularly well suited for the kind of software development that is often done here: one-of-a-kind, highly dynamic applications that must be developed on extremely tight budgets and schedules.
We see evidence of the efficacy of Lisp everywhere except in production code
Yeah, and oddly it seems like projects that grew to success with Lisp choose to embark on a rewrite journey using another language. Few examples: Viaweb, reddit.
I’m baffled by this, but then again, I’ve never written anything serious on Lisp. The language (at least the Common variant) seems to be fairly practical and has high-quality implementations.
My guess: expressive static types make refactoring easier, and easy refactoring becomes more important as your code base grows.
You’ll always need tests. A lot of behaviors are hard to prove (or even define) statically. Does this error out when it exceeds some configurable limit? In sure you can do this with dependent types or formal verification, but I won’t, and you probably don’t care enough to pay someone who will.
But a lot of behaviors are way easier to enforce with types than tests. Is this a string? Is it a valid user name, or can it be an error message too? Can it just be null? The more of these easy properties you can offload to the type checker, the more of your time you can dedicate to writing tests for the hard stuff.
Or, the less time it takes to get to fearless refactoring.
Early on a small team can move really quickly in lisp languages (I don’t buy the argument that macros magically double your effectiveness, at best you’re going to get a boost of 10-15% with a lisp, and most of that doesn’t come from macros) but when a company gets big they start to get nervous about what happens when their original team moves on. Very rarely this will happen at a company that’s got a culture that values training and spending time to invest in their new hires, and they’ll keep the language, but usually they bring in MBAs who don’t know how to operate unless every worker is a replaceable cog.
Great question. Here are the advantages I was thinking of:
Regularity of syntax
Repl-first interactive development
Conceptual simplicity
The only non-lisp I know which achieves the first is Forth, which also hasn’t seen much production uptake. Maybe Rebol?
The second one is particularly interesting because nowadays lots of languages can technically support a repl, but it’s rare to see languages which really embrace it to the extent that lispers do. Erlang is the only language I know where this happens a lot. (Other than Smalltalk, of course.) You can do it in Ruby, but … no one does, for some reason? I can’t say I understand this at all.
The last one is harder to put my finger on. But all my professional lisp experience comes from Clojure, and being able to take advantage of the JVM and its libraries without the suffocating ceremony of Java gives you a definite edge. I guess this one doesn’t apply to lisps in general though; Common Lisp’s spec is famously enormous, and Racket is a big language with certain murky corners (like the class system) that most people just pretend don’t exist. Emacs Lisp is clunky as hell but makes up for it in the second category, taking it further than anyone except Smalltalk.
So I don’t mean to say that these advantages are unique to lisps, but in the context of this thread we’re talking about lisps in comparison to languages which have seen widespread production adoption.
At least as far as common lisp goes: stability, resource-efficient implementation, gradual typing and the best REPL.
Negatives: you’re practically forced to use emacs and the language is clunky at some places. And everybody’s suspicious of common lisp because there are so few success stories.
Macros are almost totally worthless to me, frankly. Of course, when it so happens that their power is needed, they’re pretty damn good. S-expressions are artistically fascinating, but I also feel like they’re usually more difficult to read. I would expect that to change rapidly for the better if I actually worked with common lisp for a while.
Except they’re not unique to LISP. Nim has a very powerful macro feature that has been used for, among other things, adding async/await to the language. I was tempted to try to implement a SQL parser in a macro, but never got around to it.
I haven’t used Nim and am a novice Lisper so I can’t comment on it - would be interesting to hear a comparison of the macro system from someone who has used a lot of each language. Those statically typed languages with macros ala Zig, Jai etc are very interesting as successors to C
I haven’t gotten deep into Nim macros, but in a nutshell, a macro function is evaluated at compile time and is passed the ASTs of its arguments (there is an AST module defining the structs for this.) The macro code then generates a new AST and returns it.
The difference is that Nim macros operate on AST nodes which use a completely separate notation for expressing code than programmers normally use when writing it. You can accomplish the same thing in Nim as in lisps of course (turing tarpit, etc) but in non-lisps you have to translate between the two notations in your head.
I’m not trying to convince anyone, just explaining the tradeoffs. But there’s no “translation” involved in writing regular lisp code once you know it, any more than you’re translating pseudocode in your head into Algol-like notation when writing C.
That feeling you’re describing is due to unfamiliarity, not due to lisp. You would feel the same mental effort when you start writing prolog, or factor, or smalltalk.
Reddit was rewritten in another dynamic language (python) so I don’t think static types are a perfect answer here. I suspect many rewrites are for people reasons: easier to hire and bigger ecosystem access libraries.
E.g., I saw this article about writing new services in Kotlin from a company that does a lot of Elixir work.
No matter how intelligent or smart any single person, you really can’t beat the millions of people years that have been put into big ecosystems.
TBH, I’m not sure. The TIOBE Very Long Term History chart says that in 2006 Lisp was 13th and Python was 8th. (Of course, TIOBE, Red Monk, etc. are questionable.)
That said, it looks like my suspicions were wrong. Here is Aaron Swartz’s blog post on switching from Lisp to Python. Here are some salient quotes:
Over at reddit.com, we rewrote the site from Lisp to Python in the past week. It was pretty much done in one weekend. (Disclosure: We used my web.py library.) The others knew Lisp (they wrote their whole site in it) and they knew Python (they rewrote their whole site in it) and yet they decided liked Python better for this project. The Python version had less code that ran faster and was far easier to read and maintain.
Others assumed there must have been outside pressure. “Either libraries or hiring new programmers I guess.” Another concluded: “some vc suit wants a maintainable-by-joe-programmer product. I hope he pays you millions.”
The more sane argued along the lines of saying Lisp’s value lies in being able to create new linguistic constructs and that for something like a simple web app, this isn’t necessary, since the constructs have been already built. But even this isn’t true.
No other language has people claiming what the Lisp people claim. Many langs claim to be influenced by Lisp, or try to do things that Lisp can do, or do things that started in Lisp (which is just about everything but OOP). But the Lisp crowd doesn’t see a need to be like other languages. It’s like fat people want to date fit people, but fit people do not want to date fat people.
Been a while since I last got to call someone a Smug Lisp Weenie
Also, astonishingly gratuitous swipe at fat people.
What was that supposed to accomplish? Most people don’t listen better when you start off by randomly reminding them how many people don’t think they’re attractive.
It seems like everybody wants the capabilities that s-expressions give them, without actually using s-expressions. It has been tried, and nobody has been able to get it to work
If he’s talking about macros (the main thing I think of when I think “source code = runtime data structure”,) then he’s ignoring Rust and Nim.
Frankly, if you are a PhD student in CS and you do not grasp parentheses, then you deserve condescension.
This is arrogant bullshit. It’s not about “grasping” parens, it’s about being able to intuitively parse code soaked in parens. It’s more a visual-perception thing. I’ve always found it difficult to count many adjacent identical tiny objects; I’d probably be a terrible jacks player. But while I do not have a Ph.D I’ve been quite successful in a CS career. I just happen to use languages whose syntax is more readable, on the grounds that I’d rather have the compiler grunge through ASTs than be made to do so myself.
I found that particular bit of rudeness quite galling too, but it may reassure you to learn that the author’s own tolerance for parentheses is relatively low. In a recent blog post, they are studying Scheme, and they write:
I know I think the whole argument about Lisp/Scheme being hard to read because of all the parentheses is usually exaggerated, here I think for these letrec examples there might be something to it. These were hard to type correctly from the book.
Reproduced without further comment, so that you can judge for yourself, is the expression they’re talking about:
(define (union-letrec s1 s2)
(letrec
([U (lambda (set-1)
(cond [(null? set-1) s2]
[(ss-sc:member? (car set-1) s2) (U (cdr set-1))]
[else (cons (car set-1) (U (cdr set-1)))]))])
(U s1)))
To paraphrase the old saw about computers and productivity:
“We see evidence of the efficacy of Lisp everywhere except in production code. “
I’ve been using lisp nearly exclusively at work since 2009, and it’s been very successful. I believe the reason you don’t see it is that it requires companies to invest in training their people. You can’t hire many people who have the skills you need, but you can teach them, and we’ve had great success in leveling people up with lisp who have never seen it before in their life. But it requires an investment and many companies simply aren’t willing to invest in people.
There was an interesting anecdote in this story:
https://lobste.rs/s/xzzrct/lisping_at_jpl_2002
The author also offers this up at the end:
Yeah, and oddly it seems like projects that grew to success with Lisp choose to embark on a rewrite journey using another language. Few examples: Viaweb, reddit.
I’m baffled by this, but then again, I’ve never written anything serious on Lisp. The language (at least the Common variant) seems to be fairly practical and has high-quality implementations.
http://www.aaronsw.com/weblog/rewritingreddit
My guess: expressive static types make refactoring easier, and easy refactoring becomes more important as your code base grows.
You’ll always need tests. A lot of behaviors are hard to prove (or even define) statically. Does this error out when it exceeds some configurable limit? In sure you can do this with dependent types or formal verification, but I won’t, and you probably don’t care enough to pay someone who will.
But a lot of behaviors are way easier to enforce with types than tests. Is this a string? Is it a valid user name, or can it be an error message too? Can it just be null? The more of these easy properties you can offload to the type checker, the more of your time you can dedicate to writing tests for the hard stuff.
Or, the less time it takes to get to fearless refactoring.
It’s 100% about hiring. (OK, maybe 95%…)
Early on a small team can move really quickly in lisp languages (I don’t buy the argument that macros magically double your effectiveness, at best you’re going to get a boost of 10-15% with a lisp, and most of that doesn’t come from macros) but when a company gets big they start to get nervous about what happens when their original team moves on. Very rarely this will happen at a company that’s got a culture that values training and spending time to invest in their new hires, and they’ll keep the language, but usually they bring in MBAs who don’t know how to operate unless every worker is a replaceable cog.
What is the boost with a Lisp, besides macros? Afaik macros are the only unique part of lisps that haven’t been subsumed by other languages
Great question. Here are the advantages I was thinking of:
The only non-lisp I know which achieves the first is Forth, which also hasn’t seen much production uptake. Maybe Rebol?
The second one is particularly interesting because nowadays lots of languages can technically support a repl, but it’s rare to see languages which really embrace it to the extent that lispers do. Erlang is the only language I know where this happens a lot. (Other than Smalltalk, of course.) You can do it in Ruby, but … no one does, for some reason? I can’t say I understand this at all.
The last one is harder to put my finger on. But all my professional lisp experience comes from Clojure, and being able to take advantage of the JVM and its libraries without the suffocating ceremony of Java gives you a definite edge. I guess this one doesn’t apply to lisps in general though; Common Lisp’s spec is famously enormous, and Racket is a big language with certain murky corners (like the class system) that most people just pretend don’t exist. Emacs Lisp is clunky as hell but makes up for it in the second category, taking it further than anyone except Smalltalk.
So I don’t mean to say that these advantages are unique to lisps, but in the context of this thread we’re talking about lisps in comparison to languages which have seen widespread production adoption.
At least as far as common lisp goes: stability, resource-efficient implementation, gradual typing and the best REPL.
Negatives: you’re practically forced to use emacs and the language is clunky at some places. And everybody’s suspicious of common lisp because there are so few success stories.
Macros are almost totally worthless to me, frankly. Of course, when it so happens that their power is needed, they’re pretty damn good. S-expressions are artistically fascinating, but I also feel like they’re usually more difficult to read. I would expect that to change rapidly for the better if I actually worked with common lisp for a while.
Except they’re not unique to LISP. Nim has a very powerful macro feature that has been used for, among other things, adding async/await to the language. I was tempted to try to implement a SQL parser in a macro, but never got around to it.
I haven’t used Nim and am a novice Lisper so I can’t comment on it - would be interesting to hear a comparison of the macro system from someone who has used a lot of each language. Those statically typed languages with macros ala Zig, Jai etc are very interesting as successors to C
I haven’t gotten deep into Nim macros, but in a nutshell, a macro function is evaluated at compile time and is passed the ASTs of its arguments (there is an AST module defining the structs for this.) The macro code then generates a new AST and returns it.
The difference is that Nim macros operate on AST nodes which use a completely separate notation for expressing code than programmers normally use when writing it. You can accomplish the same thing in Nim as in lisps of course (turing tarpit, etc) but in non-lisps you have to translate between the two notations in your head.
Yes, but I’d much rather do that mental translation on the rare occasions that I write or edit a macro, than all the time when I write regular code.
I’m not trying to convince anyone, just explaining the tradeoffs. But there’s no “translation” involved in writing regular lisp code once you know it, any more than you’re translating pseudocode in your head into Algol-like notation when writing C.
That feeling you’re describing is due to unfamiliarity, not due to lisp. You would feel the same mental effort when you start writing prolog, or factor, or smalltalk.
Reddit was rewritten in another dynamic language (python) so I don’t think static types are a perfect answer here. I suspect many rewrites are for people reasons: easier to hire and bigger ecosystem access libraries.
E.g., I saw this article about writing new services in Kotlin from a company that does a lot of Elixir work.
No matter how intelligent or smart any single person, you really can’t beat the millions of people years that have been put into big ecosystems.
How much more prevalent was Python than Common Lisp when Reddit was rewritten?
TBH, I’m not sure. The TIOBE Very Long Term History chart says that in 2006 Lisp was 13th and Python was 8th. (Of course, TIOBE, Red Monk, etc. are questionable.)
That said, it looks like my suspicions were wrong. Here is Aaron Swartz’s blog post on switching from Lisp to Python. Here are some salient quotes:
[Comment removed by author]
Been a while since I last got to call someone a Smug Lisp Weenie
Also, astonishingly gratuitous swipe at fat people.
What was that supposed to accomplish? Most people don’t listen better when you start off by randomly reminding them how many people don’t think they’re attractive.
If he’s talking about macros (the main thing I think of when I think “source code = runtime data structure”,) then he’s ignoring Rust and Nim.
This is arrogant bullshit. It’s not about “grasping” parens, it’s about being able to intuitively parse code soaked in parens. It’s more a visual-perception thing. I’ve always found it difficult to count many adjacent identical tiny objects; I’d probably be a terrible jacks player. But while I do not have a Ph.D I’ve been quite successful in a CS career. I just happen to use languages whose syntax is more readable, on the grounds that I’d rather have the compiler grunge through ASTs than be made to do so myself.
I found that particular bit of rudeness quite galling too, but it may reassure you to learn that the author’s own tolerance for parentheses is relatively low. In a recent blog post, they are studying Scheme, and they write:
Reproduced without further comment, so that you can judge for yourself, is the expression they’re talking about: