I’m very tired of this “angry person derides everything the industry does and points at very small-scale examples as a better way” thing people often do when giving talks. We should be using notebooks, interactive or visual programming? I’d like to see demonstrations for millions-of-lines codebases. All software should contain a repl to enable live-debugging? I recently tried to download firefox’s debug symbols from debian’s repository, it’s over one gigabyte. If you’re going to make me download more than a gigabyte each time I have to update your software, while almost never needing to debug it, I’m not going to use your software.
Pointing at things and saying they’re bad without discussing the constraints that went into building the thing and how they could have been solved differently just makes you sound like an asshole. If your goal is to give a quick overview without being able to go into details, then drop the “this other thing is shit” part of your talk.
Thank you for this comment. I got the same feeling when watching the talk. The content is somewhat interesting but the tone is arrogant and insulting to every person who has different practices or opinions.
In particular, the examples are heavily biased towards things that are easy to visualize (either pictures, or small snippets of code that do simple things). It’s a bit like Bret Victor’s talks, where everything just happens to be mostly image oriented. I’d like to see how one visualizes compiler internals, SAT solvers code, etc. because it’s the kind of things I’m interested in, not this fluffy “look at little squares drawn on a canvas” thing.
And all the interactive stuff doesn’t come for free. How much code do you need to write to produce the visualizations themselves? How do you debug that code?
I also wanted to give a quick try to Glamorous Toolkit, mentioned there, because it looks very interesting. It tooks ages to install, and on my laptop it ran at a horrendous (single digit) FPS count. It was almost laughable. I find it revealing that nowhere in the talk does this person mention performance or memory consumption at all (e.g. in the cell example). The dig at C++ is also uncalled for, imho: it’s a language where people get things done, often semi interactively (in game development, for example; see Dear Imgui for how game devs add interactivity to their code), in an environment where performance and memory consumption matter.
One last rant: the tweet “stop the world typecheckers”. No. LSP and type checking inside your editor is interactive too, and I don’t believe that a runtime environment is strictly better or strictly more interactive, since you only interact with a single concrete instance of the program’s state. Typechecking gives you some interactivity on errors that might happen in at least one instance.
the examples are heavily biased towards things that are easy to visualize
And the attitude toward program representation is heavily biased toward people who can see and visualize in the first place. Yes, that’s most people. I’m afraid my comment on the orange site had a self-righteous tone to go with that of the talk. It’s just frustrating sometimes when people so completely ignore the fact that not all of us have the same abilities, and what’s constraining to some is liberating and equalizing for others. I had the same frustration with one of Bret Victor’s talks.
I also wanted to give a quick try to Glamorous Toolkit, mentioned there, because it looks very interesting. It tooks ages to install, and on my laptop it ran at a horrendous (single digit) FPS count. It was almost laughable.
Glamorous Toolkit is in beta. We should avoid judging it too harshly just because we disagree with the person who recommended it. That would be creating a cycle of dismissal and judgement.
Okay, so long as it’s a reaction to the author’s claims and not the claims of a third party. Judge people by their own actions and not how you feel about “their group”.
You’re right. I mean the FPS was almost laughable. I don’t know how hard is it going to be for GToolkit to overcome this level of performance degradation though. I would guess that a GUI toolkit written entirely in a dynamic language might have trouble being as performant as something more, hum, “dead” as the OP would say.
It’s all good! The talk was pretty dismissive and it’s hard not to take a swing at the pillars of their argument. Just want to make sure that we acknowledge that those pillars are other people who might not have as hard of takes.
I would guess that a GUI toolkit written entirely in a dynamic language might have trouble being as performant as something more, hum, “dead” as the OP would say.
How about Smalltalk on the original PARC machines (Alto, Dorado, etc.)? If they could do it back then, presumably we can implement a reasonably fast GUI toolkit in a dynamic language on today’s much more powerful machines.
One last rant: the tweet “stop the world typecheckers”.
This topic opens up the purpose of development environments. One could seriously argue that there can and should be a difference in quality during exploring code, experimenting, and checking code into the group. The goal of development environment is to speed up the developer while capturing every developer intentions along the way. This often means type checking can be ignored until after the first run.
These sort of talks are about showing alternative futures, not about presenting a complete solution. The talk should be seen to have a different aim. Not every solution has to scale. I teach high schoolers and I’d kill for a better way for my students to inspect their code. Being able to inspect state, see docs for every language construct/keyword in line, see visualizations for built-in collections. That would be tremendously useful. Maybe we can keep expanding the horizon of programs that can be worked with in this way, as hardware gets more powerful or as algorithms/languages get better.
Or even think - even a non-scalable toolset that helps you write tricky string-validation functions or regexes or SQL queries - these would be tremendously useful. We already have great tools for inspecting regexes, ofc.
I also liked the rapid-fire list of projects to look into. Plenty of jumping-off points for exploring this space.
The tone and presentation style may not be great, totally agree. But that’s a presentation layer thing.
In doesn’t have to be that bad. Common Lisp ticks many points: there are millions-of-lines codebases (or have been) (thinking about Google’s ITA Software and SISCOG, both are at least above 1 million lines). You deliver binaries smaller than your image, and commercial implementations (LispWorks, Allegro(?)) can strip out dead code.
I’m fairly certain that in some cases it does, otherwise you’re claiming that every programmer in every niche is misguided for doing things the way they’re doing it instead of your way.
Common Lisp ticks many points: there are millions-of-lines codebases (or have been) (thinking about Google’s ITA Software and SISCOG
Millions-of-lines codebases written in Common Lisp existing does not imply that the people developing these codebases use notebooks, interactive or visual programming. If they do, the orator should have demonstrated the gains they experience by using interactive programming on millions-of-lines pieces of software. This would have made his point stronger, as it would show that interactive programming provides gains across the entirety of the codebase-size spectrum.
I’ll add that the “millions-of-lines codebases” was just one of the dimensions I had in mind. There are other niches like FPGA-programming, real-time programming, high-throughtput programming and many others that would also need to be studied before claiming that there is One True Way of programming that is better for all programming niches.
You deliver binaries smaller than your image, and commercial implementations (LispWorks, Allegro(?)) can strip out dead code.
This is already something all the big pieces of software do. It’s trivial to do with GCC (use -ffunction-sections, and then pass --gc-sections to ld). The reason the binaries are still huge is because compile-time monomorphisation is used very heavily. You might argue that you don’t need to perform monomoprhisation at compile-time and delay it to run-time instead, but then you’re going to lose start-up performance, which is pretty much my point: everything is a trade-off, there is no universal answer and all you can do is work within constraints.
Having worked at ITA: the Common Lisp codebase was written using normal text editors, with code checked into version control. Emacs was quite common, and there was some sort of live prompt thing where you could push code updates during development for interactive testing… SLIME maybe?
Last I saw (been many years at this point) it was deployed on SBCL.
Very interesting, thanks for sharing your experience!
and there was some sort of live prompt thing where you could push code updates during development for interactive testing
Was the instance of the program running on developer’s machines, or was it running on servers? Was this instance shared among developers? How often would this instance be restarted (if at all?)? Do you know of any video that would demonstrate this workflow, if not on ITA’s codebase, on something similarly large?
I’ve never done much REPL development (aside from debugging JavaScript in the developer’s console) but have always been fascinated by the idea, I’d love to learn more :)
This was just a local instance, for debugging/live-testing mostly, is my impression. Production deployment was all the standard “compile from source code, deploy to N machines”.
It’s possible they used REPL for semi-live-development, but it all ended up in normal source code files in version control.
If you want a REPL-y dev experience, you can try out Jupyter notebooks, but keep in mind that people recommend restarting the notebook occasionally (or often!) so state doesn’t get messed up. Basically no one that I know of does pure REPL development for anything but one-off scripts, and for good reason.
I think the video’s orator does claim there is One True Way when he claims that compiling ahead of time is bad and that you should really go with repl-programming instead.
Millions-of-lines codebases written in Common Lisp existing does not imply that the people developing these codebases use notebooks, interactive or visual programming.
So, I’m kind of a CL newbie, but my understanding is that CL development is generally done semi-interactively. You write your code in a file, which will be under version control, and so on. But instead of stop-the-world, restart-with-changes, you’re normally working in a live image. As you update a function in your file, you run a command in your editor (usually Emacs) to evaluate the updated function, which replaces the existing function definition, and the same for CLOS class definitions - existing objects are updated to the new class definition. You can examine any aspect of the image’s internal state while it’s running.
The difference in CL images and Smalltalk images, as I understand it, is that in CL, your code in your editor can be out of sync with what’s loaded in the image (either because you haven’t evaluated the changes yet, or you’ve reverted to an older version and not evaluated that), whereas in Smalltalk, it can’t; the image is the only source of truth. Because I’m a n00b, I sometimes have to stop my Lisp process and restart it, loading my code from source, in order to be sure the image is in a state I understand.
Because I’m a n00b, I sometimes have to stop my Lisp process and restart it, loading my code from source, in order to be sure the image is in a state I understand.
People with lot of experience do that too (but not me ofc) , that’s why there’s a command named restart-inferior-lisp
True, and there are plenty of examples that could have been used instead. Fetch the Wayback Machine, Sherman!
Sabre C, renamed CodeCenter, was a popular developer tools for about $4K/seat. It included:
Live debugging. You would be in the middle of loop and want to change the logic. When you typed, the interpreter started, unloaded the currently running object code into the interpreter, let you make the change, and continue stepping though the code.
Set up displays (arrows and values) of values, all pointers in user space pointing to them, and walk through pointer chains by pointing and clicking.
Jump to locations where the last pointer to memory was lost and to where it was allocated.
Do all the usual ‘jump to declaration’ and ‘auto-reformat’ that is common now.
The only reference to Saber-C I could find is here and references a last release date of 1999. I did find a video of something named Saber-Plus but it doesn’t show the live-editing capabilities you’re mentionning. Perhaps live-editing of a running program was only practical when building a compiler whose performance is considered “adequate” was still doable for a single small company? I’d imagine building an interpreter for C++’s most recent revisions, able to deal with the intricacies of the object files produces by GCC, Clang or MSVC, that conserves the performances and behavior of the compiler being emulated might be out of reach for lone actors today.
Sorry, I didn’t put a date on it. It was very popular in 1990. The company imploded around 1995, partly from competitive pressures from Sun and HP, partly from Sun and HP changing object formats quickly to cause quality issues. Between COFF and the formal specification of ELF, changes occurred which killed the ability to swap in the interpreter, sometime in 1993 IIRC. Added to that, C++ was evolving rapidly from a C-front preprocesser to its own language, and its spec changed constantly.
Oddly, it should not be out of reach. Charging $4K/seat for development tools is out of reach.
I was in the audience for the talk and found it incredibly smug and self-righteous. At one point he makes fun of a person who left early to go to the bathroom. At one point he slags Rust for not being an “interactive language”.
I tried a few of the tools he calls the real future of programming and most are little more than demos. The most developed of them had noticeable keystroke lag on my workstation desktop.
I have one big problem with this talk, and that’s… its central thesis actually: one thing the edit-compile-run cycle has going for it is that it’s principled. Namely, you’re starting from a known state (the source code) every time you compile. With live/image based programming where you fix everything on the go, the actual source code isn’t limited to the source text, but the state of the program itself! This has to be unmanageable at some point.
I will also note that some aspects of live programming are available even to C/C++ programmers: just recompile the dll and reload it. This requires significant initial setup, but it allowed more than one game dev to change mechanics on the fly without restarting the whole game, without the help of a scripting language. The Yi editor written in Haskell demonstrated something similar. XMonad can also maintain its state across compilations (which happen whenever we change its configuration).
Well said! I do love REPLs and think they are very valuable, I also prefer “principled” approaches (as you very well put it) and think there is a great deal of value there.
Yup. There have been some threads here about “image-based” development, which I experienced with Smalltalk-80. It’s kind of magical in some ways, but also leads to real problems because your program never starts from scratch.
With live/image based programming where you fix everything on the go, the actual source code isn’t limited to the source text, but the state of the program itself! This has to be unmanageable at some point.
I am a super huge fan of Common lisp and this is something I see every single newcomer struggle with. They usually don’t struggle with it for long though. But it’s definitely something that I would like to see discussed explicitly more often.
On the bright side, Common lisp was designed with that (interactive development) in mind (mostly). What comes immediately to my mind, as an example of this, is the defparameter v.s. defvar expressions that only differ by what they do when they are re-evaluated. Making it easy to reload source code without losing the current state.
Yet another thing: I’ve rarely seen anything that deals with “cleaning up the state”. For example, if you rename a function f1 to f2, the original function f1 still exists in the image unless you explicitly remove with (with the so well named functionfmakunbound). The only time I’ve seen something that deals explicitly with renames is terraform’s refactor, but that’s hardly for “image based development”.
One last thing that I won’t develop because this is already way longer than I wanted: the problem of updating the code within an image is the same problem as updating a database.
the problem of updating the code within an image is the same problem as updating a database.
I think you meant to analogise the database with the state, not the code…
There’s a difference though: with a database, the data is concentrated in one place, and that data is mostly statically typed: you have a fairly rigid set of tables, each column has a definite type… and if it’s all normalised like it should be 99.9…% of the time the interface between it and the code will be relatively thin, and very stable.
Now replace the database with the live state of a long lived program. A dynamically typed program at that. You need to make sure your program state is just as principled as the database’s, the interface between it and the code just as thin and stable… except here all you have is programmer discipline. There’s no clear fence, and depending how your program is accessing your data, good luck determining whether a state change will break your program or not. At least with a database you know that as long as you don’t remove columns or entire tables you’re mostly safe.
So at a first glance, image based programming at scale (let’s say more than 10 devs working on the same code base & associated state) is looks utterly impossible. How does that work in practice?
How do you keep your program from irrecoverably mangling its state?
How do you deal with irrecoverably mangled state?
How do you version the code?
How do you version the state?
How do you share state among several collaborators?
How do you update the code of a production instance while keeping existing state?
How do you update the state of a production instance?
How do you update the state architecture of a production instance?
And perhaps most importantly, do you even need all of the above? Why not?
In CL: if the state becomes a mess, restart your image! No big deal, you loose 2 minutes. For many days, I can develop my program with the same running image, it’s fast, it’s interactive, I don’t loose state, I am happy. It happens indeed that I restart it. I also always run unit tests or build the project from the terminal from time to time, so from a blank test.
We use source files with git. Maybe they didn’t have this in Smalltalk land decades ago, we do now.
How do you share state among several collaborators?
To share state among developers, we can build and share a custom image. It’s nearly like an executable, but it’s an image that offers a REPL. Now instead of running a new SBCL instance from our editor, we start sbcl --core my-image, and it starts instantly.
How do you update the code of a production instance while keeping existing state?
Easy: git pull, connect to the running image (you have a tmux or you use Swank, the server side of Slime, through an SSH tunnel) and you load your source files again (today, it would quickload). The image is not restarted, so you keep state.
Also if you define variables with defvar, you don’t loose their state (versus defparameter).
But, disclaimer: one can very well update production instances without worrying keeping any state: rsync a new binary, stop the service, start the new service. Usual deployment knowledge would apply. Just as with a Go application for instance.
How do you update the state of a production instance?
wow, you mean, from the state of a dev instance to the production one? Who would care to do that? I don’t see the point. I develop on my machine, I deploy new code to the production machine. I don’t have to mix states. Maybe for very specific type of applications? If I need to send data, I send the data files.
In CL: if the state becomes a mess, restart your image! No big deal, you loose 2 minutes.
Oh, I see. So the people raging about image based programming actually don’t really care about their image. And I guess then state is not versioned either. Except of course whatever is needed to establish an initial state from scratch. I also bet the need for several people to work on the same image ranges from limited to nil.
What I thought was a completely alien realm seems actually pretty mundane all of a sudden. What I thought impossible probably is impossible… I just had the wrong idea. Kind of anticlimactic to be honest, but I think I have a better idea of what this really is now.
That’s a way of putting it :p For me, a young-ish lisper, image-based development is awesome for: development, and ease of live reload (if I want). Bonus, I use custom images from time to time (start a Lisp image with a ton of libraries pre-installed and with other tweaks). That’s it, and that’s a lot already.
Oh, I see. So the people raging about image based programming actually don’t really care about their image.
I think it’s different between CL and Smalltalk, maybe? In CL, your image is pretty important during a given development session, but not so much between developers, or between yourself on different days, I think. The main thing is that your changes are applied to a running image without a restart, and you can build up a running program iteratively like that. If you need to get back to a particular state after resetting your image (restarting your Lisp process), you’d set it up the same way you set up fixtures for a unit test in a stop-compile-run language.
Well if some programmers do care about their image to the point they will want to version it in some way, or work on it collaboratively over a long period of time… how they do it remains a total mystery to me.
the problem of updating the code within an image is the same problem as updating a database.
I think you meant to analogise the database with the state, not the code…
Yes and no, but I definitely wasn’t clear xD . I had in mind that updating functions (or methods) in an image is somewhat analog to updating views in a database. But anyway, you seem to understand the problem of updating the state of a running image well enough :)
So at a first glance, image based programming at scale (let’s say more than 10 devs working on the same code base & associated state) is looks utterly impossible. How does that work in practice?
It’s a problem that I find interesting and think about from time to time. The solution, I think, would be something like “gating” the changes made to the running images. So, one dev could redefine a function (in a sandbox, which I have no idea how that would work exactly), if it passes the tests, then the change is applied to the “real” image. It would practically be a CI-within-a-process.
People most definitely don’t care about their dev images, but some do care about the real, always running, production image. In which case they go to great length to make sure the code that is going to be evaluated during a release don’t mess up the state, don’t cause any (foreseeable) bugs and don’t pollute the image. I think this is where my analogy with database migrations came: applying successive version of the codebase to a running image is analog to running successive migrations to a database, which brings me to:
Checks, in Common lisp, you can compile the code before actually evaluating it. SBCL (the most commonly (ha!) used implementation) will catch a lot of “type errors” (which it often reports as warnings, but you can count those warnings as errors). Also, nothing keeps you from running tests for anything you can think of.
How do you update the state of a production instance?
Imagine you have an ever running python or js repl, how would you go about updating a global variable that contains all the state (to keep it simple)?
The first thing that comes to my mind is: only access the state via some specific functions (often called accessors). Update them to support the new and the old state “shape” (I think it’s what you meant by “architecture”) before updating the state.
How do you keep your program from irrecoverably mangling its state?
I covered it a bit, but you kinda just have to be mindful about the changes to the state; type-checking at compile time, assertion at run-time and automated tests all helps. Also keep in mind that you can (and are very often strongly encouraged) to use functional data structures, which helps a lot when you are wary of mangling the state. BUT, bugs can happen, same as database migrations, it can go wrong even with a lot of precaution.
How do you deal with irrecoverably mangled state?
As vindarel said, just restart the process :P
How do you version the code?
Same as with all “mainstream” programming languages. You use git.
But, @gcupc is right about Smalltalk. Smalltalk’s source code was traditionally part of the image. Smalltalk was much more “image-based” that Common lisp on that aspect. I don’t know much more about it though… Except perhaps that I know that Pharo, a recent Smalltalk implementation, tries hard to make Smalltalk work with source files, so that people can use “traditional” command line tools on those files.
How do you version the state?
I never heard anybody doing that exactly. But the “state” would be just a file, so you could use git (or better, git-lfs) to version that file. But I don’t think that’s what you had in mind. (And sbcl is super picky about running an image from a different version, so you would be stuck with a specific version of the lisp implementation.)
I think that what you had in mind is “I have a running process, I apply some code change to it, how does the code know what’s the state’s schema right now?”. I see 2 solutions: you can do it implicitly, by using predicates that checks the types at runtime, or you can keep a “global” variable that explicitly keep a version number.
How do you share state among several collaborators?
Depends what you mean. But in theory, if many collaborators have a running image that they don’t want to close for some reason, they can just git pull the most recent version of the codebase and apply that code to their running image. See point 3, about releasing updates without down-time.
How do you update the code of a production instance while keeping existing state? / How do you update the state of a production instance? / How do you update the state architecture of a production instance?
See point 3.
And perhaps most importantly, do you even need all of the above?
No, you don’t need it.
Why not?
Because it’s freaking hard xD, but I think you already guessed it.
/ disclaimer: this message is way too long, I’m not gonna proof-read it just now ;)
I’ve worked on systems in statically typed languages, as well as dynamically typed languages, some with REPLs and fancy debugging tools, and some without.
As someone who loves lisps and also languages that are quite different from lisp, there’s a reason why homoiconicity, REPLs, and time-traveling debuggers didn’t take over the world, and it’s not because people are stupider than you.
There just isn’t one true model of computation. There isn’t one development/debugging workflow that’s globally better than all others. Humans are diverse and think and communicate in different ways. Some people are more visual in their thinking; others can fit a tremendous amount of state in their brains and think procedurally.
I’m very tired of this “angry person derides everything the industry does and points at very small-scale examples as a better way” thing people often do when giving talks. We should be using notebooks, interactive or visual programming? I’d like to see demonstrations for millions-of-lines codebases. All software should contain a repl to enable live-debugging? I recently tried to download firefox’s debug symbols from debian’s repository, it’s over one gigabyte. If you’re going to make me download more than a gigabyte each time I have to update your software, while almost never needing to debug it, I’m not going to use your software.
Pointing at things and saying they’re bad without discussing the constraints that went into building the thing and how they could have been solved differently just makes you sound like an asshole. If your goal is to give a quick overview without being able to go into details, then drop the “this other thing is shit” part of your talk.
Thank you for this comment. I got the same feeling when watching the talk. The content is somewhat interesting but the tone is arrogant and insulting to every person who has different practices or opinions.
In particular, the examples are heavily biased towards things that are easy to visualize (either pictures, or small snippets of code that do simple things). It’s a bit like Bret Victor’s talks, where everything just happens to be mostly image oriented. I’d like to see how one visualizes compiler internals, SAT solvers code, etc. because it’s the kind of things I’m interested in, not this fluffy “look at little squares drawn on a canvas” thing.
And all the interactive stuff doesn’t come for free. How much code do you need to write to produce the visualizations themselves? How do you debug that code?
I also wanted to give a quick try to Glamorous Toolkit, mentioned there, because it looks very interesting. It tooks ages to install, and on my laptop it ran at a horrendous (single digit) FPS count. It was almost laughable. I find it revealing that nowhere in the talk does this person mention performance or memory consumption at all (e.g. in the cell example). The dig at C++ is also uncalled for, imho: it’s a language where people get things done, often semi interactively (in game development, for example; see Dear Imgui for how game devs add interactivity to their code), in an environment where performance and memory consumption matter.
One last rant: the tweet “stop the world typecheckers”. No. LSP and type checking inside your editor is interactive too, and I don’t believe that a runtime environment is strictly better or strictly more interactive, since you only interact with a single concrete instance of the program’s state. Typechecking gives you some interactivity on errors that might happen in at least one instance.
And the attitude toward program representation is heavily biased toward people who can see and visualize in the first place. Yes, that’s most people. I’m afraid my comment on the orange site had a self-righteous tone to go with that of the talk. It’s just frustrating sometimes when people so completely ignore the fact that not all of us have the same abilities, and what’s constraining to some is liberating and equalizing for others. I had the same frustration with one of Bret Victor’s talks.
Glamorous Toolkit is in beta. We should avoid judging it too harshly just because we disagree with the person who recommended it. That would be creating a cycle of dismissal and judgement.
Feenk has been aggressively advertising gtoolkit on Twitter since at least 2019. They seem to think it’s production ready.
Okay, so long as it’s a reaction to the author’s claims and not the claims of a third party. Judge people by their own actions and not how you feel about “their group”.
You’re right. I mean the FPS was almost laughable. I don’t know how hard is it going to be for GToolkit to overcome this level of performance degradation though. I would guess that a GUI toolkit written entirely in a dynamic language might have trouble being as performant as something more, hum, “dead” as the OP would say.
It’s all good! The talk was pretty dismissive and it’s hard not to take a swing at the pillars of their argument. Just want to make sure that we acknowledge that those pillars are other people who might not have as hard of takes.
I appreciate that, thank you :-). It’s easy to be dismissive on the internet.
How about Smalltalk on the original PARC machines (Alto, Dorado, etc.)? If they could do it back then, presumably we can implement a reasonably fast GUI toolkit in a dynamic language on today’s much more powerful machines.
This topic opens up the purpose of development environments. One could seriously argue that there can and should be a difference in quality during exploring code, experimenting, and checking code into the group. The goal of development environment is to speed up the developer while capturing every developer intentions along the way. This often means type checking can be ignored until after the first run.
These sort of talks are about showing alternative futures, not about presenting a complete solution. The talk should be seen to have a different aim. Not every solution has to scale. I teach high schoolers and I’d kill for a better way for my students to inspect their code. Being able to inspect state, see docs for every language construct/keyword in line, see visualizations for built-in collections. That would be tremendously useful. Maybe we can keep expanding the horizon of programs that can be worked with in this way, as hardware gets more powerful or as algorithms/languages get better.
Or even think - even a non-scalable toolset that helps you write tricky string-validation functions or regexes or SQL queries - these would be tremendously useful. We already have great tools for inspecting regexes, ofc.
I also liked the rapid-fire list of projects to look into. Plenty of jumping-off points for exploring this space.
The tone and presentation style may not be great, totally agree. But that’s a presentation layer thing.
In doesn’t have to be that bad. Common Lisp ticks many points: there are millions-of-lines codebases (or have been) (thinking about Google’s ITA Software and SISCOG, both are at least above 1 million lines). You deliver binaries smaller than your image, and commercial implementations (LispWorks, Allegro(?)) can strip out dead code.
I’m fairly certain that in some cases it does, otherwise you’re claiming that every programmer in every niche is misguided for doing things the way they’re doing it instead of your way.
Millions-of-lines codebases written in Common Lisp existing does not imply that the people developing these codebases use notebooks, interactive or visual programming. If they do, the orator should have demonstrated the gains they experience by using interactive programming on millions-of-lines pieces of software. This would have made his point stronger, as it would show that interactive programming provides gains across the entirety of the codebase-size spectrum.
I’ll add that the “millions-of-lines codebases” was just one of the dimensions I had in mind. There are other niches like FPGA-programming, real-time programming, high-throughtput programming and many others that would also need to be studied before claiming that there is One True Way of programming that is better for all programming niches.
This is already something all the big pieces of software do. It’s trivial to do with GCC (use
-ffunction-sections, and then pass--gc-sectionsto ld). The reason the binaries are still huge is because compile-time monomorphisation is used very heavily. You might argue that you don’t need to perform monomoprhisation at compile-time and delay it to run-time instead, but then you’re going to lose start-up performance, which is pretty much my point: everything is a trade-off, there is no universal answer and all you can do is work within constraints.Having worked at ITA: the Common Lisp codebase was written using normal text editors, with code checked into version control. Emacs was quite common, and there was some sort of live prompt thing where you could push code updates during development for interactive testing… SLIME maybe?
Last I saw (been many years at this point) it was deployed on SBCL.
Very interesting, thanks for sharing your experience!
Was the instance of the program running on developer’s machines, or was it running on servers? Was this instance shared among developers? How often would this instance be restarted (if at all?)? Do you know of any video that would demonstrate this workflow, if not on ITA’s codebase, on something similarly large?
I’ve never done much REPL development (aside from debugging JavaScript in the developer’s console) but have always been fascinated by the idea, I’d love to learn more :)
This was just a local instance, for debugging/live-testing mostly, is my impression. Production deployment was all the standard “compile from source code, deploy to N machines”.
It’s possible they used REPL for semi-live-development, but it all ended up in normal source code files in version control.
If you want a REPL-y dev experience, you can try out Jupyter notebooks, but keep in mind that people recommend restarting the notebook occasionally (or often!) so state doesn’t get messed up. Basically no one that I know of does pure REPL development for anything but one-off scripts, and for good reason.
I didn’t hear claims and I am not making any about One True Way. 2c.
I think the video’s orator does claim there is One True Way when he claims that compiling ahead of time is bad and that you should really go with repl-programming instead.
So, I’m kind of a CL newbie, but my understanding is that CL development is generally done semi-interactively. You write your code in a file, which will be under version control, and so on. But instead of stop-the-world, restart-with-changes, you’re normally working in a live image. As you update a function in your file, you run a command in your editor (usually Emacs) to evaluate the updated function, which replaces the existing function definition, and the same for CLOS class definitions - existing objects are updated to the new class definition. You can examine any aspect of the image’s internal state while it’s running.
The difference in CL images and Smalltalk images, as I understand it, is that in CL, your code in your editor can be out of sync with what’s loaded in the image (either because you haven’t evaluated the changes yet, or you’ve reverted to an older version and not evaluated that), whereas in Smalltalk, it can’t; the image is the only source of truth. Because I’m a n00b, I sometimes have to stop my Lisp process and restart it, loading my code from source, in order to be sure the image is in a state I understand.
Sounds 100% right to me
People with lot of experience do that too (but not me ofc) , that’s why there’s a command named
restart-inferior-lispTrue, and there are plenty of examples that could have been used instead. Fetch the Wayback Machine, Sherman!
Sabre C, renamed CodeCenter, was a popular developer tools for about $4K/seat. It included:
It was used on MLOC systems all the time.
The only reference to Saber-C I could find is here and references a last release date of 1999. I did find a video of something named Saber-Plus but it doesn’t show the live-editing capabilities you’re mentionning. Perhaps live-editing of a running program was only practical when building a compiler whose performance is considered “adequate” was still doable for a single small company? I’d imagine building an interpreter for C++’s most recent revisions, able to deal with the intricacies of the object files produces by GCC, Clang or MSVC, that conserves the performances and behavior of the compiler being emulated might be out of reach for lone actors today.
Sorry, I didn’t put a date on it. It was very popular in 1990. The company imploded around 1995, partly from competitive pressures from Sun and HP, partly from Sun and HP changing object formats quickly to cause quality issues. Between COFF and the formal specification of ELF, changes occurred which killed the ability to swap in the interpreter, sometime in 1993 IIRC. Added to that, C++ was evolving rapidly from a C-front preprocesser to its own language, and its spec changed constantly.
Oddly, it should not be out of reach. Charging $4K/seat for development tools is out of reach.
I was in the audience for the talk and found it incredibly smug and self-righteous. At one point he makes fun of a person who left early to go to the bathroom. At one point he slags Rust for not being an “interactive language”.
I tried a few of the tools he calls the real future of programming and most are little more than demos. The most developed of them had noticeable keystroke lag on my workstation desktop.
I have one big problem with this talk, and that’s… its central thesis actually: one thing the edit-compile-run cycle has going for it is that it’s principled. Namely, you’re starting from a known state (the source code) every time you compile. With live/image based programming where you fix everything on the go, the actual source code isn’t limited to the source text, but the state of the program itself! This has to be unmanageable at some point.
I will also note that some aspects of live programming are available even to C/C++ programmers: just recompile the dll and reload it. This requires significant initial setup, but it allowed more than one game dev to change mechanics on the fly without restarting the whole game, without the help of a scripting language. The Yi editor written in Haskell demonstrated something similar. XMonad can also maintain its state across compilations (which happen whenever we change its configuration).
Well said! I do love REPLs and think they are very valuable, I also prefer “principled” approaches (as you very well put it) and think there is a great deal of value there.
Combining both seems like fertile ground.
Exactly, it is about combining both. Principled use of excessive power.
Yup. There have been some threads here about “image-based” development, which I experienced with Smalltalk-80. It’s kind of magical in some ways, but also leads to real problems because your program never starts from scratch.
I am a super huge fan of Common lisp and this is something I see every single newcomer struggle with. They usually don’t struggle with it for long though. But it’s definitely something that I would like to see discussed explicitly more often.
On the bright side, Common lisp was designed with that (interactive development) in mind (mostly). What comes immediately to my mind, as an example of this, is the
defparameterv.s.defvarexpressions that only differ by what they do when they are re-evaluated. Making it easy to reload source code without losing the current state.Yet another thing: I’ve rarely seen anything that deals with “cleaning up the state”. For example, if you rename a function
f1tof2, the original functionf1still exists in the image unless you explicitly remove with (with the so well named functionfmakunbound). The only time I’ve seen something that deals explicitly with renames is terraform’s refactor, but that’s hardly for “image based development”.One last thing that I won’t develop
because this is already way longer than I wanted: the problem of updating the code within an image is the same problem as updating a database.I think you meant to analogise the database with the state, not the code…
There’s a difference though: with a database, the data is concentrated in one place, and that data is mostly statically typed: you have a fairly rigid set of tables, each column has a definite type… and if it’s all normalised like it should be 99.9…% of the time the interface between it and the code will be relatively thin, and very stable.
Now replace the database with the live state of a long lived program. A dynamically typed program at that. You need to make sure your program state is just as principled as the database’s, the interface between it and the code just as thin and stable… except here all you have is programmer discipline. There’s no clear fence, and depending how your program is accessing your data, good luck determining whether a state change will break your program or not. At least with a database you know that as long as you don’t remove columns or entire tables you’re mostly safe.
So at a first glance, image based programming at scale (let’s say more than 10 devs working on the same code base & associated state) is looks utterly impossible. How does that work in practice?
And perhaps most importantly, do you even need all of the above? Why not?
In CL: if the state becomes a mess, restart your image! No big deal, you loose 2 minutes. For many days, I can develop my program with the same running image, it’s fast, it’s interactive, I don’t loose state, I am happy. It happens indeed that I restart it. I also always run unit tests or build the project from the terminal from time to time, so from a blank test.
We use source files with git. Maybe they didn’t have this in Smalltalk land decades ago, we do now.
To share state among developers, we can build and share a custom image. It’s nearly like an executable, but it’s an image that offers a REPL. Now instead of running a new SBCL instance from our editor, we start
sbcl --core my-image, and it starts instantly.Easy: git pull, connect to the running image (you have a tmux or you use Swank, the server side of Slime, through an SSH tunnel) and you
loadyour source files again (today, it wouldquickload). The image is not restarted, so you keep state.Also if you define variables with
defvar, you don’t loose their state (versusdefparameter).But, disclaimer: one can very well update production instances without worrying keeping any state: rsync a new binary, stop the service, start the new service. Usual deployment knowledge would apply. Just as with a Go application for instance.
wow, you mean, from the state of a dev instance to the production one? Who would care to do that? I don’t see the point. I develop on my machine, I deploy new code to the production machine. I don’t have to mix states. Maybe for very specific type of applications? If I need to send data, I send the data files.
Oh, I see. So the people raging about image based programming actually don’t really care about their image. And I guess then state is not versioned either. Except of course whatever is needed to establish an initial state from scratch. I also bet the need for several people to work on the same image ranges from limited to nil.
What I thought was a completely alien realm seems actually pretty mundane all of a sudden. What I thought impossible probably is impossible… I just had the wrong idea. Kind of anticlimactic to be honest, but I think I have a better idea of what this really is now.
Thanks.
That’s a way of putting it :p For me, a young-ish lisper, image-based development is awesome for: development, and ease of live reload (if I want). Bonus, I use custom images from time to time (start a Lisp image with a ton of libraries pre-installed and with other tweaks). That’s it, and that’s a lot already.
I think it’s different between CL and Smalltalk, maybe? In CL, your image is pretty important during a given development session, but not so much between developers, or between yourself on different days, I think. The main thing is that your changes are applied to a running image without a restart, and you can build up a running program iteratively like that. If you need to get back to a particular state after resetting your image (restarting your Lisp process), you’d set it up the same way you set up fixtures for a unit test in a stop-compile-run language.
Well if some programmers do care about their image to the point they will want to version it in some way, or work on it collaboratively over a long period of time… how they do it remains a total mystery to me.
I think some Smalltalk implementations have had some kind of tooling for versioning images, but I have no experience of them.
Yes and no, but I definitely wasn’t clear xD . I had in mind that updating functions (or methods) in an image is somewhat analog to updating views in a database. But anyway, you seem to understand the problem of updating the state of a running image well enough :)
As @vindarel said, people don’t do that, but!
It’s a problem that I find interesting and think about from time to time. The solution, I think, would be something like “gating” the changes made to the running images. So, one dev could redefine a function (in a sandbox, which I have no idea how that would work exactly), if it passes the tests, then the change is applied to the “real” image. It would practically be a CI-within-a-process.
People most definitely don’t care about their dev images, but some do care about the real, always running, production image. In which case they go to great length to make sure the code that is going to be evaluated during a release don’t mess up the state, don’t cause any (foreseeable) bugs and don’t pollute the image. I think this is where my analogy with database migrations came: applying successive version of the codebase to a running image is analog to running successive migrations to a database, which brings me to:
Checks, in Common lisp, you can compile the code before actually evaluating it. SBCL (the most commonly (ha!) used implementation) will catch a lot of “type errors” (which it often reports as warnings, but you can count those warnings as errors). Also, nothing keeps you from running tests for anything you can think of.
Imagine you have an ever running python or js repl, how would you go about updating a global variable that contains all the state (to keep it simple)?
The first thing that comes to my mind is: only access the state via some specific functions (often called accessors). Update them to support the new and the old state “shape” (I think it’s what you meant by “architecture”) before updating the state.
I covered it a bit, but you kinda just have to be mindful about the changes to the state; type-checking at compile time, assertion at run-time and automated tests all helps. Also keep in mind that you can (and are very often strongly encouraged) to use functional data structures, which helps a lot when you are wary of mangling the state. BUT, bugs can happen, same as database migrations, it can go wrong even with a lot of precaution.
As vindarel said, just restart the process :P
Same as with all “mainstream” programming languages. You use git.
But, @gcupc is right about Smalltalk. Smalltalk’s source code was traditionally part of the image. Smalltalk was much more “image-based” that Common lisp on that aspect. I don’t know much more about it though… Except perhaps that I know that Pharo, a recent Smalltalk implementation, tries hard to make Smalltalk work with source files, so that people can use “traditional” command line tools on those files.
I never heard anybody doing that exactly. But the “state” would be just a file, so you could use git (or better, git-lfs) to version that file. But I don’t think that’s what you had in mind. (And sbcl is super picky about running an image from a different version, so you would be stuck with a specific version of the lisp implementation.)
I think that what you had in mind is “I have a running process, I apply some code change to it, how does the code know what’s the state’s schema right now?”. I see 2 solutions: you can do it implicitly, by using predicates that checks the types at runtime, or you can keep a “global” variable that explicitly keep a version number.
Depends what you mean. But in theory, if many collaborators have a running image that they don’t want to close for some reason, they can just
git pullthe most recent version of the codebase and apply that code to their running image. See point 3, about releasing updates without down-time.See point 3.
No, you don’t need it.
Because it’s freaking hard xD, but I think you already guessed it.
/ disclaimer: this message is way too long, I’m not gonna proof-read it just now ;)
I’ve worked on systems in statically typed languages, as well as dynamically typed languages, some with REPLs and fancy debugging tools, and some without.
As someone who loves lisps and also languages that are quite different from lisp, there’s a reason why homoiconicity, REPLs, and time-traveling debuggers didn’t take over the world, and it’s not because people are stupider than you.
There just isn’t one true model of computation. There isn’t one development/debugging workflow that’s globally better than all others. Humans are diverse and think and communicate in different ways. Some people are more visual in their thinking; others can fit a tremendous amount of state in their brains and think procedurally.
FYI, if you don’t want to sit through the video the author has a transcript available at https://jackrusher.com/strange-loop-2022/
It looked really bad on my desktop, so I tried reading it on mobile and it was even worse