Yes, yes, yes! When I did literate programming, I realized that the real value was not the good-looking PDF, but rather the narrative for understanding the solution (though I must admit I sometimes missed the mark and the text helped not at all). I don’t know about other people, but I typically find it difficult to jump into a code base, read it, and get a good sense of why things are the way they are, e.g., choice of data structures and algorithms, seemingly premature extension points, etc. It’s doubly true with business logic: a complex piece of code might exist only because a client 5 years ago needed that feature and it was never phased out or because of an obscure law that only exists in the province of Quebec.
One question I’ve been pondering lately is how to encourage good story telling in software. I agree with the article that git history is not it. I thought literate programming might be a way, but currently there are difficulties with the existing tools that make LP unpractical—for example, most editors don’t deal well with the mix of LaTeX and the unordered chunks that only make sense when tangled together. In addition, I found that it was more difficult to “refactor” prose than code, so I was more reluctant to make changes in my software because I had no mechanical way—either tests or a type checker—to check that I updated everything relevant.
@akkartik’s layered software approach is interesting: each individual layer could have a README file that describes why it exists. I haven’t tried it myself (maybe I should for this year’s Advent of Code in December?), so I can’t comment on how it is in practice, but I certainly like the idea and love that someone is pushing in that direction.
If people have good tips for including a narrative in a software project, I’d love to hear them. I even wonder if including a video of the author drawing a diagram might be a good idea.
I’d love to hear about your experiences if you try this out. Layers rely on tests to gain confidence that you caught all the places that need to be changed. They’re harder to refactor, yes, but I’d say they’re just exposing you to the full difficulty of the refactoring (explaining rationale) rather than letting you shove it under a carpet somewhere like conventional approaches do. They make you face up to the essential difficulty of the operation.
Since each layer is a single file, I think of the comment at the top as the README for the layer. That’s one of the surprising benefits I found from using layers: they provide a Schelling Point where readers can find the big-picture rationale for a feature. Conventional organizations often scatter the code for a feature too much to admit a strong Schelling Point. At work I often find the most useful comment about a feature in some header somewhere after weeks of wrong turns trying to understand it. In contrast, here are afewlayers with useful comments up top (copying from a recent comment).
I had always wondered why this wasn’t done at all when I first started programming. Context is for kings and we containerise everything BUT context. Thanks for linking that subreddit!
Literate programming is a very good idea that is unfortunately made impractical by our existing tools. Most technical books don’t contain just text. They contain formulas, diagrams, pictures and whatever it might take to get the important ideas across. Literate programs ought to contain formulas, diagrams and pictures too, if those happen to be the easiest way to explain their inner workings.
As much as I love LaTeX for typesetting mathematics, it is impractical for annotating programs the way I want to. I design my programs using paper and pen, which gives me the freedom to use whatever notation I want, without the inconvenience of explicitly importing such and such LaTeX package just to support the notation. Ideally, my program annotations would be (selected excerpts from) the design papers themselves. Maybe I’m willing to put some effort to redraw the diagrams more cleanly. But I’m not willing to put the effort to typeset those diagrams.
I’ll be interested to see these programs you’re writing today, and the diagrams you want to include with them that are hand-drawn cleanly but not typeset. Are you sure the diagrams are actually useful? To anybody but your self?
To summarize my argument: maybe diagrams would be useful to include with software. However:
Literate Programming would be plenty useful without them as well, so it’s not worth blocking on integrating diagrams with IDEs before we start using it, and
Diagrams will always be a lot of work. Think about blueprints in other fields today. Anything not typeset that you’re coming up with is likely to be more of a liability than an asset.
Are you sure the diagrams are actually useful? To anybody but yourself?
Completely sure. For a mundane example, if you are implementing self-balancing search trees, the easiest way to convince yourself that you have considered all possible tree rotations is to draw them all (up to symmetry).
Writing actually went through a lengthy period when every writer had his own unique-snowflake style of writing. Outgrowing this phase was a major advance.
I’m talking about expressing ideas using formulas and drawings, not about using this or that font.
Diagrams will always be a lot of work.
Diagrams can be drawn almost instantaneously using paper and pen. Of course, using LaTeX, they require a lot more work.
Ok, I understand better what sort of diagrams you mean. But the slight change in cost of diagramming based on the tools you choose is irrelevant if the software you write will be read many many times. At its root, Literate Programming is about making that aspirational assumption, that software will be read orders of magnitude more times than it’ll be written. It’s not true today, but I’d like to live in a world where it is true. And that’ll never happen without first making things easy to read. It’s a chicken-and-egg problem.
It’s fine if you’re not on board with this agenda. But this is why Literate Programming is as it is. It does not care that diagrams take a little extra effort. You just have incommensurate goals, both equally ‘impractical’ by today’s standards.
And that’ll never happen without first making things easy to read. (…) It’s fine if you’re not on board with this agenda.
I’m certainly on board. However, typesetting is neither necessary nor sufficient for documentation to be easy to read. What is actually needed is clarity of ideas and clarity of exposition.
Absolutely, 110% agreed! I think I understand your position now. You want to prioritize convenient diagramming support before typesetting. I’m totally down with that. But it seems really hard given that competing with pen and paper – while also quickly bouncing to and from a keyboard – will require significant hardware support.
Given that all our programs today don’t rely on non-text media (except for the odd bit of ASCII art), I think this new constraint you’re introducing is actually more impractical than the limitation you’re complaining about. Pure-text literate programs (with zero LaTeX or pdf) would be strictly superior to the current status quo.
Yes, yes, yes! When I did literate programming, I realized that the real value was not the good-looking PDF, but rather the narrative for understanding the solution (though I must admit I sometimes missed the mark and the text helped not at all). I don’t know about other people, but I typically find it difficult to jump into a code base, read it, and get a good sense of why things are the way they are, e.g., choice of data structures and algorithms, seemingly premature extension points, etc. It’s doubly true with business logic: a complex piece of code might exist only because a client 5 years ago needed that feature and it was never phased out or because of an obscure law that only exists in the province of Quebec.
One question I’ve been pondering lately is how to encourage good story telling in software. I agree with the article that git history is not it. I thought literate programming might be a way, but currently there are difficulties with the existing tools that make LP unpractical—for example, most editors don’t deal well with the mix of LaTeX and the unordered chunks that only make sense when tangled together. In addition, I found that it was more difficult to “refactor” prose than code, so I was more reluctant to make changes in my software because I had no mechanical way—either tests or a type checker—to check that I updated everything relevant.
@akkartik’s layered software approach is interesting: each individual layer could have a README file that describes why it exists. I haven’t tried it myself (maybe I should for this year’s Advent of Code in December?), so I can’t comment on how it is in practice, but I certainly like the idea and love that someone is pushing in that direction.
If people have good tips for including a narrative in a software project, I’d love to hear them. I even wonder if including a video of the author drawing a diagram might be a good idea.
I’d love to hear about your experiences if you try this out. Layers rely on tests to gain confidence that you caught all the places that need to be changed. They’re harder to refactor, yes, but I’d say they’re just exposing you to the full difficulty of the refactoring (explaining rationale) rather than letting you shove it under a carpet somewhere like conventional approaches do. They make you face up to the essential difficulty of the operation.
Since each layer is a single file, I think of the comment at the top as the README for the layer. That’s one of the surprising benefits I found from using layers: they provide a Schelling Point where readers can find the big-picture rationale for a feature. Conventional organizations often scatter the code for a feature too much to admit a strong Schelling Point. At work I often find the most useful comment about a feature in some header somewhere after weeks of wrong turns trying to understand it. In contrast, here are a few layers with useful comments up top (copying from a recent comment).
@smalina and I have been discussing this story quasi-privately at https://www.reddit.com/r/BarbarianProgramming/comments/74coqq/software_is_about_storytelling.
I’ll also be interested to hear your take on my critique of classic LP.
I had always wondered why this wasn’t done at all when I first started programming. Context is for kings and we containerise everything BUT context. Thanks for linking that subreddit!
Slight tangent: @litonico_ had a cool strangeloop talk on context.
Interesting. That actually reminds me of a talk from a previous Strange Loop. Submitted.
Edit: still watching your talk. It’s a really nice format, stringing pre-recorded text editor sessions together.
Literate programming is a very good idea that is unfortunately made impractical by our existing tools. Most technical books don’t contain just text. They contain formulas, diagrams, pictures and whatever it might take to get the important ideas across. Literate programs ought to contain formulas, diagrams and pictures too, if those happen to be the easiest way to explain their inner workings.
That’s why literate programs are often in LaTeX :)
As much as I love LaTeX for typesetting mathematics, it is impractical for annotating programs the way I want to. I design my programs using paper and pen, which gives me the freedom to use whatever notation I want, without the inconvenience of explicitly importing such and such LaTeX package just to support the notation. Ideally, my program annotations would be (selected excerpts from) the design papers themselves. Maybe I’m willing to put some effort to redraw the diagrams more cleanly. But I’m not willing to put the effort to typeset those diagrams.
I’ll be interested to see these programs you’re writing today, and the diagrams you want to include with them that are hand-drawn cleanly but not typeset. Are you sure the diagrams are actually useful? To anybody but your self?
Writing actually went through a lengthy period when every writer had his own unique-snowflake style of writing. Outgrowing this phase was a major advance.
To summarize my argument: maybe diagrams would be useful to include with software. However:
Literate Programming would be plenty useful without them as well, so it’s not worth blocking on integrating diagrams with IDEs before we start using it, and
Diagrams will always be a lot of work. Think about blueprints in other fields today. Anything not typeset that you’re coming up with is likely to be more of a liability than an asset.
Completely sure. For a mundane example, if you are implementing self-balancing search trees, the easiest way to convince yourself that you have considered all possible tree rotations is to draw them all (up to symmetry).
I’m talking about expressing ideas using formulas and drawings, not about using this or that font.
Diagrams can be drawn almost instantaneously using paper and pen. Of course, using LaTeX, they require a lot more work.
Ok, I understand better what sort of diagrams you mean. But the slight change in cost of diagramming based on the tools you choose is irrelevant if the software you write will be read many many times. At its root, Literate Programming is about making that aspirational assumption, that software will be read orders of magnitude more times than it’ll be written. It’s not true today, but I’d like to live in a world where it is true. And that’ll never happen without first making things easy to read. It’s a chicken-and-egg problem.
It’s fine if you’re not on board with this agenda. But this is why Literate Programming is as it is. It does not care that diagrams take a little extra effort. You just have incommensurate goals, both equally ‘impractical’ by today’s standards.
I’m certainly on board. However, typesetting is neither necessary nor sufficient for documentation to be easy to read. What is actually needed is clarity of ideas and clarity of exposition.
Absolutely, 110% agreed! I think I understand your position now. You want to prioritize convenient diagramming support before typesetting. I’m totally down with that. But it seems really hard given that competing with pen and paper – while also quickly bouncing to and from a keyboard – will require significant hardware support.
Given that all our programs today don’t rely on non-text media (except for the odd bit of ASCII art), I think this new constraint you’re introducing is actually more impractical than the limitation you’re complaining about. Pure-text literate programs (with zero LaTeX or pdf) would be strictly superior to the current status quo.