1. 7

    Cool deal. It reminds me of the classic Writing That Works, written by the former CEO of Ogilvy. I found that book to be very useful. A programmer-focused would be cool. In your outline I see some things that remind me of Team Geek. Another audience to write for are potential employers. I’d like to see a section on code review and commit messages.

    Otherwise, I’ll say that writing a whole book is difficult! In terms of tools, using Markdown will get you far. I wrote a book last year and that was my tool of choice. I extended the syntax a bit to do various formatting things I needed (see mistune for a reasonably good Python markdown parser). Some people swear by asciidoc callouts.

    Good luck!

    1. 2

      Did you ever release the code that gets Python to run code embedded in Markdown? Did you use an import hook? (I haven’t tried messing with how Python loads code, beyond just calling eval/exec.)

      1. 3
        1. 1

          Thank you!

        2. 2

          No but I need to. Thanks for the nudge.

        3. 1

          Thanks for linking to those books. Looks like I should take a look at them.

          Sections on code review and commit messages are a great idea. I’ll definitely add those.

          Haha. Yes. I suspect that writing a book should be listed as one of the classic blunders.

        1. 2

          Imagine the actual interface to this problem being a person calling out “left”, “up”, “left”, etc. To run tortoise-and-hare in your head on this input, you’d have to remember what they’d called out, to an unbounded span backwards. I don’t know about ES6 iterators, but the solution seems to amount to cloning the person calling out the directions.

          1. 2

            You are absolutely correct about that, and it was no accident. I was trying to show that the Carpenter was trying to force-fit cycle-detection onto this problem, mostly because Plissken had coached him that Thing liked to test whether Candidates knew how to solve it.

            1. 1

              I see! I wondered at the time if following your blog more would have filled in some context. (I’m on a bit of an internet diet lately… relatively at least.)

          1. 1

            It’s interesting but I’m still left wondering why? Anyone care to explain that part?

            1. 2

              The cannonical answer to that question is that all the LispMs, even later hardware like the TI lisp machines was expensive performance general purpose hardware for software development specialists of the time. As funding dried up for the AI work that Lisp had become most famously associated with in the popular press the fortunes of the LispM companies also waned. At the same time the Intell 8080 and the consequent IBM PC compatible flood dominated the low end market which the LispM companies had largely ignored.

              In ‘83 the Symbolics 3600 workstation was released, MSRP $110,000 and in '84 the Apple Macintosh MSRP $2,500 ushered in what we know as the personal computing revolution. The rest is as they say history.

              Also compilers got better and memory got cheaper. As suggested in my article, not long after the fall of the LispM companies it simply stopped making financial and technological sense to invest in special purpose hardware when better software environments enabled by larger memories could efficiently compile Lisp programs to machines lacking hardware support for the various concepts of Lisp evaluation.

              1. 2

                That’s a good overview. I’m sorry I wasn’t clear in my question. Why would someone want a “LispM like system”? Maybe there’s an assumption of FP nirvana in there that I don’t understand.

                1. 4

                  This article assumes that the reader is at least curious about LispMs which are often held up as a lost FP nirvana and seeks to distroy this collective delusion while pointing out that we have the means to build equally good tools for ourselves in exsting platforms and obvious future platforms should we choose to.

                  As to why one would want such a system, the simplest answer is pervasive system wide introspection for documentation, source and user modification the various benifits of using a memory safe host language asside and neglecting the various advantages in terms of proof by construction, pattern matching, static effect analysis and other work which the FP crowd has produced since the original fall of the LispMs.

                  1. 14

                    I’m not really sure where to start here.

                    The Lisp Machines have pretty much nothing to do with a “lost FP nirvana”. Working in Common Lisp, or the Lisp Machine Lisps like ZetaLisp or Interlisp, had little to do with functional programming.

                    Lisp Machines are about an entire experience for expert developers. It isn’t just the OS. It isn’t just the hardware (compared to other hardware of the time). It isn’t just the applications that people ran on them. It was the entire environment, how it worked, how it fit together and how it was built to be a productive environment.

                    There are some ways to run Open Genera, but often, I wouldn’t recommend it. It takes serious time and investment to fully understand what’s going on and know how to use things well enough to understand the power of the whole package. I don’t find that most people that I’ve seen try to run Open Genera have actually expended an effort beyond the tinkering stage. I, myself, don’t feel like I’ve done enough work in the environment despite spending some serious time with it.

                    If you want to read what someone who has used them extensively has to say about why they were nice, I’d suggest Ergonomics of the Symbolics Lisp Machine - Reflections on the Developer Productivity from Rainer Joswig.

                    There’s still a lot to learn from what was done on the Lisp Machines. A good example of that is the whole subject of “Presentation-based UIs”.

                    I don’t think we’ll have a situation again necessarily where everything is in Lisp (or some other language) from the ground up on top of the hardware in a world where everything can be viewed, inspected, modified at run-time. Most of our developer tools today are a shadow of what they could otherwise be and that’s sad.

                    Hopefully, we’ll have more people like Rainer Joswig tell us about the good parts of the things that have been lost. Many of the people involved in these things have moved on. Many have retired. Some have died. I hope we’ll hear more of their stories before they’re lost.

                    1. 3

                      BSD and it’s philosophy achieves a little bit of that view, inspect, modify everything ideal – though at compile time – not at run time unfortunately. From a user’s perspective (which – I understand – we’re not talking about here – we’re talking about serious experts here) that would probably lead to chaos. As a developer, do you really want people messing with the internals of your program’s execution – it would add so much more complexity – you’d have to think of everyone as a developer essentially. You’d only want that if you wanted to eliminate the masses – everybody who only want to use a computer to share music, write documents, and – at most from a programming perspective – write simple UIs. And from security perspective – this wouldn’t work at all.

                      I think the reasons why a functional operating system hasn’t really caught on is because most OS development is geared towards embedded systems development. The last I worked with embedded systems they were still using semaphores and locking shared global variables for concurrency – but I don’t think they really need something to handle a lot of multi-threading – medical devices and specific device chips only need to handle interrupts from buttons and timers. A lot of the practical improvements functional programming has brought to the table has been for web applications and servers, since they have to deal with relatively huge amounts of data. These improvements I’m talking about are namely over node, golang, and nginx (possibly).

                      From a productivity and ergonomics perspective, yeah, I don’t see why their aren’t more varieties or userlands and keyboard layouts. I suppose the way we use unix and keyboards is just culturally baked into us – with only a little reason – maybe if somebody really pushed for a change it would happen, but at the moment it’s not like its broken or anything. (Touch screen keyboards that allow for custom keyboard configurations could bring back some of those keyboard ideas from the Symbolics Lisp Machine)

                      Actually now might be a really good time to build an OS with a functional programming language. “Purely” functional probably wouldn’t be practical – because it would be too slow – but say you had an OS with a functional concurrency model in place – you’d probably see a huge gain in the ability of a single server to handle a lot of threads if the threading was all handled at the OS/hardware level instead of the application. Software is cheaper than hardware – so if it was done right – there’d be a market for it. I’ve read facebook uses specialized hardware to handle their huge amounts of network traffic. Monads could also be used to deal with buffer overflows. If the x-windows systems used a more actor based concurrency model to update the window – you’d possibly see a much smoother desktop experience.

                      Just talking loose here for conversation – so if I’ve said anything ignorant - sorry – I should go read a book on operating systems design.

                      1. 7

                        BSD and it’s philosophy achieves a little bit of that view, inspect, modify everything ideal – though at compile time – not at run time unfortunately. From a user’s perspective (which – I understand – we’re not talking about here – we’re talking about serious experts here) that would probably lead to chaos. As a developer, do you really want people messing with the internals of your program’s execution – it would add so much more complexity – you’d have to think of everyone as a developer essentially. You’d only want that if you wanted to eliminate the masses – everybody who only want to use a computer to share music, write documents, and – at most from a programming perspective – write simple UIs. And from security perspective – this wouldn’t work at all.

                        I’d like to respond to this, but I feel like doing so here would be too limiting.

                        There have been end-user friendly, multi-user, programmable systems where almost anything (above a layer in C) could be modified. They even had security models for dealing with multiple people having differing privileges with respect to different parts of the code.

                        I’ll try to find the time (in the near future) to write a blog post or two about one such system that I spent years working on, using, and that I still miss.

                        1. 1

                          Is this system publicly known? If yes, how was it named?

                          1. 1

                            I don’t know what BruceM means (and would like to), but Sandstorm gives a current example of a security design that could make it work.

                            1. 1

                              It wouldn’t be that hard would it? Have the kernel keep track of who has access to code, and those that do are allowed to modify the code while its running. You don’t have to think of the end-user as a developer, you just have the ability to do so if you want. Beyond making it so that a multi-user system is stable and secure in such a system, it kinda is a none-problem.

                              1. 1

                                I think though, that in C or Lisp based system it would only be possible to have two users. Those who have access to the run-time code and those that don’t – since if a user has access to the run-time code, there’s nothing stopping him from writing code to give him higher access privileges. Could be possible with a typed system though.

                                1. 4

                                  Typing is no panacea here. A security system can offer sufficient protections to allow this all to work.

                                  The system that I’m referring to is largely gone from the net, so when I start to write about it, it’ll mostly be the only info left online about it. It was similar (in some ways) to LambdaMOO.

                        2. 5

                          LispM were much more like Smalltalk environments than a dedicated OCaml machine (as much as I’d like to have the latter). They were designed for programmers who didn’t want an impenetrable barrier between what you as a programmer do and what the system is.

                        3. 1

                          Great reply, thank you!

                1. 3

                  This is super sweet. I’m really looking forward to seeing the code.

                  Some people have claimed he only wrote a DOS shell. It’s true that he wrote a DOS shell, but it sounds like he also ran it on a clone of MS-DOS that he also wrote. So this is legit. Edit: no, wait, he meant he wrote a clone of MS-DOS COMMAND.COM in QBASIC.

                  Discussion elsewhere: PC gaming subreddit (234 comments), Australia subreddit (150 comments), Geek subreddit (34 comments), osdev subreddit (5 comments), cyberpunk subreddit (59 comments), HN (208 comments).

                  1. 2

                    What is an Operating System though, really, anyway? I mean, Microsoft have gotten away with “a new shell on DOS” for a few iterations of their product, and ultimately .. to the Consumer .. the line is irrelevant. To a developer, we can move the line in the sand all over the tarpit, but the answer is never so black and white as to what constitutes an Operating System versus what is just a set of loosely-couple applications sharing a common core, integrated in some manner.

                    To the user the question of “OS” versus “App” is .. fundamentally .. totally irrelevant.

                    To the developer, its as simple as this: whatever the User first boots when they turn on the machine, is the Operating System. Whatever they do with it repetitively, is their Application.

                    I think he built a set of both, and its a genius and beautiful, if a little sad, story. Haven’t we all felt compelled to push the limit in the face of oppression? This is an expression of that.

                    1. 2

                      An operating system is the system that

                      • loads one program after another and sets up the I/O devices for each of them, and optionally
                      • context-switches between them to hide I/O latency and virtualizes disk and RAM so they don’t stomp on each other, and
                      • maybe keeps accounting records so you can bill users for CPU time,

                      so that

                      • you don’t have to hire a computer operator to do setup and teardown for each job and load each new card deck by hand, and so that
                      • you don’t have expensive downtime during manual setup and teardown.

                      That’s why it’s called an operating system: it’s a (software) system that automates away the computer operator’s job of operating the computer, by loading jobs into it and arbitrating between the jobs' conflicting demands on computer resources.

                      Thus ends today’s history lesson.

                      And yeah, there’s a lot of flexibility to move functionality between the operating system and the programs it runs. But I’d say that at a minimum an “operating system” has to load more than one other program into memory and run them.

                      I agree about the genius, beautiful, sad part. He could have done so much more if only he’d known…

                      1. 1

                        I like to think of an OS as a computer designed to be shared among programs (where ‘computer’ includes virtual machines broadly defined). It’s not exactly the historical meaning – I actually wanted to sort of jettison some of the history.

                  1. 5

                    I’m an advocate for TDD. But, it depends on the situation.

                    TDD is great at providing almost instantaneous feedback. When I first started programming (Java) I worked at a company where we had no tests, our codebase was set up in a way that did not accommodate tests, and it led to an extremely slow feedback loop. I remember times where I would make changes, spend many minutes rebuilding the project, testing it manually, and then hit a problem that I hadn’t anticipated and have to go through the whole thing again.

                    Later I moved on to another job, still Java but where things were TDD’d. It shortened the feedback loop by many multiples and provided a huge improvement in productivity. This is the “faster” that TDD proponents talk about.

                    More later, I moved away from Java into FP languages, and particularly REPL driven development. This is the ultimate feedback loop. If I’m ever unsure about something I can eval it and know right away. But, this led me to writing less tests. Right now I’m working on capturing my REPL experiments as tests, because…

                    TDD isn’t just about the feedback loop, the tests also leave “contracts” for future people working on the same code. When I encounter a large codebase that is covered in tests I have the confidence to go in and make changes, when there are no tests I don’t have that same confidence. Of course, passing tests doesn’t always equal everything-is-fine, but usually it does, and that’s a great thing.

                    1. 1

                      About capturing REPL experiments as tests: I use https://github.com/darius/halp – it’s for Emacs and (mainly) Python. It’s actually a tighter feedback loop than a REPL gave me, plus persistent by default.

                    1. 1

                      How about extractions or removals? Unless I were writing in a context where zippers find further use, I’d call it that:

                      def extractions(xs): 
                          for i, x in enumerate(xs):
                              yield x, xs[:i]+xs[i+1:]
                      
                      1. 4

                        I wonder if you could keep memory from exhausting by using a set comprehension with a single value..

                        {1 for nonsense in map(lambda c:
                              c[0].sendall(c[0].recv(1024)) and c[0].close(),
                              (s.accept() for _ in itertools.count(1)))}
                        
                        1. 1

                          Ooooh! That’s a fancy, fancy idea. Can make it shorter by just doing set(map(lambda c: ... ))

                          1. 3

                            Or (untested):

                            {cs.sendall(cs.recv(1024)) and cs.close()
                             for _ in itertools.count()
                             for cs,addr in [s.accept()]}
                            

                            I’ve been known to use the ... for foo in [bar] construction in real code occasionally.

                        1. 6

                          Taking something by a master and not reading their code until after a whack at the problem myself. http://norvig.com/ is a good source. Another was Thompson’s regex search paper – I did read it first, but the IBM 709 code was obscure enough to avoid spoilers. I’ve put this one up as a kind of problem set: https://github.com/darius/regexercise

                          If it’s a problem you’re interested in, you can’t really lose: you either get schooled or you find a wrinkle that your chosen master didn’t.

                          1. 4

                            I worked through Little Schemer like this. I would cover their answers and try on my own, then check, compare etc. It was a great experience: humbling in a good way.

                            Another book this works well for is K&R on C. There are tons of solutions to their exercises on Github, Bitbucket and personal websites. Try to solve first, then read around. Problem there, of course, is that the solutions are off very varied quality, and you don’t know in advance which are good, great, meh, terrible. Of course, you can think of that as training for reading real code: You never know what you’ll get.

                            1. 2

                              There’s an old book, The C Answer Book, for just this. (I haven’t read it.)