Somewhat tangent: ChatGPT-assisted development is a very cool reminder of how things aren’t equally popular.
Like most people I’ve tried to reproduce the most common examples of “look what I got ChatGPT to do!” because I was somewhat incredulous at first. And, yep, I could probably steer it through developing, say, a chatbot that fetches flightradar24 information to answer questions like “how many Boeing planes are flying over Denver right now” or something.
On the other hand, the one time I tried to get it to do something that was actually useful for me, and write me a lousy program that generated a PCAP file with a million Modbus TCP packets, the results were… nowhere near the same level.
First it insisted it can’t give me a million packets because Modbus is an industrial protocol and it should be used responsibly, and something something ethical obligations. It gladly gave me code that generated ten packets though, a truly masterful defense-in-depth scheme that would foil the evil plans of all but the l33t3st of h4x0rz.
Sadly, the code was useless because it pretty much hallucinated about Scapy’s API. I’m not talking mixing up argument functions but straight out “these functions don’t exist”. I figured, well, maybe they did in an older version. They didn’t: they’re in PyModbus, a completely different library (which does something pretty different than what Scapy does though). Needless to say, they don’t really mix.
I feel for ChatGPT: both are rather scarcely documented and pretty obscure.
The lesson for me isn’t “ChatGPT sucks and it’ll never eat programmers’ lunch” – it doesn’t and it may. What I want to point out is that, for code-generating LLMs, code triviality is measured in training data availability, not program function.
There’s quite a lot of depth to learning to use these tools for effective programming assistance.
I’m getting really productive with them because I’ve started to accumulate a bunch of different tricks, for example:
I reach for them any time I’m confident my problem can be solved using languages and APIs that were well documented before the September 2021 cutoff date. JavaScript stuff using default browser DOM APIs is a particularly good target here. I’ll sometimes tip it to use libraries I expect it will know well, like numpy or Pandas.
If I spot that it doesn’t know a library, I’ll either write that code myself or I’ll paste in just enough example code (maybe copied from a tutorial) that it “learns” enough of that library to solve my problem.
I effectively treat it as coding intern: I expect that I’ll need to review its code and provide feedback over a few rounds to get the result I need, using prompts in place of pull request comments.
The impactful difference between it and an actual human developer is that it can apply my feedback in mere seconds.
If I spot that it doesn’t know a library, I’ll either write that code myself or I’ll paste in just enough example code (maybe copied from a tutorial) that it “learns” enough of that library to solve my problem.
I might try that at some point but so far that has… kind of defeated the purpose of using it in the first place.
The reason I reached for ChatGPT in this case is that I needed a bunch of test packets for a non-trivial protocol that I know very well in order to test the performance of a packet sniffer I’m writing (hence “get me 1 million packets”). I’m very familiar with the Modbus protocol but this being 2023, I get to touch it maybe once a year, and every implementation of it is filed under “I’ve used this in the last five years or so but that’s about it”. I.e. I can’t write code without referencing the docs and maybe re-reading the introductory section just to remind myself how you hold it.
It would’ve taken me maybe ten minutes to write that code myself, but it was boring, unrewarding work that distracted me from what I actually wanted to do.
Teaching ChatGPT to use ScaPy properly sounds like fun but it takes about an order of magnitude longer than it takes me to write the stupid code myself. At that point it’s a major productivity drain, and that’s assuming I can use the program it outputs as-is, which is usually not the case.
Humans have a pretty small “context window” and a large part of programming is introducing structure so that move things can fit through that window. If you’ve seen map(f, items) before you can just represent it as one mental token instead of needing to use all the token of [f(i) for i in items] or worse whatever that would be in assembly. Using an AI that can keep track of all this junk for us will let us write “worse” code that will have other benefits, like the potential for optimization and removal of dependencies. Remember John Carmack’s advice to inline all functions? That’s bad from a programmer’s point of view, but if I can just get an AI to deal with all that crap, maybe it’s worth it in more circumstances than it was before.
In my understanding, the “inline your functions” idea from Carmack was meant for readability and not performance – it being easier to read and understand code in one place rather than having to jump around to understand all the functions and abstractions. I kind of vibe with it and tend to keep logic that’s still all “tentative” from being broken up into pieces as much possible, letting the pieces emerge rather than trying to design too hard upfront. It does depend on having APIs you’re working over that are at a decent abstraction level too.
I tend to invest the early design / modularity effort more into the data structures than the functions. It’s possible to push the code in the functions around later if the lang is good for refactoring (some make it more of a chore).
Remember John Carmack’s advice to inline all functions?
I don’t, but it’s very bad advice with a vaguely modern compiler (my guess is Carmack said it in the ‘90s). Always-inline attributes exist (and work) now and so you can structure your code for humans to read, with clear abstraction boundaries, but still have all of the optimisation benefits of writing everything in a single function.
I’ve been doing the same thing! ChatGPT is really exciting.
There’s no API for this [chatgpt] at the moment, but if you open up the browser DevTools and watch the network tab you can see it retrieving JSON any time you navigate to an older conversation.
I’d still recommend using the API, because you can also tweak the system prompt, which makes it easier to control how data is formatted. Instead of “how do I do XYZ? Provide python code”, you chat “how do I do XYZ” and put “you only output python code” in the system prompt.
(Also the playground has a “download history as JSON/CSV” option. Maybe you could script against that somehow?)
Nice post! This helped me understand what ChatGPT is good for
I do think the “helping experienced programmers” angle is much more helpful, and probably more accurate, than the “AI is taking our jobs” angle. The former is harder to understand, but the latter attracts clicks.
FWIW I have been thinking about what everyone else has thinking about, and with respect to programming, I feel like LLMs have a lot of similarities to the web and Google in ~2000.
I wrote a little comment here about how I read the MS-DOS manual from cover to cover as a teenager, and I installed a Borland C++ compiler from like 20 floppy disks … and yet I still had problems making anything interesting. I would get stuck on problems and abandon the program.
The web, Google, formal education, and time/experience helped me get a lot better. If I had to look up code snippets in a shelf full of manuals, the latter 2 might not have helped enough.
So my hypotheses:
LLMs will help experienced programmers write code faster, and take on bigger projects, as this post mentions
Less experienced programmers will also be able to write code faster. My feeling is that it will be hard to build bigger systems, because they’ll still be tripped up by “I have to read and understand the code that the LLM told me to add”, as this post shows
This is analogous to the fact that it doesn’t appear you can build huge programs purely with text from StackOverflow queries. You have to add your own knowledge, and form a theory about the code to be able to build on top of it !!!
LLMs will generate extraordinary wealth for the companies that build them, and also for society in general.
There will also be big downsides, just as with Google
But I don’t believe in blind extrapolation from current capabilities to AGI, i.e. just like blind extrapolation with self-driving cars misled people in 2016 to think their children wouldn’t need driver’s licenses in 2021.
As I mentioned in the comment, people postulated whether “Google was conscious” in 2003. It scaled exponentially for quite some time. But it clearly hit a limit, I would say a full decade ago, or even more.
There are obvious unsolved problems with LLMs (Moravec’s paradox, etc.). They are definitely not AGI, and if we get to AGI, I’d suspect they and their successors are only a small part of the solution. (I’m not an expert, just speculating mostly)
So anyway, I think if experienced programmers introspect, there are definitely things they build NOW, with the help of search queries, that they would absolutely not build if they had a shelf of paper manuals to sort through. (I’m guessing >50% of the people on this site don’t remember those days. I remember paper manuals, but I never worked profesionally when paper manuals were a thing.)
My guess is in that 5-10 years, most experienced programmers will be grateful for LLMs, just as we’re grateful for StackOverflow and search engines (leaving aside the recent decline of some :-P )
Somewhat tangent: ChatGPT-assisted development is a very cool reminder of how things aren’t equally popular.
Like most people I’ve tried to reproduce the most common examples of “look what I got ChatGPT to do!” because I was somewhat incredulous at first. And, yep, I could probably steer it through developing, say, a chatbot that fetches flightradar24 information to answer questions like “how many Boeing planes are flying over Denver right now” or something.
On the other hand, the one time I tried to get it to do something that was actually useful for me, and write me a lousy program that generated a PCAP file with a million Modbus TCP packets, the results were… nowhere near the same level.
First it insisted it can’t give me a million packets because Modbus is an industrial protocol and it should be used responsibly, and something something ethical obligations. It gladly gave me code that generated ten packets though, a truly masterful defense-in-depth scheme that would foil the evil plans of all but the l33t3st of h4x0rz.
Sadly, the code was useless because it pretty much hallucinated about Scapy’s API. I’m not talking mixing up argument functions but straight out “these functions don’t exist”. I figured, well, maybe they did in an older version. They didn’t: they’re in PyModbus, a completely different library (which does something pretty different than what Scapy does though). Needless to say, they don’t really mix.
I feel for ChatGPT: both are rather scarcely documented and pretty obscure.
The lesson for me isn’t “ChatGPT sucks and it’ll never eat programmers’ lunch” – it doesn’t and it may. What I want to point out is that, for code-generating LLMs, code triviality is measured in training data availability, not program function.
There’s quite a lot of depth to learning to use these tools for effective programming assistance.
I’m getting really productive with them because I’ve started to accumulate a bunch of different tricks, for example:
I effectively treat it as coding intern: I expect that I’ll need to review its code and provide feedback over a few rounds to get the result I need, using prompts in place of pull request comments.
The impactful difference between it and an actual human developer is that it can apply my feedback in mere seconds.
I might try that at some point but so far that has… kind of defeated the purpose of using it in the first place.
The reason I reached for ChatGPT in this case is that I needed a bunch of test packets for a non-trivial protocol that I know very well in order to test the performance of a packet sniffer I’m writing (hence “get me 1 million packets”). I’m very familiar with the Modbus protocol but this being 2023, I get to touch it maybe once a year, and every implementation of it is filed under “I’ve used this in the last five years or so but that’s about it”. I.e. I can’t write code without referencing the docs and maybe re-reading the introductory section just to remind myself how you hold it.
It would’ve taken me maybe ten minutes to write that code myself, but it was boring, unrewarding work that distracted me from what I actually wanted to do.
Teaching ChatGPT to use ScaPy properly sounds like fun but it takes about an order of magnitude longer than it takes me to write the stupid code myself. At that point it’s a major productivity drain, and that’s assuming I can use the program it outputs as-is, which is usually not the case.
Humans have a pretty small “context window” and a large part of programming is introducing structure so that move things can fit through that window. If you’ve seen map(f, items) before you can just represent it as one mental token instead of needing to use all the token of [f(i) for i in items] or worse whatever that would be in assembly. Using an AI that can keep track of all this junk for us will let us write “worse” code that will have other benefits, like the potential for optimization and removal of dependencies. Remember John Carmack’s advice to inline all functions? That’s bad from a programmer’s point of view, but if I can just get an AI to deal with all that crap, maybe it’s worth it in more circumstances than it was before.
In my understanding, the “inline your functions” idea from Carmack was meant for readability and not performance – it being easier to read and understand code in one place rather than having to jump around to understand all the functions and abstractions. I kind of vibe with it and tend to keep logic that’s still all “tentative” from being broken up into pieces as much possible, letting the pieces emerge rather than trying to design too hard upfront. It does depend on having APIs you’re working over that are at a decent abstraction level too.
I tend to invest the early design / modularity effort more into the data structures than the functions. It’s possible to push the code in the functions around later if the lang is good for refactoring (some make it more of a chore).
Maybe Carmack is an AI?
I don’t, but it’s very bad advice with a vaguely modern compiler (my guess is Carmack said it in the ‘90s). Always-inline attributes exist (and work) now and so you can structure your code for humans to read, with clear abstraction boundaries, but still have all of the optimisation benefits of writing everything in a single function.
The advice mentioned above. It’s from 2007.
It wasn’t for performance
I’ve been doing the same thing! ChatGPT is really exciting.
Yes there is :P
I meant an API for retrieving my saved conversations, not creating new ones. I just updated that paragraph to clarify.
I’d still recommend using the API, because you can also tweak the system prompt, which makes it easier to control how data is formatted. Instead of “how do I do XYZ? Provide python code”, you chat “how do I do XYZ” and put “you only output python code” in the system prompt.
(Also the playground has a “download history as JSON/CSV” option. Maybe you could script against that somehow?)
I love that download history link! Hadn’t spotted that before.
I just used GPT4 to design a Tasker task I’ve wanted for months. God I love this thing
Nice post! This helped me understand what ChatGPT is good for
I do think the “helping experienced programmers” angle is much more helpful, and probably more accurate, than the “AI is taking our jobs” angle. The former is harder to understand, but the latter attracts clicks.
FWIW I have been thinking about what everyone else has thinking about, and with respect to programming, I feel like LLMs have a lot of similarities to the web and Google in ~2000.
I wrote a little comment here about how I read the MS-DOS manual from cover to cover as a teenager, and I installed a Borland C++ compiler from like 20 floppy disks … and yet I still had problems making anything interesting. I would get stuck on problems and abandon the program.
https://news.ycombinator.com/item?id=35308305
The web, Google, formal education, and time/experience helped me get a lot better. If I had to look up code snippets in a shelf full of manuals, the latter 2 might not have helped enough.
So my hypotheses:
There are obvious unsolved problems with LLMs (Moravec’s paradox, etc.). They are definitely not AGI, and if we get to AGI, I’d suspect they and their successors are only a small part of the solution. (I’m not an expert, just speculating mostly)
So anyway, I think if experienced programmers introspect, there are definitely things they build NOW, with the help of search queries, that they would absolutely not build if they had a shelf of paper manuals to sort through. (I’m guessing >50% of the people on this site don’t remember those days. I remember paper manuals, but I never worked profesionally when paper manuals were a thing.)
My guess is in that 5-10 years, most experienced programmers will be grateful for LLMs, just as we’re grateful for StackOverflow and search engines (leaving aside the recent decline of some :-P )