Neat! I think being able to get a start on random ideas I have while I’m standing around and waiting with my phone might well be the thing that’s spoken to me most about all of this.
Absurdly, the first step is getting ChatGPT in the right “mood”
That was more of a joke about how weird these things are to work with - but in this case, I think it’s because the fine-tuning data they used to teach it how to use the Code Interpreter tool were all examples of Python, so asking it to compile C code is more likely to result in it claiming that it can’t do it.
But… it can do it, if it saves a C file on disk and then uses the Python subprocess.call() function to run the “gcc” binary against it. I know this from previous experience, so I need to get it to a state where it doesn’t claim NOT to be able to do it.
I love this idea of enabling these “sidequests” that you might not be able to progress otherwise. My limited attempts to use LLMs so far haven’t been so helpful for this though - I’ve found they work well in cases where I can instantly tell if the result is good. e.g. I don’t write much Python these days, so when I do it’s pretty handy to get snippets to jog my memory via and aichat terminal window vs looking up generic documentation or cheatsheets.
I also wanted to highlight the very end of the post, which I think is a nice reflection on the current value and limitations of LLMs for coding:
There are many legitimate criticisms of LLMs. The copyright issues involved in their training, their enormous power consumption and the risks of people trusting them when they shouldn’t (considering both accuracy and bias) are three that I think about a lot.
The one criticism I wont accept is that they aren’t useful.
One of the greatest misconceptions concerning LLMs is the idea that they are easy to use. They really aren’t: getting great results out of them requires a great deal of experience and hard-fought intuition, combined with deep domain knowledge of the problem you are applying them to.
I use these things every day. They help me take on much more interesting and ambitious problems than I could otherwise. I would miss them terribly if they were no longer available to me.
Would you be willing to share an estimate of how much you spend per month for your normal usage of Claude and GPT-4?
One thing that’s held me back from looking at Claude is my inability to map their pricing to what I might expect to use, and, since you seem to be doing useful things with these models, your cost would be a useful datapoint if you don’t mind sharing it.
I’ve really enjoyed your posts about how you use these tools.
My Claude usage is currently free (weirdly) because it fits in their API “trial” period, which I had to waitlist for.
I spend money on the OpenAI API as well, but that usually comes in at $10 or so a month and I think I’m still working through free credits I got for attending their conference back in November.
This stuff is generally SO cheap for personal use. It gets expensive when you’re running it at scale - shipping user-facing features, running bulk prompts against thousands of documents - but for personal interactive prompting the cost rarely even registers.
I just wrote up a much more ambitious sidequest that I ran this morning: getting ChatGPT Code Interpreter to write a SQLite extension in C, then compile it, load it into Python and test it out: https://simonwillison.net/2024/Mar/23/building-c-extensions-for-sqlite-with-chatgpt-code-interpreter/
Neat! I think being able to get a start on random ideas I have while I’m standing around and waiting with my phone might well be the thing that’s spoken to me most about all of this.
Do you have any idea why that is?
That was more of a joke about how weird these things are to work with - but in this case, I think it’s because the fine-tuning data they used to teach it how to use the Code Interpreter tool were all examples of Python, so asking it to compile C code is more likely to result in it claiming that it can’t do it.
But… it can do it, if it saves a C file on disk and then uses the Python subprocess.call() function to run the “gcc” binary against it. I know this from previous experience, so I need to get it to a state where it doesn’t claim NOT to be able to do it.
I love this idea of enabling these “sidequests” that you might not be able to progress otherwise. My limited attempts to use LLMs so far haven’t been so helpful for this though - I’ve found they work well in cases where I can instantly tell if the result is good. e.g. I don’t write much Python these days, so when I do it’s pretty handy to get snippets to jog my memory via and aichat terminal window vs looking up generic documentation or cheatsheets.
I also wanted to highlight the very end of the post, which I think is a nice reflection on the current value and limitations of LLMs for coding:
Would you be willing to share an estimate of how much you spend per month for your normal usage of Claude and GPT-4?
One thing that’s held me back from looking at Claude is my inability to map their pricing to what I might expect to use, and, since you seem to be doing useful things with these models, your cost would be a useful datapoint if you don’t mind sharing it.
I’ve really enjoyed your posts about how you use these tools.
I spend $20/month on ChatGPT for GPT-4.
My Claude usage is currently free (weirdly) because it fits in their API “trial” period, which I had to waitlist for.
I spend money on the OpenAI API as well, but that usually comes in at $10 or so a month and I think I’m still working through free credits I got for attending their conference back in November.
Thanks! That’s much more modest than I’d have expected based on how much you seem to be doing.
This stuff is generally SO cheap for personal use. It gets expensive when you’re running it at scale - shipping user-facing features, running bulk prompts against thousands of documents - but for personal interactive prompting the cost rarely even registers.