I’d like to see the threat model articulated a bit more clearly here. You are permitting a user to run arbitrary queries. Whether they use an LLM to prepare them or not, they can run whatever queries they’re authorised to do. If your threat model is the user using the LLM to generate malicious queries then you need to provide some kind of access control that restricts what the user can do.
Generally, when people talk about LLM security in this kind of context, they care about prompt injection or hallucination. In these scenarios, the users asks for a solution for task X, but the LLM consumes their input and either combines it with something provided by an attacker or hits a less-explored area of the similarity space and generates something that is not a solution for task X and may be actively malicious. If this is what you care about, you generally want one of two things:
A way of running the query without it being able to do any damage (or, at least, a way of undoing the damage).
A way for the user to validate what the query will do.
The problem here is not that the user can do a bad thing but that the UI will accidentally do a bad thing independent of their intention. The read-only database proposal in the article is a big hammer here: the query can tell you anything but can’t modify any data. That severely restricts the utility but eliminates much of the risk. It doesn’t, for example, prevent the user from being given bogus data. For example, if the user asks whether something is safe and a prompt injection or hallucination tells the user ‘yes’, they haven’t modified the database but they might be about to die.
If you want these queries to be able to update the database, you really want a capability system. I don’t know if there are any capability extensions to SQL, but they’re the best way of expressing the principle of intentionality: you can modify a part of the database only if you intended to modify it and you provide the LLM query with the set of capabilities that it should have to accomplish the task.
The last solution, “constraint validator” is the capability system that you describe. It’s what HeimdaLLM provides. Basically you can define what tables, columns, joins, functions, etc are allowed to be used and how they’re allowed to be used. When HeimdaLLM processes an untrusted query, it validates those constraints. It does this by using a real grammar and parser.
I’ve been investigating this problem for a few years now with my https://datasette.io/ project, which allows untrusted users to execute read-only SQL directly against SQLite. Adding LLM-assisted querying to it is an obvious next step, my first experiment with that was a ChatGPT plugin described here: https://simonwillison.net/2023/Mar/24/datasette-chatgpt-plugin/
My approach right now is that Datasette encourages users to create databases where every column should be visible to anyone using them - it’s designed such that each user can have their own database, rather than being multi-tenant in a single database (which requires row-level security).
I’ve also been experimenting with SQL views as a way to allow access to a controlled subset of data, which is something the linked article suggests too.
SQLite does have one additional tool that’s relevant here though: you can set an “authorizer” function which will get called to check if the current connection should have access to individual tables and columns. I haven’t spent much time with this, but I did use it to build a plugin that can do things like mask out the password_hash in an otherwise openly available table: https://github.com/simonw/datasette-mask-columns
My other relevant project is https://django-sql-dashboard.datasette.io/ - which provides a PostgreSQL querying dashboard interface for Django projects, protected by the Django staff user account system.
I was pleased to confirm just now that it does provide protection against the select pg_sleep(999999) attack in that article, because it encourages applying a statement_timeout=100 to the connection to limit to 100ms.
Thanks for sharing, Simon. The sqlite solution sounds like it’s working well for Datasette, cool! I look forward to seeing how the product evolves with LLM integrations.
I wasn’t aware of the sqlite authorizer function. Sqlite surprises me constantly. There are some subtleties with allowing columns to be used even if they aren’t selected, for example side channel attacks which can reveal the column values without seeing them directly https://docs.heimdallm.ai/en/main/attack_surface/sql.html#side-channel-attacks I wonder if sqlite triggers the authorizer when a column is used in a condition, but not selected from. Though it doesn’t sound like it would affect your use-cases since access to all datasette data is authorized.
I believe the next big advancement in this area is going to be LLM-assisted DML queries. Using natural language to safely insert, update, and delete records in a shared database is where the static analysis “constraint validation” solution that HeimdaLLM provides will differentiate itself meaningfully from all other solutions. If this area interests you, shoot me an email.
The “constraint validation” approach is adjacent to another approach not mentioned. Imagine defining a grammar for a restricted subset of SQL such that only specific safe tables, columns, aggregations, etc. are allowed. Every top-level production of this grammar is safe, by definition, as sketched in the approach. However, rather than parsing what the LLM generates, I would consider using something like ParserLLM to constrain the LLM itself, rather than the LLM’s output.
The practical difference is that parsing always succeeds; this is in contrast to HeimdaLLM, which might not just fail, but might always fail.
Yes, you hit on exactly what the “constraint validation” approach provides. It uses a grammar for a restricted subset of SQL, and the specific safe tables, columns, etc, are specified in the parsing step. Could this be made more clear in the post?
I’d like to see the threat model articulated a bit more clearly here. You are permitting a user to run arbitrary queries. Whether they use an LLM to prepare them or not, they can run whatever queries they’re authorised to do. If your threat model is the user using the LLM to generate malicious queries then you need to provide some kind of access control that restricts what the user can do.
Generally, when people talk about LLM security in this kind of context, they care about prompt injection or hallucination. In these scenarios, the users asks for a solution for task X, but the LLM consumes their input and either combines it with something provided by an attacker or hits a less-explored area of the similarity space and generates something that is not a solution for task X and may be actively malicious. If this is what you care about, you generally want one of two things:
The problem here is not that the user can do a bad thing but that the UI will accidentally do a bad thing independent of their intention. The read-only database proposal in the article is a big hammer here: the query can tell you anything but can’t modify any data. That severely restricts the utility but eliminates much of the risk. It doesn’t, for example, prevent the user from being given bogus data. For example, if the user asks whether something is safe and a prompt injection or hallucination tells the user ‘yes’, they haven’t modified the database but they might be about to die.
If you want these queries to be able to update the database, you really want a capability system. I don’t know if there are any capability extensions to SQL, but they’re the best way of expressing the principle of intentionality: you can modify a part of the database only if you intended to modify it and you provide the LLM query with the set of capabilities that it should have to accomplish the task.
The last solution, “constraint validator” is the capability system that you describe. It’s what HeimdaLLM provides. Basically you can define what tables, columns, joins, functions, etc are allowed to be used and how they’re allowed to be used. When HeimdaLLM processes an untrusted query, it validates those constraints. It does this by using a real grammar and parser.
I’ve been investigating this problem for a few years now with my https://datasette.io/ project, which allows untrusted users to execute read-only SQL directly against SQLite. Adding LLM-assisted querying to it is an obvious next step, my first experiment with that was a ChatGPT plugin described here: https://simonwillison.net/2023/Mar/24/datasette-chatgpt-plugin/
My approach right now is that Datasette encourages users to create databases where every column should be visible to anyone using them - it’s designed such that each user can have their own database, rather than being multi-tenant in a single database (which requires row-level security).
I’ve also been experimenting with SQL views as a way to allow access to a controlled subset of data, which is something the linked article suggests too.
SQLite does have one additional tool that’s relevant here though: you can set an “authorizer” function which will get called to check if the current connection should have access to individual tables and columns. I haven’t spent much time with this, but I did use it to build a plugin that can do things like mask out the password_hash in an otherwise openly available table: https://github.com/simonw/datasette-mask-columns
My other relevant project is https://django-sql-dashboard.datasette.io/ - which provides a PostgreSQL querying dashboard interface for Django projects, protected by the Django staff user account system.
I was pleased to confirm just now that it does provide protection against the
select pg_sleep(999999)attack in that article, because it encourages applying a statement_timeout=100 to the connection to limit to 100ms.Thanks for sharing, Simon. The sqlite solution sounds like it’s working well for Datasette, cool! I look forward to seeing how the product evolves with LLM integrations.
I wasn’t aware of the sqlite authorizer function. Sqlite surprises me constantly. There are some subtleties with allowing columns to be used even if they aren’t selected, for example side channel attacks which can reveal the column values without seeing them directly https://docs.heimdallm.ai/en/main/attack_surface/sql.html#side-channel-attacks I wonder if sqlite triggers the authorizer when a column is used in a condition, but not selected from. Though it doesn’t sound like it would affect your use-cases since access to all datasette data is authorized.
Good to hear you’re not vulnerable to
pg_sleep. I’ve manually compiled a list of postgres functions that exclude functions with side effects, feel free to use them if they are useful for what you’re doing https://github.com/amoffat/HeimdaLLM/blob/dev/heimdallm/bifrosts/sql/postgres/presets.pyI believe the next big advancement in this area is going to be LLM-assisted DML queries. Using natural language to safely insert, update, and delete records in a shared database is where the static analysis “constraint validation” solution that HeimdaLLM provides will differentiate itself meaningfully from all other solutions. If this area interests you, shoot me an email.
The “constraint validation” approach is adjacent to another approach not mentioned. Imagine defining a grammar for a restricted subset of SQL such that only specific safe tables, columns, aggregations, etc. are allowed. Every top-level production of this grammar is safe, by definition, as sketched in the approach. However, rather than parsing what the LLM generates, I would consider using something like ParserLLM to constrain the LLM itself, rather than the LLM’s output.
The practical difference is that parsing always succeeds; this is in contrast to HeimdaLLM, which might not just fail, but might always fail.
Yes, you hit on exactly what the “constraint validation” approach provides. It uses a grammar for a restricted subset of SQL, and the specific safe tables, columns, etc, are specified in the parsing step. Could this be made more clear in the post?