Blog  /  2026-11-01

From thumbs-up to better SQL: turning user feedback into verified examples

Published 2026-11-01. Written for nlqueries-core 0.4.

feedback accuracy security

A text-to-SQL feedback loop turns "that answer was right" into better SQL for the next person who asks something similar. The idea is simple: keep the question-and-SQL pairs people confirm, and show the nearest ones to the model as examples. The hard part is that an example steers every later answer on that agent, for everyone, so the loop is also a way to put words in the model's mouth. This post describes the loop in nlqueries-core 0.4: the two kinds of feedback, how each one reaches the prompt, and the checks a pair has to pass on the way.

Two kinds of feedback, two paths

Verified pairs are retrieved by meaning: for each question, up to three stored pairs whose questions are close to it (cosine similarity of 0.75 or more) are added to the prompt's example queries. They go first, ahead of the examples mined from query history, and are labelled -- verified example. Corrections go into the knowledge base itself, ahead of the mined examples, and the model sees the first few of those on every question.

Recording feedback

From the command line:

nlqueries feedback shop --question "Revenue by region last quarter" --thumbs-up \
  --generated-sql "SELECT region, SUM(amount) FROM orders WHERE ... GROUP BY region"

nlqueries feedback shop --question "Customers with no orders" --thumbs-down \
  --corrected-sql "SELECT c.id FROM customers c LEFT JOIN orders o ON o.customer_id = c.id WHERE o.id IS NULL"

Pass the SQL with a thumbs-up. --generated-sql is optional, and a thumbs-up recorded without it is stored, but promotion needs a pair: a record with no SQL is passed over without a word. nlqueries query --json prints the generated SQL to copy from. A thumbs-up that also carries --corrected-sql promotes the corrected statement.

A correction is saved, not applied. It reaches the knowledge base on the next export-kb. nlqueries feedback-stats shop counts what you have recorded so far.

The MCP server has a feedback tool too, so an assistant can rate an answer in the conversation. Those records are kept, but the loop does not use them unless you ask it to. The next section explains why.

Why only feedback that names someone

A promoted pair is shown to the model for every later question it resembles, on that agent, for every user. Whoever chose the pair chose part of everyone's prompt. So the rating has to come from someone who can be held to it.

Each record carries its origin. A record made with nlqueries feedback was run by an operator on the host, and counts. A record from the MCP tool identifies nobody: it is whatever the client sent. One written from Python with record_feedback counts only if your code sets its source; left at the default, its origin is unknown. Neither promote-feedback nor export-kb uses records of unknown origin, MCP records, or records from before origins were tracked, unless you pass --include-anonymous. Read them first:

nlqueries promote-feedback shop --include-anonymous
nlqueries export-kb shop --include-anonymous

Both commands say how many records they skipped and why, so nothing is passed over in silence, apart from the thumbs-up with no SQL above.

The checks a pair has to pass

Promotion and corrections apply the same checks, in the same order. A pair is kept only when:

• its SQL passes the SQL policy in the agent's dialect, as a single read-only query, the same test a generated statement faces before it runs;
• it names at least one table, and only tables in the knowledge base. A table named with a schema has to match the schema the knowledge base records, so other.orders does not pass for public.orders;
• the knowledge base has a schema to check against. With none (an agent never exported, say), nothing is promoted, and the warning says to run export-kb first.

What is stored is normalised too. The question is kept on one line of at most 200 characters, and the SQL without comments. Both end up in the prompt, where a line break in the question could end the line it sits on and turn the rest of the text into instructions, and a comment in the SQL would be shown as written. In a correction, literals also become typed placeholders, as in the examples mined from history, so the example shows the shape of the query rather than the values someone typed.

These checks are why a pair that passes is safe to show. They do not decide whether its SQL answers its question. That is what the rating is for, and why it matters who gave it.

Pairs that no longer fit are removed

Schemas change. After promoting, promote-feedback checks every pair already stored for the agent against the same rules and deletes the ones that fail: a pair whose table has left the knowledge base, or one promoted by an earlier release under looser rules. The command reports how many it removed. A removed pair comes back on a later promotion if its feedback qualifies by then.

There is one exception. If every stored pair fails at once, nothing is deleted and a warning is logged. That points at the check rather than the pairs: for example, an engine the SQL parser has no grammar for is read as PostgreSQL, and then no pair parses at all.

Signed, so Qdrant cannot add its own

Verified pairs live in Qdrant, in agent_<id>_verified, and anything that can write to Qdrant could add a pair. So in 0.4, promote-feedback signs each pair with the semantic-cache signing key, which is held outside Qdrant. The reader uses only pairs whose signature verifies and whose agent is the one asking.

It fetches four times as many candidates as it needs, so a few unsigned pairs near the question do not push the real ones out. More than that leaves the prompt with fewer verified examples, never with planted ones. Every process that reads the pairs needs the key that signed them: set NLQ_CACHE_SIGNING_KEY where more than one process shares a Qdrant. See configuration.

If you are upgrading from 0.3, pairs promoted then are unsigned and ignored, with a warning, until you run promote-feedback (or export-kb) once per agent.

The loop, in commands

nlqueries query shop "Revenue by region last quarter" --json   # copy the SQL
nlqueries feedback shop --question "Revenue by region last quarter" \
  --thumbs-up --generated-sql "<the SQL>"
nlqueries feedback shop --question "..." --thumbs-down --corrected-sql "<the fix>"

nlqueries export-kb shop          # folds in corrections, then promotes thumbs-ups
nlqueries promote-feedback shop   # between exports, after a batch of ratings

export-kb runs the promotion step at the end of every run, without --include-anonymous, so most of the time it is the only command you need. promote-feedback is for a batch of ratings between exports. Verified pairs need a reachable Qdrant and the embedding model; corrections need neither, since they live in the knowledge-base file. Qdrant setup covers the first.

In short

1. A thumbs-up needs its SQL, or there is nothing to promote.
2. Only feedback that names someone is used, unless you review the rest.
3. A pair must pass the SQL policy and read only known tables, and it is deleted when it stops doing so.
4. Pairs are signed, so a writer to Qdrant cannot add one.

The commands are in the CLI reference under feedback and promote-feedback. For the examples the model gets from your query history, which these sit beside, see How to build a text-to-SQL knowledge base from query history. To try it, pip install nlqueries-core and start with getting started.

All posts · RSS feed · Documentation