NLQueries GitHub

Blog  / 

Should an Assistant Answer from Database Rows or Documents?

Published . Written for nlqueries-core 0.4.

documents hybrid-answers routing

Two messages land in your team's assistant on the same morning. Operations asks, "How much paid revenue did tenant A bring in during October?" Support asks, "Which of tenant A's October orders are still eligible for a refund?" They read like siblings, with the same tenant, the same month and the same orders table in the background. But they need different evidence. The first is a count of facts, and only the database has it. The second needs those facts and a rule, and the rule lives in a policy document, not a column.

Get the routing wrong and both failures look convincing. A document-only answer to the revenue question can explain how revenue is recognized without giving you this month's number. A SQL-only answer to the refund question can list every paid October order with no idea that a refund window exists. Neither one throws an error, so nobody notices.

So the short answer is: route by the fact the question needs, not by how it's phrased. Then make the route, the evidence and any ambiguity visible to whoever reads the answer. The rest of this post works through how to do that, with NLQueries Core 0.4.1 as the concrete example. The product details come from its documentation and its source at a pinned commit. Only the SQL fixture and some date arithmetic were executed for this post; it contains no runs of the document or hybrid paths and no routing-accuracy numbers.

Start with the fact the question needs

The useful split is between three kinds of question.

Row questions ask about recorded state: counts, sums, latest values, who did what and when. "Paid revenue for tenant A in October" is one. The authoritative answer is whatever the database returns under the caller's permissions, and an assistant that answers it from a document is guessing.

Document questions ask about rules, procedures and definitions. "What's our refund window?" is one. No table holds the answer, and if one did it would be a copy of the policy that's free to drift from the real thing.

Combined questions need both. "Which October orders are still refundable?" needs the order dates from the database and the window from the policy, plus a little reasoning to join them. These are the questions where an assistant can be most useful and also the ones where it can be wrong in the most interesting ways.

Notice that phrasing doesn't sort these for you. "How much" and "which" turn up in all three kinds, and "refund" appears in both the policy question and the combined one. What separates them is whether the answer depends on a recorded fact, a written rule, or one judged against the other. That's the label to put on a question before you decide where to send it.

A small fixture makes the difference concrete. Here's the sample schema and data this post uses throughout. It's synthetic, built so that a few classic mistakes give visibly wrong answers:

-- Synthetic sample data (SQLite). Customers 1 and 2 belong to tenant A, customer 3 to tenant B.
CREATE TABLE customers(id INTEGER PRIMARY KEY, tenant_id TEXT NOT NULL, region TEXT);
CREATE TABLE orders(id INTEGER PRIMARY KEY, customer_id INTEGER REFERENCES customers(id),
                    order_date TEXT, amount_cents INTEGER, status TEXT);
CREATE TABLE order_items(id INTEGER PRIMARY KEY, order_id INTEGER REFERENCES orders(id), sku TEXT);
INSERT INTO customers VALUES (1,'A','East'),(2,'A','West'),(3,'B','North');
INSERT INTO orders VALUES
  (101,1,'2026-10-01',10000,'PAID'), (102,1,'2026-10-02',5000,'CANCELLED'),
  (103,2,'2026-10-31',20000,'PAID'), (104,3,'2026-10-02',70000,'PAID'),
  (105,1,'2026-11-01',40000,'PAID'), (106,2,'2026-10-15',NULL,'PAID');
INSERT INTO order_items VALUES (1,101,'a'),(2,101,'b'),(3,103,'c');

-- Reference answer for the row question
SELECT SUM(o.amount_cents) AS revenue_cents, COUNT(*) AS paid_order_count
FROM orders AS o JOIN customers AS c ON c.id = o.customer_id
WHERE c.tenant_id = 'A' AND o.status = 'PAID'
  AND o.order_date >= '2026-10-01' AND o.order_date < '2026-11-01';

Against this fixture on SQLite 3.45.3, the reference query returns (30000, 3): 30,000 cents from three paid October orders. The NULL-amount order (106) counts as an order but adds nothing to revenue. The cancelled order, tenant B's order and the November order all stay out, and each of them is there to catch a filter someone forgot. That's the row answer, and it doesn't need a single paragraph of text to be right.

What rows are good at, and what documents are good at

Rows are exact, and they carry their own proof: you can show the SQL that produced the number, and a reviewer can rerun it. Freshness is subtler than it looks, though. A query sees the state visible to its transaction and data source. Check commit visibility, replication lag and refresh schedules before describing that state as current. If the assistant reads from a replica or from a table that's rebuilt on a schedule, "current" really means "as of the last sync," and a careful answer says so.

What rows can't do is explain why. A status column tells you an order was cancelled. It doesn't tell you the cancellation policy.

Documents are the opposite. They hold intent, rules and definitions, and they're written for people. But they go stale quietly, they contradict each other, and retrieval hands you whichever chunks are nearest to the question, not necessarily the ones that are current or binding. A citation shows where a sentence came from. It doesn't show that the sentence is still true or that it supports the claim built on top of it.

That's why combined answers deserve the most scrutiny. Take the refund question against the fixture, using an illustrative as-of date of 5 November 2026. Tenant A's paid October orders are 101 (35 days old), 103 (5 days) and 106 (21 days). Under a 30-day refund window, orders 103 and 106 qualify. Under a 14-day window, only 103 does. Those ages and eligibility sets come from running the date arithmetic against the sample data, under two illustrative rules; they say nothing about any real retrieved policy. Same rows, same question, different answer, and the only thing that changed was which policy text the assistant found.

That example is worth holding onto, because it shows the failure you can't see from the answer alone. "Orders 103 and 106 are eligible" and "Order 103 is eligible" are both fluent, both plausible, and both computed correctly from the rows. Only the evidence trail tells you which rule each one applied.

How NLQueries picks a source, and how to see the pick

NLQueries Core makes the source choice explicit rather than magic, which is the property you want here. According to its architecture notes, an agent (NLQueries' name for a configured database connection plus its knowledge base) that lists no document sources answers from SQL only. Its questions go straight to the SQL path without classification. Once you attach document sources, query and the MCP query tool turn on the document agent and hybrid answers too. An intent classifier then sends each question to SQL, to documents, or to both in parallel, and a merger combines the results.

Wiring that up from the command line looks roughly like this. It's an untested sketch: it assumes the fixture database is already registered as a SQLite connector under the alias prod (the CLI accepts an alias anywhere it takes a connector ID), with Qdrant running for document storage.

# Untested sketch. Assumes an existing agent "prod" and a reachable Qdrant (QDRANT_URL).
nlq doc-ingest refund-policy-v3 ./policies/refund-policy-v3.md   # chunk and embed the current policy
nlqueries agent docs add prod refund-policy-v3                  # let the agent use that source

nlqueries query prod "What was paid revenue for tenant A in October 2026?" --json --explain
nlqueries query prod "Which tenant A orders from October 2026 are still eligible for a refund?" --json --explain

The CLI reference says plain query output shows the agent type alongside the SQL and rows. For document or hybrid answers it also shows the answer text and numbered sources. --explain adds answer provenance (route, cache, checks and timings) to the JSON. Note that query executes the generated SQL by default; --no-execute stops that.

The Python API is where people most often trip, because it doesn't infer the document path for you. In the pinned sync_runner.py, run_query defaults available_types to ("sql",). A backend that attaches documents to an agent but calls run_query with defaults will keep answering the refund question from rows alone. The same module provides answer_types_for(agent_id), which returns ("sql", "document", "hybrid") when the agent lists document sources and ("sql",) otherwise. Another default worth knowing is the execution policy, which is generate-only unless you pass something else (execution.py).

The same signature has a third default that matters with this fixture. run_query takes a dialect argument that defaults to "postgres", and the Python call doesn't work it out from the registered connector. The MCP server does resolve the dialect from the agent's connector itself (server.py), but a direct run_query call uses whatever you pass, or that default. Against a SQLite database, leaving it out means the SQL sub-agent is told PostgreSQL while SQLite executes the result. Pass the dialect explicitly, and take it from trusted configuration rather than from the request. The agent_id needs the same care: it's the canonical connector ID that nlqueries connect printed (SQLite connectors use sqlite:<path>), not a CLI alias like prod.

Here's a minimal sketch that asks a question, lets the agent use whatever sources it has, and records the route that was actually taken. It's written against the 0.4.1 source and hasn't been executed. Replace /absolute/path/shop.db with the path you registered, or paste the exact ID connect printed:

# Untested sketch against NLQueries Core 0.4.1 (commit 2fbf276).
# Assumes: the fixture registered as a SQLite connector, a document source attached
# to it, and a provider and Qdrant configured.
import asyncio
import json

from nlqueries.execution import ExecutionPolicy
from nlqueries.orchestrator.sync_runner import answer_types_for, run_query

# Canonical connector ID printed by `nlqueries connect sqlite`, not a CLI alias.
AGENT_ID = "sqlite:/absolute/path/shop.db"


async def ask(question: str, agent_id: str = AGENT_ID) -> dict:
    result = await run_query(
        question,
        agent_id,
        dialect="sqlite",  # this article’s fixture; derive from trusted configuration for other agents
        available_types=answer_types_for(agent_id),   # don't silently fall back to SQL-only
        execution=ExecutionPolicy.execute_read_only(),  # default is generate-only
        explain=True,                                  # populate result.provenance
    )
    return {
        "question": result.question,
        "route": result.agent_type,          # "sql" | "document" | "hybrid" | "unclear"
        "sql": result.sql,
        "sql_is_valid": result.sql_is_valid,
        "execution_mode": result.execution_mode,
        "sources": [(c.source_name, c.page_number) for c in result.citations],
        "from_cache": result.from_cache,
        "answer": result.answer,
    }


if __name__ == "__main__":
    print(json.dumps(asyncio.run(ask(
        "Which tenant A orders from October 2026 are still eligible for a refund?")), indent=2))

Because the dialect line is a fix made by reading the source rather than by running it, check it before you trust it: push one SQLite-specific expression and one PostgreSQL-specific expression through correctly configured connectors of each kind, and confirm that the grammar the SQL step works to and the engine that runs the query agree.

The returned record helps diagnose the answer. A SQL route with no sources provides no document citation. If a hybrid answer omits order 106 and cites the 14-day draft, check whether that rule accounts for the omission by comparing the retrieved text, returned dates and reference answer.

When the question is vague or the documents disagree

Two situations break naive routing, and neither is a model problem you can prompt your way out of.

The first is a question that doesn't say enough. "How are refunds looking?" doesn't name a tenant, a period, or whether it wants a count or a rule. The result type in 0.4.1 has an unclear value for agent_type, which at least gives your application something to branch on. What happens next is your call. A short clarifying question, such as "Do you mean refund requests for tenant A last month, or the refund policy?", usually beats letting a classifier pick one and answer with confidence. The cost is one extra turn; the alternative is an answer to a question nobody asked. That clarification flow is application work, not a built-in feature, so label it that way in your design docs.

The second is evidence that disagrees with itself. Put the current 30-day policy and an old 14-day draft in the same agent, and you've created a trap. The architecture notes say the document agent searches every listed source and keeps the nearest chunks overall. Nearest isn't the same as newest, and it isn't the same as binding. The cheapest fix is hygiene: only list the sources that should be authoritative. nlqueries agent docs remove stops an agent using a source while keeping its chunks. Beyond that, source precedence (effective dates, a status field, a rule that drafts never win) is something you design and test. Don't assume retrieval will sort it out for you.

Either way, a combined answer should say which document version it relied on. "Orders 103 and 106 are eligible under the refund policy effective [date]" is checkable. "Orders 103 and 106 are eligible" isn't.

A route is not a permission

It's tempting to treat routing as a security boundary, as in "this question goes to documents, so it can't touch the orders table." It isn't one. Every table and every document a route can reach has to be independently permitted for the person asking. Routing decides where to look, and authorization decides what that person may see. Keep them separate in your design and in your tests. A cheap test is to ask a document-shaped question as a user who can't read the orders table, and a row question as a user who can't read the policy, then check that neither answer leaks what the other path holds.

Two NLQueries details make this concrete. First, hybrid answers can send result rows to the model. The configuration reference documents NLQ_SEND_RESULT_ROWS_TO_LLM, which defaults to true. In that mode up to 20 rows go to the LLM for synthesis, with columns whose names look like personal data withheld. The docs are candid that the check reads the returned column name, so an alias like SELECT email AS contact slips through. Setting it to false sends only column names and the row count to the synthesis step. That's safer, but the merger then has less to reason with, so a hybrid answer may say less. Keep the flag's scope in mind, too: it governs what hybrid synthesis receives, and on its own it isn't a statement about every request the pipeline makes to a provider.

Second, caching. The architecture notes say the 0.4.1 cache doesn't store document and hybrid answers, so a newly edited policy isn't outlived by a semantic-cache entry written by this release. Entries left by older versions, and your own application caches and document indexes, still need freshness controls of their own. SQL answers can be cached, and every entry in an agent's cache is readable by everyone who may query that agent. The same notes spell out the rule that follows: anything that gives two callers of the same agent different entitlements, including per-user document ACLs, must feed into the cache context on both read and write. Treat that as an invariant your integration has to uphold. The docs state it; they can't show that a particular wrapper actually fills in the cache context or applies row filters, so test that in yours.

A source-choice table you can reuse

Here's a decision guide worth pinning next to the routing code. It's a design aid, not a measured result.

Question shape Evidence source Show the reader Failure to watch for
Count, sum, latest value ("paid revenue in October") Database rows SQL, row count, execution mode Answer drawn from a document that describes the metric
Rule, definition, procedure ("what's the refund window?") Documents Cited source name and page, document version Obsolete or draft text cited with confidence
Facts judged against a rule ("which orders are refundable?") Both (hybrid) SQL, rows used, cited policy version One half missing; rows sent to the model beyond policy
Missing scope ("how are refunds looking?") Neither yet A clarifying question Classifier guesses and answers anyway
Data you don't hold ("refunds from the old billing system") None A clear "not available" Plausible numbers or paraphrased policy filling the gap

The last row matters more than it looks. Missing evidence should stay missing. An assistant that says "I can't see the legacy billing data" is more useful than one that fills the gap with something plausible.

The "show the reader" column is also the cheapest part to build. Most of it is already in the result record from the sketch above: the route, the SQL, whether it executed, the cited sources and whether the answer came from cache. Putting those next to the answer text turns every reply into something a reviewer can check instead of something they have to trust.

Next step: label twenty questions before you tune anything

Before you adjust a prompt or a classifier threshold, write down twenty questions your users actually ask and label each with the route it should take: rows, documents, both, clarify, or not available. Include a few near-twins like the revenue and refund pair above, at least one vague question, and one that hits the 14-day draft. Load the fixture and both policy versions, run the questions through ask(), and compare the logged route and sources against your labels. Keep the raw outputs.

Then change one thing and run the refund question again. Remove the draft with nlqueries agent docs remove, and check whether order 106 comes back into the eligible set and whether the cited source changes to the current policy. If it does, you've shown that source hygiene alone fixes that case. If it doesn't, you've found the spot where you need explicit precedence rules.

That gives you something no feature list can: evidence of where your assistant picks the wrong source, on your questions, under your permissions. Then decide whether the fix is better source hygiene, a clarification step, or a tighter row-sending policy.

References

All posts · RSS feed · Documentation