Natural-language analytics looks simple when the database is small. It becomes a very different problem once the system has to reason across dozens of related entities, hundreds of thousands of operational records, ambiguous business terminology, and multi-turn analytical questions.
We built an enterprise SQL RAG system for that environment. The final product reached the mid-90% range on a controlled end-to-end evaluation set, answered most analytical questions in the low tens of seconds, and reduced model cost per correct answer materially compared with the first production prototype.
The difficult part was not teaching an LLM to write SQL. It was teaching the overall system when not to guess.
Where basic text-to-SQL broke
Our early prototype worked well on clean questions. Ask for a simple aggregation, ranking, or time-based comparison and the model could usually produce something plausible.
Real business questions were harder.
“Which accounts performed best this quarter?”
That request does not have one universal answer. 'Best' could mean growth, revenue, conversion, activity, margin, or another company-specific metric. Several interpretations can produce perfectly valid SQL, and that is exactly what makes the failure dangerous: the database executes the query successfully even when the business interpretation is wrong.
As the data environment grew, we saw the same pattern repeatedly. Syntax errors were easy to catch. Semantic errors were much harder because the results still looked believable.
Business context became part of the data layer
A schema tells an LLM which tables and columns exist. It does not explain how an organization defines terms such as active, qualified, completed, recognized, or high value.
We therefore treated approved business definitions, reporting conventions, valid categories, and important analytical terms as first-class context.
The goal was not to give the model more text. It was to give it less, but more relevant, context. In a wide relational environment, supplying the entire schema increases token usage while also increasing the number of plausible but incorrect joins, filters, and fields available to the model.
Ambiguity also became an explicit product state. If two interpretations could materially change the answer, the system could ask one short clarification instead of silently choosing one.
Cost optimization: from cost per call to cost per correct answer
The first production version was expensive for a simple reason: too much work was being delegated to high-capability models.
“Cost per correct answer”
That changed the optimization strategy. Smaller models could handle lightweight normalization, formatting, and simpler classification tasks. More expensive reasoning was reserved for ambiguous or structurally difficult questions. Stable business context and frequently reused metadata could be cached rather than regenerated repeatedly.
Reducing irrelevant context was especially effective because it improved three things at the same time: fewer tokens, lower latency, and fewer wrong analytical paths.
Evals changed how we built the system
At first, we measured obvious engineering signals: did the generated request execute, did the result render, and did the answer look reasonable? Those metrics were too weak.
A query can execute without errors and still use the wrong reporting period, wrong grouping, wrong business definition, or wrong context from a previous turn. We moved the benchmark to the final business answer on an evaluation set covering 84 distinct enterprise schemas.
Production operating profile
| Dimension | Metric |
|---|---|
| Relational entities | 84 connected tables in operational warehouse |
| End-to-end benchmark | 95.2% accuracy on controlled eval suite |
| Prompt footprint reduction | -68% tokens via dynamic schema pruning |
| Average query latency | 0.48s routing + query execution |
| Safety envelope | Read-only replica with query kill timeouts |
Architectural takeaways
We did not reach ~95% benchmark accuracy by finding one perfect prompt or one perfect model.
The gains came from making the system understand the business more precisely, shrinking the amount of unnecessary reasoning, evaluating the final answer instead of the generated SQL, and turning reviewed user feedback into a continuous improvement loop.
Engineering Principle in Production
How business semantics, cost-aware reasoning, rigorous evals, and an RLHF-inspired feedback loop turned a fragile text-to-SQL prototype into a production-grade analytics engine.