Building a Data Analyst Agent: Grounding an LLM in Tools, Not Guesses

Point a language model at a spreadsheet and it will always answer. With code from our analytics agent, here is how we keep the answer anchored to the data, and how it reports what it refused to do.

Point a language model at a spreadsheet and it will always answer. That is the problem, stated in one sentence. A model asked about a column that does not exist will not stop and tell you the column does not exist. It will reach for the nearest plausible thing, write a query, and hand back a confident, well-formatted number that is quietly fiction. The hard part of a "talk to your data" system is not getting an answer. It is making the answer one you can defend to the person who signs off on it.

This is a technical write-up of how the Data Analyst Agent does that, with the code that does it. The theme throughout is a single discipline: the model is never trusted to be right about what is in the data. It is checked, every time, against the data itself.

Plan and execute over a closed registry

The agent does not improvise. It plans. A planner chooses which tools to run and in what order; an executor loads the dataset and runs them. The planner never touches the data, and the executor never runs a tool that is not in the registry.

The registry is the leverage point. The space of good analytic actions is knowable and small, so it is enumerated: profile_dataset, data_quality_report, correlation_analysis, trend_analysis, anomaly_scan, kmeans_clusters, duckdb_query, multidim_pivot, train_supervised_model, evaluate_ml_predictions, score_with_model, explain_model, shap_explain_prediction, forecast_with_model, auto_insights, and a handful of chart specs. The planner's job is selection and sequencing from that fixed set, not invention.

Every tool declares its arguments as a Pydantic model that forbids anything it did not ask for:

class ToolArgs(BaseModel):
    model_config = ConfigDict(extra="forbid", protected_namespaces=())

That extra="forbid" is the first line of defence. A tool call carrying an argument the tool never declared does not get coerced or ignored. It fails validation. The action space is closed, which is exactly what makes it testable.

Tool selection by retrieval, not a wall of prompt

A dozen-plus tools, each with its own arguments and documentation, is more than you want in a prompt at once. Pour every schema in and the planner chooses worse, the way anyone does when handed the whole catalogue and told to pick fast.

So tool documentation lives in a local corpus indexed with FAISS, and the planner is handed a slimmed, retrieval-filtered view of the tools relevant to the question rather than the full set. The prompt stays small and the selection stays sharp. Retrieval here is not decoration. It is how the planner keeps its attention on the right shelf of the toolbox.

Validating every call against the real schema

The registry stops the model from inventing a tool. It does nothing about the model inventing a column. That is handled separately, and without exception: every proposed tool call is validated against the actual dataframe before a single row is read. The checks are specific per tool:

def validate_columns(df, columns, arg_name):
    missing = [col for col in (columns or []) if col not in df.columns]
    if missing:
        raise ValueError(f"{arg_name} contains unknown columns: {missing}")

A pivot has its index, columns, and values checked. A training call has its target_col and feature_cols checked. A chart spec has its x and y checked. SQL is the interesting case, because you cannot check SQL with a column list. So we let the database judge it, in a sandbox with external access switched off:

if call_name == "duckdb_query":
    query = args.get("query", "")
    if query:
        con = duckdb.connect(database=":memory:",
                             config={"enable_external_access": False})
        con.register("t", df.head(0))
        try:
            con.execute(f"EXPLAIN {query}")       # parse + bind, never run
        except Exception as exc:
            actual_cols = ", ".join(str(c) for c in df.columns)
            raise ValueError(
                f"SQL references unknown columns or has a syntax error. "
                f"Table 't' has columns: [{actual_cols}]. Error: {exc}"
            ) from exc

EXPLAIN binds the query against the real (empty) schema without executing it, so a reference to a column that does not exist fails here, cheaply, before any data is touched. And notice the error message: it hands back the actual column list. That is not for the human. That is the input to the next step.

The loop: detect, repair, drop, report

Validation on its own only tells you something broke. What happens next is the design, and the order of the four steps is the whole point. Here is the loop, simplified, returning both the valid calls and a list of notes:

valid, problems = validate(calls, df)

# Record every call that failed, even if repair later just drops it,
# so the caller can surface what was requested and discarded.
dropped = {p["name"]: p["error"] for p in problems}

attempts = 0
while problems and attempts < max_repair_attempts:
    attempts += 1
    repaired = repair(problems)            # hand the model back its own error
    repaired_valid, problems = validate(repaired, df)   # check again
    for call in repaired_valid:
        dropped.pop(call.name, None)
    for p in problems:
        dropped[p["name"]] = p["error"]
    valid.extend(repaired_valid)

notes = [f"Dropped tool call '{name}' after validation failed: {error}"
         for name, error in dropped.items()]
return valid, notes

Detect is validation against the live dataframe. Repair hands the model back its own invalid call plus the exact error ("column X not in dataset; the table has columns A, B, C") and asks it to fix or drop it. Then the result is validated again. The loop is bounded by a configurable number of repair attempts (default 1), because an unbounded repair loop is just a slower way to hang.

Drop is the step that takes nerve: a call that cannot be made valid is thrown away. It is never coerced onto a column that looks similar, and it never runs.

Report is the step most systems skip, and it is the one that makes the difference. Every dropped call becomes a note on the response: "Dropped tool call 'trend_analysis' after validation failed: date_col not in dataset." The caller finds out what the agent refused to do and why. A system that hides what it dropped is lying by omission, and it will be caught at the worst possible time. A system that reports it has told the truth about the edge of its own ability, and that is what you build trust on.

Measuring groundedness, including the failures

Refusing to run on invented columns stops the obvious disaster. It does not, by itself, tell you whether the answers the agent does produce are grounded in the data it looked at. For that you measure, and you measure the failures with the same seriousness as the successes.

A configurable fraction of synthesised answers is scored for groundedness by a second model acting as a judge. The sample rate is a setting (llm_judge_sample_rate), because judgement costs tokens too. Every answer carries a status, and the set of statuses is itself the honest accounting: judged, not_sampled, rule_based, llm_disabled, and failed. Nothing is swept under the rug. An answer that was produced without an LLM is marked rule_based, not silently counted as grounded. A judge call that errored is failed, not dropped from the denominator. The aggregate is exposed over HTTP, and it does not sit alone. The service ships a wall of gauges:

You cannot improve what you cannot see. The instrumentation is not scaffolding to be removed after launch. It is what lets the system get better instead of slowly, invisibly worse.

Degrading toward honesty, not toward a guess

What does the agent do when there is no model at all? Key expired, budget gone, endpoint down. The default config is llm_enabled = False, so this is not an edge case; it is the baseline the system is designed to run in.

With no LLM, the planner falls back to keyword routing over the same registry: ask for a profile and you get profile_dataset, ask about correlation and you get correlation_analysis. When nothing matches, it does not grab the nearest tool and hope. It records the miss and returns a clarifying question. The floor never drops out, and it never drops out into a confident invention.

The knobs, as proof it is a real system

None of this is a prototype's hard-coded behaviour. It is configuration, with conservative defaults:

Swap the model, loosen or tighten the repair budget, dial the judge sampling down under load. The behaviour is tunable precisely because it was built as a system, not a demo.

Takeaways

Strip away the specifics and a handful of transferable commitments remain.

• Close the action space. A planner choosing from a known registry is testable, auditable, and something you can reason about. A model improvising tools is none of those.
• Validate every model-proposed action against reality before it runs. For analytics that means columns against the live schema, and a sandboxed EXPLAIN to judge SQL rather than a regex.
• Make the loop honest: detect, repair, drop, report. Bound the repair, make the drop real, and never make the report optional.
• Measure groundedness, including the failures. Give every answer a status, and put the aggregate behind an endpoint you will actually look at.
• Degrade toward honesty. No model means rules and a clarifying question, never a confident guess.

Put a language model behind an interface, hold it to the truth of the data at every step, and let it say plainly when it cannot do what was asked. Do that, and the result is rarer than a clever demo. It is a tool that does not lie to you.