Observability and traceability for analytics agents

Traceability vs. observability

When an analytics agent gives a wrong answer, the damage isn't the wrong number in a slide deck. It's the erosion of trust. A stakeholder who gets burned once by an agent's hallucinated insight will stop trusting agent-generated analysis entirely, and they'll be right to. The core problem is accountability: without instrumentation, you have no way to know what the agent did, why it did it, or whether the answer is defensible.

The harder truth: without observability, you often won't even know the answer was wrong. A plausible-sounding wrong answer is the most dangerous kind. If an agent says churn spiked because of a pricing change, and the real cause was a data quality issue (rows duplicated during an ETL migration) nobody questions it until weeks later when the pricing rollback doesn't move the needle. By then, a bad decision has already been made.

Traceability is what prevents this. A trace file for that investigation would show exactly which tables the agent queried, whether it validated row counts or checked for duplicates, and where it jumped from correlation to causation. In this example, the trace would reveal that the agent never ran a data quality check on the source table. It went straight from "churn is up" to "pricing is the cause" without validating the underlying data. That's a reviewable, fixable gap.

Observability is the broader view. It tells you how your agents are performing across many investigations. Are they getting faster? Producing fewer errors? Which question types are they struggling with? You can't improve what you can't measure.

These are related but distinct. Build traceability first. Observability is downstream of it: once you have structured trace files, most observability metrics fall out naturally. If you've already built specialized analytics agents, adding traceability is the natural next step. And if you're using BigQuery, the job label patterns below build on the connection setup covered in the BigQuery guide.

Trace files

The foundation of traceability is a structured log that the agent writes as it works. Make it a hard requirement in every agent's instructions, not an optional add-on.

## Traceability
Every investigation MUST produce a trace file at:
  ./traces/{timestamp}-{slug}.json

The trace file must contain:
{
  "question": "the original user question",
  "started_at": "ISO timestamp",
  "completed_at": "ISO timestamp",
  "steps": [
    {
      "step": 1,
      "action": "query | hypothesis | validation | summary",
      "description": "what this step is doing and why",
      "sql": "the query if applicable",
      "result_summary": "key numbers extracted from the result",
      "finding": "what was learned from this step",
      "next_action_reason": "why the next step was chosen"
    }
  ],
  "final_answer": "the answer presented to the user",
  "confidence": "high | medium | low",
  "caveats": ["list of known limitations or assumptions"],
  "data_freshness": "most recent date in the queried data"
}

This trace file is often more valuable than the answer itself. It lets you:

• Debug wrong answers by replaying the exact query sequence
• Identify where the agent's reasoning went off track
• Build a dataset of good and bad investigations for evaluation
• Show stakeholders how the answer was derived, which matters for trust

The next_action_reason field deserves attention. It forces the agent to articulate why it's doing what it's doing, which surfaces flawed reasoning at the step level rather than only when the final answer is wrong.

But the trace file is a summary of what happened. For full reproducibility, you need the actual artifacts the agent produced along the way.

Saving artifacts to disk

A trace file tells you what the agent did. The artifacts tell you exactly how. Every query, every intermediate output, every transformation script the agent wrote during an investigation should be saved to disk as a first-class artifact. This is the single most important traceability practice: if you can't replay an investigation from its saved artifacts alone, your traceability has gaps.

Instruct the agent to write each SQL query to its own file, named by step number. Save query results as CSVs or JSON. If the agent generates Python scripts, pandas transformations, or Jupyter notebooks as part of the analysis, those go to disk too. The goal is a self-contained directory that fully reconstructs the investigation without needing to re-run anything.

## Artifacts
Every investigation MUST save all artifacts to:
  ./traces/{timestamp}-{slug}/

Directory structure:
  trace.json                    # the trace file
  queries/
    step-01-revenue-by-segment.sql
    step-02-churn-by-tier.sql
    step-03-validation-check.sql
  results/
    step-01-revenue-by-segment.csv
    step-02-churn-by-tier.csv
  scripts/
    step-04-cohort-transform.py  # if pandas/python was used
    step-05-forecast.ipynb       # if a notebook was generated
  outputs/
    final-summary.md             # the delivered answer
    charts/                      # any generated visualizations

Rules:
- Save every SQL query to disk BEFORE executing it
- Save query results as CSV (truncate to 1000 rows if larger)
- Save any Python/R scripts, notebooks, or transformation code
- Save the final output in the format delivered to the user
- Never delete artifacts from a completed investigation

This approach has several advantages over trace-file-only logging. You can re-run any individual query to verify it still produces the same result. You can diff the SQL files across investigations to see how the agent's query strategy evolved. When a stakeholder questions a number, you can point them to the exact CSV and the exact query that produced it. And when the agent generates Python scripts or notebooks for statistical analysis, forecasting, or data transformations that go beyond SQL, those scripts become reviewable, testable code rather than ephemeral computations lost to the context window.

The disk footprint is small. A typical investigation produces 5 to 15 query files, a handful of CSVs, and occasionally a script or two. Measured in kilobytes, not gigabytes. The cost of saving everything is negligible; the cost of not having it when you need to debug a wrong answer is significant.

Query tagging for cost and usage monitoring

Saving queries to disk gives you full traceability. But for cost monitoring and usage observability across many investigations, you also need instrumentation at the warehouse level. BigQuery job labels let you attribute every query to the agent and investigation that ran it, which is essential for tracking spend.

## Query labels
Every query must include these labels:
  --label=agent:{agent-type}        # e.g. revenue-analyst
  --label=trace:{trace-id}          # e.g. 20260407-q1-revenue-drop
  --label=step:{step-number}        # e.g. 3

The trace-id must match the filename of the trace file for
this investigation.

bq query --use_legacy_sql=false \
  --label=agent:revenue-analyst \
  --label=trace:20260407-q1-revenue-drop \
  --label=step:3 \
  --maximum_bytes_billed=2000000000 \
  'SELECT segment, SUM(amount_cents)/100 as revenue
   FROM `my-project.revenue.invoices`
   WHERE date BETWEEN "2026-01-01" AND "2026-03-31"
   GROUP BY segment ORDER BY revenue DESC'

Once queries are labeled, you can query INFORMATION_SCHEMA.JOBS filtered by trace ID to see every query the agent ran for a specific investigation, with bytes scanned and execution time. This closes the loop between the saved artifacts on disk (the exact queries and results) and the warehouse's own record of what executed and what it cost.

Git trace history

With trace files, saved artifacts, and query labels in place, you have the raw material. The next question is where to store it. Committing the entire investigation directory to a git repository gives you a versioned history of every investigation with no additional infrastructure.

# After each investigation, commit the full artifact directory
git add traces/
git commit -m "trace: Q1 revenue variance investigation"

This gives you git log traces/ as an audit trail and git diff to compare how the agent approached similar questions over time. You can diff the SQL files between two investigations of the same metric to see whether the agent's query strategy improved. You can review the Python scripts it generated and catch issues like incorrect join logic or missing null handling in transformations. The artifact directory makes every investigation a reviewable, self-contained package.

The review angle matters. A PR review on an investigation's artifacts is a surprisingly effective way to catch systematic errors before they become patterns. Over time, a team can develop shared standards for what good agent investigations look like by reviewing traces the same way they review code. Did the agent validate data freshness before analyzing? Did it check for obvious data quality issues? Did it consider alternative explanations? Did the Python transformation handle edge cases? These become reviewable expectations, not just hopes.

Observability metrics

Traceability gives you depth on individual investigations. You also need breadth: aggregate signals that tell you how the system is performing overall. These five metrics cover the most important dimensions.

BigQuery native monitoring

Knowing what to measure is one thing; actually measuring it without building a custom observability platform is another. The good news is you don't need a separate observability stack. BigQuery's INFORMATION_SCHEMA views capture everything you need about query execution. Combined with the job labels from your agents, this gives you per-agent cost, latency, and error rate from data BigQuery already collects.

-- Agent cost and performance over the last 30 days
SELECT
  (SELECT l.value FROM UNNEST(labels) l WHERE l.key = 'agent') as agent_type,
  COUNT(*) as total_queries,
  SUM(total_bytes_processed) / POW(1024, 3) as total_gb_scanned,
  SUM(total_bytes_processed) / POW(1024, 4) * 6.25 as estimated_cost_usd,
  AVG(TIMESTAMP_DIFF(end_time, start_time, MILLISECOND)) as avg_query_ms,
  COUNTIF(error_result IS NOT NULL) as error_count
FROM `region-us`.INFORMATION_SCHEMA.JOBS
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND EXISTS (SELECT 1 FROM UNNEST(labels) l WHERE l.key = 'agent')
GROUP BY agent_type
ORDER BY total_gb_scanned DESC

If your agents write structured trace files, load them into BigQuery to add the investigation-level metrics: step counts, confidence scores, question categories.

-- Average steps and duration per investigation
SELECT
  COUNT(*) as investigations,
  AVG(JSON_ARRAY_LENGTH(JSON_QUERY(trace, '$.steps'))) as avg_steps,
  AVG(TIMESTAMP_DIFF(
    PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', JSON_VALUE(trace, '$.completed_at')),
    PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%S', JSON_VALUE(trace, '$.started_at')),
    SECOND
  )) as avg_duration_seconds,
  COUNTIF(JSON_VALUE(trace, '$.confidence') = 'high') as high_confidence_count
FROM `my-project.observability.agent_traces`
WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)

Schedule both queries as daily jobs and route the results to a simple dashboard or a Slack notification. This is enough to catch most problems before they affect stakeholders.

Alerting

Dashboards are useful for periodic review, but the failure modes that matter most need to interrupt you. Four alerts cover the failure modes that actually happen:

• Cost spike. If any agent scans more than 10x its daily average, something is wrong: a missing date filter, a schema change that broke a partition, or the agent looping on a stuck investigation. Catch this before the bill arrives.
• Error rate increase. A sudden jump in query errors usually means a schema change, a permission revocation, or a table being deleted or renamed. The warm-up phase should catch most of these, but an alert on the INFORMATION_SCHEMA error count is a useful second layer.
• Runaway query count. If an investigation exceeds 25 queries, the agent is likely stuck in a loop. This is the most common failure mode with agentic systems: the agent keeps trying variations of a query that will never return the right answer because the underlying assumption is wrong.
• Stale data. If the warm-up freshness check returns a max date more than 2 days older than expected, the agent should stop and report the issue rather than analyzing stale data confidently.

-- Flag investigations with more than 25 queries in the last 24h
SELECT
  JSON_VALUE(trace, '$.question') as question,
  JSON_ARRAY_LENGTH(JSON_QUERY(trace, '$.steps')) as step_count,
  JSON_VALUE(trace, '$.started_at') as started_at
FROM `my-project.observability.agent_traces`
WHERE date = CURRENT_DATE()
  AND JSON_ARRAY_LENGTH(JSON_QUERY(trace, '$.steps')) > 25
ORDER BY step_count DESC

Not all of these alerts deserve the same urgency. Cost spikes and runaway queries need immediate attention because they represent active, escalating problems. Error rate increases are important but can usually wait for a morning review unless the rate is extreme. Stale data warnings should be batched into a daily digest. If you alert on everything with the same priority, you'll get alert fatigue, and a team that ignores alerts is worse off than a team with no alerts at all. Start with fewer alerts at higher thresholds and tighten them as you learn what normal looks like for your agents.

Building an observability practice

Setting up trace files, query labels, and a few alerts is the starting line, not the finish. Observability for analytics agents is an ongoing practice, not a one-time configuration.

The trace files and metrics you collect become the basis for improving the agent over time. After a few weeks of accumulated traces, patterns emerge. You'll see which types of questions the agent handles well and which it struggles with. You'll notice recurring failure modes: the agent consistently fails to check data freshness on one particular dataset, or it over-indexes on recent data when asked about trends. These patterns inform specific, targeted improvements to the agent's instructions, its available tools, or its validation steps.

Observability also pairs naturally with context management. The context you give an agent determines what it can do; observability tells you what it actually did. If your traces keep showing the agent making avoidable mistakes, the fix is often better context (more explicit instructions, better schema documentation, or tighter guardrails) rather than more monitoring.

If you're starting from scratch, the specialized agents guide covers the foundations: how to scope an agent's domain, structure its instructions, and connect it to your data. Add traceability from the beginning. It's far easier to instrument an agent on day one than to retrofit it after it's already running in production and people depend on its outputs.