2024 Apache 2.0
Spider 2.0
~600 real-world tasks
Harder successor to Spider. Real enterprise databases with thousands of tables, multi-step agentic workflows, tool use. Current frontier-model scores are under 20% solve rate.
§ Catalog
A curated list of public benchmarks relevant to agentic analytics: text-to-SQL foundations, data-analysis agents, and general agent suites whose methodology transfers. Links go to the paper and the runnable repo where available.
§ 01 · Text-to-SQL
Translation from natural language to SQL against a known schema. The foundational evaluation surface for analytics agents.
~600 real-world tasks
Harder successor to Spider. Real enterprise databases with thousands of tables, multi-step agentic workflows, tool use. Current frontier-model scores are under 20% solve rate.
12,751 questions · 95 DBs
Large-scale text-to-SQL with noisy real-world data, external knowledge grounding, and efficiency scoring alongside accuracy. Closer to production analytics than Spider.
15,269 perturbations
Robustness diagnostics on Spider: perturbs questions, schemas, and databases across 17 axes. Tests whether a model generalises or overfits to Spider surface patterns.
272 questions · 8 real DBs
Small but high-signal: Kaggle databases with natural, unedited schemas and realistic business questions. Exposes the gap between lab-clean Spider and production data.
3,007 dialogues · 15k questions
Conversational text-to-SQL: multi-turn dialogues with clarification, ambiguity resolution, and context-carrying. Closer to how agents actually interact with users.
4,298 question sequences
Sequential context-dependent SQL. Each turn depends on prior turns. Evaluates whether an agent carries investigation state forward, not just single-turn translation.
10,181 questions · 200 DBs
The foundational cross-domain text-to-SQL benchmark. 200 databases across 138 domains, split so test databases are unseen at train time. Execution accuracy and exact-match metrics.
80,654 questions · 24k tables
The original large-scale NL-to-SQL dataset. Single-table queries from Wikipedia, limited operator vocabulary. Largely saturated by modern models but still cited as a baseline.
§ 02 · Agentic data analysis
End-to-end data analysis: the agent receives a question and a dataset and must plan, run code, evaluate, and produce insights.
93 enterprise questions
Enterprise analytics workflows over real warehouse schemas with many joins, derived columns, and business logic. Designed to expose failure modes that do not show up on Spider-class benchmarks.
12 open-ended datasets
Benchmark for Language models on Data Analysis Expertise. Evaluates decision coverage against a gold set of data-analytic decisions experts would make on the same dataset.
500 tasks
Code-generation benchmark for data analysis agents. Tasks span data wrangling, machine learning, and exploratory analysis. Evaluated by executable correctness over tabular data.
100 end-to-end datasets
End-to-end data analytics: agents receive a business question and a dataset, must form hypotheses, run analyses, and produce insights. Scored with an LLaMA-3 judge against ground-truth insights.
102 tasks · 44 papers
Data-driven scientific discovery tasks extracted from published papers. Agents write code that reproduces figures and findings. Strong test of end-to-end analytical reasoning.
§ 03 · General agent (analytics-adjacent)
Broader agent benchmarks where methodology or a subset of tasks is directly relevant to agentic analytics.
1,100 tasks · 2 domains
Agents with tool use and user simulation across retail and airline domains. Not analytics-specific, but the agent-framework evaluation methodology transfers directly to analytics agents.
8 environments · 1.4k tasks
Multi-environment agent evaluation including database SQL, web shopping, OS commands, and knowledge graph. The database task aligns closely with agentic analytics workloads.
§ Internal work
Alongside tracking the public benchmarks above, we are developing a domain-specific evaluation suite focused on real business analytics workflows: revenue variance decomposition, cohort-level churn analysis, attribution across marketing channels. Work in progress. Once the methodology and reference data are stable and reproducible, it will be listed here like any other benchmark. Until then, the public benchmarks above are the authoritative reference.
If you are running an existing open benchmark against a vendor product and want to contribute results, open an issue or pull request on GitHub.
§ Contribute
Open an issue with the benchmark name, authors, year, size, license, repo URL, paper URL, and a one-line description. Analytics-relevant only: we pass on pure coding benchmarks (HumanEval, SWE-bench) unless they have a dedicated data-analysis slice.