Beyond the Baseline / Model evaluation · Table reasoning
What actually beat the baseline?
None of the tested interventions beat the strongest zero-shot baseline. Explore the prompting and fine-tuning conditions, and the separate transfer experiment.
My contributionBenchmarked prompting and fine-tuning, tested cross-dataset transfer, and analyzed prediction failures.
11 configurations · 2 seeds · One model family
Explore the project
01 / Testing the extra complexity
What actually beat the baseline?
Nothing in the tested interventions surpassed zero-shot FLAN-T5-large on WikiTableQuestions. Explore all eleven conditions, keeping the two tasks separate.
Eight conditions · answer strings must match the reference.
Dashed marker: best recorded baseline for this dataset.
Selected condition
Zero-shot baseline
- Model
- flan-t5-large
- Exact match
- 0.241
- Observed spread
- ± 0.009
Recorded spread across two seeds; observed variation, not a confidence interval.
The strongest zero-shot baseline remains visible as a reference for every condition. This result applies to the tested model family and setup, not fine-tuning in general.
Read the full results table ↗- Best WikiTableQuestions result
- 0.241
- Best TabFact result
- 0.043
exact match, zero-shot FLAN-T5-large — no intervention beat it
summary_table.md ↗classification accuracy, a different measurement from exact match
summary_table.md ↗Follow the evidence
Source revision · 0f4cbed905b24b962ea7db017053974fd3f8a1a2
Scope & limitations
- Two seeds per condition. The published spread is observed variation, not a confidence interval.
- TabFact accuracies between 0.002 and 0.043 sit far below the 0.551 majority-class floor. A near-zero score of this shape indicates the model's output does not match the label format, which is an answer-format failure rather than a narrow reasoning loss.
- One model family at two sizes. The result does not generalize to other models or to fine-tuning in general.
- Exact match and classification accuracy are not comparable and are never plotted on one axis.
The engineering decision
Keep the baseline visible, compare the same tasks and metrics, and inspect prediction formats before interpreting a low score as a reasoning failure.
What the evidence means
The conclusion is limited to these models and protocols. WikiTableQuestions exact match and TabFact classification accuracy answer different questions and remain separate.
Implementation notes
Eleven conditions across zero-shot, chain-of-thought prompting, and two styles of fine-tuning, run on FLAN-T5 at two sizes with dual seeds. Zero-shot scored 0.241 exact match on WikiTableQuestions and beat everything else. The TabFact generalization study is a separate measurement — classification accuracy, not exact match — and its near-zero results point at answer-format failure rather than a narrow reasoning loss.