Chain-of-Thought Prompting vs. Fine-Tuning for Table Reasoning

Problem

Chain-of-thought prompting and rationale-trace fine-tuning are both widely assumed to improve table reasoning. That assumption is usually tested on one model at one size against one dataset, which is not enough to support it.

Approach

11 conditions spanning zero-shot, plain and structured CoT prompting, and answer-only versus rationale-trace fine-tuning, run on FLAN-T5 at two sizes (250M and 780M) over WikiTableQuestions with dual seeds. Then a cross-dataset generalization study on TabFact and McNemar's test to check whether any difference was real.

What it measured

Zero-shot scored 0.241 exact match and beat every intervention. Fine-tuned conditions fell below the 0.551 majority-class floor — worse than always guessing the majority label. Errors were attributed across lookup, aggregation, and multi-hop reasoning types.

Result

A negative result, published as one. Fine-tuning overfit to task format rather than learning the reasoning, and the error attribution shows where it broke rather than only that it did. The honest version of this finding is more useful than a marginal positive would have been, because it tells you which intervention not to spend on.

Measured

On the engineer resume

• Benchmarked 11 conditions spanning zero-shot, plain and structured CoT prompting, and answer-only versus rationale-trace fine-tuning on FLAN-T5 transformer models (250M/780M) over WikiTableQuestions with dual seeds, isolating a defensible negative result: zero-shot (0.241 EM) beat every intervention.
• Confirmed it with a TabFact cross-dataset generalization study and McNemar's significance testing, showing fine-tuning overfit to task format and fell below the 0.551 majority-class floor, with error attribution across lookup, aggregation, and multi-hop reasoning.