A four-person team running LLM experiments against a paid API has a problem nobody puts in the methods section: the number of trials is decided by budget.
You stop early. You skip the ablation. You do not re-run after a small change because re-running costs real money and you have already spent this month's allocation. None of those are scientific decisions, but they all shape the result.
Moving it offline
The whole stack moved to local inference — Ollama running Llama 3.1 8B. Marginal cost per run went to zero.
The immediate effect was that we stopped rationing experiments. The second-order effect mattered more: we started re-running things after small changes, because it was free to do so, and that is when you catch the errors that a single expensive run hides.
What reproducibility actually required
"It runs on my machine" is not reproducibility for a four-person team. What it took:
- One-command setup. If the environment takes an afternoon to stand up, teammates work around it instead, and now you have four slightly different environments producing results you cannot compare.
- Automatic runtime discovery across macOS, Linux, and Windows, because the team was not on one platform.
- A CI matrix on GitHub Actions across 3 operating systems and 2 Python versions, so a break was caught by the pipeline instead of by a teammate losing an afternoon.
Cancelling judge bias by design
The measurement problem was separate. An LLM judge scoring a treatment and a control in separate calls carries its own calibration drift into the comparison — part of the measured difference is the thing you are studying, and part is the judge having a slightly different mood in call two.
The fix was structural rather than statistical: score both legs in a single batched, blinded call returning structured JSON. The judge sees control and treatment in the same context under the same calibration, so its bias lands on both sides and cancels in the delta.
Underneath sits a two-channel RAG pipeline — ChromaDB, MiniLM embeddings, dialogue-phase metadata filters — over 18 technique cards, with 4 matched control/treatment legs per topic.
Scoping around a constraint
Testing the original hypothesis on real people was not safe and not something we were equipped to do responsibly. Rather than proceed carefully, we changed the study: a simulated proxy over non-partisan policy topics, with no personal data involved.
That preserved the research question and removed the risk. Deciding what not to run is part of experiment design, and it is the part that tends to go unwritten.