Interrogating Agents / Agent evaluation · Team research
How do you make the comparison fairer?
I built a retrieval and evaluation harness that compares matched conversations under shared judge context. The experimental design is inspectable; its treatment effect is withheld.
My contributionBuilt the two-channel retrieval harness and owned reproducibility for a four-person team.
4 experiment legs · 2 direction-matched judge batches
Explore the project
Explore the experiment
How do you make the comparison fairer?
Run control and treatment from each starting direction. Score each matched pair together, so both receive the same judge calibration.
Leg 0 · Control
No technique intervention
Leg 2 · Treatment
Technique-guided intervention
Judge batch A
Leg 1 · Control
No technique intervention
Leg 3 · Treatment
Technique-guided intervention
Judge batch B
One call per direction. Control and treatment share calibration within their batch; the two batches do not share calibration.
Proposition + numbered utterances
The scoring input contains the proposition and numbered utterances. It withholds the control/treatment condition labels. This diagram shows the input structure, not a recorded conversation.
- Proposition
- 1. Utterance
- 2. Utterance
- … Remaining utterances
Two retrieval channels. Different jobs.
Topic knowledge
Retrieves material relevant to the proposition.
Technique cards
Retrieves intervention guidance from a library of 18 technique cards.
- Judge calls per unit
- 2
one per starting direction, each holding control and treatment together
quad.py ↗Follow the evidence
Source revision · a65c450b0ac1f6d3c7cb76c7ba7ac6c2a327430c
Scope & limitations
- Control and treatment share a call, so one calibration applies to both. Comparisons across the two starting directions are not calibration-matched.
- A single local 8B judge. Order effects within a batch are untested.
- On a parse failure the judge is re-prompted with its own prior reply, which is not an independent retry.
- The saved summary is internally inconsistent — it labels 3 x 5 x 4 as 68, reports a pooled control accuracy of 42% against per-topic values averaging 40%, and cites a housing control figure its own table contradicts. No treatment effect is published from it.
- A simulated proxy cannot establish an effect on real people; the study was scoped this way because testing the hypothesis on real users was unsafe.
The engineering decision
Score control and treatment together within each starting direction, with condition labels withheld from the judge. Two batches bound the context while keeping direction-matched comparisons together.
What the evidence means
Shared calibration within a batch does not eliminate all judge bias. The saved results summary contains conflicting totals and percentages, so this project makes no treatment-effect claim or claim about real people.
Implementation notes
A two-channel RAG harness over 18 technique cards, scoring 4 matched legs per experimental unit in 2 direction-matched judge batches. The judge receives a proposition and numbered utterances and never the condition label, so control and treatment are scored under one calibration. The saved run summary contradicts itself, so this publishes the design and withholds the effect.