Writing · September 2026

Pre-registering a reproduction

A recent paper reports that a handful of narrow, misaligned examples in a model's context can shift how it answers questions that have nothing to do with those examples. I wanted to see it myself, and I wanted to commit to what would count as seeing it before I looked.

Why reproduce it at all

The claim is not hard to believe. Models condition on what's in the window; put enough of one thing in there and the behavior moves. What's less obvious is the shape of the effect: how many examples it takes, whether it scales smoothly or has a threshold, whether it holds across model families, and, the part I find most practically important, whether it can be undone.

Existence results are cheap to produce and easy to over-read. Boundary conditions are what you'd actually need to know if you were deciding whether this failure mode matters for a system you run.

The design

The reproduction is a dose–response study. Each trial gives a model 0, 2, 4, 8, or 16 poisoned examples (narrow, misaligned advice in one domain), then asks one of 48 probe questions drawn from unrelated domains. Four frontier models, all accessed through the same interface so the prompt is identical across them.

Responses get coded for misaligned content, and the analysis is a mixed-effects logistic regression with random intercepts for probe question and model. The random effects matter more than they might seem to: without them, a handful of unusually suggestible questions can carry an entire effect estimate.

The extension

The part I actually care about is whether one example pushing the other way is enough to stop it. The extension adds a single aligned, corrective demonstration to a poisoned context and varies where it sits: first, middle, or last.

Position is a real question rather than a detail. If a corrective example only works at the end, that tells you something about how the context is being weighted, and it implies very different mitigations than if one works anywhere.

Why write the hypotheses down first

Because this design has a lot of places to hide. Five dose levels, four models, 48 questions, three insertion positions, and a coding decision on every response; that's more researcher degrees of freedom than I trust myself with after seeing the data. It's easy to decide which dose levels "really" test the hypothesis once you've seen the curve.

So the hypotheses, the coding rules, and the primary analysis were written down and committed before any prompts were run. It costs an afternoon and it makes the eventual numbers mean something. This is the same instinct that shows up in my day job, where an evaluation that wasn't specified in advance tends to produce whatever result the person running it was hoping for.

One honest caveat: the datasets here are reconstructed from public sources rather than the originals, so this is a reproduction in spirit (same question, same design), not a re-run of the exact materials.

Results

To fill in once the runs are finished: the dose–response curve, whether the corrective example blocked the effect, and whether position mattered. Say plainly which pre-registered hypotheses survived and which didn't; the ones that didn't are the more interesting half.

Code, pre-registration, and data are in the repo.