Chapter 8.1 — Execution & the Specification Curve

Maya filed her pre-analysis plan two weeks ago. It sits in her repository as a tagged commit, pap-filed, time-stamped and frozen, with one primary specification written out in full spec-discipline form and a deviation log waiting empty beside it. The hard thinking is done. What remains, this week, is the part that feels like it should be easy: run the thing. Open the notebook, point it at the confirmation data, execute the one regression she promised, and read the number off the table.

It is easy. It is also where most honest projects quietly go wrong, because the moment a result appears on the screen, the discipline of Week 7 stops being a document and starts being a test of character. The PAP told Maya which regression to run. It did not, and could not, run it for her — and it cannot stop her from running thirty more once she has seen the first. This chapter is about that moment. It teaches you to execute the pre-registered confirmatory result exactly as filed and report it whatever it shows; then to take the garden of forking paths you wrote down in Chapter 7.3 and make it visible — to run not one specification but every defensible one, plot the whole distribution of estimates, and read off whether your headline is a robust feature of the data or a lucky branch in a forest of paths. The instrument that does this is the specification curve, and it is the single most honest picture you can put in an empirical paper.

We follow the book’s reveal-the-trick structure. State what the confirmatory run is and run it; state what the garden of forking paths costs you and make it visible with a multiverse of specifications; learn to read the resulting curve honestly — fragile or robust?; and see when the curve itself lies. The intellectual frame is Simonsohn, Simmons, and Nelson’s specification-curve analysis.

Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification Curve Analysis. Nature Human Behaviour, 4(11), 1208–1214.

A boundary first, so you do not go looking for the wrong thing here. This chapter is about analytic-choice multiplicity — the fact that one dataset and one hypothesis admit many defensible regressions, and that the choice among them is a hidden multiple-testing problem. The next chapter, 8.2, is about inference validity — whether the one design you chose actually identifies what you claim, stress-tested with placebos and Oster’s \delta. They are different anxieties. The spec curve asks “does my result survive the choices I could defensibly have made differently?” The robustness tests of 8.2 ask “is the design I committed to even the right design?” Do not conflate them; this week pulls them apart on purpose.

8.1.1 The confirmatory run: execute exactly what you filed, report whatever it shows

Here is the result in one sentence: your first job in Week 8 is to run the pre-registered primary specification once, on the confirmation data, exactly as written in the PAP — no edits, no peeking-then-tweaking — and to report the number it produces no matter which way it falls.

That sounds trivial. It is the hardest professional habit in the entire course, and the reason is psychological, not technical. Go back to Chapter 1.5 and the logic of what a p-value means. You proved there that under a true null the p-value is uniformly distributed on [0,1], so a single pre-specified test at \alpha=0.05 rejects a true null exactly 5% of the time — that 5% is the contract you signed. The contract only holds if the test was fixed before you saw the outcome. The instant you run a regression, look at the coefficient, dislike it, and adjust the specification to improve it, you have re-entered the garden of forking paths of Chapter 7.3, and your p-value no longer means 5%. It means something larger and unknowable, inflated by every fork you walked with the answer in view. The PAP froze the forks. The confirmatory run is the act of honoring that freeze.

So the mechanics are deliberately rigid. Maya’s pap-filed commit named, in CONVENTIONS §4 form, a single staggered difference-in-differences: outcome the county-year minority–white denial gap; treatment a state’s active fair-lending examination program; never-adopters and not-yet-treated states as clean controls; county and year fixed effects; standard errors clustered by state; the sample pinned to a fixed HMDA vintage. Her notebook, notebooks/week-07/nb7.5-first-look-regressions.ipynb, is frozen behind a gate — a flag PAP_FILED=True that must be asserted before the analysis cell will run, the technological enforcement of “do not peek until the plan is filed.” When she ran it, exactly once, the pre-registered estimator returned

\widehat{\text{ATT}} = -1.89 \text{ percentage points}, \qquad \text{SE}_{\text{clustered by state}} = 0.065, \qquad 95\%\ \text{CI} = [-2.02,\ -1.75].

The examination programs are associated with a roughly 1.9-percentage-point narrowing of the denial gap, the sign she hypothesized, with a confidence interval comfortably away from zero. That is the confirmatory result. It is the number the rest of the paper defends, and it is the number she reports in her headline table — full stop, with the clustering flavor labeled and the CI shown.

Now the part that separates a researcher from a result-shopper. Maya must commit, in advance, to reporting that number whichever way it came out. Suppose the confirmatory run had returned \widehat{\text{ATT}} = -0.3 pp with a CI of [-1.9,\ +1.3] — a wide interval straddling zero. Her PAP’s falsification clause already told her what to do: a wide CI that includes zero is a power failure, reported as inconclusive, not as a refutation; a tight CI hugging zero would be a genuine, publishable null about fair-lending policy. Either way she reports it. The temptation — and it is enormous, because she has spent five weeks on this and wants the effect to be real — is to treat a disappointing confirmatory number as a problem to be debugged rather than a result to be reported. That instinct is the disease. The whole architecture of Week 7 exists to make the confirmatory number binding: you wrote down what it would mean before you knew what it would be, precisely so that knowing what it is cannot change what it means.

There is a subtle trap even inside an honest confirmatory run, and it is worth naming because beginners fall into it constantly. The trap is mistaking a clean execution for a robust result. Maya ran exactly one specification and got a starred coefficient. That is necessary but not sufficient for a credible finding, because the one specification she ran was one branch of a tree she could defensibly have climbed differently. Her PAP committed her to clustering by state — but clustering by county was also defensible. It committed her to county and year fixed effects — but adding state-specific linear trends was also defensible. It committed her to a particular sample window — but trimming the COVID-disrupted year 2020 was also defensible. Each of those alternatives is a fork she chose not to walk, and a reader is entitled to ask: if you had walked them, would the -1.9 still be there, or would it dissolve? The confirmatory run answers “what does the pre-registered specification say?” It does not answer “is the answer an artifact of that one specification?” For that, you need to make the whole garden visible. That is the rest of this chapter.

8.1.2 The garden of forking paths, made visible

You met the garden of forking paths in Chapter 7.3 as a threat: the reason an honest researcher can p-hack without a single dishonest thought, by making a sequence of reasonable, outcome-aware choices that each nudge the coefficient the desired way. The PAP’s defense was to nail down the forks in advance so they collapse to one confirmatory test. That defense protects your headline. But it leaves a question hanging that the PAP cannot answer, because the PAP is a single path through the garden: how many of the other defensible paths would have given you the same answer?

The specification curve, due to Simonsohn, Simmons, and Nelson (2020), answers exactly that question by refusing to hide. Their insight is to invert the usual sin. The p-hacker runs many specifications and reports the one that looks best, hiding the rest. The specification-curve analyst runs many specifications — ideally all the defensible ones — and reports all of them at once, as a distribution. Where p-hacking exploits multiplicity by concealment, specification-curve analysis defangs multiplicity by exhaustive disclosure. If your result is a robust feature of the data, it will be there across most of the forest; if it is a lucky branch, the picture will say so loudly, and you will have caught yourself before a referee does.

The procedure has three steps, and the discipline is entirely in the first one.

Step 1 — Enumerate the defensible analytic choices. This is the act of writing the garden down. You list, for each decision in your analysis, every option a reasonable, competent researcher could have chosen — and, crucially, only the defensible ones. The point is not to include absurd specifications to pad the count; a specification you would never defend in a seminar does not belong in the multiverse, because including it would let you smuggle in noise and then hide behind “but the median was fine.” The choices for Maya’s HMDA DiD partition naturally into the seven slots of CONVENTIONS §4:

Controls. Which county-year applicant-composition covariates enter? She could include the income distribution, the loan-to-value distribution, the loan-amount distribution — or any subset. Reasonable researchers disagree about how many composition controls a doubly-robust DiD needs. Call this 4 defensible combinations (none, income only, income+LTV, all three).
Fixed effects. County and year FE are the baseline. Does she also add state-specific linear time trends (a common parallel-trends hedge)? With or without: 2 options.
Sample window. Full period; drop 2020 (COVID); drop the first and last adoption cohorts (thinly identified): 3 options.
Clustering / inference. Cluster by state (her pre-registered choice); cluster by county; wild-cluster bootstrap by state (defensible when adopting states are few, per Chapter 4.1): 3 options.
Outcome definition. The denial gap (minority denial rate minus white denial rate); or two separate outcomes — the minority denial level and a within-applicant denial indicator; the gap measured as a difference vs. a log-ratio: say 2 defensible outcome operationalizations.

Multiply them: 4 \times 2 \times 3 \times 3 \times 2 = 144 specifications. Each is a single point in what the multiverse literature calls the garden of forking paths made into a grid — every combination of every defensible choice, laid out so none can hide. Notice that one of those 144 is her pre-registered primary specification. The confirmatory run is not outside the multiverse; it is a labeled, privileged point inside it.

Here the arithmetic of Week 1 comes back to bite, and it is the reason this whole exercise matters. Recall from Chapter 1.5 that if you test 20 independent true nulls at \alpha=0.05, the chance at least one crosses the line by luck is 1-(1-0.05)^{20}\approx 0.64 — a coin-flip-beating probability of a spurious “winner” purely from giving luck enough tries. With 144 specifications, even if the true effect were exactly zero, the probability that at least one lands at p<0.05 is overwhelming. The garden of forking paths is just multiple testing wearing the disguise of “I only ran one regression.” Maya did only run one regression for the confirmatory result — but the 144 specifications measure how many regressions she could defensibly have run, which is the real size of her multiple-testing problem. The specification curve does not eliminate that problem. It does something better: it shows you the whole distribution of what luck and choice together could produce, so a single starred specification can never again masquerade as the only one you ever considered.

Step 2 — Run all of them. Estimate the key coefficient — for Maya, the ATT on the denial gap — under every one of the 144 specifications, and collect the estimates and their confidence intervals into a table. This is mechanical and embarrassingly fast on modern hardware; 144 fixed-effects regressions on a county-year panel run in seconds. The output is a vector of 144 estimates and 144 inference results, plus, for each one, a record of which choice was made in each slot. That record is the load-bearing part, as Step 3 reveals.

Step 3 — Plot the distribution, with the choices underneath. The specification curve is a two-panel figure that has become the signature of this method, and once you have read one you cannot unsee it. We build it next.

8.1.3 Reading the specification curve

The figure has two stacked panels that share a horizontal axis. The horizontal axis runs over all 144 specifications, sorted by the magnitude of the estimate — most negative on the left, most positive on the right.

The top panel plots the 144 point estimates as a curve (hence the name), each with a marker for its confidence interval — colored or filled when the specification is statistically significant at your chosen level, hollow when it is not. Because the specifications are sorted by estimate, the curve rises monotonically from left to right; its shape is the whole story. A flat curve sitting entirely below zero, with nearly every marker filled, says: every defensible path gives roughly the same negative answer, and almost all are significant. A curve that sweeps from strongly negative through zero up to strongly positive, with a mix of filled and hollow markers, says: the sign and significance of my result depend entirely on which fork I walk — a result that is an artifact of analytic choice, not a feature of the world.

The bottom panel is the part that turns a pretty picture into a diagnostic. It is a grid: one row per analytic choice (cluster-by-state, cluster-by-county, drop-2020, add-state-trends, income-control-only, …), and one column per specification, aligned to the top panel’s sort order. A tick mark sits at (\text{row}, \text{column}) when that specification used that choice. So you read downward from any interesting region of the curve to see exactly which choices produced it. If all the positive, insignificant specifications on the right share a single tick — say, every one of them used “cluster by county” plus “drop 2020” — then you have learned something precise and damning: the result’s fragility is driven by those two choices. The bottom panel converts “my result is fragile” from a vague worry into a forensic finding: this specific fork is the one that flips my sign.

Let us read Maya’s curve concretely, because the reading is the skill. Suppose her 144 specifications produce a curve like this: the estimates run from about -2.4 pp on the left to about -0.4 pp on the right; the median is around -1.8 pp; about 132 of the 144 are significant at the 5% level (filled), and the dozen hollow ones cluster at the right end near zero. Reading downward from those dozen non-significant specifications, the bottom panel shows they all share two choices: the wild-cluster bootstrap (which widens intervals when adopters are few) combined with the “drop first and last cohorts” sample (which throws away identifying variation). That is an honest, interpretable picture, and here is exactly what Maya should write about it:

Across all 144 defensible specifications, the estimated effect of fair-lending examinations on the minority–white denial gap is negative, with a median of -1.8 pp and a range of [-2.4, -0.4] pp. The pre-registered primary specification (-1.9 pp) sits near the center of this distribution. The effect is statistically significant in 132 of 144 specifications (92%); the twelve exceptions combine the wild-cluster bootstrap with the most aggressive sample trim, which jointly maximize the standard error while minimizing identifying variation — i.e., the result loses significance only in the lowest-power corner of the multiverse, and never changes sign.

That is a robust result, and the curve is what licenses the word. The sign never flips; the magnitude is stable around the pre-registered estimate; loss of significance is confined to specifications that are independently the least powerful, for reasons the bottom panel makes legible. A referee reading that paragraph and that figure cannot accuse Maya of cherry-picking, because she has shown them the entire orchard and pointed to exactly which trees bear less fruit and why.

Now contrast the picture that should terrify you, and would have, had Sam built a specification curve for his momentum anomaly. Sam’s project from Week 7 asks whether past-12-month winners outperform losers next month — a portfolio sort with a long menu of defensible forks: the formation window (12 months, or 12 skipping the most recent, or 6); the holding period (1 month, 3, 6); value- versus equal-weighting; NYSE breakpoints versus all-stock breakpoints; including or excluding microcaps and penny stocks; the sample period (full history, or post-2000, or excluding 2008–09). Those forks easily multiply past 200 specifications. Suppose Sam’s curve sweeps from a significantly negative long-short return on the left, through a fat band of insignificant near-zero estimates in the middle, up to a significantly positive return on the right — and the bottom panel reveals that every significantly positive specification shares one choice: “include microcaps, equal-weighted.” Reading downward tells the whole story in one glance: the anomaly is not a feature of the cross-section of returns; it is a feature of how you weight tiny, illiquid, hard-to-trade stocks. The headline “momentum earns 80 bps a month” was a single branch — the equal-weighted-microcap branch — and the curve drags it into the light. Sam’s honest sentence is the mirror image of Maya’s: the sign of the momentum premium is not robust to the weighting and universe choices; it is positive and significant only under equal-weighting with microcaps included, and is statistically indistinguishable from zero under the value-weighted, liquid-universe specifications that a tradable strategy would actually face. That sentence kills the easy paper and replaces it with a true and far more interesting one.

The general reading rule, then, is a short checklist you apply to every specification curve you build or referee:

Where does the sign live? If estimates span both signs, your effect’s direction is choice-dependent — the most serious possible fragility. If they all share a sign, the direction is robust even if the magnitude wanders.
Where does the pre-registered specification sit? Mark it on the curve. If your confirmatory estimate sits near the center of the distribution, your headline is representative. If it sits at an extreme tail, you must explain why — and “I happened to pre-register the best-looking corner” is not a flattering explanation, even when innocent.
What fraction is significant, and where do the exceptions cluster? Read the hollow markers downward into the bottom panel. Significance lost in the low-power corner (small samples, conservative inference) is benign; significance that appears only under one or two specific choices is a red flag pointing straight at those choices.
Which rows of the bottom panel drive the spread? The choice whose ticks best separate the left of the curve from the right is the fork your result is most sensitive to. Name it. It is the thing a skeptic will attack, and the thing your robustness section (8.2) must address head-on.

A word on the formal inference that Simonsohn, Simmons, and Nelson attach to the curve, kept light because the picture is most of the value. They propose a permutation test on the whole curve: under the sharp null that the effect is zero everywhere, you re-shuffle the treatment assignment, recompute all 144 specifications on each shuffle, and ask whether the observed curve — for instance, the number of significant-and-correctly-signed specifications, or the median estimate — is more extreme than what the null curves produce. This tests the multiverse as a whole rather than any single specification, which is precisely the right unit of analysis once you admit that “which specification” was itself a choice. For a high-school capstone, computing the descriptive curve and reading it with the checklist above is the required deliverable; the permutation test is an extension in nb8.1 for those who want the formal p-value on the entire forest. [CHECK: confirm the exact page within Simonsohn, Simmons & Nelson (2020) where the joint permutation-inference procedure is stated, for a pinpoint cite.]

8.1.4 When the specification curve lies

Reveal-the-trick demands the third move: every technique has an assumption that breaks, and a picture this persuasive is dangerous precisely because it is persuasive. A specification curve can mislead in four distinct ways, and an honest analyst flags which ones apply.

The curve only contains the forks you chose to enumerate. This is the deepest limitation and the easiest to forget once the beautiful figure is on the page. A specification curve is exhaustive over the choices you listed and silent about every choice you did not think to vary. If Maya’s entire multiverse holds the outcome definition nearly fixed — if all 144 specifications quietly assume the denial gap is the right outcome and never test the within-applicant denial indicator — then her curve can look gloriously robust while resting on one unexamined fork. The curve’s robustness is conditional on the completeness of the enumeration, and completeness is a judgment call, not a guarantee. The defense is to enumerate the forks honestly and broadly, and to state in the paper which dimensions you held fixed and why — exactly the spec-discipline of CONVENTIONS §4, now applied to the meta-decision of what to vary.

Not all specifications are equally defensible, and the curve pretends they are. A naive reading treats all 144 points as equal votes, so that “the median is negative” sounds like a democratic verdict. But some of those specifications are better than others — clustering at the level treatment varies (state) is more defensible than clustering by county; a doubly-robust DiD with composition controls is better-identified than one without. If you pad the multiverse with weak-but-arguable specifications, you can drag the median wherever you like, which is p-hacking with extra steps and a fancier chart. Simonsohn, Simmons, and Nelson are explicit that the choices must be theoretically justified and statistically valid — not merely “things you could type into the regression.” The honest move is to include only specifications you would defend out loud, and, where they genuinely differ in quality, to say so rather than hide behind the median.

The curve describes analytic-choice fragility, not identification. A specification curve can be flat, tight, all-significant, and completely wrong, because every one of its 144 specifications can share the same fatal identification flaw. If Maya’s parallel-trends assumption fails — if adopting states were already on a different denial-gap trajectory — then all 144 specifications inherit that bias, and the curve will robustly report a beautifully stable artifact. Robustness across the multiverse is not the same as validity of the design. This is the boundary from this chapter’s opening, restated as a warning: the spec curve answers “did my choices drive the result?” and is mute on “did my design identify the effect?” The second question is Chapter 8.2’s job, with the placebos and Oster-\delta stress tests that probe the assumption itself. A robust spec curve on an invalid design is a confident wrong answer — the most expensive kind.

The curve can become a fishing license if you let it. There is a perverse failure mode where a researcher runs the multiverse, then picks the specification at the center of the distribution and reports that as the headline — “I chose the median specification, the most representative one.” But the median was determined by the data you have already seen, so choosing it after the fact is just outcome-driven specification selection in a lab coat. The discipline that prevents this is the one you already have: the PAP fixes the primary specification in advance, and the spec curve is reported around that pre-registered point, not used to choose it. The confirmatory run comes first and is privileged; the curve is context for it, never a substitute. This is why the order in this chapter is not negotiable — confirmatory result in §8.1.1, multiverse in §8.1.2 — and why the curve always marks where the pre-registered specification sits.

The summary is the one Simonsohn, Simmons, and Nelson themselves press: a specification curve is a tool for honesty about analytic-choice multiplicity, and it is extraordinarily good at that one job. It is not a robustness panacea, not a substitute for a credible identifying assumption, and not a way to launder a post-hoc choice into a pre-registered one. Use it for what it is — the most complete possible disclosure of the garden of forking paths — and say plainly what it does not do.

8.1.5 The honest deviations log

You filed a PAP. You ran the confirmatory specification. You built the spec curve. Somewhere in that process — it is nearly inevitable, as Chapter 7.3 warned via Olken’s perils — reality forced you off the plan. A control you swore you would include turned out to be collinear with a fixed effect and dropped out. A clustering variable was corrupted for 4% of rows. The COVID year behaved so strangely you had to handle it differently than you guessed. The deviations log is the running, dated record of every departure from the PAP — what changed, why, and, the crucial part, whether the change was prompted by seeing the outcome.

Deviating is not the sin; Week 7 said this and it bears repeating now that you are actually doing the analysis and the temptation is live rather than hypothetical. Hiding the deviation is the sin, and outcome-driven deviation is the specific sin the whole apparatus exists to catch. So every entry in the log carries a verdict on one binary question: was this change forced on me by a mechanical reality, or did I make it because I saw the coefficient and did not like it?

A deviation forced by a mechanical reality — collinearity, a coding bug, a data-availability surprise, a cluster identifier that was corrupted — is innocent. You note it, you fix it, you move on. It is not outcome-driven; nothing about the result steered the change. Example entry: “PAP §2 specified clustering by state. On the confirmation data, the state identifier was missing for 4% of county-year cells due to a FIPS-merge gap. We clustered by the state-equivalent census region for those cells and re-ran; the ATT moved from -1.89 to -1.91 pp, immaterial. Not outcome-driven.”
A deviation prompted by seeing the outcome — you added a control after noticing it improved your t-stat, you trimmed a year after seeing it weakened the effect, you switched clustering because the pre-registered choice was insignificant — is exactly the garden of forking paths, and the log forces you to confront it as such. The honest response is not to pretend it didn’t happen but to report the result both ways and flag the deviating version as exploratory. Example entry: “After observing anomalous denial patterns in 2020, we added a year-2020 indicator interacted with treatment. This was prompted by the data, so it is outcome-driven. We report the pre-registered specification (no 2020 interaction) as the confirmatory result and the 2020-adjusted version as a labeled exploratory analysis, not as the headline.”

That last clause is the entire ethic of Week 8 in one move. The PAP is not a prison and the deviations log is not a confession booth; together they are the instrument that draws the line between confirmatory and exploratory honestly, in public, after the fact. A referee — or future-Maya, six months on — reads the deviations log and trusts the paper more, not less, because they can see exactly where the analysis was a test and where it became a search. The deepest point, carried straight from Olken in Chapter 7.3: the value of pre-registration is not that you never deviate, but that the reader can see where and why you did.

And the deviations log connects directly to the spec curve you just built, in a way that makes the two tools one system. Every deviation you were tempted to make for outcome-driven reasons — “should I have clustered by county instead? added that control? dropped that year?” — is already a fork in your multiverse. So when you are tempted, the disciplined response is not to silently switch the primary specification; it is to point at the spec curve and say: that alternative is specification number 87; here is where it sits; the headline does not depend on it. The spec curve is the honest home for every fork you were tempted to walk, and the deviations log is the honest record of the few you actually had to. Between them, there is no fork left in the dark.

8.1.6 Where this goes next

You arrived at this chapter with a frozen plan and a frozen notebook. You leave it with three things on the page: a confirmatory result run exactly as pre-registered and reported whatever it showed; a specification curve that makes the entire garden of forking paths visible and tells you, with a forensic bottom panel, whether your headline is a robust feature of the data or a lucky branch; and an honest deviations log that draws the confirmatory/exploratory line in public. Together they are the answer to the question a skeptic asks first: would your result have been the same if you had made your defensible choices differently?

But notice what the spec curve cannot tell you, and what therefore comes next. Every specification in Maya’s robust, all-negative curve assumes parallel trends. If that assumption is false, the curve is a confident wrong answer. Chapter 8.2 turns from analytic-choice multiplicity to inference validity: the placebo tests that would have fired if the design were broken, and Oster’s \delta, which asks how much unobserved selection it would take to explain away your effect. The spec curve stress-tests your choices; 8.2 stress-tests your assumption. You need both before you defend the paper in 8.3 and hand it to a hostile peer reviewer in 8.4. A result that survives the multiverse and the placebos and the bound on omitted-variable bias is one you can present at the conference without flinching — because every honest attack has already been made, by you, first.

Your Turn

Open nb8.1 — the specification-curve generator (notebooks/week-08/nb8.1-specification-curve.ipynb). It takes your project’s pre-registered primary specification — the one your pap-filed commit names and your nb7.5 first-look ran once — and builds the full multiverse around it. You will: (1) enumerate your defensible analytic choices slot by slot in CONVENTIONS §4 form (controls, fixed effects, sample window, clustering, outcome definition), and count the number of specifications the cross-product implies; (2) run every specification and collect the estimates, confidence intervals, and the choice-record for each; (3) draw the two-panel specification curve — estimates sorted on top, the choice grid aligned underneath — with your pre-registered specification marked; and (4) read it with the four-point checklist from §8.1.3, then write the one honest paragraph that describes your curve. The notebook ships Maya’s HMDA multiverse as the worked example (her curve sits robustly below zero, median \approx -1.8 pp) and Sam’s momentum multiverse as the cautionary one (sign flips with the weighting/universe choice). The optional extension implements the Simonsohn–Simmons–Nelson permutation test for a p-value on the whole forest.

Reflection prompts to carry into Lab 8 and the peer review.

Find the fork that flips your sign. Build your spec curve, then read the bottom panel: which single analytic choice, if any, best separates the significant specifications from the insignificant ones — or the negative estimates from the positive? Write the one sentence you will put in your paper naming that fork and stating honestly how much your headline depends on it. If no single fork drives the spread, say that, and explain why it is the stronger result.
Where does your pre-registered specification sit? Mark your confirmatory estimate on your own curve. Is it near the center (representative) or out in a tail? If it sits in a tail, write the honest explanation — and ask yourself whether, had you peeked first, you would have been tempted to pre-register exactly that corner. What in your Week-7 process prevented that?
Audit your deviations log against your multiverse. List every departure you made from your PAP on the confirmation data. For each, mark it mechanical or outcome-driven, and locate the corresponding specification in your spec curve. For any outcome-driven deviation, confirm that you reported the pre-registered version as the headline and the deviation as labeled-exploratory. If you cannot find a deviation’s alternative anywhere in your multiverse, your enumeration of forks was incomplete — go back and add it.

# Chapter 8.1 — Execution & the Specification Curve Maya filed her pre-analysis plan two weeks ago. It sits in her repository as a tagged commit, `pap-filed`, time-stamped and frozen, with one primary specification written out in full spec-discipline form and a deviation log waiting empty beside it. The hard thinking is done. What remains, this week, is the part that feels like it should be easy: *run the thing.* Open the notebook, point it at the confirmation data, execute the one regression she promised, and read the number off the table. It is easy. It is also where most honest projects quietly go wrong, because the moment a result appears on the screen, the discipline of Week 7 stops being a document and starts being a test of character. The PAP told Maya *which* regression to run. It did not, and could not, run it for her — and it cannot stop her from running thirty more once she has seen the first. This chapter is about that moment. It teaches you to execute the pre-registered confirmatory result exactly as filed and report it whatever it shows; then to take the garden of forking paths you wrote down in Chapter 7.3 and make it *visible* — to run not one specification but every defensible one, plot the whole distribution of estimates, and read off whether your headline is a robust feature of the data or a lucky branch in a forest of paths. The instrument that does this is the **specification curve**, and it is the single most honest picture you can put in an empirical paper. We follow the book's reveal-the-trick structure. State what the confirmatory run is and run it; state what the garden of forking paths costs you and make it visible with a multiverse of specifications; learn to read the resulting curve honestly — fragile or robust?; and see when the curve itself lies. The intellectual frame is Simonsohn, Simmons, and Nelson's specification-curve analysis. > Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification Curve Analysis. *Nature Human Behaviour*, 4(11), 1208–1214. A boundary first, so you do not go looking for the wrong thing here. This chapter is about **analytic-choice multiplicity** — the fact that one dataset and one hypothesis admit many defensible regressions, and that the choice among them is a hidden multiple-testing problem. The *next* chapter, 8.2, is about **inference validity** — whether the one design you chose actually identifies what you claim, stress-tested with placebos and Oster's $\delta$. They are different anxieties. The spec curve asks "does my result survive the choices I could defensibly have made differently?" The robustness tests of 8.2 ask "is the design I committed to even the right design?" Do not conflate them; this week pulls them apart on purpose. --- ## 8.1.1 The confirmatory run: execute exactly what you filed, report whatever it shows Here is the result in one sentence: **your first job in Week 8 is to run the pre-registered primary specification once, on the confirmation data, exactly as written in the PAP — no edits, no peeking-then-tweaking — and to report the number it produces no matter which way it falls.** That sounds trivial. It is the hardest professional habit in the entire course, and the reason is psychological, not technical. Go back to Chapter 1.5 and the logic of what a p-value *means*. You proved there that under a true null the p-value is uniformly distributed on $[0,1]$, so a single pre-specified test at $\alpha=0.05$ rejects a true null exactly 5% of the time — that 5% is the contract you signed. The contract only holds if the test was *fixed before you saw the outcome.* The instant you run a regression, look at the coefficient, dislike it, and adjust the specification to improve it, you have re-entered the garden of forking paths of Chapter 7.3, and your p-value no longer means 5%. It means something larger and unknowable, inflated by every fork you walked with the answer in view. The PAP froze the forks. The confirmatory run is the act of *honoring* that freeze. So the mechanics are deliberately rigid. Maya's `pap-filed` commit named, in CONVENTIONS §4 form, a single staggered difference-in-differences: outcome the county-year minority–white denial gap; treatment a state's active fair-lending examination program; never-adopters and not-yet-treated states as clean controls; county and year fixed effects; standard errors clustered by state; the sample pinned to a fixed HMDA vintage. Her notebook, `notebooks/week-07/nb7.5-first-look-regressions.ipynb`, is *frozen behind a gate* — a flag `PAP_FILED=True` that must be asserted before the analysis cell will run, the technological enforcement of "do not peek until the plan is filed." When she ran it, exactly once, the pre-registered estimator returned $$\widehat{\text{ATT}} = -1.89 \text{ percentage points}, \qquad \text{SE}_{\text{clustered by state}} = 0.065, \qquad 95\%\ \text{CI} = [-2.02,\ -1.75].$$ The examination programs are associated with a roughly 1.9-percentage-point *narrowing* of the denial gap, the sign she hypothesized, with a confidence interval comfortably away from zero. That is the confirmatory result. It is the number the rest of the paper defends, and it is the number she reports in her headline table — full stop, with the clustering flavor labeled and the CI shown. Now the part that separates a researcher from a result-shopper. Maya must commit, *in advance*, to reporting that number whichever way it came out. Suppose the confirmatory run had returned $\widehat{\text{ATT}} = -0.3$ pp with a CI of $[-1.9,\ +1.3]$ — a wide interval straddling zero. Her PAP's falsification clause already told her what to do: a wide CI that includes zero is a *power failure*, reported as inconclusive, not as a refutation; a tight CI hugging zero would be a genuine, publishable null about fair-lending policy. Either way she reports it. The temptation — and it is enormous, because she has spent five weeks on this and *wants* the effect to be real — is to treat a disappointing confirmatory number as a problem to be debugged rather than a result to be reported. That instinct is the disease. The whole architecture of Week 7 exists to make the confirmatory number *binding*: you wrote down what it would mean before you knew what it would be, precisely so that knowing what it is cannot change what it means. There is a subtle trap even inside an honest confirmatory run, and it is worth naming because beginners fall into it constantly. The trap is mistaking a *clean execution* for a *robust result*. Maya ran exactly one specification and got a starred coefficient. That is necessary but not sufficient for a credible finding, because the one specification she ran was *one branch of a tree she could defensibly have climbed differently.* Her PAP committed her to clustering by state — but clustering by county was also defensible. It committed her to county and year fixed effects — but adding state-specific linear trends was also defensible. It committed her to a particular sample window — but trimming the COVID-disrupted year 2020 was also defensible. Each of those alternatives is a fork she *chose not to walk*, and a reader is entitled to ask: if you had walked them, would the $-1.9$ still be there, or would it dissolve? The confirmatory run answers "what does the pre-registered specification say?" It does not answer "is the answer an artifact of that one specification?" For that, you need to make the whole garden visible. That is the rest of this chapter. --- ## 8.1.2 The garden of forking paths, made visible You met the garden of forking paths in Chapter 7.3 as a *threat*: the reason an honest researcher can p-hack without a single dishonest thought, by making a sequence of reasonable, outcome-aware choices that each nudge the coefficient the desired way. The PAP's defense was to nail down the forks in advance so they collapse to one confirmatory test. That defense protects your *headline*. But it leaves a question hanging that the PAP cannot answer, because the PAP is a single path through the garden: **how many of the other defensible paths would have given you the same answer?** The specification curve, due to Simonsohn, Simmons, and Nelson (2020), answers exactly that question by refusing to hide. Their insight is to invert the usual sin. The p-hacker runs many specifications and reports the *one* that looks best, hiding the rest. The specification-curve analyst runs many specifications — ideally *all* the defensible ones — and reports *all of them at once*, as a distribution. Where p-hacking exploits multiplicity by concealment, specification-curve analysis defangs multiplicity by *exhaustive disclosure.* If your result is a robust feature of the data, it will be there across most of the forest; if it is a lucky branch, the picture will say so loudly, and you will have caught yourself before a referee does. The procedure has three steps, and the discipline is entirely in the first one. **Step 1 — Enumerate the defensible analytic choices.** This is the act of *writing the garden down*. You list, for each decision in your analysis, every option a reasonable, competent researcher could have chosen — and, crucially, *only* the defensible ones. The point is not to include absurd specifications to pad the count; a specification you would never defend in a seminar does not belong in the multiverse, because including it would let you smuggle in noise and then hide behind "but the median was fine." The choices for Maya's HMDA DiD partition naturally into the seven slots of CONVENTIONS §4: - **Controls.** Which county-year applicant-composition covariates enter? She could include the income distribution, the loan-to-value distribution, the loan-amount distribution — or any subset. Reasonable researchers disagree about how many composition controls a doubly-robust DiD needs. Call this 4 defensible combinations (none, income only, income+LTV, all three). - **Fixed effects.** County and year FE are the baseline. Does she also add state-specific linear time trends (a common parallel-trends hedge)? With or without: 2 options. - **Sample window.** Full period; drop 2020 (COVID); drop the first and last adoption cohorts (thinly identified): 3 options. - **Clustering / inference.** Cluster by state (her pre-registered choice); cluster by county; wild-cluster bootstrap by state (defensible when adopting states are few, per Chapter 4.1): 3 options. - **Outcome definition.** The denial *gap* (minority denial rate minus white denial rate); or two separate outcomes — the minority denial *level* and a within-applicant denial indicator; the gap measured as a difference vs. a log-ratio: say 2 defensible outcome operationalizations. Multiply them: $4 \times 2 \times 3 \times 3 \times 2 = 144$ specifications. Each is a single point in what the multiverse literature calls the **garden of forking paths made into a grid** — every combination of every defensible choice, laid out so none can hide. Notice that one of those 144 *is* her pre-registered primary specification. The confirmatory run is not outside the multiverse; it is a labeled, privileged point inside it. Here the arithmetic of Week 1 comes back to bite, and it is the reason this whole exercise matters. Recall from Chapter 1.5 that if you test 20 independent true nulls at $\alpha=0.05$, the chance at least one crosses the line by luck is $1-(1-0.05)^{20}\approx 0.64$ — a coin-flip-beating probability of a spurious "winner" purely from giving luck enough tries. With 144 specifications, even if the true effect were exactly zero, the probability that *at least one* lands at $p<0.05$ is overwhelming. The garden of forking paths is just multiple testing wearing the disguise of "I only ran one regression." Maya did only run one regression *for the confirmatory result* — but the 144 specifications measure how many regressions she *could defensibly have run*, which is the real size of her multiple-testing problem. The specification curve does not eliminate that problem. It does something better: it shows you the whole distribution of what luck and choice together could produce, so a single starred specification can never again masquerade as the only one you ever considered. **Step 2 — Run all of them.** Estimate the key coefficient — for Maya, the ATT on the denial gap — under every one of the 144 specifications, and collect the estimates and their confidence intervals into a table. This is mechanical and embarrassingly fast on modern hardware; 144 fixed-effects regressions on a county-year panel run in seconds. The output is a vector of 144 estimates and 144 inference results, plus, for each one, a record of *which choice was made in each slot.* That record is the load-bearing part, as Step 3 reveals. **Step 3 — Plot the distribution, with the choices underneath.** The specification curve is a two-panel figure that has become the signature of this method, and once you have read one you cannot unsee it. We build it next. --- ## 8.1.3 Reading the specification curve The figure has two stacked panels that share a horizontal axis. The horizontal axis runs over all 144 specifications, *sorted by the magnitude of the estimate* — most negative on the left, most positive on the right. The **top panel** plots the 144 point estimates as a curve (hence the name), each with a marker for its confidence interval — colored or filled when the specification is statistically significant at your chosen level, hollow when it is not. Because the specifications are sorted by estimate, the curve rises monotonically from left to right; its *shape* is the whole story. A flat curve sitting entirely below zero, with nearly every marker filled, says: *every defensible path gives roughly the same negative answer, and almost all are significant.* A curve that sweeps from strongly negative through zero up to strongly positive, with a mix of filled and hollow markers, says: *the sign and significance of my result depend entirely on which fork I walk* — a result that is an artifact of analytic choice, not a feature of the world. The **bottom panel** is the part that turns a pretty picture into a diagnostic. It is a grid: one row per analytic *choice* (cluster-by-state, cluster-by-county, drop-2020, add-state-trends, income-control-only, …), and one column per specification, *aligned* to the top panel's sort order. A tick mark sits at $(\text{row}, \text{column})$ when that specification used that choice. So you read *downward* from any interesting region of the curve to see exactly which choices produced it. If all the *positive, insignificant* specifications on the right share a single tick — say, every one of them used "cluster by county" plus "drop 2020" — then you have learned something precise and damning: **the result's fragility is driven by those two choices.** The bottom panel converts "my result is fragile" from a vague worry into a forensic finding: *this specific fork is the one that flips my sign.* Let us read Maya's curve concretely, because the reading is the skill. Suppose her 144 specifications produce a curve like this: the estimates run from about $-2.4$ pp on the left to about $-0.4$ pp on the right; the median is around $-1.8$ pp; about 132 of the 144 are significant at the 5% level (filled), and the dozen hollow ones cluster at the right end near zero. Reading downward from those dozen non-significant specifications, the bottom panel shows they *all* share two choices: the wild-cluster bootstrap (which widens intervals when adopters are few) combined with the "drop first and last cohorts" sample (which throws away identifying variation). That is an honest, interpretable picture, and here is exactly what Maya should write about it: > Across all 144 defensible specifications, the estimated effect of fair-lending examinations on the minority–white denial gap is negative, with a median of $-1.8$ pp and a range of $[-2.4, -0.4]$ pp. The pre-registered primary specification ($-1.9$ pp) sits near the center of this distribution. The effect is statistically significant in 132 of 144 specifications (92%); the twelve exceptions combine the wild-cluster bootstrap with the most aggressive sample trim, which jointly maximize the standard error while minimizing identifying variation — i.e., the result loses significance only in the lowest-power corner of the multiverse, and never changes sign. That is a **robust** result, and the curve is what licenses the word. The sign never flips; the magnitude is stable around the pre-registered estimate; loss of significance is confined to specifications that are independently the least powerful, for reasons the bottom panel makes legible. A referee reading that paragraph and that figure cannot accuse Maya of cherry-picking, because she has shown them the entire orchard and pointed to exactly which trees bear less fruit and why. Now contrast the picture that should *terrify* you, and would have, had Sam built a specification curve for his momentum anomaly. Sam's project from Week 7 asks whether past-12-month winners outperform losers next month — a portfolio sort with a long menu of defensible forks: the formation window (12 months, or 12 skipping the most recent, or 6); the holding period (1 month, 3, 6); value- versus equal-weighting; NYSE breakpoints versus all-stock breakpoints; including or excluding microcaps and penny stocks; the sample period (full history, or post-2000, or excluding 2008–09). Those forks easily multiply past 200 specifications. Suppose Sam's curve sweeps from a significantly *negative* long-short return on the left, through a fat band of insignificant near-zero estimates in the middle, up to a significantly *positive* return on the right — and the bottom panel reveals that *every* significantly positive specification shares one choice: "include microcaps, equal-weighted." Reading downward tells the whole story in one glance: the anomaly is *not* a feature of the cross-section of returns; it is a feature of how you weight tiny, illiquid, hard-to-trade stocks. The headline "momentum earns 80 bps a month" was a single branch — the equal-weighted-microcap branch — and the curve drags it into the light. Sam's honest sentence is the mirror image of Maya's: *the sign of the momentum premium is not robust to the weighting and universe choices; it is positive and significant only under equal-weighting with microcaps included, and is statistically indistinguishable from zero under the value-weighted, liquid-universe specifications that a tradable strategy would actually face.* That sentence kills the easy paper and replaces it with a true and far more interesting one. The general reading rule, then, is a short checklist you apply to every specification curve you build or referee: 1. **Where does the sign live?** If estimates span both signs, your effect's *direction* is choice-dependent — the most serious possible fragility. If they all share a sign, the direction is robust even if the magnitude wanders. 2. **Where does the pre-registered specification sit?** Mark it on the curve. If your confirmatory estimate sits near the center of the distribution, your headline is representative. If it sits at an extreme tail, you must explain why — and "I happened to pre-register the best-looking corner" is not a flattering explanation, even when innocent. 3. **What fraction is significant, and where do the exceptions cluster?** Read the hollow markers downward into the bottom panel. Significance lost in the low-power corner (small samples, conservative inference) is benign; significance that appears *only* under one or two specific choices is a red flag pointing straight at those choices. 4. **Which rows of the bottom panel drive the spread?** The choice whose ticks best separate the left of the curve from the right is the fork your result is most sensitive to. Name it. It is the thing a skeptic will attack, and the thing your robustness section (8.2) must address head-on. A word on the formal inference that Simonsohn, Simmons, and Nelson attach to the curve, kept light because the picture is most of the value. They propose a **permutation test on the whole curve**: under the sharp null that the effect is zero everywhere, you re-shuffle the treatment assignment, recompute all 144 specifications on each shuffle, and ask whether the *observed* curve — for instance, the number of significant-and-correctly-signed specifications, or the median estimate — is more extreme than what the null curves produce. This tests the multiverse *as a whole* rather than any single specification, which is precisely the right unit of analysis once you admit that "which specification" was itself a choice. For a high-school capstone, computing the descriptive curve and reading it with the checklist above is the required deliverable; the permutation test is an extension in nb8.1 for those who want the formal p-value on the entire forest. [CHECK: confirm the exact page within Simonsohn, Simmons & Nelson (2020) where the joint permutation-inference procedure is stated, for a pinpoint cite.] --- ## 8.1.4 When the specification curve lies Reveal-the-trick demands the third move: every technique has an assumption that breaks, and a picture this persuasive is dangerous precisely *because* it is persuasive. A specification curve can mislead in four distinct ways, and an honest analyst flags which ones apply. **The curve only contains the forks you chose to enumerate.** This is the deepest limitation and the easiest to forget once the beautiful figure is on the page. A specification curve is exhaustive over the choices *you listed* and silent about every choice you did not think to vary. If Maya's entire multiverse holds the *outcome definition* nearly fixed — if all 144 specifications quietly assume the denial gap is the right outcome and never test the within-applicant denial indicator — then her curve can look gloriously robust while resting on one unexamined fork. The curve's robustness is conditional on the completeness of the enumeration, and completeness is a judgment call, not a guarantee. The defense is to enumerate the forks *honestly and broadly*, and to state in the paper which dimensions you held fixed and why — exactly the spec-discipline of CONVENTIONS §4, now applied to the meta-decision of what to vary. **Not all specifications are equally defensible, and the curve pretends they are.** A naive reading treats all 144 points as equal votes, so that "the median is negative" sounds like a democratic verdict. But some of those specifications are *better* than others — clustering at the level treatment varies (state) is more defensible than clustering by county; a doubly-robust DiD with composition controls is better-identified than one without. If you pad the multiverse with weak-but-arguable specifications, you can drag the median wherever you like, which is p-hacking with extra steps and a fancier chart. Simonsohn, Simmons, and Nelson are explicit that the choices must be *theoretically justified and statistically valid* — not merely "things you could type into the regression." The honest move is to include only specifications you would defend out loud, and, where they genuinely differ in quality, to say so rather than hide behind the median. **The curve describes analytic-choice fragility, not identification.** A specification curve can be flat, tight, all-significant, and *completely wrong*, because every one of its 144 specifications can share the same fatal identification flaw. If Maya's parallel-trends assumption fails — if adopting states were already on a different denial-gap trajectory — then all 144 specifications inherit that bias, and the curve will robustly report a beautifully stable artifact. Robustness across the multiverse is *not* the same as validity of the design. This is the boundary from this chapter's opening, restated as a warning: the spec curve answers "did my choices drive the result?" and is mute on "did my design identify the effect?" The second question is Chapter 8.2's job, with the placebos and Oster-$\delta$ stress tests that probe the assumption itself. A robust spec curve on an invalid design is a confident wrong answer — the most expensive kind. **The curve can become a fishing license if you let it.** There is a perverse failure mode where a researcher runs the multiverse, *then* picks the specification at the center of the distribution and reports *that* as the headline — "I chose the median specification, the most representative one." But the median was determined *by the data you have already seen*, so choosing it after the fact is just outcome-driven specification selection in a lab coat. The discipline that prevents this is the one you already have: the **PAP fixes the primary specification in advance**, and the spec curve is reported *around* that pre-registered point, not used to *choose* it. The confirmatory run comes first and is privileged; the curve is context for it, never a substitute. This is why the order in this chapter is not negotiable — confirmatory result in §8.1.1, multiverse in §8.1.2 — and why the curve always marks where the pre-registered specification sits. The summary is the one Simonsohn, Simmons, and Nelson themselves press: a specification curve is a tool for *honesty about analytic-choice multiplicity*, and it is extraordinarily good at that one job. It is not a robustness panacea, not a substitute for a credible identifying assumption, and not a way to launder a post-hoc choice into a pre-registered one. Use it for what it is — the most complete possible disclosure of the garden of forking paths — and say plainly what it does not do. --- ## 8.1.5 The honest deviations log You filed a PAP. You ran the confirmatory specification. You built the spec curve. Somewhere in that process — it is nearly inevitable, as Chapter 7.3 warned via Olken's *perils* — reality forced you off the plan. A control you swore you would include turned out to be collinear with a fixed effect and dropped out. A clustering variable was corrupted for 4% of rows. The COVID year behaved so strangely you had to handle it differently than you guessed. **The deviations log is the running, dated record of every departure from the PAP — what changed, why, and, the crucial part, whether the change was prompted by seeing the outcome.** Deviating is not the sin; Week 7 said this and it bears repeating now that you are actually doing the analysis and the temptation is live rather than hypothetical. *Hiding* the deviation is the sin, and *outcome-driven* deviation is the specific sin the whole apparatus exists to catch. So every entry in the log carries a verdict on one binary question: **was this change forced on me by a mechanical reality, or did I make it because I saw the coefficient and did not like it?** - A deviation forced by a *mechanical reality* — collinearity, a coding bug, a data-availability surprise, a cluster identifier that was corrupted — is innocent. You note it, you fix it, you move on. It is not outcome-driven; nothing about the result steered the change. Example entry: *"PAP §2 specified clustering by state. On the confirmation data, the state identifier was missing for 4% of county-year cells due to a FIPS-merge gap. We clustered by the state-equivalent census region for those cells and re-ran; the ATT moved from $-1.89$ to $-1.91$ pp, immaterial. Not outcome-driven."* - A deviation prompted by *seeing the outcome* — you added a control after noticing it improved your t-stat, you trimmed a year after seeing it weakened the effect, you switched clustering because the pre-registered choice was insignificant — is exactly the garden of forking paths, and the log forces you to confront it as such. The honest response is not to pretend it didn't happen but to *report the result both ways and flag the deviating version as exploratory.* Example entry: *"After observing anomalous denial patterns in 2020, we added a year-2020 indicator interacted with treatment. This was prompted by the data, so it is outcome-driven. We report the pre-registered specification (no 2020 interaction) as the confirmatory result and the 2020-adjusted version as a labeled exploratory analysis, not as the headline."* That last clause is the entire ethic of Week 8 in one move. The PAP is not a prison and the deviations log is not a confession booth; together they are the instrument that draws the line between confirmatory and exploratory *honestly, in public, after the fact.* A referee — or future-Maya, six months on — reads the deviations log and trusts the paper *more*, not less, because they can see exactly where the analysis was a test and where it became a search. The deepest point, carried straight from Olken in Chapter 7.3: the value of pre-registration is not that you never deviate, but that the reader can see *where* and *why* you did. And the deviations log connects directly to the spec curve you just built, in a way that makes the two tools one system. Every deviation you were tempted to make for outcome-driven reasons — "should I have clustered by county instead? added that control? dropped that year?" — is *already a fork in your multiverse.* So when you are tempted, the disciplined response is not to silently switch the primary specification; it is to point at the spec curve and say: *that alternative is specification number 87; here is where it sits; the headline does not depend on it.* The spec curve is the honest home for every fork you were tempted to walk, and the deviations log is the honest record of the few you actually had to. Between them, there is no fork left in the dark. --- ## 8.1.6 Where this goes next You arrived at this chapter with a frozen plan and a frozen notebook. You leave it with three things on the page: a **confirmatory result** run exactly as pre-registered and reported whatever it showed; a **specification curve** that makes the entire garden of forking paths visible and tells you, with a forensic bottom panel, whether your headline is a robust feature of the data or a lucky branch; and an **honest deviations log** that draws the confirmatory/exploratory line in public. Together they are the answer to the question a skeptic asks first: *would your result have been the same if you had made your defensible choices differently?* But notice what the spec curve cannot tell you, and what therefore comes next. Every specification in Maya's robust, all-negative curve assumes parallel trends. If that assumption is false, the curve is a confident wrong answer. Chapter 8.2 turns from analytic-choice multiplicity to *inference validity*: the placebo tests that would have fired if the design were broken, and Oster's $\delta$, which asks how much *unobserved* selection it would take to explain away your effect. The spec curve stress-tests your choices; 8.2 stress-tests your *assumption.* You need both before you defend the paper in 8.3 and hand it to a hostile peer reviewer in 8.4. A result that survives the multiverse *and* the placebos *and* the bound on omitted-variable bias is one you can present at the conference without flinching — because every honest attack has already been made, by you, first. --- ## Your Turn Open **`nb8.1` — the specification-curve generator** (`notebooks/week-08/nb8.1-specification-curve.ipynb`). It takes your project's pre-registered primary specification — the one your `pap-filed` commit names and your nb7.5 first-look ran once — and builds the full multiverse around it. You will: (1) enumerate your defensible analytic choices slot by slot in CONVENTIONS §4 form (controls, fixed effects, sample window, clustering, outcome definition), and count the number of specifications the cross-product implies; (2) run every specification and collect the estimates, confidence intervals, and the choice-record for each; (3) draw the two-panel specification curve — estimates sorted on top, the choice grid aligned underneath — with your pre-registered specification marked; and (4) read it with the four-point checklist from §8.1.3, then write the one honest paragraph that describes your curve. The notebook ships Maya's HMDA multiverse as the worked example (her curve sits robustly below zero, median $\approx -1.8$ pp) and Sam's momentum multiverse as the cautionary one (sign flips with the weighting/universe choice). The optional extension implements the Simonsohn–Simmons–Nelson permutation test for a p-value on the whole forest. **Reflection prompts to carry into Lab 8 and the peer review.** 1. *Find the fork that flips your sign.* Build your spec curve, then read the bottom panel: which single analytic choice, if any, best separates the significant specifications from the insignificant ones — or the negative estimates from the positive? Write the one sentence you will put in your paper naming that fork and stating honestly how much your headline depends on it. If no single fork drives the spread, say *that*, and explain why it is the stronger result. 2. *Where does your pre-registered specification sit?* Mark your confirmatory estimate on your own curve. Is it near the center (representative) or out in a tail? If it sits in a tail, write the honest explanation — and ask yourself whether, had you peeked first, you would have been tempted to pre-register exactly that corner. What in your Week-7 process prevented that? 3. *Audit your deviations log against your multiverse.* List every departure you made from your PAP on the confirmation data. For each, mark it mechanical or outcome-driven, and locate the corresponding specification in your spec curve. For any outcome-driven deviation, confirm that you reported the pre-registered version as the headline and the deviation as labeled-exploratory. If you cannot find a deviation's alternative anywhere in your multiverse, your enumeration of forks was incomplete — go back and add it.