Chapter 7.3 — The Pre-Analysis Plan (Short Form)

Maya has a question, a dataset, and a problem she has not noticed yet.

The question is the one she has been circling since Week 6: holding creditworthiness fixed as well as the public data allow, do minority applicants face higher mortgage denial rates than otherwise-similar white applicants, and does the gap shrink when the lender is algorithmic rather than face-to-face? The dataset is a HMDA extract she pulled and documented in Chapter 7.2 — millions of loan applications, public, real, hers. She is excited, which is the problem.

Because here is what is about to happen if no one stops her. Maya will load the data, run her main regression, and stare at the race coefficient. Suppose it comes out small and insignificant. She will frown, and — being a good and resourceful student — she will try things. Maybe drop the smallest lenders. Maybe winsorize income at the 1st and 99th percentiles instead of leaving it raw. Maybe add a quadratic in loan-to-value. Maybe restrict to purchase loans, since refinances are “noisier.” Maybe cluster by county instead of by lender, because that tightens the standard error. Somewhere in that thicket of reasonable-sounding choices, a specification will appear in which the coefficient is large and starred. She will feel relief, write it up, and report that one specification as though it were the only one she ever considered.

She will not be lying, exactly. Every choice she made was defensible in isolation. That is precisely what makes the result worthless. Maya has just p-hacked her own honest project, and she did it without a single dishonest thought, because the search happened after she saw the outcomes and was steered, choice by choice, toward the answer she was hoping for. This chapter is the device that stops it. The device is a pre-analysis plan — a short document, written and time-stamped before you look at your outcomes, that commits you to your hypotheses, your primary specification, your multiple-testing plan, and what would prove you wrong. We will follow the reveal-the-trick structure the book always uses: state what a PAP does and why, see exactly the failure it prevents, see when it itself fails or over-promises, and then write a real one for Maya.

The intellectual frame is Benjamin Olken’s “Promises and Perils of Pre-Analysis Plans” — the most balanced thing written on the topic, because it takes both the promises and the perils seriously.

Olken, B. A. (2015). Promises and Perils of Pre-Analysis Plans. Journal of Economic Perspectives, 29(3), 61–80.

7.3.1 Why pre-registration exists: confirmatory vs. exploratory

Here is the whole idea in one sentence: a pre-analysis plan separates the analysis you committed to in advance (confirmatory) from the analysis you invented after seeing the data (exploratory), so that a reader can tell which of your results were tests and which were searches. Both kinds of analysis are legitimate and valuable. The sin is not exploring; the sin is presenting an exploratory result as if it had been a confirmatory test.

To see why that distinction is the entire game, go back to the cold arithmetic of Chapter 1.5. You proved there that when the null is true, the p-value is uniformly distributed on [0,1] — every value from 0 to 1 is equally likely. A direct consequence: a single test of a true null produces p < 0.05 exactly 5% of the time, by construction. That 5% is the price of admission you agreed to pay. But you also showed what happens when you run many tests. If twenty independent true nulls are each tested at \alpha = 0.05, the chance that at least one crosses the line by pure luck is

1 - (1 - 0.05)^{20} = 1 - 0.95^{20} \approx 0.64.

A 64% chance of a spurious “winner” — better than a coin flip — purely from giving luck twenty tries. The per-test error rate \alpha was never built to survive being applied many times and then cherry-picked.

Now connect that to Maya’s afternoon. She did not consciously run twenty tests. But every “maybe” in her thicket — drop small lenders or not, winsorize or not, quadratic LTV or not, purchase-only or not, cluster by lender or by county — is a fork in the road, and each fork doubles the number of paths to a starred result. Five binary choices give 2^5 = 32 specifications. Even if her question is dead — even if the true conditional gap is exactly zero — the probability that at least one of 32 specifications crosses p < 0.05 is enormous. And because she walked the forks with the outcome in view, she did not pick a path at random; she was pulled, at each fork, toward the branch that nudged the coefficient her way. This is Andrew Gelman’s “garden of forking paths,” and it is more insidious than classic p-hacking precisely because it does not require running twenty explicit regressions. It only requires making a sequence of reasonable, outcome-aware choices in a single analysis. The number of tests you “really” ran is the number of paths you would have been willing to take — and that number is invisible, even to you, unless you wrote it down in advance.

You met the AI-era version of exactly this trap last week. In Chapter 6.5, the danger was prompt-induced p-hacking: because an LLM’s label column depends on the prompt, you can rewrite the rubric — “negative” defined one way, then another — until the labels happen to produce the regression result you wanted, then report the lucky prompt. Same disease, new vector. The prompt wording is just one more fork in the garden, and an unusually easy one to fiddle with. The defense proposed there was to freeze the rubric against a gold set before looking at the downstream coefficient. A pre-analysis plan generalizes that defense to your entire design: freeze every fork — outcome, controls, sample, clustering, prompt — against your stated hypotheses, before the outcomes are visible, so that the choices cannot be steered by the answer.

The mechanism is almost embarrassingly simple, and it is the whole trick. You cannot fish in a pond whose boundaries you nailed down before you knew where the fish were. If Maya writes, in a dated and committed file, “my primary outcome is the loan denial indicator; my primary specification clusters by lender; my sample is purchase loans only; I will winsorize income at 1%/99%” — and she writes it before she has seen a single coefficient — then the 32 forks collapse to one. There is exactly one confirmatory test. Its p-value means what Chapter 1.5 said a p-value means. Everything she tries afterward is still allowed, still valuable, but it is now labeled honestly as exploratory: a generator of new hypotheses for the next study, not a test of this one.

That is the promise. Olken’s contribution — and the reason this chapter is named after his paper rather than a cheerleading one — is that he holds the promise and the perils in the same hand. The promise is credibility: a confirmatory result with a pre-registered plan is one a referee can trust, because the researcher’s degrees of freedom were spent in advance, in the open. The perils are real too, and we will not hide them: a PAP can be gamed (written vaguely so it constrains nothing, or filed after a sneak peek); it can be too rigid, forcing you to ignore something genuinely important you discovered along the way; and the pressure to register everything can crowd out exploratory work that is the actual engine of discovery. The resolution is not “register nothing” or “register and then never deviate.” It is: be precise about what you committed to, honest about what you changed, and clear about which results are confirmatory and which are exploratory. A PAP done in that spirit makes your confirmatory claims bulletproof without amputating your curiosity.

7.3.2 The five components of a short-form PAP

A full PAP for a randomized field experiment can run to forty pages. You do not need that, and writing it would teach you the wrong lesson — that pre-registration is a bureaucratic burden rather than a thinking tool. The short form has exactly five components, and each one forces a specific kind of honesty before the data can flatter you. Write them in order; the order is the discipline.

Component 1 — Hypotheses, stated directionally

Write down what you predict, and which way. Not “race is related to denial” — that is a coin that lands heads on any nonzero coefficient — but a directional claim: “minority applicants face higher denial rates conditional on observables,” and “the gap is smaller for algorithmic lenders than for face-to-face lenders.” Directional hypotheses are the honest default for a reason you proved in Chapter 1.5: a directional claim commits you to a one-sided test, and you may only claim a one-sided test if the direction came from the science before you saw the data. Writing the direction down in the PAP is exactly the evidence that it did. (If you genuinely have no prior on direction — either way would be a finding — say so, and commit to a two-sided test. That is honest too; it just costs you the easier critical value, 1.96 instead of 1.65, which is the price of not having had a prior.)

State the hypotheses as numbered, falsifiable sentences. Number them, because the numbering will reappear in the multiple-testing plan: H1 is your primary hypothesis, the one the whole project lives or dies on; H2, H3, … are secondary, the ones that would enrich the story but whose failure would not sink it. Deciding which hypothesis is primary, in advance, is one of the most consequential choices in the whole plan, and we return to it in Component 3.

Component 2 — The primary specification, in spec-discipline form

This is the heart of the PAP, and it is where this book’s empirical-spec discipline (CONVENTIONS §4) earns its keep. Your primary specification is the single regression that tests H1 — the one number that, if it broke, sinks the paper. Write it out naming, explicitly, every one of the seven slots:

Outcome — the exact dependent variable, defined to the level of a variable name. Not “denial” but “the binary indicator denied = 1 if HMDA action_taken ∈ {3, 7}, else 0, on the application sample defined below.”
Treatment / key regressor — the one coefficient you will read off the table to answer H1. For Maya, the minority indicator minority (a single coefficient \beta_1), interacted with the FinTech flag for H2.
Controls — the legitimate covariates you hold fixed, named. Loan amount, applicant income, loan-to-value where available. And — this is the part the PAP forces — you state why each control belongs and, crucially, name any variable you are deliberately not controlling for because it sits on the causal pathway (Mentor 4’s over-controlling trap). Maya will not control for the loan-product type, because steering into a worse product is itself a channel of discrimination, and conditioning on a channel would regress away the very effect she is hunting.
Fixed effects — which absorbing dummies. Lender fixed effects (so the comparison is within a lender, not across lenders with different clienteles) and census-tract fixed effects (so the comparison is between applicants in the same neighborhood).
Clustering — the level at which the standard errors are clustered, chosen in advance and justified. This is a notorious fork: clustering by lender versus by county versus by tract can change your t-stat materially, and choosing the level after seeing which one gives significance is p-hacking on the standard error. Maya commits to clustering by lender, because the plausible correlation in errors is at the lender level (a lender’s idiosyncratic underwriting style affects all its loans).
Sample — the exact universe of rows: which years, which loan purposes, which geographies, which exclusions, with the row count you expect. “First-lien, owner-occupied, one-to-four-family purchase mortgage applications, 2019–2021, conventional conforming loans, dropping applications withdrawn or closed for incompleteness.” Every exclusion is a fork; nail them all down now.
Identifying assumption, in one sentence — the single sentence (CONVENTIONS §4) that, if true, makes \beta_1 the thing you claim it is. For Maya: “Conditional on the named controls, lender fixed effects, and tract fixed effects, the minority indicator is uncorrelated with the unobserved creditworthiness (the missing credit score) that the lender legitimately prices — an assumption I know to be imperfect, which is why my headline claim is a bound, not a clean causal effect.” You cannot write that sentence vaguely. The PAP demands it because, as Maya learned in Mentor 4, no list of controls closes the gap between a disparity and discrimination by itself; only a named assumption — defended or admitted as a bound — does.

The mathematical object underneath is the one you have built since Week 2. In the notation of CONVENTIONS §3, the primary specification is a linear model

\texttt{denied}_{i} = \beta_0 + \beta_1\,\texttt{minority}_{i} + \mathbf{x}_i'\boldsymbol{\gamma} + \alpha_{\ell(i)} + \delta_{t(i)} + \varepsilon_i,

where \mathbf{x}_i is the named control vector, \alpha_{\ell(i)} is a lender fixed effect, \delta_{t(i)} is a tract fixed effect, errors are clustered by lender, and \beta_1 is the single coefficient H1 is about. Writing the equation is not optional ornament; it pins down, unambiguously, which number is the test. A referee — or future-Maya, six months from now — can read this line and know exactly what was promised.

Component 3 — The multiple-testing plan

You will compute more than one number. Maya has a primary gap (H1), a FinTech interaction (H2), and she will be tempted to also look at the pricing margin, at subgroups by income band, at separate coefficients for Black and Hispanic applicants, at different years. Each of those is a test, and §7.3.1 showed where unmanaged tests lead. The multiple-testing plan is the part of the PAP that decides, in advance, how the army of tests will be governed so that the headline is not a lucky draw from a crowd.

It has three moving parts.

First, declare primary versus secondary outcomes. This is the single most powerful multiplicity control there is, and it is free. You commit, before seeing anything, that H1’s coefficient on the denial outcome is the test — the one held to the strict \alpha = 0.05 and reported as the result. Everything else is secondary: interesting, reported, but explicitly labeled as not the confirmatory claim. By promoting one test to primary, you have shrunk the multiplicity problem for your headline from “however many things I tried” to one. Olken stresses this hierarchy precisely because it is the cheapest and most honest defense: most of the multiplicity disappears the moment you say which question is the question.

Second, group your secondary tests into families and control the false discovery rate within each. When you genuinely have a family of related tests — say, the gap measured separately for five income bands, or for Black, Hispanic, and Asian applicants — you cannot report each at \alpha = 0.05 and call them all findings; that is the 64% trap again. You correct for multiplicity. Chapter 1.5 introduced the crude Bonferroni rule (test each of m hypotheses at \alpha/m), which controls the chance of any false positive but is brutally conservative when m is large — it can leave you unable to detect real effects. The more sensible target when screening a family of candidate effects is the false discovery rate (FDR): the expected fraction of your declared discoveries that are false. The Benjamini–Hochberg procedure controls it with a simple, mechanical recipe you can do by hand and which you will code in nb7.3:

Take your m p-values from the family and sort them ascending: p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}. Find the largest rank k for which p_{(k)} \le \frac{k}{m}\,\alpha. Reject (declare a discovery for) every hypothesis with rank \le k.

The genius of the threshold \frac{k}{m}\alpha is that it is lenient where Bonferroni is harsh: the smallest p-value is held to \frac{1}{m}\alpha (exactly Bonferroni’s bar), but the k-th smallest is allowed up to \frac{k}{m}\alpha, so a family with several genuinely significant effects lets the later ones in at a much friendlier threshold. FDR says, in effect, “I am willing to tolerate that, say, 5% of the things I call discoveries are flukes” — a far more useful contract than Bonferroni’s “I will tolerate a 5% chance that any single one is a fluke” when you are scanning many real candidates at once. A worked example: with m = 5 income-band p-values \{0.008,\,0.012,\,0.039,\,0.041,\,0.330\} and \alpha = 0.05, the BH thresholds \frac{k}{5}(0.05) are \{0.01, 0.02, 0.03, 0.04, 0.05\}. Compare from the top down: rank 5, 0.330 > 0.05, no; rank 4, 0.041 > 0.04, no; rank 3, 0.039 > 0.03, no; rank 2, 0.012 \le 0.02, yes — so k = 2, and you declare the two smallest (the 0.008 and 0.012 bands) as discoveries and the rest as not. Bonferroni would have held every band to 0.05/5 = 0.01 and declared only the 0.008 band, throwing away the 0.012 effect that BH keeps. That is the cost of conservatism, and it is why FDR is the modern default for families.

Third — and this is the part the PAP exists to enforce — state the families and the procedure now, before you have the p-values. The reason is the same one that makes everything in this chapter work. If you decide after seeing the numbers which tests count as a “family,” you can shrink or grow the family until your favorite effect survives correction. The family must be a pre-committed list. Write: “Family A = the primary denial gap (H1), reported at \alpha = 0.05, no correction needed as it is a single primary test. Family B = the gap by {Black, Hispanic, Asian} applicants, three tests, BH-FDR at 0.05. Family C = the gap by {five income bands}, BH-FDR at 0.05.” Now the correction is automatic and unspoofable.

Component 4 — What would falsify the hypothesis

This is the component beginners skip and Olken would not let you. For each hypothesis, write the result that would make you abandon it. Not “fail to confirm” — falsify. Karl Popper’s point, dragged into empirical finance: a hypothesis that no possible result could contradict is not a scientific claim, it is a mood. If you cannot name the coefficient pattern that would force you to say “I was wrong,” you have not stated a testable hypothesis.

For Maya’s H1, falsification is concrete: “If \hat{\beta}_1 on the minority indicator is statistically indistinguishable from zero in the primary specification — that is, if the 95% confidence interval for the conditional gap contains zero and excludes economically meaningful values — I will conclude this dataset does not support a conditional denial gap, and I will say so.” Note the care, drawing on Chapter 1.5.8: she does not write “if it’s insignificant I’m wrong,” because an insignificant-but-huge estimate measured noisily is a power failure, not a refutation (the third of the four significance/effect-size cases). She writes the falsification in terms of the confidence interval excluding meaningful effects — a result that is genuinely informative about absence, not merely silent. Writing falsification conditions in advance does something subtle and valuable: it pre-commits you to believing your own null result if it comes. The single biggest reason p-hacking happens is that researchers cannot stomach a null, so they search for a star. If you have already written, in a dated file, “here is the result I will accept as a no,” you have inoculated yourself against the search before the temptation arrives.

Component 5 — Planned robustness

Finally, list the robustness checks you plan to run — and label them as planned, to distinguish them from the post-hoc robustness you might add later. The distinction matters: a planned robustness check is part of your confirmatory design (“I will show H1 survives clustering by county as well as by lender, and survives dropping 2020 for COVID disruption”); a post-hoc one discovered later is exploratory and labeled as such. Maya’s planned list: re-estimate with county clustering; drop 2020; use the continuous rate-spread outcome alongside the binary denial outcome; restrict to the largest 50 lenders to check the result is not driven by thin-data small lenders. Each is a fork she is choosing to walk deliberately and in advance, which converts it from a p-hacking liability into a strength. The key discipline: a planned robustness check is one whose result you commit to reporting whichever way it comes out. If H1 holds in the main spec but evaporates under county clustering, the PAP obligates her to say so. Robustness is not a search for confirmation; it is a stress test you promised to report honestly.

7.3.3 Registering before you peek: the hold-out and the deviation log

The five components are what you write. The discipline that makes them work is when you write them — before you have looked at your outcomes — and how you handle the inevitable surprises. Three mechanics enforce this.

Split exploratory from confirmatory, mechanically. The cleanest way to keep yourself honest is to never let the confirmatory data touch your eyes during the design phase. Two routes. The first, available whenever your study is prospective — you write the plan before the data even exist (a forward-looking experiment, or a sample period that has not happened yet) — needs no split: the future is your hold-out, because you cannot peek at data that does not exist. Maya’s HMDA study is not like that; the data already exist and she could peek. So she uses the second route, a hold-out sample. She randomly partitions her HMDA extract — say, a 30% exploration set and a 70% confirmation set, splitting on a stable key (a hash of the loan identifier, so the split is reproducible and pinned, per CONVENTIONS §5). She is free to play in the exploration set: run regressions, look at coefficients, discover patterns, tune her specification, find the controls that matter, even (especially) p-hack to her heart’s content — because nothing she finds there is a confirmatory result. The exploration set is the laboratory where she designs the experiment. She then freezes the PAP, files it, and only then runs the frozen primary specification once on the untouched 70% confirmation set. Whatever that one run produces is the confirmatory answer. The hold-out converts “don’t peek” from a request for superhuman willpower into a mechanical fact: she could not have tuned the specification to the confirmation data because she never saw it.

This is the same train/test discipline you met in Chapter 6.5’s classifier validation — develop on a development set, report on a touched-once held-out test set — applied now to the whole research design rather than to one classifier. And it is the same discipline behind the camp’s own workflow: nb7.5 (“First-look regressions”) is frozen until the PAP is filed. The notebook of confirmatory regressions does not run until the plan that governs it is a committed, time-stamped artifact.

Register by time-stamping, in public or in version control. A PAP only constrains you if its contents are fixed before the analysis, and the only way to prove that to a skeptical reader is a credible time stamp. Professional researchers register on a public platform — the AEA RCT Registry for experiments, OSF or AsPredicted for observational designs — which date-stamps the document so it cannot be silently edited afterward. In this camp, your registry is your Git history: Lab 7 has you file the PAP as a tagged commit (git tag pap-filed) in your project repository. The tag is your time stamp. A tagged commit is a cryptographic promise: anyone can verify that the PAP existed, in exactly this form, before the commits that contain your confirmatory results. That is why the PAP is a deliverable of Lab 7 and not a private note — a plan no one can date is a plan that constrains no one.

Log your deviations honestly. Here is where Olken’s perils become operational, and where naive pre-registration goes wrong. You will deviate from your PAP. You will discover, on the confirmation data, that a control you swore you would include is collinear and drops out; that 2020 is so disrupted it must be handled differently than you guessed; that a coding bug in your outcome definition needs fixing. Deviating is not the sin. Hiding the deviation is the sin. The PAP is not a prison; it is a baseline. So you keep a deviation log — a running, dated record, committed alongside the PAP, of every departure from the plan, what changed, why, and whether the change was prompted by seeing the outcome. That last clause is the crux: a deviation forced by a mechanical reality (collinearity, a bug, a data-availability surprise) is innocent and you note it; a deviation that happened because you saw the coefficient and didn’t like it is exactly the p-hacking the PAP exists to prevent, and the log forces you to confront which kind it is. The honest researcher’s report reads: “We pre-registered specification X. On the confirmation data we deviated in two ways: (1) we clustered by county rather than lender, because the lender identifier was corrupted for 4% of rows — this was not outcome-driven; (2) we added a year-2020 indicator after observing anomalous denial patterns — this was prompted by the data, so we report the result both with and without it and flag the latter as exploratory.” A referee reading that trusts you more, not less, because you have shown them the seams. Olken’s deepest point is here: the value of a PAP is not that you never deviate, but that the reader can see exactly where your analysis was confirmatory and where it became exploratory. The deviation log is the instrument that draws that line.

7.3.4 A worked short-PAP for Maya

Here is the whole thing assembled — the short-form PAP Maya files as her pap-filed tagged commit. It is deliberately compact; the discipline is in the precision, not the length.

PRE-ANALYSIS PLAN — Conditional Racial Disparity in Mortgage Denial, and the FinTech Margin Author: Maya · Filed (tagged commit pap-filed): before any confirmatory regression · Data: HMDA 2019–2021 extract, snapshot pinned in data-cards/hmda.md

§1. Hypotheses (directional). - H1 (primary). Conditional on observable risk factors, lender, and neighborhood, minority applicants face a higher denial rate than otherwise-similar non-minority applicants (\beta_1 > 0). One-sided; the direction is the long-standing prior in the fair-lending literature (Bartlett et al. 2022; BHR 2022). - H2 (secondary). The conditional gap is smaller for algorithmic (FinTech) lenders than for face-to-face lenders (interaction coefficient < 0). One-sided. - H3 (secondary, exploratory family). The conditional gap differs across applicant subgroups (Black / Hispanic / Asian) and across income bands. No directional prior; two-sided.

§2. Primary specification (CONVENTIONS §4 form). \texttt{denied}_{i} = \beta_0 + \beta_1\,\texttt{minority}_{i} + \mathbf{x}_i'\boldsymbol{\gamma} + \alpha_{\ell(i)} + \delta_{t(i)} + \varepsilon_i - Outcome: denied = 1 if HMDA action_taken ∈ {3, 7}, else 0. - Treatment / key regressor: minority (1 if applicant race/ethnicity is Black or Hispanic; the single coefficient \beta_1 that answers H1). - Controls \mathbf{x}_i: log loan amount, log applicant income, loan-to-value, loan term. Deliberately excluded: loan-product type and high-cost flag — these sit on the steering/discrimination pathway (Mentor 4 over-controlling trap), so conditioning on them would absorb the effect. - Fixed effects: lender (\alpha_\ell) and census tract (\delta_t). - Clustering: by lender (errors plausibly correlated within a lender’s underwriting style). - Sample: first-lien, owner-occupied, 1–4-family, conventional conforming purchase applications, 2019–2021; drop withdrawn / closed-for-incompleteness / preapproval-only; expected N \approx [from exploration set]. - Identifying assumption (one sentence): Conditional on the named controls and the two sets of fixed effects, minority is uncorrelated with the unobserved creditworthiness (the credit score HMDA omits) that lenders legitimately price — an assumption known to be imperfect, so \hat{\beta}_1 is reported as an upper bound on differential treatment, not a clean causal effect.

§3. Multiple-testing plan. - Family A (primary): H1 alone, reported at \alpha = 0.05, one-sided. No correction (single primary test). - Family B: H2 FinTech interaction, \alpha = 0.05, one-sided. - Family C (exploratory): the {Black, Hispanic, Asian} × {5 income-band} subgroup gaps. Benjamini–Hochberg FDR control at 0.05 within this family; results labeled exploratory regardless of outcome.

§4. Falsification. - H1 is falsified if the 95% CI for \beta_1 in the primary specification contains zero and excludes economically meaningful values (a gap of, say, ≥ 2 percentage points) — i.e., the data place a tight bound near zero. A wide CI that merely includes zero is a power failure, reported as inconclusive, not as a refutation. - H2 is falsified if the interaction is indistinguishable from zero (FinTech and face-to-face gaps statistically equal) or positive (FinTech worse).

§5. Planned robustness (reported whichever way they come out). - Re-cluster by county. Drop 2020 (COVID disruption). Add the continuous rate-spread outcome alongside binary denial. Restrict to the 50 largest lenders. Re-estimate on the full sample (exploration + confirmation) as a precision check, clearly labeled.

§6. Hold-out. Data split 30% exploration / 70% confirmation on hash(loan_id) mod 10 (pinned). Specification designed only on the 30%; the primary regression runs once on the untouched 70% after this PAP is tagged. Deviations logged in DEVIATIONS.md.

Read that PAP and notice what it has done to Maya’s dangerous afternoon. The 32 forks of §7.3.1 are gone: there is exactly one primary specification, one key coefficient, one clustering choice, one sample, all fixed before she could see a result. The robustness checks she might have used to fish are now promised in advance and reported either way. The subgroup analyses that could have generated a lucky star are quarantined into an explicitly exploratory family with FDR control. And the hold-out means she physically cannot have tuned the confirmatory spec to the confirmatory data. If \hat{\beta}_1 comes out large and significant on the 70%, that is a result a referee can trust. If it comes out a tight zero, the PAP has already obligated her to report a null — and a clean, pre-registered null about fair lending is a genuine contribution, not a failure.

That is the reveal. A pre-analysis plan is not paperwork. It is the difference between a number that tested a hypothesis and a number that was selected to look like it did — and the only person it is really protecting you from is yourself, eight hours and thirty-two specifications from now.

7.3.5 When the PAP itself fails: Olken’s perils, honestly

A chapter that sold pre-registration as a free lunch would be doing exactly the thing the technique is meant to prevent — hiding the inconvenient results. So, in Olken’s spirit, the perils.

A PAP can be written so vaguely it constrains nothing. “I will study the relationship between race and lending using appropriate controls and standard errors” is a PAP-shaped object that licenses every fork in the garden. The defense is the spec-discipline of Component 2: if the plan does not name the outcome to a variable name, the clustering level, and the exact sample, it is not a plan. Vagueness is p-hacking with extra steps.

A PAP can be too rigid for its own good. Suppose Maya, running the confirmation data, discovers something she never imagined — a sharp discontinuity in denials at a regulatory threshold that suggests a far better identification strategy than her conditioning regression. A fundamentalist reading of “never deviate” would force her to ignore the best idea in her project. Olken’s resolution, and ours: she reports the pre-registered confirmatory result as planned (it is what she promised), and separately reports the discovery as clearly-labeled exploratory analysis and a hypothesis for the next study. Both live in the paper, honestly partitioned. The PAP did not cost her the discovery; it cost her only the right to pretend the discovery was a confirmatory test, which she should not want anyway.

A PAP can crowd out exploration if a field over-fetishizes it — if “was it pre-registered?” becomes the only question and the messy, inductive, pattern-hunting work that actually generates good hypotheses gets treated as second-class. This is the peril Olken weighs most carefully. The answer is the confirmatory/exploratory distinction held with a steady hand: exploration is not lesser science, it is earlier science. The hold-out design honors this directly — the entire exploration set exists for unconstrained fishing, because that is where Maya’s best ideas will come from. The plan does not forbid exploration; it forbids laundering exploration as confirmation.

And one peril particular to a high-schooler’s first project: the PAP can give false confidence. A pre-registered design is not automatically a good design. Maya’s identifying assumption is still imperfect — the credit score is still missing, the residual is still a bound — and pre-registering that imperfect design does not make it clean. Pre-registration governs the honesty of your inference (did you test or did you search?), not its validity (is the test the right one?). Validity is the job of the identification memo you write next, in Chapter 7.5. A PAP and an identification memo are complementary: one promises you did not fish, the other argues that what you caught is the fish you claim. You need both.

7.3.6 Where this goes next

You arrived at this chapter with an idea (7.1) and data in hand (7.2). You now have the instrument that turns them into a falsifiable, pre-registered design: hypotheses stated directionally, one primary specification written in spec-discipline form, a multiple-testing plan that quarantines the garden of forking paths, explicit falsification conditions, planned robustness, a hold-out that makes “don’t peek” mechanical, and a deviation log that keeps you honest when reality intrudes. Next you will build the analysis dataset the PAP refers to (7.4) — the merge keys, the survivorship and look-ahead checks, the winsorizing you already committed to here — and then write the identification memo (7.5) that defends the one assumption your \beta_1 rests on. The PAP says you will not fish; the memo says the pond is the right pond. Filed together as your Lab 7 tagged commit, they are the spine of a project that could survive a hostile referee.

Your Turn

Open nb7.3 (notebooks/week-07/nb7.3-pap-companion-power-calc.ipynb), the PAP companion. It walks you from the five components to a filed plan for your own project, and its core is a power calculation for your design — the direct sequel to the power curves you built in nb1.5. You will: (1) state your effect size of interest in the units of your outcome (for Maya, the smallest conditional denial gap, in percentage points, that would matter to a real borrower); (2) estimate the residual standard deviation and the effective sample size from your exploration set only; (3) compute, exactly as in Chapter 1.5.4, the N your design needs to detect that effect at \alpha = 0.05 with 80% power — and discover whether your data are even capable of testing your hypothesis before you waste a confirmation run on a study that was underpowered from the start; and (4) implement the Benjamini–Hochberg procedure on a family of p-values and watch it admit the right number of discoveries. The notebook ends by emitting a filled-in PAP template you tag as pap-filed, the deliverable for PS 7.3 and Lab 7.

Reflection prompts to carry into the lab and Mentor Session 7.

Name your three most dangerous forks. In your own project, list the three specification choices (an outcome definition, a sample exclusion, a clustering level, a control, a prompt rubric) where, if you peeked at the result first, you would be most tempted to choose the branch that helps your hypothesis. For each, write the one sentence you will put in your PAP to nail it down in advance.
Write your falsification sentence, then sit with it. For your primary hypothesis, write the exact coefficient-and-confidence-interval pattern that would make you abandon it — phrased, as in §7.3.2, to distinguish a genuine refutation from a mere power failure. Would you actually report that null if it came? If your honest answer is “I’d keep looking,” your PAP is not yet protecting you; find the fork you are still leaving open.
Power before peeking. Suppose your nb7.3 power calculation reveals that detecting the smallest effect you care about, at 80% power, would need three times the sample you have. What are your options before you run the confirmation regression — and why is “run it anyway and hope” the one option a pre-registered researcher has already foreclosed?

# Chapter 7.3 — The Pre-Analysis Plan (Short Form) Maya has a question, a dataset, and a problem she has not noticed yet. The question is the one she has been circling since Week 6: holding creditworthiness fixed as well as the public data allow, do minority applicants face higher mortgage denial rates than otherwise-similar white applicants, and does the gap shrink when the lender is algorithmic rather than face-to-face? The dataset is a HMDA extract she pulled and documented in Chapter 7.2 — millions of loan applications, public, real, hers. She is excited, which is the problem. Because here is what is about to happen if no one stops her. Maya will load the data, run her main regression, and stare at the race coefficient. Suppose it comes out small and insignificant. She will frown, and — being a good and resourceful student — she will *try things*. Maybe drop the smallest lenders. Maybe winsorize income at the 1st and 99th percentiles instead of leaving it raw. Maybe add a quadratic in loan-to-value. Maybe restrict to purchase loans, since refinances are "noisier." Maybe cluster by county instead of by lender, because that tightens the standard error. Somewhere in that thicket of reasonable-sounding choices, a specification will appear in which the coefficient is large and starred. She will feel relief, write it up, and report that one specification as though it were the only one she ever considered. She will not be lying, exactly. Every choice she made was defensible in isolation. That is precisely what makes the result worthless. Maya has just *p-hacked her own honest project*, and she did it without a single dishonest thought, because the search happened *after* she saw the outcomes and was steered, choice by choice, toward the answer she was hoping for. This chapter is the device that stops it. The device is a **pre-analysis plan** — a short document, written and time-stamped *before* you look at your outcomes, that commits you to your hypotheses, your primary specification, your multiple-testing plan, and what would prove you wrong. We will follow the reveal-the-trick structure the book always uses: state what a PAP does and why, see exactly the failure it prevents, see when it itself fails or over-promises, and then write a real one for Maya. The intellectual frame is Benjamin Olken's "Promises and Perils of Pre-Analysis Plans" — the most balanced thing written on the topic, because it takes both the promises *and* the perils seriously. > Olken, B. A. (2015). Promises and Perils of Pre-Analysis Plans. *Journal of Economic Perspectives*, 29(3), 61–80. --- ## 7.3.1 Why pre-registration exists: confirmatory vs. exploratory Here is the whole idea in one sentence: **a pre-analysis plan separates the analysis you committed to in advance (confirmatory) from the analysis you invented after seeing the data (exploratory), so that a reader can tell which of your results were tests and which were searches.** Both kinds of analysis are legitimate and valuable. The sin is not exploring; the sin is *presenting an exploratory result as if it had been a confirmatory test.* To see why that distinction is the entire game, go back to the cold arithmetic of Chapter 1.5. You proved there that when the null is true, the p-value is *uniformly distributed* on $[0,1]$ — every value from 0 to 1 is equally likely. A direct consequence: a single test of a true null produces $p < 0.05$ exactly 5% of the time, by construction. That 5% is the price of admission you agreed to pay. But you also showed what happens when you run *many* tests. If twenty independent true nulls are each tested at $\alpha = 0.05$, the chance that *at least one* crosses the line by pure luck is $$1 - (1 - 0.05)^{20} = 1 - 0.95^{20} \approx 0.64.$$ A 64% chance of a spurious "winner" — better than a coin flip — purely from giving luck twenty tries. The per-test error rate $\alpha$ was never built to survive being applied many times and then cherry-picked. Now connect that to Maya's afternoon. She did not consciously run twenty tests. But every "maybe" in her thicket — drop small lenders or not, winsorize or not, quadratic LTV or not, purchase-only or not, cluster by lender or by county — is a fork in the road, and each fork doubles the number of paths to a starred result. Five binary choices give $2^5 = 32$ specifications. Even if her question is dead — even if the true conditional gap is exactly zero — the probability that *at least one* of 32 specifications crosses $p < 0.05$ is enormous. And because she walked the forks *with the outcome in view*, she did not pick a path at random; she was pulled, at each fork, toward the branch that nudged the coefficient her way. This is Andrew Gelman's **"garden of forking paths,"** and it is more insidious than classic p-hacking precisely because it does not require running twenty explicit regressions. It only requires making a sequence of reasonable, outcome-aware choices in a single analysis. The number of tests you "really" ran is the number of paths you *would have been willing to take* — and that number is invisible, even to you, unless you wrote it down in advance. You met the AI-era version of exactly this trap last week. In Chapter 6.5, the danger was **prompt-induced p-hacking**: because an LLM's label column depends on the prompt, you can rewrite the rubric — "negative" defined one way, then another — until the labels happen to produce the regression result you wanted, then report the lucky prompt. Same disease, new vector. The prompt wording is just one more fork in the garden, and an unusually easy one to fiddle with. The defense proposed there was to *freeze the rubric against a gold set before looking at the downstream coefficient*. A pre-analysis plan generalizes that defense to your *entire* design: freeze every fork — outcome, controls, sample, clustering, prompt — against your stated hypotheses, before the outcomes are visible, so that the choices cannot be steered by the answer. The mechanism is almost embarrassingly simple, and it is the whole trick. **You cannot fish in a pond whose boundaries you nailed down before you knew where the fish were.** If Maya writes, in a dated and committed file, "my primary outcome is the loan denial indicator; my primary specification clusters by lender; my sample is purchase loans only; I will winsorize income at 1%/99%" — and she writes it *before* she has seen a single coefficient — then the 32 forks collapse to *one*. There is exactly one confirmatory test. Its p-value means what Chapter 1.5 said a p-value means. Everything she tries afterward is still allowed, still valuable, but it is now labeled honestly as exploratory: a *generator of new hypotheses for the next study*, not a test of this one. That is the promise. Olken's contribution — and the reason this chapter is named after his paper rather than a cheerleading one — is that he holds the promise and the perils in the same hand. The promise is credibility: a confirmatory result with a pre-registered plan is one a referee can trust, because the researcher's degrees of freedom were spent in advance, in the open. The perils are real too, and we will not hide them: a PAP can be gamed (written vaguely so it constrains nothing, or filed after a sneak peek); it can be *too* rigid, forcing you to ignore something genuinely important you discovered along the way; and the pressure to register everything can crowd out exploratory work that is the actual engine of discovery. The resolution is not "register nothing" or "register and then never deviate." It is: **be precise about what you committed to, honest about what you changed, and clear about which results are confirmatory and which are exploratory.** A PAP done in that spirit makes your confirmatory claims bulletproof *without* amputating your curiosity. --- ## 7.3.2 The five components of a short-form PAP A full PAP for a randomized field experiment can run to forty pages. You do not need that, and writing it would teach you the wrong lesson — that pre-registration is a bureaucratic burden rather than a thinking tool. The short form has exactly five components, and each one forces a specific kind of honesty *before* the data can flatter you. Write them in order; the order is the discipline. ### Component 1 — Hypotheses, stated directionally Write down what you predict, *and which way*. Not "race is related to denial" — that is a coin that lands heads on any nonzero coefficient — but a **directional** claim: "minority applicants face *higher* denial rates conditional on observables," and "the gap is *smaller* for algorithmic lenders than for face-to-face lenders." Directional hypotheses are the honest default for a reason you proved in Chapter 1.5: a directional claim commits you to a *one-sided* test, and you may only claim a one-sided test if the direction came from the science *before* you saw the data. Writing the direction down in the PAP is exactly the evidence that it did. (If you genuinely have no prior on direction — either way would be a finding — say so, and commit to a two-sided test. That is honest too; it just costs you the easier critical value, $1.96$ instead of $1.65$, which is the price of not having had a prior.) State the hypotheses as numbered, falsifiable sentences. Number them, because the numbering will reappear in the multiple-testing plan: H1 is your *primary* hypothesis, the one the whole project lives or dies on; H2, H3, … are secondary, the ones that would enrich the story but whose failure would not sink it. Deciding which hypothesis is primary, *in advance*, is one of the most consequential choices in the whole plan, and we return to it in Component 3. ### Component 2 — The primary specification, in spec-discipline form This is the heart of the PAP, and it is where this book's empirical-spec discipline (CONVENTIONS §4) earns its keep. Your **primary specification** is the single regression that tests H1 — the one number that, if it broke, sinks the paper. Write it out naming, explicitly, every one of the seven slots: - **Outcome** — the exact dependent variable, defined to the level of a variable name. Not "denial" but "the binary indicator `denied` = 1 if HMDA `action_taken` ∈ {3, 7}, else 0, on the application sample defined below." - **Treatment / key regressor** — the one coefficient you will read off the table to answer H1. For Maya, the minority indicator `minority` (a single coefficient $\beta_1$), interacted with the FinTech flag for H2. - **Controls** — the legitimate covariates you hold fixed, *named*. Loan amount, applicant income, loan-to-value where available. And — this is the part the PAP forces — you state *why each control belongs* and, crucially, name any variable you are deliberately *not* controlling for because it sits on the causal pathway (Mentor 4's over-controlling trap). Maya will not control for the loan-product type, because steering into a worse product is itself a channel of discrimination, and conditioning on a channel would regress away the very effect she is hunting. - **Fixed effects** — which absorbing dummies. Lender fixed effects (so the comparison is *within* a lender, not across lenders with different clienteles) and census-tract fixed effects (so the comparison is between applicants in the same neighborhood). - **Clustering** — the level at which the standard errors are clustered, *chosen in advance and justified*. This is a notorious fork: clustering by lender versus by county versus by tract can change your t-stat materially, and choosing the level *after* seeing which one gives significance is p-hacking on the standard error. Maya commits to clustering by lender, because the plausible correlation in errors is at the lender level (a lender's idiosyncratic underwriting style affects all its loans). - **Sample** — the exact universe of rows: which years, which loan purposes, which geographies, which exclusions, with the row count you expect. "First-lien, owner-occupied, one-to-four-family purchase mortgage applications, 2019–2021, conventional conforming loans, dropping applications withdrawn or closed for incompleteness." Every exclusion is a fork; nail them all down now. - **Identifying assumption, in one sentence** — the single sentence (CONVENTIONS §4) that, if true, makes $\beta_1$ the thing you claim it is. For Maya: *"Conditional on the named controls, lender fixed effects, and tract fixed effects, the minority indicator is uncorrelated with the unobserved creditworthiness (the missing credit score) that the lender legitimately prices — an assumption I know to be imperfect, which is why my headline claim is a bound, not a clean causal effect."* You cannot write that sentence vaguely. The PAP demands it because, as Maya learned in Mentor 4, no list of controls closes the gap between a disparity and discrimination by itself; only a named assumption — defended or admitted as a bound — does. The mathematical object underneath is the one you have built since Week 2. In the notation of CONVENTIONS §3, the primary specification is a linear model $$\texttt{denied}_{i} = \beta_0 + \beta_1\,\texttt{minority}_{i} + \mathbf{x}_i'\boldsymbol{\gamma} + \alpha_{\ell(i)} + \delta_{t(i)} + \varepsilon_i,$$ where $\mathbf{x}_i$ is the named control vector, $\alpha_{\ell(i)}$ is a lender fixed effect, $\delta_{t(i)}$ is a tract fixed effect, errors are clustered by lender, and $\beta_1$ is the *single* coefficient H1 is about. Writing the equation is not optional ornament; it pins down, unambiguously, which number is the test. A referee — or future-Maya, six months from now — can read this line and know exactly what was promised. ### Component 3 — The multiple-testing plan You will compute more than one number. Maya has a primary gap (H1), a FinTech interaction (H2), and she will be tempted to also look at the pricing margin, at subgroups by income band, at separate coefficients for Black and Hispanic applicants, at different years. Each of those is a test, and §7.3.1 showed where unmanaged tests lead. The multiple-testing plan is the part of the PAP that decides, *in advance*, how the army of tests will be governed so that the headline is not a lucky draw from a crowd. It has three moving parts. First, **declare primary versus secondary outcomes.** This is the single most powerful multiplicity control there is, and it is free. You commit, before seeing anything, that H1's coefficient on the denial outcome is *the* test — the one held to the strict $\alpha = 0.05$ and reported as the result. Everything else is secondary: interesting, reported, but explicitly labeled as not the confirmatory claim. By promoting one test to primary, you have shrunk the multiplicity problem for your *headline* from "however many things I tried" to *one*. Olken stresses this hierarchy precisely because it is the cheapest and most honest defense: most of the multiplicity disappears the moment you say which question is the question. Second, **group your secondary tests into families and control the false discovery rate within each.** When you genuinely have a family of related tests — say, the gap measured separately for five income bands, or for Black, Hispanic, and Asian applicants — you cannot report each at $\alpha = 0.05$ and call them all findings; that is the 64% trap again. You correct for multiplicity. Chapter 1.5 introduced the crude **Bonferroni** rule (test each of $m$ hypotheses at $\alpha/m$), which controls the chance of *any* false positive but is brutally conservative when $m$ is large — it can leave you unable to detect real effects. The more sensible target when screening a family of candidate effects is the **false discovery rate (FDR)**: the *expected fraction of your declared discoveries that are false*. The **Benjamini–Hochberg procedure** controls it with a simple, mechanical recipe you can do by hand and which you will code in nb7.3: > Take your $m$ p-values from the family and sort them ascending: $p_{(1)} \le p_{(2)} \le \dots \le p_{(m)}$. Find the *largest* rank $k$ for which > $$p_{(k)} \le \frac{k}{m}\,\alpha.$$ > Reject (declare a discovery for) every hypothesis with rank $\le k$. The genius of the threshold $\frac{k}{m}\alpha$ is that it is lenient where Bonferroni is harsh: the smallest p-value is held to $\frac{1}{m}\alpha$ (exactly Bonferroni's bar), but the $k$-th smallest is allowed up to $\frac{k}{m}\alpha$, so a family with *several* genuinely significant effects lets the later ones in at a much friendlier threshold. FDR says, in effect, "I am willing to tolerate that, say, 5% of the things I call discoveries are flukes" — a far more useful contract than Bonferroni's "I will tolerate a 5% chance that *any single one* is a fluke" when you are scanning many real candidates at once. A worked example: with $m = 5$ income-band p-values $\{0.008,\,0.012,\,0.039,\,0.041,\,0.330\}$ and $\alpha = 0.05$, the BH thresholds $\frac{k}{5}(0.05)$ are $\{0.01, 0.02, 0.03, 0.04, 0.05\}$. Compare from the top down: rank 5, $0.330 > 0.05$, no; rank 4, $0.041 > 0.04$, no; rank 3, $0.039 > 0.03$, no; rank 2, $0.012 \le 0.02$, yes — so $k = 2$, and you declare the two smallest (the $0.008$ and $0.012$ bands) as discoveries and the rest as not. Bonferroni would have held *every* band to $0.05/5 = 0.01$ and declared only the $0.008$ band, throwing away the $0.012$ effect that BH keeps. That is the cost of conservatism, and it is why FDR is the modern default for families. Third — and this is the part the PAP exists to enforce — **state the families and the procedure now, before you have the p-values.** The reason is the same one that makes everything in this chapter work. If you decide *after* seeing the numbers which tests count as a "family," you can shrink or grow the family until your favorite effect survives correction. The family must be a pre-committed list. Write: "Family A = the primary denial gap (H1), reported at $\alpha = 0.05$, no correction needed as it is a single primary test. Family B = the gap by {Black, Hispanic, Asian} applicants, three tests, BH-FDR at 0.05. Family C = the gap by {five income bands}, BH-FDR at 0.05." Now the correction is automatic and unspoofable. ### Component 4 — What would falsify the hypothesis This is the component beginners skip and Olken would not let you. For each hypothesis, write the result that would make you *abandon* it. Not "fail to confirm" — *falsify*. Karl Popper's point, dragged into empirical finance: a hypothesis that no possible result could contradict is not a scientific claim, it is a mood. If you cannot name the coefficient pattern that would force you to say "I was wrong," you have not stated a testable hypothesis. For Maya's H1, falsification is concrete: *"If $\hat{\beta}_1$ on the minority indicator is statistically indistinguishable from zero in the primary specification — that is, if the 95% confidence interval for the conditional gap contains zero and excludes economically meaningful values — I will conclude this dataset does not support a conditional denial gap, and I will say so."* Note the care, drawing on Chapter 1.5.8: she does not write "if it's insignificant I'm wrong," because an insignificant-but-huge estimate measured noisily is a *power* failure, not a refutation (the third of the four significance/effect-size cases). She writes the falsification in terms of the *confidence interval excluding meaningful effects* — a result that is genuinely informative about absence, not merely silent. Writing falsification conditions in advance does something subtle and valuable: it pre-commits you to *believing your own null result* if it comes. The single biggest reason p-hacking happens is that researchers cannot stomach a null, so they search for a star. If you have already written, in a dated file, "here is the result I will accept as a no," you have inoculated yourself against the search before the temptation arrives. ### Component 5 — Planned robustness Finally, list the robustness checks you *plan* to run — and label them as planned, to distinguish them from the post-hoc robustness you might add later. The distinction matters: a planned robustness check is part of your confirmatory design ("I will show H1 survives clustering by county as well as by lender, and survives dropping 2020 for COVID disruption"); a post-hoc one discovered later is exploratory and labeled as such. Maya's planned list: re-estimate with county clustering; drop 2020; use the continuous rate-spread outcome alongside the binary denial outcome; restrict to the largest 50 lenders to check the result is not driven by thin-data small lenders. Each is a fork she is choosing to walk *deliberately and in advance*, which converts it from a p-hacking liability into a strength. The key discipline: a planned robustness check is one whose result you commit to *reporting whichever way it comes out*. If H1 holds in the main spec but evaporates under county clustering, the PAP obligates her to say so. Robustness is not a search for confirmation; it is a stress test you promised to report honestly. --- ## 7.3.3 Registering before you peek: the hold-out and the deviation log The five components are *what* you write. The discipline that makes them work is *when* you write them — before you have looked at your outcomes — and *how you handle the inevitable surprises*. Three mechanics enforce this. **Split exploratory from confirmatory, mechanically.** The cleanest way to keep yourself honest is to never let the confirmatory data touch your eyes during the design phase. Two routes. The first, available whenever your study is *prospective* — you write the plan before the data even exist (a forward-looking experiment, or a sample period that has not happened yet) — needs no split: the future is your hold-out, because you cannot peek at data that does not exist. Maya's HMDA study is not like that; the data already exist and she could peek. So she uses the second route, a **hold-out sample.** She randomly partitions her HMDA extract — say, a 30% *exploration* set and a 70% *confirmation* set, splitting on a stable key (a hash of the loan identifier, so the split is reproducible and pinned, per CONVENTIONS §5). She is free to play in the exploration set: run regressions, look at coefficients, discover patterns, tune her specification, find the controls that matter, even (especially) p-hack to her heart's content — because *nothing* she finds there is a confirmatory result. The exploration set is the laboratory where she designs the experiment. She then *freezes the PAP*, files it, and only then runs the frozen primary specification *once* on the untouched 70% confirmation set. Whatever that one run produces is the confirmatory answer. The hold-out converts "don't peek" from a request for superhuman willpower into a mechanical fact: she could not have tuned the specification to the confirmation data because she never saw it. This is the same train/test discipline you met in Chapter 6.5's classifier validation — develop on a development set, report on a touched-once held-out test set — applied now to the *whole research design* rather than to one classifier. And it is the same discipline behind the camp's own workflow: nb7.5 ("First-look regressions") is *frozen until the PAP is filed*. The notebook of confirmatory regressions does not run until the plan that governs it is a committed, time-stamped artifact. **Register by time-stamping, in public or in version control.** A PAP only constrains you if its contents are fixed *before* the analysis, and the only way to prove that to a skeptical reader is a credible time stamp. Professional researchers register on a public platform — the AEA RCT Registry for experiments, OSF or AsPredicted for observational designs — which date-stamps the document so it cannot be silently edited afterward. In this camp, your registry is **your Git history**: Lab 7 has you file the PAP as a **tagged commit** (`git tag pap-filed`) in your project repository. The tag is your time stamp. A tagged commit is a cryptographic promise: anyone can verify that the PAP existed, in exactly this form, before the commits that contain your confirmatory results. That is why the PAP is a deliverable of Lab 7 and not a private note — a plan no one can date is a plan that constrains no one. **Log your deviations honestly.** Here is where Olken's *perils* become operational, and where naive pre-registration goes wrong. You *will* deviate from your PAP. You will discover, on the confirmation data, that a control you swore you would include is collinear and drops out; that 2020 is so disrupted it must be handled differently than you guessed; that a coding bug in your outcome definition needs fixing. Deviating is not the sin. *Hiding* the deviation is the sin. The PAP is not a prison; it is a baseline. So you keep a **deviation log** — a running, dated record, committed alongside the PAP, of every departure from the plan, *what* changed, *why*, and *whether the change was prompted by seeing the outcome*. That last clause is the crux: a deviation forced by a mechanical reality (collinearity, a bug, a data-availability surprise) is innocent and you note it; a deviation that happened *because you saw the coefficient and didn't like it* is exactly the p-hacking the PAP exists to prevent, and the log forces you to confront which kind it is. The honest researcher's report reads: "We pre-registered specification X. On the confirmation data we deviated in two ways: (1) we clustered by county rather than lender, because the lender identifier was corrupted for 4% of rows — this was not outcome-driven; (2) we added a year-2020 indicator after observing anomalous denial patterns — this *was* prompted by the data, so we report the result both with and without it and flag the latter as exploratory." A referee reading that trusts you *more*, not less, because you have shown them the seams. Olken's deepest point is here: the value of a PAP is not that you never deviate, but that the reader can see exactly where your analysis was confirmatory and where it became exploratory. The deviation log is the instrument that draws that line. --- ## 7.3.4 A worked short-PAP for Maya Here is the whole thing assembled — the short-form PAP Maya files as her `pap-filed` tagged commit. It is deliberately compact; the discipline is in the precision, not the length. --- > **PRE-ANALYSIS PLAN — Conditional Racial Disparity in Mortgage Denial, and the FinTech Margin** > *Author: Maya · Filed (tagged commit `pap-filed`): before any confirmatory regression · Data: HMDA 2019–2021 extract, snapshot pinned in `data-cards/hmda.md`* > > **§1. Hypotheses (directional).** > - **H1 (primary).** Conditional on observable risk factors, lender, and neighborhood, minority applicants face a *higher* denial rate than otherwise-similar non-minority applicants ($\beta_1 > 0$). One-sided; the direction is the long-standing prior in the fair-lending literature (Bartlett et al. 2022; BHR 2022). > - **H2 (secondary).** The conditional gap is *smaller* for algorithmic (FinTech) lenders than for face-to-face lenders (interaction coefficient $< 0$). One-sided. > - **H3 (secondary, exploratory family).** The conditional gap differs across applicant subgroups (Black / Hispanic / Asian) and across income bands. No directional prior; two-sided. > > **§2. Primary specification (CONVENTIONS §4 form).** > $$\texttt{denied}_{i} = \beta_0 + \beta_1\,\texttt{minority}_{i} + \mathbf{x}_i'\boldsymbol{\gamma} + \alpha_{\ell(i)} + \delta_{t(i)} + \varepsilon_i$$ > - **Outcome:** `denied` = 1 if HMDA `action_taken` ∈ {3, 7}, else 0. > - **Treatment / key regressor:** `minority` (1 if applicant race/ethnicity is Black or Hispanic; the single coefficient $\beta_1$ that answers H1). > - **Controls $\mathbf{x}_i$:** log loan amount, log applicant income, loan-to-value, loan term. *Deliberately excluded:* loan-product type and high-cost flag — these sit on the steering/discrimination pathway (Mentor 4 over-controlling trap), so conditioning on them would absorb the effect. > - **Fixed effects:** lender ($\alpha_\ell$) and census tract ($\delta_t$). > - **Clustering:** by lender (errors plausibly correlated within a lender's underwriting style). > - **Sample:** first-lien, owner-occupied, 1–4-family, conventional conforming *purchase* applications, 2019–2021; drop withdrawn / closed-for-incompleteness / preapproval-only; expected $N \approx$ [from exploration set]. > - **Identifying assumption (one sentence):** Conditional on the named controls and the two sets of fixed effects, `minority` is uncorrelated with the unobserved creditworthiness (the credit score HMDA omits) that lenders legitimately price — an assumption known to be imperfect, so $\hat{\beta}_1$ is reported as an *upper bound* on differential treatment, not a clean causal effect. > > **§3. Multiple-testing plan.** > - *Family A (primary):* H1 alone, reported at $\alpha = 0.05$, one-sided. No correction (single primary test). > - *Family B:* H2 FinTech interaction, $\alpha = 0.05$, one-sided. > - *Family C (exploratory):* the {Black, Hispanic, Asian} × {5 income-band} subgroup gaps. Benjamini–Hochberg FDR control at 0.05 *within this family*; results labeled exploratory regardless of outcome. > > **§4. Falsification.** > - **H1 is falsified** if the 95% CI for $\beta_1$ in the primary specification contains zero *and* excludes economically meaningful values (a gap of, say, ≥ 2 percentage points) — i.e., the data place a tight bound near zero. A wide CI that merely includes zero is a *power* failure, reported as inconclusive, not as a refutation. > - **H2 is falsified** if the interaction is indistinguishable from zero (FinTech and face-to-face gaps statistically equal) or *positive* (FinTech worse). > > **§5. Planned robustness (reported whichever way they come out).** > - Re-cluster by county. Drop 2020 (COVID disruption). Add the continuous rate-spread outcome alongside binary denial. Restrict to the 50 largest lenders. Re-estimate on the full sample (exploration + confirmation) as a precision check, clearly labeled. > > **§6. Hold-out.** Data split 30% exploration / 70% confirmation on `hash(loan_id) mod 10` (pinned). Specification designed *only* on the 30%; the primary regression runs *once* on the untouched 70% after this PAP is tagged. Deviations logged in `DEVIATIONS.md`. --- Read that PAP and notice what it has done to Maya's dangerous afternoon. The 32 forks of §7.3.1 are gone: there is exactly one primary specification, one key coefficient, one clustering choice, one sample, all fixed before she could see a result. The robustness checks she might have used to fish are now *promised in advance and reported either way*. The subgroup analyses that could have generated a lucky star are quarantined into an explicitly exploratory family with FDR control. And the hold-out means she physically cannot have tuned the confirmatory spec to the confirmatory data. If $\hat{\beta}_1$ comes out large and significant on the 70%, that is a result a referee can trust. If it comes out a tight zero, the PAP has already obligated her to report a null — and a clean, pre-registered null about fair lending is a genuine contribution, not a failure. That is the reveal. A pre-analysis plan is not paperwork. It is the difference between a number that *tested* a hypothesis and a number that was *selected* to look like it did — and the only person it is really protecting you from is yourself, eight hours and thirty-two specifications from now. --- ## 7.3.5 When the PAP itself fails: Olken's perils, honestly A chapter that sold pre-registration as a free lunch would be doing exactly the thing the technique is meant to prevent — hiding the inconvenient results. So, in Olken's spirit, the perils. A PAP can be **written so vaguely it constrains nothing.** "I will study the relationship between race and lending using appropriate controls and standard errors" is a PAP-shaped object that licenses every fork in the garden. The defense is the spec-discipline of Component 2: if the plan does not name the outcome to a variable name, the clustering level, and the exact sample, it is not a plan. Vagueness is p-hacking with extra steps. A PAP can be **too rigid for its own good.** Suppose Maya, running the confirmation data, discovers something she never imagined — a sharp discontinuity in denials at a regulatory threshold that suggests a far better identification strategy than her conditioning regression. A fundamentalist reading of "never deviate" would force her to ignore the best idea in her project. Olken's resolution, and ours: she *reports the pre-registered confirmatory result as planned* (it is what she promised), and *separately* reports the discovery as clearly-labeled exploratory analysis and a hypothesis for the next study. Both live in the paper, honestly partitioned. The PAP did not cost her the discovery; it cost her only the right to *pretend the discovery was a confirmatory test*, which she should not want anyway. A PAP can **crowd out exploration** if a field over-fetishizes it — if "was it pre-registered?" becomes the only question and the messy, inductive, pattern-hunting work that actually generates good hypotheses gets treated as second-class. This is the peril Olken weighs most carefully. The answer is the confirmatory/exploratory distinction held with a steady hand: exploration is not lesser science, it is *earlier* science. The hold-out design honors this directly — the entire exploration set exists *for* unconstrained fishing, because that is where Maya's best ideas will come from. The plan does not forbid exploration; it forbids *laundering* exploration as confirmation. And one peril particular to a high-schooler's first project: the PAP can give **false confidence**. A pre-registered design is not automatically a *good* design. Maya's identifying assumption is still imperfect — the credit score is still missing, the residual is still a bound — and pre-registering that imperfect design does not make it clean. Pre-registration governs the *honesty* of your inference (did you test or did you search?), not its *validity* (is the test the right one?). Validity is the job of the identification memo you write next, in Chapter 7.5. A PAP and an identification memo are complementary: one promises you did not fish, the other argues that what you caught is the fish you claim. You need both. --- ## 7.3.6 Where this goes next You arrived at this chapter with an idea (7.1) and data in hand (7.2). You now have the instrument that turns them into a *falsifiable, pre-registered design*: hypotheses stated directionally, one primary specification written in spec-discipline form, a multiple-testing plan that quarantines the garden of forking paths, explicit falsification conditions, planned robustness, a hold-out that makes "don't peek" mechanical, and a deviation log that keeps you honest when reality intrudes. Next you will *build the analysis dataset* the PAP refers to (7.4) — the merge keys, the survivorship and look-ahead checks, the winsorizing you already committed to here — and then write the *identification memo* (7.5) that defends the one assumption your $\beta_1$ rests on. The PAP says you will not fish; the memo says the pond is the right pond. Filed together as your Lab 7 tagged commit, they are the spine of a project that could survive a hostile referee. --- ## Your Turn Open **`nb7.3`** (`notebooks/week-07/nb7.3-pap-companion-power-calc.ipynb`), the **PAP companion**. It walks you from the five components to a filed plan for *your own* project, and its core is a **power calculation for your design** — the direct sequel to the power curves you built in nb1.5. You will: (1) state your effect size of interest in the *units of your outcome* (for Maya, the smallest conditional denial gap, in percentage points, that would matter to a real borrower); (2) estimate the residual standard deviation and the effective sample size *from your exploration set only*; (3) compute, exactly as in Chapter 1.5.4, the $N$ your design needs to detect that effect at $\alpha = 0.05$ with 80% power — and discover whether your data are even *capable* of testing your hypothesis before you waste a confirmation run on a study that was underpowered from the start; and (4) implement the Benjamini–Hochberg procedure on a family of p-values and watch it admit the right number of discoveries. The notebook ends by emitting a filled-in PAP template you tag as `pap-filed`, the deliverable for PS 7.3 and Lab 7. **Reflection prompts to carry into the lab and Mentor Session 7.** 1. *Name your three most dangerous forks.* In your own project, list the three specification choices (an outcome definition, a sample exclusion, a clustering level, a control, a prompt rubric) where, if you peeked at the result first, you would be most tempted to choose the branch that helps your hypothesis. For each, write the one sentence you will put in your PAP to nail it down in advance. 2. *Write your falsification sentence, then sit with it.* For your primary hypothesis, write the exact coefficient-and-confidence-interval pattern that would make you abandon it — phrased, as in §7.3.2, to distinguish a genuine refutation from a mere power failure. Would you actually report that null if it came? If your honest answer is "I'd keep looking," your PAP is not yet protecting you; find the fork you are still leaving open. 3. *Power before peeking.* Suppose your nb7.3 power calculation reveals that detecting the smallest effect you care about, at 80% power, would need three times the sample you have. What are your options *before* you run the confirmation regression — and why is "run it anyway and hope" the one option a pre-registered researcher has already foreclosed?