PS 8.1 — Specification Curve for Your Main Result

Course: 8-Week Empirical Finance Camp · Week 8 · Problem Set 8.1 Covers: Ch 8.1 (Execution & the Specification Curve), building on the PAP you froze in Ch 7.3, the identification memo you wrote in Ch 7.5, and the analysis dataset you built in Ch 7.4. Uses nb8.1 — the specification-curve generator. Type: Project deliverable, not a numeric problem set. You run your own pre-registered primary specification once on the confirmation data, then build and read the full specification curve around it, for the project you committed to in Week 7. There is no single right answer, because the answer is your curve — and what is graded is whether you executed the confirmatory run honestly, enumerated your forks defensibly, read the curve with the §8.1.3 checklist, and logged your deviations without flinching.

Total: 100 points. Point values are stated per part. A model deliverable — a full A-grade specification-curve write-up for one cast project, with instructor grading notes keyed to the rubric — is in Appendix E (E-w8-ps8.1-solutions.md). Read it after you draft your own, the way Ch 8.1 §8.1.3 walked through Maya’s curve: not to copy, but to calibrate.

A boundary you must hold onto for the whole assignment. This problem set probes analytic-choice multiplicity — whether your headline survives the defensible choices you could have made differently. It does not probe identification — whether the one design you committed to is even the right design. That is PS 8.2’s job, with placebos and Oster’s \delta. A specification curve can be flat, tight, and all-significant while every one of its specifications shares the same fatal parallel-trends violation (Ch 8.1 §8.1.4). Robustness across the multiverse is not validity of the design. Do not let this deliverable lull you into claiming your result is “proven”; it is, at most, not an artifact of your analytic choices. The intellectual frame throughout is Simonsohn, Simmons & Nelson (2020).

What you submit

A single Markdown file, spec-curve.md, committed to your project repository, plus the two-panel specification-curve figure it references (spec-curve.png or equivalent, generated by your nb8.1). The write-up has exactly five sections, in this order:

  1. The confirmatory result — your pre-registered primary specification, run once, reported whatever it shows (Part 1).
  2. The multiverse enumeration — your defensible forks, slot by slot, with the specification count (Part 2).
  3. The specification curve — the two-panel figure, with the pre-registered point marked (Part 3).
  4. The read — the one honest paragraph plus the four-point checklist applied to your curve (Part 4).
  5. The deviations log — the dated record of every departure from the PAP, each marked mechanical or outcome-driven (Part 5).

The whole thing is two to three pages plus the figure. The “read” paragraph is the one a referee will read first; write it as prose, not as notes.

Part 1 — The confirmatory result (15 points)

Run your pre-registered primary specification — the one your pap-filed commit names and your nb7.5 first-look executed once — on the confirmation data, exactly as written, no edits, no peeking-then-tweaking (Ch 8.1 §8.1.1). Report:

  • The specification, in CONVENTIONS §4 form (5 pts): all seven slots named to the level of a variable name — outcome, treatment/key regressor, controls, fixed effects, clustering, sample, identifying assumption — identical to the PAP. If it has drifted, that is a contract violation; revert, or flag it in the deviations log (Part 5) and explain why the drift was forced rather than chosen.
  • The number, with its inference (5 pts): the point estimate, the standard-error flavor labeled (clustered? HC3? wild-bootstrap?), and the 95% confidence interval. One headline coefficient, reported full stop.
  • The honest reading of it (5 pts): one or two sentences stating what the confirmatory number means, per the falsification clause you wrote in your PAP — a CI comfortably away from zero is a confirmation; a tight CI hugging zero is a genuine null; a wide CI straddling zero is a power failure reported as inconclusive, not a refutation. You commit to this reading whichever way the number fell.

A note worth re-reading from Ch 8.1 §8.1.1: a clean execution is not the same as a robust result. Part 1 answers “what does the pre-registered specification say?” It does not answer “is the answer an artifact of that one specification?” That is what the rest of this problem set is for. Do not let a starred coefficient tempt you to skip the multiverse — the single regression is one branch of a tree you could defensibly have climbed differently, and Parts 2–4 are where you find out whether the branch holds.

Part 2 — The multiverse enumeration (25 points)

This is the act of writing the garden down (Ch 8.1 §8.1.2, Step 1), and the discipline of the whole assignment lives here.

(a) Enumerate the defensible forks, slot by slot (15 pts). For each decision in your analysis, list every option a reasonable, competent researcher could have chosen — partitioned into the CONVENTIONS §4 slots (controls, fixed effects, sample window, clustering/inference, outcome definition). For each option, write one clause on why it is defensible. The grading is on the word defensible, in both directions:

  • Omitting an obvious defensible fork is penalized: if your design has a standard clustering debate (cluster at the treatment level vs. the unit) and you only list one, your enumeration is incomplete, and a reader cannot trust the curve.
  • Padding with indefensible forks is penalized harder, because it is p-hacking with extra steps (Ch 8.1 §8.1.4). A specification you would not defend out loud in a seminar does not belong in the multiverse; including it lets you smuggle in noise and then hide behind “but the median was fine.” Do not include a 6-month formation window you would never use just to inflate the count.

(b) Count the specifications (5 pts). Multiply the options across slots and state the cross-product — e.g., 4 \times 2 \times 3 \times 3 \times 2 = 144. Confirm that one of those specifications is your pre-registered primary — the confirmatory run is not outside the multiverse; it is a labeled, privileged point inside it.

(c) Connect to the multiple-testing arithmetic (5 pts): in one or two sentences, state why this count matters — the Week-1 result that with k specifications, even if the true effect were exactly zero, the probability that at least one lands at p<0.05 is 1-(1-0.05)^{k}, which is overwhelming for k in the dozens or hundreds (Ch 1.5, recalled in Ch 8.1 §8.1.2). Your multiverse measures the real size of your multiple-testing problem — the regressions you could defensibly have run — even though you ran only one for the confirmatory result. The spec curve does not eliminate that problem; it makes the whole distribution of what luck-plus-choice could produce visible, so one starred specification can never masquerade as the only one you considered.

Part 3 — The specification curve (20 points)

Using nb8.1, run every specification in your grid, collect the estimate, the confidence interval, and the choice-record for each, and draw the canonical two-panel figure (Ch 8.1 §8.1.3). Embed it in spec-curve.md and commit the image file.

  • Top panel (8 pts): the point estimates plotted as a curve, sorted by magnitude (most negative left, most positive right), each with a confidence-interval marker — filled/colored when significant at your chosen level, hollow when not. Because they are sorted, the curve rises monotonically; its shape is the story.
  • Bottom panel (8 pts): the choice grid — one row per analytic choice, one column per specification, aligned to the top panel’s sort order — with a tick at (row, column) when that specification used that choice. This is the panel that turns a pretty picture into a diagnostic: you read downward from any region of the curve to see exactly which choices produced it.
  • Mark the pre-registered point (4 pts): highlight where your confirmatory specification sits in the sorted order (a distinct marker, e.g., a diamond). This is non-negotiable — the curve is reported around the pre-registered point, never used to choose a new headline (Ch 8.1 §8.1.4, the fishing-license failure mode).

The figure must be legible: axis labeled in your estimate’s units (percentage points, basis points, log points), a zero line drawn, and a legend distinguishing significant from insignificant markers and the primary specification.

Part 4 — The read (25 points)

This is the conceptual heart of the assignment. Apply the four-point checklist from Ch 8.1 §8.1.3 to your curve, then distill it into one honest paragraph.

(a) The four-point checklist (15 pts), three to four points each:

  1. Where does the sign live? If estimates span both signs, your effect’s direction is choice-dependent — the most serious fragility. If they all share a sign, the direction is robust even if the magnitude wanders. State which.
  2. Where does the pre-registered specification sit? Near the center (representative) or out in a tail? If in a tail, you must explain why — and “I happened to pre-register the best-looking corner” is not flattering, even when innocent.
  3. What fraction is significant, and where do the exceptions cluster? Read the hollow markers downward into the bottom panel. Significance lost in the low-power corner (small samples, conservative inference) is benign; significance that appears only under one or two specific choices is a red flag.
  4. Which rows of the bottom panel drive the spread? Name the choice whose ticks best separate the left of the curve from the right. It is the fork your result is most sensitive to — the thing a skeptic will attack and your robustness section (PS 8.2) must address head-on. If no single fork drives the spread, say that, and explain why it is the stronger result.

(b) The one honest paragraph (10 pts). Write the single paragraph that goes in your Week-8 paper, in the style of Maya’s in Ch 8.1 §8.1.3: the range and median of the estimates across all specifications, where the pre-registered estimate sits, the fraction significant, and — if any specifications lose significance or flip sign — which choices produce them and whether that pattern is benign (low-power corner) or damning (one fork flips your sign). The verdict word is robust or fragile, and the paragraph must earn it. “Robust” is licensed only when the sign never flips, the magnitude is stable, and any loss of significance is confined to specifications that are independently the least powerful, for reasons the bottom panel makes legible. A curve that sweeps across zero with a mix of filled and hollow markers is fragile, and the honest sentence names the fork that flips it — the way Sam’s momentum curve confesses that the premium is positive only under equal-weighting with microcaps included (Ch 8.1 §8.1.3). Do not write “robust” because you want it to be; write what the curve says.

Part 5 — The honest deviations log (15 points)

Paste the running, dated record of every departure from your PAP on the confirmation data (Ch 8.1 §8.1.5). Deviating is not the sin; hiding the deviation is, and outcome-driven deviation is the specific sin the whole apparatus exists to catch. Every entry carries a verdict on one binary question: was this change forced on me by a mechanical reality, or did I make it because I saw the coefficient and did not like it?

  • Mechanical deviations (innocent): collinearity, a coding bug, a data-availability surprise, a corrupted cluster identifier. Note what changed, why, and — where possible — that the estimate barely moved. Example: “PAP §2 specified clustering by state; the state identifier was missing for 4% of county-year cells from a FIPS-merge gap, so those cells were clustered by census region; ATT moved -1.89 \to -1.91, immaterial. Not outcome-driven.”
  • Outcome-driven deviations (the dangerous kind): you added a control after seeing it improved your t-stat, trimmed a year after seeing it weakened the effect, switched clustering because the pre-registered choice was insignificant. The honest response is not to pretend it didn’t happen but to report the result both ways and flag the deviating version as exploratory, never as the headline.

Grading:

(a) Completeness and honesty (7 pts). Every departure logged, dated, with the mechanical/outcome-driven verdict stated explicitly. If you genuinely made zero deviations, say so and state that the confirmatory run executed exactly as filed — that is a valid and strong entry, not an empty one.

(b) The deviations-to-multiverse audit (8 pts). For every deviation you were tempted to make for outcome-driven reasons — “should I have clustered by county? added that control? dropped that year?” — confirm it is already a fork in your multiverse, and locate it: “that alternative is specification number 87; here is where it sits; the headline does not depend on it” (Ch 8.1 §8.1.5). The spec curve is the honest home for every fork you were tempted to walk; the deviations log is the record of the few you actually had to. If you cannot find a deviation’s alternative anywhere in your curve, your Part-2 enumeration was incomplete — go back and add it.

Submission checklist

Before you commit, confirm:

How this is graded

Assessment 8 grades this specification-curve deliverable and the PS 8.2 robustness deliverable together, on the empirical-execution rubric. The point allocation above maps onto it: Part 1 → confirmatory discipline (run-what-you-filed, report-whatever-it-shows); Part 2 → the enumerate-the-garden skill and the multiple-testing logic that motivates it; Parts 3–4 → reading the curve forensically (the checklist, the earned verdict, the named fork); Part 5 → intellectual honesty (the confirmatory/exploratory line drawn in public). The single highest-signal question a grader asks: does the verdict word match what the curve actually shows, and is the pre-registered point marked and reported around — not chosen by — the multiverse? A student who clears that has the Week-8 mindset. The model deliverable in Appendix E shows the A-grade standard for one cast project, annotated with the rubric levels each section hits.

The bridge to PS 8.2

Hold the boundary one more time: this deliverable answered “did my choices drive the result?” It is silent on “did my design identify the effect?” Every specification in even a gloriously robust, all-one-sign curve can share the same fatal identification flaw — a broken parallel-trends assumption, a leaky exclusion restriction — and the curve will robustly report a beautifully stable artifact (Ch 8.1 §8.1.4). PS 8.2 turns from analytic-choice multiplicity to inference validity: it takes the threats-and-responses table you built in PS 7.5 and makes column 3 come due — the placebos that would have fired if the design were broken, the alternative standard errors, and Oster’s \delta, which asks how much unobserved selection it would take to explain away your effect. The fork you named in Part 4(a)4 — the choice your result is most sensitive to — is exactly the one PS 8.2 must address head-on. A result that survives the multiverse and the placebos and the bound on omitted-variable bias is one you can defend in Ch 8.3 and hand to a hostile peer reviewer in Ch 8.4 without flinching — because every honest attack has already been made, by you, first.

End of PS 8.1. The model deliverable is in book/appendices/E-solutions-manual/E-w8-ps8.1-solutions.md. Draft your own write-up first; read the exemplar to calibrate, not to copy. The skeptic who reads a stranger’s spec curve and the author who builds an honest one are the same person — this assignment is you showing the whole orchard, on purpose, before a referee can accuse you of picking one tree.