Ch 8.4 — Peer Review & Revision

You have spent three chapters building your own paper: you ran the pre-registered specification and logged your deviations (8.1), you broke it on purpose with placebos and alternative standard errors (8.2), and you wrote it up to the style guide (8.3). This chapter teaches the skill that sits on the other side of the desk — the one that turns a room full of student papers into a room full of better student papers. You are going to learn to referee someone else’s paper, and then to respond when someone referees yours.

Here is the trick this chapter reveals, stated once before the machinery. Refereeing and writing are the same skill aimed in opposite directions, and you already learned the skill in Weeks 5 and 6. When you read Fama–French tables-first, asking “what is the identifying assumption and what threatens it,” you were rehearsing for exactly this. A referee is just a Week-5 reader with a deadline and a verdict to render. The only thing new in Week 8 is that the paper on the desk was written by a peer who will read what you say, and that you owe them a particular kind of toughness: tough on the work, kind to the person. That distinction is not a politeness; it is the entire professional norm, and a referee who confuses the two — who is soft on a broken design because the author is nice, or cruel to a person because the paper is weak — is failing at the job in both directions.

We will build the referee report first, because you cannot respond to one you cannot write. Then we will build the revise-and-resubmit memo — the point-by-point response letter that is the most underrated document in all of empirical research, and the one that, more than the paper itself, tells an editor whether you are someone worth publishing. Throughout, the running example is Maya’s fair-lending difference-in-differences, the project whose identification memo you watched her draft in Chapter 7.5. Now a classmate — call him Devon, wearing his referee hat — has her draft on his desk.

1. What a referee is actually for

Strip away the ceremony and a referee does two jobs. The first is a gatekeeping verdict: should this paper, in some form, exist in the record? The second, and far more important for you, is improvement: how does this specific paper become the best version of itself? In a journal, an anonymous referee sends a confidential report to an editor, who weighs it against one or two other reports and decides among reject, revise-and-resubmit, or (rarely, on a first pass) accept. In the camp’s peer-review workshop, the structure is the same minus the anonymity: you are assigned a peer’s draft, you write a report, and you both sit in a room where the report is the agenda.

The mindset that makes a referee useful is not “find what is wrong.” It is “figure out what the paper is claiming, decide whether to believe it, and tell the author exactly what would make you believe it more.” Notice that this is the Week-5 self-check rubric — the question, the identifying assumption, the threat, the headline table — turned into a service you perform for someone else. A referee who only catches typos has read the abstract. A referee who reconstructs the identifying assumption, names the one alternative explanation that would sink the headline result, and says precisely what evidence would rule it out has done the job. The author may not enjoy that report, but they will be unable to do without it.

There is an asymmetry worth internalizing now, because it shapes everything that follows. Your job is not to rewrite the paper into the one you would have written. It is the author’s paper, their question, their design. A common failure of beginner referees is to spend three paragraphs explaining how they would have approached the topic — a different dataset, a different identification strategy, a different framing entirely. That is not refereeing; it is hijacking. The referee’s question is narrower and harder: given the paper the author chose to write, is the claim credible, and what is the shortest path to making it more so? You may, in one sentence, note that an alternative design exists. You may not demand the author become a different researcher.

2. The anatomy of a referee report

A good referee report has a fixed skeleton, and the order is load-bearing in exactly the way the Week-5 reading order was. You read tables-first; you write the report concern-first. Here is the structure, top to bottom.

(0) A summary, in your own words. Open with one paragraph — three to six sentences — that states what the paper does: the question, the data, the design, and the headline finding with its magnitude. This is not filler and it is not flattery. It is proof that you understood the paper, and it is the single most reassuring thing an author can read, because the deepest fear of anyone who has just submitted a paper is that the reader did not get it. Write the summary without looking at the author’s abstract — write it the way you would summarize Fama–French from the tables — and then the author can check your summary against their intent. If your summary describes a paper they did not write, that mismatch is itself the most important finding in your report: the paper failed to communicate its own claim, and no amount of robustness fixes that.

(1) The single biggest concern, stated first. After the summary, lead with the one issue that most threatens the paper — the thing that, if the author cannot answer it, sinks the result. Do not bury it; do not build to it; do not soften it with five smaller points first. A referee report that opens with “Table 3’s caption has a typo” and reaches the broken identifying assumption on page two has mis-ranked its own concerns and wasted the author’s attention. Put the dagger on the table first. In almost every empirical paper, this biggest concern is an identification threat — the alternative explanation for the headline result that the design did not rule out. Naming it well is the highest-value sentence you will write as a referee.

(2) Then identification threats, in descending order of danger. Having led with the biggest one, work through the rest of the threats to the causal (or at least credible) interpretation. This is the heart of the report and it is pure Week-5 reading: for the design the author chose, what is on the standard menu of failures from Chapter 7.5? For a difference-in-differences, that menu is pre-trends, anticipation, and — in staggered designs — the forbidden comparison of already-treated units. You go threat by threat and ask, for each: did the author address it, did they address it convincingly, and if not, what specific evidence would settle it?

(3) Then specification and robustness. Below identification sit the questions of whether, granting the design, the estimate is fragile. Are the standard errors the right flavor — clustered at the level the treatment varies, as Petersen and Bertrand–Duflo–Mullainathan drilled in Week 5? Does the headline coefficient survive the march across columns as controls are added (8.1’s specification curve), or does it collapse? Are the obvious placebo and falsification tests run (8.2)? Is the multiple-testing problem handled if the author reports many outcomes? These are real and they matter, but they are downstream of identification: a perfectly clustered standard error on a coefficient with no credible design is a precise estimate of nothing.

(4) Then exposition. Last come the writing issues: an introduction that overclaims relative to the tables, a contribution sentence you cannot find, a results section that narrates the coefficients instead of interpreting them, tables that violate the Appendix D conventions, a figure with no axis labels. Exposition matters — a paper a reader cannot follow is a paper a reader will not believe — but it goes last because fixing prose on a paper with a fatal design is rearranging deck chairs. Put it last so the author reads it last, after they have absorbed what actually matters.

You will notice this is the exact inversion of how the author experiences the paper. The author spent the most time on the prose and the least on naming their own worst threat; a good referee report reverses that ranking and hands it back. That reversal is the gift.

3. Major vs. minor, and the verdict

Inside the report, sort every point into major or minor, and label them as such — most reports use two numbered lists under those headings. A major point is one whose resolution could change the paper’s conclusion or your belief in it: an unaddressed identification threat, a standard-error flavor that inflates the t-stats, a sample filter that might drive the result, a contribution claim the evidence does not support. A minor point is one that improves the paper without touching the verdict: a clearer figure, a missing citation, a confusing sentence, a table that should report the within-R^2. The labeling is a courtesy and a discipline at once. It tells the author where to spend their limited revision energy, and it forces you to decide, for each comment, whether you actually think it could change the result — which is a sharper question than “is this worth mentioning.”

A rule that separates a generous referee from a lazy one: for every major point, say what evidence would change your mind. Do not write “I am not convinced the parallel-trends assumption holds.” Write “I am not convinced parallel trends holds; an event-study plot of the denial gap with at least four pre-period leads would let me judge, and flat, insignificant leads would substantially reassure me.” The first is a complaint; the second is a contract — it tells the author exactly what to produce and tells them, in advance, that producing it will move you. A referee who states the falsifying or reassuring evidence for every major concern has written a report the author can act on, which is the only kind worth writing.

Then, separate from the report itself, comes your confidential note to the editor with the verdict. In the camp workshop this is the line you write at the top of your report: is this, in your judgment, a reject, a major revision, a minor revision, or near-ready? Be honest, and be calibrated. A student paper that ran a clean design on real data, found a result, and stress-tested it is a major or even minor revision even if you found ten things to fix — the bones are sound. A paper whose headline rests on a design that cannot identify the claimed effect is a reject as currently identified, no matter how clean the prose, and saying so plainly is kinder than a vague “promising but needs work” that wastes the author’s next month.

4. Tough on the work, kind to the person

This is the norm that holds the whole enterprise together, and it is worth stating as flatly as possible. Criticize the paper as hard as the evidence demands; never criticize the author at all. The two are easy to conflate when you are tired and the paper is frustrating, and the conflation is corrosive — it makes the author defensive, which makes them argue back instead of fixing things, which makes the paper worse and the workshop hostile.

The mechanics are concrete. Write about the paper, not the person: “the identification section does not rule out anticipation,” not “you didn’t think about anticipation.” Attribute claims to the text, not to a supposed failing of mind: “Table 3 reports robust standard errors on panel data, which Petersen (2009) shows understates the true sampling variability,” not “you used the wrong standard errors because you don’t understand clustering.” Assume competence and good faith — the author is a colleague who made defensible choices you happen to disagree with, not a student who needs scolding. And calibrate the warmth to the toughness: the harder a point, the more carefully you phrase it, not because the point softens but because a hard point delivered carelessly gets rejected for its tone instead of accepted for its content.

None of this means going easy. The kindest thing you can do for a peer is to find the fatal flaw before the assessment grader does, and to find it so clearly that they can fix it. False reassurance is the true cruelty: a referee who waves through a broken design to spare someone’s feelings has guaranteed that someone a harsher audience will break it later, when it costs more. Tough on the work is the kindness. Be precise, be specific, be relentless about the evidence — and never once make it about the person holding the pen.

5. A worked mini referee report — Devon on Maya’s paper

Maya’s draft, the one whose identification memo you read in Chapter 7.5, asks whether state fair-lending examinations narrowed the county-level minority–white mortgage-denial gap, using a staggered difference-in-differences on HMDA data. Devon has it on his desk. Here is the core of his report — abridged, but in the real shape.

Referee Report — “Do State Fair-Lending Examinations Narrow the Mortgage-Denial Gap?”

Summary. The paper asks whether state-level fair-lending examination programs reduced the gap in mortgage-denial rates between minority and white applicants. Using HMDA loan-application data aggregated to county-year, the author estimates a staggered difference-in-differences comparing counties in adopting states to counties in not-yet- and never-adopting states, with county and year fixed effects and standard errors clustered by state. The headline result (Table 2, column 3) is that examinations narrow the denial gap by about 1.4 percentage points, roughly a fifth of the pre-period mean gap, significant at the 1% level. The design and the question are both well-chosen, and the institutional setup is clearly explained.

Major points.

(Biggest concern — identification: anticipation and pre-trends.) The entire causal claim rests on parallel trends, and I cannot currently assess it because the paper reports no event study. This is my central concern. Fair-lending examination programs are typically debated in a state legislature for a year or more before adoption, which is exactly the condition under which lenders might change behavior in anticipation — tightening or loosening before the official examination date — contaminating the pre-period. I would need an event-study plot of the denial gap with at least four pre-period leads and several post-period lags. Flat, insignificant leads would substantially reassure me; a downward pre-trend in adopting states would be close to fatal, because it would mean the gap was already closing before the policy and the design is mislabeling a trend as an effect. As written, I cannot tell which world we are in.

(Identification: the forbidden comparison.) The design is staggered, and the methods section says “two-way fixed effects” without addressing the negative-weights problem (Chapter 4.2): already-treated counties are serving as controls for later-treated ones, which can bias the estimate when the effect is dynamic — and an anticipation story makes dynamic effects likely. I would want the headline re-estimated with a heterogeneity-robust estimator (Callaway–Sant’Anna against clean controls only) and the two numbers compared. If they agree, the TWFE number is safe to report; if they diverge, the robust one is the result.

(Specification: the denominator of the denial gap.) The county-year denial gap is constructed from raw HMDA application counts, but the text does not say whether counties with very few minority applications are dropped or winsorized. A handful of small counties where the gap is mechanically volatile could be driving the estimate. A robustness column weighting by application volume, or dropping counties below a minimum application threshold, would tell us whether the result is real or a small-cell artifact.

Minor points.

Table 2’s note does not state the significance thresholds for the stars; per the convention, the note should be explicit (read-the-note, never-assume).

The introduction claims the paper “establishes that examinations cause fairer lending,” which is stronger than a credible-conditional-on-parallel-trends DiD can support; I would soften “establishes… cause” to match the design.

Figure 1 has no y-axis label; I assume it is the denial gap in percentage points.

The Bertrand–Duflo–Mullainathan (2004) citation is the right one for the clustering choice and should be cited where clustering is introduced, not only in the references.

Verdict. Major revision. The question is good, the data are real, and the design is the right one; the paper is one event study and one robust-estimator column away from being convincing. None of my major points asks the author to change the project — only to defend the design it already has.

Read what Devon did against Section 2. He summarized in his own words first (proving he got it). He led with the biggest concern and named it specifically — anticipation, not “endogeneity.” He ordered the majors by danger and put exposition in the minors. For every major point he stated the evidence that would move him, and for the worst one he stated what would reassure and what would be fatal. He was tough — he flagged a potentially fatal pre-trend and a softened-this-claim overreach — and entirely kind, closing with a sentence that tells Maya her project is sound and the path is short. That last sentence is not flattery; it is an accurate verdict, and an accurate encouraging verdict is the most motivating thing a referee can write.

6. Responding to a referee — the R&R memo

Now flip the desk. The editor has returned your paper with a “revise and resubmit” — the most common good outcome in empirical research, and a small victory, because it means the design was credible enough to be worth saving. Attached are the referee reports. Your job is to produce a revised paper and a document that matters at least as much: the response memo, also called the response-to-referees or the R&R letter. It is a point-by-point reply to every comment, and an editor reads it before re-reading the paper, because it tells them whether you are someone who can take criticism and act on it — which is, in the long run, more predictive of a good career than any single result.

The structure is rigid and merciful in its rigidity. For each comment, you do three things, in this order: quote the comment, state what you changed, and point to the evidence. Quote it so the editor does not have to hold two documents side by side — paste the referee’s words verbatim, usually in a blockquote or italics, then respond beneath. State what you changed in plain, specific language: “We have added an event study as new Figure 2 and discuss it in Section 4.2.” Then point to the result: the new table number, the new figure, the page, the actual number that the new analysis produced. A response that says “we have addressed this concern” without saying where and with what number is worthless; the editor cannot verify it and will assume you did nothing.

The deepest norm of the whole exercise, and the one the camp grades hardest, is this: do exactly what you say you did. If your memo claims you added an event study, the event study must be in the paper, it must be the one you describe, and its leads must say what your memo says they say. The fastest way to lose a referee’s trust permanently — in the workshop and for the rest of your career — is a response memo whose claims the revised paper does not back up. Referees check. The whole institution of peer review runs on the assumption that the response memo is honest, and a single caught lie poisons every other claim in the document. Honesty here is not optional ethics; it is the load-bearing wall.

7. When to concede, and when to defend

Not every referee comment is right, and a memo that caves to all of them is as weak as one that fights all of them. The skill is knowing which is which, and the rule is the same one that runs through this whole book: follow the evidence, not the ego.

Concede when the referee is right, and concede gracefully. If they caught a real flaw, fix it, say so plainly, and thank them — “The referee is correct; we had clustered at the wrong level. We now cluster by state, the level at which examinations vary, and the t-statistic on the main effect falls from 4.1 to 2.6, which we report and discuss.” Note what just happened: conceding made the result weaker, and reporting the weaker number honestly is exactly the move that earns trust. A referee who sees you report a t-stat that dropped from 4.1 to 2.6 because they were right believes the 2.6 far more than they would have believed the 4.1. Honest concession is not surrender; it is the strongest evidence you can offer that the surviving result is real.

Defend when the referee is wrong — respectfully, with evidence, never with attitude. Sometimes a referee misreads the paper, asks for a test that does not apply, or proposes an alternative explanation you can actually rule out. You are allowed to disagree, and a good editor respects a well-argued rebuttal more than a reflexive capitulation. But the defense must be evidence, not assertion: “We respectfully disagree that small-cell volatility drives the result. As the referee suggests we have checked this directly: weighting by application volume (new column 4) leaves the estimate essentially unchanged at 1.3 pp, and dropping counties below 50 applications (column 5) gives 1.5 pp. We have added these columns and a sentence noting the result is not a small-cell artifact.” That is a defense, and it is a better paper for the challenge: Maya disagreed, but she ran the test the referee implied, reported it, and her result is now armored against exactly the concern the referee raised. The cleanest defenses do the requested work and then explain why the worry does not bite.

There is also the comment you can neither fully fix nor fully rebut — the untestable threat, like the exclusion restriction in an IV or the parallel-trends counterfactual itself. Here you neither concede the paper is broken nor pretend the concern is gone. You bound it: acknowledge the threat is real and untestable, present every piece of indirect evidence you have, and state the residual concern honestly in the paper’s limitations — exactly the fourth column of the threats-and-responses table from Chapter 7.5, now promoted into the response memo. “The referee is correct that anticipation cannot be ruled out by the data alone. We present the event study (flat leads, Figure 2) as the best available evidence against it, and we now state in Section 6 that anticipation remains the limitation most likely to bias our estimate, and in which direction.” Acknowledging a residual limitation is not weakness; a memo that pretends every threat is vanquished reads as naïve, and a referee trusts the author who says “here is what still worries me” far more than the one who claims a clean sweep.

8. A worked R&R response excerpt — Maya answers Devon

Maya has revised. Here is the core of her response memo — the three major points, in the quote-change-evidence shape, showing one concession, one defense, and one bounded threat.

Response to Referee 1

We thank the referee for an unusually constructive report. Below we respond to each point; referee comments are in italics, our responses follow. New material is flagged in the revised manuscript in blue.

Major point 1 (anticipation / pre-trends). “The entire causal claim rests on parallel trends, and I cannot currently assess it because the paper reports no event study… Flat, insignificant leads would substantially reassure me; a downward pre-trend would be close to fatal.”

This was the right thing to ask first, and we are glad we ran it. We have added an event study as new Figure 2 and discuss it in Section 4.2. The four pre-period leads are individually insignificant and jointly insignificant (joint F-test p = 0.41), with no visible downward drift in the denial gap before adoption; the effect emerges in the year of examination and grows over the following two years. We read this as consistent with — though, as we now stress, not proof of — parallel trends and against an anticipation story. We have added one sentence to the limitations (Section 6) noting that flat leads cannot confirm the counterfactual, only fail to refute it.

Major point 2 (forbidden comparison / staggered DiD). “The design is staggered… already-treated counties are serving as controls… I would want the headline re-estimated with a heterogeneity-robust estimator and the two numbers compared.”

We agree this was a real gap. We have re-estimated the headline using the Callaway–Sant’Anna estimator against never- and not-yet-treated controls only (new Table 3). The robust estimate is −1.5 pp, very close to the −1.4 pp TWFE estimate, suggesting the negative-weights problem is not material here — consistent with effects that, while dynamic, are similarly signed across cohorts. We now report both numbers and note the agreement in Section 5.1. We retain TWFE as the headline for comparability but flag the robust estimate as the one we would defend if the two diverged.

Major point 3 (small-cell denominator). “A handful of small counties where the gap is mechanically volatile could be driving the estimate… a robustness column weighting by application volume, or dropping small counties, would tell us whether the result is real.”

We respectfully believe the result is not a small-cell artifact, and we have checked this directly as the referee suggests. New columns 4 and 5 of Table 2 weight by application volume (estimate −1.3 pp) and drop counties below 50 minority applications per year (−1.5 pp). The estimate is stable across both. We have added a sentence in Section 5.2 reporting this and concluding the result is not driven by mechanically volatile small counties.

We believe the paper is substantially stronger for these three additions — particularly the event study, which we should have included in the first submission. We are grateful for the report.

Look at what each response does. Point 1: a clean addition of requested evidence, with the actual joint-test p-value and an honest note that flat leads cannot confirm the counterfactual — she bounded the untestable threat rather than overclaiming it away. Point 2: a concession that the gap was real, the requested estimator run, and both numbers reported side by side — and crucially, the numbers agree, so the concession cost her nothing and bought her enormous credibility. Point 3: a respectful defense that does the requested work first and then states the conclusion, so the disagreement is settled by evidence rather than argument. Every response quotes, states the change, and points to a specific new table or figure with a specific number. And every change Maya claims is a change the revised paper actually contains — the norm of Section 6, honored. Devon, re-reading, can verify each one in under a minute, and his next report to the editor will say “the author addressed every point; recommend acceptance.”

9. Why this is the same skill as Week 5, one more time

It is worth closing the loop explicitly, because the symmetry is the point. In Week 5 you learned to read a paper tables-first and to ask, of any design, “what is the identifying assumption, and what threatens it.” In Chapter 7.5 you learned to write that assumption down for your own paper and to list its threats in a four-column table. The referee report is that same threats table, written about someone else’s paper. The R&R memo is your own threats table, promoted from a private worksheet into a public defense, with the referee’s concerns slotted in beside the ones you anticipated. A referee reads tables-first and asks the identification question first because that is how the credibility of a result is actually decided; you respond table-by-table and evidence-by-evidence because that is how credibility is actually rebuilt.

The student who internalizes this stops experiencing peer review as a hazing ritual and starts experiencing it as the engine it is. A good referee makes your paper better whether you agree with them or not — if they are right you fix a flaw, and if they are wrong you produce the evidence that proves they are wrong, which armors the paper either way. There is no losing move except the dishonest one. Reveal the trick to a peer so they can fix theirs; let a peer reveal the trick on yours so you can fix it before the grader does. That exchange, performed in good faith and on the evidence, is the whole of what a research community is.

Your Turn

Workshop pointer. This chapter is the engine of the Week-8 peer-review workshop and of PS 8.4, where you will be assigned a classmate’s capstone draft, write a full referee report on it, and then — after receiving the report someone else wrote on your paper — produce an R&R memo responding point by point. Bring your draft (from 8.1–8.3) and come ready to be both referee and author in the same session. Mentor Session 8, “Defending a result: what a referee actually asks,” will rehearse the live version of this exchange — the questions a referee fires in a seminar — tied to Prof. Gao’s work on disclosure (Elnahas, Gao, Hossain & Kim, 2024, JFQA). The revised paper and replication packet you defend in Chapter 8.5 should already reflect the changes your R&R memo promised: do what you said you did.

When you write your report, work the skeleton in order: summary in your own words, biggest concern first, then identification, specification, exposition; major and minor labeled; for every major point, the evidence that would change your mind. When you write your memo, work each comment in the quote-change-evidence shape, and decide consciously for each whether you are conceding, defending, or bounding.

Reflection prompts.

Pick one comment from a report you received that you initially wanted to argue against. Reconstruct, honestly, whether your instinct to defend was about the evidence or about your ego — and what the evidence actually says now that you have looked.
A referee asks for a test that, when you run it, weakens your headline result (a t-stat falls, a coefficient shrinks). Walk through why reporting the weaker number in your memo can make a reader believe your paper more, not less. What does this tell you about why honest concession is a strength?
Your single biggest concern about your own paper — the threat at the top of your Chapter 7.5 table — was it raised by your referee? If yes, did your response actually resolve it or only bound it? If no, why do you think a fresh reader did not see what you saw, and does that change how worried you should be?

# Ch 8.4 — Peer Review & Revision You have spent three chapters building your own paper: you ran the pre-registered specification and logged your deviations (8.1), you broke it on purpose with placebos and alternative standard errors (8.2), and you wrote it up to the style guide (8.3). This chapter teaches the skill that sits on the other side of the desk — the one that turns a room full of student papers into a room full of *better* student papers. You are going to learn to **referee** someone else's paper, and then to **respond** when someone referees yours. Here is the trick this chapter reveals, stated once before the machinery. **Refereeing and writing are the same skill aimed in opposite directions, and you already learned the skill in Weeks 5 and 6.** When you read Fama–French tables-first, asking "what is the identifying assumption and what threatens it," you were rehearsing for exactly this. A referee is just a Week-5 reader with a deadline and a verdict to render. The only thing new in Week 8 is that the paper on the desk was written by a peer who will read what you say, and that you owe them a particular *kind* of toughness: tough on the work, kind to the person. That distinction is not a politeness; it is the entire professional norm, and a referee who confuses the two — who is soft on a broken design because the author is nice, or cruel to a person because the paper is weak — is failing at the job in both directions. We will build the referee report first, because you cannot respond to one you cannot write. Then we will build the **revise-and-resubmit memo** — the point-by-point response letter that is the most underrated document in all of empirical research, and the one that, more than the paper itself, tells an editor whether you are someone worth publishing. Throughout, the running example is Maya's fair-lending difference-in-differences, the project whose identification memo you watched her draft in Chapter 7.5. Now a classmate — call him Devon, wearing his referee hat — has her draft on his desk. --- ## 1. What a referee is actually for Strip away the ceremony and a referee does two jobs. The first is a *gatekeeping* verdict: should this paper, in some form, exist in the record? The second, and far more important for you, is *improvement*: how does this specific paper become the best version of itself? In a journal, an anonymous referee sends a confidential report to an editor, who weighs it against one or two other reports and decides among reject, revise-and-resubmit, or (rarely, on a first pass) accept. In the camp's peer-review workshop, the structure is the same minus the anonymity: you are assigned a peer's draft, you write a report, and you both sit in a room where the report is the agenda. The mindset that makes a referee useful is not "find what is wrong." It is **"figure out what the paper is claiming, decide whether to believe it, and tell the author exactly what would make you believe it more."** Notice that this is the Week-5 self-check rubric — the question, the identifying assumption, the threat, the headline table — turned into a service you perform for someone else. A referee who only catches typos has read the abstract. A referee who reconstructs the identifying assumption, names the one alternative explanation that would sink the headline result, and says precisely what evidence would rule it out has done the job. The author may not enjoy that report, but they will be unable to do without it. There is an asymmetry worth internalizing now, because it shapes everything that follows. **Your job is not to rewrite the paper into the one you would have written.** It is the author's paper, their question, their design. A common failure of beginner referees is to spend three paragraphs explaining how *they* would have approached the topic — a different dataset, a different identification strategy, a different framing entirely. That is not refereeing; it is hijacking. The referee's question is narrower and harder: *given the paper the author chose to write, is the claim credible, and what is the shortest path to making it more so?* You may, in one sentence, note that an alternative design exists. You may not demand the author become a different researcher. --- ## 2. The anatomy of a referee report A good referee report has a fixed skeleton, and the order is load-bearing in exactly the way the Week-5 reading order was. You read tables-first; you *write* the report concern-first. Here is the structure, top to bottom. **(0) A summary, in your own words.** Open with one paragraph — three to six sentences — that states what the paper does: the question, the data, the design, and the headline finding with its magnitude. This is not filler and it is not flattery. It is *proof that you understood the paper*, and it is the single most reassuring thing an author can read, because the deepest fear of anyone who has just submitted a paper is that the reader did not get it. Write the summary without looking at the author's abstract — write it the way you would summarize Fama–French from the tables — and then the author can check your summary against their intent. If your summary describes a paper they did not write, that mismatch is itself the most important finding in your report: the paper failed to communicate its own claim, and no amount of robustness fixes that. **(1) The single biggest concern, stated first.** After the summary, lead with the one issue that most threatens the paper — the thing that, if the author cannot answer it, sinks the result. Do not bury it; do not build to it; do not soften it with five smaller points first. A referee report that opens with "Table 3's caption has a typo" and reaches the broken identifying assumption on page two has mis-ranked its own concerns and wasted the author's attention. Put the dagger on the table first. In almost every empirical paper, this biggest concern is an **identification threat** — the alternative explanation for the headline result that the design did not rule out. Naming it well is the highest-value sentence you will write as a referee. **(2) Then identification threats, in descending order of danger.** Having led with the biggest one, work through the rest of the threats to the *causal* (or at least credible) interpretation. This is the heart of the report and it is pure Week-5 reading: for the design the author chose, what is on the standard menu of failures from Chapter 7.5? For a difference-in-differences, that menu is pre-trends, anticipation, and — in staggered designs — the forbidden comparison of already-treated units. You go threat by threat and ask, for each: did the author address it, did they address it *convincingly*, and if not, what specific evidence would settle it? **(3) Then specification and robustness.** Below identification sit the questions of whether, *granting* the design, the estimate is fragile. Are the standard errors the right flavor — clustered at the level the treatment varies, as Petersen and Bertrand–Duflo–Mullainathan drilled in Week 5? Does the headline coefficient survive the march across columns as controls are added (8.1's specification curve), or does it collapse? Are the obvious placebo and falsification tests run (8.2)? Is the multiple-testing problem handled if the author reports many outcomes? These are real and they matter, but they are *downstream* of identification: a perfectly clustered standard error on a coefficient with no credible design is a precise estimate of nothing. **(4) Then exposition.** Last come the writing issues: an introduction that overclaims relative to the tables, a contribution sentence you cannot find, a results section that narrates the coefficients instead of interpreting them, tables that violate the Appendix D conventions, a figure with no axis labels. Exposition matters — a paper a reader cannot follow is a paper a reader will not believe — but it goes last because fixing prose on a paper with a fatal design is rearranging deck chairs. Put it last so the author reads it last, after they have absorbed what actually matters. You will notice this is the exact inversion of how the author *experiences* the paper. The author spent the most time on the prose and the least on naming their own worst threat; a good referee report reverses that ranking and hands it back. That reversal is the gift. --- ## 3. Major vs. minor, and the verdict Inside the report, sort every point into **major** or **minor**, and label them as such — most reports use two numbered lists under those headings. A **major point** is one whose resolution could change the paper's conclusion or your belief in it: an unaddressed identification threat, a standard-error flavor that inflates the t-stats, a sample filter that might drive the result, a contribution claim the evidence does not support. A **minor point** is one that improves the paper without touching the verdict: a clearer figure, a missing citation, a confusing sentence, a table that should report the within-$R^2$. The labeling is a courtesy and a discipline at once. It tells the author where to spend their limited revision energy, and it forces *you* to decide, for each comment, whether you actually think it could change the result — which is a sharper question than "is this worth mentioning." A rule that separates a generous referee from a lazy one: **for every major point, say what evidence would change your mind.** Do not write "I am not convinced the parallel-trends assumption holds." Write "I am not convinced parallel trends holds; an event-study plot of the denial gap with at least four pre-period leads would let me judge, and flat, insignificant leads would substantially reassure me." The first is a complaint; the second is a contract — it tells the author exactly what to produce and tells them, in advance, that producing it will move you. A referee who states the falsifying or reassuring evidence for every major concern has written a report the author can *act on*, which is the only kind worth writing. Then, separate from the report itself, comes your confidential note to the editor with the verdict. In the camp workshop this is the line you write at the top of your report: is this, in your judgment, a **reject**, a **major revision**, a **minor revision**, or near-ready? Be honest, and be calibrated. A student paper that ran a clean design on real data, found a result, and stress-tested it is a *major* or even *minor* revision even if you found ten things to fix — the bones are sound. A paper whose headline rests on a design that cannot identify the claimed effect is a reject *as currently identified*, no matter how clean the prose, and saying so plainly is kinder than a vague "promising but needs work" that wastes the author's next month. --- ## 4. Tough on the work, kind to the person This is the norm that holds the whole enterprise together, and it is worth stating as flatly as possible. **Criticize the paper as hard as the evidence demands; never criticize the author at all.** The two are easy to conflate when you are tired and the paper is frustrating, and the conflation is corrosive — it makes the author defensive, which makes them argue back instead of fixing things, which makes the paper worse and the workshop hostile. The mechanics are concrete. Write about the paper, not the person: "the identification section does not rule out anticipation," not "you didn't think about anticipation." Attribute claims to the text, not to a supposed failing of mind: "Table 3 reports robust standard errors on panel data, which Petersen (2009) shows understates the true sampling variability," not "you used the wrong standard errors because you don't understand clustering." Assume competence and good faith — the author is a colleague who made defensible choices you happen to disagree with, not a student who needs scolding. And calibrate the warmth to the toughness: the harder a point, the more carefully you phrase it, not because the point softens but because a hard point delivered carelessly gets rejected for its tone instead of accepted for its content. None of this means going easy. The kindest thing you can do for a peer is to find the fatal flaw *before* the assessment grader does, and to find it so clearly that they can fix it. False reassurance is the true cruelty: a referee who waves through a broken design to spare someone's feelings has guaranteed that someone a harsher audience will break it later, when it costs more. Tough on the work *is* the kindness. Be precise, be specific, be relentless about the evidence — and never once make it about the person holding the pen. --- ## 5. A worked mini referee report — Devon on Maya's paper Maya's draft, the one whose identification memo you read in Chapter 7.5, asks whether state fair-lending examinations narrowed the county-level minority–white mortgage-denial gap, using a staggered difference-in-differences on HMDA data. Devon has it on his desk. Here is the core of his report — abridged, but in the real shape. > **Referee Report — "Do State Fair-Lending Examinations Narrow the Mortgage-Denial Gap?"** > > **Summary.** The paper asks whether state-level fair-lending examination programs reduced the gap in mortgage-denial rates between minority and white applicants. Using HMDA loan-application data aggregated to county-year, the author estimates a staggered difference-in-differences comparing counties in adopting states to counties in not-yet- and never-adopting states, with county and year fixed effects and standard errors clustered by state. The headline result (Table 2, column 3) is that examinations narrow the denial gap by about 1.4 percentage points, roughly a fifth of the pre-period mean gap, significant at the 1% level. The design and the question are both well-chosen, and the institutional setup is clearly explained. > > **Major points.** > > 1. *(Biggest concern — identification: anticipation and pre-trends.)* The entire causal claim rests on parallel trends, and I cannot currently assess it because the paper reports no event study. This is my central concern. Fair-lending examination programs are typically debated in a state legislature for a year or more before adoption, which is exactly the condition under which lenders might change behavior *in anticipation* — tightening or loosening before the official examination date — contaminating the pre-period. I would need an event-study plot of the denial gap with at least four pre-period leads and several post-period lags. *Flat, insignificant leads would substantially reassure me; a downward pre-trend in adopting states would be close to fatal,* because it would mean the gap was already closing before the policy and the design is mislabeling a trend as an effect. As written, I cannot tell which world we are in. > > 2. *(Identification: the forbidden comparison.)* The design is staggered, and the methods section says "two-way fixed effects" without addressing the negative-weights problem (Chapter 4.2): already-treated counties are serving as controls for later-treated ones, which can bias the estimate when the effect is dynamic — and an anticipation story makes dynamic effects likely. I would want the headline re-estimated with a heterogeneity-robust estimator (Callaway–Sant'Anna against clean controls only) and the two numbers compared. If they agree, the TWFE number is safe to report; if they diverge, the robust one is the result. > > 3. *(Specification: the denominator of the denial gap.)* The county-year denial gap is constructed from raw HMDA application counts, but the text does not say whether counties with very few minority applications are dropped or winsorized. A handful of small counties where the gap is mechanically volatile could be driving the estimate. A robustness column weighting by application volume, or dropping counties below a minimum application threshold, would tell us whether the result is real or a small-cell artifact. > > **Minor points.** > > - Table 2's note does not state the significance thresholds for the stars; per the convention, the note should be explicit (read-the-note, never-assume). > - The introduction claims the paper "establishes that examinations cause fairer lending," which is stronger than a credible-conditional-on-parallel-trends DiD can support; I would soften "establishes... cause" to match the design. > - Figure 1 has no y-axis label; I assume it is the denial gap in percentage points. > - The Bertrand–Duflo–Mullainathan (2004) citation is the right one for the clustering choice and should be cited where clustering is introduced, not only in the references. > > **Verdict.** Major revision. The question is good, the data are real, and the design is the right one; the paper is one event study and one robust-estimator column away from being convincing. None of my major points asks the author to change the project — only to defend the design it already has. Read what Devon did against Section 2. He summarized in his own words first (proving he got it). He led with the biggest concern and named it specifically — anticipation, not "endogeneity." He ordered the majors by danger and put exposition in the minors. For every major point he stated the evidence that would move him, and for the worst one he stated what would *reassure* and what would be *fatal*. He was tough — he flagged a potentially fatal pre-trend and a softened-this-claim overreach — and entirely kind, closing with a sentence that tells Maya her project is sound and the path is short. That last sentence is not flattery; it is an accurate verdict, and an accurate encouraging verdict is the most motivating thing a referee can write. --- ## 6. Responding to a referee — the R&R memo Now flip the desk. The editor has returned your paper with a "revise and resubmit" — the most common good outcome in empirical research, and a small victory, because it means the design was credible enough to be worth saving. Attached are the referee reports. Your job is to produce a revised paper *and* a document that matters at least as much: the **response memo**, also called the response-to-referees or the R&R letter. It is a point-by-point reply to every comment, and an editor reads it before re-reading the paper, because it tells them whether you are someone who can take criticism and act on it — which is, in the long run, more predictive of a good career than any single result. The structure is rigid and merciful in its rigidity. **For each comment, you do three things, in this order: quote the comment, state what you changed, and point to the evidence.** Quote it so the editor does not have to hold two documents side by side — paste the referee's words verbatim, usually in a blockquote or italics, then respond beneath. State what you changed in plain, specific language: "We have added an event study as new Figure 2 and discuss it in Section 4.2." Then point to the result: the new table number, the new figure, the page, the actual number that the new analysis produced. A response that says "we have addressed this concern" without saying *where* and *with what number* is worthless; the editor cannot verify it and will assume you did nothing. The deepest norm of the whole exercise, and the one the camp grades hardest, is this: **do exactly what you say you did.** If your memo claims you added an event study, the event study must be in the paper, it must be the one you describe, and its leads must say what your memo says they say. The fastest way to lose a referee's trust permanently — in the workshop and for the rest of your career — is a response memo whose claims the revised paper does not back up. Referees check. The whole institution of peer review runs on the assumption that the response memo is honest, and a single caught lie poisons every other claim in the document. Honesty here is not optional ethics; it is the load-bearing wall. --- ## 7. When to concede, and when to defend Not every referee comment is right, and a memo that caves to all of them is as weak as one that fights all of them. The skill is knowing which is which, and the rule is the same one that runs through this whole book: *follow the evidence, not the ego.* **Concede when the referee is right, and concede gracefully.** If they caught a real flaw, fix it, say so plainly, and thank them — "The referee is correct; we had clustered at the wrong level. We now cluster by state, the level at which examinations vary, and the t-statistic on the main effect falls from 4.1 to 2.6, which we report and discuss." Note what just happened: conceding made the result *weaker*, and reporting the weaker number honestly is exactly the move that earns trust. A referee who sees you report a t-stat that dropped from 4.1 to 2.6 because they were right believes the 2.6 far more than they would have believed the 4.1. Honest concession is not surrender; it is the strongest evidence you can offer that the surviving result is real. **Defend when the referee is wrong — respectfully, with evidence, never with attitude.** Sometimes a referee misreads the paper, asks for a test that does not apply, or proposes an alternative explanation you can actually rule out. You are allowed to disagree, and a good editor respects a well-argued rebuttal more than a reflexive capitulation. But the defense must be evidence, not assertion: "We respectfully disagree that small-cell volatility drives the result. As the referee suggests we have checked this directly: weighting by application volume (new column 4) leaves the estimate essentially unchanged at 1.3 pp, and dropping counties below 50 applications (column 5) gives 1.5 pp. We have added these columns and a sentence noting the result is not a small-cell artifact." That is a defense, and it is a *better* paper for the challenge: Maya disagreed, but she ran the test the referee implied, reported it, and her result is now armored against exactly the concern the referee raised. The cleanest defenses do the requested work and *then* explain why the worry does not bite. There is also the comment you can neither fully fix nor fully rebut — the untestable threat, like the exclusion restriction in an IV or the parallel-trends counterfactual itself. Here you neither concede the paper is broken nor pretend the concern is gone. You **bound** it: acknowledge the threat is real and untestable, present every piece of indirect evidence you have, and state the residual concern honestly in the paper's limitations — exactly the fourth column of the threats-and-responses table from Chapter 7.5, now promoted into the response memo. "The referee is correct that anticipation cannot be ruled out by the data alone. We present the event study (flat leads, Figure 2) as the best available evidence against it, and we now state in Section 6 that anticipation remains the limitation most likely to bias our estimate, and in which direction." Acknowledging a residual limitation is not weakness; a memo that pretends every threat is vanquished reads as naïve, and a referee trusts the author who says "here is what still worries me" far more than the one who claims a clean sweep. --- ## 8. A worked R&R response excerpt — Maya answers Devon Maya has revised. Here is the core of her response memo — the three major points, in the quote-change-evidence shape, showing one concession, one defense, and one bounded threat. > **Response to Referee 1** > > We thank the referee for an unusually constructive report. Below we respond to each point; referee comments are in italics, our responses follow. New material is flagged in the revised manuscript in blue. > > **Major point 1 (anticipation / pre-trends).** *"The entire causal claim rests on parallel trends, and I cannot currently assess it because the paper reports no event study... Flat, insignificant leads would substantially reassure me; a downward pre-trend would be close to fatal."* > > > This was the right thing to ask first, and we are glad we ran it. We have added an event study as **new Figure 2** and discuss it in **Section 4.2**. The four pre-period leads are individually insignificant and jointly insignificant (joint F-test p = 0.41), with no visible downward drift in the denial gap before adoption; the effect emerges in the year of examination and grows over the following two years. We read this as consistent with — though, as we now stress, not proof of — parallel trends and against an anticipation story. We have added one sentence to the limitations (Section 6) noting that flat leads cannot confirm the counterfactual, only fail to refute it. > > **Major point 2 (forbidden comparison / staggered DiD).** *"The design is staggered... already-treated counties are serving as controls... I would want the headline re-estimated with a heterogeneity-robust estimator and the two numbers compared."* > > > We agree this was a real gap. We have re-estimated the headline using the Callaway–Sant'Anna estimator against never- and not-yet-treated controls only (**new Table 3**). The robust estimate is **−1.5 pp**, very close to the **−1.4 pp** TWFE estimate, suggesting the negative-weights problem is not material here — consistent with effects that, while dynamic, are similarly signed across cohorts. We now report both numbers and note the agreement in **Section 5.1**. We retain TWFE as the headline for comparability but flag the robust estimate as the one we would defend if the two diverged. > > **Major point 3 (small-cell denominator).** *"A handful of small counties where the gap is mechanically volatile could be driving the estimate... a robustness column weighting by application volume, or dropping small counties, would tell us whether the result is real."* > > > We respectfully believe the result is not a small-cell artifact, and we have checked this directly as the referee suggests. **New columns 4 and 5 of Table 2** weight by application volume (estimate −1.3 pp) and drop counties below 50 minority applications per year (−1.5 pp). The estimate is stable across both. We have added a sentence in **Section 5.2** reporting this and concluding the result is not driven by mechanically volatile small counties. > > We believe the paper is substantially stronger for these three additions — particularly the event study, which we should have included in the first submission. We are grateful for the report. Look at what each response does. Point 1: a clean addition of requested evidence, with the actual joint-test p-value and an honest note that flat leads cannot confirm the counterfactual — she *bounded* the untestable threat rather than overclaiming it away. Point 2: a concession that the gap was real, the requested estimator run, and both numbers reported side by side — and crucially, the numbers agree, so the concession cost her nothing and bought her enormous credibility. Point 3: a respectful defense that *does the requested work first* and then states the conclusion, so the disagreement is settled by evidence rather than argument. Every response quotes, states the change, and points to a specific new table or figure with a specific number. And every change Maya claims is a change the revised paper actually contains — the norm of Section 6, honored. Devon, re-reading, can verify each one in under a minute, and his next report to the editor will say "the author addressed every point; recommend acceptance." --- ## 9. Why this is the same skill as Week 5, one more time It is worth closing the loop explicitly, because the symmetry is the point. In Week 5 you learned to read a paper tables-first and to ask, of any design, "what is the identifying assumption, and what threatens it." In Chapter 7.5 you learned to write that assumption down for your own paper and to list its threats in a four-column table. The referee report is that same threats table, written about someone else's paper. The R&R memo is your own threats table, promoted from a private worksheet into a public defense, with the referee's concerns slotted in beside the ones you anticipated. A referee reads tables-first and asks the identification question first because that is how the credibility of a result is actually decided; you respond table-by-table and evidence-by-evidence because that is how credibility is actually rebuilt. The student who internalizes this stops experiencing peer review as a hazing ritual and starts experiencing it as the engine it is. A good referee makes your paper better whether you agree with them or not — if they are right you fix a flaw, and if they are wrong you produce the evidence that proves they are wrong, which armors the paper either way. There is no losing move except the dishonest one. Reveal the trick to a peer so they can fix theirs; let a peer reveal the trick on yours so you can fix it before the grader does. That exchange, performed in good faith and on the evidence, is the whole of what a research community is. --- ## Your Turn **Workshop pointer.** This chapter is the engine of the **Week-8 peer-review workshop** and of **PS 8.4**, where you will be assigned a classmate's capstone draft, write a full referee report on it, and then — after receiving the report someone else wrote on *your* paper — produce an R&R memo responding point by point. Bring your draft (from 8.1–8.3) and come ready to be both referee and author in the same session. Mentor Session 8, *"Defending a result: what a referee actually asks,"* will rehearse the live version of this exchange — the questions a referee fires in a seminar — tied to Prof. Gao's work on disclosure (Elnahas, Gao, Hossain & Kim, 2024, *JFQA*). The revised paper and replication packet you defend in Chapter 8.5 should already reflect the changes your R&R memo promised: do what you said you did. When you write your report, work the skeleton in order: summary in your own words, biggest concern first, then identification, specification, exposition; major and minor labeled; for every major point, the evidence that would change your mind. When you write your memo, work each comment in the quote-change-evidence shape, and decide consciously for each whether you are conceding, defending, or bounding. **Reflection prompts.** 1. Pick one comment from a report you received that you initially wanted to argue against. Reconstruct, honestly, whether your instinct to defend was about the evidence or about your ego — and what the evidence actually says now that you have looked. 2. A referee asks for a test that, when you run it, *weakens* your headline result (a t-stat falls, a coefficient shrinks). Walk through why reporting the weaker number in your memo can make a reader believe your paper *more*, not less. What does this tell you about why honest concession is a strength? 3. Your single biggest concern about your *own* paper — the threat at the top of your Chapter 7.5 table — was it raised by your referee? If yes, did your response actually resolve it or only bound it? If no, why do you think a fresh reader did not see what you saw, and does that change how worried you should be?