nb8.4 — The Reproducibility Check: Can a Stranger Regenerate Your Number?

Here is the bar the whole camp has been walking toward, stated as bluntly as Chapter 8.5 states it: a result a stranger cannot reconstruct is not yet a result — it is a claim about a result. In Lab 7 you built the repository that holds an analysis reproducibly; in Lab 8 you assemble the one-click packet from Appendix D that lets a skeptic rebuild every table and figure with a single command. This notebook is the rehearsal of the test that packet has to pass. We are going to take a tiny end-to-end analysis — simulate raw data, build a dataset, estimate one coefficient, draw one figure — and then we are going to attack our own work the way a referee would.

The reveal, up front, so you have it before the machinery: reproducibility is not a vibe, it is a property you can assert with ==. “I think I’d get the same answer if I re-ran it” is a hope. “The coefficient is bit-identical and the content hash of the output table matches across two clean runs” is a test that passes or fails, and a failing one points at exactly the bug. So the plan of this notebook is: (1) write the analysis as plain functions driven by a single named seed; (2) run the whole pipeline twice and assert the result is identical — then watch an unseeded variant differ run-to-run, which is the bug that quietly irreproduces a real paper; (3) wrap it in a run_all() that writes every output to disk with a content-hash manifest (the “did anything change?” check); (4) describe the fresh-environment discipline — make clean && make all on a clean conda env — and simulate that check; (5) stamp the run with its Python, package, and seed provenance. Every piece traces to Chapter 8.5 §5 and the Lab 7 repository it finishes.

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")  # non-interactive backend: render to files, never pop a GUI window
import matplotlib.pyplot as plt

import hashlib          # content hashing -> the "did anything change?" check
import json             # the manifest and the provenance stamp are JSON
import sys, platform    # for the provenance stamp (python + OS)
import tempfile, shutil, os, importlib.metadata

# ----------------------------------------------------------------------------
# THE SEED. One named constant, set once, threaded through every stochastic step.
# This is the non-negotiable from Chapter 8.5 §5: a single fixed, *named* seed,
# recorded so a reviewer knows it was fixed BEFORE results, not tuned after.
# The value is the camp's conference date (2026-08-15); any fixed int is fine.
# ----------------------------------------------------------------------------
SEED = 20260815

print("pandas", pd.__version__, "| numpy", np.__version__, "| matplotlib", matplotlib.__version__)
print("SEED =", SEED)

1. The analysis, written as functions driven by `rng`

The first discipline of a reproducible analysis is that it is callable, not a pile of top-to-bottom cells you have to remember the order of. We write four small functions — simulate_raw, build, estimate, make_figure — and every one of them that touches randomness takes an rng argument. It never reaches for a hidden global np.random.*. This is the rule from Chapter 8.5 §5: pass rng into your functions rather than reaching for the global, so a reader can see exactly what randomness each step consumes.

The toy question is deliberately the kind you have met all camp. Sam wants to know whether a stock’s past month’s return predicts this month’s return — a one-factor momentum-ish regression r_t = \beta_0 + \beta_1 r_{t-1} + \varepsilon_t. We simulate a true \beta_1 = 0.15 so we know the right answer, build the lagged-return panel (no look-ahead — the regressor is strictly the prior month, the Chapter 7.4 lesson in miniature), estimate \hat\beta_1 by ordinary least squares with numpy, and draw the scatter with the fitted line. The “result” is the single number \hat\beta_1 plus a small results table; the figure is its picture.

def simulate_raw(rng: np.random.Generator, n_firms: int = 40, n_months: int = 36) -> pd.DataFrame:
    # Simulate a raw long panel of monthly returns with built-in 1-month persistence.
    # True data-generating process: r_t = 0.15 * r_{t-1} + noise. ALL randomness via rng.
    TRUE_BETA = 0.15
    frames = []
    for firm in range(n_firms):
        eps = rng.normal(0.0, 0.05, size=n_months)   # idiosyncratic shocks
        r = np.empty(n_months)
        r[0] = eps[0]
        for t in range(1, n_months):
            r[t] = TRUE_BETA * r[t - 1] + eps[t]      # AR(1) -> real momentum signal
        frames.append(pd.DataFrame({
            "firm": firm,
            "month": np.arange(n_months),
            "ret": r,
        }))
    return pd.concat(frames, ignore_index=True)


def build(raw: pd.DataFrame) -> pd.DataFrame:
    # Build the analysis frame: attach each firm's PRIOR-month return as the regressor.
    # The shift is within firm and strictly backward -> no look-ahead (Chapter 7.4).
    df = raw.sort_values(["firm", "month"]).copy()    # .copy(): no chained-assignment surprises
    df["ret_lag"] = df.groupby("firm")["ret"].shift(1)
    df = df.dropna(subset=["ret_lag"]).reset_index(drop=True)
    return df


def estimate(df: pd.DataFrame) -> dict:
    # OLS of ret on ret_lag via numpy.linalg.lstsq. Returns a tidy dict of results.
    y = df["ret"].to_numpy()
    X = np.column_stack([np.ones(len(df)), df["ret_lag"].to_numpy()])  # [intercept, slope]
    beta, *_ = np.linalg.lstsq(X, y, rcond=None)
    resid = y - X @ beta
    n, k = X.shape
    sigma2 = resid @ resid / (n - k)
    se = np.sqrt(np.diag(sigma2 * np.linalg.inv(X.T @ X)))
    return {
        "n_obs": int(n),
        "beta0": float(beta[0]),
        "beta1": float(beta[1]),
        "se_beta1": float(se[1]),
        "t_beta1": float(beta[1] / se[1]),
    }


def make_figure(df: pd.DataFrame, res: dict, path: str) -> str:
    # Scatter of ret vs ret_lag with the fitted line; save to path. Returns path.
    fig, ax = plt.subplots(figsize=(5, 3.5))
    ax.scatter(df["ret_lag"], df["ret"], s=6, alpha=0.3)
    xs = np.linspace(df["ret_lag"].min(), df["ret_lag"].max(), 50)
    ax.plot(xs, res["beta0"] + res["beta1"] * xs, color="C1", lw=2)
    ax.set_xlabel("prior-month return  $r_{t-1}$")
    ax.set_ylabel("this-month return  $r_t$")
    ax.set_title(fr"$\hat\beta_1 = {res['beta1']:.4f}$  (t = {res['t_beta1']:.2f})")
    fig.tight_layout()
    fig.savefig(path, dpi=80)
    plt.close(fig)   # close so a 40-cell rerun never leaks figure handles
    return path

2. One pass through the pipeline

Now wire the four functions together with a fresh rng seeded from our one named SEED, and look at the estimate. Because the true \beta_1 is 0.15, we expect \hat\beta_1 to land near there — but the exact digits are what we care about in this notebook, because those exact digits are what a reviewer will get if, and only if, the run is reproducible.

def pipeline(seed: int) -> tuple[pd.DataFrame, dict]:
    # The whole analysis, seeded from ONE named integer. Returns (analysis_df, results).
    rng = np.random.default_rng(seed)     # modern NumPy: a seeded Generator, not legacy np.random.seed
    raw = simulate_raw(rng)
    df = build(raw)
    res = estimate(df)
    return df, res

df1, res1 = pipeline(SEED)
print("estimate from a single seeded run:")
for k, v in res1.items():
    print(f"  {k:10s} = {v}")

3. Determinism: run it twice, assert bit-identical

Here is the test that turns “I think it’s reproducible” into a fact. We run the entire pipeline a second time with the same SEED and check two things, both with assert so a failure stops the notebook loudly:

The coefficient is bit-identical. Not “close” — identical, the same float down to the last bit. We use == on the raw float, because a seeded generator threaded through deterministic code must produce the same draws, hence the same regression, hence the same number. If 0.15-ish came out as 0.1500000001 the second time, something is consuming randomness you didn’t account for.
The content hash of the output table matches. A content hash is a short fixed-length fingerprint (here SHA-256) of the table’s bytes. Two tables with identical contents hash to the same string; change a single cell and the hash changes completely. Hashing the whole results table at once is how the packet checks “did anything in this output change?” without eyeballing every number. We canonicalize the dict to sorted-key JSON first so the hash depends on contents, not dict ordering.

def content_hash(obj) -> str:
    # SHA-256 of a canonical JSON encoding. Order-independent, contents-sensitive.
    payload = json.dumps(obj, sort_keys=True, separators=(",", ":")).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()

# Second pass, same seed.
df2, res2 = pipeline(SEED)

# (1) the coefficient is bit-identical
assert res1["beta1"] == res2["beta1"], "SEEDED run is not deterministic -- a bug!"

# (2) the content hash of the full results table matches
h1, h2 = content_hash(res1), content_hash(res2)
assert h1 == h2, "results-table hash differs across two seeded runs -- a bug!"

print("PASS  seeded run is reproducible.")
print(f"  beta1 run #1 == run #2 : {res1['beta1']} == {res2['beta1']}")
print(f"  results hash (both)    : {h1[:16]}...  (full 64 hex chars match)")

4. The bug: an unseeded variant differs run-to-run

Now we plant the exact failure that quietly irreproduces real papers. Suppose someone — maybe a past version of you — wrote simulate_raw to reach for the global np.random.default_rng() with no seed instead of taking an rng argument. Every call seeds itself from the operating system’s entropy, so every run draws different data and the regression lands somewhere new each time.

This is not a hypothetical. Chapter 8.5 §5 spells out the consequence: a reviewer runs your code and gets [1.2, 2.4] where your paper said [1.3, 2.3], and now they doubt everything, fairly. Below we run the unseeded version twice and assert that the two coefficients differ — demonstrating the disease, so that the cure (the seeded pipeline above) is unmistakable. Note the asymmetry: the seeded test asserts equality; this one asserts inequality. A reproducible pipeline is one where the first assert passes and this kind of code does not exist.

def simulate_raw_UNSEEDED(n_firms: int = 40, n_months: int = 36) -> pd.DataFrame:
    # THE BUG: no seed argument -> a fresh OS-seeded generator on every call.
    rng = np.random.default_rng()         # <-- no seed: different data every single run
    frames = []
    for firm in range(n_firms):
        eps = rng.normal(0.0, 0.05, size=n_months)
        r = np.empty(n_months); r[0] = eps[0]
        for t in range(1, n_months):
            r[t] = 0.15 * r[t - 1] + eps[t]
        frames.append(pd.DataFrame({"firm": firm, "month": np.arange(n_months), "ret": r}))
    return pd.concat(frames, ignore_index=True)

def pipeline_unseeded() -> dict:
    return estimate(build(simulate_raw_UNSEEDED()))

bad1 = pipeline_unseeded()
bad2 = pipeline_unseeded()

# The bug, made visible: two runs of the SAME code give DIFFERENT numbers.
assert bad1["beta1"] != bad2["beta1"], "unexpected: unseeded runs matched (astronomically unlikely)"

print("BUG DEMONSTRATED  unseeded pipeline is NOT reproducible:")
print(f"  beta1 run #1 = {bad1['beta1']:.6f}")
print(f"  beta1 run #2 = {bad2['beta1']:.6f}")
print(f"  difference   = {abs(bad1['beta1'] - bad2['beta1']):.6f}  (a reviewer would get neither of YOUR numbers)")

5. `run_all()`: one entry point, every output, a content-hash manifest

This is the one-click from Chapter 8.5 §5, in miniature. A single run_all() regenerates the result and writes every output to disk: the results table (results.json), the analysis dataset (analysis.csv), and the figure (figure.png). Then it writes a manifest — a JSON listing, for every output file, its content hash. The manifest is the answer to “did anything change?”: re-run run_all later, re-hash the outputs, and compare them to the manifest. If every hash matches, nothing drifted; if one differs, you know exactly which file changed before a reviewer does.

We hash the bytes on disk here (not the in-memory dict), because that is what actually ships in the packet — the file a stranger downloads. Everything goes to a temp directory, never the repo, so this notebook leaves no litter behind; in a real capstone these would be paper/tables/, paper/figures/, and a committed manifest.json. The seed is written into the manifest too, so the manifest records the conditions that produced it.

def hash_file(path: str) -> str:
    # SHA-256 of a file's raw bytes -> the fingerprint of the shipped artifact.
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()


def run_all(outdir: str, seed: int = SEED) -> dict:
    # The one-click entry point: regenerate the result, write every output to `outdir`,
    # and emit a content-hash manifest of all outputs. Returns the manifest dict.
    os.makedirs(outdir, exist_ok=True)
    df, res = pipeline(seed)

    results_path = os.path.join(outdir, "results.json")
    data_path    = os.path.join(outdir, "analysis.csv")
    fig_path     = os.path.join(outdir, "figure.png")

    with open(results_path, "w") as f:
        json.dump(res, f, sort_keys=True, indent=2)
    df.to_csv(data_path, index=False)
    make_figure(df, res, fig_path)

    outputs = {
        "results.json": hash_file(results_path),
        "analysis.csv": hash_file(data_path),
        "figure.png":   hash_file(fig_path),
    }
    manifest = {"seed": seed, "headline_beta1": res["beta1"], "outputs": outputs}
    with open(os.path.join(outdir, "manifest.json"), "w") as f:
        json.dump(manifest, f, sort_keys=True, indent=2)
    return manifest


WORKDIR = tempfile.mkdtemp(prefix="nb84_")     # scratch dir; cleaned at the end
run1_dir = os.path.join(WORKDIR, "run1")
manifest1 = run_all(run1_dir)

print("run_all() wrote outputs to:", run1_dir)
for name in sorted(os.listdir(run1_dir)):
    print("   ", name)
print("\nmanifest:")
print(json.dumps(manifest1, indent=2))

6. Fresh-environment discipline: `make clean && make all`

The manifest checks that your machine regenerates the same outputs. The harder, truer test is whether a different machine — a clean conda env on a reviewer’s laptop, a fresh git clone with no caches — does too. Chapter 8.5 §5 names the honesty test: make clean && make all. Delete every derived file and rebuild from nothing. If the paper comes back identical, the packet is real; if it doesn’t, you just found a hand-edit (or a missing seed, or an unpinned dependency) before a reviewer did.

In a real capstone this is literal: on a fresh clone you’d run

conda env create -f environment.lock.yml   # the exact stack from Lab 7, Step 3
conda activate capstone
make clean && make all                      # delete derived outputs, rebuild raw -> PDF

and then diff the rebuilt manifest.json against the committed one. We can’t spin up a conda env inside this notebook, but we can simulate the essential check: throw away the first run’s directory entirely, run run_all() again into a brand-new directory (a stand-in for the clean machine), and confirm the regenerated manifest’s hashes match the original. That is the from-scratch reproduction the discipline promises, asserted with ==.

# Simulate `make clean`: obliterate the first run's outputs entirely.
shutil.rmtree(run1_dir)
assert not os.path.exists(run1_dir), "clean failed -- derived outputs survived"
print("make clean  -> deleted", run1_dir)

# Simulate `make all` on a fresh machine: a NEW directory, nothing carried over.
run2_dir = os.path.join(WORKDIR, "run2_freshenv")
manifest2 = run_all(run2_dir)
print("make all    -> rebuilt into", run2_dir, "from scratch")

# The from-scratch reproduction check: every output hash matches the original manifest.
assert manifest2["outputs"] == manifest1["outputs"], \
    "REPRODUCTION FAILED: a from-scratch rebuild produced different bytes!"
assert manifest2["headline_beta1"] == manifest1["headline_beta1"], "headline number drifted!"

print("\nPASS  fresh-env rebuild reproduces the manifest byte-for-byte:")
for name, h in sorted(manifest2["outputs"].items()):
    print(f"   {name:14s} {h[:16]}...  matches")

6a. What a failed check looks like

The manifest is only useful if it actually catches drift. To prove it does, we tamper with one output the way a careless hand-edit would — append a stray byte to analysis.csv — re-hash it, and watch the comparison flag exactly that file while the others stay clean. This is the “you just found a hand-edit before a reviewer did” moment made concrete: the check does not merely say something changed, it says which file changed.

# A careless hand-edit: someone opens analysis.csv and "fixes" a number.
tampered = os.path.join(run2_dir, "analysis.csv")
with open(tampered, "a") as f:
    f.write("\n# stray edit a reviewer never approved\n")

# Re-verify every output against the committed manifest.
print("re-checking each output against the manifest:")
drift = []
for name, expected in sorted(manifest2["outputs"].items()):
    actual = hash_file(os.path.join(run2_dir, name))
    ok = (actual == expected)
    print(f"   {name:14s} {'OK' if ok else 'CHANGED <--'}")
    if not ok:
        drift.append(name)

assert drift == ["analysis.csv"], "drift detector failed to pinpoint the edited file"
print("\nManifest pinpointed the drifted file:", drift)

7. The provenance stamp: Python, packages, and seed

The last piece of a reproducible run is a record of the conditions that produced it: the Python version, the exact versions of the packages whose numerics could move a last decimal (Lab 7, Step 3’s whole point — “a numerical routine whose algorithm was improved so the last decimal moved”), the operating system, and the seed. Stamping this into the run is how a future reviewer — or future-you on a new laptop — knows what to reconstruct. It is the in-notebook companion to environment.lock.yml: that file pins the environment; this stamp records what was actually used, so a mismatch is visible.

The sentence you must be able to say out loud (Lab 7, Step 3): “my results were produced under this stack and this seed; rebuild them and you rebuild my numbers.”

def provenance_stamp(seed: int) -> dict:
    # Record python + package versions + seed + OS: the conditions of this run.
    def ver(pkg):
        try:
            return importlib.metadata.version(pkg)
        except importlib.metadata.PackageNotFoundError:
            return "not-installed"
    return {
        "seed": seed,
        "python": sys.version.split()[0],
        "platform": platform.platform(),
        "packages": {p: ver(p) for p in ("numpy", "pandas", "matplotlib", "scipy")},
    }

stamp = provenance_stamp(SEED)
# Ship it alongside the outputs, exactly like the manifest.
stamp_path = os.path.join(run2_dir, "provenance.json")
with open(stamp_path, "w") as f:
    json.dump(stamp, f, sort_keys=True, indent=2)

print("provenance stamp (written to provenance.json):")
print(json.dumps(stamp, indent=2))

8. Clean up the scratch directory

A reproducibility notebook that left a trail of temp files would be quietly hypocritical, so we delete the entire scratch tree. In a real capstone these artifacts are committed (under paper/tables/, paper/figures/, manifest.json) — here they were only ever scratch, and the discipline is to leave the working tree exactly as we found it.

shutil.rmtree(WORKDIR, ignore_errors=True)
assert not os.path.exists(WORKDIR), "scratch dir survived cleanup"
print("cleaned up scratch dir:", WORKDIR)
print("\nReproducibility check complete:")
print("  - seeded pipeline is bit-identical across runs (hash match)")
print("  - unseeded variant differs run-to-run (the bug)")
print("  - run_all() writes outputs + a content-hash manifest")
print("  - a from-scratch rebuild reproduces the manifest")
print("  - provenance (python/packages/seed) is stamped")

Your Turn

You have now run the test your Lab 8 packet must pass: a seeded analysis that regenerates bit-for-bit, a run_all() that writes a content-hash manifest, a simulated fresh-env rebuild, and a provenance stamp. Extend it toward your own capstone:

Bootstrap a confidence interval, seeded. Add a function that bootstraps a CI for \hat\beta_1 by resampling firms through the same rng. Run it twice with SEED and assert the interval endpoints are identical; then bootstrap without a seed and watch the interval wobble — the exact “[1.2, 2.4] vs [1.3, 2.3]” failure from Chapter 8.5 §5. A confidence interval is part of the result, so it must be reproducible too.
Add a real Makefile. Translate run_all() into the make data / analysis / paper / clean targets from Appendix D, and confirm make clean && make all regenerates your manifest.json identically. This is the literal one-click your packet ships.
Hash-pin your raw inputs. Extend the manifest to also hash the raw simulated panel, not just the outputs. Now the manifest answers a sharper question: did the inputs change, or only the code? That distinction is what tells a reviewer whether a re-run drift is a data problem or a code problem.
Break it on purpose, then fix it. Introduce one subtle irreproducibility — a set() whose iteration order leaks into a result, or a dict you hash without sort_keys — and use the manifest to catch it. The lesson of this whole notebook: a rule that is verified beats a rule that is merely written.

1. The analysis, written as functions driven by rng