nb6.2 — 10-K Text Vectorization & Similarity (Hoberg–Phillips, TNIC-style)

In Chapter 1.2 we said something that sounded almost too cute to be useful: correlation is the cosine of an angle between two vectors. Two assets that move together point in nearly the same direction; two that move oppositely point apart. The whole idea of a covariance matrix was just bookkeeping for those angles.

This notebook cashes that idea in for words. Hoberg and Phillips (2016) take the product-description section that every U.S. public firm is legally required to write in its 10-K, turn each description into a vector of numbers, and measure how similar two firms are with the exact same cosine Maya used for returns. The numerator is large when two firms emphasize the same words; the denominators normalize away description length. Two software firms point the same way; a software firm and a bank point apart — and the code never had to be told what an industry is. It rediscovers industries from text alone.

That is the trick. Reveal it early, then make it mechanical:

  1. Write a small self-contained corpus of synthetic “10-K product descriptions” across a few latent industries (software, banking, retail) plus a couple of deliberate cross-over firms — so the whole notebook runs offline.
  2. Build bag-of-words and TF–IDF vectors with scikit-learn, compute the pairwise cosine-similarity matrix, and draw it as a heatmap. Watch the industry blocks light up on the diagonal.
  3. Construct a TNIC-style firm-specific industry: each firm’s “industry” is the set of firms above a similarity threshold. Show it is firm-specific, intransitive, and changes as you move the threshold.
  4. Contrast with a fixed, hand-coded “SIC-like” classification — and see TNIC place the cross-over firms more sensibly.
  5. Tie the cosine back to Chapter 1.2 (correlation as cosine) and forward to embeddings / LLMs (Chapter 6.5).

Leah, our text-analysis student, has been waiting for this one. Sam will recognize the heatmap — it is the return-correlation heatmap from Week 1, pointed at prose.

Citation. Hoberg, G., & Phillips, G. (2016). Text-Based Network Industries and Endogenous Product Differentiation. Journal of Political Economy, 124(5), 1423–1465.

import numpy as np
import pandas as pd
import matplotlib
matplotlib.use("Agg")  # non-interactive backend: render to figures, never pop a window
import matplotlib.pyplot as plt

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

rng = np.random.default_rng(42)  # seed any randomness for reproducibility
np.set_printoptions(precision=3, suppress=True)
pd.set_option("display.width", 120)
print("Setup complete. sklearn TfidfVectorizer / CountVectorizer / cosine_similarity loaded.")

1. A tiny corpus of synthetic 10-K product descriptions

The real TNIC pipeline parses Item 1 (“Business”) out of tens of thousands of EDGAR filings — a months-long, unglamorous extraction job. We do the idea in miniature: 16 hand-written paragraphs, each a caricature of a real firm’s product-description prose, across three latent industries:

  • a software / cloud cluster (SoftA–SoftD),
  • a bank / lending cluster (BankA–BankD),
  • a retail / apparel cluster (RetA–RetD).

Then two firms we plant on purpose to break clean boxes:

  • OmniMart — an Amazon-like firm that sells retail goods and runs a cloud platform. It should neighbor both the retail and software clusters.
  • FinShop — a buy-now-pay-later (BNPL) firm that talks like a bank and a retailer at once.

These are the “cross-over firms” SIC codes handle badly — each gets forced into one box. We will watch text similarity refuse to force them. Note these paragraphs are written to share vocabulary within a cluster (the software firms all say “cloud,” “software,” “platform,” “subscription”) and to avoid sharing it across clusters — that is the only thing that makes the trick work, and it is exactly what real product descriptions do.

# Synthetic "10-K Item 1" product descriptions. Written in the notebook so it runs offline.
# Vocabulary deliberately clusters within an industry; cross-over firms borrow from two.
descriptions = {
    # ---- software / cloud cluster ----
    "SoftA": ("We develop cloud software and subscription platform services for enterprise "
              "customers. Our applications run on a scalable cloud platform with data analytics "
              "and developer tools delivered as software."),
    "SoftB": ("Our company provides software as a service and cloud computing platforms. "
              "Subscription revenue comes from enterprise applications, data analytics dashboards, "
              "and developer software tools hosted in the cloud."),
    "SoftC": ("We offer enterprise cloud platform software with subscription licensing. The "
              "platform supports application development, data analytics, and integration tools "
              "for software developers and corporate customers."),
    "SoftD": ("Our cloud-based software platform delivers subscription analytics and application "
              "services. We sell developer tools and enterprise software hosted on our scalable "
              "cloud computing infrastructure."),
    # ---- bank / lending cluster ----
    "BankA": ("We are a bank holding company offering deposit accounts, consumer loans, mortgage "
              "lending, and credit products. Our banking services include savings deposits, "
              "interest-bearing accounts, and commercial lending to borrowers."),
    "BankB": ("Our bank provides lending and deposit services, including mortgage loans, credit "
              "cards, and savings accounts. We earn interest income from consumer and commercial "
              "loans and manage credit risk on our lending portfolio."),
    "BankC": ("We offer banking and lending products: deposit accounts, personal loans, mortgage "
              "credit, and interest-bearing savings. Our lending business serves consumer borrowers "
              "and small commercial banking clients."),
    "BankD": ("As a regional bank we provide consumer and commercial lending, deposit accounts, "
              "mortgage origination, and credit services. Interest income on loans and deposits "
              "drives our banking revenue."),
    # ---- retail / apparel cluster ----
    "RetA": ("We are a retailer of apparel and consumer merchandise sold through retail stores "
             "and our online shopping website. Our merchandise includes clothing, footwear, and "
             "seasonal consumer goods for shoppers."),
    "RetB": ("Our company operates retail stores selling apparel, footwear, and household "
             "merchandise. Customers shop in stores and online for clothing and consumer goods at "
             "discount retail prices."),
    "RetC": ("We sell apparel, accessories, and consumer merchandise through a chain of retail "
             "stores and e-commerce shopping. Our clothing and footwear products target value "
             "retail shoppers."),
    "RetD": ("As a specialty retailer we offer clothing, footwear, and consumer merchandise in "
             "retail stores and through online shopping. Seasonal apparel and accessories make up "
             "our retail product assortment."),
    # ---- deliberate cross-over firms ----
    "OmniMart": ("We are a retailer selling apparel and consumer merchandise through online "
                 "shopping and retail stores, and we also operate a cloud computing platform "
                 "providing software, data analytics, and developer subscription services to "
                 "enterprise customers."),
    "FinShop": ("Our buy-now-pay-later platform offers consumer credit and lending at the point of "
                "retail shopping. We provide installment loans for apparel and merchandise "
                "purchases, combining banking-style credit with online retail checkout."),
    # ---- one more pure-ish software firm to round out the corpus ----
    "SoftE": ("We provide a subscription cloud software platform for data analytics. Enterprise "
              "customers use our application and developer tools, and we recognize recurring "
              "subscription revenue from the software."),
    # ---- one more pure-ish bank to round out the corpus ----
    "BankE": ("We operate a savings bank offering deposit accounts, mortgage lending, and consumer "
              "credit. Interest income from loans and deposits and our credit underwriting are the "
              "core of our banking model."),
}

firms = list(descriptions.keys())
docs = list(descriptions.values())
print(f"Corpus: {len(firms)} firms")
for f in firms:
    print(f"  {f:9s} ({len(descriptions[f].split())} words)")

2. A hand-coded “SIC-like” classification (the thing TNIC competes against)

Before we let the text speak, let us write down the official-style answer: a fixed, top-down code that drops each firm into exactly one box. This is the SIC/NAICS convention in miniature — transitive (everyone in a box is declared a mutual competitor) and forced (one box per firm, no overlap). We will use it as the foil in Section 6.

Notice the problem already: where does OmniMart go? It is half retailer, half cloud platform. Where does FinShop go? Half lender, half store. A single code must pick one and discard the other half of what the firm actually does. We make the arbitrary choice a human classifier would — and that arbitrariness is the whole point.

# Fixed, hand-coded "SIC-like" industry: one box per firm, no overlap (transitive).
sic_like = {
    "SoftA": "Software", "SoftB": "Software", "SoftC": "Software",
    "SoftD": "Software", "SoftE": "Software",
    "BankA": "Banking", "BankB": "Banking", "BankC": "Banking",
    "BankD": "Banking", "BankE": "Banking",
    "RetA": "Retail", "RetB": "Retail", "RetC": "Retail", "RetD": "Retail",
    # The cross-over firms: a single code MUST pick one box and discard the rest.
    "OmniMart": "Retail",    # filed as a retailer -- its cloud business is invisible here
    "FinShop": "Banking",    # filed as a lender -- its retail-checkout business is invisible
}
sic_df = pd.DataFrame({"firm": firms, "sic_like": [sic_like[f] for f in firms]})
print(sic_df.to_string(index=False))
print("\nNote OmniMart -> Retail and FinShop -> Banking: each loses half its identity to the box.")

3. Step one — turn each description into a bag-of-words vector

The first thing we do to a document is forget that it is a document. We throw away word order and keep only which words appear and how often — a bag of words. Build a dictionary of every non-trivial word across all 16 descriptions; firm i’s vector \mathbf{w}_i has one entry per dictionary word, equal to the count of that word in firm i’s description.

scikit-learn’s CountVectorizer does this in one line. We pass stop_words="english" to drop ubiquitous filler (“the”, “and”, “of”, “our”, “we”) that carries no industry signal — the same stop-word pruning the real pipeline does. The result is a sparse 16 \times V matrix, where V is the vocabulary size: 16 firms, V words.

# Bag-of-words: count of each (non-stopword) term per firm.
count_vec = CountVectorizer(stop_words="english")
X_counts = count_vec.fit_transform(docs)        # sparse 16 x V matrix of raw counts
vocab = count_vec.get_feature_names_out()
print(f"Vocabulary size V = {len(vocab)} distinct words")
print(f"Bag-of-words matrix shape: {X_counts.shape}  (firms x words)")

# Peek: the 12 most common words across the whole corpus (these are the LEAST informative).
total_counts = np.asarray(X_counts.sum(axis=0)).ravel()
top = np.argsort(total_counts)[::-1][:12]
print("\nMost frequent words (carry little industry signal):")
print("  " + ", ".join(f"{vocab[j]}({total_counts[j]})" for j in top))

4. Step two — reweight with TF–IDF so distinctive words count more

Look at that top-words list: words like “cloud,” “loans,” or “retail” are common because whole clusters share them — useful — but words like “services,” “products,” or “customers” are common because everyone uses them, and they tell you nothing about which firm is which. Raw counts let the boring shared words pull every firm toward every other firm.

TF–IDF — term frequency × inverse document frequency — fixes this. It multiplies a word’s within-document frequency (TF) by a weight that grows when the word appears in fewer documents:

\text{idf}_j = \log\!\frac{D}{d_j},

with D the number of documents and d_j the number containing word j. A word in every document has d_j = D, so \text{idf}_j = \log 1 = 0 — silenced. A word in one description out of sixteen gets a large weight. Firm i’s entry for word j becomes \text{tf}_{ij}\cdot\text{idf}_j: distinctive words amplified, ubiquitous words muted.

TfidfVectorizer does TF and IDF together and, by default, L2-normalizes each row to unit length — which is convenient, because once vectors are unit length their cosine similarity is just their dot product. (scikit-learn uses a smoothed IDF, \log\frac{1+D}{1+d_j}+1, to avoid division by zero; same idea, gentler.)

# TF-IDF: reweight counts so distinctive words dominate, ubiquitous words fade.
tfidf_vec = TfidfVectorizer(stop_words="english")   # default: L2-normalized rows, smoothed idf
X_tfidf = tfidf_vec.fit_transform(docs)             # sparse 16 x V, unit-length rows
assert (tfidf_vec.get_feature_names_out() == vocab).all()  # same vocabulary as the count version

# Inspect the idf weights: high idf = rare/distinctive, low idf = common/boring.
idf = tfidf_vec.idf_
order = np.argsort(idf)
print("Lowest idf (most common, least informative words):")
print("  " + ", ".join(f"{vocab[j]}({idf[j]:.2f})" for j in order[:8]))
print("Highest idf (rarest, most distinctive words):")
print("  " + ", ".join(f"{vocab[j]}({idf[j]:.2f})" for j in order[::-1][:8]))

# Confirm rows are unit length (so cosine == dot product later).
row_norms = np.sqrt(np.asarray(X_tfidf.multiply(X_tfidf).sum(axis=1)).ravel())
print(f"\nTF-IDF row norms all ~1.0?  min={row_norms.min():.4f}  max={row_norms.max():.4f}")

5. Step three — the cosine similarity matrix (Chapter 1.2, pointed at words)

Here is the payoff and the bridge back to Week 1. For two firm vectors \mathbf{w}_i and \mathbf{w}_k, the cosine similarity is

\text{sim}(i,k) = \cos\theta = \frac{\mathbf{w}_i \cdot \mathbf{w}_k}{\lVert \mathbf{w}_i\rVert\,\lVert \mathbf{w}_k\rVert} = \frac{\sum_j w_{ij}\,w_{kj}}{\sqrt{\sum_j w_{ij}^2}\,\sqrt{\sum_j w_{kj}^2}}.

This is exactly the correlation-as-a-cosine formula from Chapter 1.2 — only the vectors hold word weights instead of centered returns. It runs from 0 (no shared vocabulary: orthogonal, \theta = 90^\circ) to 1 (identical direction, \theta = 0^\circ). Two software firms score near 1; a software firm and a bank near 0.

cosine_similarity from scikit-learn computes it for every pair at once, returning a symmetric 16 \times 16 matrix with ones on the diagonal (every firm is identical to itself). Because our TF–IDF rows are already unit length, this is just X X^\top — but we let sklearn do it so the normalization is explicit and correct even for the (un-normalized) count vectors.

# Pairwise cosine similarity for every firm pair -- the full TNIC similarity matrix.
S_tfidf = cosine_similarity(X_tfidf)     # 16 x 16, symmetric, 1.0 on diagonal
S_counts = cosine_similarity(X_counts)   # same for raw bag-of-words (for comparison in Sec 7)

sim_tfidf = pd.DataFrame(S_tfidf, index=firms, columns=firms)
print("TF-IDF cosine similarity matrix (rounded):")
print(sim_tfidf.round(2).to_string())

# Confirm the structure we expect: 1.0 on the diagonal, symmetric, all in [0, 1].
assert np.allclose(np.diag(S_tfidf), 1.0)
assert np.allclose(S_tfidf, S_tfidf.T)
assert S_tfidf.min() >= -1e-9 and S_tfidf.max() <= 1 + 1e-9
print("\nMatrix is symmetric, unit-diagonal, entries in [0, 1].")

6. See it: the heatmap, with the industry blocks lighting up

A similarity matrix is hard to read as numbers and easy to read as a picture. We order the firms so that the three pure clusters sit together, with the two cross-over firms (OmniMart, FinShop) at the end, then draw the matrix as a heatmap — hot where two firms share vocabulary, cool where they don’t.

Watch for two things. First, three bright blocks on the diagonal: the software firms are near-twins of each other, likewise the banks, likewise the retailers — and the code was never told these groups exist. That is Exercise 1 of the chapter, reproduced in your own hands. Second, the two cross-over firms light up against two blocks at once — the first hint of the intransitivity that breaks SIC.

# Order firms by cluster so the block structure is visible; cross-overs last.
order_firms = ["SoftA", "SoftB", "SoftC", "SoftD", "SoftE",
               "BankA", "BankB", "BankC", "BankD", "BankE",
               "RetA", "RetB", "RetC", "RetD",
               "OmniMart", "FinShop"]
idx = [firms.index(f) for f in order_firms]
S_ord = S_tfidf[np.ix_(idx, idx)]

fig, ax = plt.subplots(figsize=(8, 7))
im = ax.imshow(S_ord, cmap="magma", vmin=0, vmax=1)
ax.set_xticks(range(len(order_firms)), order_firms, rotation=90, fontsize=8)
ax.set_yticks(range(len(order_firms)), order_firms, fontsize=8)
for i in range(len(order_firms)):
    for j in range(len(order_firms)):
        v = S_ord[i, j]
        ax.text(j, i, f"{v:.1f}", ha="center", va="center",
                color="white" if v < 0.55 else "black", fontsize=6)
fig.colorbar(im, ax=ax, label="cosine similarity (TF-IDF)")
ax.set_title("10-K text similarity: three industry blocks emerge from words alone")
fig.tight_layout()
fig.savefig("nb62_similarity_heatmap.png", dpi=110)
plt.close(fig)
print("Heatmap saved to nb62_similarity_heatmap.png")
print("Three bright diagonal blocks = software / banking / retail, discovered without labels.")

7. The headline check: within-cluster similarity beats cross-cluster

The heatmap is suggestive; let us make it a number. Using the three pure clusters (ignore the two cross-over firms for this test), compute the average cosine similarity for pairs within the same cluster versus pairs in different clusters. The claim of the whole enterprise is that within \gg cross. If that fails, the measure is worthless.

We also re-run it on the raw bag-of-words matrix to make the chapter’s knob-stress point. Be careful how you compare the two: raw counts inflate all the magnitudes, so the raw difference within-cross can look larger for counts even though counts are worse. The honest measures are (a) the cross-cluster average — IDF should push the spurious cross-industry similarity down, because it silences the ubiquitous words (“services,” “products,” “customers”) that made unrelated firms look alike — and (b) the separation ratio within/cross, which should be higher for TF–IDF. Those are what we assert.

def within_vs_cross(S, firm_list, label_map, clusters=("Software", "Banking", "Retail")):
    """Mean off-diagonal similarity for within-cluster vs cross-cluster pairs."""
    within, cross = [], []
    n = len(firm_list)
    for a in range(n):
        for b in range(a + 1, n):           # off-diagonal pairs only
            la, lb = label_map[firm_list[a]], label_map[firm_list[b]]
            if la not in clusters or lb not in clusters:
                continue                     # skip cross-over firms for this clean test
            (within if la == lb else cross).append(S[a, b])
    return float(np.mean(within)), float(np.mean(cross))

w_tfidf, c_tfidf = within_vs_cross(S_tfidf, firms, sic_like)
w_count, c_count = within_vs_cross(S_counts, firms, sic_like)

print("Average cosine similarity (pure-cluster firms only):")
print(f"  TF-IDF : within={w_tfidf:.3f}   cross={c_tfidf:.3f}   "
      f"separation ratio within/cross={w_tfidf / c_tfidf:.2f}")
print(f"  Counts : within={w_count:.3f}   cross={c_count:.3f}   "
      f"separation ratio within/cross={w_count / c_count:.2f}")

# (1) The core claim: within-cluster similarity far exceeds cross-cluster.
assert w_tfidf > c_tfidf, "within-cluster similarity must exceed cross-cluster"
# (2) IDF pushes SPURIOUS cross-industry similarity DOWN vs raw counts...
assert c_tfidf < c_count, "TF-IDF should lower spurious cross-cluster similarity vs counts"
# (3) ...and raises the within/cross SEPARATION ratio (the honest comparison; the raw
#     difference is misleading because counts inflate all magnitudes at once).
assert (w_tfidf / c_tfidf) > (w_count / c_count), "TF-IDF should improve the separation ratio"
print("\nWithin-cluster >> cross-cluster. TF-IDF lowers cross-cluster contamination and")
print("raises the within/cross separation ratio -- IDF sharpens the industry boundaries.")

8. Step four — build a TNIC-style firm-specific industry

Now the move that earns the word “network.” A SIC code partitions firms into boxes: pick a firm, read off its box, and its competitors are everyone else in that box — the same set for all of them. TNIC does something different. Firm i’s industry is the set of firms whose similarity to i clears a threshold \tau. Every firm centers its own neighborhood. There is no global partition.

This object has three properties SIC cannot represent, all of which you can verify on our data:

  • Firm-specific. OmniMart’s neighbor set is not identical to any single cluster’s — it is its own list.
  • Intransitive. A can neighbor B and B neighbor C without A neighboring C (the Amazon-to-Microsoft-and-Walmart case).
  • Threshold-dependent. Raise \tau and neighborhoods shrink; lower it and they balloon. The “industry” is only as fixed as the knob you chose — exactly the chapter’s “knobs are levers” lesson.

Let us write the neighborhood function and read off a few firms’ industries at a moderate threshold.

def tnic_industry(firm, S, firm_list, tau):
    """TNIC neighborhood of `firm`: all OTHER firms with cosine similarity >= tau."""
    i = firm_list.index(firm)
    return [firm_list[j] for j in range(len(firm_list))
            if j != i and S[i, j] >= tau]

tau = 0.15
print(f"TNIC industries at threshold tau = {tau}\n")
for f in ["SoftA", "BankA", "RetA", "OmniMart", "FinShop"]:
    nbrs = tnic_industry(f, S_tfidf, firms, tau)
    print(f"  {f:9s} -> {nbrs}")

print("\nEach firm has its OWN competitor list -- no shared global box.")
print("OmniMart and FinShop reach into TWO clusters at once (cross-over firms).")

9. Firm-specific and intransitive — the two properties SIC cannot fake

Two checks make the abstract properties concrete.

Firm-specific. If TNIC were secretly a partition in disguise, every member of a cluster would have the identical neighbor set. It does not: even within the software cluster, SoftA and SoftB have slightly different neighbor lists, because each measures distance from its own position. We confirm at least one pair of same-cluster firms whose TNIC industries differ.

Intransitive. This is the headline property and the Amazon case. We hunt for a concrete triple (A, B, C) where \text{sim}(A,B) \ge \tau and \text{sim}(B,C) \ge \tau but \text{sim}(A,C) < \tau — a chain of neighbors that does not close into a triangle. Our planted cross-over firms make this happen: OmniMart neighbors both a software firm and a retail firm, but those two are strangers to each other.

# (a) Firm-specific: same-cluster firms have DIFFERENT neighbor sets.
na = set(tnic_industry("SoftA", S_tfidf, firms, tau))
nb = set(tnic_industry("SoftB", S_tfidf, firms, tau))
print("SoftA industry:", sorted(na))
print("SoftB industry:", sorted(nb))
print("Identical neighbor sets?", na == nb, "-> firm-specific, not a shared box.\n")

# (b) Intransitive: find a triple A-B-C with sim(A,B), sim(B,C) >= tau but sim(A,C) < tau.
def find_intransitive_triple(S, firm_list, tau):
    n = len(firm_list)
    for a in range(n):
        for b in range(n):
            if b == a or S[a, b] < tau:
                continue
            for c in range(n):
                if c in (a, b) or S[b, c] < tau:
                    continue
                if S[a, c] < tau:
                    return firm_list[a], firm_list[b], firm_list[c]
    return None

triple = find_intransitive_triple(S_tfidf, firms, tau)
A, B, C = triple
ia, ib, ic = firms.index(A), firms.index(B), firms.index(C)
print(f"Intransitive triple at tau={tau}:  {A} -- {B} -- {C}")
print(f"  sim({A},{B}) = {S_tfidf[ia, ib]:.3f}  >= tau")
print(f"  sim({B},{C}) = {S_tfidf[ib, ic]:.3f}  >= tau")
print(f"  sim({A},{C}) = {S_tfidf[ia, ic]:.3f}  <  tau   <-- chain does NOT close")
assert S_tfidf[ia, ib] >= tau and S_tfidf[ib, ic] >= tau and S_tfidf[ia, ic] < tau
print(f"\n{B} bridges {A} and {C}, but {A} and {C} are not neighbors -- SIC cannot represent this.")

10. The threshold is a knob: sweep it and watch industries grow and shrink

The chapter’s Exercise 2(c): move the similarity threshold and watch a firm’s competitor count balloon or collapse. At a high \tau, only near-twins count as competitors and most firms are loners; at a low \tau, the faintest shared word makes everyone a neighbor and the network melts into one blob. The “right” threshold is a judgment call — and the measure’s claims are only as stable as that call. We sweep \tau and plot each example firm’s TNIC industry size.

taus = np.linspace(0.02, 0.50, 25)
watch = ["SoftA", "BankA", "RetA", "OmniMart", "FinShop"]
sizes = {f: [len(tnic_industry(f, S_tfidf, firms, t)) for t in taus] for f in watch}

fig, ax = plt.subplots(figsize=(7.5, 4.5))
for f in watch:
    ax.plot(taus, sizes[f], marker="o", ms=3, label=f)
ax.axvline(0.15, ls="--", color="gray", lw=1, label="tau used above")
ax.set_xlabel("similarity threshold  tau")
ax.set_ylabel("number of TNIC competitors")
ax.set_title("The threshold is a lever: industry size depends on the knob you pick")
ax.legend(fontsize=8)
fig.tight_layout()
fig.savefig("nb62_threshold_sweep.png", dpi=110)
plt.close(fig)

print("TNIC industry size at a few thresholds:")
print(f"{'firm':9s}" + "".join(f"  tau={t:.2f}" for t in (0.05, 0.15, 0.30, 0.45)))
for f in watch:
    row = [len(tnic_industry(f, S_tfidf, firms, t)) for t in (0.05, 0.15, 0.30, 0.45)]
    print(f"{f:9s}" + "".join(f"{v:9d}" for v in row))
# Monotonic: a higher threshold can only shrink (never grow) a neighborhood.
for f in watch:
    s = sizes[f]
    assert all(s[k] >= s[k + 1] for k in range(len(s) - 1)), "industry size must be non-increasing in tau"
print("\nRaising tau can only shrink a neighborhood -- the knob directly sets who counts as a rival.")

11. Step five — TNIC vs the SIC-like box on the cross-over firms

Now the horse race in miniature. Take the two cross-over firms and ask each classification: who are this firm’s competitors?

  • The SIC-like code gives one box: OmniMart is “Retail,” full stop — its entire cloud business is invisible, and it is declared not a competitor of any software firm. FinShop is “Banking” — its retail-checkout half vanishes.
  • TNIC reads the words and refuses to discard half the firm: OmniMart neighbors both retail and software firms; FinShop neighbors both banks and retailers.

This is the chapter’s claim made tactile: when a firm genuinely straddles two businesses, the fixed box must lie about one of them, while the text-based neighborhood reports both. We print the contrast side by side.

def cluster_of(firm):
    return sic_like[firm]

for f in ["OmniMart", "FinShop"]:
    box = sic_like[f]
    sic_peers = [g for g in firms if g != f and sic_like[g] == box]
    tnic_peers = tnic_industry(f, S_tfidf, firms, tau)
    tnic_clusters = sorted(set(cluster_of(g) for g in tnic_peers))
    print(f"=== {f} ===")
    print(f"  SIC-like box : {box}  (declares competitors only within '{box}')")
    print(f"  SIC peers    : {sic_peers}")
    print(f"  TNIC peers   : {tnic_peers}")
    print(f"  TNIC spans clusters: {tnic_clusters}")
    print()

# Verify the headline: TNIC reaches into MORE than one industry for each cross-over firm.
for f in ["OmniMart", "FinShop"]:
    tnic_peers = tnic_industry(f, S_tfidf, firms, tau)
    spanned = set(cluster_of(g) for g in tnic_peers)
    assert len(spanned) >= 2, f"{f} should neighbor more than one SIC-like industry"
print("Both cross-over firms span >= 2 industries under TNIC -- the box could only show one.")

12. Where this connects: back to Chapter 1.2, forward to embeddings (Ch 6.5)

Step back and notice you used one operation the whole way down. In Chapter 1.2 the cosine measured the angle between two centered return vectors and we called it correlation. Here the cosine measured the angle between two word-count vectors and we called it product similarity. The formula never changed:

\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}.

The cosine does not care whether its vectors hold returns, word counts, or — next — meanings. That last step is Chapter 6.5. The bag-of-words vector has a famous ceiling: it sees vocabulary overlap, not meaning. “We do not compete in semiconductors” and “we compete in semiconductors” produce nearly identical bags; “automobile” and “car” are treated as unrelated strangers. A modern embedding model fixes exactly this — it maps each passage to a few-hundred-dimensional vector where texts that mean similar things land near each other even with no shared words, and you compare them with the same cosine. The embedding is the grown-up cousin of the bag-of-words vector you just built; Retrieval-Augmented Generation (Ch 6.5) is nearest-neighbor search under that same cosine. You have already met the metric; Ch 6.5 only upgrades the vectors.

The cell below makes the bag-of-words ceiling concrete — and explains why it motivates the next chapter.

# The bag-of-words ceiling: negation and synonyms are invisible to word-overlap.
pair = {
    "claims_yes": "We compete directly in the semiconductor and chip fabrication market.",
    "claims_no":  "We do not compete in the semiconductor or chip fabrication market.",
}
demo_vec = TfidfVectorizer(stop_words="english")
Xd = demo_vec.fit_transform(list(pair.values()))
sim_neg = cosine_similarity(Xd)[0, 1]
print(f'cosine("{list(pair)[0]}", "{list(pair)[1]}") = {sim_neg:.3f}')
print("Opposite claims -- yet near-identical bags, because 'not' is a dropped stop-word.")
print("Bag-of-words sees the shared 'semiconductor/chip/fabrication', not the negation.")
print()
print("This is the ceiling Hoberg-Phillips note and Chapter 6.5 (embeddings/LLMs) breaks:")
print("an embedding model would place these two sentences FAR apart because they MEAN")
print("opposite things -- same cosine metric, smarter vectors.")

13. The real path (not executed here): EDGAR + the public TNIC library

Everything above ran on 16 paragraphs we typed by hand so the notebook works offline. The real pipeline replaces our descriptions dict with thousands of actual 10-K Item 1 sections pulled from SEC EDGAR — then runs the identical sklearn vectorize-and-cosine machinery. And for most research you do not even re-parse EDGAR: Hoberg and Phillips publish their computed industries as a free public data library you merge in, the way you use Fama–French factors without re-sorting portfolios.

The cell below shows the real shape of both. It is intentionally not executed — it needs network access, a declared SEC user-agent (EDGAR requires it), and packages you may not have installed. Treat it as a map, not a command to run.

# === REAL PATH (DO NOT RUN HERE) — needs network, an SEC user-agent, and extra packages. ===
#
# --- Option A: pull a real 10-K Item 1 from SEC EDGAR ---------------------------------------
# SEC requires a descriptive User-Agent ("Name email") on every request; be polite (<=10 req/s).
#
#   # pip install edgartools
#   from edgar import set_identity, Company
#   set_identity("Your Name your.email@example.com")        # SEC fair-access requirement
#   filing = Company("AAPL").get_filings(form="10-K").latest(1)
#   text = filing.obj().business            # Item 1 "Business" section as plain text
#
#   # ...or with plain requests against the EDGAR REST API:
#   # import requests
#   # headers = {"User-Agent": "Your Name your.email@example.com"}
#   # subs = requests.get("https://data.sec.gov/submissions/CIK0000320193.json",
#   #                     headers=headers, timeout=30).json()
#   # then fetch the filing document and parse Item 1 out of the HTML/text.
#
# Once you have a {ticker: item1_text} dict, the rest is IDENTICAL to this notebook:
#   docs = list(real_descriptions.values())
#   X = TfidfVectorizer(stop_words="english").fit_transform(docs)
#   S = cosine_similarity(X)                 # same cosine, real firms
#
# --- Option B: just download the precomputed Hoberg-Phillips TNIC industries ----------------
# The "Hoberg-Phillips Data Library" publishes TNIC industries + pairwise similarity scores as
# downloadable text files (firm-pair-year similarity above their threshold). You merge them onto
# your Compustat/CRSP panel by CIK/GVKEY -- no re-parsing of EDGAR needed.
#   [CHECK: confirm the current canonical URL/host of the Hoberg-Phillips Data Library before use.]
#
#   # tnic = pd.read_csv("tnic3_data.txt", sep="\t")   # columns ~ year, gvkey1, gvkey2, score
#   # my_panel = my_panel.merge(tnic, on=["year", "gvkey"], how="left")
#
# Reference: Hoberg & Phillips (2016), J. Political Economy 124(5):1423-1465.
print("Reference cell only -- not executed. See comments for the EDGAR + TNIC-library path.")

Your Turn

You have the full mini-pipeline. Pick at least one and change exactly one thing at a time — that is how you learn what each knob does.

  1. Add a firm. Write a new product-description paragraph for, say, a fintech payments company (Devon’s territory — crypto/FinTech) and append it to descriptions. Re-run from Section 3. Where does it land on the heatmap? Whose TNIC neighborhood does it join, and at what threshold? Does it create a new intransitive triple?

  2. Change the threshold. Re-read Section 9’s intransitive triple at \tau = 0.05, 0.20, and 0.35. At what \tau does the triple stop being intransitive (either the chain breaks, or it closes into a triangle)? This is the chapter’s “knobs are levers” point: the network’s topology depends on the knob.

  3. Try bigrams. Pass ngram_range=(1, 2) to TfidfVectorizer so the vocabulary includes two-word phrases (“cloud platform”, “consumer loans”, “do not”). Recompute the similarity matrix. Does “do not compete” now sit differently from “compete”? (Bigrams are the cheapest step toward capturing meaning — a baby version of what embeddings do in Ch 6.5.) Compare the within-vs-cross gap to the unigram version.

  4. Stress the stop-words (Exercise 2b). Drop stop_words="english" entirely and re-run Section 7. Watch ubiquitous filler inflate every similarity and shrink the within-vs-cross gap — the mechanical reason the real pipeline prunes vocabulary so carefully.

A scaffold for option 3 to get you started:

# Your Turn -- option 3 scaffold: bigrams (1- and 2-word phrases) instead of single words.
tfidf_bigram = TfidfVectorizer(stop_words="english", ngram_range=(1, 2))
X_bi = tfidf_bigram.fit_transform(docs)
S_bi = cosine_similarity(X_bi)

w_bi, c_bi = within_vs_cross(S_bi, firms, sic_like)
print(f"bigram vocabulary size: {len(tfidf_bigram.get_feature_names_out())} "
      f"(vs {len(vocab)} unigrams)")
print(f"unigram: within={w_tfidf:.3f} cross={c_tfidf:.3f} "
      f"separation ratio={w_tfidf / c_tfidf:.2f}")
print(f"bigram : within={w_bi:.3f} cross={c_bi:.3f} "
      f"separation ratio={w_bi / c_bi:.2f}")
print("(Bigrams shrink within-cluster similarity too -- two firms must now share whole")
print(" PHRASES, not just words. Decide for yourself whether that helps or hurts here.)")
print("\nNow try: does an intransitive triple still exist with bigrams? Re-run Section 9's")
print("finder with S_bi. And add your own firm to `descriptions`, then re-run from Section 3.")