gbanyanandClaude Opus 4.8
6781c00d5b
Resolve the Firm C/D "two-snapshot" inconsistency from the co-author rev9.1 review. Root cause was NOT stale data but two live firm-assignment keys (assigned_accountant-> registry firm vs the excel_firm column) used inconsistently across tables; standardize on the registry key, which the headline Table IV/II-b already use. Sole difference = 379 signatures with excel_firm=C but registry firm=D; A and B identical under both keys. Numbers (all DB-verified, registry key, n=150,442): - Table II-c Firm C/D recomputed (was mixing keys within firm C across periods, which manufactured the spurious "third value" 38,934); now C 22,449+16,164=38,613, D 9,945+7,188=17,133, all four firms reconcile. - Table VI C/D counts + S III-B prose + Fig 3/6 captions -> 150,442. - S V-B HC-rate text -> Firm C 21.6->26.7%, Firm D 22.0->28.0%. Note on R4: the reviewer (PDF-only) asked to change S IV-C to 26.5/28.5 to match Table II-c; DB verification showed the reverse - S IV-C's 26.7/28.0 are correct and Table II-c was the stale outlier, so II-c was aligned to S IV-C (data-correct, opposite to the literal instruction). Accountant counts (R2 reviewer "179>171 impossible" = false positive; three distinct, all-reproducible universes): Table I + S III-B -> 457 (>=2 sig, owns the 150,442 signatures); Table III documented as the 437 with >=10 signatures (K=3 GMM subset, reproduces A=82.5/B=0.0/C=1.0/D=1.9% exactly); bootstrap 179/280 unchanged (accountant_id key, correct and invariant to the A-vs-BCD contrast). R3 (corpus scope): S III-B reworded - corpus = all retrievable reports, Big-4 as the primary analysis sample (removes the "corpus = four firms" vs "non-Big-4 in robustness" contradiction); per-firm counts now explicitly labelled A/B/C/D. R5 (spelling): unify to American (artefact->artifact x11, centred->centered, behaviour->behavior, analyse(d)->analyze(d), favours->favors). R6: delete non-standard "(+)" marker in S IV-C. Figures regenerated under the registry key: make_fig3_density.py and make_fig6_sensitivity.py switched to the assigned_accountant join (fig3/fig6 n=150,442); fig4/fig5 refreshed. FE/LOYO/bootstrap re-validated exactly (ORs 0.116/0.061/0.070, LOYO 53.1-54.9pp, full 53.7pp). Add CANONICAL_NUMBERS_rev9.1.md with full provenance, the analyzable/GMM definitions, and the firm-key root cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K35dXhb9XEM1mnYz6SSHpU
85 lines
3.9 KiB
Python
85 lines
3.9 KiB
Python
"""Figure 3 (real data version): 2D density of the two measures over the five-region scheme.
|
|
Replaces the earlier schematic with the actual distribution, with axis ticks and the rule cuts.
|
|
Reproduces from signature_analysis.db; Big-4, is_valid=1, both measures present."""
|
|
import matplotlib
|
|
matplotlib.use('Agg')
|
|
import matplotlib.pyplot as plt
|
|
from matplotlib.colors import LogNorm
|
|
from matplotlib.patches import Rectangle
|
|
import numpy as np
|
|
import sqlite3
|
|
|
|
DB = "/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db"
|
|
BIG4 = ('勤業眾信聯合', '資誠聯合', '安侯建業聯合', '安永聯合')
|
|
|
|
con = sqlite3.connect(DB)
|
|
cur = con.cursor()
|
|
cur.execute(f"""
|
|
SELECT s.max_similarity_to_same_accountant, s.min_dhash_independent
|
|
FROM signatures s JOIN accountants a ON s.assigned_accountant=a.name
|
|
WHERE s.max_similarity_to_same_accountant IS NOT NULL
|
|
AND s.min_dhash_independent IS NOT NULL
|
|
AND a.firm IN ({','.join(['?']*4)})
|
|
""", BIG4)
|
|
rows = cur.fetchall()
|
|
con.close()
|
|
cos = np.array([r[0] for r in rows], dtype=float)
|
|
dh = np.array([r[1] for r in rows], dtype=float)
|
|
n = len(cos)
|
|
|
|
LO, HI = 0.8547, 0.95
|
|
DH1, DH2 = 5, 15
|
|
xmin, xmax = 0.70, 1.002
|
|
ymin, ymax = -0.5, 30
|
|
ycap = 30 # display cap; values above are piled into the top row for visibility
|
|
|
|
dh_disp = np.minimum(dh, ycap - 0.5)
|
|
|
|
fig, ax = plt.subplots(figsize=(5.6, 4.4))
|
|
|
|
# faint region tint behind the density
|
|
ax.add_patch(Rectangle((xmin, ymin), LO - xmin, ymax - ymin, facecolor='#bdc3c7', alpha=0.12, zorder=0))
|
|
ax.add_patch(Rectangle((LO, ymin), HI - LO, ymax - ymin, facecolor='#f7dc6f', alpha=0.12, zorder=0))
|
|
ax.add_patch(Rectangle((HI, ymin), xmax - HI, DH1 - ymin, facecolor='#cb4335', alpha=0.14, zorder=0))
|
|
ax.add_patch(Rectangle((HI, DH1), xmax - HI, DH2 - DH1, facecolor='#eb984e', alpha=0.14, zorder=0))
|
|
ax.add_patch(Rectangle((HI, DH2), xmax - HI, ymax - DH2, facecolor='#aed6f1', alpha=0.14, zorder=0))
|
|
|
|
# real 2D density (log counts)
|
|
xedges = np.linspace(xmin, xmax, 90)
|
|
yedges = np.arange(-0.5, ycap + 0.5, 1.0) # integer dHash bins
|
|
H, xe, ye = np.histogram2d(cos, dh_disp, bins=[xedges, yedges])
|
|
pcm = ax.pcolormesh(xe, ye, H.T, norm=LogNorm(vmin=1, vmax=H.max()),
|
|
cmap='viridis', zorder=1, shading='flat')
|
|
cb = fig.colorbar(pcm, ax=ax, pad=0.02)
|
|
cb.set_label('signatures per cell (log scale)', fontsize=8)
|
|
cb.ax.tick_params(labelsize=7)
|
|
|
|
# cut lines
|
|
ax.axvline(LO, color='gray', ls=':', lw=1.1, zorder=3)
|
|
ax.axvline(HI, color='black', ls='--', lw=1.1, zorder=3)
|
|
ax.plot([HI, xmax], [DH1, DH1], 'k--', lw=0.9, zorder=3)
|
|
ax.plot([HI, xmax], [DH2, DH2], 'k--', lw=0.9, zorder=3)
|
|
|
|
# region labels
|
|
ax.text((xmin + LO) / 2, 24, 'LH', ha='center', fontsize=10, weight='bold', color='#34495e', zorder=4)
|
|
ax.text((LO + HI) / 2, 24, 'UN', ha='center', fontsize=10, weight='bold', color='#7d6608', zorder=4)
|
|
ax.text((HI + xmax) / 2, 2.2, 'HC', ha='center', fontsize=10, weight='bold', color='#cb4335', zorder=4)
|
|
ax.text((HI + xmax) / 2, 9.7, 'MC', ha='center', fontsize=10, weight='bold', color='#a04000', zorder=4)
|
|
ax.text((HI + xmax) / 2, 24, 'HSC', ha='center', fontsize=9, weight='bold', color='#21618c', zorder=4)
|
|
|
|
ax.set_xlim(xmin, xmax)
|
|
ax.set_ylim(ymin, ymax)
|
|
ax.set_xticks([0.70, 0.75, 0.80, 0.8547, 0.90, 0.95, 1.00])
|
|
ax.set_xticklabels(['0.70', '0.75', '0.80', '0.855', '0.90', '0.95', '1.00'], fontsize=7.5)
|
|
ax.set_yticks([0, 5, 10, 15, 20, 25, 30])
|
|
ax.set_yticklabels(['0', '5', '10', '15', '20', '25', '≥30'], fontsize=7.5)
|
|
ax.set_xlabel('cosine similarity to same accountant (style)', fontsize=9)
|
|
ax.set_ylabel('min dHash distance (structure)', fontsize=9)
|
|
ax.set_title(f'Figure 3. Two-measure plane: real density over the five regions (Big-4, n={n:,})',
|
|
fontsize=8.5)
|
|
fig.tight_layout()
|
|
out = '/Volumes/NV2/pdf_recognize/paper/v13_build/figures/fig3.png'
|
|
fig.savefig(out, dpi=200, bbox_inches='tight')
|
|
plt.close(fig)
|
|
print(f'fig3 density OK: n={n:,}, dHash>=30 piled: {(dh>=ycap).sum()}, written {out}')
|