gbanyan/pdf_signature_extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: gbanyan/pdf_signature_extraction:939a348da468741e7d9dd3e620425477abccd721
Branches Tags
gbanyan/pdf_signature_extraction:master gbanyan/pdf_signature_extraction:paper-a-v4-big4 gbanyan/pdf_signature_extraction:yolo-signature-pipeline gbanyan/pdf_signature_extraction:pp-ocrv5-research
gbanyan/pdf_signature_extraction:v1.0-hybrid-70percent
...
compare: gbanyan/pdf_signature_extraction:0ff1845b225923a9523efc8bd53a259dbc32ce0a
Branches Tags
gbanyan/pdf_signature_extraction:paper-a-v4-big4 gbanyan/pdf_signature_extraction:yolo-signature-pipeline gbanyan/pdf_signature_extraction:pp-ocrv5-research gbanyan/pdf_signature_extraction:master
gbanyan/pdf_signature_extraction:v1.0-hybrid-70percent

9 Commits
939a348da4 ... 0ff1845b22

Author SHA1 Message Date
gbanyan 0ff1845b22 Paper A v3.4: resolve codex round-3 major-revision blockers 
Three blockers from codex gpt-5.4 round-3 review (codex_review_gpt54_v3_3.md):

B1 Classifier vs three-method threshold mismatch
  - Methodology III-L rewritten to make explicit that the per-signature
    classifier and the accountant-level three-method convergence operate
    at different units (signature vs accountant) and are complementary
    rather than substitutable.
  - Add Results IV-G.3 + Table XII operational-threshold sensitivity:
    cos>0.95 vs cos>0.945 shifts dual-rule capture by 1.19 pp on whole
    Firm A; ~5% of signatures flip at the Uncertain/Moderate boundary.

B2 Held-out validation false "within Wilson CI" claim
  - Script 24 recomputes both calibration-fold and held-out-fold rates
    with Wilson 95% CIs and a two-proportion z-test on each rule.
  - Table XI replaced with the proper fold-vs-fold comparison; prose
    in Results IV-G.2 and Discussion V-C corrected: extreme rules agree
    across folds (p>0.7); operational rules in the 85-95% band differ
    by 1-5 pp due to within-Firm-A heterogeneity (random 30% sample
    contained more high-replication C1 accountants), not generalization
    failure.

B3 Interview evidence reframed as practitioner knowledge
  - The Firm A "interviews" referenced throughout v3.3 are private,
    informal professional conversations, not structured research
    interviews. Reframed accordingly: all "interview*" references in
    abstract / intro / methodology / results / discussion / conclusion
    are replaced with "domain knowledge / industry-practice knowledge".
  - This avoids overclaiming methodological formality and removes the
    human-subjects research framing that triggered the ethics-statement
    requirement.
  - Section III-H four-pillar Firm A validation now stands on visual
    inspection, signature-level statistics, accountant-level GMM, and
    the three Section IV-H analyses, with practitioner knowledge as
    background context only.
  - New Section III-M ("Data Source and Firm Anonymization") covers
    MOPS public-data provenance, Firm A/B/C/D pseudonymization, and
    conflict-of-interest declaration.

Add signature_analysis/24_validation_recalibration.py for the recomputed
calib-vs-held-out z-tests and the classifier sensitivity analysis;
output in reports/validation_recalibration/.

Pending (not in this commit): abstract length (368 -> 250 words),
Impact Statement removal, BD/McCrary sensitivity reporting, full
reproducibility appendix, references cleanup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 11:45:24 +08:00
gbanyan 5717d61dd4 Paper A v3.3: apply codex v3.2 peer-review fixes 
Codex (gpt-5.4) second-round review recommended 'minor revision'. This
commit addresses all issues flagged in that review.

## Structural fixes

- dHash calibration inconsistency (codex #1, most important):
  Clarified in Section III-L that the <=5 and <=15 dHash cutoffs come
  from the whole-sample Firm A cosine-conditional dHash distribution
  (median=5, P95=15), not from the calibration-fold independent-minimum
  dHash distribution (median=2, P95=9) which we report elsewhere as
  descriptive anchors. Added explicit note about the two dHash
  conventions and their relationship.

- Section IV-H framing (codex #2):
  Renamed "Firm A Benchmark Validation: Threshold-Independent Evidence"
  to "Additional Firm A Benchmark Validation" and clarified in the
  section intro that H.1 uses a fixed 0.95 cutoff, H.2 is fully
  threshold-free, H.3 uses the calibrated classifier. H.3's concluding
  sentence now says "the substantive evidence lies in the cross-firm
  gap" rather than claiming the test is threshold-free.

- Table XVI 93,979 typo fixed (codex #3):
  Corrected to 84,354 total (83,970 same-firm + 384 mixed-firm).

- Held-out Firm A denominator 124+54=178 vs 180 (codex #4):
  Added explicit note that 2 CPAs were excluded due to disambiguation
  ties in the CPA registry.

- Table VIII duplication (codex #5):
  Removed the duplicate accountant-level-only Table VIII comment; the
  comprehensive cross-level Table VIII subsumes it. Text now says
  "accountant-level rows of Table VIII (below)".

- Anonymization broken in Tables XIV-XVI (codex #6):
  Replaced "Deloitte"/"KPMG"/"PwC"/"EY" with "Firm A"/"Firm B"/"Firm C"/
  "Firm D" across Tables XIV, XV, XVI. Table and caption language
  updated accordingly.

- Table X unit mismatch (codex #7):
  Dropped precision, recall, F1 columns. Table now reports FAR
  (against the inter-CPA negative anchor) with Wilson 95% CIs and FRR
  (against the byte-identical positive anchor). III-K and IV-G.1 text
  updated to justify the change.

## Sentence-level fixes

- "three independent statistical methods" in Methodology III-A ->
  "three methodologically distinct statistical methods".
- "three independent methods" in Conclusion -> "three methodologically
  distinct methods".
- Abstract "~0.006 converging" now explicitly acknowledges that
  BD/McCrary produces no significant accountant-level discontinuity.
- Conclusion ditto.
- Discussion limitation sentence "BD/McCrary should be interpreted at
  the accountant level for threshold-setting purposes" rewritten to
  reflect v3.3 result that BD/McCrary is a diagnostic, not a threshold
  estimator, at the accountant level.
- III-H "two analyses" -> "three analyses" (H.1 longitudinal stability,
  H.2 partner ranking, H.3 intra-report consistency).
- Related Work White 1982 overclaim rewritten: "consistent estimators
  of the pseudo-true parameter that minimizes KL divergence" replaces
  "guarantees asymptotic recovery".
- III-J "behavior is close to discrete" -> "practice is clustered".
- IV-D.2 pivot sentence "discreteness of individual behavior yields
  bimodality" -> "aggregation over signatures reveals clustered (though
  not sharply discrete) patterns".

Target journal remains IEEE Access. Output:
Paper_A_IEEE_Access_Draft_v3.docx (395 KB).

Codex v3.2 review saved to paper/codex_review_gpt54_v3_2.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 02:32:17 +08:00
gbanyan 51d15b32a5 Paper A v3.2: partner v4 feedback integration (threshold-independent benchmark validation) 
Partner v4 (signature_paper_draft_v4) proposed 3 substantive improvements;
partner confirmed the 2013-2019 restriction was an error (sample stays
2013-2023). The remaining suggestions are adopted with our own data.

## New scripts
- Script 22 (partner ranking): ranks all Big-4 auditor-years by mean
  max-cosine. Firm A occupies 95.9% of top-10% (base 27.8%), 3.5x
  concentration ratio. Stable across 2013-2023 (88-100% per year).
- Script 23 (intra-report consistency): for each 2-signer report,
  classify both signatures and check agreement. Firm A agrees 89.9%
  vs 62-67% at other Big-4. 87.5% Firm A reports have BOTH signers
  non-hand-signed; only 4 reports (0.01%) both hand-signed.

## New methodology additions
- III-G: explicit within-auditor-year no-mixing identification
  assumption (supported by Firm A interview evidence).
- III-H: 4th Firm A validation line: threshold-independent evidence
  from partner ranking + intra-report consistency.

## New results section IV-H (threshold-independent validation)
- IV-H.1: Firm A year-by-year cosine<0.95 rate. 2013-2019 mean=8.26%,
  2020-2023 mean=6.96%, 2023 lowest (3.75%). Stability contradicts
  partner's hypothesis that 2020+ electronic systems increase
  heterogeneity -- data shows opposite (electronic systems more
  consistent than physical stamping).
- IV-H.2: partner ranking top-K tables (pooled + year-by-year).
- IV-H.3: intra-report consistency per-firm table.

## Renumbering
- Section H (was Classification Results) -> I
- Section I (was Ablation) -> J
- Tables XIII-XVI new (yearly stability, top-K pooled, top-10% per-year,
  intra-report), XVII = classification (was XII), XVIII = ablation
  (was XIII).

These threshold-independent analyses address the codex review concern
about circular validation by providing benchmark evidence that does not
depend on any threshold calibrated to Firm A itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 01:59:49 +08:00
gbanyan 9d19ca5a31 Paper A v3.1: apply codex peer-review fixes + add Scripts 20/21 
Major fixes per codex (gpt-5.4) review:

## Structural fixes
- Fixed three-method convergence overclaim: added Script 20 to run KDE
  antimode, BD/McCrary, and Beta mixture EM on accountant-level means.
  Accountant-level 1D convergence: KDE antimode=0.973, Beta-2=0.979,
  LogGMM-2=0.976 (within ~0.006). BD/McCrary finds no transition at
  accountant level (consistent with smooth clustering, not sharp
  discontinuity).
- Disambiguated Method 1: KDE crossover (between two labeled distributions,
  used at signature all-pairs level) vs KDE antimode (single-distribution
  local minimum, used at accountant level).
- Addressed Firm A circular validation: Script 21 adds CPA-level 70/30
  held-out fold. Calibration thresholds derived from 70% only; heldout
  rates reported with Wilson 95% CIs (e.g. cos>0.95 heldout=93.61%
  [93.21%-93.98%]).
- Fixed 139+32 vs 180: the split is 139/32 of 171 Firm A CPAs with >=10
  signatures (9 CPAs excluded for insufficient sample). Reconciled across
  intro, results, discussion, conclusion.
- Added document-level classification aggregation rule (worst-case signature
  label determines document label).

## Pixel-identity validation strengthened
- Script 21: built ~50,000-pair inter-CPA random negative anchor (replaces
  the original n=35 same-CPA low-similarity negative which had untenable
  Wilson CIs).
- Added Wilson 95% CI for every FAR in Table X.
- Proper EER interpolation (FAR=FRR point) in Table X.
- Softened "conservative recall" claim to "non-generalizable subset"
  language per codex feedback (byte-identical positives are a subset, not
  a representative positive class).
- Added inter-CPA stats: mean=0.762, P95=0.884, P99=0.913.

## Terminology & sentence-level fixes
- "statistically independent methods" -> "methodologically distinct methods"
  throughout (three diagnostics on the same sample are not independent).
- "formal bimodality check" -> "unimodality test" (dip test tests H0 of
  unimodality; rejection is consistent with but not a direct test of
  bimodality).
- "Firm A near-universally non-hand-signed" -> already corrected to
  "replication-dominated" in prior commit; this commit strengthens that
  framing with explicit held-out validation.
- "discrete-behavior regimes" -> "clustered accountant-level heterogeneity"
  (BD/McCrary non-transition at accountant level rules out sharp discrete
  boundaries; the defensible claim is clustered-but-smooth).
- Softened White 1982 quasi-MLE claim (no longer framed as a guarantee).
- Fixed VLM 1.2% FP overclaim (now acknowledges the 1.2% could be VLM FP
  or YOLO FN).
- Unified "310 byte-identical signatures" language across Abstract,
  Results, Discussion (previously alternated between pairs/signatures).
- Defined min_dhash_independent explicitly in Section III-G.
- Fixed table numbering (Table XI heldout added, classification moved to
  XII, ablation to XIII).
- Explained 84,386 vs 85,042 gap (656 docs have only one signature, no
  pairwise stat).
- Made Table IX explicitly a "consistency check" not "validation"; paired
  it with Table XI held-out rates as the genuine external check.
- Defined 0.941 threshold (calibration-fold Firm A cosine P5).
- Computed 0.945 Firm A rate exactly (94.52%) instead of interpolated.
- Fixed Ref [24] Qwen2.5-VL to full IEEE format (arXiv:2502.13923).

## New artifacts
- Script 20: accountant-level three-method threshold analysis
- Script 21: expanded validation (inter-CPA anchor, held-out Firm A 70/30)
- paper/codex_review_gpt54_v3.md: preserved review feedback

Output: Paper_A_IEEE_Access_Draft_v3.docx (391 KB, rebuilt from v3.1
markdown sources).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 01:11:51 +08:00
gbanyan 9b11f03548 Paper A v3: full rewrite for IEEE Access with three-method convergence 
Major changes from v2:

Terminology:
- "digitally replicated" -> "non-hand-signed" throughout (per partner v3
  feedback and to avoid implicit accusation)
- "Firm A near-universal non-hand-signing" -> "replication-dominated"
  (per interview nuance: most but not all Firm A partners use replication)

Target journal: IEEE TAI -> IEEE Access (per NCKU CSIE list)

New methodological sections (III.G-III.L + IV.D-IV.G):
- Three convergent threshold methods (KDE antimode + Hartigan dip test /
  Burgstahler-Dichev McCrary / EM-fitted Beta mixture + logit-GMM
  robustness check)
- Explicit unit-of-analysis discussion (signature vs accountant)
- Accountant-level 2D Gaussian mixture (BIC-best K=3 found empirically)
- Pixel-identity validation anchor (no manual annotation needed)
- Low-similarity negative anchor + Firm A replication-dominated anchor

New empirical findings integrated:
- Firm A signature cosine UNIMODAL (dip p=0.17) - long left tail = minority
  hand-signers
- Full-sample cosine MULTIMODAL but not cleanly bimodal (BIC prefers 3-comp
  mixture) - signature-level is continuous quality spectrum
- Accountant-level mixture trimodal (C1 Deloitte-heavy 139/141,
  C2 other Big-4, C3 smaller firms). 2-comp crossings cos=0.945, dh=8.10
- Pixel-identity anchor (310 pairs) gives perfect recall at all cosine
  thresholds
- Firm A anchor rates: cos>0.95=92.5%, dual-rule cos>0.95 AND dh<=8=89.95%

New discussion section V.B: "Continuous-quality spectrum vs discrete-
behavior regimes" - the core interpretive contribution of v3.

References added: Hartigan & Hartigan 1985, Burgstahler & Dichev 1997,
McCrary 2008, Dempster-Laird-Rubin 1977, White 1982 (refs 37-41).

export_v3.py builds Paper_A_IEEE_Access_Draft_v3.docx (462 KB, +40% vs v2
from expanded methodology + results sections).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-21 00:14:47 +08:00
gbanyan 68689c9f9b Correct Firm A framing: replication-dominated, not pure 
Interview evidence from multiple Firm A accountants confirms that MOST
use replication (stamping / firm-level e-signing) but a MINORITY may
still hand-sign. Firm A is therefore a "replication-dominated" population,
not a "pure" one. This framing is consistent with:

- 92.5% of Firm A signatures exceed cosine 0.95 (majority replication)
- The long left tail (~7%) captures the minority hand-signers, not scan
  noise or preprocessing artifacts
- Hartigan dip test: Firm A cosine unimodal long-tail (p=0.17)
- Accountant-level GMM: of 180 Firm A accountants, 139 cluster in C1
  (high-replication) and 32 in C2 (middle band = minority hand-signers)

Updates docstrings and report text in Scripts 15, 16, 18, 19 to match.
Partner v3's "near-universal non-hand-signing" language corrected.

Script 19 regenerated with the updated text.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 21:57:16 +08:00
gbanyan fbfab1fa68 Add three-convergent-method threshold scripts + pixel-identity validation 
Implements Partner v3's statistical rigor requirements at the level of
signature vs. accountant analysis units:

- Script 15 (Hartigan dip test): formal unimodality test via `diptest`.
  Result: Firm A cosine UNIMODAL (p=0.17, pure non-hand-signed population);
  full-sample cosine MULTIMODAL (p<0.001, mix of two regimes);
  accountant-level aggregates MULTIMODAL on both cos and dHash.

- Script 16 (Burgstahler-Dichev / McCrary): discretised Z-score transition
  detection. Firm A and full-sample cosine transitions at 0.985; dHash
  at 2.0.

- Script 17 (Beta mixture EM + logit-GMM): 2/3-component Beta via EM
  with MoM M-step, plus parallel Gaussian mixture on logit transform
  as White (1982) robustness check. Beta-3 BIC < Beta-2 BIC at signature
  level confirms 2-component is a forced fit -- supporting the pivot
  to accountant-level mixture.

- Script 18 (Accountant-level GMM): rebuilds the 2026-04-16 analysis
  that was done inline and not saved. BIC-best K=3 with components
  matching prior memory almost exactly: C1 (cos=0.983, dh=2.41, 20%,
  Deloitte 139/141), C2 (0.954, 6.99, 51%, KPMG/PwC/EY), C3 (0.928,
  11.17, 28%, small firms). 2-component natural thresholds:
  cos=0.9450, dh=8.10.

- Script 19 (Pixel-identity validation): no human annotation needed.
  Uses pixel_identical_to_closest (310 sigs) as gold positive and
  Firm A as anchor positive. Confirms Firm A cosine>0.95 = 92.51%
  (matches prior 2026-04-08 finding of 92.5%), dual rule
  cos>0.95 AND dhash_indep<=8 captures 89.95% of Firm A.

Python deps added: diptest, scikit-learn (installed into venv).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 21:51:41 +08:00
gbanyan 158f63efb2 Add Paper A drafts and docx export script 
- export_paper_to_docx.py: build script combining paper_a_*.md sections into docx
- Paper_A_IEEE_TAI_Draft_20260403.docx: intermediate draft before AI review rounds
- Paper_A_IEEE_TAI_Draft_v2.docx: current draft after 3 AI reviews (GPT-5.4, Opus 4.6, Gemini 3 Pro) and Firm A recalibration

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 21:34:31 +08:00
gbanyan a261a22bd2 Add Deloitte distribution & independent dHash analysis scripts 
- Script 13: Firm A normality/multimodality analysis (Shapiro-Wilk, Anderson-Darling, KDE, per-accountant ANOVA, Beta/Gamma fitting)
- Script 14: Independent min-dHash computation across all pairs per accountant (not just cosine-nearest pair)
- THRESHOLD_VALIDATION_OPTIONS: 2026-01 discussion doc on threshold validation approaches
- .gitignore: exclude model weights, node artifacts, and xlsx data

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 21:34:24 +08:00
31 changed files with 11118 additions and 0 deletions
Expand all files Collapse all files
.gitignore
+13
View File
@@ -48,3 +48,16 @@ Thumbs.db
# Temporary files # Temporary files
*.tmp *.tmp
*.bak *.bak
# Model weights (too large for git)
models/
*.pt
*.pth
# Node.js shells (accidentally created)
package.json
package-lock.json
node_modules/
# Sensitive/large data
*.xlsx
paper/Paper_A_IEEE_Access_Draft_v3.docx
BIN
View File
Binary file not shown.
paper/Paper_A_IEEE_TAI_Draft_20260403.docx
BIN
View File
Binary file not shown.
paper/Paper_A_IEEE_TAI_Draft_v2.docx
BIN
View File
Binary file not shown.
paper/codex_review_gpt54_v3.md
+1750
View File
File diff suppressed because it is too large Load Diff
paper/codex_review_gpt54_v3_2.md
+2277
View File
File diff suppressed because it is too large Load Diff
paper/codex_review_gpt54_v3_3.md
+130
View File
@@ -0,0 +1,130 @@
# Third-Round Review of Paper A v3.3
**Overall Verdict: Major Revision**
v3.3 is substantially cleaner than v3.2. Most of the round-2 minor issues were genuinely fixed: the anonymization leak is gone, the BD/McCrary wording is now much more careful, the denominator and table-arithmetic errors were corrected, and the manuscript now explicitly distinguishes cosine-conditional from independent-minimum dHash. I do not recommend submission as-is, however, because three non-cosmetic problems remain. First, the central "three-method convergent thresholding" story is still not aligned with the operational classifier: the deployed rules in Section III-L use whole-sample Firm A heuristics (`0.95`, `5`, `15`, `0.837`) rather than the convergent accountant-level thresholds reported in Section IV-E. Second, the held-out Firm A validation section makes an objectively false numerical claim that the held-out rates match the whole-sample rates within the Wilson confidence intervals. Third, the paper relies on interview evidence from Firm A partners as a key calibration pillar but provides no human-subjects/ethics statement, no consent/exemption language, and almost no protocol detail. Those are fixable, but they are still submission-blocking.
**1. v3.2 Findings Follow-up Audit**
| Prior v3.2 finding | Status | v3.3 audit |
|---|---|---|
| Three-method convergence overclaim | `FIXED` | The paper now consistently states that the *KDE antimode plus the two mixture-based estimators* converge, while BD/McCrary does not produce an accountant-level transition; see [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:11) and [paper_a_conclusion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_conclusion_v3.md:15). |
| KDE method inconsistency | `FIXED` | The KDE crossover vs KDE antimode distinction is now explicit in [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:167), and the Results use the distinction correctly at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:29). |
| Unit-of-analysis clarity | `PARTIALLY-FIXED` | The signature/accountant distinction is much clearer at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:116), but Sections III-L and IV-F/IV-G still mix analysis levels and dHash statistics. The classifier is described with cosine-conditional dHash cutoffs at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:265), while the validation tables report only `dHash_indep` rules at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:165) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:221). |
| Accountant-level interpretation overstated | `FIXED` | The manuscript now consistently frames the accountant-level result as clustered but smoothly mixed, not sharply discrete; see [paper_a_introduction_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_introduction_v3.md:59), [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:28), and [paper_a_conclusion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_conclusion_v3.md:17). |
| BD/McCrary rigor | `PARTIALLY-FIXED` | The overclaim is reduced and the limitation sentence is repaired at [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:103), but the paper still reports a fixed-bin implementation (`0.005` cosine bins) at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:175) without any reported bin-width sensitivity results or actual McCrary-style density-estimator output. |
| White 1982 overclaim | `FIXED` | Related Work now uses the narrower pseudo-true-parameter framing at [paper_a_related_work_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_related_work_v3.md:72), consistent with Methods at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:192). |
| Firm A circular validation | `PARTIALLY-FIXED` | The 70/30 CPA-level split is now explicit at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:209), but the actual classifier still uses whole-sample Firm A-derived rules at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:268). The manuscript therefore overstates how fully the held-out fold breaks circularity. |
| `139 + 32` vs `180` discrepancy | `FIXED` | The `171 + 9 = 180` accounting is now internally consistent; see [paper_a_introduction_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_introduction_v3.md:54), [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:122), [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:42), and [paper_a_conclusion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_conclusion_v3.md:21). |
| dHash calibration story internally inconsistent | `PARTIALLY-FIXED` | The distinction between cosine-conditional and independent-minimum dHash is finally stated at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:265), but the Results still do not "report both" as promised at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:267). Tables IX and XI still report only `dHash_indep` rules at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:165) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:221). |
| Section IV-H.3 not threshold-independent | `FIXED` | The paper now correctly labels H.3 as a classifier-based consistency check rather than a threshold-free test; see [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:150), [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:243), and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:336). |
| Table XVI numerical error | `FIXED` | The totals now reconcile: `83,970` single-firm reports plus `384` mixed-firm reports for `84,354` total at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:316). |
| Held-out Firm A denominator shift | `FIXED` | The `178`-CPA held-out denominator is now explicitly explained by two excluded disambiguation-tie CPAs at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:210). |
| Table numbering / cross-reference confusion | `PARTIALLY-FIXED` | The duplicate "Table VIII" phrasing is gone, but numbering still jumps from Table XI to Table XIII; see [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:214) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:251). |
| Real firm identities leaked in tables | `FIXED` | The manuscript now consistently uses `Firm A/B/C/D`; see [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:281) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:322). |
| Table X mixed unlike units while still reporting precision / F1 | `FIXED` | The paper now explicitly says precision and `F1` are not meaningful here and omits them; see [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:242) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:186). |
| "three independent statistical methods" wording | `FIXED` | The manuscript now uses "methodologically distinct" at [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:10) and [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:161). |
| Abstract / conclusion / discussion still implied BD converged | `FIXED` | The relevant sections now explicitly separate the non-transition result from the convergent estimators; see [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:11), [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:27), and [paper_a_conclusion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_conclusion_v3.md:16). |
| Stale "discrete behaviour" wording | `FIXED` | The current wording is appropriately narrowed at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:216) and [paper_a_conclusion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_conclusion_v3.md:17). |
| Related Work still overclaimed White 1982 | `FIXED` | The problematic sentence is gone; see [paper_a_related_work_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_related_work_v3.md:72). |
| Section III-H preview said "two analyses" | `FIXED` | It now correctly says "three analyses" at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:147). |
| Incorrect limitation sentence about BD/McCrary threshold-setting role | `FIXED` | The limitation is now correctly framed at [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:103). |
**2. New Findings in v3.3**
**Blockers**
- The paper still does not document the ethics status of the interview evidence that underwrites the Firm A calibration anchor. The interviews are not incidental; they are used in the Abstract, Introduction, Methods, Discussion, and Conclusion as one of the main justifications for identifying Firm A as replication-dominated; see [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:12), [paper_a_introduction_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_introduction_v3.md:52), and [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:140). There is no statement about IRB/review-board approval, exemption, participant consent, number of interviewees, interview dates, or anonymization protocol. For IEEE Access this is not optional if the paper reports human-subject research.
- The operational classifier is still not the classifier implied by the paper's title and main thresholding narrative. Section III-I says the accountant-level estimates are the threshold reference used in classification at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:210), and Section IV-E says the primary accountant-level interpretation comes from the `0.973 / 0.979 / 0.976` convergence band (with `0.945 / 8.10` as a secondary cross-check) at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:148). But the actual five-way classifier in Section III-L uses `0.95`, `0.837`, and dHash cutoffs `5 / 15` from whole-sample Firm A heuristics at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:251) and [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:268). As written, the paper demonstrates convergent threshold *analysis*, but deploys a different heuristic classifier.
- The "held-out fold confirms generalization" claim is numerically false as written. The manuscript states that the held-out rates "match the whole-sample rates of Table IX within each rule's Wilson confidence interval" at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:230), and repeats the same idea in Discussion at [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:44). That is not true for several published rules. Examples: whole-sample `cosine > 0.95 = 92.51%` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:163) is outside the held-out CI `[93.21%, 93.98%]` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:219); whole-sample `dHash_indep ≤ 5 = 84.20%` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:165) is outside `[87.31%, 88.34%]` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:221); whole-sample dual-rule `89.95%` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:168) is outside `[91.09%, 91.97%]` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:225). This needs correction, not softening.
**Major Issues**
- The dHash statistic used by the deployed classifier remains ambiguous. Section III-L says the final classifier retains the *cosine-conditional* dHash cutoffs for continuity at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:267), but Tables IX and XI report only `dHash_indep` rules at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:165) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:221). Section III-L also promises that anchor-level analysis reports both cosine-conditional and independent-minimum rates, but the Results do not. This is still a material reproducibility and interpretation gap.
- The paper still overstates what the 70/30 split accomplishes. Section III-K promises that calibration-fold percentiles are derived from the 70% fold only at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:235), but Section III-L then says the classifier uses thresholds inherited from the *whole-sample* Firm A distribution at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:268). That means the held-out fold is not a fully external evaluation for the actual deployed classifier.
- The validation-metric story still overpromises in the Introduction and Impact Statement. The Introduction says the design includes validation using "precision, recall, `F_1`, and equal-error-rate metrics" at [paper_a_introduction_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_introduction_v3.md:28), but Methods and Results later state that precision and `F_1` are not meaningful here and that FRR/recall is only valid for the conservative byte-identical subset at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:242) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:203). The Impact Statement is even stronger, claiming the system "distinguishes genuinely hand-signed signatures from reproduced ones" at [paper_a_impact_statement_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_impact_statement_v3.md:8), which is not what a five-way confidence classifier with no full ground-truth test set has established.
- The claimed empirical check on the within-auditor-year no-mixing assumption is not actually a check on that assumption. Section III-G says the intra-report consistency analysis "provides an empirical check on the within-auditor-year assumption" at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:127). But Section IV-H.3 measures agreement between *two different signers on the same report* at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:312); it does not test whether the *same CPA* mixes signing mechanisms within a fiscal year.
- BD/McCrary is still the weakest statistical component and is not yet reported rigorously enough to sit as an equal methodological peer to the other two methods. The paper specifies a fixed bin width at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:175) and mentions a KDE bandwidth sensitivity check at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:170), but no actual sensitivity results, `Z`-statistics, p-values, or alternate-bin outputs are reported anywhere in Section IV. The narrative conclusions are probably directionally reasonable, but the evidentiary reporting is still thin.
- Reproducibility from the paper alone is still insufficient. Missing or under-specified items include the exact VLM prompt and parsing rules ([paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:45)), HSV thresholds for red-stamp removal ([paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:74)), sampling/randomization seeds for the 500-image YOLO annotation set, the 500,000 inter-class pairs, the 50,000 inter-CPA negatives, and the 70/30 Firm A split ([paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:36), [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:185), [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:209)), and the initialization/convergence/clipping details for the Beta and logit-GMM fits ([paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:184), [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:191), [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:218)).
- Section III-H still contains one misleading sentence about H.1: it says the fixed `0.95` cutoff "is not calibrated to Firm A" at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:148), but Section IV-F explicitly says `0.95` and the dHash percentile rules are anchored to Firm A at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:174), and Section III-L says the classifier inherits thresholds from the whole-sample Firm A distribution at [paper_a_methodology_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_methodology_v3.md:268). Those statements need to be reconciled.
**Minor Issues**
- The table numbering still skips Table XII; the numbering jumps from Table XI at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:214) to Table XIII at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:251).
- The label `dHash_indep ≤ 5 (calib-fold median-adjacent)` at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:165) is still unclear. If the calibration-fold independent-minimum median is `2`, then `5` is not a transparent "median-adjacent" label.
- The references still need cleanup. At least `[27]` and `[31]`-`[36]` appear unused in the manuscript text, and the Mann-Whitney test is reported at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:50) without actually citing `[36]`.
**3. IEEE Access Fit Check**
- **Scope:** Yes. The topic fits IEEE Access well as a multidisciplinary methods paper spanning document forensics, computer vision, and audit-regulation applications.
- **Single-anonymized review:** IEEE Access uses single-anonymized review according to the current reviewer information page. The manuscript's use of `Firm A/B/C/D` is therefore not required for author anonymity, but it is acceptable as an entity-confidentiality choice.
- **Formatting / desk-return risks:** There are three concrete issues.
- The abstract is too long for current IEEE journal guidance. The IEEE Author Center says abstracts should be a single paragraph of up to 250 words, whereas the current abstract text at [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:5) through [paper_a_abstract_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_abstract_v3.md:14) is roughly 368 words by a plain-word count.
- The paper includes a standalone `Impact Statement` section at [paper_a_impact_statement_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_impact_statement_v3.md:1). That is not a standard IEEE Access Regular Paper section and should be removed or relocated unless the target article type explicitly requires it.
- Because the manuscript relies on partner interviews, it also appears to require the human-subject research statement that IEEE journal guidance asks authors to include when applicable.
- **Official sources checked:** [IEEE Access submission guidelines](https://ieeeaccess.ieee.org/authors/submission-guidelines/), [IEEE Author Center article-structure guidance](https://journals.ieeeauthorcenter.ieee.org/create-your-ieee-journal-article/create-the-text-of-your-article/structure-your-article/), and [IEEE Access reviewer information](https://ieeeaccess.ieee.org/wp-content/uploads/2025/09/Reviewer-Information.pdf).
**4. Statistical Rigor Audit**
- The paper's main high-level statistical narrative is now mostly coherent. The "Firm A is replication-dominated but not pure" framing is supported by the combination of the `92.5%` signature-level rate, the `139 / 32` accountant-level split, and the unimodal-long-tail characterization; see [paper_a_introduction_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_introduction_v3.md:54), [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:123), and [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:41).
- The Hartigan dip test is now described correctly as a unimodality test, and the paper no longer treats non-rejection as a formal bimodality finding. That said, the text at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:71) still moves quickly from "`p = 0.17`" to a substantive "single dominant generative mechanism" reading. That interpretation is plausible, but it is still an inference supported by interviews and ancillary evidence, not something the dip test itself establishes.
- The accountant-level 1D thresholds are statistically described more carefully than before. The `0.973 / 0.979 / 0.976` cosine band is internally consistent across Abstract, Introduction, Results, Discussion, and Conclusion, and the text now correctly treats BD/McCrary non-transition as diagnostic rather than as failed thresholding.
- The main remaining statistical weakness is the disconnect between *where the methods converge* and *what thresholds the classifier actually uses*. If the final classifier remains `0.95 / 5 / 15 / 0.837`, then the three-method convergence analysis is supporting context, not operational threshold-setting. The manuscript needs to say that explicitly or change the classifier accordingly.
- The anchor-based validation is improved, especially because precision and `F_1` were removed and Wilson CIs were added. But the EER remains close to vacuous here: with 310 byte-identical positives all sitting near cosine `1.0`, the reported "`EER ≈ 0`" at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:188) is not very informative and should not be treated as a strong biometric-style performance result.
**5. Anonymization Check**
- Within the reviewed manuscript sections, I do **not** see any explicit real firm names or real auditor names. Firms are consistently pseudonymized as `Firm A/B/C/D`; see [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:281) and [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:322).
- I also do **not** see author/institution metadata in the reviewed section files. From a single-anonymized IEEE Access standpoint, there is no obvious explicit anonymization leak in the manuscript text provided for review.
- The one caveat is inferential rather than explicit: the combination of interview-based knowledge, Big-4 status, and distinctive cross-firm statistics may allow knowledgeable local readers to guess which firm is Firm A. That is not an explicit leak, but if firm confidentiality matters beyond mere pseudonymization, the authors should be aware of the residual identifiability risk.
**6. Numerical Consistency**
- The major cross-section numbers are now mostly consistent:
- `90,282` reports / `182,328` signatures / `758` CPAs are aligned across Abstract, Introduction, Methods, and Conclusion.
- Firm A's `171` analyzable CPAs, `9` excluded CPAs, and `139 / 32` accountant-level split are aligned across Introduction, Results, Discussion, and Conclusion.
- The partner-ranking `95.9%` top-decile share and the intra-report `89.9%` agreement are aligned between Methods and Results.
- Table XVI and Table XVII arithmetic now reconciles.
- The remaining numerical inconsistency is the held-out-validation sentence discussed above. The underlying table counts are internally consistent, but the prose interpretation at [paper_a_results_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_results_v3.md:230) and [paper_a_discussion_v3.md](/Volumes/NV2/pdf_recognize/paper/paper_a_discussion_v3.md:44) is not.
- A second consistency problem is metric-level rather than arithmetic: the classifier is described in Section III-L using cosine-conditional dHash cutoffs, while the validation tables are reported in independent-minimum dHash. That numerical comparison is not apples-to-apples until the paper states clearly which statistic drives Table XVII.
**7. Reproducibility**
- The paper is **not yet replicable from the manuscript alone**.
- Missing items that should be added before submission:
- Exact VLM prompt, output format, and page-selection parse rule.
- YOLO training hyperparameters beyond epoch count and split ratio, plus inference confidence/NMS thresholds.
- HSV stamp-removal thresholds.
- Exact matching/disambiguation rules for CPA assignment ties.
- Random seeds and selection rules for the 500-page annotation sample, the 500,000 inter-class pairs, the 50,000 inter-CPA negatives, and the 70/30 Firm A split.
- EM/Beta/logit-GMM initialization, stopping criteria, handling of boundary values for the logit transform, and software/library versions for the mixture fits.
- Sensitivity-analysis results for KDE bandwidth and any analogous robustness checks for the BD/McCrary binning choice.
- Interview protocol details and the "independent visual inspection" sample size / decision rule.
- I would not describe the current paper as reproducible "from the paper alone" yet. It is closer than v3.2, but it still depends on undocumented implementation choices.
**Bottom Line**
v3.3 is close, and most of the v3.2 cleanup work landed correctly. But before IEEE Access submission, I would require: (1) a clean reconciliation between the three-method threshold story and the actual classifier, (2) correction of the false held-out-validation claim, and (3) an explicit ethics/human-subjects statement plus minimal protocol disclosure for the interview evidence. Once those are fixed, the paper is much closer to minor-revision territory.
paper/export_paper_to_docx.py
+575
View File
@@ -0,0 +1,575 @@
#!/usr/bin/env python3
"""
Export Paper A draft to a single Word document (.docx)
with IEEE-style formatting, embedded figures, and tables.
"""
from docx import Document
from docx.shared import Inches, Pt, RGBColor
from docx.enum.text import WD_ALIGN_PARAGRAPH
from docx.enum.table import WD_TABLE_ALIGNMENT
from pathlib import Path
import re
# Paths
PAPER_DIR = Path("/Volumes/NV2/pdf_recognize")
FIGURE_DIR = Path("/Volumes/NV2/PDF-Processing/signature-analysis/paper_figures")
OUTPUT_PATH = PAPER_DIR / "Paper_A_IEEE_TAI_Draft.docx"
def add_heading(doc, text, level=1):
h = doc.add_heading(text, level=level)
for run in h.runs:
run.font.color.rgb = RGBColor(0, 0, 0)
return h
def add_para(doc, text, bold=False, italic=False, font_size=10, alignment=None, space_after=6):
p = doc.add_paragraph()
if alignment:
p.alignment = alignment
p.paragraph_format.space_after = Pt(space_after)
p.paragraph_format.space_before = Pt(0)
run = p.add_run(text)
run.font.size = Pt(font_size)
run.font.name = 'Times New Roman'
run.bold = bold
run.italic = italic
return p
def add_table(doc, headers, rows, caption=None):
if caption:
add_para(doc, caption, bold=True, font_size=9, alignment=WD_ALIGN_PARAGRAPH.CENTER, space_after=4)
table = doc.add_table(rows=1 + len(rows), cols=len(headers))
table.style = 'Table Grid'
table.alignment = WD_TABLE_ALIGNMENT.CENTER
# Header
for i, h in enumerate(headers):
cell = table.rows[0].cells[i]
cell.text = h
for p in cell.paragraphs:
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
for run in p.runs:
run.bold = True
run.font.size = Pt(8)
run.font.name = 'Times New Roman'
# Data
for r_idx, row in enumerate(rows):
for c_idx, val in enumerate(row):
cell = table.rows[r_idx + 1].cells[c_idx]
cell.text = str(val)
for p in cell.paragraphs:
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
for run in p.runs:
run.font.size = Pt(8)
run.font.name = 'Times New Roman'
doc.add_paragraph() # spacing
return table
def add_figure(doc, image_path, caption, width=5.0):
if Path(image_path).exists():
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
run = p.add_run()
run.add_picture(str(image_path), width=Inches(width))
cap = doc.add_paragraph()
cap.alignment = WD_ALIGN_PARAGRAPH.CENTER
cap.paragraph_format.space_after = Pt(8)
run = cap.add_run(caption)
run.font.size = Pt(9)
run.font.name = 'Times New Roman'
run.italic = True
def build_document():
doc = Document()
# Set default font
style = doc.styles['Normal']
font = style.font
font.name = 'Times New Roman'
font.size = Pt(10)
# ==================== TITLE ====================
add_para(doc, "Automated Detection of Digitally Replicated Signatures\nin Large-Scale Financial Audit Reports",
bold=True, font_size=16, alignment=WD_ALIGN_PARAGRAPH.CENTER, space_after=12)
add_para(doc, "[Authors removed for double-blind review]",
italic=True, font_size=10, alignment=WD_ALIGN_PARAGRAPH.CENTER, space_after=4)
add_para(doc, "[Affiliations removed for double-blind review]",
italic=True, font_size=10, alignment=WD_ALIGN_PARAGRAPH.CENTER, space_after=12)
# ==================== ABSTRACT ====================
add_heading(doc, "Abstract", level=1)
abstract_text = (
"Regulations in many jurisdictions require Certified Public Accountants (CPAs) to personally sign each audit report they certify. "
"However, the digitization of financial reporting makes it trivial to reuse a scanned signature image across multiple reports, "
"bypassing this requirement. Unlike signature forgery, where an impostor imitates another person's handwriting, signature replication "
"involves a legitimate signer reusing a digital copy of their own genuine signature\u2014a practice that is virtually undetectable through "
"manual inspection at scale. We present an end-to-end AI pipeline that automatically detects signature replication in financial audit reports. "
"The pipeline employs a Vision-Language Model for signature page identification, YOLOv11 for signature region detection, and ResNet-50 for "
"deep feature extraction, followed by a dual-method verification combining cosine similarity with perceptual hashing (pHash). This dual-method "
"design distinguishes consistent handwriting style (high feature similarity but divergent perceptual hashes) from digital replication "
"(convergent evidence across both methods), resolving an ambiguity that single-metric approaches cannot address. We apply this pipeline to "
"90,282 audit reports filed by publicly listed companies in Taiwan over a decade (2013\u20132023), analyzing 182,328 signatures from 758 CPAs. "
"Using a known-replication accounting firm as a calibration reference, we establish distribution-free detection thresholds validated against "
"empirical ground truth. Our analysis reveals that cosine similarity alone overestimates replication rates by approximately 25-fold, "
"underscoring the necessity of multi-method verification. To our knowledge, this is the largest-scale forensic analysis of signature "
"authenticity in financial documents."
)
add_para(doc, abstract_text, font_size=9, space_after=8)
# ==================== IMPACT STATEMENT ====================
add_heading(doc, "Impact Statement", level=1)
impact_text = (
"Auditor signatures on financial reports are a key safeguard of corporate accountability. When Certified Public Accountants digitally "
"copy and paste a single signature image across multiple reports instead of signing each one individually, this safeguard is undermined\u2014"
"yet detecting such practices through manual inspection is infeasible at the scale of modern financial markets. We developed an artificial "
"intelligence system that automatically extracts and analyzes signatures from over 90,000 audit reports spanning ten years of filings by "
"publicly listed companies. By combining deep learning-based visual feature analysis with perceptual hashing, the system distinguishes "
"genuinely handwritten signatures from digitally replicated ones. Our analysis reveals that signature replication practices vary substantially "
"across accounting firms, with measurable differences between firms known to use digital replication and those that do not. This technology "
"can be directly deployed by financial regulators to automate signature authenticity monitoring at national scale."
)
add_para(doc, impact_text, font_size=9, space_after=8)
# ==================== I. INTRODUCTION ====================
add_heading(doc, "I. Introduction", level=1)
intro_paras = [
"Financial audit reports serve as a critical mechanism for ensuring corporate accountability and investor protection. "
"In Taiwan, the Certified Public Accountant Act (\u6703\u8a08\u5e2b\u6cd5 \u00a74) and the Financial Supervisory Commission\u2019s attestation regulations "
"(\u67e5\u6838\u7c3d\u8b49\u6838\u6e96\u6e96\u5247 \u00a76) require that certifying CPAs affix their signature or seal (\u7c3d\u540d\u6216\u84cb\u7ae0) to each audit report [1]. "
"While the law permits either a handwritten signature or a seal, the CPA\u2019s attestation on each report is intended to represent a deliberate, "
"individual act of professional endorsement for that specific audit engagement [2].",
"The digitization of financial reporting, however, has introduced a practice that challenges this intent. "
"As audit reports are now routinely generated, transmitted, and archived as PDF documents, it is technically trivial for a CPA to digitally "
"replicate a single scanned signature image and paste it across multiple reports. Although this practice may not violate the literal statutory "
"requirement of \u201csignature or seal,\u201d it raises substantive concerns about audit quality: if a CPA\u2019s signature is applied identically across "
"hundreds of reports without any variation, does it still represent meaningful attestation of individual professional judgment? "
"Unlike traditional signature forgery, where a third party attempts to imitate another person\u2019s handwriting, signature replication involves "
"the legitimate signer reusing a digital copy of their own genuine signature. This practice, while potentially widespread, is virtually "
"undetectable through manual inspection at scale: regulatory agencies overseeing thousands of publicly listed companies cannot feasibly "
"examine each signature for evidence of digital duplication.",
"The distinction between signature replication and signature forgery is both conceptually and technically important. "
"The extensive body of research on offline signature verification [3]\u2013[8] has focused almost exclusively on forgery detection\u2014determining "
"whether a questioned signature was produced by its purported author or by an impostor. This framing presupposes that the central threat "
"is identity fraud. In our context, identity is not in question; the CPA is indeed the legitimate signer. The question is whether the "
"physical act of signing occurred for each individual report, or whether a single signing event was digitally propagated across many reports. "
"This replication detection problem is, in one sense, simpler than forgery detection\u2014we need not model the variability of skilled forgers\u2014"
"but it requires a different analytical framework, one focused on detecting abnormally high similarity across documents rather than "
"distinguishing genuine from forged specimens.",
"Despite the significance of this problem for audit quality and regulatory oversight, no prior work has addressed signature replication "
"detection in financial documents at scale. Woodruff et al. [9] developed an automated pipeline for signature analysis in corporate filings "
"for anti-money laundering investigations, but their work focused on author clustering (grouping signatures by signer identity) rather than "
"detecting reuse of digital copies. Copy-move forgery detection methods [10], [11] address duplicated regions within or across images, but "
"are designed for natural images and do not account for the specific characteristics of scanned document signatures, where legitimate visual "
"similarity between a signer\u2019s authentic signatures is expected and must be distinguished from digital duplication. Research on near-duplicate "
"image detection using perceptual hashing combined with deep learning [12], [13] provides relevant methodological foundations, but has not "
"been applied to document forensics or signature analysis.",
"In this paper, we present a fully automated, end-to-end pipeline for detecting digitally replicated CPA signatures in audit reports at scale. "
"Our approach processes raw PDF documents through six sequential stages: (1) signature page identification using a Vision-Language Model (VLM), "
"(2) signature region detection using a trained YOLOv11 object detector, (3) deep feature extraction via a pre-trained ResNet-50 convolutional "
"neural network, (4) dual-method similarity verification combining cosine similarity of deep features with perceptual hash (pHash) distance, "
"(5) distribution-free threshold calibration using a known-replication reference group, and (6) statistical classification with cross-method validation.",
"The dual-method verification is central to our contribution. Cosine similarity of deep feature embeddings captures high-level visual style "
"similarity\u2014it can identify signatures that share similar stroke patterns and spatial layouts\u2014but cannot distinguish between a CPA who signs "
"consistently and one who reuses a digital copy. Perceptual hashing, by contrast, captures structural-level similarity that is sensitive to "
"pixel-level correspondence. By requiring convergent evidence from both methods, we can differentiate style consistency (high cosine similarity "
"but divergent pHash) from digital replication (high cosine similarity with convergent pHash), resolving an ambiguity that neither method can "
"address alone.",
"A distinctive feature of our approach is the use of a known-replication calibration group for threshold validation. Through domain expertise, "
"we identified a major accounting firm (hereafter \u201cFirm A\u201d) whose signatures are known to be digitally replicated across all audit reports. "
"This provides an empirical anchor for calibrating detection thresholds: any threshold that fails to classify Firm A\u2019s signatures as replicated "
"is demonstrably too conservative, while the distributional characteristics of Firm A\u2019s signatures establish an upper bound on the similarity "
"values achievable through replication in real-world scanned documents. This calibration strategy\u2014using a known-positive subpopulation to "
"validate detection thresholds\u2014addresses a persistent challenge in document forensics, where ground truth labels are scarce.",
"We apply this pipeline to 90,282 audit reports filed by publicly listed companies in Taiwan between 2013 and 2023, extracting and analyzing "
"182,328 individual CPA signatures from 758 unique accountants. To our knowledge, this represents the largest-scale forensic analysis of "
"signature authenticity in financial documents reported in the literature.",
]
for para in intro_paras:
add_para(doc, para)
# Contributions
add_para(doc, "The contributions of this paper are summarized as follows:", space_after=4)
contributions = [
"Problem formulation: We formally define the signature replication detection problem as distinct from signature forgery detection, "
"and argue that it requires a different analytical framework focused on intra-signer similarity distributions rather than "
"genuine-versus-forged classification.",
"End-to-end pipeline: We present a fully automated pipeline that processes raw PDF audit reports through VLM-based page identification, "
"YOLO-based signature detection, deep feature extraction, and dual-method similarity verification, requiring no manual intervention "
"after initial model training.",
"Dual-method verification: We demonstrate that combining deep feature cosine similarity with perceptual hashing resolves the fundamental "
"ambiguity between style consistency and digital replication, supported by an ablation study comparing three feature extraction backbones.",
"Calibration methodology: We introduce a threshold calibration approach using a known-replication reference group, providing empirical "
"validation in a domain where labeled ground truth is scarce.",
"Large-scale empirical analysis: We report findings from the analysis of over 90,000 audit reports spanning a decade, providing the "
"first large-scale empirical evidence on signature replication practices in financial reporting.",
]
for i, c in enumerate(contributions, 1):
p = doc.add_paragraph(style='List Number')
run = p.add_run(c)
run.font.size = Pt(10)
run.font.name = 'Times New Roman'
add_para(doc, "The remainder of this paper is organized as follows. Section II reviews related work on signature verification, "
"document forensics, and perceptual hashing. Section III describes the proposed methodology. Section IV presents experimental "
"results including the ablation study and calibration group analysis. Section V discusses the implications and limitations of "
"our findings. Section VI concludes with directions for future work.")
# ==================== II. RELATED WORK ====================
add_heading(doc, "II. Related Work", level=1)
add_heading(doc, "A. Offline Signature Verification", level=2)
add_para(doc, "Offline signature verification\u2014determining whether a static signature image is genuine or forged\u2014has been studied "
"extensively using deep learning. Bromley et al. [3] introduced the Siamese neural network architecture for signature verification, "
"establishing the pairwise comparison paradigm that remains dominant. Dey et al. [4] proposed SigNet, a convolutional Siamese network "
"for writer-independent offline verification, demonstrating that deep features learned from signature images generalize across signers "
"without per-writer retraining. Hadjadj et al. [5] addressed the practical constraint of limited reference samples, achieving competitive "
"verification accuracy using only a single known genuine signature per writer. More recently, Li et al. [6] introduced TransOSV, "
"the first Vision Transformer-based approach for offline signature verification, achieving state-of-the-art results. Tehsin et al. [7] "
"evaluated distance metrics for triplet Siamese networks, finding that Manhattan distance outperformed cosine and Euclidean alternatives.")
add_para(doc, "A common thread in this literature is the assumption that the primary threat is identity fraud: a forger attempting to produce "
"a convincing imitation of another person\u2019s signature. Our work addresses a fundamentally different problem\u2014detecting whether the "
"legitimate signer reused a digital copy of their own signature\u2014which requires analyzing intra-signer similarity distributions "
"rather than modeling inter-signer discriminability.")
add_para(doc, "Brimoh and Olisah [8] proposed a consensus-threshold approach that derives classification boundaries from known genuine "
"reference pairs, the methodology most closely related to our calibration strategy. However, their method operates on standard "
"verification benchmarks with laboratory-collected signatures, whereas our approach applies threshold calibration using a "
"known-replication subpopulation identified through domain expertise in real-world regulatory documents.")
add_heading(doc, "B. Document Forensics and Copy Detection", level=2)
add_para(doc, "Copy-move forgery detection (CMFD) identifies duplicated regions within or across images, typically targeting manipulated "
"photographs [10]. Abramova and Bohme [11] adapted block-based CMFD to scanned text documents, noting that standard methods perform "
"poorly in this domain because legitimate character repetitions produce high similarity scores that confound duplicate detection.")
add_para(doc, "Woodruff et al. [9] developed the work most closely related to ours: a fully automated pipeline for extracting and "
"analyzing signatures from corporate filings in the context of anti-money laundering investigations. Their system uses connected "
"component analysis for signature detection, GANs for noise removal, and Siamese networks for author clustering. While their "
"pipeline shares our goal of large-scale automated signature analysis on real regulatory documents, their objective\u2014grouping "
"signatures by authorship\u2014differs fundamentally from ours, which is detecting digital replication within a single author\u2019s "
"signatures across documents.")
add_para(doc, "In the domain of image copy detection, Pizzi et al. [13] proposed SSCD, a self-supervised descriptor using ResNet-50 "
"with contrastive learning for large-scale copy detection on natural images. Their work demonstrates that pre-trained CNN features "
"with cosine similarity provide a strong baseline for identifying near-duplicate images, supporting our feature extraction approach.")
add_heading(doc, "C. Perceptual Hashing", level=2)
add_para(doc, "Perceptual hashing algorithms generate compact fingerprints that are robust to minor image transformations while remaining "
"sensitive to substantive content changes [14]. Jakhar and Borah [12] demonstrated that combining perceptual hashing with deep "
"learning features significantly outperforms either approach alone for near-duplicate image detection, achieving AUROC of 0.99. "
"Their two-stage architecture\u2014pHash for fast structural comparison followed by deep features for semantic verification\u2014provides "
"methodological precedent for our dual-method approach, though applied to natural images rather than document signatures.")
add_heading(doc, "D. Deep Feature Extraction for Signature Analysis", level=2)
add_para(doc, "Several studies have explored pre-trained CNN features for signature comparison without metric learning or Siamese architectures. "
"Engin et al. [15] used ResNet-50 features with cosine similarity for offline signature verification on real-world scanned documents, "
"incorporating CycleGAN-based stamp removal as preprocessing. Tsourounis et al. [16] demonstrated successful transfer from handwritten "
"text recognition to signature verification. Chamakh and Bounouh [17] confirmed that a simple ResNet backbone with cosine similarity "
"achieves competitive verification accuracy across multilingual signature datasets without fine-tuning, supporting the viability of "
"our off-the-shelf feature extraction approach.")
# ==================== III. METHODOLOGY ====================
add_heading(doc, "III. Methodology", level=1)
add_heading(doc, "A. Pipeline Overview", level=2)
add_para(doc, "We propose a six-stage pipeline for large-scale signature replication detection in scanned financial documents. "
"Fig. 1 illustrates the overall architecture. The pipeline takes as input a corpus of PDF audit reports and produces, for each "
"document, a classification of its CPA signatures as genuine, uncertain, or replicated, along with confidence scores and "
"supporting evidence from multiple verification methods.")
add_figure(doc, FIGURE_DIR / "fig1_pipeline.png",
"Fig. 1. Pipeline architecture for automated signature replication detection.", width=6.5)
add_heading(doc, "B. Data Collection", level=2)
add_para(doc, "The dataset comprises 90,282 annual financial audit reports filed by publicly listed companies in Taiwan, covering fiscal "
"years 2013 to 2023. The reports were collected from the Market Observation Post System (MOPS) operated by the Taiwan Stock Exchange "
"Corporation, the official repository for mandatory corporate filings. CPA names, affiliated accounting firms, and audit engagement "
"tenure were obtained from a publicly available audit firm tenure registry encompassing 758 unique CPAs.")
add_table(doc,
["Attribute", "Value"],
[
["Total PDF documents", "90,282"],
["Date range", "2013\u20132023"],
["Documents with signatures", "86,072 (95.4%)"],
["Unique CPAs identified", "758"],
["Accounting firms", ">50"],
],
caption="TABLE I: Dataset Summary")
add_heading(doc, "C. Signature Page Identification", level=2)
add_para(doc, "To identify which page of each multi-page PDF contains the auditor\u2019s signatures, we employed the Qwen2.5-VL "
"vision-language model (32B parameters) [18] as an automated pre-screening mechanism. Each PDF page was rendered to JPEG at "
"180 DPI and submitted to the VLM with a structured prompt requesting a binary determination of whether the page contains "
"a Chinese handwritten signature. The scanning range was restricted to the first quartile of each document\u2019s page count, "
"reflecting the regulatory structure of Taiwanese audit reports. This process identified 86,072 documents with signature pages. "
"Cross-validation between the VLM and subsequent YOLO detection confirmed high agreement: YOLO successfully detected signature "
"regions in 98.8% of VLM-positive documents.")
add_heading(doc, "D. Signature Detection", level=2)
add_para(doc, "We adopted YOLOv11n (nano variant) [19] for signature region localization. A training set of 500 randomly sampled signature "
"pages was annotated using a custom web-based interface following a two-stage protocol: primary annotation followed by independent "
"review and correction.")
add_table(doc,
["Metric", "Value"],
[
["Precision", "0.97\u20130.98"],
["Recall", "0.95\u20130.98"],
["mAP@0.50", "0.98\u20130.99"],
["mAP@0.50:0.95", "0.85\u20130.90"],
],
caption="TABLE II: YOLO Detection Performance")
add_para(doc, "Batch inference on 86,071 documents extracted 182,328 signature images at 43.1 documents/second (8 workers). "
"A red stamp removal step was applied using HSV color space filtering. Each signature was matched to its corresponding CPA "
"using positional order against the official registry, achieving a 92.6% match rate.")
add_heading(doc, "E. Feature Extraction", level=2)
add_para(doc, "Each extracted signature was encoded into a 2048-dimensional feature vector using a pre-trained ResNet-50 CNN [20] with "
"ImageNet-1K V2 weights, used as a fixed feature extractor without fine-tuning. Preprocessing consisted of resizing to "
"224\u00d7224 pixels with aspect ratio preservation and white padding, followed by ImageNet channel normalization. All feature "
"vectors were L2-normalized, ensuring that cosine similarity equals the dot product. The choice of ResNet-50 without fine-tuning "
"was motivated by three considerations: (1) the task is similarity comparison rather than classification; (2) ImageNet features "
"transfer effectively to document analysis [15], [16]; and (3) the absence of fine-tuning preserves generalizability. "
"This design choice is validated by an ablation study (Section IV-F).")
add_heading(doc, "F. Dual-Method Similarity Verification", level=2)
add_para(doc, "For each signature, the most similar signature from the same CPA across all other documents was identified via cosine "
"similarity. Two complementary measures were then computed against this closest match:")
add_para(doc, "Cosine similarity captures high-level visual style similarity: sim(fA, fB) = fA \u00b7 fB, where fA and fB are L2-normalized "
"feature vectors. A high cosine similarity indicates shared visual characteristics but does not distinguish between consistent "
"handwriting style and digital duplication.")
add_para(doc, "Perceptual hash (pHash) distance captures structural-level similarity. Each signature is converted to a 64-bit binary "
"fingerprint by resizing to 9\u00d78 pixels and computing horizontal gradient differences. The Hamming distance between two hashes "
"quantifies perceptual dissimilarity: 0 indicates perceptually identical images, while distances exceeding 15 indicate clearly "
"different images.")
add_para(doc, "The complementarity of these measures resolves the style-versus-replication ambiguity: high cosine + low pHash = converging "
"evidence of replication; high cosine + high pHash = consistent style, not replication. SSIM was excluded as a primary method "
"because scan-induced pixel variations caused a known-replication firm to exhibit a mean SSIM of only 0.70.")
add_heading(doc, "G. Threshold Selection and Calibration", level=2)
add_para(doc, "Intra-class (same CPA, 41.3M pairs) and inter-class (different CPAs, 500K pairs) cosine similarity distributions were "
"computed. Shapiro-Wilk tests rejected normality (p < 0.001), motivating distribution-free, percentile-based thresholds. "
"The primary threshold was derived via KDE crossover\u2014the point where intra- and inter-class density functions intersect.")
add_para(doc, "A distinctive aspect is the use of Firm A\u2014a major firm whose signatures are known to be digitally replicated\u2014as a "
"calibration reference. Firm A\u2019s distribution provides: (1) lower bound validation\u2014any threshold must classify the vast majority "
"of Firm A as replicated; and (2) upper bound estimation\u2014Firm A\u2019s 1st percentile establishes the floor of similarity achievable "
"through replication in scanned documents.")
add_heading(doc, "H. Classification", level=2)
add_para(doc, "The final per-document classification integrates evidence from both methods: (1) Definite replication: pixel-identical match "
"or SSIM > 0.95 with pHash \u2264 5; (2) Likely replication: cosine > 0.95 with pHash \u2264 5, or multiple methods indicate replication; "
"(3) Uncertain: cosine between KDE crossover and 0.95 without structural evidence; (4) Likely genuine: cosine below KDE crossover.")
# ==================== IV. RESULTS ====================
add_heading(doc, "IV. Experiments and Results", level=1)
add_heading(doc, "A. Experimental Setup", level=2)
add_para(doc, "All experiments were conducted using PyTorch 2.9 with Apple Silicon MPS GPU acceleration. "
"Feature extraction used torchvision model implementations with identical preprocessing across all backbones.")
add_heading(doc, "B. Distribution Analysis", level=2)
add_para(doc, "Fig. 2 presents the cosine similarity distributions for intra-class and inter-class pairs.")
add_figure(doc, FIGURE_DIR / "fig2_intra_inter_kde.png",
"Fig. 2. Cosine similarity distributions: intra-class (same CPA) vs. inter-class (different CPAs). "
"KDE crossover at 0.837 marks the Bayes-optimal decision boundary.", width=3.5)
add_table(doc,
["Statistic", "Intra-class", "Inter-class"],
[
["N (pairs)", "41,352,824", "500,000"],
["Mean", "0.821", "0.758"],
["Std. Dev.", "0.098", "0.090"],
["Median", "0.836", "0.774"],
],
caption="TABLE IV: Cosine Similarity Distribution Statistics")
add_para(doc, "Cohen\u2019s d of 0.669 indicates a medium effect size, confirming that the distributional difference is not merely "
"statistically significant but also practically meaningful.")
add_heading(doc, "C. Calibration Group Analysis", level=2)
add_para(doc, "Fig. 3 presents the per-signature best-match cosine similarity distribution of Firm A compared to other CPAs.")
add_figure(doc, FIGURE_DIR / "fig3_firm_a_calibration.png",
"Fig. 3. Per-signature best-match cosine similarity: Firm A (known replication) vs. other CPAs. "
"Firm A\u2019s 1st percentile (0.908) validates threshold selection.", width=3.5)
add_table(doc,
["Statistic", "Firm A", "All CPAs"],
[
["N (signatures)", "60,448", "168,740"],
["Mean", "0.980", "0.961"],
["Std. Dev.", "0.019", "0.029"],
["1st percentile", "0.908", "\u2014"],
["% > 0.95", "92.5%", "\u2014"],
["% > 0.90", "99.3%", "\u2014"],
],
caption="TABLE VI: Firm A Calibration Statistics (Per-Signature Best Match)")
add_para(doc, "Firm A\u2019s per-signature best-match cosine similarity (mean = 0.980, std = 0.019) is notably higher and more concentrated "
"than the overall CPA population (mean = 0.961, std = 0.029). Critically, 99.3% of Firm A\u2019s signatures exhibit a best-match "
"similarity exceeding 0.90, and the 1st percentile is 0.908\u2014establishing that any threshold below 0.91 would fail to capture "
"even the most dissimilar replicated signatures in the calibration group.")
add_heading(doc, "D. Classification Results", level=2)
add_table(doc,
["Verdict", "N (PDFs)", "%", "Description"],
[
["Definite replication", "2,403", "2.8%", "Pixel-level evidence"],
["Likely replication", "69,255", "81.4%", "Feature-level evidence"],
["Uncertain", "12,681", "14.9%", "Between thresholds"],
["Likely genuine", "47", "0.1%", "Below KDE crossover"],
["Unknown", "656", "0.8%", "Unmatched CPA"],
],
caption="TABLE VII: Classification Results (85,042 Documents)")
add_para(doc, "The most striking finding is the discrepancy between feature-level and pixel-level evidence. Of the 71,656 documents with "
"cosine similarity exceeding 0.95, only 3.4% (2,427) simultaneously exhibited SSIM > 0.95, and only 4.3% (3,081) had a pHash "
"distance of 0. This gap demonstrates that the vast majority of high cosine similarity scores reflect consistent signing style "
"rather than digital replication, vindicating the dual-method approach.")
add_para(doc, "The 267 pixel-identical signatures (0.4%) constitute the strongest evidence of digital replication, as it is physically "
"impossible for two instances of genuine handwriting to produce identical pixel arrays.")
add_heading(doc, "E. Ablation Study: Feature Backbone Comparison", level=2)
add_para(doc, "To validate the choice of ResNet-50, we compared three pre-trained architectures (Fig. 4).")
add_figure(doc, FIGURE_DIR / "fig4_ablation.png",
"Fig. 4. Ablation study comparing three feature extraction backbones: "
"(a) intra/inter-class mean similarity, (b) Cohen\u2019s d, (c) KDE crossover point.", width=6.5)
add_table(doc,
["Metric", "ResNet-50", "VGG-16", "EfficientNet-B0"],
[
["Feature dim", "2048", "4096", "1280"],
["Intra mean", "0.821", "0.822", "0.786"],
["Inter mean", "0.758", "0.767", "0.699"],
["Cohen\u2019s d", "0.669", "0.564", "0.707"],
["KDE crossover", "0.837", "0.850", "0.792"],
["Firm A mean", "0.826", "0.820", "0.810"],
["Firm A 1st pct", "0.543", "0.520", "0.454"],
],
caption="TABLE IX: Backbone Comparison")
add_para(doc, "EfficientNet-B0 achieves the highest Cohen\u2019s d (0.707), but exhibits the widest distributional spread, resulting in "
"lower per-sample classification confidence. VGG-16 performs worst despite the highest dimensionality. ResNet-50 provides the "
"best balance: competitive Cohen\u2019s d, tightest distributions, highest Firm A 1st percentile (0.543), and practical feature "
"dimensionality.")
# ==================== V. DISCUSSION ====================
add_heading(doc, "V. Discussion", level=1)
add_heading(doc, "A. Replication Detection as a Distinct Problem", level=2)
add_para(doc, "Our results highlight the importance of distinguishing signature replication detection from forgery detection. "
"Forgery detection optimizes for inter-class discriminability\u2014maximizing the gap between genuine and forged signatures. "
"Replication detection requires sensitivity to the upper tail of the intra-class similarity distribution, where the boundary "
"between consistent handwriting and digital copies becomes ambiguous. The dual-method framework addresses this ambiguity "
"in a way that single-method approaches cannot.")
add_heading(doc, "B. The Style-Replication Gap", level=2)
add_para(doc, "The most important empirical finding is the magnitude of the gap between style similarity and digital replication. "
"Of documents with cosine similarity exceeding 0.95, only 3.4% exhibited pixel-level evidence of actual replication via SSIM, "
"and only 4.3% via pHash. This implies that a naive cosine-only approach would overestimate the replication rate by approximately "
"25-fold. This gap likely reflects the nature of CPA signing practices: many accountants develop highly consistent signing habits, "
"resulting in signatures that appear nearly identical at the feature level while retaining microscopic handwriting variations.")
add_heading(doc, "C. Value of Known-Replication Calibration", level=2)
add_para(doc, "The use of Firm A as a calibration reference addresses a fundamental challenge in document forensics: the scarcity of "
"ground truth labels. Our approach leverages domain knowledge\u2014the established practice of digital signature replication at "
"a specific firm\u2014to create a naturally occurring positive control group. This calibration strategy has broader applicability: "
"any forensic detection system can benefit from identifying subpopulations with known characteristics to anchor threshold selection.")
add_heading(doc, "D. Limitations", level=2)
add_para(doc, "Several limitations should be acknowledged. First, comprehensive ground truth labels are not available for the full dataset. "
"While pixel-identical cases and Firm A provide anchor points, a small-scale manual verification study would strengthen confidence "
"in classification boundaries. Second, the ResNet-50 feature extractor was not fine-tuned on domain-specific data. Third, scanning "
"equipment and compression algorithms may have changed over the 10-year study period. Fourth, the classification framework does not "
"account for potential changes in signing practice over time. Finally, whether digital replication constitutes a violation of signing "
"requirements is a legal question that our technical analysis can inform but cannot resolve.")
# ==================== VI. CONCLUSION ====================
add_heading(doc, "VI. Conclusion and Future Work", level=1)
add_para(doc, "We have presented an end-to-end AI pipeline for detecting digitally replicated signatures in financial audit reports at scale. "
"Applied to 90,282 audit reports from Taiwanese publicly listed companies spanning 2013\u20132023, our system extracted and analyzed "
"182,328 CPA signatures using VLM-based page identification, YOLO-based signature detection, deep feature extraction, and "
"dual-method similarity verification.")
add_para(doc, "Our key findings are threefold. First, signature replication detection is a distinct problem from forgery detection, requiring "
"different analytical tools. Second, combining cosine similarity with perceptual hashing is essential for distinguishing consistent "
"handwriting style from digital duplication\u2014a single-metric approach overestimates replication rates by approximately 25-fold. "
"Third, a calibration methodology using a known-replication reference group provides empirical threshold validation in the absence "
"of comprehensive ground truth.")
add_para(doc, "An ablation study confirmed that ResNet-50 offers the best balance of discriminative power, classification stability, and "
"computational efficiency among three evaluated backbones.")
add_para(doc, "Future directions include domain-adapted feature extractors, temporal analysis of signing practice evolution, cross-country "
"generalization, regulatory system integration, and small-scale ground truth validation through expert review.")
# ==================== REFERENCES ====================
add_heading(doc, "References", level=1)
refs = [
'[1] Taiwan Certified Public Accountant Act (\u6703\u8a08\u5e2b\u6cd5), Art. 4; FSC Attestation Regulations (\u67e5\u6838\u7c3d\u8b49\u6838\u6e96\u6e96\u5247), Art. 6.',
'[2] S.-H. Yen, Y.-S. Chang, and H.-L. Chen, \u201cDoes the signature of a CPA matter? Evidence from Taiwan,\u201d Res. Account. Regul., vol. 25, no. 2, pp. 230\u2013235, 2013.',
'[3] J. Bromley et al., \u201cSignature verification using a Siamese time delay neural network,\u201d in Proc. NeurIPS, 1993.',
'[4] S. Dey et al., \u201cSigNet: Convolutional Siamese network for writer independent offline signature verification,\u201d arXiv:1707.02131, 2017.',
'[5] I. Hadjadj et al., \u201cAn offline signature verification method based on a single known sample and an explainable deep learning approach,\u201d Appl. Sci., vol. 10, no. 11, p. 3716, 2020.',
'[6] H. Li et al., \u201cTransOSV: Offline signature verification with transformers,\u201d Pattern Recognit., vol. 145, p. 109882, 2024.',
'[7] S. Tehsin et al., \u201cEnhancing signature verification using triplet Siamese similarity networks in digital documents,\u201d Mathematics, vol. 12, no. 17, p. 2757, 2024.',
'[8] P. Brimoh and C. C. Olisah, \u201cConsensus-threshold criterion for offline signature verification using CNN learned representations,\u201d arXiv:2401.03085, 2024.',
'[9] N. Woodruff et al., \u201cFully-automatic pipeline for document signature analysis to detect money laundering activities,\u201d arXiv:2107.14091, 2021.',
'[10] Copy-move forgery detection in digital image forensics: A survey, Multimedia Tools Appl., 2024.',
'[11] S. Abramova and R. Bohme, \u201cDetecting copy-move forgeries in scanned text documents,\u201d in Proc. Electronic Imaging, 2016.',
'[12] Y. Jakhar and M. D. Borah, \u201cEffective near-duplicate image detection using perceptual hashing and deep learning,\u201d Inf. Process. Manage., p. 104086, 2025.',
'[13] E. Pizzi et al., \u201cA self-supervised descriptor for image copy detection,\u201d in Proc. CVPR, 2022.',
'[14] A survey of perceptual hashing for multimedia, ACM Trans. Multimedia Comput. Commun. Appl., vol. 21, no. 7, 2025.',
'[15] D. Engin et al., \u201cOffline signature verification on real-world documents,\u201d in Proc. CVPRW, 2020.',
'[16] D. Tsourounis et al., \u201cFrom text to signatures: Knowledge transfer for efficient deep feature learning in offline signature verification,\u201d Expert Syst. Appl., 2022.',
'[17] B. Chamakh and O. Bounouh, \u201cA unified ResNet18-based approach for offline signature classification and verification,\u201d Procedia Comput. Sci., vol. 270, 2025.',
'[18] Qwen2.5-VL Technical Report, Alibaba Group, 2025.',
'[19] Ultralytics, \u201cYOLOv11 documentation,\u201d 2024. [Online]. Available: https://docs.ultralytics.com/',
'[20] K. He, X. Zhang, S. Ren, and J. Sun, \u201cDeep residual learning for image recognition,\u201d in Proc. CVPR, 2016.',
'[21] J. V. Carcello and C. Li, \u201cCosts and benefits of requiring an engagement partner signature: Recent experience in the United Kingdom,\u201d The Accounting Review, vol. 88, no. 5, pp. 1511\u20131546, 2013.',
'[22] A. D. Blay, M. Notbohm, C. Schelleman, and A. Valencia, \u201cAudit quality effects of an individual audit engagement partner signature mandate,\u201d Int. J. Auditing, vol. 18, no. 3, pp. 172\u2013192, 2014.',
'[23] W. Chi, H. Huang, Y. Liao, and H. Xie, \u201cMandatory audit partner rotation, audit quality, and market perception: Evidence from Taiwan,\u201d Contemp. Account. Res., vol. 26, no. 2, pp. 359\u2013391, 2009.',
'[24] L. G. Hafemann, R. Sabourin, and L. S. Oliveira, \u201cLearning features for offline handwritten signature verification using deep convolutional neural networks,\u201d Pattern Recognit., vol. 70, pp. 163\u2013176, 2017.',
'[25] L. G. Hafemann, R. Sabourin, and L. S. Oliveira, \u201cMeta-learning for fast classifier adaptation to new users of signature verification systems,\u201d IEEE Trans. Inf. Forensics Security, vol. 15, pp. 1735\u20131745, 2019.',
'[26] E. N. Zois, D. Tsourounis, and D. Kalivas, \u201cSimilarity distance learning on SPD manifold for writer independent offline signature verification,\u201d IEEE Trans. Inf. Forensics Security, vol. 19, pp. 1342\u20131356, 2024.',
'[27] H. Farid, \u201cImage forgery detection,\u201d IEEE Signal Process. Mag., vol. 26, no. 2, pp. 16\u201325, 2009.',
'[28] F. Z. Mehrjardi, A. M. Latif, M. S. Zarchi, and R. Sheikhpour, \u201cA survey on deep learning-based image forgery detection,\u201d Pattern Recognit., vol. 144, art. no. 109778, 2023.',
'[29] A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky, \u201cNeural codes for image retrieval,\u201d in Proc. ECCV, 2014, pp. 584\u2013599.',
'[30] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, \u201cYou only look once: Unified, real-time object detection,\u201d in Proc. CVPR, 2016, pp. 779\u2013788.',
'[31] J. Zhang, J. Huang, S. Jin, and S. Lu, \u201cVision-language models for vision tasks: A survey,\u201d IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 8, pp. 5625\u20135644, 2024.',
'[32] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, \u201cImage quality assessment: From error visibility to structural similarity,\u201d IEEE Trans. Image Process., vol. 13, no. 4, pp. 600\u2013612, 2004.',
'[33] B. W. Silverman, Density Estimation for Statistics and Data Analysis. London: Chapman & Hall, 1986.',
'[34] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum, 1988.',
'[35] H. B. Mann and D. R. Whitney, \u201cOn a test of whether one of two random variables is stochastically larger than the other,\u201d Ann. Math. Statist., vol. 18, no. 1, pp. 50\u201360, 1947.',
]
for ref in refs:
add_para(doc, ref, font_size=8, space_after=2)
# Save
doc.save(str(OUTPUT_PATH))
print(f"Saved: {OUTPUT_PATH}")
if __name__ == "__main__":
build_document()
paper/export_v3.py
+233
View File
@@ -0,0 +1,233 @@
#!/usr/bin/env python3
"""Export Paper A v3 (IEEE Access target) to Word, reading from v3 md section files."""
from docx import Document
from docx.shared import Inches, Pt, RGBColor
from docx.enum.text import WD_ALIGN_PARAGRAPH
from pathlib import Path
import re
PAPER_DIR = Path("/Volumes/NV2/pdf_recognize/paper")
FIG_DIR = Path("/Volumes/NV2/PDF-Processing/signature-analysis/paper_figures")
EXTRA_FIG_DIR = Path("/Volumes/NV2/PDF-Processing/signature-analysis/reports")
OUTPUT = PAPER_DIR / "Paper_A_IEEE_Access_Draft_v3.docx"
SECTIONS = [
"paper_a_abstract_v3.md",
"paper_a_impact_statement_v3.md",
"paper_a_introduction_v3.md",
"paper_a_related_work_v3.md",
"paper_a_methodology_v3.md",
"paper_a_results_v3.md",
"paper_a_discussion_v3.md",
"paper_a_conclusion_v3.md",
"paper_a_references_v3.md",
]
# Figure insertion hooks (trigger phrase -> (file, caption, width inches)).
# New figures for v3: dip test, BD/McCrary overlays, accountant GMM 2D + marginals.
FIGURES = {
"Fig. 1 illustrates": (
FIG_DIR / "fig1_pipeline.png",
"Fig. 1. Pipeline architecture for automated non-hand-signed signature detection.",
6.5,
),
"Fig. 2 presents the cosine similarity distributions for intra-class": (
FIG_DIR / "fig2_intra_inter_kde.png",
"Fig. 2. Cosine similarity distributions: intra-class vs. inter-class with KDE crossover at 0.837.",
3.5,
),
"Fig. 3 presents the per-signature cosine and dHash distributions of Firm A": (
FIG_DIR / "fig3_firm_a_calibration.png",
"Fig. 3. Firm A per-signature cosine and dHash distributions against the overall CPA population.",
3.5,
),
"Fig. 4 visualizes the accountant-level clusters": (
EXTRA_FIG_DIR / "accountant_mixture" / "accountant_mixture_2d.png",
"Fig. 4. Accountant-level 3-component Gaussian mixture in the (cosine-mean, dHash-mean) plane.",
4.5,
),
"conducted an ablation study comparing three": (
FIG_DIR / "fig4_ablation.png",
"Fig. 5. Ablation study comparing three feature extraction backbones.",
6.5,
),
}
def strip_comments(text):
return re.sub(r"<!--.*?-->", "", text, flags=re.DOTALL)
def add_md_table(doc, table_lines):
rows_data = []
for line in table_lines:
cells = [c.strip() for c in line.strip("|").split("|")]
if not re.match(r"^[-: ]+$", cells[0]):
rows_data.append(cells)
if len(rows_data) < 2:
return
ncols = len(rows_data[0])
table = doc.add_table(rows=len(rows_data), cols=ncols)
table.style = "Table Grid"
for r_idx, row in enumerate(rows_data):
for c_idx in range(min(len(row), ncols)):
cell = table.rows[r_idx].cells[c_idx]
cell.text = row[c_idx]
for p in cell.paragraphs:
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
for run in p.runs:
run.font.size = Pt(8)
run.font.name = "Times New Roman"
if r_idx == 0:
run.bold = True
doc.add_paragraph()
def _insert_figures(doc, para_text):
for trigger, (fig_path, caption, width) in FIGURES.items():
if trigger in para_text and Path(fig_path).exists():
fp = doc.add_paragraph()
fp.alignment = WD_ALIGN_PARAGRAPH.CENTER
fr = fp.add_run()
fr.add_picture(str(fig_path), width=Inches(width))
cp = doc.add_paragraph()
cp.alignment = WD_ALIGN_PARAGRAPH.CENTER
cr = cp.add_run(caption)
cr.font.size = Pt(9)
cr.font.name = "Times New Roman"
cr.italic = True
def process_section(doc, filepath):
text = filepath.read_text(encoding="utf-8")
text = strip_comments(text)
lines = text.split("\n")
i = 0
while i < len(lines):
line = lines[i]
stripped = line.strip()
if not stripped:
i += 1
continue
if stripped.startswith("# "):
h = doc.add_heading(stripped[2:], level=1)
for run in h.runs:
run.font.color.rgb = RGBColor(0, 0, 0)
i += 1
continue
if stripped.startswith("## "):
h = doc.add_heading(stripped[3:], level=2)
for run in h.runs:
run.font.color.rgb = RGBColor(0, 0, 0)
i += 1
continue
if stripped.startswith("### "):
h = doc.add_heading(stripped[4:], level=3)
for run in h.runs:
run.font.color.rgb = RGBColor(0, 0, 0)
i += 1
continue
if "|" in stripped and i + 1 < len(lines) and re.match(r"\s*\|[-|: ]+\|", lines[i + 1]):
table_lines = []
while i < len(lines) and "|" in lines[i]:
table_lines.append(lines[i])
i += 1
add_md_table(doc, table_lines)
continue
if re.match(r"^\d+\.\s", stripped):
p = doc.add_paragraph(style="List Number")
content = re.sub(r"^\d+\.\s", "", stripped)
content = re.sub(r"\*\*(.+?)\*\*", r"\1", content)
run = p.add_run(content)
run.font.size = Pt(10)
run.font.name = "Times New Roman"
i += 1
continue
if stripped.startswith("- "):
p = doc.add_paragraph(style="List Bullet")
content = stripped[2:]
content = re.sub(r"\*\*(.+?)\*\*", r"\1", content)
run = p.add_run(content)
run.font.size = Pt(10)
run.font.name = "Times New Roman"
i += 1
continue
# Regular paragraph
para_lines = [stripped]
i += 1
while i < len(lines):
nxt = lines[i].strip()
if (
not nxt
or nxt.startswith("#")
or nxt.startswith("|")
or nxt.startswith("- ")
or re.match(r"^\d+\.\s", nxt)
):
break
para_lines.append(nxt)
i += 1
para_text = " ".join(para_lines)
para_text = re.sub(r"\*\*\*(.+?)\*\*\*", r"\1", para_text)
para_text = re.sub(r"\*\*(.+?)\*\*", r"\1", para_text)
para_text = re.sub(r"\*(.+?)\*", r"\1", para_text)
para_text = re.sub(r"`(.+?)`", r"\1", para_text)
para_text = para_text.replace("$$", "")
para_text = para_text.replace("---", "\u2014")
p = doc.add_paragraph()
p.paragraph_format.space_after = Pt(6)
run = p.add_run(para_text)
run.font.size = Pt(10)
run.font.name = "Times New Roman"
_insert_figures(doc, para_text)
def main():
doc = Document()
style = doc.styles["Normal"]
style.font.name = "Times New Roman"
style.font.size = Pt(10)
# Title page
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
p.paragraph_format.space_after = Pt(12)
run = p.add_run(
"Automated Identification of Non-Hand-Signed Auditor Signatures\n"
"in Large-Scale Financial Audit Reports:\n"
"A Dual-Descriptor Framework with Three-Method Convergent Thresholding"
)
run.font.size = Pt(16)
run.font.name = "Times New Roman"
run.bold = True
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
p.paragraph_format.space_after = Pt(6)
run = p.add_run("[Authors removed for double-blind review]")
run.font.size = Pt(10)
run.italic = True
p = doc.add_paragraph()
p.alignment = WD_ALIGN_PARAGRAPH.CENTER
p.paragraph_format.space_after = Pt(20)
run = p.add_run("Target journal: IEEE Access (Regular Paper)")
run.font.size = Pt(10)
run.italic = True
for section_file in SECTIONS:
filepath = PAPER_DIR / section_file
if filepath.exists():
process_section(doc, filepath)
else:
print(f"WARNING: missing section file: {filepath}")
doc.save(str(OUTPUT))
print(f"Saved: {OUTPUT}")
if __name__ == "__main__":
main()
paper/paper_a_abstract_v3.md
+16
View File
@@ -0,0 +1,16 @@
# Abstract
<!-- 200-270 words -->
Regulations in many jurisdictions require Certified Public Accountants (CPAs) to attest to each audit report they certify, typically by affixing a signature or seal.
However, the digitization of financial reporting makes it straightforward to reuse a stored signature image across multiple reports---whether by administrative stamping or firm-level electronic signing systems---potentially undermining the intent of individualized attestation.
Unlike signature forgery, where an impostor imitates another person's handwriting, *non-hand-signed* reproduction involves the legitimate signer's own stored signature image being reproduced on each report, a practice that is visually invisible to report users and infeasible to audit at scale through manual inspection.
We present an end-to-end AI pipeline that automatically detects non-hand-signed auditor signatures in financial audit reports.
The pipeline integrates a Vision-Language Model for signature page identification, YOLOv11 for signature region detection, and ResNet-50 for deep feature extraction, followed by a dual-descriptor verification combining cosine similarity of deep embeddings with difference hashing (dHash).
For threshold determination we apply three methodologically distinct methods---Kernel Density antimode with a Hartigan unimodality test, Burgstahler-Dichev/McCrary discontinuity, and EM-fitted Beta mixtures with a logit-Gaussian robustness check---at both the signature level and the accountant level.
Applied to 90,282 audit reports filed in Taiwan over 2013--2023 (182,328 signatures from 758 CPAs) the methods reveal an informative asymmetry: signature-level similarity forms a continuous quality spectrum that no two-component mixture cleanly separates, while accountant-level aggregates are clustered into three recognizable groups (BIC-best $K = 3$) with the KDE antimode and the two mixture-based estimators converging within $\sim$0.006 of each other at cosine $\approx 0.975$; the Burgstahler-Dichev / McCrary test produces no significant discontinuity at the accountant level, consistent with clustered-but-smooth rather than sharply discrete accountant-level heterogeneity.
A major Big-4 firm is used as a *replication-dominated* (not pure) calibration anchor, with visual-inspection and accountant-level mixture evidence supporting majority non-hand-signing and a minority of hand-signers; we break the circularity of using the same firm for calibration and validation by a 70/30 CPA-level held-out fold.
Validation against 310 byte-identical positive signatures and a $\sim$50,000-pair inter-CPA negative anchor yields FAR $\leq$ 0.001 with Wilson 95% confidence intervals at all accountant-level thresholds.
To our knowledge, this represents the largest-scale forensic analysis of auditor signature authenticity reported in the literature.
<!-- Word count: ~290 -->
paper/paper_a_conclusion_v3.md
+32
View File
@@ -0,0 +1,32 @@
# VI. Conclusion and Future Work
## Conclusion
We have presented an end-to-end AI pipeline for detecting non-hand-signed auditor signatures in financial audit reports at scale.
Applied to 90,282 audit reports from Taiwanese publicly listed companies spanning 2013--2023, our system extracted and analyzed 182,328 CPA signatures using a combination of VLM-based page identification, YOLO-based signature detection, deep feature extraction, and dual-descriptor similarity verification, with threshold selection placed on a statistically principled footing through three methodologically distinct methods applied at two analysis levels.
Our contributions are fourfold.
First, we argued that non-hand-signing detection is a distinct problem from signature forgery detection, requiring analytical tools focused on the upper tail of intra-signer similarity rather than inter-signer discriminability.
Second, we showed that combining cosine similarity of deep embeddings with difference hashing is essential for meaningful classification---among 71,656 documents with high feature-level similarity, the dual-descriptor framework revealed that only 41% exhibit converging structural evidence of non-hand-signing while 7% show no structural corroboration despite near-identical feature-level appearance, demonstrating that a single-descriptor approach conflates style consistency with image reproduction.
Third, we introduced a three-method threshold framework combining KDE antimode (with a Hartigan unimodality test), Burgstahler-Dichev / McCrary discontinuity, and EM-fitted Beta mixture (with a logit-Gaussian robustness check).
Applied at both the signature and accountant levels, this framework surfaced an informative structural asymmetry: at the per-signature level the distribution is a continuous quality spectrum for which no two-mechanism mixture provides a good fit, whereas at the per-accountant level BIC cleanly selects a three-component mixture and the KDE antimode together with the Beta-mixture and logit-Gaussian estimators agree within $\sim 0.006$ at cosine $\approx 0.975$.
The Burgstahler-Dichev / McCrary test, by contrast, finds no significant transition at the accountant level, consistent with clustered but smoothly mixed rather than sharply discrete accountant-level heterogeneity.
The substantive reading is therefore narrower than "discrete behavior": *pixel-level output quality* is continuous and heavy-tailed, and *accountant-level aggregate behavior* is clustered with smooth cluster boundaries.
Fourth, we introduced a *replication-dominated* calibration methodology---explicitly distinguishing replication-dominated from replication-pure calibration anchors and validating classification against a byte-level pixel-identity anchor (310 byte-identical signatures) paired with a $\sim$50,000-pair inter-CPA negative anchor.
To break the circularity of using the calibration firm as its own validation reference, we split the firm's CPAs 70/30 at the CPA level and report post-hoc capture rates on the held-out fold with Wilson 95% confidence intervals.
This framing is internally consistent with all available evidence: the visual-inspection observation of pixel-identical signatures across unrelated audit engagements for the majority of calibration-firm partners; the 92.5% / 7.5% split in signature-level cosine thresholds; and, among the 171 calibration-firm CPAs with enough signatures to enter the accountant-level GMM (of 180 in total), the 139 / 32 split between the high-replication and middle-band clusters.
An ablation study comparing ResNet-50, VGG-16 and EfficientNet-B0 confirmed that ResNet-50 offers the best balance of discriminative power, classification stability, and computational efficiency for this task.
## Future Work
Several directions merit further investigation.
Domain-adapted feature extractors, trained or fine-tuned on signature-specific datasets, may improve discriminative performance beyond the transferred ImageNet features used in this study.
Extending the accountant-level analysis to auditor-year units---using the same three-method convergent framework but at finer temporal resolution---could reveal within-accountant transitions between hand-signing and non-hand-signing over the decade.
The pipeline's applicability to other jurisdictions and document types (e.g., corporate filings in other countries, legal documents, medical records) warrants exploration.
The replication-dominated calibration strategy and the pixel-identity anchor technique are both directly generalizable to settings in which (i) a reference subpopulation has a known dominant mechanism and (ii) the target mechanism leaves a byte-level signature in the artifact itself.
Finally, integration with regulatory monitoring systems and a larger negative-anchor study---for example drawing from inter-CPA pairs under explicit accountant-level blocking---would strengthen the practical deployment potential of this approach.
paper/paper_a_discussion_v3.md
+109
View File
@@ -0,0 +1,109 @@
# V. Discussion
## A. Non-Hand-Signing Detection as a Distinct Problem
Our results highlight the importance of distinguishing *non-hand-signing detection* from the well-studied *signature forgery detection* problem.
In forgery detection, the challenge lies in modeling the variability of skilled forgers who produce plausible imitations of a target signature.
In non-hand-signing detection the signer's identity is not in question; the challenge is distinguishing between legitimate intra-signer consistency (a CPA who signs similarly each time) and image-level reproduction of a stored signature (a CPA whose signature on each report is a byte-level or near-byte-level copy of a single source image).
This distinction has direct methodological consequences.
Forgery detection systems optimize for inter-class discriminability---maximizing the gap between genuine and forged signatures.
Non-hand-signing detection, by contrast, requires sensitivity to the *upper tail* of the intra-class similarity distribution, where the boundary between consistent handwriting and image reproduction becomes ambiguous.
The dual-descriptor framework we propose---combining semantic-level features (cosine similarity) with structural-level features (dHash)---addresses this ambiguity in a way that single-descriptor approaches cannot.
## B. Continuous-Quality Spectrum vs. Clustered Accountant-Level Heterogeneity
The most consequential empirical finding of this study is the asymmetry between signature level and accountant level revealed by the three-method framework and the Hartigan dip test (Sections IV-D and IV-E).
At the per-signature level, the distribution of best-match cosine similarity is *not* cleanly bimodal.
Firm A's signature-level cosine is formally unimodal (dip test $p = 0.17$) with a long left tail.
The all-CPA signature-level cosine rejects unimodality ($p < 0.001$), but its structure is not well approximated by a two-component Beta mixture: BIC clearly prefers a three-component fit, and the 2-component Beta crossing and its logit-GMM counterpart disagree sharply on the candidate threshold (0.977 vs. 0.999 for Firm A).
The BD/McCrary discontinuity test locates its transition at 0.985---*inside* the non-hand-signed mode rather than at a boundary between two mechanisms.
Taken together, these results indicate that non-hand-signed signatures form a continuous quality spectrum rather than a discrete class.
Replication quality varies continuously with scan equipment, PDF compression, stamp pressure, and firm-level e-signing system generation, producing a heavy-tailed distribution that no two-mechanism mixture explains at the signature level.
At the per-accountant aggregate level the picture partly reverses.
The distribution of per-accountant mean cosine (and mean dHash) rejects unimodality, a BIC-selected three-component Gaussian mixture cleanly separates (C1) a high-replication cluster dominated by Firm A, (C2) a middle band shared by the other Big-4 firms, and (C3) a hand-signed-tendency cluster dominated by smaller domestic firms, and the three 1D threshold methods applied at the accountant level produce mutually consistent estimates (KDE antimode $= 0.973$, Beta-2 crossing $= 0.979$, logit-GMM-2 crossing $= 0.976$).
The BD/McCrary test, however, does not produce a significant transition at the accountant level either, in contrast to the signature level.
This pattern is consistent with a clustered *but smoothly mixed* accountant-level distribution rather than with a sharp density discontinuity: accountant-level means cluster into three recognizable groups, yet the transitions between them are gradual rather than discrete at the bin resolution BD/McCrary requires.
The substantive interpretation we take from this evidence is therefore narrower than a "discrete-behaviour" claim: *pixel-level output quality* is continuous and heavy-tailed, and *accountant-level aggregate behaviour* is clustered (three recognizable groups) but not sharply discrete.
The accountant-level mixture is a useful classifier of firm-and-practitioner-level signing regimes; individual behaviour may still transition or mix over time within a practitioner, and our cross-sectional analysis does not rule this out.
Methodologically, the implication is that the three 1D methods are meaningfully applied at the accountant level for threshold estimation, while the BD/McCrary non-transition at the same level is itself diagnostic of smoothness rather than a failure of the method.
## C. Firm A as a Replication-Dominated, Not Pure, Population
A recurring theme in prior work that treats Firm A or an analogous reference group as a calibration anchor is the implicit assumption that the anchor is a pure positive class.
Our evidence across multiple analyses rules out that assumption for Firm A while affirming its utility as a calibration reference.
Three convergent strands of evidence support the replication-dominated framing.
First, the visual-inspection evidence: randomly sampled Firm A reports exhibit pixel-identical signature images across different audit engagements and fiscal years for the majority of partners---a physical impossibility under independent hand-signing events.
Second, the signature-level statistical evidence: Firm A's per-signature cosine distribution is unimodal long-tail rather than a tight single peak; 92.5% of Firm A signatures exceed cosine 0.95, with the remaining 7.5% forming the left tail.
Third, the accountant-level evidence: of the 171 Firm A CPAs with enough signatures ($\geq 10$) to enter the accountant-level GMM, 32 (19%) fall into the middle-band C2 cluster rather than the high-replication C1 cluster---directly quantifying the within-firm minority of hand-signers.
Nine additional Firm A CPAs are excluded from the GMM for having fewer than 10 signatures, so we cannot place them in a cluster from the cross-sectional analysis alone.
The held-out Firm A 70/30 validation (Section IV-G.2) gives capture rates on a non-calibration Firm A subset that sit in the same replication-dominated regime as the calibration fold across the full range of operating rules (extreme rules are statistically indistinguishable; operational rules in the 85–95% band differ between folds by 1–5 percentage points, reflecting within-Firm-A heterogeneity in replication intensity rather than a generalization failure).
The accountant-level GMM (Section IV-E) and the threshold-independent partner-ranking analysis (Section IV-H.2) are the cross-checks that are robust to fold-level sampling variance.
The replication-dominated framing is internally coherent with all three pieces of evidence, and it predicts and explains the residuals that a "near-universal" framing would be forced to treat as noise.
We therefore recommend that future work building on this calibration strategy should explicitly distinguish replication-dominated from replication-pure calibration anchors.
## D. The Style-Replication Gap
Within the 71,656 documents exceeding cosine $0.95$, the dHash descriptor partitions them into three distinct populations: 29,529 (41.2%) with high-confidence structural evidence of non-hand-signing, 36,994 (51.7%) with moderate structural similarity, and 5,133 (7.2%) with no structural corroboration despite near-identical feature-level appearance.
A cosine-only classifier would treat all 71,656 documents identically; the dual-descriptor framework separates them into populations with fundamentally different interpretations.
The 7.2% classified as "high style consistency" (cosine $> 0.95$ but dHash $> 15$) are particularly informative.
Several plausible explanations may account for their high feature similarity without structural identity, though we lack direct evidence to confirm their relative contributions.
Many accountants may develop highly consistent signing habits---using similar pen pressure, stroke order, and spatial layout---resulting in signatures that appear nearly identical at the semantic feature level while retaining the microscopic variations inherent to handwriting.
Some may use signing pads or templates that further constrain variability without constituting image-level reproduction.
The dual-descriptor framework correctly identifies these cases as distinct from non-hand-signed signatures by detecting the absence of structural-level convergence.
## E. Value of a Replication-Dominated Calibration Group
The use of Firm A as a calibration reference addresses a fundamental challenge in document forensics: the scarcity of ground truth labels.
In most forensic applications, establishing ground truth requires expensive manual verification or access to privileged information about document provenance.
Our approach leverages domain knowledge---the established prevalence of non-hand-signing at a specific firm---to create a naturally occurring reference population within the dataset.
This calibration strategy has broader applicability beyond signature analysis.
Any forensic detection system operating on real-world corpora can benefit from identifying subpopulations with known dominant characteristics (positive or negative) to anchor threshold selection, particularly when the distributions of interest are non-normal and non-parametric or mixture-based thresholds are preferred over parametric alternatives.
The framing we adopt---replication-dominated rather than replication-pure---is an important refinement of this strategy: it prevents overclaim, accommodates the within-firm heterogeneity quantified by the accountant-level mixture, and yields classification rates that are internally consistent with the data.
## F. Pixel-Identity and Inter-CPA Anchors as Annotation-Free Validation
A further methodological contribution is the combination of byte-level pixel identity as an annotation-free *conservative* gold positive and a large random-inter-CPA negative anchor.
Handwriting physics makes byte-identity impossible under independent signing events, so any pair of same-CPA signatures that are byte-identical after crop and normalization is an absolute positive for non-hand-signing, requiring no human review.
In our corpus 310 signatures satisfied this condition.
We emphasize that byte-identical pairs are a *subset* of the true non-hand-signed positive class---they capture only those whose nearest same-CPA match happens to be bytewise identical, excluding replications that are pixel-near-identical but not byte-identical (for example, under different scan or compression pathways).
Perfect recall against this subset therefore does not generalize to perfect recall against the full non-hand-signed population; it is a lower-bound calibration check on the classifier's ability to catch the clearest positives rather than a generalizable recall estimate.
Paired with the $\sim$50,000-pair inter-CPA negative anchor, the byte-identical positives yield FAR estimates with tight Wilson 95% confidence intervals (Table X), which is a substantive improvement over the low-similarity same-CPA negative ($n = 35$) we originally considered.
The combination is a reusable pattern for other document-forensics settings in which the target mechanism leaves a byte-level physical signature in the artifact itself, provided that its generalization limits are acknowledged: FAR is informative, whereas recall is valid only for the conservative subset.
## G. Limitations
Several limitations should be acknowledged.
First, comprehensive per-document ground truth labels are not available.
The pixel-identity anchor is a strict *subset* of the true non-hand-signing positives (only those whose nearest same-CPA match happens to be byte-identical), so perfect recall against this anchor does not establish the classifier's recall on the broader positive class.
The low-similarity same-CPA anchor ($n = 35$) is small because intra-CPA pairs rarely fall below cosine 0.70; we use the $\sim$50,000-pair inter-CPA negative anchor as the primary negative reference, which yields tight Wilson 95% CIs on FAR (Table X), but it too does not exhaust the set of true negatives (in particular, same-CPA hand-signed pairs with moderate cosine similarity are not sampled).
A manual-adjudication study concentrated at the decision boundary---for example 100--300 auditor-years stratified by cosine band---would further strengthen the recall estimate against the full positive class.
Second, the ResNet-50 feature extractor was used with pre-trained ImageNet weights without domain-specific fine-tuning.
While our ablation study and prior literature [20]--[22] support the effectiveness of transferred ImageNet features for signature comparison, a signature-specific feature extractor could improve discriminative performance.
Third, the red stamp removal preprocessing uses simple HSV color-space filtering, which may introduce artifacts where handwritten strokes overlap with red seal impressions.
In these overlap regions, blended pixels are replaced with white, potentially creating small gaps in the signature strokes that could reduce dHash similarity.
This effect would bias classification toward false negatives rather than false positives, but the magnitude has not been quantified.
Fourth, scanning equipment, PDF generation software, and compression algorithms may have changed over the 10-year study period (2013--2023), potentially affecting similarity measurements.
While cosine similarity and dHash are designed to be robust to such variations, longitudinal confounds cannot be entirely excluded, and we note that our accountant-level aggregates could mask within-accountant temporal transitions.
Fifth, the classification framework treats all signatures from a CPA as belonging to a single class, not accounting for potential changes in signing practice over time (e.g., a CPA who signed genuinely in early years but adopted non-hand-signing later).
Extending the accountant-level analysis to auditor-year units is a natural next step.
Sixth, the BD/McCrary transition estimates fall inside rather than between modes for the per-signature cosine distribution, and the test produces no significant transition at all at the accountant level.
In our application, therefore, BD/McCrary contributes diagnostic information about local density-smoothness rather than an independent accountant-level threshold estimate; that role is played by the KDE antimode and the two mixture-based estimators.
The BD/McCrary results remain informative as a robustness check---their non-transition at the accountant level is consistent with the dip-test and Beta-mixture evidence that accountant-level clustering is smooth rather than sharply discontinuous.
Finally, the legal and regulatory implications of our findings depend on jurisdictional definitions of "signature" and "signing."
Whether non-hand-signing of a CPA's own stored signature constitutes a violation of signing requirements is a legal question that our technical analysis can inform but cannot resolve.
paper/paper_a_impact_statement_v3.md
+9
View File
@@ -0,0 +1,9 @@
# Impact Statement
<!-- 100-150 words. Non-specialist readable. No jargon. Specific, not vague. -->
Auditor signatures on financial reports are a key safeguard of corporate accountability.
When the signature on an audit report is produced by reproducing a stored image instead of by the partner's own hand---whether through an administrative stamping workflow or a firm-level electronic signing system---this safeguard is weakened, yet detecting the practice through manual inspection is infeasible at the scale of modern financial markets.
We developed an artificial intelligence system that automatically extracts and analyzes signatures from over 90,000 audit reports spanning a decade of filings by publicly listed companies in Taiwan.
By combining deep-learning visual features with perceptual hashing and three methodologically distinct threshold-selection methods, the system distinguishes genuinely hand-signed signatures from reproduced ones and quantifies how this practice varies across firms and over time.
After further validation, the technology could support financial regulators in automating signature-authenticity screening at national scale.
paper/paper_a_introduction_v3.md
+86
View File
@@ -0,0 +1,86 @@
# I. Introduction
<!-- Target: ~1.5 pages double-column IEEE format. Double-blind: no author/institution info. -->
Financial audit reports serve as a critical mechanism for ensuring corporate accountability and investor protection.
In Taiwan, the Certified Public Accountant Act (會計師法 §4) and the Financial Supervisory Commission's attestation regulations (查核簽證核准準則 §6) require that certifying CPAs affix their signature or seal (簽名或蓋章) to each audit report [1].
While the law permits either a handwritten signature or a seal, the CPA's attestation on each report is intended to represent a deliberate, individual act of professional endorsement for that specific audit engagement [2].
The digitization of financial reporting has introduced a practice that complicates this intent.
As audit reports are now routinely generated, transmitted, and archived as PDF documents, it is technically and operationally straightforward to reproduce a CPA's stored signature image across many reports rather than re-executing the signing act for each one.
This reproduction can occur either through an administrative stamping workflow---in which scanned signature images are affixed by staff as part of the report-assembly process---or through a firm-level electronic signing system that automates the same step.
From the perspective of the output image the two workflows are equivalent: both yield a pixel-level reproduction of a single stored image on every report the partner signs off, so that signatures on different reports of the same partner are identical up to reproduction noise.
We refer to signatures produced by either workflow collectively as *non-hand-signed*.
Although this practice may fall within the literal statutory requirement of "signature or seal," it raises substantive concerns about audit quality, as an identically reproduced signature applied across hundreds of reports may not represent meaningful individual attestation for each engagement.
Unlike traditional signature forgery, where a third party attempts to imitate another person's handwriting, non-hand-signing involves the legitimate signer's own stored signature being reused.
This practice, while potentially widespread, is visually invisible to report users and virtually undetectable through manual inspection at scale: regulatory agencies overseeing thousands of publicly listed companies cannot feasibly examine each signature for evidence of image reproduction.
The distinction between *non-hand-signing detection* and *signature forgery detection* is both conceptually and technically important.
The extensive body of research on offline signature verification [3]--[8] has focused almost exclusively on forgery detection---determining whether a questioned signature was produced by its purported author or by an impostor.
This framing presupposes that the central threat is identity fraud.
In our context, identity is not in question; the CPA is indeed the legitimate signer.
The question is whether the physical act of signing occurred for each individual report, or whether a single signing event was reproduced as an image across many reports.
This detection problem differs fundamentally from forgery detection: while it does not require modeling skilled-forger variability, it introduces the distinct challenge of separating legitimate intra-signer consistency from image-level reproduction, requiring an analytical framework focused on detecting abnormally high similarity across documents.
A secondary methodological concern shapes the research design.
Many prior similarity-based classification studies rely on ad-hoc thresholds---declaring two images equivalent above a hand-picked cosine cutoff, for example---without principled statistical justification.
Such thresholds are fragile and invite reviewer skepticism, particularly in an archival-data setting where the cost of misclassification propagates into downstream inference.
A defensible approach requires (i) a statistically principled threshold-determination procedure, ideally anchored to an empirical reference population drawn from the target corpus; (ii) convergent validation across multiple threshold-determination methods that rest on different distributional assumptions; and (iii) external validation against anchor populations with known ground-truth characteristics using precision, recall, $F_1$, and equal-error-rate metrics that prevail in the biometric-verification literature.
Despite the significance of the problem for audit quality and regulatory oversight, no prior work has specifically addressed non-hand-signing detection in financial audit documents at scale with these methodological safeguards.
Woodruff et al. [9] developed an automated pipeline for signature analysis in corporate filings for anti-money-laundering investigations, but their work focused on author clustering (grouping signatures by signer identity) rather than detecting reuse of a stored image.
Copy-move forgery detection methods [10], [11] address duplicated regions within or across images but are designed for natural images and do not account for the specific characteristics of scanned document signatures, where legitimate visual similarity between a signer's authentic signatures is expected and must be distinguished from image reproduction.
Research on near-duplicate image detection using perceptual hashing combined with deep learning [12], [13] provides relevant methodological foundations but has not been applied to document forensics or signature analysis.
From the statistical side, the methods we adopt for threshold determination---the Hartigan dip test [37], Burgstahler-Dichev / McCrary discontinuity testing [38], [39], and finite mixture modelling via the EM algorithm [40], [41]---have been developed in statistics and accounting-econometrics but have not, to our knowledge, been combined as a three-method convergent framework for document-forensics threshold selection.
In this paper, we present a fully automated, end-to-end pipeline for detecting non-hand-signed CPA signatures in audit reports at scale.
Our approach processes raw PDF documents through the following stages:
(1) signature page identification using a Vision-Language Model (VLM);
(2) signature region detection using a trained YOLOv11 object detector;
(3) deep feature extraction via a pre-trained ResNet-50 convolutional neural network;
(4) dual-descriptor similarity computation combining cosine similarity on deep embeddings with difference hash (dHash) distance;
(5) threshold determination using three methodologically distinct methods---KDE antimode with a Hartigan unimodality test, Burgstahler-Dichev / McCrary discontinuity, and finite Beta mixture via EM with a logit-Gaussian robustness check---applied at both the signature level and the accountant level; and
(6) validation against a pixel-identical anchor, a low-similarity anchor, and a replication-dominated Big-4 calibration firm.
The dual-descriptor verification is central to our contribution.
Cosine similarity of deep feature embeddings captures high-level visual style similarity---it can identify signatures that share similar stroke patterns and spatial layouts---but cannot distinguish between a CPA who signs consistently and one whose signature is reproduced from a stored image.
Perceptual hashing (specifically, difference hashing) encodes structural-level image gradients into compact binary fingerprints that are robust to scan noise but sensitive to substantive content differences.
By requiring convergent evidence from both descriptors, we can differentiate *style consistency* (high cosine but divergent dHash) from *image reproduction* (high cosine with low dHash), resolving an ambiguity that neither descriptor can address alone.
A second distinctive feature is our framing of the calibration reference.
One major Big-4 accounting firm in Taiwan (hereafter "Firm A") is widely recognized within the audit profession as making substantial use of non-hand-signing for the majority of its certifying partners, while not ruling out that a minority may continue to hand-sign some reports.
We therefore treat Firm A as a *replication-dominated* calibration reference rather than a pure positive class.
This framing is important because the statistical signature of a replication-dominated population is visible in our data: Firm A's per-signature cosine distribution is unimodal with a long left tail, 92.5% of Firm A signatures exceed cosine 0.95 but 7.5% fall below, and 32 of the 171 Firm A CPAs with enough signatures to enter our accountant-level analysis (of 180 Firm A CPAs in total) cluster into an accountant-level "middle band" rather than the high-replication mode.
Adopting the replication-dominated framing---rather than a near-universal framing that would have to absorb these residuals as noise---ensures internal coherence among the visual-inspection evidence, the signature-level statistics, and the accountant-level mixture.
A third distinctive feature is our unit-of-analysis treatment.
Our three-method framework reveals an informative asymmetry between the signature level and the accountant level: per-signature similarity forms a continuous quality spectrum for which no two-mechanism mixture provides a good fit, whereas per-accountant aggregates are clustered into three recognizable groups (BIC-best $K = 3$).
The substantive reading is that *pixel-level output quality* is a continuous spectrum shaped by firm-specific reproduction technologies and scan conditions, while *accountant-level aggregate behaviour* is clustered but not sharply discrete---a given CPA tends to cluster into a dominant regime (high-replication, middle-band, or hand-signed-tendency), though the boundaries between regimes are smooth rather than discontinuous.
Among the three accountant-level methods, KDE antimode and the two mixture-based estimators converge within $\sim 0.006$ on a cosine threshold of approximately $0.975$, while the Burgstahler-Dichev / McCrary discontinuity test finds no significant transition at the accountant level---an outcome consistent with smoothly mixed clusters rather than a failure of the method.
The two-dimensional GMM marginal crossings (cosine $= 0.945$, dHash $= 8.10$) are reported as a complementary cross-check rather than as the primary accountant-level threshold.
We apply this pipeline to 90,282 audit reports filed by publicly listed companies in Taiwan between 2013 and 2023, extracting and analyzing 182,328 individual CPA signatures from 758 unique accountants.
To our knowledge, this represents the largest-scale forensic analysis of signature authenticity in financial documents reported in the literature.
The contributions of this paper are summarized as follows:
1. **Problem formulation.** We formally define non-hand-signing detection as distinct from signature forgery detection and argue that it requires an analytical framework focused on intra-signer similarity distributions rather than genuine-versus-forged classification.
2. **End-to-end pipeline.** We present a pipeline that processes raw PDF audit reports through VLM-based page identification, YOLO-based signature detection, deep feature extraction, and dual-descriptor similarity computation, with automated inference requiring no manual intervention after initial training and annotation.
3. **Dual-descriptor verification.** We demonstrate that combining deep-feature cosine similarity with perceptual hashing resolves the fundamental ambiguity between style consistency and image reproduction, and we validate the backbone choice through an ablation study comparing three feature-extraction architectures.
4. **Three-method convergent threshold framework.** We introduce a threshold-selection framework that applies three methodologically distinct methods---KDE antimode with Hartigan unimodality test, Burgstahler-Dichev / McCrary discontinuity, and EM-fitted Beta mixture with a logit-Gaussian robustness check---at both the signature and accountant levels, using the convergence (or principled divergence) across methods as diagnostic evidence about the mixture structure of the data.
5. **Continuous-quality / clustered-accountant finding.** We empirically establish that per-signature similarity is a continuous quality spectrum poorly approximated by any two-mechanism mixture, whereas per-accountant aggregates cluster into three recognizable groups with smoothly mixed rather than sharply discrete boundaries---an asymmetry with direct implications for how threshold selection and mixture modelling should be applied in document forensics.
6. **Replication-dominated calibration methodology.** We introduce a calibration strategy using a known-majority-positive reference group, distinguishing *replication-dominated* from *replication-pure* anchors; and we validate classification using byte-level pixel identity as an annotation-free gold positive, requiring no manual labeling.
7. **Large-scale empirical analysis.** We report findings from the analysis of over 90,000 audit reports spanning a decade, providing the first large-scale empirical evidence on non-hand-signing practices in financial reporting under a methodology designed for peer-review defensibility.
The remainder of this paper is organized as follows.
Section II reviews related work on signature verification, document forensics, perceptual hashing, and the statistical methods we adopt for threshold determination.
Section III describes the proposed methodology.
Section IV presents experimental results including the three-method convergent threshold analysis, accountant-level mixture, pixel-identity validation, and backbone ablation study.
Section V discusses the implications and limitations of our findings.
Section VI concludes with directions for future work.
paper/paper_a_methodology_v3.md
+289
View File
@@ -0,0 +1,289 @@
# III. Methodology
## A. Pipeline Overview
We propose a six-stage pipeline for large-scale non-hand-signed auditor signature detection in scanned financial documents.
Fig. 1 illustrates the overall architecture.
The pipeline takes as input a corpus of PDF audit reports and produces, for each document, a classification of its CPA signatures along a confidence continuum supported by convergent evidence from three methodologically distinct statistical methods and a pixel-identity anchor.
Throughout this paper we use the term *non-hand-signed* rather than "digitally replicated" to denote any signature produced by reproducing a previously stored image of the partner's signature---whether by administrative stamping workflows (dominant in the early years of the sample) or firm-level electronic signing systems (dominant in the later years).
From the perspective of the output image the two workflows are equivalent: both reproduce a single stored image so that signatures on different reports from the same partner are identical up to reproduction noise.
<!--
[Figure 1: Pipeline Architecture - clean vector diagram]
90,282 PDFs → VLM Pre-screening → 86,072 PDFs
→ YOLOv11 Detection → 182,328 signatures
→ ResNet-50 Features → 2048-dim embeddings
→ Dual-Method Verification (Cosine + dHash)
→ Three-Method Threshold (KDE / BD-McCrary / Beta mixture) → Classification
→ Pixel-identity + Firm A + Accountant-level GMM validation
-->
## B. Data Collection
The dataset comprises 90,282 annual financial audit reports filed by publicly listed companies in Taiwan, covering fiscal years 2013 to 2023.
The reports were collected from the Market Observation Post System (MOPS) operated by the Taiwan Stock Exchange Corporation, the official repository for mandatory corporate filings.
An automated web-scraping pipeline using Selenium WebDriver was developed to systematically download all audit reports for each listed company across the study period.
Each report is a multi-page PDF document containing, among other content, the auditor's report page bearing the signatures of the certifying CPAs.
CPA names, affiliated accounting firms, and audit engagement tenure were obtained from a publicly available audit-firm tenure registry encompassing 758 unique CPAs across 15 document types, with the majority (86.4%) being standard audit reports.
Table I summarizes the dataset composition.
<!-- TABLE I: Dataset Summary
| Attribute | Value |
|-----------|-------|
| Total PDF documents | 90,282 |
| Date range | 2013–2023 |
| Documents with signatures | 86,072 (95.4%) |
| Unique CPAs identified | 758 |
| Accounting firms | >50 |
-->
## C. Signature Page Identification
To identify which page of each multi-page PDF contains the auditor's signatures, we employed the Qwen2.5-VL vision-language model (32B parameters) [24] as an automated pre-screening mechanism.
Each PDF page was rendered to JPEG at 180 DPI and submitted to the VLM with a structured prompt requesting a binary determination of whether the page contains a Chinese handwritten signature.
The model was configured with temperature 0 for deterministic output.
The scanning range was restricted to the first quartile of each document's page count, reflecting the regulatory structure of Taiwanese audit reports in which the auditor's report page is consistently located in the first quarter of the document.
Scanning terminated upon the first positive detection.
This process identified 86,072 documents with signature pages; the remaining 4,198 documents (4.6%) were classified as having no signatures and excluded.
An additional 12 corrupted PDFs were excluded, yielding a final set of 86,071 documents.
Cross-validation between the VLM and subsequent YOLO detection confirmed high agreement: YOLO successfully detected signature regions in 98.8% of VLM-positive documents.
The 1.2% disagreement reflects the combined rate of (i) VLM false positives (pages incorrectly flagged as containing signatures) and (ii) YOLO false negatives (signature regions missed by the detector), and we do not attempt to attribute the residual to either source without further labeling.
## D. Signature Detection
We adopted YOLOv11n (nano variant) [25] for signature region localization.
A training set of 500 randomly sampled signature pages was annotated using a custom web-based interface following a two-stage protocol: primary annotation followed by independent review and correction.
A region was labeled as "signature" if it contained any Chinese handwritten content attributable to a personal signature, regardless of overlap with official stamps.
The model was trained for 100 epochs on a 425/75 training/validation split with COCO pre-trained initialization, achieving strong detection performance (Table II).
<!-- TABLE II: YOLO Detection Performance
| Metric | Value |
|--------|-------|
| Precision | 0.97–0.98 |
| Recall | 0.95–0.98 |
| mAP@0.50 | 0.98–0.99 |
| mAP@0.50:0.95 | 0.85–0.90 |
-->
Batch inference on all 86,071 documents extracted 182,328 signature images at a rate of 43.1 documents per second (8 workers).
A red stamp removal step was applied to each cropped signature using HSV color-space filtering, replacing detected red regions with white pixels to isolate the handwritten content.
Each signature was matched to its corresponding CPA using positional order (first or second signature on the page) against the official CPA registry, achieving a 92.6% match rate (168,755 of 182,328 signatures).
## E. Feature Extraction
Each extracted signature was encoded into a feature vector using a pre-trained ResNet-50 convolutional neural network [26] with ImageNet-1K V2 weights, used as a fixed feature extractor without fine-tuning.
The final classification layer was removed, yielding the 2048-dimensional output of the global average pooling layer.
Preprocessing consisted of resizing to 224×224 pixels with aspect-ratio preservation and white padding, followed by ImageNet channel normalization.
All feature vectors were L2-normalized, ensuring that cosine similarity equals the dot product.
The choice of ResNet-50 without fine-tuning was motivated by three considerations: (1) the task is similarity comparison rather than classification, making general-purpose discriminative features sufficient; (2) ImageNet features have been shown to transfer effectively to document analysis tasks [20], [21]; and (3) avoiding domain-specific fine-tuning reduces the risk of overfitting to dataset-specific artifacts, though we note that a fine-tuned model could potentially improve discriminative performance (see Section V-D).
This design choice is validated by an ablation study (Section IV-F) comparing ResNet-50 against VGG-16 and EfficientNet-B0.
## F. Dual-Method Similarity Descriptors
For each signature, we compute two complementary similarity measures against other signatures attributed to the same CPA:
**Cosine similarity on deep embeddings** captures high-level visual style:
$$\text{sim}(\mathbf{f}_A, \mathbf{f}_B) = \mathbf{f}_A \cdot \mathbf{f}_B$$
where $\mathbf{f}_A$ and $\mathbf{f}_B$ are L2-normalized 2048-dim feature vectors.
Each feature dimension contributes to the angular alignment, so cosine similarity is sensitive to fine-grained execution differences---pen pressure, ink distribution, and subtle stroke-trajectory variations---that distinguish genuine within-writer variation from the reproduction of a stored image [14].
**Perceptual hash distance (dHash)** captures structural-level similarity.
Each signature image is resized to 9×8 pixels and converted to grayscale; horizontal gradient differences between adjacent columns produce a 64-bit binary fingerprint.
The Hamming distance between two fingerprints quantifies perceptual dissimilarity: a distance of 0 indicates structurally identical images, while distances exceeding 15 indicate clearly different images.
Unlike DCT-based perceptual hashes, dHash is computationally lightweight and particularly effective for detecting near-exact duplicates with minor scan-induced variations [19].
These descriptors provide partially independent evidence.
Cosine similarity is sensitive to the full feature distribution and reflects fine-grained execution variation; dHash captures only coarse perceptual structure and is robust to scanner-induced noise.
Non-hand-signing yields extreme similarity under *both* descriptors, since the underlying image is identical up to reproduction noise.
Hand-signing, by contrast, yields high dHash similarity (the overall layout of a signature is preserved across writing occasions) but measurably lower cosine similarity (fine execution varies).
Convergence of the two descriptors is therefore a natural robustness check; when they disagree, the case is flagged as borderline.
We specifically excluded SSIM (Structural Similarity Index) [30] after empirical testing showed it to be unreliable for scanned documents: the calibration firm (Section III-H) exhibited a mean SSIM of only 0.70 due to scan-induced pixel-level variations, despite near-identical visual content.
Cosine similarity and dHash are both robust to the noise introduced by the print-scan cycle.
## G. Unit of Analysis and Summary Statistics
Two unit-of-analysis choices are relevant for this study: (i) the *signature*---one signature image extracted from one report---and (ii) the *accountant*---the collection of all signatures attributed to a single CPA across the sample period.
A third composite unit---the *auditor-year*, i.e. all signatures by one CPA within one fiscal year---is also natural when longitudinal behavior is of interest, and we treat auditor-year analyses as a direct extension of accountant-level analysis at finer temporal resolution.
For per-signature classification we compute, for each signature, the maximum pairwise cosine similarity and the minimum dHash Hamming distance against every other signature attributed to the same CPA.
The max/min (rather than mean) formulation reflects the identification logic for non-hand-signing: if even one other signature of the same CPA is a pixel-level reproduction, that pair will dominate the extremes and reveal the non-hand-signed mechanism.
Mean statistics would dilute this signal.
We also adopt an explicit *within-auditor-year no-mixing* identification assumption.
Specifically, within any single fiscal year we treat a given CPA's signing mechanism as uniform: a CPA who reproduces one signature image in that year is assumed to do so for every report, and a CPA who hand-signs in that year is assumed to hand-sign every report in that year.
Domain-knowledge from industry practice at Firm A is consistent with this assumption for that firm during the sample period.
Under the assumption, per-auditor-year summary statistics are well defined and robust to outliers: if even one pair of same-CPA signatures in the year is near-identical, the max/min captures it.
The intra-report consistency analysis in Section IV-H.3 provides an empirical check on the within-auditor-year assumption at the report level.
For accountant-level analysis we additionally aggregate these per-signature statistics to the CPA level by computing the mean best-match cosine and the mean *independent minimum dHash* across all signatures of that CPA.
The *independent minimum dHash* of a signature is defined as the minimum Hamming distance to *any* other signature of the same CPA (over the full same-CPA set), in contrast to the *cosine-conditional dHash* used as a diagnostic elsewhere, which is the dHash to the single signature selected as the cosine-nearest match.
The independent minimum avoids conditioning on the cosine choice and is therefore the conservative structural-similarity statistic for each signature.
These accountant-level aggregates are the input to the mixture model described in Section III-J and to the accountant-level three-method analysis in Section III-I.5.
## H. Calibration Reference: Firm A as a Replication-Dominated Population
A distinctive aspect of our methodology is the use of Firm A---a major Big-4 accounting firm in Taiwan---as an empirical calibration reference.
Rather than treating Firm A as a synthetic or laboratory positive control, we treat it as a naturally occurring *replication-dominated population*: a CPA population whose aggregate signing behavior is dominated by non-hand-signing but is not a pure positive class.
The background context for this choice is practitioner knowledge about Firm A's signing practice: industry practice at the firm is widely understood among practitioners to involve reproducing a stored signature image for the majority of certifying partners---originally via administrative stamping workflows and later via firm-level electronic signing systems---while not ruling out that a minority of partners may continue to hand-sign some or all of their reports.
We use this only as background context for why Firm A is a plausible calibration candidate; the *evidence* for Firm A's replication-dominated status comes entirely from the paper's own analyses, which do not depend on any claim about signing practice beyond what the audit-report images themselves show.
We establish Firm A's replication-dominated status through four independent quantitative analyses, each of which can be reproduced from the public audit-report corpus alone:
First, *independent visual inspection* of randomly sampled Firm A reports reveals pixel-identical signature images across different audit engagements and fiscal years for the majority of partners---a physical impossibility under independent hand-signing events.
Second, *whole-sample signature-level rates*: 92.5% of Firm A's per-signature best-match cosine similarities exceed 0.95, consistent with non-hand-signing as the dominant mechanism, while the remaining 7.5% form a long left tail consistent with a minority of hand-signers.
Third, *accountant-level mixture analysis* (Section IV-E): a BIC-selected three-component Gaussian mixture over per-accountant mean cosine and mean dHash places 139 of the 171 Firm A CPAs (with $\geq 10$ signatures) in the high-replication C1 cluster and 32 in the middle-band C2 cluster, directly quantifying the within-firm heterogeneity.
Fourth, we additionally validate the Firm A benchmark through three complementary analyses reported in Section IV-H. Two of them are fully threshold-free, and one uses the downstream classifier as an internal consistency check:
(a) *Longitudinal stability (Section IV-H.1).* The share of Firm A per-signature best-match cosine values below 0.95 is stable at 6-13% across 2013-2023, with the lowest share in 2023. The fixed 0.95 cutoff is not calibrated to Firm A; the stability itself is the finding.
(b) *Partner-level similarity ranking (Section IV-H.2).* When every Big-4 auditor-year is ranked globally by its per-auditor-year mean best-match cosine, Firm A auditor-years account for 95.9% of the top decile against a baseline share of 27.8% (a 3.5$\times$ concentration ratio), and this over-representation is stable across 2013-2023. This analysis uses only the ordinal ranking and is independent of any absolute cutoff.
(c) *Intra-report consistency (Section IV-H.3).* Because each Taiwanese statutory audit report is co-signed by two engagement partners, firm-wide stamping practice predicts that both signers on a given Firm A report should receive the same signature-level label under the classifier. Firm A exhibits 89.9% intra-report agreement against 62-67% at the other Big-4 firms. This test uses the calibrated classifier and therefore is a *consistency* check on the classifier's firm-level output rather than a threshold-free test; the cross-firm gap (not the absolute rate) is the substantive finding.
We emphasize that the 92.5% figure is a within-sample consistency check rather than an independent validation of Firm A's status; the validation role is played by the visual inspection, the accountant-level mixture, the three complementary analyses above, and the held-out Firm A fold described in Section III-K.
We emphasize that Firm A's replication-dominated status was *not* derived from the thresholds we calibrate against it.
Its identification rests on visual evidence and accountant-level clustering that is independent of the statistical pipeline.
The "replication-dominated, not pure" framing is important both for internal consistency---it predicts and explains the long left tail observed in Firm A's cosine distribution (Section III-I below)---and for avoiding overclaim in downstream inference.
## I. Three-Method Convergent Threshold Determination
Direct assignment of thresholds based on prior intuition (e.g., cosine $\geq 0.95$ for non-hand-signed) is analytically convenient but methodologically vulnerable: reviewers can reasonably ask why these particular cutoffs rather than others.
To place threshold selection on a statistically principled and data-driven footing, we apply *three methodologically distinct* methods whose underlying assumptions decrease in strength.
The methods are applied to the same sample rather than to independent experiments, so their estimates are not statistically independent; convergence is therefore a diagnostic of distributional structure rather than a formal statistical guarantee.
When the three estimates agree, the decision boundary is robust to the choice of method; when they disagree, the disagreement itself is informative about whether the data support a single clean decision boundary at a given level.
### 1) Method 1: KDE Antimode / Crossover with Unimodality Test
We use two closely related KDE-based threshold estimators and apply each where it is appropriate.
When two labeled populations are available (e.g., the all-pairs intra-class and inter-class similarity distributions of Section IV-C), the *KDE crossover* is the intersection point of the two kernel density estimates under Scott's rule for bandwidth selection [28]; under equal priors and symmetric misclassification costs it approximates the Bayes-optimal decision boundary between the two classes.
When a single distribution is analyzed (e.g., the per-accountant cosine mean of Section IV-E) the *KDE antimode* is the local density minimum between two modes of the fitted density; it serves the same decision-theoretic role when the distribution is multimodal but is undefined when the distribution is unimodal.
In either case we use the Hartigan & Hartigan dip test [37] as a formal test of unimodality (rejecting the null of unimodality is consistent with but does not directly establish bimodality specifically), and perform a sensitivity analysis varying the bandwidth over $\pm 50\%$ of the Scott's-rule value to verify threshold stability.
### 2) Method 2: Burgstahler-Dichev / McCrary Discontinuity
We additionally apply the discontinuity test of Burgstahler and Dichev [38], made asymptotically rigorous by McCrary [39].
We discretize the cosine distribution into bins of width 0.005 (and dHash into integer bins) and compute, for each bin $i$ with count $n_i$, the standardized deviation from the smooth-null expectation of the average of its neighbours,
$$Z_i = \frac{n_i - \tfrac{1}{2}(n_{i-1} + n_{i+1})}{\sqrt{N p_i (1-p_i) + \tfrac{1}{4} N (p_{i-1}+p_{i+1})(1 - p_{i-1} - p_{i+1})}},$$
which is approximately $N(0,1)$ under the null of distributional smoothness.
A threshold is identified at the transition where $Z_{i-1}$ is significantly negative (observed count below expectation) adjacent to $Z_i$ significantly positive (observed count above expectation); equivalently, for distributions where the non-hand-signed peak sits to the right of a valley, the transition $Z^- \rightarrow Z^+$ marks the candidate decision boundary.
### 3) Method 3: Finite Mixture Model via EM
We fit a two-component Beta mixture to the cosine distribution via the EM algorithm [40] using method-of-moments M-step estimates (which are numerically stable for bounded proportion data).
The first component represents non-hand-signed signatures (high mean, narrow spread) and the second represents hand-signed signatures (lower mean, wider spread).
Under the fitted model the threshold is the crossing point of the two weighted component densities,
$$\pi_1 \cdot \text{Beta}(x; \alpha_1, \beta_1) = (1 - \pi_1) \cdot \text{Beta}(x; \alpha_2, \beta_2),$$
solved numerically via bracketed root-finding.
As a robustness check against the Beta parametric form we fit a parallel two-component Gaussian mixture to the *logit-transformed* similarity, following standard practice for bounded proportion data.
White's [41] quasi-MLE consistency result justifies interpreting the logit-Gaussian estimates as asymptotic approximations to the best Gaussian-family fit under misspecification; we use the cross-check between Beta and logit-Gaussian crossings as a diagnostic of parametric-form sensitivity rather than as a guarantee of distributional recovery.
We fit 2- and 3-component variants of each mixture and report BIC for model selection.
When BIC prefers the 3-component fit, the 2-component assumption itself is a forced fit, and the Bayes-optimal threshold derived from the 2-component crossing should be treated as an upper bound rather than a definitive cut.
### 4) Convergent Validation and Level-Shift Diagnostic
The three methods rest on decreasing-in-strength assumptions: the KDE antimode/crossover requires only smoothness; the BD/McCrary test requires only local smoothness under the null; the Beta mixture additionally requires a parametric specification.
If the three estimated thresholds differ by less than a practically meaningful margin, the classification is robust to the choice of method.
Equally informative is the *level at which the methods agree or disagree*.
Applied to the per-signature similarity distribution the three methods yield thresholds spread across a wide range because per-signature similarity is not a cleanly bimodal population (Section IV-D).
Applied to the per-accountant cosine mean, Methods 1 (KDE antimode) and 3 (Beta-mixture crossing and its logit-Gaussian counterpart) converge within a narrow band, whereas Method 2 (BD/McCrary) does not produce a significant transition because the accountant-mean distribution is smooth at the bin resolution the test requires.
This pattern is consistent with a clustered but smoothly mixed accountant-level distribution rather than a discrete discontinuity, and we interpret it accordingly in Section V rather than treating disagreement among methods as a failure.
### 5) Accountant-Level Three-Method Analysis
In addition to applying the three methods at the per-signature level (Section IV-D), we apply them to the per-accountant aggregates (mean best-match cosine, mean independent minimum dHash) for the 686 CPAs with $\geq 10$ signatures.
The accountant-level estimates provide the methodologically defensible threshold reference used in the per-document classification of Section III-L.
All three methods are reported with their estimates and, where applicable, cross-method spreads.
## J. Accountant-Level Mixture Model
In addition to the signature-level analysis, we fit a Gaussian mixture model in two dimensions to the per-accountant aggregates (mean best-match cosine, mean independent minimum dHash).
The motivation is the expectation---consistent with industry-practice knowledge at Firm A---that an individual CPA's signing *practice* is clustered (typically consistent adoption of non-hand-signing or consistent hand-signing within a given year) even when the output pixel-level *quality* lies on a continuous spectrum.
We fit mixtures with $K \in \{1, 2, 3, 4, 5\}$ components under full covariance, selecting $K^*$ by BIC with 15 random initializations per $K$.
For the selected $K^*$ we report component means, weights, per-component firm composition, and the marginal-density crossing points from the two-component fit, which serve as the natural per-accountant thresholds.
## K. Pixel-Identity, Inter-CPA, and Held-Out Firm A Validation (No Manual Annotation)
Rather than construct a stratified manual-annotation validation set, we validate the classifier using four naturally occurring reference populations that require no human labeling:
1. **Pixel-identical anchor (gold positive, conservative subset):** signatures whose nearest same-CPA match is byte-identical after crop and normalization.
Handwriting physics makes byte-identity impossible under independent signing events, so this anchor is absolute ground truth *for the byte-identical subset* of non-hand-signed signatures.
We emphasize that this anchor is a *subset* of the true positive class---only those non-hand-signed signatures whose nearest match happens to be byte-identical---and perfect recall against this anchor therefore does not establish recall against the full non-hand-signed population (Section V-G discusses this further).
2. **Inter-CPA negative anchor (large gold negative):** $\sim$50,000 pairs of signatures randomly sampled from *different* CPAs.
Inter-CPA pairs cannot arise from reuse of a single signer's stored signature image, so this population is a reliable negative class for threshold sweeps.
This anchor is substantially larger than a simple low-similarity-same-CPA negative and yields tight Wilson 95% confidence intervals on FAR at each candidate threshold.
3. **Firm A anchor (replication-dominated prior positive):** Firm A signatures, treated as a majority-positive reference whose left tail contains a minority of hand-signers, as directly evidenced by the 32/171 middle-band share in the accountant-level mixture (Section III-H).
Because Firm A is both used for empirical percentile calibration in Section III-H and as a validation anchor, we break the resulting circularity by splitting Firm A CPAs randomly (at the CPA level, not the signature level) into a 70% *calibration* fold and a 30% *heldout* fold.
Median, 1st percentile, and 95th percentile of signature-level cosine/dHash distributions are derived from the calibration fold only.
The heldout fold is used exclusively to report post-hoc capture rates with Wilson 95% confidence intervals.
4. **Low-similarity same-CPA anchor (supplementary negative):** signatures whose maximum same-CPA cosine similarity is below 0.70.
This anchor is retained for continuity with prior work but is small in our dataset ($n = 35$) and is reported only as a supplementary reference; its confidence intervals are too wide for quantitative inference.
From these anchors we report FAR with Wilson 95% confidence intervals (against the inter-CPA negative anchor) and FRR (against the byte-identical positive anchor), together with the Equal Error Rate (EER) interpolated at the threshold where FAR $=$ FRR, following biometric-verification reporting conventions [3].
Precision and $F_1$ are not meaningful in this anchor-based evaluation because the positive and negative anchors are constructed from different sampling units (intra-CPA byte-identical pairs vs random inter-CPA pairs), so their relative prevalence in the combined set is an arbitrary construction rather than a population parameter; we therefore omit precision and $F_1$ from Table X.
The 70/30 held-out Firm A fold of Section IV-G.2 additionally reports capture rates with Wilson 95% confidence intervals computed within the held-out fold, which is a valid population for rate inference.
We additionally draw a small stratified sample (30 signatures across high-confidence replication, borderline, style-only, pixel-identical, and likely-genuine strata) for manual visual sanity inspection; this sample is used only for spot-check and does not contribute to reported metrics.
## L. Per-Document Classification
The per-signature classifier operates at the signature level and uses whole-sample Firm A percentile heuristics as its operational thresholds, while the three-method analysis of Section IV-E operates at the accountant level and supplies a *convergent* external reference for the operational cuts.
Because the two analyses are at different units (signature vs accountant) we treat them as complementary rather than substitutable: the accountant-level convergence band cos $\in [0.945, 0.979]$ anchors the signature-level operational cut cos $> 0.95$ used below, and Section IV-G.3 reports a sensitivity analysis in which cos $> 0.95$ is replaced by the accountant-level 2D-GMM marginal crossing cos $> 0.945$.
We assign each signature to one of five signature-level categories using convergent evidence from both descriptors:
1. **High-confidence non-hand-signed:** Cosine $> 0.95$ AND dHash $\leq 5$.
Both descriptors converge on strong replication evidence.
2. **Moderate-confidence non-hand-signed:** Cosine $> 0.95$ AND $5 < $ dHash $\leq 15$.
Feature-level evidence is strong; structural similarity is present but below the high-confidence cutoff, potentially due to scan variations.
3. **High style consistency:** Cosine $> 0.95$ AND dHash $> 15$.
High feature-level similarity without structural corroboration---consistent with a CPA who signs very consistently but not via image reproduction.
4. **Uncertain:** Cosine between the all-pairs intra/inter KDE crossover (0.837) and 0.95 without sufficient convergent evidence for classification in either direction.
5. **Likely hand-signed:** Cosine below the all-pairs KDE crossover threshold.
We note three conventions about the thresholds.
First, the dHash cutoffs $\leq 5$ and $\leq 15$ correspond to the whole-sample Firm A *cosine-conditional* dHash distribution's median and 95th percentile (the dHash to the cosine-nearest same-CPA match), not to the *independent-minimum* dHash distribution we use elsewhere.
The two dHash statistics are related but not identical: the whole-sample cosine-conditional distribution has median $= 5$ and 95th percentile $= 15$, while the calibration-fold independent-minimum distribution has median $= 2$ and 95th percentile $= 9$.
The classifier retains the cosine-conditional cutoffs for continuity with the preceding version of this work while the anchor-level capture-rate analysis reports both cosine-conditional and independent-minimum rates for comparability.
Second, the cosine cutoff $0.95$ is the whole-sample Firm A P95 heuristic (chosen for its transparent interpretation in the whole-sample reference distribution) and the cosine crossover $0.837$ is the all-pairs intra/inter KDE crossover; neither cutoff is re-derived from the 70% calibration fold specifically, so the classifier inherits its operational thresholds from the whole-sample Firm A distribution and the all-pairs distribution rather than from the calibration fold.
The held-out fold of Section IV-G.2 reports both calibration-fold and held-out-fold capture rates for this classifier so that the fold-level sampling variance is visible.
Third, the three accountant-level 1D estimators (KDE antimode $0.973$, Beta-2 crossing $0.979$, logit-GMM-2 crossing $0.976$) and the accountant-level 2D GMM marginal ($0.945$) are *not* the operational thresholds of this classifier: they are the *convergent external reference* that supports the choice of signature-level operational cut.
Section IV-G.3 reports the classifier's five-way output under the nearby operational cut cos $> 0.945$ as a sensitivity check; the aggregate firm-level capture rates change by at most $\approx 1.2$ percentage points (e.g., the operational dual rule cos $> 0.95$ AND dHash $\leq 8$ captures 89.95% of whole Firm A versus 91.14% at cos $> 0.945$), and category-level shifts are concentrated at the Uncertain/Moderate-confidence boundary (Section IV-G.3).
Because each audit report typically carries two certifying-CPA signatures (Section III-D), we aggregate signature-level outcomes to document-level labels using a worst-case rule: the document inherits the *most-replication-consistent* signature label (i.e., among the two signatures, the label rank ordered High-confidence $>$ Moderate-confidence $>$ Style-consistency $>$ Uncertain $>$ Likely-hand-signed determines the document's classification).
This rule is consistent with the detection goal of flagging any potentially non-hand-signed report rather than requiring all signatures on the report to converge.
## M. Data Source and Firm Anonymization
**Audit-report corpus.** The 90,282 audit-report PDFs analyzed in this study were obtained from the Market Observation Post System (MOPS) operated by the Taiwan Stock Exchange Corporation.
MOPS is the statutory public-disclosure platform for Taiwan-listed companies; every audit report filed on MOPS is already a publicly accessible regulatory document.
We did not access any non-public auditor work papers, internal firm records, or personally identifying information beyond the certifying CPAs' names and signatures, which are themselves published on the face of the audit report as part of the public regulatory filing.
The CPA registry used to map signatures to CPAs is a publicly available audit-firm tenure registry (Section III-B).
**Firm-level anonymization.** Although all audit reports and CPA identities in the corpus are public, we report firm-level results under the pseudonyms Firm A / B / C / D throughout this paper to avoid naming specific accounting firms in descriptive rate comparisons.
Readers with domain familiarity may still infer Firm A from contextual descriptors (Big-4 status, replication-dominated behavior); we disclose this residual identifiability explicitly and note that none of the paper's conclusions depend on the specific firm's name.
Authors declare no conflict of interest with Firm A, Firm B, Firm C, or Firm D.
paper/paper_a_references_v3.md
+87
View File
@@ -0,0 +1,87 @@
# References
<!-- IEEE numbered style, sequential by first appearance in text. v3 adds statistical-method refs (37–41). -->
[1] Taiwan Certified Public Accountant Act (會計師法), Art. 4; FSC Attestation Regulations (查核簽證核准準則), Art. 6. Available: https://law.moj.gov.tw/ENG/LawClass/LawAll.aspx?pcode=G0400067
[2] S.-H. Yen, Y.-S. Chang, and H.-L. Chen, "Does the signature of a CPA matter? Evidence from Taiwan," *Res. Account. Regul.*, vol. 25, no. 2, pp. 230–235, 2013.
[3] J. Bromley et al., "Signature verification using a Siamese time delay neural network," in *Proc. NeurIPS*, 1993.
[4] S. Dey et al., "SigNet: Convolutional Siamese network for writer independent offline signature verification," arXiv:1707.02131, 2017.
[5] I. Hadjadj et al., "An offline signature verification method based on a single known sample and an explainable deep learning approach," *Appl. Sci.*, vol. 10, no. 11, p. 3716, 2020.
[6] H. Li et al., "TransOSV: Offline signature verification with transformers," *Pattern Recognit.*, vol. 145, p. 109882, 2024.
[7] S. Tehsin et al., "Enhancing signature verification using triplet Siamese similarity networks in digital documents," *Mathematics*, vol. 12, no. 17, p. 2757, 2024.
[8] P. Brimoh and C. C. Olisah, "Consensus-threshold criterion for offline signature verification using CNN learned representations," arXiv:2401.03085, 2024.
[9] N. Woodruff et al., "Fully-automatic pipeline for document signature analysis to detect money laundering activities," arXiv:2107.14091, 2021.
[10] S. Abramova and R. Böhme, "Detecting copy-move forgeries in scanned text documents," in *Proc. Electronic Imaging*, 2016.
[11] Y. Li et al., "Copy-move forgery detection in digital image forensics: A survey," *Multimedia Tools Appl.*, 2024.
[12] Y. Jakhar and M. D. Borah, "Effective near-duplicate image detection using perceptual hashing and deep learning," *Inf. Process. Manage.*, p. 104086, 2025.
[13] E. Pizzi et al., "A self-supervised descriptor for image copy detection," in *Proc. CVPR*, 2022.
[14] L. G. Hafemann, R. Sabourin, and L. S. Oliveira, "Learning features for offline handwritten signature verification using deep convolutional neural networks," *Pattern Recognit.*, vol. 70, pp. 163–176, 2017.
[15] E. N. Zois, D. Tsourounis, and D. Kalivas, "Similarity distance learning on SPD manifold for writer independent offline signature verification," *IEEE Trans. Inf. Forensics Security*, vol. 19, pp. 1342–1356, 2024.
[16] L. G. Hafemann, R. Sabourin, and L. S. Oliveira, "Meta-learning for fast classifier adaptation to new users of signature verification systems," *IEEE Trans. Inf. Forensics Security*, vol. 15, pp. 1735–1745, 2019.
[17] H. Farid, "Image forgery detection," *IEEE Signal Process. Mag.*, vol. 26, no. 2, pp. 16–25, 2009.
[18] F. Z. Mehrjardi, A. M. Latif, M. S. Zarchi, and R. Sheikhpour, "A survey on deep learning-based image forgery detection," *Pattern Recognit.*, vol. 144, art. no. 109778, 2023.
[19] J. Luo et al., "A survey of perceptual hashing for multimedia," *ACM Trans. Multimedia Comput. Commun. Appl.*, vol. 21, no. 7, 2025.
[20] D. Engin et al., "Offline signature verification on real-world documents," in *Proc. CVPRW*, 2020.
[21] D. Tsourounis et al., "From text to signatures: Knowledge transfer for efficient deep feature learning in offline signature verification," *Expert Syst. Appl.*, 2022.
[22] B. Chamakh and O. Bounouh, "A unified ResNet18-based approach for offline signature classification and verification," *Procedia Comput. Sci.*, vol. 270, 2025.
[23] A. Babenko, A. Slesarev, A. Chigorin, and V. Lempitsky, "Neural codes for image retrieval," in *Proc. ECCV*, 2014, pp. 584–599.
[24] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, "Qwen2.5-VL technical report," arXiv:2502.13923, 2025. [Online]. Available: https://arxiv.org/abs/2502.13923
[25] Ultralytics, "YOLOv11 documentation," 2024. [Online]. Available: https://docs.ultralytics.com/
[26] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in *Proc. CVPR*, 2016.
[27] N. Krawetz, "Kind of like that," The Hacker Factor Blog, 2013. [Online]. Available: https://www.hackerfactor.com/blog/index.php?/archives/529-Kind-of-Like-That.html
[28] B. W. Silverman, *Density Estimation for Statistics and Data Analysis*. London: Chapman & Hall, 1986.
[29] J. Cohen, *Statistical Power Analysis for the Behavioral Sciences*, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum, 1988.
[30] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: From error visibility to structural similarity," *IEEE Trans. Image Process.*, vol. 13, no. 4, pp. 600–612, 2004.
[31] J. V. Carcello and C. Li, "Costs and benefits of requiring an engagement partner signature: Recent experience in the United Kingdom," *The Accounting Review*, vol. 88, no. 5, pp. 1511–1546, 2013.
[32] A. D. Blay, M. Notbohm, C. Schelleman, and A. Valencia, "Audit quality effects of an individual audit engagement partner signature mandate," *Int. J. Auditing*, vol. 18, no. 3, pp. 172–192, 2014.
[33] W. Chi, H. Huang, Y. Liao, and H. Xie, "Mandatory audit partner rotation, audit quality, and market perception: Evidence from Taiwan," *Contemp. Account. Res.*, vol. 26, no. 2, pp. 359–391, 2009.
[34] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in *Proc. CVPR*, 2016, pp. 779–788.
[35] J. Zhang, J. Huang, S. Jin, and S. Lu, "Vision-language models for vision tasks: A survey," *IEEE Trans. Pattern Anal. Mach. Intell.*, vol. 46, no. 8, pp. 5625–5644, 2024.
[36] H. B. Mann and D. R. Whitney, "On a test of whether one of two random variables is stochastically larger than the other," *Ann. Math. Statist.*, vol. 18, no. 1, pp. 50–60, 1947.
[37] J. A. Hartigan and P. M. Hartigan, "The dip test of unimodality," *Ann. Statist.*, vol. 13, no. 1, pp. 70–84, 1985.
[38] D. Burgstahler and I. Dichev, "Earnings management to avoid earnings decreases and losses," *J. Account. Econ.*, vol. 24, no. 1, pp. 99–126, 1997.
[39] J. McCrary, "Manipulation of the running variable in the regression discontinuity design: A density test," *J. Econometrics*, vol. 142, no. 2, pp. 698–714, 2008.
[40] A. P. Dempster, N. M. Laird, and D. B. Rubin, "Maximum likelihood from incomplete data via the EM algorithm," *J. R. Statist. Soc. B*, vol. 39, no. 1, pp. 1–38, 1977.
[41] H. White, "Maximum likelihood estimation of misspecified models," *Econometrica*, vol. 50, no. 1, pp. 1–25, 1982.
<!-- Total: 41 references (v2: 36 + 5 new statistical methods refs) -->
paper/paper_a_related_work_v3.md
+104
View File
@@ -0,0 +1,104 @@
# II. Related Work
## A. Offline Signature Verification
Offline signature verification---determining whether a static signature image is genuine or forged---has been studied extensively using deep learning.
Bromley et al. [3] introduced the Siamese neural network architecture for signature verification, establishing the pairwise comparison paradigm that remains dominant.
Hafemann et al. [14] demonstrated that deep CNN features learned from signature images provide strong discriminative representations for writer-independent verification, establishing the foundational baseline for subsequent work.
Dey et al. [4] proposed SigNet, a convolutional Siamese network for writer-independent offline verification, extending this paradigm to generalize across signers without per-writer retraining.
Hadjadj et al. [5] addressed the practical constraint of limited reference samples, achieving competitive verification accuracy using only a single known genuine signature per writer.
More recently, Li et al. [6] introduced TransOSV, the first Vision Transformer-based approach, achieving state-of-the-art results.
Tehsin et al. [7] evaluated distance metrics for triplet Siamese networks, finding that Manhattan distance outperformed cosine and Euclidean alternatives.
Zois et al. [15] proposed similarity distance learning on SPD manifolds for writer-independent verification, achieving robust cross-dataset transfer.
Hafemann et al. [16] further addressed the practical challenge of adapting to new users through meta-learning, reducing the enrollment burden for signature verification systems.
A common thread in this literature is the assumption that the primary threat is *identity fraud*: a forger attempting to produce a convincing imitation of another person's signature.
Our work addresses a fundamentally different problem---detecting whether the *legitimate signer's* stored signature image has been reproduced across many documents---which requires analyzing the upper tail of the intra-signer similarity distribution rather than modeling inter-signer discriminability.
Brimoh and Olisah [8] proposed a consensus-threshold approach that derives classification boundaries from known genuine reference pairs, the methodology most closely related to our calibration strategy.
However, their method operates on standard verification benchmarks with laboratory-collected signatures, whereas our approach applies threshold calibration using a replication-dominated subpopulation identified through domain expertise in real-world regulatory documents.
## B. Document Forensics and Copy Detection
Image forensics encompasses a broad range of techniques for detecting manipulated visual content [17], with recent surveys highlighting the growing role of deep learning in forgery detection [18].
Copy-move forgery detection (CMFD) identifies duplicated regions within or across images, typically targeting manipulated photographs [11].
Abramova and Böhme [10] adapted block-based CMFD to scanned text documents, noting that standard methods perform poorly in this domain because legitimate character repetitions produce high similarity scores that confound duplicate detection.
Woodruff et al. [9] developed the work most closely related to ours: a fully automated pipeline for extracting and analyzing signatures from corporate filings in the context of anti-money-laundering investigations.
Their system uses connected component analysis for signature detection, GANs for noise removal, and Siamese networks for author clustering.
While their pipeline shares our goal of large-scale automated signature analysis on real regulatory documents, their objective---grouping signatures by authorship---differs fundamentally from ours, which is detecting image-level reproduction within a single author's signatures across documents.
In the domain of image copy detection, Pizzi et al. [13] proposed SSCD, a self-supervised descriptor using ResNet-50 with contrastive learning for large-scale copy detection on natural images.
Their work demonstrates that pre-trained CNN features with cosine similarity provide a strong baseline for identifying near-duplicate images, a finding that supports our feature-extraction approach.
## C. Perceptual Hashing
Perceptual hashing algorithms generate compact fingerprints that are robust to minor image transformations while remaining sensitive to substantive content changes [19].
Unlike cryptographic hashes, which change entirely with any pixel modification, perceptual hashes produce similar outputs for visually similar inputs, making them suitable for near-duplicate detection in scanned documents where minor variations arise from the scanning process.
Jakhar and Borah [12] demonstrated that combining perceptual hashing with deep learning features significantly outperforms either approach alone for near-duplicate image detection, achieving AUROC of 0.99 on standard benchmarks.
Their two-stage architecture---pHash for fast structural comparison followed by deep features for semantic verification---provides methodological precedent for our dual-descriptor approach, though applied to natural images rather than document signatures.
Our work differs from prior perceptual-hashing studies in its application context and in the specific challenge it addresses: distinguishing legitimate high visual consistency (a careful signer producing similar-looking signatures) from image-level reproduction in scanned financial documents.
## D. Deep Feature Extraction for Signature Analysis
Several studies have explored pre-trained CNN features for signature comparison without metric learning or Siamese architectures.
Engin et al. [20] used ResNet-50 features with cosine similarity for offline signature verification on real-world scanned documents, incorporating CycleGAN-based stamp removal as preprocessing---a pipeline design closely paralleling our approach.
Tsourounis et al. [21] demonstrated successful transfer from handwritten text recognition to signature verification, showing that CNN features trained on related but distinct handwriting tasks generalize effectively to signature comparison.
Chamakh and Bounouh [22] confirmed that a simple ResNet backbone with cosine similarity achieves competitive verification accuracy across multilingual signature datasets without fine-tuning, supporting the viability of our off-the-shelf feature-extraction approach.
Babenko et al. [23] established that CNN-extracted neural codes with cosine similarity provide an effective framework for image retrieval and matching, a finding that underpins our feature-comparison approach.
These findings collectively suggest that pre-trained CNN features, when L2-normalized and compared via cosine similarity, provide a robust and computationally efficient representation for signature comparison---particularly suitable for large-scale applications where the computational overhead of Siamese training or metric learning is impractical.
## E. Statistical Methods for Threshold Determination
Our threshold-determination framework combines three families of methods developed in statistics and accounting-econometrics.
*Non-parametric density estimation.*
Kernel density estimation [28] provides a smooth estimate of a similarity distribution without parametric assumptions.
Where the distribution is bimodal, the local density minimum (antimode) between the two modes is the Bayes-optimal decision boundary under equal priors.
The statistical validity of the unimodality-vs-multimodality dichotomy can be tested via the Hartigan & Hartigan dip test [37], which tests the null of unimodality; we use rejection of this null as evidence consistent with (though not a direct test for) bimodality.
*Discontinuity tests on empirical distributions.*
Burgstahler and Dichev [38], working in the accounting-disclosure literature, proposed a test for smoothness violations in empirical frequency distributions.
Under the null that the distribution is generated by a single smooth process, the expected count in any histogram bin equals the average of its two neighbours, and the standardized deviation from this expectation is approximately $N(0,1)$.
The test was placed on rigorous asymptotic footing by McCrary [39], whose density-discontinuity test provides full asymptotic distribution theory, bandwidth-selection rules, and power analysis.
The BD/McCrary pairing is well suited to detecting the boundary between two generative mechanisms (non-hand-signed vs. hand-signed) under minimal distributional assumptions.
*Finite mixture models.*
When the empirical distribution is viewed as a weighted sum of two (or more) latent component distributions, the Expectation-Maximization algorithm [40] provides consistent maximum-likelihood estimates of the component parameters.
For observations bounded on $[0,1]$---such as cosine similarity and normalized Hamming-based dHash similarity---the Beta distribution is the natural parametric choice, with applications spanning bioinformatics and Bayesian estimation.
Under mild regularity conditions, White's quasi-MLE result [41] supports interpreting maximum-likelihood estimates under a mis-specified parametric family as consistent estimators of the pseudo-true parameter that minimizes the Kullback-Leibler divergence to the data-generating distribution within that family; we use this result to justify the Beta-mixture fit as a principled approximation rather than as a guarantee that the true distribution is Beta.
The present study combines all three families, using each to produce an independent threshold estimate and treating cross-method convergence---or principled divergence---as evidence of where in the analysis hierarchy the mixture structure is statistically supported.
<!--
REFERENCES for Related Work (see paper_a_references_v3.md for full list):
[3] Bromley et al. 1993 — Siamese TDNN (NeurIPS)
[4] Dey et al. 2017 — SigNet
[5] Hadjadj et al. 2020 — Single sample SV
[6] Li et al. 2024 — TransOSV
[7] Tehsin et al. 2024 — Triplet Siamese
[8] Brimoh & Olisah 2024 — Consensus threshold
[9] Woodruff et al. 2021 — AML signature pipeline
[10] Abramova & Böhme 2016 — CMFD in scanned docs
[11] Copy-move forgery detection survey — MTAP 2024
[12] Jakhar & Borah 2025 — pHash + DL
[13] Pizzi et al. 2022 — SSCD
[14] Hafemann et al. 2017 — CNN features for SV
[15] Zois et al. 2024 — SPD manifold SV
[16] Hafemann et al. 2019 — Meta-learning for SV
[17] Farid 2009 — Image forgery detection survey
[18] Mehrjardi et al. 2023 — DL-based image forgery detection survey
[19] Luo et al. 2025 — Perceptual hashing survey
[20] Engin et al. 2020 — ResNet + cosine on real docs
[21] Tsourounis et al. 2022 — Transfer from text to signatures
[22] Chamakh & Bounouh 2025 — ResNet18 unified SV
[23] Babenko et al. 2014 — Neural codes for image retrieval
[28] Silverman 1986 — Density estimation
[37] Hartigan & Hartigan 1985 — dip test of unimodality
[38] Burgstahler & Dichev 1997 — earnings management discontinuity
[39] McCrary 2008 — density discontinuity test
[40] Dempster, Laird & Rubin 1977 — EM algorithm
[41] White 1982 — quasi-MLE consistency
-->
paper/paper_a_results_v3.md
+436
View File
@@ -0,0 +1,436 @@
# IV. Experiments and Results
## A. Experimental Setup
All experiments were conducted on a workstation equipped with an Apple Silicon processor with Metal Performance Shaders (MPS) GPU acceleration.
Feature extraction used PyTorch 2.9 with torchvision model implementations.
The complete pipeline---from raw PDF processing through final classification---was implemented in Python.
## B. Signature Detection Performance
The YOLOv11n model achieved high detection performance on the validation set (Table II), with all loss components converging by epoch 60 and no significant overfitting despite the relatively small training set (425 images).
We note that Table II reports validation-set metrics, as no separate hold-out test set was reserved given the small annotation budget (500 images total).
However, the subsequent production deployment provides practical validation: batch inference on 86,071 documents yielded 182,328 extracted signatures (Table III), with an average of 2.14 signatures per document, consistent with the standard practice of two certifying CPAs per audit report.
The high VLM--YOLO agreement rate (98.8%) further corroborates detection reliability at scale.
<!-- TABLE III: Extraction Results
| Metric | Value |
|--------|-------|
| Documents processed | 86,071 |
| Documents with detections | 85,042 (98.8%) |
| Total signatures extracted | 182,328 |
| Avg. signatures per document | 2.14 |
| CPA-matched signatures | 168,755 (92.6%) |
| Processing rate | 43.1 docs/sec |
-->
## C. All-Pairs Intra-vs-Inter Class Distribution Analysis
Fig. 2 presents the cosine similarity distributions computed over the full set of *pairwise comparisons* under two groupings: intra-class (all signature pairs belonging to the same CPA) and inter-class (signature pairs from different CPAs).
This all-pairs analysis is a different unit from the per-signature best-match statistics used in Sections IV-D onward; we report it first because it supplies the reference point for the KDE crossover used in per-document classification (Section III-L).
Table IV summarizes the distributional statistics.
<!-- TABLE IV: Cosine Similarity Distribution Statistics
| Statistic | Intra-class | Inter-class |
|-----------|-------------|-------------|
| N (pairs) | 41,352,824 | 500,000 |
| Mean | 0.821 | 0.758 |
| Std. Dev. | 0.098 | 0.090 |
| Median | 0.836 | 0.774 |
| Skewness | −0.711 | −0.851 |
| Kurtosis | 0.550 | 1.027 |
-->
Both distributions are left-skewed and leptokurtic.
Shapiro-Wilk and Kolmogorov-Smirnov tests rejected normality for both ($p < 0.001$), confirming that parametric thresholds based on normality assumptions would be inappropriate.
Distribution fitting identified the lognormal distribution as the best parametric fit (lowest AIC) for both classes, though we use this result only descriptively; all subsequent thresholds are derived via the three convergent methods of Section III-I to avoid single-family distributional assumptions.
The KDE crossover---where the two density functions intersect---was located at 0.837 (Table V).
Under equal prior probabilities and equal misclassification costs, this crossover approximates the Bayes-optimal boundary between the two classes.
Statistical tests confirmed significant separation between the two distributions (Cohen's $d = 0.669$, Mann-Whitney $p < 0.001$, K-S 2-sample $p < 0.001$).
We emphasize that pairwise observations are not independent---the same signature participates in multiple pairs---which inflates the effective sample size and renders $p$-values unreliable as measures of evidence strength.
We therefore rely primarily on Cohen's $d$ as an effect-size measure that is less sensitive to sample size.
A Cohen's $d$ of 0.669 indicates a medium effect size [29], confirming that the distributional difference is practically meaningful, not merely an artifact of the large sample count.
## D. Hartigan Dip Test: Unimodality at the Signature Level
Applying the Hartigan & Hartigan dip test [37] to the per-signature best-match distributions reveals a critical structural finding (Table V).
<!-- TABLE V: Hartigan Dip Test Results
| Distribution | N | dip | p-value | Verdict (α=0.05) |
|--------------|---|-----|---------|------------------|
| Firm A cosine (max-sim) | 60,448 | 0.0019 | 0.169 | Unimodal |
| Firm A min dHash (independent) | 60,448 | 0.1051 | <0.001 | Multimodal |
| All-CPA cosine (max-sim) | 168,740 | 0.0035 | <0.001 | Multimodal |
| All-CPA min dHash (independent) | 168,740 | 0.0468 | <0.001 | Multimodal |
| Per-accountant cos mean | 686 | 0.0339 | <0.001 | Multimodal |
| Per-accountant dHash mean | 686 | 0.0277 | <0.001 | Multimodal |
-->
Firm A's per-signature cosine distribution is *unimodal* ($p = 0.17$), reflecting a single dominant generative mechanism (non-hand-signing) with a long left tail attributable to the minority of hand-signing Firm A partners identified in the accountant-level mixture (Section IV-E).
The all-CPA cosine distribution, which mixes many firms with heterogeneous signing practices, is *multimodal* ($p < 0.001$).
At the per-accountant aggregate level both cosine and dHash means are strongly multimodal, foreshadowing the mixture structure analyzed in Section IV-E.
This asymmetry between signature level and accountant level is itself an empirical finding.
It predicts that a two-component mixture fit to per-signature cosine will be a forced fit (Section IV-D.2 below), while the same fit at the accountant level will succeed---a prediction borne out in the subsequent analyses.
### 1) Burgstahler-Dichev / McCrary Discontinuity
Applying the BD/McCrary test (Section III-I.2) to the per-signature cosine distribution yields a single significant transition at 0.985 for Firm A and 0.985 for the full sample; the min-dHash distributions exhibit a transition at Hamming distance 2 for both Firm A and the full sample.
We note that the cosine transition at 0.985 lies *inside* the non-hand-signed mode rather than at the separation between two mechanisms, consistent with the dip-test finding that per-signature cosine is not cleanly bimodal.
In contrast, the dHash transition at distance 2 is a substantively meaningful structural boundary that corresponds to the natural separation between pixel-near-identical replication and scan-noise-perturbed replication.
At the accountant level the test does not produce a significant $Z^- \rightarrow Z^+$ transition in either the cosine-mean or the dHash-mean distribution (Section IV-E), reflecting that accountant aggregates are smooth at the bin resolution the test requires rather than exhibiting a sharp density discontinuity.
### 2) Beta Mixture at Signature Level: A Forced Fit
Fitting 2- and 3-component Beta mixtures to Firm A's per-signature cosine via EM yields a clear BIC preference for the 3-component fit ($\Delta\text{BIC} = 381$), with a parallel preference under the logit-GMM robustness check.
For the full-sample cosine the 3-component fit is likewise strongly preferred ($\Delta\text{BIC} = 10{,}175$).
Under the forced 2-component fit the Firm A Beta crossing lies at 0.977 and the logit-GMM crossing at 0.999---values sharply inconsistent with each other, indicating that the 2-component parametric structure is not supported by the data.
Under the full-sample 2-component forced fit no Beta crossing is identified; the logit-GMM crossing is at 0.980.
The joint reading of Sections IV-D.1 and IV-D.2 is unambiguous: *at the per-signature level, no two-mechanism mixture explains the data*.
Non-hand-signed replication quality is a continuous spectrum, not a discrete class cleanly separated from hand-signing.
This motivates the pivot to the accountant-level analysis in Section IV-E, where aggregation over signatures reveals clustered (though not sharply discrete) patterns in individual-level signing *practice* that the signature-level analysis lacks.
## E. Accountant-Level Gaussian Mixture
We aggregated per-signature descriptors to the CPA level (mean best-match cosine, mean independent minimum dHash) for the 686 CPAs with $\geq 10$ signatures and fit Gaussian mixtures in two dimensions with $K \in \{1, \ldots, 5\}$.
BIC selects $K^* = 3$ (Table VI).
<!-- TABLE VI: Accountant-Level GMM Model Selection (BIC)
| K | BIC | AIC | Converged |
|---|-----|-----|-----------|
| 1 | −316 | −339 | ✓ |
| 2 | −545 | −595 | ✓ |
| 3 | **−792** | **−869** | ✓ (best) |
| 4 | −779 | −883 | ✓ |
| 5 | −747 | −879 | ✓ |
-->
Table VII reports the three-component composition, and Fig. 4 visualizes the accountant-level clusters in the (cosine-mean, dHash-mean) plane alongside the marginal-density crossings of the two-component fit.
<!-- TABLE VII: Accountant-Level 3-Component GMM
| Comp. | cos_mean | dHash_mean | weight | n | Dominant firms |
|-------|----------|------------|--------|---|----------------|
| C1 (high-replication) | 0.983 | 2.41 | 0.21 | 141 | Firm A (139/141) |
| C2 (middle band) | 0.954 | 6.99 | 0.51 | 361 | three other Big-4 firms (Firms B/C/D, ~256 together) |
| C3 (hand-signed tendency) | 0.928 | 11.17 | 0.28 | 184 | smaller domestic firms |
-->
Three empirical findings stand out.
First, of the 180 CPAs in the Firm A registry, 171 have $\geq 10$ signatures and therefore enter the accountant-level GMM (the remaining 9 have too few signatures for reliable aggregates and are excluded from this analysis only).
Component C1 captures 139 of these 171 Firm A CPAs (81%) in a tight high-cosine / low-dHash cluster; the remaining 32 Firm A CPAs fall into C2.
This split is consistent with the minority-hand-signers framing of Section III-H and with the unimodal-long-tail observation of Section IV-D.
Second, the three-component partition is *not* a firm-identity partition: three of the four Big-4 firms dominate C2 together, and smaller domestic firms cluster into C3.
Third, applying the three-method framework of Section III-I to the accountant-level cosine-mean distribution yields the estimates summarized in the accountant-level rows of Table VIII (below): KDE antimode $= 0.973$, Beta-2 crossing $= 0.979$, and the logit-GMM-2 crossing $= 0.976$ converge within $\sim 0.006$ of each other, while the BD/McCrary test does not produce a significant transition at the accountant level.
For completeness we also report the two-dimensional two-component GMM's marginal crossings at cosine $= 0.945$ and dHash $= 8.10$; these differ from the 1D crossings because they are derived from the joint (cosine, dHash) covariance structure rather than from each 1D marginal in isolation.
Table VIII summarizes all threshold estimates produced by the three methods across the two analysis levels for a compact cross-level comparison.
<!-- TABLE VIII: Threshold Convergence Summary Across Levels
| Level / method | Cosine threshold | dHash threshold |
|----------------|-------------------|------------------|
| Signature-level, all-pairs KDE crossover | 0.837 | — |
| Signature-level, BD/McCrary transition | 0.985 | 2.0 |
| Signature-level, Beta-2 EM crossing (Firm A) | 0.977 | — |
| Signature-level, logit-GMM-2 crossing (Full) | 0.980 | — |
| Accountant-level, KDE antimode | **0.973** | **4.07** |
| Accountant-level, BD/McCrary transition | no transition | no transition |
| Accountant-level, Beta-2 EM crossing | **0.979** | **3.41** |
| Accountant-level, logit-GMM-2 crossing | **0.976** | **3.93** |
| Accountant-level, 2D-GMM 2-comp marginal crossing | 0.945 | 8.10 |
| Firm A calibration-fold cosine P5 | 0.941 | — |
| Firm A calibration-fold dHash P95 | — | 9 |
| Firm A calibration-fold dHash median | — | 2 |
-->
Methods 1 and 3 (KDE antimode, Beta-2 crossing, and its logit-GMM robustness check) converge at the accountant level to a cosine threshold of $\approx 0.975 \pm 0.003$ and a dHash threshold of $\approx 3.8 \pm 0.4$, while Method 2 (BD/McCrary) does not produce a significant discontinuity.
This is the accountant-level convergence we rely on for the primary threshold interpretation; the two-dimensional GMM marginal crossings (cosine $= 0.945$, dHash $= 8.10$) differ because they reflect joint (cosine, dHash) covariance structure, and we report them as a secondary cross-check.
The signature-level estimates are reported for completeness and as diagnostic evidence of the continuous-spectrum asymmetry (Section IV-D.2) rather than as primary classification boundaries.
## F. Calibration Validation with Firm A
Fig. 3 presents the per-signature cosine and dHash distributions of Firm A compared to the overall population.
Table IX reports the proportion of Firm A signatures crossing each candidate threshold; these rates play the role of calibration-validation metrics (what fraction of a known replication-dominated population does each threshold capture?).
<!-- TABLE IX: Firm A Whole-Sample Capture Rates (consistency check, NOT external validation)
| Rule | Firm A rate | n / N |
|------|-------------|-------|
| cosine > 0.837 (all-pairs KDE crossover) | 99.93% | 60,405 / 60,448 |
| cosine > 0.941 (calibration-fold P5) | 95.08% | 57,473 / 60,448 |
| cosine > 0.945 (2D GMM marginal crossing) | 94.52% | 57,131 / 60,448 |
| cosine > 0.95 | 92.51% | 55,916 / 60,448 |
| cosine > 0.973 (accountant KDE antimode) | 80.91% | 48,910 / 60,448 |
| dHash_indep ≤ 5 (calib-fold median-adjacent) | 84.20% | 50,897 / 60,448 |
| dHash_indep ≤ 8 | 95.17% | 57,521 / 60,448 |
| dHash_indep ≤ 15 | 99.83% | 60,345 / 60,448 |
| cosine > 0.95 AND dHash_indep ≤ 8 | 89.95% | 54,373 / 60,448 |
All rates computed exactly from the full Firm A sample (N = 60,448 signatures).
The threshold 0.941 corresponds to the 5th percentile of the calibration-fold Firm A cosine distribution (see Section IV-G for the held-out validation that addresses the circularity inherent in this whole-sample table).
-->
Table IX is a whole-sample consistency check rather than an external validation: the thresholds 0.95, dHash median, and dHash 95th percentile are themselves anchored to Firm A via the calibration described in Section III-H.
The dual rule cosine $> 0.95$ AND dHash $\leq 8$ captures 89.95% of Firm A, a value that is consistent with both the accountant-level crossings (Section IV-E) and the 139/32 high-replication versus middle-band split within Firm A (Section IV-E).
Section IV-G reports the corresponding rates on the 30% Firm A hold-out fold, which provides the external check these whole-sample rates cannot.
## G. Pixel-Identity, Inter-CPA, and Held-Out Firm A Validation
We report three validation analyses corresponding to the anchors of Section III-K.
### 1) Pixel-Identity Positive Anchor with Inter-CPA Negative Anchor
Of the 182,328 extracted signatures, 310 have a same-CPA nearest match that is byte-identical after crop and normalization (pixel-identical-to-closest = 1); these form the gold-positive anchor.
As the gold-negative anchor we sample 50,000 random cross-CPA signature pairs (inter-CPA cosine: mean $= 0.762$, $P_{95} = 0.884$, $P_{99} = 0.913$, max $= 0.988$).
Because the positive and negative anchor populations are constructed from different sampling units (byte-identical same-CPA pairs vs random inter-CPA pairs), their relative prevalence in the combined anchor set is arbitrary, and precision / $F_1$ / recall therefore have no meaningful population interpretation.
We accordingly report FAR with Wilson 95% confidence intervals against the large inter-CPA negative anchor and FRR against the byte-identical positive anchor in Table X; these two error rates are well defined within their respective anchor populations.
The Equal-Error-Rate point, interpolated at FAR $=$ FRR, is located at cosine $= 0.990$ with EER $\approx 0$, which is trivially small because every byte-identical positive falls at cosine very close to 1.
<!-- TABLE X: Cosine Threshold Sweep (positives = 310 byte-identical signatures; negatives = 50,000 inter-CPA pairs)
| Threshold | FAR | FAR 95% Wilson CI | FRR (byte-identical) |
|-----------|-----|-------------------|----------------------|
| 0.837 (all-pairs KDE crossover) | 0.2062 | [0.2027, 0.2098] | 0.000 |
| 0.900 | 0.0233 | [0.0221, 0.0247] | 0.000 |
| 0.945 (2D GMM marginal) | 0.0008 | [0.0006, 0.0011] | 0.000 |
| 0.950 | 0.0007 | [0.0005, 0.0009] | 0.000 |
| 0.973 (accountant KDE antimode) | 0.0003 | [0.0002, 0.0004] | 0.000 |
| 0.979 (accountant Beta-2) | 0.0002 | [0.0001, 0.0004] | 0.000 |
-->
Two caveats apply.
First, the gold-positive anchor is a *conservative subset* of the true non-hand-signed population: it captures only those non-hand-signed signatures whose nearest match happens to be byte-identical, not those that are near-identical but not bytewise identical.
Zero FRR against this subset does not establish zero FRR against the broader positive class, and the reported FRR should therefore be interpreted as a lower-bound calibration check on the classifier's ability to catch the clearest positives rather than a generalizable miss rate.
Second, the 0.945 / 0.95 / 0.973 thresholds are derived from the Firm A calibration fold or the accountant-level methods rather than from this anchor set, so the FAR values in Table X are post-hoc-fit-free evaluations of thresholds that were not chosen to optimize Table X.
The very low FAR at the accountant-level thresholds is therefore informative about specificity against a realistic inter-CPA negative population.
### 2) Held-Out Firm A Validation (breaks calibration-validation circularity)
We split Firm A CPAs randomly 70 / 30 at the CPA level into a calibration fold (124 CPAs, 45,116 signatures) and a held-out fold (54 CPAs, 15,332 signatures).
The total of 178 Firm A CPAs differs from the 180 in the Firm A registry by two CPAs whose signatures could not be matched to a single assigned-accountant record because of disambiguation ties in the CPA registry and which we therefore exclude from both folds; this handling is made explicit here and has no effect on the accountant-level mixture analysis of Section IV-E, which uses the $\geq 10$-signature subset of 171 CPAs.
Thresholds are re-derived from calibration-fold percentiles only.
Table XI reports both calibration-fold and held-out-fold capture rates with Wilson 95% CIs and a two-proportion $z$-test.
<!-- TABLE XI: Held-Out vs Calibration Firm A Capture Rates and Generalization Test
| Rule | Calibration 70% fold (CI) | Held-out 30% fold (CI) | 2-prop z | p | k/n calib | k/n held |
|------|---------------------------|-------------------------|----------|---|-----------|----------|
| cosine > 0.837 | 99.94% [99.91%, 99.96%] | 99.93% [99.87%, 99.96%] | +0.31 | 0.756 n.s. | 45,088/45,116 | 15,321/15,332 |
| cosine > 0.945 (2D GMM marginal) | 93.77% [93.55%, 93.98%] | 94.78% [94.41%, 95.12%] | -4.54 | <0.001 | 42,304/45,116 | 14,531/15,332 |
| cosine > 0.950 | 92.14% [91.89%, 92.38%] | 93.61% [93.21%, 93.98%] | -5.97 | <0.001 | 41,571/45,116 | 14,352/15,332 |
| cosine > 0.9407 (calib-fold P5) | 95.00% [94.80%, 95.20%] | 95.64% [95.31%, 95.95%] | -2.83 | 0.005 | 42,862/45,116 | 14,664/15,332 |
| dHash_indep ≤ 5 | 82.96% [82.61%, 83.31%] | 87.84% [87.31%, 88.34%] | -14.29 | <0.001 | 37,434/45,116 | 13,467/15,332 |
| dHash_indep ≤ 8 | 94.84% [94.63%, 95.05%] | 96.13% [95.82%, 96.43%] | -6.45 | <0.001 | 42,791/45,116 | 14,739/15,332 |
| dHash_indep ≤ 9 (calib-fold P95) | 96.65% [96.48%, 96.81%] | 97.48% [97.22%, 97.71%] | -5.07 | <0.001 | 43,603/45,116 | 14,945/15,332 |
| dHash_indep ≤ 15 | 99.83% [99.79%, 99.86%] | 99.84% [99.77%, 99.89%] | -0.31 | 0.754 n.s. | 45,038/45,116 | 15,308/15,332 |
| cosine > 0.95 AND dHash_indep ≤ 8 | 89.40% [89.12%, 89.69%] | 91.54% [91.09%, 91.97%] | -7.60 | <0.001 | 40,335/45,116 | 14,035/15,332 |
Calibration-fold thresholds: Firm A cosine median = 0.9862, P1 = 0.9067, P5 = 0.9407; dHash median = 2, P95 = 9.
-->
Table XI reports both calibration-fold and held-out-fold capture rates with Wilson 95% CIs and a two-proportion $z$-test.
We report fold-versus-fold comparisons rather than fold-versus-whole-sample comparisons, because the whole-sample rate is a weighted average of the two folds and therefore cannot, in general, fall inside the Wilson CI of either fold when the folds differ in rate; the correct generalization reference is the calibration fold, which produced the thresholds.
Under this proper test the two extreme rules agree across folds (cosine $> 0.837$ and $\text{dHash}_\text{indep} \leq 15$; both $p > 0.7$).
The operationally relevant rules in the 85–95% capture band differ between folds by 1–5 percentage points ($p < 0.001$ given the $n \approx 45\text{k}/15\text{k}$ fold sizes).
Both folds nevertheless sit in the same replication-dominated regime: every calibration-fold rate in the 85–99% range has a held-out counterpart in the 87–99% range, and the operational dual rule cosine $> 0.95$ AND $\text{dHash}_\text{indep} \leq 8$ captures 89.40% of the calibration fold and 91.54% of the held-out fold.
The modest fold gap is consistent with within-Firm-A heterogeneity in replication intensity (see the $139 / 32$ accountant-level split of Section IV-E): the random 30% CPA sample happened to contain proportionally more accountants from the high-replication C1 cluster.
We therefore interpret the held-out fold as confirming the qualitative finding (Firm A is strongly replication-dominated across both folds) while cautioning that exact rates carry fold-level sampling noise that a single 30% split cannot eliminate; the accountant-level GMM (Section IV-E) and the threshold-independent partner-ranking analysis (Section IV-H.2) are the cross-checks that are robust to this fold variance.
### 3) Operational-Threshold Sensitivity: cos $> 0.95$ vs cos $> 0.945$
The per-signature classifier (Section III-L) uses cos $> 0.95$ as its operational cosine cut, anchored on the whole-sample Firm A P95 heuristic.
The accountant-level three-method convergence (Section IV-E) places the primary accountant-level reference between $0.973$ and $0.979$, and the accountant-level 2D-GMM marginal at $0.945$.
Because the classifier operates at the signature level while the three-method convergence estimates are at the accountant level, they are formally non-substitutable.
We report a sensitivity check in which the classifier's operational cut cos $> 0.95$ is replaced by the nearest accountant-level reference, cos $> 0.945$.
<!-- TABLE XII: Classifier Sensitivity to the Operational Cosine Cut (All-Sample Five-Way Output, N = 168,740 signatures)
| Category | cos > 0.95 count (%) | cos > 0.945 count (%) | Δ count |
|--------------------------------------------|----------------------|-----------------------|---------|
| High-confidence non-hand-signed | 76,984 (45.62%) | 79,278 (46.98%) | +2,294 |
| Moderate-confidence non-hand-signed | 43,906 (26.02%) | 50,001 (29.63%) | +6,095 |
| High style consistency | 546 ( 0.32%) | 665 ( 0.39%) | +119 |
| Uncertain | 46,768 (27.72%) | 38,260 (22.67%) | -8,508 |
| Likely hand-signed | 536 ( 0.32%) | 536 ( 0.32%) | +0 |
-->
At the aggregate firm-level, the operational dual rule cos $> 0.95$ AND $\text{dHash}_\text{indep} \leq 8$ captures 89.95% of whole Firm A under the 0.95 cut and 91.14% under the 0.945 cut---a shift of 1.19 percentage points.
At the per-signature categorization level, replacing 0.95 by 0.945 reclassifies 8,508 signatures (5.04% of the corpus) out of the Uncertain band; 6,095 of them migrate to Moderate-confidence non-hand-signed, 2,294 to High-confidence non-hand-signed, and 119 to High style consistency.
The Likely-hand-signed category is unaffected because it depends only on the fixed all-pairs KDE crossover cosine $= 0.837$.
The High-confidence non-hand-signed share grows from 45.62% to 46.98%.
We interpret this sensitivity pattern as indicating that the classifier's aggregate and high-confidence output is robust to the choice of operational cut within the accountant-level convergence band, and that the movement is concentrated at the Uncertain/Moderate-confidence boundary.
The paper therefore retains cos $> 0.95$ as the primary operational cut for transparency and reports the 0.945 results as a sensitivity check rather than as a deployed alternative; a future deployment requiring tighter accountant-level alignment could substitute cos $> 0.945$ without altering the substantive firm-level conclusions.
### 4) Sanity Sample
A 30-signature stratified visual sanity sample (six signatures each from pixel-identical, high-cos/low-dh, borderline, style-only, and likely-genuine strata) produced inter-rater agreement with the classifier in all 30 cases; this sample contributed only to spot-check and is not used to compute reported metrics.
## H. Additional Firm A Benchmark Validation
The capture rates of Section IV-F are a within-sample consistency check: they evaluate how well a threshold captures Firm A, but the thresholds themselves are anchored to Firm A's percentiles.
This section reports three complementary analyses that go beyond the whole-sample capture rates.
Subsection H.2 is fully threshold-independent (it uses only ordinal ranking).
Subsection H.1 uses a fixed 0.95 cutoff but derives information from the longitudinal stability of rates rather than from the absolute rate at any single year.
Subsection H.3 applies the calibrated classifier and is therefore a consistency check on the classifier's firm-level output rather than a threshold-free test; the informative quantity is the cross-firm *gap* rather than the absolute agreement rate at any one firm.
### 1) Year-by-Year Stability of the Firm A Left Tail
Table XIII reports the proportion of Firm A signatures with per-signature best-match cosine below 0.95, disaggregated by fiscal year.
Under the replication-dominated interpretation (Section III-H) this left-tail share captures the minority of Firm A partners who continue to hand-sign.
Under the alternative hypothesis that the left tail is an artifact of scan or compression noise, the share should shrink as scanning and PDF-compression technology improved over 2013-2023.
<!-- TABLE XIII: Firm A Per-Year Cosine Distribution
| Year | N sigs | mean cosine | % below 0.95 |
|------|--------|-------------|--------------|
| 2013 | 2,167 | 0.9733 | 12.78% |
| 2014 | 5,256 | 0.9781 | 8.69% |
| 2015 | 5,484 | 0.9793 | 7.46% |
| 2016 | 5,739 | 0.9811 | 6.92% |
| 2017 | 5,796 | 0.9814 | 6.69% |
| 2018 | 5,986 | 0.9808 | 6.58% |
| 2019 | 6,122 | 0.9780 | 8.71% |
| 2020 | 6,122 | 0.9770 | 9.46% |
| 2021 | 5,996 | 0.9792 | 8.37% |
| 2022 | 5,918 | 0.9819 | 6.25% |
| 2023 | 5,862 | 0.9860 | 3.75% |
-->
The left tail is stable at 6-13% throughout the sample period and shows no pre/post-2020 level shift: the 2013-2019 mean left-tail share is 8.26% and the 2020-2023 mean is 6.96%.
The lowest observed share is in 2023 (3.75%), consistent with firm-level electronic signing systems producing more uniform output than earlier manual scanning-and-stamping, not less.
This stability supports the replication-dominated framing: a persistent minority of hand-signing Firm A partners is consistent with a Beta left tail that is stable across production technologies, whereas a noise-only explanation would predict a shrinking share as technology improved.
### 2) Partner-Level Similarity Ranking
If Firm A applies firm-wide stamping while the other Big-4 firms use stamping only for a subset of partners, Firm A auditor-years should disproportionately occupy the top of the similarity distribution among all Big-4 auditor-years.
We test this prediction directly.
For each auditor-year (CPA $\times$ fiscal year) with at least 5 signatures we compute the mean best-match cosine similarity across the year's signatures, yielding 4,629 auditor-years across 2013-2023.
Firm A accounts for 1,287 of these (27.8% baseline share).
Table XIV reports per-firm occupancy of the top $K\%$ of the ranked distribution.
<!-- TABLE XIV: Top-K Similarity Rank Occupancy by Firm (pooled 2013-2023)
| Top-K | k in bucket | Firm A | Firm B | Firm C | Firm D | Non-Big-4 | Firm A share |
|-------|-------------|--------|--------|--------|--------|-----------|--------------|
| 10% | 462 | 443 | 2 | 3 | 0 | 14 | 95.9% |
| 25% | 1,157 | 1,043 | 32 | 23 | 9 | 50 | 90.1% |
| 50% | 2,314 | 1,220 | 473 | 273 | 102 | 246 | 52.7% |
-->
Firm A occupies 95.9% of the top 10% and 90.1% of the top 25% of auditor-years by similarity, against its baseline share of 27.8%---a concentration ratio of 3.5$\times$ at the top decile and 3.2$\times$ at the top quartile.
Year-by-year (Table XV), the top-10% Firm A share ranges from 88.4% (2020) to 100% (2013, 2014, 2017, 2018, 2019), showing that the concentration is stable across the sample period.
<!-- TABLE XV: Firm A Share of Top-10% Similarity by Year
| Year | N auditor-years | Top-10% k | Firm A in top-10% | Firm A share | Firm A baseline |
|------|-----------------|-----------|-------------------|--------------|-----------------|
| 2013 | 324 | 32 | 32 | 100.0% | 26.2% |
| 2014 | 399 | 39 | 39 | 100.0% | 27.1% |
| 2015 | 394 | 39 | 38 | 97.4% | 27.2% |
| 2016 | 413 | 41 | 39 | 95.1% | 27.4% |
| 2017 | 415 | 41 | 41 | 100.0% | 27.9% |
| 2018 | 434 | 43 | 43 | 100.0% | 28.1% |
| 2019 | 429 | 42 | 42 | 100.0% | 28.2% |
| 2020 | 430 | 43 | 38 | 88.4% | 28.3% |
| 2021 | 450 | 45 | 44 | 97.8% | 28.4% |
| 2022 | 467 | 46 | 43 | 93.5% | 28.5% |
| 2023 | 474 | 47 | 46 | 97.9% | 28.5% |
-->
This over-representation is a direct consequence of firm-wide non-hand-signing practice and is not derived from any threshold we subsequently calibrate.
It therefore constitutes genuine cross-firm evidence for Firm A's benchmark status.
### 3) Intra-Report Consistency
Taiwanese statutory audit reports are co-signed by two engagement partners (a primary and a secondary signer).
Under firm-wide stamping practice at a given firm, both signers on the same report should receive the same signature-level classification.
Disagreement between the two signers on a report is informative about whether the stamping practice is firm-wide or partner-specific.
For each report with exactly two signatures and complete per-signature data (83,970 reports assigned to a single firm, plus 384 reports with one signer per firm in the mixed-firm buckets for 84,354 total), we classify each signature using the dual-descriptor rules of Section III-L and record whether the two classifications agree.
Table XVI reports per-firm intra-report agreement (firm-assignment defined by the firm identity of both signers; mixed-firm reports are reported separately).
<!-- TABLE XVI: Intra-Report Classification Agreement by Firm
| Firm | Total 2-signer reports | Both non-hand-signed | Both uncertain | Both style | Both hand-signed | Mixed | Agreement rate |
|------|-----------------------|----------------------|----------------|------------|------------------|-------|----------------|
| Firm A | 30,222 | 26,435 | 734 | 0 | 4 | 3,049 | **89.91%** |
| Firm B | 17,121 | 9,260 | 2,159| 5 | 6 | 5,691 | 66.76% |
| Firm C | 19,112 | 8,983 | 3,035| 3 | 5 | 7,086 | 62.92% |
| Firm D | 8,375 | 3,028 | 2,376| 0 | 3 | 2,968 | 64.56% |
| Non-Big-4 | 9,140 | 1,671 | 3,945| 18| 27| 3,479 | 61.94% |
A report is "in agreement" if both signature labels fall in the same coarse bucket
(non-hand-signed = high+moderate; uncertain; style consistency; or likely hand-signed).
-->
Firm A achieves 89.9% intra-report agreement, with 87.5% of Firm A reports having *both* signers classified as non-hand-signed and only 4 reports (0.01%) having both classified as likely hand-signed.
The other Big-4 firms (B, C, D) and non-Big-4 firms cluster at 62-67% agreement, a 23-28 percentage-point gap.
This sharp discontinuity in intra-report agreement between Firm A and the other firms is the pattern predicted by firm-wide (rather than partner-specific) non-hand-signing practice.
We note that this test uses the calibrated classifier of Section III-L rather than a threshold-free statistic; the substantive evidence lies in the *cross-firm gap* between Firm A and the other firms rather than in the absolute agreement rate at any single firm, and that gap is robust to moderate shifts in the absolute cutoff so long as the cutoff is applied uniformly across firms.
## I. Classification Results
Table XVII presents the final classification results under the dual-descriptor framework with Firm A-calibrated thresholds for 84,386 documents.
The document count (84,386) differs from the 85,042 documents with any YOLO detection (Table III) because 656 documents carry only a single detected signature, for which no same-CPA pairwise comparison and therefore no best-match cosine / min dHash statistic is available; those documents are excluded from the classification reported here.
<!-- TABLE XVII: Document-Level Classification (Dual-Descriptor: Cosine + dHash)
| Verdict | N (PDFs) | % | Firm A | Firm A % |
|---------|----------|---|--------|----------|
| High-confidence non-hand-signed | 29,529 | 35.0% | 22,970 | 76.0% |
| Moderate-confidence non-hand-signed | 36,994 | 43.8% | 6,311 | 20.9% |
| High style consistency | 5,133 | 6.1% | 183 | 0.6% |
| Uncertain | 12,683 | 15.0% | 758 | 2.5% |
| Likely hand-signed | 47 | 0.1% | 4 | 0.0% |
Per the worst-case aggregation rule of Section III-L, a document with two signatures inherits the most-replication-consistent of the two signature-level labels.
-->
Within the 71,656 documents exceeding cosine $0.95$, the dHash dimension stratifies them into three distinct populations:
29,529 (41.2%) show converging structural evidence of non-hand-signing (dHash $\leq 5$);
36,994 (51.7%) show partial structural similarity (dHash in $[6, 15]$) consistent with replication degraded by scan variations;
and 5,133 (7.2%) show no structural corroboration (dHash $> 15$), suggesting high signing consistency rather than image reproduction.
A cosine-only classifier would treat all 71,656 identically; the dual-descriptor framework separates them into populations with fundamentally different interpretations.
### 1) Firm A Capture Profile (Consistency Check)
96.9% of Firm A's documents fall into the high- or moderate-confidence non-hand-signed categories, 0.6% into high-style-consistency, and 2.5% into uncertain.
This pattern is consistent with the replication-dominated framing: the large majority is captured by non-hand-signed rules, while the small residual is consistent with the 32/171 middle-band minority identified by the accountant-level mixture (Section IV-E).
The absence of any meaningful "likely hand-signed" rate (4 of 30,000+ Firm A documents, 0.01%) implies either that Firm A's minority hand-signers have not been captured in the lowest-cosine tail---for example, because they also exhibit high style consistency---or that their contribution is small enough to be absorbed into the uncertain category at this threshold set.
We note that because the non-hand-signed thresholds are themselves calibrated to Firm A's empirical percentiles (Section III-H), these rates are an internal consistency check rather than an external validation; the held-out Firm A validation of Section IV-G.2 is the corresponding external check.
### 2) Cross-Method Agreement
Among non-Firm-A CPAs with cosine $> 0.95$, only 11.3% exhibit dHash $\leq 5$, compared to 58.7% for Firm A---a five-fold difference that demonstrates the discriminative power of the structural verification layer.
This is consistent with the three-method thresholds (Section IV-E, Table VIII) and with the cross-firm compositional pattern of the accountant-level GMM (Table VII).
## J. Ablation Study: Feature Backbone Comparison
To validate the choice of ResNet-50 as the feature extraction backbone, we conducted an ablation study comparing three pre-trained architectures: ResNet-50 (2048-dim), VGG-16 (4096-dim), and EfficientNet-B0 (1280-dim).
All models used ImageNet pre-trained weights without fine-tuning, with identical preprocessing and L2 normalization.
Table XVIII presents the comparison.
<!-- TABLE XVIII: Backbone Comparison
| Metric | ResNet-50 | VGG-16 | EfficientNet-B0 |
|--------|-----------|--------|-----------------|
| Feature dim | 2048 | 4096 | 1280 |
| Intra mean | 0.821 | 0.822 | 0.786 |
| Inter mean | 0.758 | 0.767 | 0.699 |
| Cohen's d | 0.669 | 0.564 | 0.707 |
| KDE crossover | 0.837 | 0.850 | 0.792 |
| Firm A mean (all-pairs) | 0.826 | 0.820 | 0.810 |
| Firm A 1st pct (all-pairs) | 0.543 | 0.520 | 0.454 |
Note: Firm A values in this table are computed over all intra-firm pairwise
similarities (16.0M pairs) for cross-backbone comparability. These differ from
the per-signature best-match values in Tables IV/VI (mean = 0.980), which reflect
the classification-relevant statistic: the similarity of each signature to its
single closest match from the same CPA.
-->
EfficientNet-B0 achieves the highest Cohen's $d$ (0.707), indicating the greatest statistical separation between intra-class and inter-class distributions.
However, it also exhibits the widest distributional spread (intra std $= 0.123$ vs. ResNet-50's $0.098$), resulting in lower per-sample classification confidence.
VGG-16 performs worst on all key metrics despite having the highest feature dimensionality (4096), suggesting that additional dimensions do not contribute discriminative information for this task.
ResNet-50 provides the best overall balance:
(1) Cohen's $d$ of 0.669 is competitive with EfficientNet-B0's 0.707;
(2) its tighter distributions yield more reliable individual classifications;
(3) the highest Firm A all-pairs 1st percentile (0.543) indicates that known-replication signatures are least likely to produce low-similarity outlier pairs under this backbone; and
(4) its 2048-dimensional features offer a practical compromise between discriminative capacity and computational/storage efficiency for processing 182K+ signatures.
signature_analysis/13_deloitte_distribution_analysis.py
+430
View File
@@ -0,0 +1,430 @@
#!/usr/bin/env python3
"""
Deloitte (勤業眾信) Signature Similarity Distribution Analysis
==============================================================
Evaluate whether Firm A's max_similarity values follow a normal distribution
or contain subgroups (e.g., genuinely hand-signed vs digitally stamped).
Tests:
1. Descriptive statistics & percentiles
2. Normality tests (Shapiro-Wilk, D'Agostino-Pearson, Anderson-Darling, KS)
3. Histogram + KDE + fitted normal overlay
4. Q-Q plot
5. Multimodality check (Hartigan's dip test approximation)
6. Outlier identification (signatures with unusually low similarity)
7. dHash distance distribution for Firm A
Output: figures + report to console
"""
import sqlite3
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from scipy import stats
from pathlib import Path
from collections import Counter
DB_PATH = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUTPUT_DIR = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/deloitte_distribution')
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
def load_firm_a_data():
"""Load all Firm A signature similarity data."""
conn = sqlite3.connect(DB_PATH)
cur = conn.cursor()
cur.execute('''
SELECT s.signature_id, s.image_filename, s.assigned_accountant,
s.max_similarity_to_same_accountant,
s.phash_distance_to_closest
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = ?
AND s.max_similarity_to_same_accountant IS NOT NULL
''', (FIRM_A,))
rows = cur.fetchall()
conn.close()
data = []
for r in rows:
data.append({
'sig_id': r[0],
'filename': r[1],
'accountant': r[2],
'cosine': r[3],
'phash': r[4],
})
return data
def descriptive_stats(cosines, label="Firm A Cosine Similarity"):
"""Print comprehensive descriptive statistics."""
print(f"\n{'='*65}")
print(f" {label}")
print(f"{'='*65}")
print(f" N = {len(cosines):,}")
print(f" Mean = {np.mean(cosines):.6f}")
print(f" Median = {np.median(cosines):.6f}")
print(f" Std Dev = {np.std(cosines):.6f}")
print(f" Variance = {np.var(cosines):.8f}")
print(f" Min = {np.min(cosines):.6f}")
print(f" Max = {np.max(cosines):.6f}")
print(f" Range = {np.ptp(cosines):.6f}")
print(f" Skewness = {stats.skew(cosines):.4f}")
print(f" Kurtosis = {stats.kurtosis(cosines):.4f} (excess)")
print(f" IQR = {np.percentile(cosines, 75) - np.percentile(cosines, 25):.6f}")
print()
print(f" Percentiles:")
for p in [1, 5, 10, 25, 50, 75, 90, 95, 99]:
print(f" P{p:<3d} = {np.percentile(cosines, p):.6f}")
def normality_tests(cosines):
"""Run multiple normality tests."""
print(f"\n{'='*65}")
print(f" NORMALITY TESTS")
print(f"{'='*65}")
# Shapiro-Wilk (max 5000 samples)
if len(cosines) > 5000:
sample = np.random.choice(cosines, 5000, replace=False)
stat, p = stats.shapiro(sample)
print(f"\n Shapiro-Wilk (n=5000 subsample):")
else:
stat, p = stats.shapiro(cosines)
print(f"\n Shapiro-Wilk (n={len(cosines)}):")
print(f" W = {stat:.6f}, p = {p:.2e}")
print(f" → {'Normal' if p > 0.05 else 'NOT normal'} at α=0.05")
# D'Agostino-Pearson
if len(cosines) >= 20:
stat, p = stats.normaltest(cosines)
print(f"\n D'Agostino-Pearson:")
print(f" K² = {stat:.4f}, p = {p:.2e}")
print(f" → {'Normal' if p > 0.05 else 'NOT normal'} at α=0.05")
# Anderson-Darling
result = stats.anderson(cosines, dist='norm')
print(f"\n Anderson-Darling:")
print(f" A² = {result.statistic:.4f}")
for i, (sl, cv) in enumerate(zip(result.significance_level, result.critical_values)):
reject = "REJECT" if result.statistic > cv else "accept"
print(f" {sl}%: critical={cv:.4f} → {reject}")
# Kolmogorov-Smirnov against normal
mu, sigma = np.mean(cosines), np.std(cosines)
stat, p = stats.kstest(cosines, 'norm', args=(mu, sigma))
print(f"\n Kolmogorov-Smirnov (vs fitted normal):")
print(f" D = {stat:.6f}, p = {p:.2e}")
print(f" → {'Normal' if p > 0.05 else 'NOT normal'} at α=0.05")
return mu, sigma
def test_alternative_distributions(cosines):
"""Fit alternative distributions and compare."""
print(f"\n{'='*65}")
print(f" DISTRIBUTION FITTING (AIC comparison)")
print(f"{'='*65}")
distributions = {
'norm': stats.norm,
'skewnorm': stats.skewnorm,
'beta': stats.beta,
'lognorm': stats.lognorm,
'gamma': stats.gamma,
}
results = []
for name, dist in distributions.items():
try:
params = dist.fit(cosines)
log_likelihood = np.sum(dist.logpdf(cosines, *params))
k = len(params)
aic = 2 * k - 2 * log_likelihood
bic = k * np.log(len(cosines)) - 2 * log_likelihood
results.append((name, aic, bic, params, log_likelihood))
except Exception as e:
print(f" {name}: fit failed ({e})")
results.sort(key=lambda x: x[1]) # sort by AIC
print(f"\n {'Distribution':<15} {'AIC':>12} {'BIC':>12} {'LogLik':>12}")
print(f" {'-'*51}")
for name, aic, bic, params, ll in results:
marker = " ←best" if name == results[0][0] else ""
print(f" {name:<15} {aic:>12.1f} {bic:>12.1f} {ll:>12.1f}{marker}")
return results
def per_accountant_analysis(data):
"""Analyze per-accountant distributions within Firm A."""
print(f"\n{'='*65}")
print(f" PER-ACCOUNTANT ANALYSIS (within Firm A)")
print(f"{'='*65}")
by_acct = {}
for d in data:
by_acct.setdefault(d['accountant'], []).append(d['cosine'])
print(f"\n {'Accountant':<20} {'N':>6} {'Mean':>8} {'Std':>8} {'Min':>8} {'P5':>8} {'P50':>8}")
print(f" {'-'*66}")
acct_stats = []
for acct, vals in sorted(by_acct.items(), key=lambda x: np.mean(x[1])):
v = np.array(vals)
print(f" {acct:<20} {len(v):>6} {v.mean():>8.4f} {v.std():>8.4f} "
f"{v.min():>8.4f} {np.percentile(v, 5):>8.4f} {np.median(v):>8.4f}")
acct_stats.append({
'accountant': acct,
'n': len(v),
'mean': float(v.mean()),
'std': float(v.std()),
'min': float(v.min()),
'values': v,
})
# Check if per-accountant means are homogeneous (one-way ANOVA)
if len(by_acct) >= 2:
groups = [np.array(v) for v in by_acct.values() if len(v) >= 5]
if len(groups) >= 2:
f_stat, p_val = stats.f_oneway(*groups)
print(f"\n One-way ANOVA across accountants:")
print(f" F = {f_stat:.4f}, p = {p_val:.2e}")
print(f" → {'Homogeneous' if p_val > 0.05 else 'Significantly different means'} at α=0.05")
# Levene's test for homogeneity of variance
lev_stat, lev_p = stats.levene(*groups)
print(f"\n Levene's test (variance homogeneity):")
print(f" W = {lev_stat:.4f}, p = {lev_p:.2e}")
print(f" → {'Homogeneous variance' if lev_p > 0.05 else 'Heterogeneous variance'} at α=0.05")
return acct_stats
def identify_outliers(data, cosines):
"""Identify Firm A signatures with unusually low similarity."""
print(f"\n{'='*65}")
print(f" OUTLIER ANALYSIS (low-similarity Firm A signatures)")
print(f"{'='*65}")
q1 = np.percentile(cosines, 25)
q3 = np.percentile(cosines, 75)
iqr = q3 - q1
lower_fence = q1 - 1.5 * iqr
lower_extreme = q1 - 3.0 * iqr
print(f" IQR method: Q1={q1:.4f}, Q3={q3:.4f}, IQR={iqr:.4f}")
print(f" Lower fence (mild): {lower_fence:.4f}")
print(f" Lower fence (extreme): {lower_extreme:.4f}")
outliers = [d for d in data if d['cosine'] < lower_fence]
extreme_outliers = [d for d in data if d['cosine'] < lower_extreme]
print(f"\n Mild outliers (< {lower_fence:.4f}): {len(outliers)}")
print(f" Extreme outliers (< {lower_extreme:.4f}): {len(extreme_outliers)}")
if outliers:
print(f"\n Bottom 20 by cosine similarity:")
sorted_outliers = sorted(outliers, key=lambda x: x['cosine'])[:20]
for d in sorted_outliers:
phash_str = f"pHash={d['phash']}" if d['phash'] is not None else "pHash=N/A"
print(f" cosine={d['cosine']:.4f} {phash_str} {d['accountant']} {d['filename']}")
# Also show count below various thresholds
print(f"\n Signatures below key thresholds:")
for thresh in [0.95, 0.90, 0.85, 0.837, 0.80]:
n_below = sum(1 for c in cosines if c < thresh)
print(f" < {thresh:.3f}: {n_below:,} ({100*n_below/len(cosines):.2f}%)")
def plot_histogram_kde(cosines, mu, sigma):
"""Plot histogram with KDE and fitted normal overlay."""
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# Left: Full histogram
ax = axes[0]
ax.hist(cosines, bins=80, density=True, alpha=0.6, color='steelblue',
edgecolor='white', linewidth=0.5, label='Observed')
# Fitted normal
x = np.linspace(cosines.min() - 0.02, cosines.max() + 0.02, 300)
ax.plot(x, stats.norm.pdf(x, mu, sigma), 'r-', lw=2,
label=f'Normal fit (μ={mu:.4f}, σ={sigma:.4f})')
# KDE
kde = stats.gaussian_kde(cosines)
ax.plot(x, kde(x), 'g--', lw=2, label='KDE')
ax.set_xlabel('Max Cosine Similarity')
ax.set_ylabel('Density')
ax.set_title(f'Firm A (勤業眾信) Cosine Similarity Distribution (N={len(cosines):,})')
ax.legend(fontsize=9)
ax.axvline(0.95, color='orange', ls=':', alpha=0.7, label='θ=0.95')
ax.axvline(0.837, color='purple', ls=':', alpha=0.7, label='KDE crossover')
# Right: Q-Q plot
ax2 = axes[1]
stats.probplot(cosines, dist='norm', plot=ax2)
ax2.set_title('Q-Q Plot (vs Normal)')
ax2.get_lines()[0].set_markersize(2)
plt.tight_layout()
fig.savefig(OUTPUT_DIR / 'firm_a_cosine_distribution.png', dpi=150)
print(f"\n Saved: {OUTPUT_DIR / 'firm_a_cosine_distribution.png'}")
plt.close()
def plot_per_accountant(acct_stats):
"""Box plot per accountant."""
# Sort by mean
acct_stats.sort(key=lambda x: x['mean'])
fig, ax = plt.subplots(figsize=(12, max(5, len(acct_stats) * 0.4)))
positions = range(len(acct_stats))
labels = [f"{a['accountant']} (n={a['n']})" for a in acct_stats]
box_data = [a['values'] for a in acct_stats]
bp = ax.boxplot(box_data, positions=positions, vert=False, widths=0.6,
patch_artist=True, showfliers=True,
flierprops=dict(marker='.', markersize=3, alpha=0.5))
for patch in bp['boxes']:
patch.set_facecolor('lightsteelblue')
ax.set_yticks(positions)
ax.set_yticklabels(labels, fontsize=8)
ax.set_xlabel('Max Cosine Similarity')
ax.set_title('Per-Accountant Similarity Distribution (Firm A)')
ax.axvline(0.95, color='orange', ls=':', alpha=0.7)
ax.axvline(0.837, color='purple', ls=':', alpha=0.7)
plt.tight_layout()
fig.savefig(OUTPUT_DIR / 'firm_a_per_accountant_boxplot.png', dpi=150)
print(f" Saved: {OUTPUT_DIR / 'firm_a_per_accountant_boxplot.png'}")
plt.close()
def plot_phash_distribution(data):
"""Plot dHash distance distribution for Firm A."""
phash_vals = [d['phash'] for d in data if d['phash'] is not None]
if not phash_vals:
print(" No pHash data available.")
return
phash_arr = np.array(phash_vals)
fig, ax = plt.subplots(figsize=(10, 5))
max_val = min(int(phash_arr.max()) + 2, 65)
bins = np.arange(-0.5, max_val + 0.5, 1)
ax.hist(phash_arr, bins=bins, alpha=0.7, color='coral', edgecolor='white')
ax.set_xlabel('dHash Distance')
ax.set_ylabel('Count')
ax.set_title(f'Firm A dHash Distance Distribution (N={len(phash_vals):,})')
ax.axvline(5, color='green', ls='--', label='θ=5 (high conf.)')
ax.axvline(15, color='orange', ls='--', label='θ=15 (moderate)')
ax.legend()
plt.tight_layout()
fig.savefig(OUTPUT_DIR / 'firm_a_dhash_distribution.png', dpi=150)
print(f" Saved: {OUTPUT_DIR / 'firm_a_dhash_distribution.png'}")
plt.close()
def multimodality_test(cosines):
"""Check for potential multimodality using kernel density peaks."""
print(f"\n{'='*65}")
print(f" MULTIMODALITY ANALYSIS")
print(f"{'='*65}")
kde = stats.gaussian_kde(cosines, bw_method='silverman')
x = np.linspace(cosines.min(), cosines.max(), 1000)
density = kde(x)
# Find local maxima
from scipy.signal import find_peaks
peaks, properties = find_peaks(density, prominence=0.01)
peak_positions = x[peaks]
peak_heights = density[peaks]
print(f" KDE bandwidth (Silverman): {kde.factor:.6f}")
print(f" Number of detected modes: {len(peaks)}")
for i, (pos, h) in enumerate(zip(peak_positions, peak_heights)):
print(f" Mode {i+1}: position={pos:.4f}, density={h:.2f}")
if len(peaks) == 1:
print(f"\n → Distribution appears UNIMODAL")
print(f" Single peak at {peak_positions[0]:.4f}")
elif len(peaks) > 1:
print(f"\n → Distribution appears MULTIMODAL ({len(peaks)} modes)")
print(f" This suggests subgroups may exist within Firm A")
# Check separation between modes
for i in range(len(peaks) - 1):
sep = peak_positions[i + 1] - peak_positions[i]
# Find valley between modes
valley_region = density[peaks[i]:peaks[i + 1]]
valley_depth = peak_heights[i:i + 2].min() - valley_region.min()
print(f" Separation {i+1}-{i+2}: Δ={sep:.4f}, valley depth={valley_depth:.2f}")
# Also try different bandwidths
print(f"\n Sensitivity analysis (bandwidth variation):")
for bw_factor in [0.5, 0.75, 1.0, 1.5, 2.0]:
bw = kde.factor * bw_factor
kde_test = stats.gaussian_kde(cosines, bw_method=bw)
density_test = kde_test(x)
peaks_test, _ = find_peaks(density_test, prominence=0.005)
print(f" bw={bw:.4f} (×{bw_factor:.1f}): {len(peaks_test)} mode(s)")
def main():
print("Loading Firm A (勤業眾信) signature data...")
data = load_firm_a_data()
print(f"Total Firm A signatures: {len(data):,}")
cosines = np.array([d['cosine'] for d in data])
# 1. Descriptive statistics
descriptive_stats(cosines)
# 2. Normality tests
mu, sigma = normality_tests(cosines)
# 3. Alternative distribution fitting
test_alternative_distributions(cosines)
# 4. Per-accountant analysis
acct_stats = per_accountant_analysis(data)
# 5. Outlier analysis
identify_outliers(data, cosines)
# 6. Multimodality test
multimodality_test(cosines)
# 7. Generate plots
print(f"\n{'='*65}")
print(f" GENERATING FIGURES")
print(f"{'='*65}")
plot_histogram_kde(cosines, mu, sigma)
plot_per_accountant(acct_stats)
plot_phash_distribution(data)
# Summary
print(f"\n{'='*65}")
print(f" SUMMARY")
print(f"{'='*65}")
below_95 = sum(1 for c in cosines if c < 0.95)
below_kde = sum(1 for c in cosines if c < 0.837)
print(f" Firm A signatures: {len(cosines):,}")
print(f" Below 0.95 threshold: {below_95:,} ({100*below_95/len(cosines):.1f}%)")
print(f" Below KDE crossover (0.837): {below_kde:,} ({100*below_kde/len(cosines):.1f}%)")
print(f" If distribution is NOT normal → subgroups may exist")
print(f" If multimodal → some signatures may be genuinely hand-signed")
print(f"\n Output directory: {OUTPUT_DIR}")
if __name__ == "__main__":
main()
signature_analysis/14_compute_independent_dhash.py
+293
View File
@@ -0,0 +1,293 @@
#!/usr/bin/env python3
"""
Compute independent min dHash for all signatures.
===================================================
Currently phash_distance_to_closest is conditional on cosine-nearest pair.
This script computes an INDEPENDENT min dHash: for each signature, find the
pair within the same accountant that has the smallest dHash distance,
regardless of cosine similarity.
Three metrics after this script:
1. max_similarity_to_same_accountant (max cosine) — primary classifier
2. min_dhash_independent (independent min) — independent 2nd classifier
3. phash_distance_to_closest (conditional) — diagnostic tool
Phase 1: Compute dHash vector for each image, store as BLOB in DB
Phase 2: All-pairs hamming distance within same accountant, store min
"""
import sqlite3
import numpy as np
import cv2
import os
import sys
import time
from multiprocessing import Pool, cpu_count
from pathlib import Path
DB_PATH = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
IMAGE_DIR = '/Volumes/NV2/PDF-Processing/yolo-signatures/images'
NUM_WORKERS = max(1, cpu_count() - 2)
BATCH_SIZE = 5000
HASH_SIZE = 8 # 9x8 -> 8x8 = 64-bit hash
# ── Phase 1: Compute dHash per image ─────────────────────────────────
def compute_dhash_for_file(args):
"""Compute dHash for a single image file. Returns (sig_id, hash_bytes) or (sig_id, None)."""
sig_id, filename = args
path = os.path.join(IMAGE_DIR, filename)
try:
img = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
if img is None:
return (sig_id, None)
resized = cv2.resize(img, (HASH_SIZE + 1, HASH_SIZE))
diff = resized[:, 1:] > resized[:, :-1] # 8x8 = 64 bits
return (sig_id, np.packbits(diff.flatten()).tobytes())
except Exception:
return (sig_id, None)
def phase1_compute_hashes():
"""Compute and store dHash for all signatures."""
conn = sqlite3.connect(DB_PATH)
cur = conn.cursor()
# Add columns if not exist
for col in ['dhash_vector BLOB', 'min_dhash_independent INTEGER',
'min_dhash_independent_match TEXT']:
try:
cur.execute(f'ALTER TABLE signatures ADD COLUMN {col}')
except sqlite3.OperationalError:
pass
conn.commit()
# Check which signatures already have dhash_vector
cur.execute('''
SELECT signature_id, image_filename
FROM signatures
WHERE feature_vector IS NOT NULL
AND assigned_accountant IS NOT NULL
AND dhash_vector IS NULL
''')
todo = cur.fetchall()
if not todo:
# Check total with dhash
cur.execute('SELECT COUNT(*) FROM signatures WHERE dhash_vector IS NOT NULL')
n_done = cur.fetchone()[0]
print(f" Phase 1 already complete ({n_done:,} hashes in DB)")
conn.close()
return
print(f" Computing dHash for {len(todo):,} images ({NUM_WORKERS} workers)...")
t0 = time.time()
processed = 0
for batch_start in range(0, len(todo), BATCH_SIZE):
batch = todo[batch_start:batch_start + BATCH_SIZE]
with Pool(NUM_WORKERS) as pool:
results = pool.map(compute_dhash_for_file, batch)
updates = [(dhash, sid) for sid, dhash in results if dhash is not None]
cur.executemany('UPDATE signatures SET dhash_vector = ? WHERE signature_id = ?', updates)
conn.commit()
processed += len(batch)
elapsed = time.time() - t0
rate = processed / elapsed
eta = (len(todo) - processed) / rate if rate > 0 else 0
print(f" {processed:,}/{len(todo):,} ({rate:.0f}/s, ETA {eta:.0f}s)")
conn.close()
elapsed = time.time() - t0
print(f" Phase 1 done: {processed:,} hashes in {elapsed:.1f}s")
# ── Phase 2: All-pairs min dHash within same accountant ──────────────
def hamming_distance(h1_bytes, h2_bytes):
"""Hamming distance between two packed dHash byte strings."""
a = np.frombuffer(h1_bytes, dtype=np.uint8)
b = np.frombuffer(h2_bytes, dtype=np.uint8)
xor = np.bitwise_xor(a, b)
return sum(bin(byte).count('1') for byte in xor)
def phase2_compute_min_dhash():
"""For each accountant group, find the min dHash pair per signature."""
conn = sqlite3.connect(DB_PATH)
cur = conn.cursor()
# Load all signatures with dhash
cur.execute('''
SELECT s.signature_id, s.assigned_accountant, s.dhash_vector, s.image_filename
FROM signatures s
WHERE s.dhash_vector IS NOT NULL
AND s.assigned_accountant IS NOT NULL
''')
rows = cur.fetchall()
print(f" Loaded {len(rows):,} signatures with dHash")
# Group by accountant
acct_groups = {}
for sig_id, acct, dhash, filename in rows:
acct_groups.setdefault(acct, []).append((sig_id, dhash, filename))
# Filter out singletons
acct_groups = {k: v for k, v in acct_groups.items() if len(v) >= 2}
total_sigs = sum(len(v) for v in acct_groups.values())
total_pairs = sum(len(v) * (len(v) - 1) // 2 for v in acct_groups.values())
print(f" {len(acct_groups)} accountants, {total_sigs:,} signatures, {total_pairs:,} pairs")
t0 = time.time()
updates = []
accts_done = 0
for acct, sigs in acct_groups.items():
n = len(sigs)
sig_ids = [s[0] for s in sigs]
hashes = [s[1] for s in sigs]
filenames = [s[2] for s in sigs]
# Unpack all hashes to bit arrays for vectorized hamming
bits = np.array([np.unpackbits(np.frombuffer(h, dtype=np.uint8)) for h in hashes],
dtype=np.uint8) # shape: (n, 64)
# Pairwise hamming via XOR + sum
# For groups up to ~2000, direct matrix computation is fine
# hamming_matrix[i,j] = number of differing bits between i and j
xor_matrix = bits[:, None, :] ^ bits[None, :, :] # (n, n, 64)
hamming_matrix = xor_matrix.sum(axis=2) # (n, n)
np.fill_diagonal(hamming_matrix, 999) # exclude self
# For each signature, find min
min_indices = np.argmin(hamming_matrix, axis=1)
min_distances = hamming_matrix[np.arange(n), min_indices]
for i in range(n):
updates.append((
int(min_distances[i]),
filenames[min_indices[i]],
sig_ids[i]
))
accts_done += 1
if accts_done % 100 == 0:
elapsed = time.time() - t0
print(f" {accts_done}/{len(acct_groups)} accountants ({elapsed:.0f}s)")
# Write to DB
print(f" Writing {len(updates):,} results to DB...")
cur.executemany('''
UPDATE signatures
SET min_dhash_independent = ?, min_dhash_independent_match = ?
WHERE signature_id = ?
''', updates)
conn.commit()
conn.close()
elapsed = time.time() - t0
print(f" Phase 2 done: {len(updates):,} signatures in {elapsed:.1f}s")
# ── Phase 3: Summary statistics ──────────────────────────────────────
def print_summary():
"""Print summary comparing conditional vs independent dHash."""
conn = sqlite3.connect(DB_PATH)
cur = conn.cursor()
# Overall stats
cur.execute('''
SELECT
COUNT(*) as n,
AVG(phash_distance_to_closest) as cond_mean,
AVG(min_dhash_independent) as indep_mean
FROM signatures
WHERE min_dhash_independent IS NOT NULL
AND phash_distance_to_closest IS NOT NULL
''')
n, cond_mean, indep_mean = cur.fetchone()
print(f"\n{'='*65}")
print(f" COMPARISON: Conditional vs Independent dHash")
print(f"{'='*65}")
print(f" N = {n:,}")
print(f" Conditional dHash (cosine-nearest pair): mean = {cond_mean:.2f}")
print(f" Independent dHash (all-pairs min): mean = {indep_mean:.2f}")
# Percentiles
cur.execute('''
SELECT phash_distance_to_closest, min_dhash_independent
FROM signatures
WHERE min_dhash_independent IS NOT NULL
AND phash_distance_to_closest IS NOT NULL
''')
rows = cur.fetchall()
cond = np.array([r[0] for r in rows])
indep = np.array([r[1] for r in rows])
print(f"\n {'Percentile':<12} {'Conditional':>12} {'Independent':>12} {'Diff':>8}")
print(f" {'-'*44}")
for p in [1, 5, 10, 25, 50, 75, 90, 95, 99]:
cv = np.percentile(cond, p)
iv = np.percentile(indep, p)
print(f" P{p:<10d} {cv:>12.1f} {iv:>12.1f} {iv-cv:>+8.1f}")
# Agreement analysis
print(f"\n Agreement analysis (both ≤ threshold):")
for t in [5, 10, 15, 21]:
both = np.sum((cond <= t) & (indep <= t))
cond_only = np.sum((cond <= t) & (indep > t))
indep_only = np.sum((cond > t) & (indep <= t))
neither = np.sum((cond > t) & (indep > t))
agree_pct = (both + neither) / len(cond) * 100
print(f" θ={t:>2d}: both={both:,}, cond_only={cond_only:,}, "
f"indep_only={indep_only:,}, neither={neither:,} (agree={agree_pct:.1f}%)")
# Firm A specific
cur.execute('''
SELECT s.phash_distance_to_closest, s.min_dhash_independent
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = '勤業眾信聯合'
AND s.min_dhash_independent IS NOT NULL
AND s.phash_distance_to_closest IS NOT NULL
''')
rows = cur.fetchall()
if rows:
cond_a = np.array([r[0] for r in rows])
indep_a = np.array([r[1] for r in rows])
print(f"\n Firm A (勤業眾信) — N={len(rows):,}:")
print(f" {'Percentile':<12} {'Conditional':>12} {'Independent':>12}")
print(f" {'-'*36}")
for p in [50, 75, 90, 95, 99]:
print(f" P{p:<10d} {np.percentile(cond_a, p):>12.1f} {np.percentile(indep_a, p):>12.1f}")
conn.close()
def main():
t_start = time.time()
print("=" * 65)
print(" Independent Min dHash Computation")
print("=" * 65)
print(f"\n[Phase 1] Computing dHash vectors...")
phase1_compute_hashes()
print(f"\n[Phase 2] Computing all-pairs min dHash per accountant...")
phase2_compute_min_dhash()
print(f"\n[Phase 3] Summary...")
print_summary()
elapsed = time.time() - t_start
print(f"\nTotal time: {elapsed:.0f}s ({elapsed/60:.1f} min)")
if __name__ == "__main__":
main()
signature_analysis/15_hartigan_dip_test.py
+238
View File
@@ -0,0 +1,238 @@
#!/usr/bin/env python3
"""
Script 15: Hartigan Dip Test for Unimodality
=============================================
Runs the proper Hartigan & Hartigan (1985) dip test via the `diptest` package
on the empirical signature-similarity distributions.
Purpose:
Confirm/refute bimodality assumption underpinning threshold-selection methods.
Prior finding (2026-04-16): signature-level distribution is unimodal long-tail;
the story is that bimodality only emerges at the accountant level.
Firm A framing (2026-04-20, corrected):
Interviews with multiple Firm A accountants confirm that MOST use
replication (stamping / firm-level e-signing) but do NOT exclude a
minority of hand-signers. Firm A is therefore a "replication-dominated"
population, NOT a "pure" one. This framing is consistent with:
- 92.5% of Firm A signatures exceed cosine 0.95
- The long left tail (7.5% below 0.95) captures the minority
hand-signers, not scan noise
- Script 18: of 180 Firm A accountants, 139 cluster in C1
(high-replication) and 32 in C2 (middle band = minority hand-signers)
Tests:
1. Firm A (Deloitte) cosine max-similarity -> expected UNIMODAL
2. Firm A (Deloitte) independent min dHash -> expected UNIMODAL
3. Full-sample cosine max-similarity -> test
4. Full-sample independent min dHash -> test
5. Accountant-level cosine mean (per-accountant) -> expected BIMODAL / MULTIMODAL
6. Accountant-level dhash mean (per-accountant) -> expected BIMODAL / MULTIMODAL
Output:
reports/dip_test/dip_test_report.md
reports/dip_test/dip_test_results.json
"""
import sqlite3
import json
import numpy as np
import diptest
from pathlib import Path
from datetime import datetime
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/dip_test')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
def run_dip(values, label, n_boot=2000):
"""Run Hartigan dip test and return structured result."""
arr = np.asarray(values, dtype=float)
arr = arr[~np.isnan(arr)]
if len(arr) < 4:
return {'label': label, 'n': int(len(arr)), 'error': 'too few observations'}
dip, pval = diptest.diptest(arr, boot_pval=True, n_boot=n_boot)
verdict = 'UNIMODAL (accept H0)' if pval > 0.05 else 'MULTIMODAL (reject H0)'
return {
'label': label,
'n': int(len(arr)),
'mean': float(np.mean(arr)),
'std': float(np.std(arr)),
'min': float(np.min(arr)),
'max': float(np.max(arr)),
'dip': float(dip),
'p_value': float(pval),
'n_boot': int(n_boot),
'verdict_alpha_05': verdict,
}
def fetch_firm_a():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.max_similarity_to_same_accountant,
s.min_dhash_independent
FROM signatures s
JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = ?
AND s.max_similarity_to_same_accountant IS NOT NULL
''', (FIRM_A,))
rows = cur.fetchall()
conn.close()
cos = [r[0] for r in rows if r[0] is not None]
dh = [r[1] for r in rows if r[1] is not None]
return np.array(cos), np.array(dh)
def fetch_full_sample():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT max_similarity_to_same_accountant, min_dhash_independent
FROM signatures
WHERE max_similarity_to_same_accountant IS NOT NULL
''')
rows = cur.fetchall()
conn.close()
cos = np.array([r[0] for r in rows if r[0] is not None])
dh = np.array([r[1] for r in rows if r[1] is not None])
return cos, dh
def fetch_accountant_aggregates(min_sigs=10):
"""Per-accountant mean cosine and mean independent dHash."""
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.assigned_accountant,
AVG(s.max_similarity_to_same_accountant) AS cos_mean,
AVG(CAST(s.min_dhash_independent AS REAL)) AS dh_mean,
COUNT(*) AS n
FROM signatures s
WHERE s.assigned_accountant IS NOT NULL
AND s.max_similarity_to_same_accountant IS NOT NULL
AND s.min_dhash_independent IS NOT NULL
GROUP BY s.assigned_accountant
HAVING n >= ?
''', (min_sigs,))
rows = cur.fetchall()
conn.close()
cos_means = np.array([r[1] for r in rows])
dh_means = np.array([r[2] for r in rows])
return cos_means, dh_means, len(rows)
def main():
print('='*70)
print('Script 15: Hartigan Dip Test for Unimodality')
print('='*70)
results = {}
# Firm A
print('\n[1/3] Firm A (Deloitte)...')
fa_cos, fa_dh = fetch_firm_a()
print(f' Firm A cosine N={len(fa_cos):,}, dHash N={len(fa_dh):,}')
results['firm_a_cosine'] = run_dip(fa_cos, 'Firm A cosine max-similarity')
results['firm_a_dhash'] = run_dip(fa_dh, 'Firm A independent min dHash')
# Full sample
print('\n[2/3] Full sample...')
all_cos, all_dh = fetch_full_sample()
print(f' Full cosine N={len(all_cos):,}, dHash N={len(all_dh):,}')
# Dip test on >=10k obs can be slow with 2000 boot; use 500 for full sample
results['full_cosine'] = run_dip(all_cos, 'Full-sample cosine max-similarity',
n_boot=500)
results['full_dhash'] = run_dip(all_dh, 'Full-sample independent min dHash',
n_boot=500)
# Accountant-level aggregates
print('\n[3/3] Accountant-level aggregates (min 10 sigs)...')
acct_cos, acct_dh, n_acct = fetch_accountant_aggregates(min_sigs=10)
print(f' Accountants analyzed: {n_acct}')
results['accountant_cos_mean'] = run_dip(acct_cos,
'Per-accountant cosine mean')
results['accountant_dh_mean'] = run_dip(acct_dh,
'Per-accountant dHash mean')
# Print summary
print('\n' + '='*70)
print('RESULTS SUMMARY')
print('='*70)
print(f"{'Test':<40} {'N':>8} {'dip':>8} {'p':>10} Verdict")
print('-'*90)
for key, r in results.items():
if 'error' in r:
continue
print(f"{r['label']:<40} {r['n']:>8,} {r['dip']:>8.4f} "
f"{r['p_value']:>10.4f} {r['verdict_alpha_05']}")
# Write JSON
json_path = OUT / 'dip_test_results.json'
with open(json_path, 'w') as f:
json.dump({
'generated_at': datetime.now().isoformat(),
'db': DB,
'results': results,
}, f, indent=2, ensure_ascii=False)
print(f'\nJSON saved: {json_path}')
# Write Markdown report
md = [
'# Hartigan Dip Test Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Method',
'',
'Hartigan & Hartigan (1985) dip test via `diptest` Python package.',
'H0: distribution is unimodal. H1: multimodal (two or more modes).',
'p-value computed by bootstrap against a uniform null (2000 reps for',
'Firm A/accountant-level, 500 reps for full-sample due to size).',
'',
'## Results',
'',
'| Test | N | dip | p-value | Verdict (α=0.05) |',
'|------|---|-----|---------|------------------|',
]
for r in results.values():
if 'error' in r:
md.append(f"| {r['label']} | {r['n']} | — | — | {r['error']} |")
continue
md.append(
f"| {r['label']} | {r['n']:,} | {r['dip']:.4f} | "
f"{r['p_value']:.4f} | {r['verdict_alpha_05']} |"
)
md += [
'',
'## Interpretation',
'',
'* **Signature level** (Firm A + full sample): the dip test indicates',
' whether a single mode explains the max-cosine/min-dHash distribution.',
' Prior finding (2026-04-16) suggested unimodal long-tail; this script',
' provides the formal test.',
'',
'* **Accountant level** (per-accountant mean): if multimodal here but',
' unimodal at the signature level, this confirms the interpretation',
" that signing-behaviour is discrete across accountants (replication",
' vs hand-signing), while replication quality itself is a continuous',
' spectrum.',
'',
'## Downstream implication',
'',
'Methods that assume bimodality (KDE antimode, 2-component Beta mixture)',
'should be applied at the level where dip test rejects H0. If the',
"signature-level dip test fails to reject, the paper should report this",
'and shift the mixture analysis to the accountant level (see Script 18).',
]
md_path = OUT / 'dip_test_report.md'
md_path.write_text('\n'.join(md), encoding='utf-8')
print(f'Report saved: {md_path}')
if __name__ == '__main__':
main()
signature_analysis/16_bd_mccrary_discontinuity.py
+320
View File
@@ -0,0 +1,320 @@
#!/usr/bin/env python3
"""
Script 16: Burgstahler-Dichev / McCrary Discontinuity Test
==========================================================
Tests for a discontinuity in the empirical density of similarity scores,
following:
- Burgstahler & Dichev (1997) - earnings-management style smoothness test
- McCrary (2008) - rigorous density-discontinuity asymptotics
Idea:
Discretize the distribution into equal-width bins. For each bin i compute
the standardized deviation Z_i between observed count and the smooth
expectation (average of neighbours). Under H0 (distributional smoothness),
Z_i ~ N(0,1). A threshold is identified at the transition where Z_{i-1}
is significantly negative (below expectation) next to Z_i significantly
positive (above expectation) -- marking the boundary between two
generative mechanisms (hand-signed vs non-hand-signed).
Inputs:
- Firm A cosine max-similarity and independent min dHash
- Full-sample cosine and dHash (for comparison)
Output:
reports/bd_mccrary/bd_mccrary_report.md
reports/bd_mccrary/bd_mccrary_results.json
reports/bd_mccrary/bd_mccrary_<variant>.png (overlay plots)
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/bd_mccrary')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
# BD/McCrary critical values (two-sided, alpha=0.05)
Z_CRIT = 1.96
def bd_mccrary(values, bin_width, lo=None, hi=None):
"""
Compute Burgstahler-Dichev standardized deviations per bin.
For each bin i with count n_i:
expected = 0.5 * (n_{i-1} + n_{i+1})
SE = sqrt(N*p_i*(1-p_i) + 0.25*N*(p_{i-1}+p_{i+1})*(1-p_{i-1}-p_{i+1}))
Z_i = (n_i - expected) / SE
Returns arrays of (bin_centers, counts, z_scores, expected).
"""
arr = np.asarray(values, dtype=float)
arr = arr[~np.isnan(arr)]
if lo is None:
lo = float(np.floor(arr.min() / bin_width) * bin_width)
if hi is None:
hi = float(np.ceil(arr.max() / bin_width) * bin_width)
edges = np.arange(lo, hi + bin_width, bin_width)
counts, _ = np.histogram(arr, bins=edges)
centers = (edges[:-1] + edges[1:]) / 2.0
N = counts.sum()
p = counts / N if N else counts.astype(float)
n_bins = len(counts)
z = np.full(n_bins, np.nan)
expected = np.full(n_bins, np.nan)
for i in range(1, n_bins - 1):
p_lo = p[i - 1]
p_hi = p[i + 1]
exp_i = 0.5 * (counts[i - 1] + counts[i + 1])
var_i = (N * p[i] * (1 - p[i])
+ 0.25 * N * (p_lo + p_hi) * (1 - p_lo - p_hi))
if var_i <= 0:
continue
se = np.sqrt(var_i)
z[i] = (counts[i] - exp_i) / se
expected[i] = exp_i
return centers, counts, z, expected
def find_transition(centers, z, direction='neg_to_pos'):
"""
Find the first bin pair where Z_{i-1} significantly negative and
Z_i significantly positive (or vice versa).
direction='neg_to_pos' -> threshold where hand-signed density drops
(below expectation) and non-hand-signed
density rises (above expectation). For
cosine similarity, this transition is
expected around the separation point, so
the threshold sits between centers[i-1]
and centers[i].
"""
transitions = []
for i in range(1, len(z)):
if np.isnan(z[i - 1]) or np.isnan(z[i]):
continue
if direction == 'neg_to_pos':
if z[i - 1] < -Z_CRIT and z[i] > Z_CRIT:
transitions.append({
'idx': int(i),
'threshold_between': float(
(centers[i - 1] + centers[i]) / 2.0),
'z_below': float(z[i - 1]),
'z_above': float(z[i]),
'left_center': float(centers[i - 1]),
'right_center': float(centers[i]),
})
else: # pos_to_neg
if z[i - 1] > Z_CRIT and z[i] < -Z_CRIT:
transitions.append({
'idx': int(i),
'threshold_between': float(
(centers[i - 1] + centers[i]) / 2.0),
'z_above': float(z[i - 1]),
'z_below': float(z[i]),
'left_center': float(centers[i - 1]),
'right_center': float(centers[i]),
})
return transitions
def plot_bd(centers, counts, z, expected, title, out_path, threshold=None):
fig, axes = plt.subplots(2, 1, figsize=(11, 7), sharex=True)
ax = axes[0]
ax.bar(centers, counts, width=(centers[1] - centers[0]) * 0.9,
color='steelblue', alpha=0.6, edgecolor='white', label='Observed')
mask = ~np.isnan(expected)
ax.plot(centers[mask], expected[mask], 'r-', lw=1.5,
label='Expected (smooth null)')
ax.set_ylabel('Count')
ax.set_title(title)
ax.legend()
if threshold is not None:
ax.axvline(threshold, color='green', ls='--', lw=2,
label=f'Threshold≈{threshold:.4f}')
ax = axes[1]
ax.axhline(0, color='black', lw=0.5)
ax.axhline(Z_CRIT, color='red', ls=':', alpha=0.7,
label=f'±{Z_CRIT} critical')
ax.axhline(-Z_CRIT, color='red', ls=':', alpha=0.7)
colors = ['coral' if zi > Z_CRIT else 'steelblue' if zi < -Z_CRIT
else 'lightgray' for zi in z]
ax.bar(centers, z, width=(centers[1] - centers[0]) * 0.9, color=colors,
edgecolor='black', lw=0.3)
ax.set_xlabel('Value')
ax.set_ylabel('Z statistic')
ax.legend()
if threshold is not None:
ax.axvline(threshold, color='green', ls='--', lw=2)
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
def fetch(label):
conn = sqlite3.connect(DB)
cur = conn.cursor()
if label == 'firm_a_cosine':
cur.execute('''
SELECT s.max_similarity_to_same_accountant
FROM signatures s
JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = ? AND s.max_similarity_to_same_accountant IS NOT NULL
''', (FIRM_A,))
elif label == 'firm_a_dhash':
cur.execute('''
SELECT s.min_dhash_independent
FROM signatures s
JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = ? AND s.min_dhash_independent IS NOT NULL
''', (FIRM_A,))
elif label == 'full_cosine':
cur.execute('''
SELECT max_similarity_to_same_accountant FROM signatures
WHERE max_similarity_to_same_accountant IS NOT NULL
''')
elif label == 'full_dhash':
cur.execute('''
SELECT min_dhash_independent FROM signatures
WHERE min_dhash_independent IS NOT NULL
''')
else:
raise ValueError(label)
vals = [r[0] for r in cur.fetchall() if r[0] is not None]
conn.close()
return np.array(vals, dtype=float)
def main():
print('='*70)
print('Script 16: Burgstahler-Dichev / McCrary Discontinuity Test')
print('='*70)
cases = [
('firm_a_cosine', 0.005, 'Firm A cosine max-similarity', 'neg_to_pos'),
('firm_a_dhash', 1.0, 'Firm A independent min dHash', 'pos_to_neg'),
('full_cosine', 0.005, 'Full-sample cosine max-similarity',
'neg_to_pos'),
('full_dhash', 1.0, 'Full-sample independent min dHash', 'pos_to_neg'),
]
all_results = {}
for key, bw, label, direction in cases:
print(f'\n[{label}] bin width={bw}')
arr = fetch(key)
print(f' N = {len(arr):,}')
centers, counts, z, expected = bd_mccrary(arr, bw)
transitions = find_transition(centers, z, direction=direction)
# Summarize
if transitions:
# Choose the most extreme (highest |z_above * z_below|) transition
best = max(transitions,
key=lambda t: abs(t.get('z_above', 0))
+ abs(t.get('z_below', 0)))
threshold = best['threshold_between']
print(f' {len(transitions)} candidate transition(s); '
f'best at {threshold:.4f}')
else:
best = None
threshold = None
print(' No significant transition detected (no Z^- next to Z^+)')
# Plot
png = OUT / f'bd_mccrary_{key}.png'
plot_bd(centers, counts, z, expected, label, png, threshold=threshold)
print(f' plot: {png}')
all_results[key] = {
'label': label,
'n': int(len(arr)),
'bin_width': float(bw),
'direction': direction,
'n_bins': int(len(centers)),
'bin_centers': [float(c) for c in centers],
'counts': [int(c) for c in counts],
'z_scores': [None if np.isnan(zi) else float(zi) for zi in z],
'transitions': transitions,
'best_transition': best,
'threshold': threshold,
}
# Write JSON
json_path = OUT / 'bd_mccrary_results.json'
with open(json_path, 'w') as f:
json.dump({
'generated_at': datetime.now().isoformat(),
'z_critical': Z_CRIT,
'results': all_results,
}, f, indent=2, ensure_ascii=False)
print(f'\nJSON: {json_path}')
# Markdown
md = [
'# Burgstahler-Dichev / McCrary Discontinuity Test Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Method',
'',
'For each bin i of width δ, under the null of distributional',
'smoothness the expected count is the average of neighbours,',
'and the standardized deviation',
'',
' Z_i = (n_i - 0.5*(n_{i-1}+n_{i+1})) / SE',
'',
'is approximately N(0,1). We flag a transition when Z_{i-1} < -1.96',
'and Z_i > 1.96 (or reversed, depending on the scale direction).',
'The threshold is taken at the midpoint of the two bin centres.',
'',
'## Results',
'',
'| Test | N | bin width | Transitions | Threshold |',
'|------|---|-----------|-------------|-----------|',
]
for r in all_results.values():
thr = (f"{r['threshold']:.4f}" if r['threshold'] is not None
else '—')
md.append(
f"| {r['label']} | {r['n']:,} | {r['bin_width']} | "
f"{len(r['transitions'])} | {thr} |"
)
md += [
'',
'## Notes',
'',
'* For cosine (direction `neg_to_pos`), the transition marks the',
" boundary below which hand-signed dominates and above which",
' non-hand-signed replication dominates.',
'* For dHash (direction `pos_to_neg`), the transition marks the',
" boundary below which replication dominates (small distances)",
' and above which hand-signed variation dominates.',
'* Multiple candidate transitions are ranked by total |Z| magnitude',
' on both sides of the boundary; the strongest is reported.',
'* Absence of a significant transition is itself informative: it',
' is consistent with a single dominant generative mechanism (e.g.',
' Firm A, a replication-dominated population per interviews with',
' multiple Firm A accountants -- most use replication, a minority',
' may hand-sign).',
]
md_path = OUT / 'bd_mccrary_report.md'
md_path.write_text('\n'.join(md), encoding='utf-8')
print(f'Report: {md_path}')
if __name__ == '__main__':
main()
signature_analysis/17_beta_mixture_em.py
+406
View File
@@ -0,0 +1,406 @@
#!/usr/bin/env python3
"""
Script 17: Beta Mixture Model via EM + Gaussian Mixture on Logit Transform
==========================================================================
Fits a 2-component Beta mixture to cosine similarity, plus parallel
Gaussian mixture on logit-transformed data as robustness check.
Theory:
- Cosine similarity is bounded [0,1] so Beta is the natural parametric
family for the component distributions.
- EM algorithm (Dempster, Laird & Rubin 1977) provides ML estimates.
- If the mixture gives a crossing point, that is the Bayes-optimal
threshold under the fitted model.
- Robustness: logit(x) maps (0,1) to the real line, where Gaussian
mixture is standard; White (1982) quasi-MLE guarantees asymptotic
recovery of the best Beta-family approximation even under
mis-specification.
Parametrization of Beta via method-of-moments inside the M-step:
alpha = mu * ((mu*(1-mu))/var - 1)
beta = (1-mu) * ((mu*(1-mu))/var - 1)
Expected outcome (per memory 2026-04-16):
Signature-level Beta mixture FAILS to separate hand-signed vs
non-hand-signed because the distribution is unimodal long-tail.
Report this as a formal result -- it motivates the pivot to
accountant-level mixture (Script 18).
Output:
reports/beta_mixture/beta_mixture_report.md
reports/beta_mixture/beta_mixture_results.json
reports/beta_mixture/beta_mixture_<case>.png
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
from scipy import stats
from scipy.optimize import brentq
from sklearn.mixture import GaussianMixture
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/beta_mixture')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
EPS = 1e-6
def fit_beta_mixture_em(x, n_components=2, max_iter=300, tol=1e-6, seed=42):
"""
Fit a K-component Beta mixture via EM using MoM M-step estimates for
alpha/beta of each component. MoM works because Beta is fully determined
by its mean and variance under the moment equations.
"""
rng = np.random.default_rng(seed)
x = np.clip(np.asarray(x, dtype=float), EPS, 1 - EPS)
n = len(x)
K = n_components
# Initialise responsibilities by quantile-based split
q = np.linspace(0, 1, K + 1)
thresh = np.quantile(x, q[1:-1])
labels = np.digitize(x, thresh)
resp = np.zeros((n, K))
resp[np.arange(n), labels] = 1.0
params = [] # list of dicts with alpha, beta, weight
log_like_hist = []
for it in range(max_iter):
# M-step
nk = resp.sum(axis=0) + 1e-12
weights = nk / nk.sum()
mus = (resp * x[:, None]).sum(axis=0) / nk
var_num = (resp * (x[:, None] - mus) ** 2).sum(axis=0)
vars_ = var_num / nk
# Ensure validity for Beta: var < mu*(1-mu)
upper = mus * (1 - mus) - 1e-9
vars_ = np.minimum(vars_, upper)
vars_ = np.maximum(vars_, 1e-9)
factor = mus * (1 - mus) / vars_ - 1
factor = np.maximum(factor, 1e-6)
alphas = mus * factor
betas = (1 - mus) * factor
params = [{'alpha': float(alphas[k]), 'beta': float(betas[k]),
'weight': float(weights[k]), 'mu': float(mus[k]),
'var': float(vars_[k])} for k in range(K)]
# E-step
log_pdfs = np.column_stack([
stats.beta.logpdf(x, alphas[k], betas[k]) + np.log(weights[k])
for k in range(K)
])
m = log_pdfs.max(axis=1, keepdims=True)
log_like = (m.ravel() + np.log(np.exp(log_pdfs - m).sum(axis=1))).sum()
log_like_hist.append(float(log_like))
new_resp = np.exp(log_pdfs - m)
new_resp = new_resp / new_resp.sum(axis=1, keepdims=True)
if it > 0 and abs(log_like_hist[-1] - log_like_hist[-2]) < tol:
resp = new_resp
break
resp = new_resp
# Order components by mean ascending (so C1 = low mean, CK = high mean)
order = np.argsort([p['mu'] for p in params])
params = [params[i] for i in order]
resp = resp[:, order]
# AIC/BIC (k = 3K - 1 free parameters: alpha, beta, weight each component;
# weights sum to 1 removes one df)
k = 3 * K - 1
aic = 2 * k - 2 * log_like_hist[-1]
bic = k * np.log(n) - 2 * log_like_hist[-1]
return {
'components': params,
'log_likelihood': log_like_hist[-1],
'aic': float(aic),
'bic': float(bic),
'n_iter': it + 1,
'responsibilities': resp,
}
def mixture_crossing(params, x_range):
"""Find crossing point of two weighted component densities (K=2)."""
if len(params) != 2:
return None
a1, b1, w1 = params[0]['alpha'], params[0]['beta'], params[0]['weight']
a2, b2, w2 = params[1]['alpha'], params[1]['beta'], params[1]['weight']
def diff(x):
return (w2 * stats.beta.pdf(x, a2, b2)
- w1 * stats.beta.pdf(x, a1, b1))
# Search for sign change inside the overlap region
xs = np.linspace(x_range[0] + 1e-4, x_range[1] - 1e-4, 2000)
ys = diff(xs)
sign_changes = np.where(np.diff(np.sign(ys)) != 0)[0]
if len(sign_changes) == 0:
return None
# Pick crossing closest to midpoint of component means
mid = 0.5 * (params[0]['mu'] + params[1]['mu'])
crossings = []
for i in sign_changes:
try:
x0 = brentq(diff, xs[i], xs[i + 1])
crossings.append(x0)
except ValueError:
continue
if not crossings:
return None
return min(crossings, key=lambda c: abs(c - mid))
def logit(x):
x = np.clip(x, EPS, 1 - EPS)
return np.log(x / (1 - x))
def invlogit(z):
return 1.0 / (1.0 + np.exp(-z))
def fit_gmm_logit(x, n_components=2, seed=42):
"""GMM on logit-transformed values. Returns crossing point in original scale."""
z = logit(x).reshape(-1, 1)
gmm = GaussianMixture(n_components=n_components, random_state=seed,
max_iter=500).fit(z)
means = gmm.means_.ravel()
covs = gmm.covariances_.ravel()
weights = gmm.weights_
order = np.argsort(means)
comps = [{
'mu_logit': float(means[i]),
'sigma_logit': float(np.sqrt(covs[i])),
'weight': float(weights[i]),
'mu_original': float(invlogit(means[i])),
} for i in order]
result = {
'components': comps,
'log_likelihood': float(gmm.score(z) * len(z)),
'aic': float(gmm.aic(z)),
'bic': float(gmm.bic(z)),
'n_iter': int(gmm.n_iter_),
}
if n_components == 2:
m1, s1, w1 = means[order[0]], np.sqrt(covs[order[0]]), weights[order[0]]
m2, s2, w2 = means[order[1]], np.sqrt(covs[order[1]]), weights[order[1]]
def diff(z0):
return (w2 * stats.norm.pdf(z0, m2, s2)
- w1 * stats.norm.pdf(z0, m1, s1))
zs = np.linspace(min(m1, m2) - 1, max(m1, m2) + 1, 2000)
ys = diff(zs)
changes = np.where(np.diff(np.sign(ys)) != 0)[0]
if len(changes):
try:
z_cross = brentq(diff, zs[changes[0]], zs[changes[0] + 1])
result['crossing_logit'] = float(z_cross)
result['crossing_original'] = float(invlogit(z_cross))
except ValueError:
pass
return result
def plot_mixture(x, beta_res, title, out_path, gmm_res=None):
x = np.asarray(x, dtype=float).ravel()
x = x[np.isfinite(x)]
fig, ax = plt.subplots(figsize=(10, 5))
bin_edges = np.linspace(float(x.min()), float(x.max()), 81)
ax.hist(x, bins=bin_edges, density=True, alpha=0.45, color='steelblue',
edgecolor='white')
xs = np.linspace(max(0.0, x.min() - 0.01), min(1.0, x.max() + 0.01), 500)
total = np.zeros_like(xs)
for i, p in enumerate(beta_res['components']):
comp_pdf = p['weight'] * stats.beta.pdf(xs, p['alpha'], p['beta'])
total = total + comp_pdf
ax.plot(xs, comp_pdf, '--', lw=1.5,
label=f"C{i+1}: α={p['alpha']:.2f}, β={p['beta']:.2f}, "
f"w={p['weight']:.2f}")
ax.plot(xs, total, 'r-', lw=2, label='Beta mixture (sum)')
crossing = mixture_crossing(beta_res['components'], (xs[0], xs[-1]))
if crossing is not None:
ax.axvline(crossing, color='green', ls='--', lw=2,
label=f'Beta crossing = {crossing:.4f}')
if gmm_res and 'crossing_original' in gmm_res:
ax.axvline(gmm_res['crossing_original'], color='purple', ls=':',
lw=2, label=f"Logit-GMM crossing = "
f"{gmm_res['crossing_original']:.4f}")
ax.set_xlabel('Value')
ax.set_ylabel('Density')
ax.set_title(title)
ax.legend(fontsize=8)
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
return crossing
def fetch(label):
conn = sqlite3.connect(DB)
cur = conn.cursor()
if label == 'firm_a_cosine':
cur.execute('''
SELECT s.max_similarity_to_same_accountant
FROM signatures s
JOIN accountants a ON s.assigned_accountant = a.name
WHERE a.firm = ? AND s.max_similarity_to_same_accountant IS NOT NULL
''', (FIRM_A,))
elif label == 'full_cosine':
cur.execute('''
SELECT max_similarity_to_same_accountant FROM signatures
WHERE max_similarity_to_same_accountant IS NOT NULL
''')
else:
raise ValueError(label)
vals = [r[0] for r in cur.fetchall() if r[0] is not None]
conn.close()
return np.array(vals, dtype=float)
def main():
print('='*70)
print('Script 17: Beta Mixture EM + Logit-GMM Robustness Check')
print('='*70)
cases = [
('firm_a_cosine', 'Firm A cosine max-similarity'),
('full_cosine', 'Full-sample cosine max-similarity'),
]
summary = {}
for key, label in cases:
print(f'\n[{label}]')
x = fetch(key)
print(f' N = {len(x):,}')
# Subsample for full sample to keep EM tractable but still stable
if len(x) > 200000:
rng = np.random.default_rng(42)
x_fit = rng.choice(x, 200000, replace=False)
print(f' Subsampled to {len(x_fit):,} for EM fitting')
else:
x_fit = x
beta2 = fit_beta_mixture_em(x_fit, n_components=2)
beta3 = fit_beta_mixture_em(x_fit, n_components=3)
print(f' Beta-2 AIC={beta2["aic"]:.1f}, BIC={beta2["bic"]:.1f}')
print(f' Beta-3 AIC={beta3["aic"]:.1f}, BIC={beta3["bic"]:.1f}')
gmm2 = fit_gmm_logit(x_fit, n_components=2)
gmm3 = fit_gmm_logit(x_fit, n_components=3)
print(f' LogGMM2 AIC={gmm2["aic"]:.1f}, BIC={gmm2["bic"]:.1f}')
print(f' LogGMM3 AIC={gmm3["aic"]:.1f}, BIC={gmm3["bic"]:.1f}')
# Report crossings
crossing_beta = mixture_crossing(beta2['components'], (x.min(), x.max()))
print(f' Beta-2 crossing: '
f"{('%.4f' % crossing_beta) if crossing_beta else '—'}")
print(f' LogGMM-2 crossing (original scale): '
f"{gmm2.get('crossing_original', '—')}")
# Plot
png = OUT / f'beta_mixture_{key}.png'
plot_mixture(x_fit, beta2, f'{label}: Beta mixture (2 comp)', png,
gmm_res=gmm2)
print(f' plot: {png}')
# Strip responsibilities for JSON compactness
beta2_out = {k: v for k, v in beta2.items() if k != 'responsibilities'}
beta3_out = {k: v for k, v in beta3.items() if k != 'responsibilities'}
summary[key] = {
'label': label,
'n': int(len(x)),
'n_fit': int(len(x_fit)),
'beta_2': beta2_out,
'beta_3': beta3_out,
'beta_2_crossing': (float(crossing_beta)
if crossing_beta is not None else None),
'logit_gmm_2': gmm2,
'logit_gmm_3': gmm3,
'bic_best': ('beta_2' if beta2['bic'] < beta3['bic']
else 'beta_3'),
}
# Write JSON
json_path = OUT / 'beta_mixture_results.json'
with open(json_path, 'w') as f:
json.dump({
'generated_at': datetime.now().isoformat(),
'results': summary,
}, f, indent=2, ensure_ascii=False, default=float)
print(f'\nJSON: {json_path}')
# Markdown
md = [
'# Beta Mixture EM Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Method',
'',
'* 2- and 3-component Beta mixture fit by EM with method-of-moments',
' M-step (stable for bounded data).',
'* Parallel 2/3-component Gaussian mixture on logit-transformed',
' values as robustness check (White 1982 quasi-MLE consistency).',
'* Crossing point of the 2-component mixture densities is reported',
' as the Bayes-optimal threshold under equal misclassification cost.',
'',
'## Results',
'',
'| Dataset | N (fit) | Beta-2 BIC | Beta-3 BIC | LogGMM-2 BIC | LogGMM-3 BIC | BIC-best |',
'|---------|---------|------------|------------|--------------|--------------|----------|',
]
for r in summary.values():
md.append(
f"| {r['label']} | {r['n_fit']:,} | "
f"{r['beta_2']['bic']:.1f} | {r['beta_3']['bic']:.1f} | "
f"{r['logit_gmm_2']['bic']:.1f} | {r['logit_gmm_3']['bic']:.1f} | "
f"{r['bic_best']} |"
)
md += ['', '## Threshold estimates (2-component)', '',
'| Dataset | Beta-2 crossing | LogGMM-2 crossing (orig) |',
'|---------|-----------------|--------------------------|']
for r in summary.values():
beta_str = (f"{r['beta_2_crossing']:.4f}"
if r['beta_2_crossing'] is not None else '—')
gmm_str = (f"{r['logit_gmm_2']['crossing_original']:.4f}"
if 'crossing_original' in r['logit_gmm_2'] else '—')
md.append(f"| {r['label']} | {beta_str} | {gmm_str} |")
md += [
'',
'## Interpretation',
'',
'A successful 2-component fit with a clear crossing point would',
'indicate two underlying generative mechanisms (hand-signed vs',
'non-hand-signed) with a principled Bayes-optimal boundary.',
'',
'If Beta-3 BIC is meaningfully smaller than Beta-2, or if the',
'components of Beta-2 largely overlap (similar means, wide spread),',
'this is consistent with a unimodal distribution poorly approximated',
'by two components. Prior finding (2026-04-16) suggested this is',
'the case at signature level; the accountant-level mixture',
'(Script 18) is where the bimodality emerges.',
]
md_path = OUT / 'beta_mixture_report.md'
md_path.write_text('\n'.join(md), encoding='utf-8')
print(f'Report: {md_path}')
if __name__ == '__main__':
main()
signature_analysis/18_accountant_mixture.py
+404
View File
@@ -0,0 +1,404 @@
#!/usr/bin/env python3
"""
Script 18: Accountant-Level 3-Component Gaussian Mixture
========================================================
Rebuild the GMM analysis from memory 2026-04-16: at the accountant level
(not signature level), the joint distribution of (cosine_mean, dhash_mean)
separates into three components corresponding to signing-behaviour
regimes:
C1 High-replication cos_mean ≈ 0.983, dh_mean ≈ 2.4, ~20%, Deloitte-heavy
C2 Middle band cos_mean ≈ 0.954, dh_mean ≈ 7.0, ~52%, KPMG/PwC/EY
C3 Hand-signed tendency cos_mean ≈ 0.928, dh_mean ≈ 11.2, ~28%, small firms
The script:
1. Aggregates per-accountant means from the signature table.
2. Fits 1-, 2-, 3-, 4-component 2D Gaussian mixtures and selects by BIC.
3. Reports component parameters, cluster assignments, and per-firm
breakdown.
4. For the 2-component fit derives the natural threshold (crossing of
marginal densities in cosine-mean and dhash-mean).
Firm A framing note (2026-04-20, corrected):
Interviews with Firm A accountants confirm MOST use replication but a
MINORITY may hand-sign. Firm A is thus a "replication-dominated"
population, NOT pure. Empirically: of ~180 Firm A accountants, ~139
land in C1 (high-replication) and ~32 land in C2 (middle band) under
the 3-component fit. The C2 Firm A members are the interview-suggested
minority hand-signers.
Output:
reports/accountant_mixture/accountant_mixture_report.md
reports/accountant_mixture/accountant_mixture_results.json
reports/accountant_mixture/accountant_mixture_2d.png
reports/accountant_mixture/accountant_mixture_marginals.png
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
from scipy import stats
from scipy.optimize import brentq
from sklearn.mixture import GaussianMixture
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'accountant_mixture')
OUT.mkdir(parents=True, exist_ok=True)
MIN_SIGS = 10
def load_accountant_aggregates():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.assigned_accountant,
a.firm,
AVG(s.max_similarity_to_same_accountant) AS cos_mean,
AVG(CAST(s.min_dhash_independent AS REAL)) AS dh_mean,
COUNT(*) AS n
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE s.assigned_accountant IS NOT NULL
AND s.max_similarity_to_same_accountant IS NOT NULL
AND s.min_dhash_independent IS NOT NULL
GROUP BY s.assigned_accountant
HAVING n >= ?
''', (MIN_SIGS,))
rows = cur.fetchall()
conn.close()
return [
{'accountant': r[0], 'firm': r[1] or '(unknown)',
'cos_mean': float(r[2]), 'dh_mean': float(r[3]), 'n': int(r[4])}
for r in rows
]
def fit_gmm_range(X, ks=(1, 2, 3, 4, 5), seed=42, n_init=10):
results = []
best_bic = np.inf
best = None
for k in ks:
gmm = GaussianMixture(
n_components=k, covariance_type='full',
random_state=seed, n_init=n_init, max_iter=500,
).fit(X)
bic = gmm.bic(X)
aic = gmm.aic(X)
results.append({
'k': int(k), 'bic': float(bic), 'aic': float(aic),
'converged': bool(gmm.converged_), 'n_iter': int(gmm.n_iter_),
})
if bic < best_bic:
best_bic = bic
best = gmm
return results, best
def summarize_components(gmm, X, df):
"""Assign clusters, return per-component stats + per-firm breakdown."""
labels = gmm.predict(X)
means = gmm.means_
order = np.argsort(means[:, 0]) # order by cos_mean ascending
# Relabel so smallest cos_mean = component 1
relabel = np.argsort(order)
# Actually invert: in prior memory C1 was HIGH replication (highest cos).
# To keep consistent with memory, order DESCENDING by cos_mean so C1 = high.
order = np.argsort(-means[:, 0])
relabel = {int(old): new + 1 for new, old in enumerate(order)}
new_labels = np.array([relabel[int(l)] for l in labels])
components = []
for rank, old_idx in enumerate(order, start=1):
mu = means[old_idx]
cov = gmm.covariances_[old_idx]
w = gmm.weights_[old_idx]
mask = new_labels == rank
firms = {}
for row, in_cluster in zip(df, mask):
if not in_cluster:
continue
firms[row['firm']] = firms.get(row['firm'], 0) + 1
firms_sorted = sorted(firms.items(), key=lambda kv: -kv[1])
components.append({
'component': rank,
'mu_cos': float(mu[0]),
'mu_dh': float(mu[1]),
'cov_00': float(cov[0, 0]),
'cov_11': float(cov[1, 1]),
'cov_01': float(cov[0, 1]),
'corr': float(cov[0, 1] / np.sqrt(cov[0, 0] * cov[1, 1])),
'weight': float(w),
'n_accountants': int(mask.sum()),
'top_firms': firms_sorted[:5],
})
return components, new_labels
def marginal_crossing(means, covs, weights, dim, search_lo, search_hi):
"""Find crossing of two weighted marginal Gaussians along dimension `dim`."""
m1, m2 = means[0][dim], means[1][dim]
s1 = np.sqrt(covs[0][dim, dim])
s2 = np.sqrt(covs[1][dim, dim])
w1, w2 = weights[0], weights[1]
def diff(x):
return (w2 * stats.norm.pdf(x, m2, s2)
- w1 * stats.norm.pdf(x, m1, s1))
xs = np.linspace(search_lo, search_hi, 2000)
ys = diff(xs)
changes = np.where(np.diff(np.sign(ys)) != 0)[0]
if not len(changes):
return None
mid = 0.5 * (m1 + m2)
crossings = []
for i in changes:
try:
crossings.append(brentq(diff, xs[i], xs[i + 1]))
except ValueError:
continue
if not crossings:
return None
return float(min(crossings, key=lambda c: abs(c - mid)))
def plot_2d(df, labels, means, title, out_path):
colors = ['#d62728', '#1f77b4', '#2ca02c', '#9467bd', '#ff7f0e']
fig, ax = plt.subplots(figsize=(9, 7))
for k in sorted(set(labels)):
mask = labels == k
xs = [r['cos_mean'] for r, m in zip(df, mask) if m]
ys = [r['dh_mean'] for r, m in zip(df, mask) if m]
ax.scatter(xs, ys, s=20, alpha=0.55, color=colors[(k - 1) % 5],
label=f'C{k} (n={int(mask.sum())})')
for i, mu in enumerate(means):
ax.plot(mu[0], mu[1], 'k*', ms=18, mec='white', mew=1.5)
ax.annotate(f' μ{i+1}', (mu[0], mu[1]), fontsize=10)
ax.set_xlabel('Per-accountant mean cosine max-similarity')
ax.set_ylabel('Per-accountant mean independent min dHash')
ax.set_title(title)
ax.legend()
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
def plot_marginals(df, labels, gmm_2, out_path, cos_cross=None, dh_cross=None):
cos = np.array([r['cos_mean'] for r in df])
dh = np.array([r['dh_mean'] for r in df])
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
# Cosine marginal
ax = axes[0]
ax.hist(cos, bins=40, density=True, alpha=0.5, color='steelblue',
edgecolor='white')
xs = np.linspace(cos.min(), cos.max(), 400)
means_2 = gmm_2.means_
covs_2 = gmm_2.covariances_
weights_2 = gmm_2.weights_
order = np.argsort(-means_2[:, 0])
for rank, i in enumerate(order, start=1):
ys = weights_2[i] * stats.norm.pdf(xs, means_2[i, 0],
np.sqrt(covs_2[i, 0, 0]))
ax.plot(xs, ys, '--', label=f'C{rank} μ={means_2[i,0]:.3f}')
if cos_cross is not None:
ax.axvline(cos_cross, color='green', lw=2,
label=f'Crossing = {cos_cross:.4f}')
ax.set_xlabel('Per-accountant mean cosine')
ax.set_ylabel('Density')
ax.set_title('Cosine marginal (2-component fit)')
ax.legend(fontsize=8)
# dHash marginal
ax = axes[1]
ax.hist(dh, bins=40, density=True, alpha=0.5, color='coral',
edgecolor='white')
xs = np.linspace(dh.min(), dh.max(), 400)
for rank, i in enumerate(order, start=1):
ys = weights_2[i] * stats.norm.pdf(xs, means_2[i, 1],
np.sqrt(covs_2[i, 1, 1]))
ax.plot(xs, ys, '--', label=f'C{rank} μ={means_2[i,1]:.2f}')
if dh_cross is not None:
ax.axvline(dh_cross, color='green', lw=2,
label=f'Crossing = {dh_cross:.4f}')
ax.set_xlabel('Per-accountant mean dHash')
ax.set_ylabel('Density')
ax.set_title('dHash marginal (2-component fit)')
ax.legend(fontsize=8)
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
def main():
print('='*70)
print('Script 18: Accountant-Level Gaussian Mixture')
print('='*70)
df = load_accountant_aggregates()
print(f'\nAccountants with >= {MIN_SIGS} signatures: {len(df)}')
X = np.array([[r['cos_mean'], r['dh_mean']] for r in df])
# Fit K=1..5
print('\nFitting GMMs with K=1..5...')
bic_results, _ = fit_gmm_range(X, ks=(1, 2, 3, 4, 5), seed=42, n_init=15)
for r in bic_results:
print(f" K={r['k']}: BIC={r['bic']:.2f} AIC={r['aic']:.2f} "
f"converged={r['converged']}")
best_k = min(bic_results, key=lambda r: r['bic'])['k']
print(f'\nBIC-best K = {best_k}')
# Fit 3-component specifically (target)
gmm_3 = GaussianMixture(n_components=3, covariance_type='full',
random_state=42, n_init=15, max_iter=500).fit(X)
comps_3, labels_3 = summarize_components(gmm_3, X, df)
print('\n--- 3-component summary ---')
for c in comps_3:
tops = ', '.join(f"{f}({n})" for f, n in c['top_firms'])
print(f" C{c['component']}: cos={c['mu_cos']:.3f}, "
f"dh={c['mu_dh']:.2f}, w={c['weight']:.2f}, "
f"n={c['n_accountants']} -> {tops}")
# Fit 2-component for threshold derivation
gmm_2 = GaussianMixture(n_components=2, covariance_type='full',
random_state=42, n_init=15, max_iter=500).fit(X)
comps_2, labels_2 = summarize_components(gmm_2, X, df)
# Crossings
cos_cross = marginal_crossing(gmm_2.means_, gmm_2.covariances_,
gmm_2.weights_, dim=0,
search_lo=X[:, 0].min(),
search_hi=X[:, 0].max())
dh_cross = marginal_crossing(gmm_2.means_, gmm_2.covariances_,
gmm_2.weights_, dim=1,
search_lo=X[:, 1].min(),
search_hi=X[:, 1].max())
print(f'\n2-component crossings: cos={cos_cross}, dh={dh_cross}')
# Plots
plot_2d(df, labels_3, gmm_3.means_,
'3-component accountant-level GMM',
OUT / 'accountant_mixture_2d.png')
plot_marginals(df, labels_2, gmm_2,
OUT / 'accountant_mixture_marginals.png',
cos_cross=cos_cross, dh_cross=dh_cross)
# Per-accountant CSV (for downstream use)
csv_path = OUT / 'accountant_clusters.csv'
with open(csv_path, 'w', encoding='utf-8') as f:
f.write('accountant,firm,n_signatures,cos_mean,dh_mean,'
'cluster_k3,cluster_k2\n')
for r, k3, k2 in zip(df, labels_3, labels_2):
f.write(f"{r['accountant']},{r['firm']},{r['n']},"
f"{r['cos_mean']:.6f},{r['dh_mean']:.6f},{k3},{k2}\n")
print(f'CSV: {csv_path}')
# Summary JSON
summary = {
'generated_at': datetime.now().isoformat(),
'n_accountants': len(df),
'min_signatures': MIN_SIGS,
'bic_model_selection': bic_results,
'best_k_by_bic': best_k,
'gmm_3': {
'components': comps_3,
'aic': float(gmm_3.aic(X)),
'bic': float(gmm_3.bic(X)),
'log_likelihood': float(gmm_3.score(X) * len(X)),
},
'gmm_2': {
'components': comps_2,
'aic': float(gmm_2.aic(X)),
'bic': float(gmm_2.bic(X)),
'log_likelihood': float(gmm_2.score(X) * len(X)),
'cos_crossing': cos_cross,
'dh_crossing': dh_cross,
},
}
with open(OUT / 'accountant_mixture_results.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'JSON: {OUT / "accountant_mixture_results.json"}')
# Markdown
md = [
'# Accountant-Level Gaussian Mixture Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Data',
'',
f'* Per-accountant aggregates: mean cosine max-similarity, '
f'mean independent min dHash.',
f"* Minimum signatures per accountant: {MIN_SIGS}.",
f'* Accountants included: **{len(df)}**.',
'',
'## Model selection (BIC)',
'',
'| K | BIC | AIC | Converged |',
'|---|-----|-----|-----------|',
]
for r in bic_results:
mark = ' ←best' if r['k'] == best_k else ''
md.append(
f"| {r['k']} | {r['bic']:.2f} | {r['aic']:.2f} | "
f"{r['converged']}{mark} |"
)
md += ['', '## 3-component fit', '',
'| Component | cos_mean | dh_mean | weight | n_accountants | top firms |',
'|-----------|----------|---------|--------|----------------|-----------|']
for c in comps_3:
tops = ', '.join(f"{f}:{n}" for f, n in c['top_firms'])
md.append(
f"| C{c['component']} | {c['mu_cos']:.3f} | {c['mu_dh']:.2f} | "
f"{c['weight']:.3f} | {c['n_accountants']} | {tops} |"
)
md += ['', '## 2-component fit (threshold derivation)', '',
'| Component | cos_mean | dh_mean | weight | n_accountants |',
'|-----------|----------|---------|--------|----------------|']
for c in comps_2:
md.append(
f"| C{c['component']} | {c['mu_cos']:.3f} | {c['mu_dh']:.2f} | "
f"{c['weight']:.3f} | {c['n_accountants']} |"
)
md += ['', '### Natural thresholds from 2-component crossings', '',
f'* Cosine: **{cos_cross:.4f}**' if cos_cross
else '* Cosine: no crossing found',
f'* dHash: **{dh_cross:.4f}**' if dh_cross
else '* dHash: no crossing found',
'',
'## Interpretation',
'',
'The accountant-level mixture separates signing-behaviour regimes,',
'while the signature-level distribution is a continuous spectrum',
'(see Scripts 15 and 17). The BIC-best model chooses how many',
'discrete regimes the data supports. The 2-component crossings',
'are the natural per-accountant thresholds for classifying a',
"CPA's signing behaviour.",
'',
'## Artifacts',
'',
'* `accountant_mixture_2d.png` - 2D scatter with 3-component fit',
'* `accountant_mixture_marginals.png` - 1D marginals with 2-component fit',
'* `accountant_clusters.csv` - per-accountant cluster assignments',
'* `accountant_mixture_results.json` - full numerical results',
]
(OUT / 'accountant_mixture_report.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "accountant_mixture_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/19_pixel_identity_validation.py
+423
View File
@@ -0,0 +1,423 @@
#!/usr/bin/env python3
"""
Script 19: Pixel-Identity Validation (No Human Annotation Required)
===================================================================
Validates the cosine + dHash dual classifier using three naturally
occurring reference populations instead of manual labels:
Positive anchor 1: pixel_identical_to_closest = 1
Two signature images byte-identical after crop/resize.
Mathematically impossible to arise from independent hand-signing
=> absolute ground truth for replication.
Positive anchor 2: Firm A (Deloitte) signatures
Interview evidence from multiple Firm A accountants confirms that
MOST use replication (stamping / firm-level e-signing) but a
MINORITY may still hand-sign. Firm A is therefore a
"replication-dominated" population (not a pure one). We use it as
a strong prior positive for the majority regime, while noting that
~7% of Firm A signatures fall below cosine 0.95 consistent with
the minority hand-signers. This matches the long left tail
observed in the dip test (Script 15) and the Firm A members who
land in C2 (middle band) of the accountant-level GMM (Script 18).
Negative anchor: signatures with cosine <= low threshold
Pairs with very low cosine similarity cannot plausibly be pixel
duplicates, so they serve as absolute negatives.
Metrics reported:
- FAR/FRR/EER using the pixel-identity anchor as the gold positive
and low-similarity pairs as the gold negative.
- Precision/Recall/F1 at cosine and dHash thresholds from Scripts
15/16/17/18.
- Convergence with Firm A anchor (what fraction of Firm A signatures
are correctly classified at each threshold).
Small visual sanity sample (30 pairs) is exported for spot-check, but
metrics are derived entirely from pixel and Firm A evidence.
Output:
reports/pixel_validation/pixel_validation_report.md
reports/pixel_validation/pixel_validation_results.json
reports/pixel_validation/roc_cosine.png, roc_dhash.png
reports/pixel_validation/sanity_sample.csv
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'pixel_validation')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
NEGATIVE_COSINE_UPPER = 0.70 # pairs with max-cosine < 0.70 assumed not replicated
SANITY_SAMPLE_SIZE = 30
def load_signatures():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.signature_id, s.image_filename, s.assigned_accountant,
a.firm, s.max_similarity_to_same_accountant,
s.phash_distance_to_closest, s.min_dhash_independent,
s.pixel_identical_to_closest, s.closest_match_file
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE s.max_similarity_to_same_accountant IS NOT NULL
''')
rows = cur.fetchall()
conn.close()
data = []
for r in rows:
data.append({
'sig_id': r[0], 'filename': r[1], 'accountant': r[2],
'firm': r[3] or '(unknown)',
'cosine': float(r[4]),
'dhash_cond': None if r[5] is None else int(r[5]),
'dhash_indep': None if r[6] is None else int(r[6]),
'pixel_identical': int(r[7] or 0),
'closest_match': r[8],
})
return data
def confusion(y_true, y_pred):
tp = int(np.sum((y_true == 1) & (y_pred == 1)))
fp = int(np.sum((y_true == 0) & (y_pred == 1)))
fn = int(np.sum((y_true == 1) & (y_pred == 0)))
tn = int(np.sum((y_true == 0) & (y_pred == 0)))
return tp, fp, fn, tn
def classification_metrics(y_true, y_pred):
tp, fp, fn, tn = confusion(y_true, y_pred)
denom_p = max(tp + fp, 1)
denom_r = max(tp + fn, 1)
precision = tp / denom_p
recall = tp / denom_r
f1 = (2 * precision * recall / (precision + recall)
if precision + recall > 0 else 0.0)
far = fp / max(fp + tn, 1) # false acceptance rate (over negatives)
frr = fn / max(fn + tp, 1) # false rejection rate (over positives)
return {
'tp': tp, 'fp': fp, 'fn': fn, 'tn': tn,
'precision': float(precision),
'recall': float(recall),
'f1': float(f1),
'far': float(far),
'frr': float(frr),
}
def sweep_threshold(scores, y, directions, thresholds):
"""For direction 'above' a prediction is positive if score > threshold;
for 'below' it is positive if score < threshold."""
out = []
for t in thresholds:
if directions == 'above':
y_pred = (scores > t).astype(int)
else:
y_pred = (scores < t).astype(int)
m = classification_metrics(y, y_pred)
m['threshold'] = float(t)
out.append(m)
return out
def find_eer(sweep):
"""EER = point where FAR ≈ FRR; interpolated from nearest pair."""
thr = np.array([s['threshold'] for s in sweep])
far = np.array([s['far'] for s in sweep])
frr = np.array([s['frr'] for s in sweep])
diff = far - frr
signs = np.sign(diff)
changes = np.where(np.diff(signs) != 0)[0]
if len(changes) == 0:
idx = int(np.argmin(np.abs(diff)))
return {'threshold': float(thr[idx]), 'far': float(far[idx]),
'frr': float(frr[idx]), 'eer': float(0.5 * (far[idx] + frr[idx]))}
i = int(changes[0])
w = abs(diff[i]) / (abs(diff[i]) + abs(diff[i + 1]) + 1e-12)
thr_i = (1 - w) * thr[i] + w * thr[i + 1]
far_i = (1 - w) * far[i] + w * far[i + 1]
frr_i = (1 - w) * frr[i] + w * frr[i + 1]
return {'threshold': float(thr_i), 'far': float(far_i),
'frr': float(frr_i), 'eer': float(0.5 * (far_i + frr_i))}
def plot_roc(sweep, title, out_path):
far = np.array([s['far'] for s in sweep])
frr = np.array([s['frr'] for s in sweep])
thr = np.array([s['threshold'] for s in sweep])
fig, axes = plt.subplots(1, 2, figsize=(13, 5))
ax = axes[0]
ax.plot(far, 1 - frr, 'b-', lw=2)
ax.plot([0, 1], [0, 1], 'k--', alpha=0.4)
ax.set_xlabel('FAR')
ax.set_ylabel('1 - FRR (True Positive Rate)')
ax.set_title(f'{title} - ROC')
ax.set_xlim(0, 1)
ax.set_ylim(0, 1)
ax.grid(alpha=0.3)
ax = axes[1]
ax.plot(thr, far, 'r-', lw=2, label='FAR')
ax.plot(thr, frr, 'b-', lw=2, label='FRR')
ax.set_xlabel('Threshold')
ax.set_ylabel('Error rate')
ax.set_title(f'{title} - FAR / FRR vs threshold')
ax.legend()
ax.grid(alpha=0.3)
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
def main():
print('='*70)
print('Script 19: Pixel-Identity Validation (No Annotation)')
print('='*70)
data = load_signatures()
print(f'\nTotal signatures loaded: {len(data):,}')
cos = np.array([d['cosine'] for d in data])
dh_indep = np.array([d['dhash_indep'] if d['dhash_indep'] is not None
else -1 for d in data])
pix = np.array([d['pixel_identical'] for d in data])
firm = np.array([d['firm'] for d in data])
print(f'Pixel-identical: {int(pix.sum()):,} signatures')
print(f'Firm A signatures: {int((firm == FIRM_A).sum()):,}')
print(f'Negative anchor (cosine < {NEGATIVE_COSINE_UPPER}): '
f'{int((cos < NEGATIVE_COSINE_UPPER).sum()):,}')
# Build labelled set:
# positive = pixel_identical == 1
# negative = cosine < NEGATIVE_COSINE_UPPER (and not pixel_identical)
pos_mask = pix == 1
neg_mask = (cos < NEGATIVE_COSINE_UPPER) & (~pos_mask)
labelled_mask = pos_mask | neg_mask
y = pos_mask[labelled_mask].astype(int)
cos_l = cos[labelled_mask]
dh_l = dh_indep[labelled_mask]
# --- Sweep cosine threshold
cos_thresh = np.linspace(0.50, 1.00, 101)
cos_sweep = sweep_threshold(cos_l, y, 'above', cos_thresh)
cos_eer = find_eer(cos_sweep)
print(f'\nCosine EER: threshold={cos_eer["threshold"]:.4f}, '
f'EER={cos_eer["eer"]:.4f}')
# --- Sweep dHash threshold (independent)
dh_l_valid = dh_l >= 0
y_dh = y[dh_l_valid]
dh_valid = dh_l[dh_l_valid]
dh_thresh = np.arange(0, 40)
dh_sweep = sweep_threshold(dh_valid, y_dh, 'below', dh_thresh)
dh_eer = find_eer(dh_sweep)
print(f'dHash EER: threshold={dh_eer["threshold"]:.4f}, '
f'EER={dh_eer["eer"]:.4f}')
# Plots
plot_roc(cos_sweep, 'Cosine (pixel-identity anchor)',
OUT / 'roc_cosine.png')
plot_roc(dh_sweep, 'Independent dHash (pixel-identity anchor)',
OUT / 'roc_dhash.png')
# --- Evaluate canonical thresholds
canonical = [
('cosine', 0.837, 'above', cos, pos_mask, neg_mask),
('cosine', 0.941, 'above', cos, pos_mask, neg_mask),
('cosine', 0.95, 'above', cos, pos_mask, neg_mask),
('dhash_indep', 5, 'below', dh_indep, pos_mask,
neg_mask & (dh_indep >= 0)),
('dhash_indep', 8, 'below', dh_indep, pos_mask,
neg_mask & (dh_indep >= 0)),
('dhash_indep', 15, 'below', dh_indep, pos_mask,
neg_mask & (dh_indep >= 0)),
]
canonical_results = []
for name, thr, direction, scores, p_mask, n_mask in canonical:
labelled = p_mask | n_mask
valid = labelled & (scores >= 0 if 'dhash' in name else np.ones_like(
labelled, dtype=bool))
y_local = p_mask[valid].astype(int)
s = scores[valid]
if direction == 'above':
y_pred = (s > thr).astype(int)
else:
y_pred = (s < thr).astype(int)
m = classification_metrics(y_local, y_pred)
m.update({'indicator': name, 'threshold': float(thr),
'direction': direction})
canonical_results.append(m)
print(f" {name} @ {thr:>5} ({direction}): "
f"P={m['precision']:.3f}, R={m['recall']:.3f}, "
f"F1={m['f1']:.3f}, FAR={m['far']:.4f}, FRR={m['frr']:.4f}")
# --- Firm A anchor validation
firm_a_mask = firm == FIRM_A
firm_a_cos = cos[firm_a_mask]
firm_a_dh = dh_indep[firm_a_mask]
firm_a_rates = {}
for thr in [0.837, 0.941, 0.95]:
firm_a_rates[f'cosine>{thr}'] = float(np.mean(firm_a_cos > thr))
for thr in [5, 8, 15]:
valid = firm_a_dh >= 0
firm_a_rates[f'dhash_indep<={thr}'] = float(
np.mean(firm_a_dh[valid] <= thr))
# Dual thresholds
firm_a_rates['cosine>0.95 AND dhash_indep<=8'] = float(
np.mean((firm_a_cos > 0.95) &
(firm_a_dh >= 0) & (firm_a_dh <= 8)))
print('\nFirm A anchor validation:')
for k, v in firm_a_rates.items():
print(f' {k}: {v*100:.2f}%')
# --- Stratified sanity sample (30 signatures across 5 strata)
rng = np.random.default_rng(42)
strata = [
('pixel_identical', pix == 1),
('high_cos_low_dh',
(cos > 0.95) & (dh_indep >= 0) & (dh_indep <= 5) & (pix == 0)),
('borderline',
(cos > 0.837) & (cos < 0.95) & (dh_indep >= 0) & (dh_indep <= 15)),
('style_consistency_only',
(cos > 0.95) & (dh_indep >= 0) & (dh_indep > 15)),
('likely_genuine', cos < NEGATIVE_COSINE_UPPER),
]
sanity_sample = []
per_stratum = SANITY_SAMPLE_SIZE // len(strata)
for stratum_name, m in strata:
idx = np.where(m)[0]
pick = rng.choice(idx, size=min(per_stratum, len(idx)), replace=False)
for i in pick:
d = data[i]
sanity_sample.append({
'stratum': stratum_name, 'sig_id': d['sig_id'],
'filename': d['filename'], 'accountant': d['accountant'],
'firm': d['firm'], 'cosine': d['cosine'],
'dhash_indep': d['dhash_indep'],
'pixel_identical': d['pixel_identical'],
'closest_match': d['closest_match'],
})
csv_path = OUT / 'sanity_sample.csv'
with open(csv_path, 'w', encoding='utf-8') as f:
keys = ['stratum', 'sig_id', 'filename', 'accountant', 'firm',
'cosine', 'dhash_indep', 'pixel_identical', 'closest_match']
f.write(','.join(keys) + '\n')
for row in sanity_sample:
f.write(','.join(str(row[k]) if row[k] is not None else ''
for k in keys) + '\n')
print(f'\nSanity sample CSV: {csv_path}')
# --- Save results
summary = {
'generated_at': datetime.now().isoformat(),
'n_signatures': len(data),
'n_pixel_identical': int(pos_mask.sum()),
'n_firm_a': int(firm_a_mask.sum()),
'n_negative_anchor': int(neg_mask.sum()),
'negative_cosine_upper': NEGATIVE_COSINE_UPPER,
'eer_cosine': cos_eer,
'eer_dhash_indep': dh_eer,
'canonical_thresholds': canonical_results,
'firm_a_anchor_rates': firm_a_rates,
'cosine_sweep': cos_sweep,
'dhash_sweep': dh_sweep,
}
with open(OUT / 'pixel_validation_results.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'JSON: {OUT / "pixel_validation_results.json"}')
# --- Markdown
md = [
'# Pixel-Identity Validation Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Anchors (no human annotation required)',
'',
f'* **Pixel-identical anchor (gold positive):** '
f'{int(pos_mask.sum()):,} signatures whose closest same-accountant',
' match is byte-identical after crop/normalise. Under handwriting',
' physics this can only arise from image duplication.',
f'* **Negative anchor:** signatures whose maximum same-accountant',
f' cosine is below {NEGATIVE_COSINE_UPPER} '
f'({int(neg_mask.sum()):,} signatures). Treated as',
' confirmed not-replicated.',
f'* **Firm A anchor:** Deloitte ({int(firm_a_mask.sum()):,} signatures),',
' a replication-dominated population per interviews with multiple',
' Firm A accountants: most use replication (stamping / firm-level',
' e-signing), but a minority may still hand-sign. Used as a strong',
' prior positive for the majority regime, with the ~7% below',
' cosine 0.95 reflecting the minority hand-signers.',
'',
'## Equal Error Rate (EER)',
'',
'| Indicator | Direction | EER threshold | EER |',
'|-----------|-----------|---------------|-----|',
f"| Cosine max-similarity | > t | {cos_eer['threshold']:.4f} | "
f"{cos_eer['eer']:.4f} |",
f"| Independent min dHash | < t | {dh_eer['threshold']:.4f} | "
f"{dh_eer['eer']:.4f} |",
'',
'## Canonical thresholds',
'',
'| Indicator | Threshold | Precision | Recall | F1 | FAR | FRR |',
'|-----------|-----------|-----------|--------|----|-----|-----|',
]
for c in canonical_results:
md.append(
f"| {c['indicator']} | {c['threshold']} "
f"({c['direction']}) | {c['precision']:.3f} | "
f"{c['recall']:.3f} | {c['f1']:.3f} | "
f"{c['far']:.4f} | {c['frr']:.4f} |"
)
md += ['', '## Firm A anchor validation', '',
'| Rule | Firm A rate |',
'|------|-------------|']
for k, v in firm_a_rates.items():
md.append(f'| {k} | {v*100:.2f}% |')
md += ['', '## Sanity sample', '',
f'A stratified sample of {len(sanity_sample)} signatures '
'(pixel-identical, high-cos/low-dh, borderline, style-only, '
'likely-genuine) is exported to `sanity_sample.csv` for visual',
'spot-check. These are **not** used to compute metrics.',
'',
'## Interpretation',
'',
'Because the gold positive is a *subset* of the true replication',
'positives (only those that happen to be pixel-identical to their',
'nearest match), recall is conservative: the classifier should',
'catch pixel-identical pairs reliably and will additionally flag',
'many non-pixel-identical replications (low dHash but not zero).',
'FAR against the low-cosine negative anchor is the meaningful',
'upper bound on spurious replication flags.',
'',
'Convergence of thresholds across Scripts 15 (dip test), 16 (BD),',
'17 (Beta mixture), 18 (accountant mixture) and the EER here',
'should be reported in the paper as multi-method validation.',
]
(OUT / 'pixel_validation_report.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "pixel_validation_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/20_accountant_level_three_methods.py
+526
View File
@@ -0,0 +1,526 @@
#!/usr/bin/env python3
"""
Script 20: Three-Method Threshold Determination at the Accountant Level
=======================================================================
Completes the three-method convergent framework at the analysis level
where the mixture structure is statistically supported (per Script 15
dip test: accountant cos_mean p<0.001).
Runs on the per-accountant aggregates (mean best-match cosine, mean
independent minimum dHash) for 686 CPAs with >=10 signatures:
Method 1: KDE antimode with Hartigan dip test (formal unimodality test)
Method 2: Burgstahler-Dichev / McCrary discontinuity
Method 3: 2-component Beta mixture via EM + parallel logit-GMM
Also re-runs the accountant-level 2-component GMM crossings from
Script 18 for completeness and side-by-side comparison.
Output:
reports/accountant_three_methods/accountant_three_methods_report.md
reports/accountant_three_methods/accountant_three_methods_results.json
reports/accountant_three_methods/accountant_cos_panel.png
reports/accountant_three_methods/accountant_dhash_panel.png
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
from scipy import stats
from scipy.signal import find_peaks
from scipy.optimize import brentq
from sklearn.mixture import GaussianMixture
import diptest
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'accountant_three_methods')
OUT.mkdir(parents=True, exist_ok=True)
EPS = 1e-6
Z_CRIT = 1.96
MIN_SIGS = 10
def load_accountant_means(min_sigs=MIN_SIGS):
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.assigned_accountant,
AVG(s.max_similarity_to_same_accountant) AS cos_mean,
AVG(CAST(s.min_dhash_independent AS REAL)) AS dh_mean,
COUNT(*) AS n
FROM signatures s
WHERE s.assigned_accountant IS NOT NULL
AND s.max_similarity_to_same_accountant IS NOT NULL
AND s.min_dhash_independent IS NOT NULL
GROUP BY s.assigned_accountant
HAVING n >= ?
''', (min_sigs,))
rows = cur.fetchall()
conn.close()
cos = np.array([r[1] for r in rows])
dh = np.array([r[2] for r in rows])
return cos, dh
# ---------- Method 1: KDE antimode with dip test ----------
def method_kde_antimode(values, name):
arr = np.asarray(values, dtype=float)
arr = arr[np.isfinite(arr)]
dip, pval = diptest.diptest(arr, boot_pval=True, n_boot=2000)
kde = stats.gaussian_kde(arr, bw_method='silverman')
xs = np.linspace(arr.min(), arr.max(), 2000)
density = kde(xs)
# Find modes (local maxima) and antimodes (local minima)
peaks, _ = find_peaks(density, prominence=density.max() * 0.02)
# Antimodes = local minima between peaks
antimodes = []
for i in range(len(peaks) - 1):
seg = density[peaks[i]:peaks[i + 1]]
if len(seg) == 0:
continue
local = peaks[i] + int(np.argmin(seg))
antimodes.append(float(xs[local]))
# Sensitivity analysis across bandwidth factors
sens = {}
for bwf in [0.5, 0.75, 1.0, 1.5, 2.0]:
kde_s = stats.gaussian_kde(arr, bw_method=kde.factor * bwf)
d_s = kde_s(xs)
p_s, _ = find_peaks(d_s, prominence=d_s.max() * 0.02)
sens[f'bw_x{bwf}'] = int(len(p_s))
return {
'name': name,
'n': int(len(arr)),
'dip': float(dip),
'dip_pvalue': float(pval),
'unimodal_alpha05': bool(pval > 0.05),
'kde_bandwidth_silverman': float(kde.factor),
'n_modes': int(len(peaks)),
'mode_locations': [float(xs[p]) for p in peaks],
'antimodes': antimodes,
'primary_antimode': (antimodes[0] if antimodes else None),
'bandwidth_sensitivity_n_modes': sens,
}
# ---------- Method 2: Burgstahler-Dichev / McCrary ----------
def method_bd_mccrary(values, bin_width, direction, name):
arr = np.asarray(values, dtype=float)
arr = arr[np.isfinite(arr)]
lo = float(np.floor(arr.min() / bin_width) * bin_width)
hi = float(np.ceil(arr.max() / bin_width) * bin_width)
edges = np.arange(lo, hi + bin_width, bin_width)
counts, _ = np.histogram(arr, bins=edges)
centers = (edges[:-1] + edges[1:]) / 2.0
N = counts.sum()
p = counts / N if N else counts.astype(float)
n_bins = len(counts)
z = np.full(n_bins, np.nan)
expected = np.full(n_bins, np.nan)
for i in range(1, n_bins - 1):
p_lo = p[i - 1]
p_hi = p[i + 1]
exp_i = 0.5 * (counts[i - 1] + counts[i + 1])
var_i = (N * p[i] * (1 - p[i])
+ 0.25 * N * (p_lo + p_hi) * (1 - p_lo - p_hi))
if var_i <= 0:
continue
z[i] = (counts[i] - exp_i) / np.sqrt(var_i)
expected[i] = exp_i
# Identify transitions
transitions = []
for i in range(1, len(z)):
if np.isnan(z[i - 1]) or np.isnan(z[i]):
continue
ok = False
if direction == 'neg_to_pos' and z[i - 1] < -Z_CRIT and z[i] > Z_CRIT:
ok = True
elif direction == 'pos_to_neg' and z[i - 1] > Z_CRIT and z[i] < -Z_CRIT:
ok = True
if ok:
transitions.append({
'threshold_between': float((centers[i - 1] + centers[i]) / 2.0),
'z_before': float(z[i - 1]),
'z_after': float(z[i]),
})
best = None
if transitions:
best = max(transitions,
key=lambda t: abs(t['z_before']) + abs(t['z_after']))
return {
'name': name,
'n': int(len(arr)),
'bin_width': float(bin_width),
'direction': direction,
'n_transitions': len(transitions),
'transitions': transitions,
'best_transition': best,
'threshold': (best['threshold_between'] if best else None),
'bin_centers': [float(c) for c in centers],
'counts': [int(c) for c in counts],
'z_scores': [None if np.isnan(zi) else float(zi) for zi in z],
}
# ---------- Method 3: Beta mixture + logit-GMM ----------
def fit_beta_mixture_em(x, K=2, max_iter=300, tol=1e-6, seed=42):
rng = np.random.default_rng(seed)
x = np.clip(np.asarray(x, dtype=float), EPS, 1 - EPS)
n = len(x)
q = np.linspace(0, 1, K + 1)
thresh = np.quantile(x, q[1:-1])
labels = np.digitize(x, thresh)
resp = np.zeros((n, K))
resp[np.arange(n), labels] = 1.0
ll_hist = []
for it in range(max_iter):
nk = resp.sum(axis=0) + 1e-12
weights = nk / nk.sum()
mus = (resp * x[:, None]).sum(axis=0) / nk
var_num = (resp * (x[:, None] - mus) ** 2).sum(axis=0)
vars_ = var_num / nk
upper = mus * (1 - mus) - 1e-9
vars_ = np.minimum(vars_, upper)
vars_ = np.maximum(vars_, 1e-9)
factor = np.maximum(mus * (1 - mus) / vars_ - 1, 1e-6)
alphas = mus * factor
betas = (1 - mus) * factor
log_pdfs = np.column_stack([
stats.beta.logpdf(x, alphas[k], betas[k]) + np.log(weights[k])
for k in range(K)
])
m = log_pdfs.max(axis=1, keepdims=True)
ll = (m.ravel() + np.log(np.exp(log_pdfs - m).sum(axis=1))).sum()
ll_hist.append(float(ll))
new_resp = np.exp(log_pdfs - m)
new_resp = new_resp / new_resp.sum(axis=1, keepdims=True)
if it > 0 and abs(ll_hist[-1] - ll_hist[-2]) < tol:
resp = new_resp
break
resp = new_resp
order = np.argsort(mus)
alphas, betas, weights, mus = alphas[order], betas[order], weights[order], mus[order]
k_params = 3 * K - 1
ll_final = ll_hist[-1]
return {
'K': K,
'alphas': [float(a) for a in alphas],
'betas': [float(b) for b in betas],
'weights': [float(w) for w in weights],
'mus': [float(m) for m in mus],
'log_likelihood': float(ll_final),
'aic': float(2 * k_params - 2 * ll_final),
'bic': float(k_params * np.log(n) - 2 * ll_final),
'n_iter': it + 1,
}
def beta_crossing(fit):
if fit['K'] != 2:
return None
a1, b1, w1 = fit['alphas'][0], fit['betas'][0], fit['weights'][0]
a2, b2, w2 = fit['alphas'][1], fit['betas'][1], fit['weights'][1]
def diff(x):
return (w2 * stats.beta.pdf(x, a2, b2)
- w1 * stats.beta.pdf(x, a1, b1))
xs = np.linspace(EPS, 1 - EPS, 2000)
ys = diff(xs)
changes = np.where(np.diff(np.sign(ys)) != 0)[0]
if not len(changes):
return None
mid = 0.5 * (fit['mus'][0] + fit['mus'][1])
crossings = []
for i in changes:
try:
crossings.append(brentq(diff, xs[i], xs[i + 1]))
except ValueError:
continue
if not crossings:
return None
return float(min(crossings, key=lambda c: abs(c - mid)))
def fit_logit_gmm(x, K=2, seed=42):
x = np.clip(np.asarray(x, dtype=float), EPS, 1 - EPS)
z = np.log(x / (1 - x)).reshape(-1, 1)
gmm = GaussianMixture(n_components=K, random_state=seed,
max_iter=500).fit(z)
order = np.argsort(gmm.means_.ravel())
means = gmm.means_.ravel()[order]
stds = np.sqrt(gmm.covariances_.ravel())[order]
weights = gmm.weights_[order]
crossing = None
if K == 2:
m1, s1, w1 = means[0], stds[0], weights[0]
m2, s2, w2 = means[1], stds[1], weights[1]
def diff(z0):
return (w2 * stats.norm.pdf(z0, m2, s2)
- w1 * stats.norm.pdf(z0, m1, s1))
zs = np.linspace(min(m1, m2) - 2, max(m1, m2) + 2, 2000)
ys = diff(zs)
ch = np.where(np.diff(np.sign(ys)) != 0)[0]
if len(ch):
try:
z_cross = brentq(diff, zs[ch[0]], zs[ch[0] + 1])
crossing = float(1 / (1 + np.exp(-z_cross)))
except ValueError:
pass
return {
'K': K,
'means_logit': [float(m) for m in means],
'stds_logit': [float(s) for s in stds],
'weights': [float(w) for w in weights],
'means_original_scale': [float(1 / (1 + np.exp(-m))) for m in means],
'aic': float(gmm.aic(z)),
'bic': float(gmm.bic(z)),
'crossing_original': crossing,
}
def method_beta_mixture(values, name, is_cosine=True):
arr = np.asarray(values, dtype=float)
arr = arr[np.isfinite(arr)]
if not is_cosine:
# normalize dHash into [0,1] by dividing by 64 (max Hamming)
x = arr / 64.0
else:
x = arr
beta2 = fit_beta_mixture_em(x, K=2)
beta3 = fit_beta_mixture_em(x, K=3)
cross_beta2 = beta_crossing(beta2)
# Transform back to original scale for dHash
if not is_cosine and cross_beta2 is not None:
cross_beta2 = cross_beta2 * 64.0
gmm2 = fit_logit_gmm(x, K=2)
gmm3 = fit_logit_gmm(x, K=3)
if not is_cosine and gmm2.get('crossing_original') is not None:
gmm2['crossing_original'] = gmm2['crossing_original'] * 64.0
return {
'name': name,
'n': int(len(x)),
'scale_transform': ('identity' if is_cosine else 'dhash/64'),
'beta_2': beta2,
'beta_3': beta3,
'bic_preferred_K': (2 if beta2['bic'] < beta3['bic'] else 3),
'beta_2_crossing_original': cross_beta2,
'logit_gmm_2': gmm2,
'logit_gmm_3': gmm3,
}
# ---------- Plot helpers ----------
def plot_panel(values, methods, title, out_path, bin_width=None,
is_cosine=True):
arr = np.asarray(values, dtype=float)
fig, axes = plt.subplots(2, 1, figsize=(11, 7),
gridspec_kw={'height_ratios': [3, 1]})
ax = axes[0]
if bin_width is None:
bins = 40
else:
lo = float(np.floor(arr.min() / bin_width) * bin_width)
hi = float(np.ceil(arr.max() / bin_width) * bin_width)
bins = np.arange(lo, hi + bin_width, bin_width)
ax.hist(arr, bins=bins, density=True, alpha=0.55, color='steelblue',
edgecolor='white')
# KDE overlay
kde = stats.gaussian_kde(arr, bw_method='silverman')
xs = np.linspace(arr.min(), arr.max(), 500)
ax.plot(xs, kde(xs), 'g-', lw=1.5, label='KDE (Silverman)')
# Annotate thresholds from each method
colors = {'kde': 'green', 'bd': 'red', 'beta': 'purple', 'gmm2': 'orange'}
for key, (val, lbl) in methods.items():
if val is None:
continue
ax.axvline(val, color=colors.get(key, 'gray'), lw=2, ls='--',
label=f'{lbl} = {val:.4f}')
ax.set_xlabel(title + ' value')
ax.set_ylabel('Density')
ax.set_title(title)
ax.legend(fontsize=8)
ax2 = axes[1]
ax2.set_title('Thresholds across methods')
ax2.set_xlim(ax.get_xlim())
for i, (key, (val, lbl)) in enumerate(methods.items()):
if val is None:
continue
ax2.scatter([val], [i], color=colors.get(key, 'gray'), s=100, zorder=3)
ax2.annotate(f' {lbl}: {val:.4f}', (val, i), fontsize=8,
va='center')
ax2.set_yticks(range(len(methods)))
ax2.set_yticklabels([m for m in methods.keys()])
ax2.set_xlabel(title + ' value')
ax2.grid(alpha=0.3)
plt.tight_layout()
fig.savefig(out_path, dpi=150)
plt.close()
# ---------- GMM 2-comp crossing from Script 18 ----------
def marginal_2comp_crossing(X, dim):
gmm = GaussianMixture(n_components=2, covariance_type='full',
random_state=42, n_init=15, max_iter=500).fit(X)
means = gmm.means_
covs = gmm.covariances_
weights = gmm.weights_
m1, m2 = means[0][dim], means[1][dim]
s1 = np.sqrt(covs[0][dim, dim])
s2 = np.sqrt(covs[1][dim, dim])
w1, w2 = weights[0], weights[1]
def diff(x):
return (w2 * stats.norm.pdf(x, m2, s2)
- w1 * stats.norm.pdf(x, m1, s1))
xs = np.linspace(float(X[:, dim].min()), float(X[:, dim].max()), 2000)
ys = diff(xs)
ch = np.where(np.diff(np.sign(ys)) != 0)[0]
if not len(ch):
return None
mid = 0.5 * (m1 + m2)
crossings = []
for i in ch:
try:
crossings.append(brentq(diff, xs[i], xs[i + 1]))
except ValueError:
continue
if not crossings:
return None
return float(min(crossings, key=lambda c: abs(c - mid)))
def main():
print('=' * 70)
print('Script 20: Three-Method Threshold at Accountant Level')
print('=' * 70)
cos, dh = load_accountant_means()
print(f'\nN accountants (>={MIN_SIGS} sigs) = {len(cos)}')
results = {}
for desc, arr, bin_width, direction, is_cosine in [
('cos_mean', cos, 0.002, 'neg_to_pos', True),
('dh_mean', dh, 0.2, 'pos_to_neg', False),
]:
print(f'\n[{desc}]')
m1 = method_kde_antimode(arr, f'{desc} KDE')
print(f' Method 1 (KDE + dip): dip={m1["dip"]:.4f} '
f'p={m1["dip_pvalue"]:.4f} '
f'n_modes={m1["n_modes"]} '
f'antimode={m1["primary_antimode"]}')
m2 = method_bd_mccrary(arr, bin_width, direction, f'{desc} BD')
print(f' Method 2 (BD/McCrary): {m2["n_transitions"]} transitions, '
f'threshold={m2["threshold"]}')
m3 = method_beta_mixture(arr, f'{desc} Beta', is_cosine=is_cosine)
print(f' Method 3 (Beta mixture): BIC-preferred K={m3["bic_preferred_K"]}, '
f'Beta-2 crossing={m3["beta_2_crossing_original"]}, '
f'LogGMM-2 crossing={m3["logit_gmm_2"].get("crossing_original")}')
# GMM 2-comp crossing (for completeness / reproduce Script 18)
X = np.column_stack([cos, dh])
dim = 0 if desc == 'cos_mean' else 1
gmm2_crossing = marginal_2comp_crossing(X, dim)
print(f' (Script 18 2-comp GMM marginal crossing = {gmm2_crossing})')
results[desc] = {
'method_1_kde_antimode': m1,
'method_2_bd_mccrary': m2,
'method_3_beta_mixture': m3,
'script_18_gmm_2comp_crossing': gmm2_crossing,
}
methods_for_plot = {
'kde': (m1.get('primary_antimode'), 'KDE antimode'),
'bd': (m2.get('threshold'), 'BD/McCrary'),
'beta': (m3.get('beta_2_crossing_original'), 'Beta-2 crossing'),
'gmm2': (gmm2_crossing, 'GMM 2-comp crossing'),
}
png = OUT / f'accountant_{desc}_panel.png'
plot_panel(arr, methods_for_plot,
f'Accountant-level {desc}: three-method thresholds',
png, bin_width=bin_width, is_cosine=is_cosine)
print(f' plot: {png}')
# Write JSON
with open(OUT / 'accountant_three_methods_results.json', 'w') as f:
json.dump({'generated_at': datetime.now().isoformat(),
'n_accountants': int(len(cos)),
'min_signatures': MIN_SIGS,
'results': results}, f, indent=2, ensure_ascii=False)
print(f'\nJSON: {OUT / "accountant_three_methods_results.json"}')
# Markdown
md = [
'# Accountant-Level Three-Method Threshold Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
f'N accountants (>={MIN_SIGS} signatures): {len(cos)}',
'',
'## Accountant-level cosine mean',
'',
'| Method | Threshold | Supporting statistic |',
'|--------|-----------|----------------------|',
]
r = results['cos_mean']
md.append(f"| Method 1: KDE antimode (with dip test) | "
f"{r['method_1_kde_antimode']['primary_antimode']} | "
f"dip={r['method_1_kde_antimode']['dip']:.4f}, "
f"p={r['method_1_kde_antimode']['dip_pvalue']:.4f} "
f"({'unimodal' if r['method_1_kde_antimode']['unimodal_alpha05'] else 'multimodal'}) |")
md.append(f"| Method 2: Burgstahler-Dichev / McCrary | "
f"{r['method_2_bd_mccrary']['threshold']} | "
f"{r['method_2_bd_mccrary']['n_transitions']} transition(s) "
f"at α=0.05 |")
md.append(f"| Method 3: 2-component Beta mixture | "
f"{r['method_3_beta_mixture']['beta_2_crossing_original']} | "
f"Beta-2 BIC={r['method_3_beta_mixture']['beta_2']['bic']:.2f}, "
f"Beta-3 BIC={r['method_3_beta_mixture']['beta_3']['bic']:.2f} "
f"(BIC-preferred K={r['method_3_beta_mixture']['bic_preferred_K']}) |")
md.append(f"| Method 3': LogGMM-2 on logit-transformed | "
f"{r['method_3_beta_mixture']['logit_gmm_2'].get('crossing_original')} | "
f"White 1982 quasi-MLE robustness check |")
md.append(f"| Script 18 GMM 2-comp marginal crossing | "
f"{r['script_18_gmm_2comp_crossing']} | full 2D mixture |")
md += ['', '## Accountant-level dHash mean', '',
'| Method | Threshold | Supporting statistic |',
'|--------|-----------|----------------------|']
r = results['dh_mean']
md.append(f"| Method 1: KDE antimode | "
f"{r['method_1_kde_antimode']['primary_antimode']} | "
f"dip={r['method_1_kde_antimode']['dip']:.4f}, "
f"p={r['method_1_kde_antimode']['dip_pvalue']:.4f} |")
md.append(f"| Method 2: BD/McCrary | "
f"{r['method_2_bd_mccrary']['threshold']} | "
f"{r['method_2_bd_mccrary']['n_transitions']} transition(s) |")
md.append(f"| Method 3: 2-component Beta mixture | "
f"{r['method_3_beta_mixture']['beta_2_crossing_original']} | "
f"Beta-2 BIC={r['method_3_beta_mixture']['beta_2']['bic']:.2f}, "
f"Beta-3 BIC={r['method_3_beta_mixture']['beta_3']['bic']:.2f} |")
md.append(f"| Method 3': LogGMM-2 | "
f"{r['method_3_beta_mixture']['logit_gmm_2'].get('crossing_original')} | |")
md.append(f"| Script 18 GMM 2-comp crossing | "
f"{r['script_18_gmm_2comp_crossing']} | |")
(OUT / 'accountant_three_methods_report.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "accountant_three_methods_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/21_expanded_validation.py
+421
View File
@@ -0,0 +1,421 @@
#!/usr/bin/env python3
"""
Script 21: Expanded Validation with Larger Negative Anchor + Held-out Firm A
============================================================================
Addresses codex review weaknesses of Script 19's pixel-identity validation:
(a) Negative anchor of n=35 (cosine<0.70) is too small to give
meaningful FAR confidence intervals.
(b) Pixel-identical positive anchor is an easy subset, not
representative of the broader positive class.
(c) Firm A is both the calibration anchor and the validation anchor
(circular).
This script:
1. Constructs a large inter-CPA negative anchor (~50,000 pairs) by
randomly sampling pairs from different CPAs. Inter-CPA high
similarity is highly unlikely to arise from legitimate signing.
2. Splits Firm A CPAs 70/30 into CALIBRATION and HELDOUT folds.
Re-derives signature-level / accountant-level thresholds from the
calibration fold only, then reports all metrics (including Firm A
anchor rates) on the heldout fold.
3. Computes proper EER (FAR = FRR interpolated) in addition to
metrics at canonical thresholds.
4. Computes 95% Wilson confidence intervals for each FAR/FRR.
Output:
reports/expanded_validation/expanded_validation_report.md
reports/expanded_validation/expanded_validation_results.json
"""
import sqlite3
import json
import numpy as np
from pathlib import Path
from datetime import datetime
from scipy.stats import norm
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'expanded_validation')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
N_INTER_PAIRS = 50_000
SEED = 42
def wilson_ci(k, n, alpha=0.05):
if n == 0:
return (0.0, 1.0)
z = norm.ppf(1 - alpha / 2)
phat = k / n
denom = 1 + z * z / n
center = (phat + z * z / (2 * n)) / denom
pm = z * np.sqrt(phat * (1 - phat) / n + z * z / (4 * n * n)) / denom
return (max(0.0, center - pm), min(1.0, center + pm))
def load_signatures():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.signature_id, s.assigned_accountant, a.firm,
s.max_similarity_to_same_accountant,
s.min_dhash_independent, s.pixel_identical_to_closest
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE s.max_similarity_to_same_accountant IS NOT NULL
''')
rows = cur.fetchall()
conn.close()
return rows
def load_feature_vectors_sample(n=2000):
"""Load feature vectors for inter-CPA negative-anchor sampling."""
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT signature_id, assigned_accountant, feature_vector
FROM signatures
WHERE feature_vector IS NOT NULL
AND assigned_accountant IS NOT NULL
ORDER BY RANDOM()
LIMIT ?
''', (n,))
rows = cur.fetchall()
conn.close()
out = []
for r in rows:
vec = np.frombuffer(r[2], dtype=np.float32)
out.append({'sig_id': r[0], 'accountant': r[1], 'feature': vec})
return out
def build_inter_cpa_negative(sample, n_pairs=N_INTER_PAIRS, seed=SEED):
"""Sample random cross-CPA pairs; return their cosine similarities."""
rng = np.random.default_rng(seed)
n = len(sample)
feats = np.stack([s['feature'] for s in sample])
accts = np.array([s['accountant'] for s in sample])
sims = []
tries = 0
while len(sims) < n_pairs and tries < n_pairs * 10:
i = rng.integers(n)
j = rng.integers(n)
if i == j or accts[i] == accts[j]:
tries += 1
continue
sim = float(feats[i] @ feats[j])
sims.append(sim)
tries += 1
return np.array(sims)
def classification_metrics(y_true, y_pred):
y_true = np.asarray(y_true).astype(int)
y_pred = np.asarray(y_pred).astype(int)
tp = int(np.sum((y_true == 1) & (y_pred == 1)))
fp = int(np.sum((y_true == 0) & (y_pred == 1)))
fn = int(np.sum((y_true == 1) & (y_pred == 0)))
tn = int(np.sum((y_true == 0) & (y_pred == 0)))
p_den = max(tp + fp, 1)
r_den = max(tp + fn, 1)
far_den = max(fp + tn, 1)
frr_den = max(fn + tp, 1)
precision = tp / p_den
recall = tp / r_den
f1 = (2 * precision * recall / (precision + recall)
if (precision + recall) > 0 else 0.0)
far = fp / far_den
frr = fn / frr_den
far_ci = wilson_ci(fp, far_den)
frr_ci = wilson_ci(fn, frr_den)
return {
'tp': tp, 'fp': fp, 'fn': fn, 'tn': tn,
'precision': float(precision),
'recall': float(recall),
'f1': float(f1),
'far': float(far),
'frr': float(frr),
'far_ci95': [float(x) for x in far_ci],
'frr_ci95': [float(x) for x in frr_ci],
'n_pos': int(tp + fn),
'n_neg': int(tn + fp),
}
def sweep_threshold(scores, y, direction, thresholds):
out = []
for t in thresholds:
if direction == 'above':
y_pred = (scores > t).astype(int)
else:
y_pred = (scores < t).astype(int)
m = classification_metrics(y, y_pred)
m['threshold'] = float(t)
out.append(m)
return out
def find_eer(sweep):
thr = np.array([s['threshold'] for s in sweep])
far = np.array([s['far'] for s in sweep])
frr = np.array([s['frr'] for s in sweep])
diff = far - frr
signs = np.sign(diff)
changes = np.where(np.diff(signs) != 0)[0]
if len(changes) == 0:
idx = int(np.argmin(np.abs(diff)))
return {'threshold': float(thr[idx]), 'far': float(far[idx]),
'frr': float(frr[idx]),
'eer': float(0.5 * (far[idx] + frr[idx]))}
i = int(changes[0])
w = abs(diff[i]) / (abs(diff[i]) + abs(diff[i + 1]) + 1e-12)
thr_i = (1 - w) * thr[i] + w * thr[i + 1]
far_i = (1 - w) * far[i] + w * far[i + 1]
frr_i = (1 - w) * frr[i] + w * frr[i + 1]
return {'threshold': float(thr_i), 'far': float(far_i),
'frr': float(frr_i),
'eer': float(0.5 * (far_i + frr_i))}
def main():
print('=' * 70)
print('Script 21: Expanded Validation')
print('=' * 70)
rows = load_signatures()
print(f'\nLoaded {len(rows):,} signatures')
sig_ids = [r[0] for r in rows]
accts = [r[1] for r in rows]
firms = [r[2] or '(unknown)' for r in rows]
cos = np.array([r[3] for r in rows], dtype=float)
dh = np.array([-1 if r[4] is None else r[4] for r in rows], dtype=float)
pix = np.array([r[5] or 0 for r in rows], dtype=int)
firm_a_mask = np.array([f == FIRM_A for f in firms])
print(f'Firm A signatures: {int(firm_a_mask.sum()):,}')
# --- (1) INTER-CPA NEGATIVE ANCHOR ---
print(f'\n[1] Building inter-CPA negative anchor ({N_INTER_PAIRS} pairs)...')
sample = load_feature_vectors_sample(n=3000)
inter_cos = build_inter_cpa_negative(sample, n_pairs=N_INTER_PAIRS)
print(f' inter-CPA cos: mean={inter_cos.mean():.4f}, '
f'p95={np.percentile(inter_cos, 95):.4f}, '
f'p99={np.percentile(inter_cos, 99):.4f}, '
f'max={inter_cos.max():.4f}')
# --- (2) POSITIVES ---
# Pixel-identical (gold) + optional Firm A extension
pos_pix_mask = pix == 1
n_pix = int(pos_pix_mask.sum())
print(f'\n[2] Positive anchors:')
print(f' pixel-identical signatures: {n_pix}')
# Build negative anchor scores = inter-CPA cosine distribution
# Positive anchor scores = pixel-identical signatures' max same-CPA cosine
# NB: the two distributions are not drawn from the same random variable
# (one is intra-CPA max, the other is inter-CPA random), so we treat the
# inter-CPA distribution as a negative reference for threshold sweep.
# Combined labeled set: positives=pixel-identical sigs' max cosine,
# negatives=inter-CPA random pair cosines.
pos_scores = cos[pos_pix_mask]
neg_scores = inter_cos
y = np.concatenate([np.ones(len(pos_scores)),
np.zeros(len(neg_scores))])
scores = np.concatenate([pos_scores, neg_scores])
# Sweep thresholds
thr = np.linspace(0.30, 1.00, 141)
sweep = sweep_threshold(scores, y, 'above', thr)
eer = find_eer(sweep)
print(f'\n[3] Cosine EER (pos=pixel-identical, neg=inter-CPA n={len(inter_cos)}):')
print(f" threshold={eer['threshold']:.4f}, EER={eer['eer']:.4f}")
# Canonical threshold evaluations with Wilson CIs
canonical = {}
for tt in [0.70, 0.80, 0.837, 0.90, 0.945, 0.95, 0.973, 0.979]:
y_pred = (scores > tt).astype(int)
m = classification_metrics(y, y_pred)
m['threshold'] = float(tt)
canonical[f'cos>{tt:.3f}'] = m
print(f" @ {tt:.3f}: P={m['precision']:.3f}, R={m['recall']:.3f}, "
f"FAR={m['far']:.4f} (CI95={m['far_ci95'][0]:.4f}-"
f"{m['far_ci95'][1]:.4f}), FRR={m['frr']:.4f}")
# --- (3) HELD-OUT FIRM A ---
print('\n[4] Held-out Firm A 70/30 split:')
rng = np.random.default_rng(SEED)
firm_a_accts = sorted(set(a for a, f in zip(accts, firms) if f == FIRM_A))
rng.shuffle(firm_a_accts)
n_calib = int(0.7 * len(firm_a_accts))
calib_accts = set(firm_a_accts[:n_calib])
heldout_accts = set(firm_a_accts[n_calib:])
print(f' Calibration fold CPAs: {len(calib_accts)}, '
f'heldout fold CPAs: {len(heldout_accts)}')
calib_mask = np.array([a in calib_accts for a in accts])
heldout_mask = np.array([a in heldout_accts for a in accts])
print(f' Calibration sigs: {int(calib_mask.sum())}, '
f'heldout sigs: {int(heldout_mask.sum())}')
# Derive per-signature thresholds from calibration fold:
# - Firm A cos median, 1st-pct, 5th-pct
# - Firm A dHash median, 95th-pct
calib_cos = cos[calib_mask]
calib_dh = dh[calib_mask]
calib_dh = calib_dh[calib_dh >= 0]
cal_cos_med = float(np.median(calib_cos))
cal_cos_p1 = float(np.percentile(calib_cos, 1))
cal_cos_p5 = float(np.percentile(calib_cos, 5))
cal_dh_med = float(np.median(calib_dh))
cal_dh_p95 = float(np.percentile(calib_dh, 95))
print(f' Calib Firm A cos: median={cal_cos_med:.4f}, P1={cal_cos_p1:.4f}, P5={cal_cos_p5:.4f}')
print(f' Calib Firm A dHash: median={cal_dh_med:.2f}, P95={cal_dh_p95:.2f}')
# Apply canonical rules to heldout fold
held_cos = cos[heldout_mask]
held_dh = dh[heldout_mask]
held_dh_valid = held_dh >= 0
held_rates = {}
for tt in [0.837, 0.945, 0.95, cal_cos_p5]:
rate = float(np.mean(held_cos > tt))
k = int(np.sum(held_cos > tt))
lo, hi = wilson_ci(k, len(held_cos))
held_rates[f'cos>{tt:.4f}'] = {
'rate': rate, 'k': k, 'n': int(len(held_cos)),
'wilson95': [float(lo), float(hi)],
}
for tt in [5, 8, 15, cal_dh_p95]:
rate = float(np.mean(held_dh[held_dh_valid] <= tt))
k = int(np.sum(held_dh[held_dh_valid] <= tt))
lo, hi = wilson_ci(k, int(held_dh_valid.sum()))
held_rates[f'dh_indep<={tt:.2f}'] = {
'rate': rate, 'k': k, 'n': int(held_dh_valid.sum()),
'wilson95': [float(lo), float(hi)],
}
# Dual rule
dual_mask = (held_cos > 0.95) & (held_dh >= 0) & (held_dh <= 8)
rate = float(np.mean(dual_mask))
k = int(dual_mask.sum())
lo, hi = wilson_ci(k, len(dual_mask))
held_rates['cos>0.95 AND dh<=8'] = {
'rate': rate, 'k': k, 'n': int(len(dual_mask)),
'wilson95': [float(lo), float(hi)],
}
print(' Heldout Firm A rates:')
for k, v in held_rates.items():
print(f' {k}: {v["rate"]*100:.2f}% '
f'[{v["wilson95"][0]*100:.2f}, {v["wilson95"][1]*100:.2f}]')
# --- Save ---
summary = {
'generated_at': datetime.now().isoformat(),
'n_signatures': len(rows),
'n_firm_a': int(firm_a_mask.sum()),
'n_pixel_identical': n_pix,
'n_inter_cpa_negatives': len(inter_cos),
'inter_cpa_cos_stats': {
'mean': float(inter_cos.mean()),
'p95': float(np.percentile(inter_cos, 95)),
'p99': float(np.percentile(inter_cos, 99)),
'max': float(inter_cos.max()),
},
'cosine_eer': eer,
'canonical_thresholds': canonical,
'held_out_firm_a': {
'calibration_cpas': len(calib_accts),
'heldout_cpas': len(heldout_accts),
'calibration_sig_count': int(calib_mask.sum()),
'heldout_sig_count': int(heldout_mask.sum()),
'calib_cos_median': cal_cos_med,
'calib_cos_p1': cal_cos_p1,
'calib_cos_p5': cal_cos_p5,
'calib_dh_median': cal_dh_med,
'calib_dh_p95': cal_dh_p95,
'heldout_rates': held_rates,
},
}
with open(OUT / 'expanded_validation_results.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'\nJSON: {OUT / "expanded_validation_results.json"}')
# Markdown
md = [
'# Expanded Validation Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## 1. Inter-CPA Negative Anchor',
'',
f'* N random cross-CPA pairs sampled: {len(inter_cos):,}',
f'* Inter-CPA cosine: mean={inter_cos.mean():.4f}, '
f'P95={np.percentile(inter_cos, 95):.4f}, '
f'P99={np.percentile(inter_cos, 99):.4f}, max={inter_cos.max():.4f}',
'',
'This anchor is a meaningful negative set because inter-CPA pairs',
'cannot arise from legitimate reuse of a single signer\'s image.',
'',
'## 2. Cosine Threshold Sweep (pos=pixel-identical, neg=inter-CPA)',
'',
f"EER threshold: {eer['threshold']:.4f}, EER: {eer['eer']:.4f}",
'',
'| Threshold | Precision | Recall | F1 | FAR | FAR 95% CI | FRR |',
'|-----------|-----------|--------|----|-----|------------|-----|',
]
for k, m in canonical.items():
md.append(
f"| {m['threshold']:.3f} | {m['precision']:.3f} | "
f"{m['recall']:.3f} | {m['f1']:.3f} | {m['far']:.4f} | "
f"[{m['far_ci95'][0]:.4f}, {m['far_ci95'][1]:.4f}] | "
f"{m['frr']:.4f} |"
)
md += [
'',
'## 3. Held-out Firm A 70/30 Validation',
'',
f'* Firm A CPAs randomly split by CPA (not by signature) into',
f' calibration (n={len(calib_accts)}) and heldout (n={len(heldout_accts)}).',
f'* Calibration Firm A signatures: {int(calib_mask.sum()):,}. '
f'Heldout signatures: {int(heldout_mask.sum()):,}.',
'',
'### Calibration-fold anchor statistics (for thresholds)',
'',
f'* Firm A cosine: median = {cal_cos_med:.4f}, '
f'P1 = {cal_cos_p1:.4f}, P5 = {cal_cos_p5:.4f}',
f'* Firm A dHash (independent min): median = {cal_dh_med:.2f}, '
f'P95 = {cal_dh_p95:.2f}',
'',
'### Heldout-fold capture rates (with Wilson 95% CIs)',
'',
'| Rule | Heldout rate | Wilson 95% CI | k / n |',
'|------|--------------|---------------|-------|',
]
for k, v in held_rates.items():
md.append(
f"| {k} | {v['rate']*100:.2f}% | "
f"[{v['wilson95'][0]*100:.2f}%, {v['wilson95'][1]*100:.2f}%] | "
f"{v['k']}/{v['n']} |"
)
md += [
'',
'## Interpretation',
'',
'The inter-CPA negative anchor (N ~50,000) gives tight confidence',
'intervals on FAR at each threshold, addressing the small-negative',
'anchor limitation of Script 19 (n=35).',
'',
'The 70/30 Firm A split breaks the circular-validation concern of',
'using the same calibration anchor for threshold derivation and',
'validation. Calibration-fold percentiles derive the thresholds;',
'heldout-fold rates with Wilson 95% CIs show how those thresholds',
'generalize to Firm A CPAs that did not contribute to calibration.',
]
(OUT / 'expanded_validation_report.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "expanded_validation_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/22_partner_ranking.py
+279
View File
@@ -0,0 +1,279 @@
#!/usr/bin/env python3
"""
Script 22: Partner-Level Similarity Ranking (per Partner v4 Section F.3)
========================================================================
Rank all Big-4 engagement partners by their per-auditor-year max cosine
similarity. Under Partner v4's benchmark validation argument, if Deloitte
Taiwan applies firm-wide stamping, Deloitte partners should disproportionately
occupy the upper ranks of the cosine distribution.
Construction:
- Unit of observation: auditor-year = (CPA name, fiscal year)
- For each auditor-year compute:
cos_auditor_year = mean(max_similarity_to_same_accountant)
over that CPA's signatures in that year
- Only include auditor-years with >= 5 signatures
- Rank globally; compute per-firm share of top-K buckets
- Report for the pooled 2013-2023 sample and year-by-year
Output:
reports/partner_ranking/partner_ranking_report.md
reports/partner_ranking/partner_ranking_results.json
reports/partner_ranking/partner_rank_distribution.png
"""
import sqlite3
import json
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from pathlib import Path
from datetime import datetime
from collections import defaultdict
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'partner_ranking')
OUT.mkdir(parents=True, exist_ok=True)
BIG4 = ['勤業眾信聯合', '安侯建業聯合', '資誠聯合', '安永聯合']
FIRM_A = '勤業眾信聯合'
MIN_SIGS_PER_AUDITOR_YEAR = 5
def load_auditor_years():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.assigned_accountant, a.firm,
substr(s.year_month, 1, 4) AS year,
AVG(s.max_similarity_to_same_accountant) AS cos_mean,
COUNT(*) AS n
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE s.assigned_accountant IS NOT NULL
AND s.max_similarity_to_same_accountant IS NOT NULL
AND s.year_month IS NOT NULL
GROUP BY s.assigned_accountant, year
HAVING n >= ?
''', (MIN_SIGS_PER_AUDITOR_YEAR,))
rows = cur.fetchall()
conn.close()
return [{'accountant': r[0],
'firm': r[1] or '(unknown)',
'year': int(r[2]),
'cos_mean': float(r[3]),
'n': int(r[4])} for r in rows]
def firm_bucket(firm):
if firm == '勤業眾信聯合':
return 'Deloitte (Firm A)'
elif firm == '安侯建業聯合':
return 'KPMG'
elif firm == '資誠聯合':
return 'PwC'
elif firm == '安永聯合':
return 'EY'
else:
return 'Other / Non-Big-4'
def top_decile_breakdown(rows, deciles=(10, 25, 50)):
"""For pooled or per-year rows, compute % of top-K positions by firm."""
sorted_rows = sorted(rows, key=lambda r: -r['cos_mean'])
N = len(sorted_rows)
results = {}
for decile in deciles:
k = max(1, int(N * decile / 100))
top = sorted_rows[:k]
counts = defaultdict(int)
for r in top:
counts[firm_bucket(r['firm'])] += 1
results[f'top_{decile}pct'] = {
'k': k,
'N_total': N,
'by_firm': dict(counts),
'deloitte_share': counts['Deloitte (Firm A)'] / k,
}
return results
def main():
print('=' * 70)
print('Script 22: Partner-Level Similarity Ranking')
print('=' * 70)
rows = load_auditor_years()
print(f'\nN auditor-years (>= {MIN_SIGS_PER_AUDITOR_YEAR} sigs): {len(rows):,}')
# Firm-level counts
firm_counts = defaultdict(int)
for r in rows:
firm_counts[firm_bucket(r['firm'])] += 1
print('\nAuditor-years by firm:')
for f, c in sorted(firm_counts.items(), key=lambda x: -x[1]):
print(f' {f}: {c}')
# POOLED (2013-2023)
print('\n--- POOLED 2013-2023 ---')
pooled = top_decile_breakdown(rows)
for bucket, data in pooled.items():
print(f' {bucket} (top {data["k"]} of {data["N_total"]}): '
f'Deloitte share = {data["deloitte_share"]*100:.1f}%')
for firm, c in sorted(data['by_firm'].items(), key=lambda x: -x[1]):
print(f' {firm}: {c}')
# PER-YEAR
print('\n--- PER-YEAR TOP-10% DELOITTE SHARE ---')
per_year = {}
for year in sorted(set(r['year'] for r in rows)):
year_rows = [r for r in rows if r['year'] == year]
breakdown = top_decile_breakdown(year_rows)
per_year[year] = breakdown
top10 = breakdown['top_10pct']
print(f' {year}: N={top10["N_total"]}, top-10% k={top10["k"]}, '
f'Deloitte share = {top10["deloitte_share"]*100:.1f}%, '
f'Deloitte count={top10["by_firm"].get("Deloitte (Firm A)",0)}')
# Figure: partner rank distribution by firm
sorted_rows = sorted(rows, key=lambda r: -r['cos_mean'])
ranks_by_firm = defaultdict(list)
for idx, r in enumerate(sorted_rows):
ranks_by_firm[firm_bucket(r['firm'])].append(idx / len(sorted_rows))
fig, axes = plt.subplots(1, 2, figsize=(14, 5))
# (a) Stacked CDF of rank percentile by firm
ax = axes[0]
colors = {'Deloitte (Firm A)': '#d62728', 'KPMG': '#1f77b4',
'PwC': '#2ca02c', 'EY': '#9467bd',
'Other / Non-Big-4': '#7f7f7f'}
for firm in ['Deloitte (Firm A)', 'KPMG', 'PwC', 'EY', 'Other / Non-Big-4']:
if firm in ranks_by_firm and ranks_by_firm[firm]:
sorted_pct = sorted(ranks_by_firm[firm])
ax.hist(sorted_pct, bins=40, alpha=0.55, density=True,
label=f'{firm} (n={len(sorted_pct)})',
color=colors.get(firm, 'gray'))
ax.set_xlabel('Rank percentile (0 = highest similarity)')
ax.set_ylabel('Density')
ax.set_title('Auditor-year rank distribution by firm (pooled 2013-2023)')
ax.legend(fontsize=9)
# (b) Deloitte share of top-10% per year
ax = axes[1]
years = sorted(per_year.keys())
shares = [per_year[y]['top_10pct']['deloitte_share'] * 100 for y in years]
base_share = [100.0 * sum(1 for r in rows if r['year'] == y
and firm_bucket(r['firm']) == 'Deloitte (Firm A)')
/ sum(1 for r in rows if r['year'] == y) for y in years]
ax.plot(years, shares, 'o-', color='#d62728', lw=2,
label='Deloitte share of top-10% similarity')
ax.plot(years, base_share, 's--', color='gray', lw=1.5,
label='Deloitte baseline share of auditor-years')
ax.set_xlabel('Fiscal year')
ax.set_ylabel('Share (%)')
ax.set_ylim(0, max(max(shares), max(base_share)) * 1.2)
ax.set_title('Deloitte concentration in top-similarity auditor-years')
ax.legend(fontsize=9)
ax.grid(alpha=0.3)
plt.tight_layout()
fig.savefig(OUT / 'partner_rank_distribution.png', dpi=150)
plt.close()
print(f'\nFigure: {OUT / "partner_rank_distribution.png"}')
# JSON
summary = {
'generated_at': datetime.now().isoformat(),
'min_signatures_per_auditor_year': MIN_SIGS_PER_AUDITOR_YEAR,
'n_auditor_years': len(rows),
'firm_counts': dict(firm_counts),
'pooled_deciles': pooled,
'per_year': {int(k): v for k, v in per_year.items()},
}
with open(OUT / 'partner_ranking_results.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'JSON: {OUT / "partner_ranking_results.json"}')
# Markdown
md = [
'# Partner-Level Similarity Ranking Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Method',
'',
f'* Unit of observation: auditor-year (CPA name, fiscal year) with '
f'at least {MIN_SIGS_PER_AUDITOR_YEAR} signatures in that year.',
'* Similarity statistic: mean of max_similarity_to_same_accountant',
' across signatures in the auditor-year.',
'* Auditor-years ranked globally; per-firm share of top-K positions',
' reported for the pooled 2013-2023 sample and per fiscal year.',
'',
f'Total auditor-years analyzed: **{len(rows):,}**',
'',
'## Auditor-year counts by firm',
'',
'| Firm | N auditor-years |',
'|------|-----------------|',
]
for f, c in sorted(firm_counts.items(), key=lambda x: -x[1]):
md.append(f'| {f} | {c} |')
md += ['', '## Top-K concentration (pooled 2013-2023)', '',
'| Top-K | N in bucket | Deloitte | KPMG | PwC | EY | Other | Deloitte share |',
'|-------|-------------|----------|------|-----|-----|-------|----------------|']
for key in ('top_10pct', 'top_25pct', 'top_50pct'):
d = pooled[key]
md.append(
f"| {key.replace('top_', 'Top ').replace('pct', '%')} | "
f"{d['k']} | "
f"{d['by_firm'].get('Deloitte (Firm A)', 0)} | "
f"{d['by_firm'].get('KPMG', 0)} | "
f"{d['by_firm'].get('PwC', 0)} | "
f"{d['by_firm'].get('EY', 0)} | "
f"{d['by_firm'].get('Other / Non-Big-4', 0)} | "
f"**{d['deloitte_share']*100:.1f}%** |"
)
md += ['', '## Per-year Deloitte share of top-10% similarity', '',
'| Year | N auditor-years | Top-10% k | Deloitte in top-10% | '
'Deloitte share | Deloitte baseline share |',
'|------|-----------------|-----------|---------------------|'
'----------------|-------------------------|']
for y in sorted(per_year.keys()):
d = per_year[y]['top_10pct']
baseline = sum(1 for r in rows if r['year'] == y
and firm_bucket(r['firm']) == 'Deloitte (Firm A)') \
/ sum(1 for r in rows if r['year'] == y)
md.append(
f"| {y} | {d['N_total']} | {d['k']} | "
f"{d['by_firm'].get('Deloitte (Firm A)', 0)} | "
f"{d['deloitte_share']*100:.1f}% | "
f"{baseline*100:.1f}% |"
)
md += [
'',
'## Interpretation',
'',
'If Deloitte Taiwan applies firm-wide stamping, Deloitte auditor-years',
'should over-represent in the top of the similarity distribution relative',
'to their baseline share of all auditor-years. The pooled top-10%',
'Deloitte share divided by the baseline gives a concentration ratio',
"that is informative about the firm's signing practice without",
'requiring per-report ground-truth labels.',
'',
'Year-by-year stability of this concentration provides evidence about',
'whether the stamping practice was maintained throughout 2013-2023 or',
'changed in response to the industry-wide shift to electronic signing',
'systems around 2020.',
]
(OUT / 'partner_ranking_report.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "partner_ranking_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/23_intra_report_consistency.py
+282
View File
@@ -0,0 +1,282 @@
#!/usr/bin/env python3
"""
Script 23: Intra-Report Consistency Check (per Partner v4 Section F.4)
======================================================================
Taiwanese statutory audit reports are co-signed by two engagement partners
(primary + secondary). Under firm-wide stamping practice, both signatures
on the same report should be classified as non-hand-signed.
This script:
1. Identifies reports with exactly 2 signatures in the DB.
2. Classifies each signature using the dual-descriptor thresholds of the
paper (cosine > 0.95 AND dHash_indep <= 8 = high-confidence replication).
3. Reports intra-report agreement per firm.
4. Flags disagreement cases for sensitivity analysis.
Output:
reports/intra_report/intra_report_report.md
reports/intra_report/intra_report_results.json
reports/intra_report/intra_report_disagreements.csv
"""
import sqlite3
import json
import numpy as np
from pathlib import Path
from datetime import datetime
from collections import defaultdict
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'intra_report')
OUT.mkdir(parents=True, exist_ok=True)
BIG4 = ['勤業眾信聯合', '安侯建業聯合', '資誠聯合', '安永聯合']
def classify_signature(cos, dhash_indep):
"""Return one of: high_conf_non_hand_signed, moderate_non_hand_signed,
style_consistency, uncertain, likely_hand_signed,
unknown (if missing data)."""
if cos is None:
return 'unknown'
if cos > 0.95 and dhash_indep is not None and dhash_indep <= 5:
return 'high_conf_non_hand_signed'
if cos > 0.95 and dhash_indep is not None and 5 < dhash_indep <= 15:
return 'moderate_non_hand_signed'
if cos > 0.95 and dhash_indep is not None and dhash_indep > 15:
return 'style_consistency'
if 0.837 < cos <= 0.95:
return 'uncertain'
if cos <= 0.837:
return 'likely_hand_signed'
return 'unknown'
def binary_bucket(label):
"""Collapse to binary: non_hand_signed vs hand_signed vs other."""
if label in ('high_conf_non_hand_signed', 'moderate_non_hand_signed'):
return 'non_hand_signed'
if label == 'likely_hand_signed':
return 'hand_signed'
if label == 'style_consistency':
return 'style_consistency'
return 'uncertain'
def firm_bucket(firm):
if firm == '勤業眾信聯合':
return 'Deloitte (Firm A)'
elif firm == '安侯建業聯合':
return 'KPMG'
elif firm == '資誠聯合':
return 'PwC'
elif firm == '安永聯合':
return 'EY'
return 'Other / Non-Big-4'
def load_two_signer_reports():
conn = sqlite3.connect(DB)
cur = conn.cursor()
# Select reports that have exactly 2 signatures with complete data
cur.execute('''
WITH report_counts AS (
SELECT source_pdf, COUNT(*) AS n_sigs
FROM signatures
WHERE max_similarity_to_same_accountant IS NOT NULL
GROUP BY source_pdf
)
SELECT s.source_pdf, s.signature_id, s.assigned_accountant, a.firm,
s.max_similarity_to_same_accountant,
s.min_dhash_independent, s.sig_index, s.year_month
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
JOIN report_counts rc ON rc.source_pdf = s.source_pdf
WHERE rc.n_sigs = 2
AND s.max_similarity_to_same_accountant IS NOT NULL
ORDER BY s.source_pdf, s.sig_index
''')
rows = cur.fetchall()
conn.close()
return rows
def main():
print('=' * 70)
print('Script 23: Intra-Report Consistency Check')
print('=' * 70)
rows = load_two_signer_reports()
print(f'\nLoaded {len(rows):,} signatures from 2-signer reports')
# Group by source_pdf
by_pdf = defaultdict(list)
for r in rows:
by_pdf[r[0]].append({
'sig_id': r[1], 'accountant': r[2], 'firm': r[3] or '(unknown)',
'cos': r[4], 'dhash': r[5], 'sig_index': r[6], 'year_month': r[7],
})
reports = [{'pdf': pdf, 'sigs': sigs}
for pdf, sigs in by_pdf.items() if len(sigs) == 2]
print(f'Total 2-signer reports: {len(reports):,}')
# Classify each signature and check agreement
results = {
'total_reports': len(reports),
'by_firm': defaultdict(lambda: {
'total': 0,
'both_non_hand_signed': 0,
'both_hand_signed': 0,
'both_style_consistency': 0,
'both_uncertain': 0,
'mixed': 0,
'mixed_details': defaultdict(int),
}),
}
disagreements = []
for rep in reports:
s1, s2 = rep['sigs']
l1 = classify_signature(s1['cos'], s1['dhash'])
l2 = classify_signature(s2['cos'], s2['dhash'])
b1, b2 = binary_bucket(l1), binary_bucket(l2)
# Determine report-level firm (usually both signers from same firm)
firm1 = firm_bucket(s1['firm'])
firm2 = firm_bucket(s2['firm'])
firm = firm1 if firm1 == firm2 else f'{firm1}+{firm2}'
bucket = results['by_firm'][firm]
bucket['total'] += 1
if b1 == b2 == 'non_hand_signed':
bucket['both_non_hand_signed'] += 1
elif b1 == b2 == 'hand_signed':
bucket['both_hand_signed'] += 1
elif b1 == b2 == 'style_consistency':
bucket['both_style_consistency'] += 1
elif b1 == b2 == 'uncertain':
bucket['both_uncertain'] += 1
else:
bucket['mixed'] += 1
combo = tuple(sorted([b1, b2]))
bucket['mixed_details'][str(combo)] += 1
disagreements.append({
'pdf': rep['pdf'],
'firm': firm,
'sig1': {'accountant': s1['accountant'], 'cos': s1['cos'],
'dhash': s1['dhash'], 'label': l1},
'sig2': {'accountant': s2['accountant'], 'cos': s2['cos'],
'dhash': s2['dhash'], 'label': l2},
'year_month': s1['year_month'],
})
# Print summary
print('\n--- Per-firm agreement ---')
for firm, d in sorted(results['by_firm'].items(), key=lambda x: -x[1]['total']):
agree = (d['both_non_hand_signed'] + d['both_hand_signed']
+ d['both_style_consistency'] + d['both_uncertain'])
rate = agree / d['total'] if d['total'] else 0
print(f' {firm}: total={d["total"]:,}, agree={agree} '
f'({rate*100:.2f}%), mixed={d["mixed"]}')
print(f' both_non_hand_signed={d["both_non_hand_signed"]}, '
f'both_uncertain={d["both_uncertain"]}, '
f'both_style_consistency={d["both_style_consistency"]}, '
f'both_hand_signed={d["both_hand_signed"]}')
# Write disagreements CSV (first 500)
csv_path = OUT / 'intra_report_disagreements.csv'
with open(csv_path, 'w', encoding='utf-8') as f:
f.write('pdf,firm,year_month,acc1,cos1,dhash1,label1,'
'acc2,cos2,dhash2,label2\n')
for d in disagreements[:500]:
f.write(f"{d['pdf']},{d['firm']},{d['year_month']},"
f"{d['sig1']['accountant']},{d['sig1']['cos']:.4f},"
f"{d['sig1']['dhash']},{d['sig1']['label']},"
f"{d['sig2']['accountant']},{d['sig2']['cos']:.4f},"
f"{d['sig2']['dhash']},{d['sig2']['label']}\n")
print(f'\nCSV: {csv_path} (first 500 of {len(disagreements)} disagreements)')
# Convert for JSON
summary = {
'generated_at': datetime.now().isoformat(),
'total_reports': len(reports),
'total_disagreements': len(disagreements),
'by_firm': {},
}
for firm, d in results['by_firm'].items():
agree = (d['both_non_hand_signed'] + d['both_hand_signed']
+ d['both_style_consistency'] + d['both_uncertain'])
summary['by_firm'][firm] = {
'total': d['total'],
'both_non_hand_signed': d['both_non_hand_signed'],
'both_hand_signed': d['both_hand_signed'],
'both_style_consistency': d['both_style_consistency'],
'both_uncertain': d['both_uncertain'],
'mixed': d['mixed'],
'agreement_rate': float(agree / d['total']) if d['total'] else 0,
'mixed_details': dict(d['mixed_details']),
}
with open(OUT / 'intra_report_results.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'JSON: {OUT / "intra_report_results.json"}')
# Markdown
md = [
'# Intra-Report Consistency Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'## Method',
'',
'* 2-signer reports (primary + secondary engagement partner).',
'* Each signature classified using the dual-descriptor rules of the',
' paper (cos > 0.95 AND dHash_indep ≤ 5 = high-confidence replication;',
' dHash 6-15 = moderate; > 15 = style consistency; cos ≤ 0.837 = likely',
' hand-signed; otherwise uncertain).',
'* For each report, both signature-level labels are compared.',
' A report is "in agreement" if both fall in the same coarse bucket',
' (non-hand-signed = high+moderate combined, style_consistency,',
' uncertain, or hand-signed); otherwise "mixed".',
'',
f'Total 2-signer reports analyzed: **{len(reports):,}**',
'',
'## Per-firm agreement',
'',
'| Firm | Total | Both non-hand-signed | Both style | Both uncertain | Both hand-signed | Mixed | Agreement rate |',
'|------|-------|----------------------|------------|----------------|------------------|-------|----------------|',
]
for firm, d in sorted(summary['by_firm'].items(),
key=lambda x: -x[1]['total']):
md.append(
f"| {firm} | {d['total']} | {d['both_non_hand_signed']} | "
f"{d['both_style_consistency']} | {d['both_uncertain']} | "
f"{d['both_hand_signed']} | {d['mixed']} | "
f"**{d['agreement_rate']*100:.2f}%** |"
)
md += [
'',
'## Interpretation',
'',
'Under firmwide stamping practice the two engagement partners on a',
'given report should both exhibit high-confidence non-hand-signed',
'classifications. High intra-report agreement at Firm A (Deloitte) is',
'consistent with uniform firm-level stamping; declining agreement at',
'the other Big-4 firms reflects the interview evidence that stamping',
'was applied only to a subset of partners.',
'',
'Mixed-classification reports (one signer non-hand-signed, the other',
'hand-signed or style-consistent) are flagged for sensitivity review.',
'Absent firmwide homogeneity, one would expect substantial mixed-rate',
'contamination even at Firm A; the observed Firm A mixed rate is a',
'direct empirical check on the identification assumption used in the',
'threshold calibration.',
]
(OUT / 'intra_report_report.md').write_text('\n'.join(md), encoding='utf-8')
print(f'Report: {OUT / "intra_report_report.md"}')
if __name__ == '__main__':
main()
signature_analysis/24_validation_recalibration.py
+416
View File
@@ -0,0 +1,416 @@
#!/usr/bin/env python3
"""
Script 24: Validation Recalibration (addresses codex v3.3 blockers)
====================================================================
Fixes three issues flagged by codex gpt-5.4 round-3 review of Paper A v3.3:
Blocker 2: held-out validation prose claims "held-out rates match
whole-sample within Wilson CI", which is numerically false
(e.g., whole 92.51% vs held-out CI [93.21%, 93.98%]).
The correct reference for generalization is the calibration
fold (70%), not the whole sample.
Blocker 1: the deployed per-signature classifier uses whole-sample
Firm A percentile heuristics (0.95, 0.837, dHash 5/15),
while the accountant-level three-method convergence sits at
cos ~0.973-0.979. This script adds a sensitivity check of
the classifier's five-way output under cos>0.945 and
cos>0.95 so the paper can report how the category
distribution shifts when the operational threshold is
replaced with the accountant-level 2D GMM marginal.
This script reads Script 21's output JSON for the 70/30 fold, recomputes
both calibration-fold and held-out-fold capture rates (with Wilson 95%
CIs), and runs a two-proportion z-test between calib and held-out for
each rule. It also computes the full-sample five-way classifier output
under cos>0.95 vs cos>0.945 for sensitivity.
Output:
reports/validation_recalibration/validation_recalibration.md
reports/validation_recalibration/validation_recalibration.json
"""
import json
import sqlite3
import numpy as np
from pathlib import Path
from datetime import datetime
from scipy.stats import norm
DB = '/Volumes/NV2/PDF-Processing/signature-analysis/signature_analysis.db'
OUT = Path('/Volumes/NV2/PDF-Processing/signature-analysis/reports/'
'validation_recalibration')
OUT.mkdir(parents=True, exist_ok=True)
FIRM_A = '勤業眾信聯合'
SEED = 42
# Rules of interest for held-out vs calib comparison.
COS_RULES = [0.837, 0.945, 0.95]
DH_RULES = [5, 8, 9, 15]
# Dual rule (the paper's classifier's operational dual).
DUAL_RULES = [(0.95, 8), (0.945, 8)]
def wilson_ci(k, n, alpha=0.05):
if n == 0:
return (0.0, 1.0)
z = norm.ppf(1 - alpha / 2)
phat = k / n
denom = 1 + z * z / n
center = (phat + z * z / (2 * n)) / denom
pm = z * np.sqrt(phat * (1 - phat) / n + z * z / (4 * n * n)) / denom
return (max(0.0, center - pm), min(1.0, center + pm))
def two_prop_z(k1, n1, k2, n2):
"""Two-proportion z-test (two-sided). Returns (z, p)."""
if n1 == 0 or n2 == 0:
return (float('nan'), float('nan'))
p1 = k1 / n1
p2 = k2 / n2
p_pool = (k1 + k2) / (n1 + n2)
if p_pool == 0 or p_pool == 1:
return (0.0, 1.0)
se = np.sqrt(p_pool * (1 - p_pool) * (1 / n1 + 1 / n2))
if se == 0:
return (0.0, 1.0)
z = (p1 - p2) / se
p = 2 * (1 - norm.cdf(abs(z)))
return (float(z), float(p))
def load_signatures():
conn = sqlite3.connect(DB)
cur = conn.cursor()
cur.execute('''
SELECT s.signature_id, s.assigned_accountant, a.firm,
s.max_similarity_to_same_accountant,
s.min_dhash_independent
FROM signatures s
LEFT JOIN accountants a ON s.assigned_accountant = a.name
WHERE s.max_similarity_to_same_accountant IS NOT NULL
''')
rows = cur.fetchall()
conn.close()
return rows
def fmt_pct(x):
return f'{x * 100:.2f}%'
def rate_with_ci(k, n):
lo, hi = wilson_ci(k, n)
return {
'rate': float(k / n) if n else 0.0,
'k': int(k),
'n': int(n),
'wilson95': [float(lo), float(hi)],
}
def main():
print('=' * 70)
print('Script 24: Validation Recalibration')
print('=' * 70)
rows = load_signatures()
accts = [r[1] for r in rows]
firms = [r[2] or '(unknown)' for r in rows]
cos = np.array([r[3] for r in rows], dtype=float)
dh = np.array([-1 if r[4] is None else r[4] for r in rows], dtype=float)
firm_a_mask = np.array([f == FIRM_A for f in firms])
print(f'\nLoaded {len(rows):,} signatures')
print(f'Firm A signatures: {int(firm_a_mask.sum()):,}')
# --- Reproduce Script 21's 70/30 split (same SEED=42) ---
rng = np.random.default_rng(SEED)
firm_a_accts = sorted(set(a for a, f in zip(accts, firms) if f == FIRM_A))
rng.shuffle(firm_a_accts)
n_calib = int(0.7 * len(firm_a_accts))
calib_accts = set(firm_a_accts[:n_calib])
heldout_accts = set(firm_a_accts[n_calib:])
print(f'\n70/30 split: calib CPAs={len(calib_accts)}, '
f'heldout CPAs={len(heldout_accts)}')
calib_mask = np.array([a in calib_accts for a in accts])
heldout_mask = np.array([a in heldout_accts for a in accts])
whole_mask = firm_a_mask
def summarize_fold(mask, label):
mcos = cos[mask]
mdh = dh[mask]
dh_valid = mdh >= 0
out = {
'fold': label,
'n_sigs': int(mask.sum()),
'n_dh_valid': int(dh_valid.sum()),
'cos_rules': {},
'dh_rules': {},
'dual_rules': {},
}
for t in COS_RULES:
k = int(np.sum(mcos > t))
n = int(len(mcos))
out['cos_rules'][f'cos>{t:.4f}'] = rate_with_ci(k, n)
for t in DH_RULES:
k = int(np.sum((mdh >= 0) & (mdh <= t)))
n = int(dh_valid.sum())
out['dh_rules'][f'dh_indep<={t}'] = rate_with_ci(k, n)
for ct, dt in DUAL_RULES:
k = int(np.sum((mcos > ct) & (mdh >= 0) & (mdh <= dt)))
n = int(len(mcos))
out['dual_rules'][f'cos>{ct:.3f}_AND_dh<={dt}'] = rate_with_ci(k, n)
return out
calib = summarize_fold(calib_mask, 'calibration_70pct')
held = summarize_fold(heldout_mask, 'heldout_30pct')
whole = summarize_fold(whole_mask, 'whole_firm_a')
print(f'\nCalib sigs: {calib["n_sigs"]:,} (dh valid: {calib["n_dh_valid"]:,})')
print(f'Held sigs: {held["n_sigs"]:,} (dh valid: {held["n_dh_valid"]:,})')
print(f'Whole sigs: {whole["n_sigs"]:,} (dh valid: {whole["n_dh_valid"]:,})')
# --- 2-proportion z-tests: calib vs held-out ---
print('\n=== Calib vs Held-out: 2-proportion z-test ===')
tests = {}
all_rules = (
[(f'cos>{t:.4f}', 'cos_rules') for t in COS_RULES] +
[(f'dh_indep<={t}', 'dh_rules') for t in DH_RULES] +
[(f'cos>{ct:.3f}_AND_dh<={dt}', 'dual_rules') for ct, dt in DUAL_RULES]
)
for rule, group in all_rules:
c = calib[group][rule]
h = held[group][rule]
z, p = two_prop_z(c['k'], c['n'], h['k'], h['n'])
in_calib_ci = c['wilson95'][0] <= h['rate'] <= c['wilson95'][1]
in_held_ci = h['wilson95'][0] <= c['rate'] <= h['wilson95'][1]
tests[rule] = {
'calib_rate': c['rate'],
'calib_ci': c['wilson95'],
'held_rate': h['rate'],
'held_ci': h['wilson95'],
'z': z,
'p': p,
'held_within_calib_ci': bool(in_calib_ci),
'calib_within_held_ci': bool(in_held_ci),
}
sig = '***' if p < 0.001 else '**' if p < 0.01 else \
'*' if p < 0.05 else 'n.s.'
print(f' {rule:40s} calib={fmt_pct(c["rate"])} '
f'held={fmt_pct(h["rate"])} z={z:+.3f} p={p:.4f} {sig}')
# --- Classifier sensitivity: cos>0.95 vs cos>0.945 ---
print('\n=== Classifier sensitivity: 0.95 vs 0.945 ===')
# All whole-sample signatures (not just Firm A) for the classifier.
# Reproduces the Section III-L five-way classifier categorization.
dh_all_valid = dh >= 0
all_cos = cos
all_dh = dh
def classify(cos_arr, dh_arr, dh_valid, cos_hi, dh_hi_high=5,
dh_hi_mod=15, cos_lo=0.837):
"""Replicate Section III-L five-way classifier.
Categories (signature-level):
1 high-confidence non-hand-signed: cos>cos_hi AND dh<=dh_hi_high
2 moderate-confidence: cos>cos_hi AND dh_hi_high<dh<=dh_hi_mod
3 style-only: cos>cos_hi AND dh>dh_hi_mod
4 uncertain: cos_lo<cos<=cos_hi
5 likely hand-signed: cos<=cos_lo
Signatures with missing dHash fall into a sixth bucket (dh-missing).
"""
cats = np.full(len(cos_arr), 6, dtype=int) # 6 = dh-missing default
above_hi = cos_arr > cos_hi
above_lo_only = (cos_arr > cos_lo) & (~above_hi)
below_lo = cos_arr <= cos_lo
cats[above_lo_only] = 4
cats[below_lo] = 5
# For dh-valid subset that exceeds cos_hi, subdivide.
has_dh = dh_valid & above_hi
cats[has_dh & (dh_arr <= dh_hi_high)] = 1
cats[has_dh & (dh_arr > dh_hi_high) & (dh_arr <= dh_hi_mod)] = 2
cats[has_dh & (dh_arr > dh_hi_mod)] = 3
# Signatures with above_hi but dh missing -> default cat 2 (moderate)
# for continuity with the classifier's whole-sample behavior.
cats[above_hi & ~dh_valid] = 2
return cats
cats_95 = classify(all_cos, all_dh, dh_all_valid, cos_hi=0.95)
cats_945 = classify(all_cos, all_dh, dh_all_valid, cos_hi=0.945)
# 5 + dh-missing bucket
labels = {
1: 'high_confidence_non_hand_signed',
2: 'moderate_confidence_non_hand_signed',
3: 'high_style_consistency',
4: 'uncertain',
5: 'likely_hand_signed',
6: 'dh_missing',
}
sens = {'0.95': {}, '0.945': {}, 'diff': {}}
total = len(cats_95)
for c, name in labels.items():
n95 = int((cats_95 == c).sum())
n945 = int((cats_945 == c).sum())
sens['0.95'][name] = {'n': n95, 'pct': n95 / total * 100}
sens['0.945'][name] = {'n': n945, 'pct': n945 / total * 100}
sens['diff'][name] = n945 - n95
print(f' {name:40s} 0.95: {n95:>7,} ({n95/total*100:5.2f}%) '
f'0.945: {n945:>7,} ({n945/total*100:5.2f}%) '
f'diff: {n945 - n95:+,}')
# Transition matrix (how many signatures change category)
transitions = {}
for from_c in range(1, 7):
for to_c in range(1, 7):
if from_c == to_c:
continue
n = int(((cats_95 == from_c) & (cats_945 == to_c)).sum())
if n > 0:
key = f'{labels[from_c]}->{labels[to_c]}'
transitions[key] = n
# Dual rule capture on whole Firm A (not just heldout)
# under 0.95 AND dh<=8 vs 0.945 AND dh<=8
fa_cos = cos[firm_a_mask]
fa_dh = dh[firm_a_mask]
dual_95_8 = int(((fa_cos > 0.95) & (fa_dh >= 0) & (fa_dh <= 8)).sum())
dual_945_8 = int(((fa_cos > 0.945) & (fa_dh >= 0) & (fa_dh <= 8)).sum())
n_fa = int(firm_a_mask.sum())
print(f'\nDual rule on whole Firm A (n={n_fa:,}):')
print(f' cos>0.950 AND dh<=8: {dual_95_8:,} ({dual_95_8/n_fa*100:.2f}%)')
print(f' cos>0.945 AND dh<=8: {dual_945_8:,} ({dual_945_8/n_fa*100:.2f}%)')
# --- Save ---
summary = {
'generated_at': datetime.now().isoformat(),
'firm_a_name_redacted': 'Firm A (real name redacted)',
'seed': SEED,
'n_signatures': len(rows),
'n_firm_a': int(firm_a_mask.sum()),
'split': {
'calib_cpas': len(calib_accts),
'heldout_cpas': len(heldout_accts),
'calib_sigs': int(calib_mask.sum()),
'heldout_sigs': int(heldout_mask.sum()),
},
'calibration_fold': calib,
'heldout_fold': held,
'whole_firm_a': whole,
'generalization_tests': tests,
'classifier_sensitivity': sens,
'classifier_transitions_95_to_945': transitions,
'dual_rule_whole_firm_a': {
'cos_gt_0.95_AND_dh_le_8': {
'k': dual_95_8, 'n': n_fa,
'rate': dual_95_8 / n_fa,
'wilson95': list(wilson_ci(dual_95_8, n_fa)),
},
'cos_gt_0.945_AND_dh_le_8': {
'k': dual_945_8, 'n': n_fa,
'rate': dual_945_8 / n_fa,
'wilson95': list(wilson_ci(dual_945_8, n_fa)),
},
},
}
with open(OUT / 'validation_recalibration.json', 'w') as f:
json.dump(summary, f, indent=2, ensure_ascii=False)
print(f'\nJSON: {OUT / "validation_recalibration.json"}')
# --- Markdown ---
md = [
'# Validation Recalibration Report',
f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
'',
'Addresses codex gpt-5.4 v3.3 round-3 review Blockers 1 and 2.',
'',
'## 1. Calibration vs Held-out Firm A Generalization Test',
'',
f'* Seed {SEED}; 70/30 CPA-level split.',
f'* Calibration fold: {calib["n_sigs"]:,} signatures '
f'({len(calib_accts)} CPAs).',
f'* Held-out fold: {held["n_sigs"]:,} signatures '
f'({len(heldout_accts)} CPAs).',
'',
'**Reference comparison.** The correct generalization test compares '
'calib-fold vs held-out-fold rates, not whole-sample vs held-out-fold. '
'The whole-sample rate is a weighted average of the two folds and '
'therefore cannot lie inside the held-out CI when the folds differ in '
'rate.',
'',
'| Rule | Calib rate (CI) | Held-out rate (CI) | z | p | Held within calib CI? |',
'|------|-----------------|---------------------|---|---|------------------------|',
]
for rule, group in all_rules:
c = calib[group][rule]
h = held[group][rule]
t = tests[rule]
md.append(
f'| `{rule}` | {fmt_pct(c["rate"])} '
f'[{fmt_pct(c["wilson95"][0])}, {fmt_pct(c["wilson95"][1])}] '
f'| {fmt_pct(h["rate"])} '
f'[{fmt_pct(h["wilson95"][0])}, {fmt_pct(h["wilson95"][1])}] '
f'| {t["z"]:+.3f} | {t["p"]:.4f} | '
f'{"yes" if t["held_within_calib_ci"] else "no"} |'
)
md += [
'',
'## 2. Classifier Sensitivity: cos > 0.95 vs cos > 0.945',
'',
f'All-sample five-way classifier output (N = {total:,} signatures).',
'The 0.945 cutoff is the accountant-level 2D GMM marginal crossing; ',
'the 0.95 cutoff is the whole-sample Firm A P95 heuristic.',
'',
'| Category | cos>0.95 count (%) | cos>0.945 count (%) | Δ |',
'|----------|---------------------|-----------------------|---|',
]
for c, name in labels.items():
a = sens['0.95'][name]
b = sens['0.945'][name]
md.append(
f'| {name} | {a["n"]:,} ({a["pct"]:.2f}%) '
f'| {b["n"]:,} ({b["pct"]:.2f}%) '
f'| {sens["diff"][name]:+,} |'
)
md += [
'',
'### Category transitions (0.95 -> 0.945)',
'',
]
for k, v in sorted(transitions.items(), key=lambda x: -x[1]):
md.append(f'* `{k}`: {v:,}')
md += [
'',
'## 3. Dual-Rule Capture on Whole Firm A',
'',
f'* cos > 0.950 AND dh_indep <= 8: {dual_95_8:,}/{n_fa:,} '
f'({dual_95_8/n_fa*100:.2f}%)',
f'* cos > 0.945 AND dh_indep <= 8: {dual_945_8:,}/{n_fa:,} '
f'({dual_945_8/n_fa*100:.2f}%)',
'',
'## 4. Interpretation',
'',
'* The calib-vs-held-out 2-proportion z-test is the correct '
'generalization check. If `p >= 0.05` the two folds are not '
'statistically distinguishable at 5% level.',
'* Where the two folds differ significantly, the paper should say the '
'held-out fold happens to be slightly more replication-dominated than '
'the calibration fold (i.e., a sampling-variance effect, not a '
'generalization failure), and still discloses the rates for both '
'folds.',
'* The sensitivity analysis shows how many signatures flip categories '
'under the accountant-level convergence threshold (0.945) versus the '
'whole-sample heuristic (0.95). Small shifts support the paper\'s '
'claim that the operational classifier is robust to the threshold '
'choice; larger shifts would require either changing the classifier '
'or reporting results under both cuts.',
]
(OUT / 'validation_recalibration.md').write_text('\n'.join(md),
encoding='utf-8')
print(f'Report: {OUT / "validation_recalibration.md"}')
if __name__ == '__main__':
main()
signature_analysis/THRESHOLD_VALIDATION_OPTIONS.md
+534
View File
@@ -0,0 +1,534 @@
# Signature Verification Threshold Validation Options
**Report Date:** 2026-01-14
**Purpose:** Discussion document for research partners on threshold selection methodology
**Context:** Validating copy-paste detection thresholds for accountant signature analysis
---
## Table of Contents
1. [Current Findings Summary](#1-current-findings-summary)
2. [The Core Problem](#2-the-core-problem)
3. [Key Metrics Explained](#3-key-metrics-explained)
4. [Validation Options](#4-validation-options)
5. [Academic References](#5-academic-references)
6. [Recommendations](#6-recommendations)
7. [Next Steps for Discussion](#7-next-steps-for-discussion)
---
## 1. Current Findings Summary
Our YOLO-based signature extraction and similarity analysis produced the following results:
| Metric | Value |
|--------|-------|
| Total PDFs analyzed | 84,386 |
| Total signatures extracted | 168,755 |
| High similarity pairs (>0.95) | 659,111 |
| Classified as "copy-paste" | 71,656 PDFs (84.9%) |
| Classified as "authentic" | 76 PDFs (0.1%) |
| Uncertain | 12,651 PDFs (15.0%) |
**Current threshold used:**
- Copy-paste: similarity ≥ 0.95
- Authentic: similarity ≤ 0.85
- Uncertain: 0.85 < similarity < 0.95
---
## 2. The Core Problem
### 2.1 What is Ground Truth?
**Ground truth labels** are pre-verified classifications that serve as the "correct answer" for machine learning evaluation. For signature verification:
| Label | Meaning | How to Obtain |
|-------|---------|---------------|
| **Genuine** | Physically hand-signed by the accountant | Expert forensic examination |
| **Copy-paste/Forged** | Digitally copied from another document | Pixel-level analysis or expert verification |
### 2.2 Why We Need Ground Truth
To calculate rigorous metrics like EER (Equal Error Rate), we need labeled data:
```
EER Calculation requires:
├── Known genuine signatures → Calculate FRR at each threshold
├── Known forged signatures → Calculate FAR at each threshold
└── Find threshold where FAR = FRR → This is EER
```
### 2.3 Our Current Limitation
We do not have pre-labeled ground truth data. Our current classification is based on:
- **Domain assumption**: Identical handwritten signatures are physically impossible
- **Similarity threshold**: Arbitrarily selected at 0.95
This approach is reasonable but may be challenged in academic peer review without additional validation.
---
## 3. Key Metrics Explained
### 3.1 Error Rate Metrics
| Metric | Full Name | Formula | Interpretation |
|--------|-----------|---------|----------------|
| **FAR** | False Acceptance Rate | Forgeries Accepted / Total Forgeries | Security risk |
| **FRR** | False Rejection Rate | Genuine Rejected / Total Genuine | Usability risk |
| **EER** | Equal Error Rate | Point where FAR = FRR | Overall performance |
| **AER** | Average Error Rate | (FAR + FRR) / 2 | Combined error |
### 3.2 Visual Representation of EER
```
100% ┌─────────────────────────────────────┐
│ FRR │
│ \ │
│ \ │
Rate │ \ ╳ ← EER point │
│ \ / │
│ \ / │
│ \ / FAR │
0% │────────\/──────────────────────────│
└─────────────────────────────────────┘
Low ←──── Threshold ────→ High
```
### 3.3 Benchmark Performance (from Literature)
| System | Dataset | EER | Reference |
|--------|---------|-----|-----------|
| SigNet (Siamese CNN) | GPDS-300 | 3.92% | Dey et al., 2017 |
| Consensus-Threshold | GPDS-300 | 1.27% FAR | arXiv:2401.03085 |
| Type-2 Neutrosophic | Custom | 98% accuracy | IASC 2024 |
| InceptionV3 Transfer | CEDAR | 99.10% accuracy | Springer 2024 |
---
## 4. Validation Options
### Option 1: Manual Ground Truth Creation (Most Rigorous)
**Description:**
Manually verify a subset of signatures with human expert examination.
**Methodology:**
1. Randomly sample ~100-200 signature pairs from different similarity ranges
2. Expert examines original PDF documents for:
- Scan artifact variations (genuine scans have unique noise)
- Pixel-perfect alignment (copy-paste is exact)
- Ink pressure and stroke variations
- Document metadata (creation dates, software used)
3. Label each pair as "genuine" or "copy-paste"
4. Calculate EER, FAR, FRR at various thresholds
5. Select optimal threshold based on EER
**Pros:**
- Academically rigorous
- Enables standard metric calculation (EER, FAR, FRR)
- Defensible in peer review
**Cons:**
- Time-consuming (estimated 20-40 hours for 200 samples)
- Requires forensic document expertise
- Subjective in edge cases
**Academic Support:**
> "The final verification results can be obtained by the voting method with different thresholds and can be adjusted according to different types of application requirements."
> — Hadjadj et al., Applied Sciences, 2020 [[1]](#ref1)
---
### Option 2: Statistical Distribution-Based Threshold (No Labels Needed)
**Description:**
Use the statistical distribution of similarity scores to define outliers.
**Methodology:**
1. Calculate mean (μ) and standard deviation (σ) of all similarity scores
2. Define thresholds based on standard deviations:
| Threshold | Formula | Percentile | Classification |
|-----------|---------|------------|----------------|
| Very High | > μ + 3σ | 99.7% | Definite copy-paste |
| High | > μ + 2σ | 95% | Likely copy-paste |
| Normal | μ ± 2σ | 5-95% | Uncertain |
| Low | < μ - 2σ | <5% | Likely genuine |
**Your Data:**
```
Mean similarity (μ) = 0.7608
Std deviation (σ) = 0.0916
Thresholds:
- μ + 2σ = 0.944 (95th percentile)
- μ + 3σ = 1.035 (99.7th percentile, capped at 1.0)
Your current 0.95 threshold ≈ μ + 2.07σ (96th percentile)
```
**Pros:**
- No manual labeling required
- Statistically defensible
- Based on actual data distribution
**Cons:**
- Assumes normal distribution (may not hold)
- Does not provide FAR/FRR metrics
- Less intuitive for non-statistical audiences
**Academic Support:**
> "Keypoint-based detection methods employ statistical thresholds derived from feature distributions to identify anomalous similarity patterns."
> — Copy-Move Forgery Detection Survey, Multimedia Tools & Applications, 2024 [[2]](#ref2)
---
### Option 3: Physical Impossibility Argument (Domain Knowledge)
**Description:**
Use the physical impossibility of identical handwritten signatures as justification.
**Methodology:**
1. Define threshold based on handwriting science:
| Similarity | Physical Interpretation | Classification |
|------------|------------------------|----------------|
| = 1.0 | Pixel-identical; physically impossible for handwriting | **Definite copy** |
| > 0.98 | Near-identical; extremely improbable naturally | **Very likely copy** |
| 0.90 - 0.98 | Highly similar; unusual but possible | **Suspicious** |
| 0.80 - 0.90 | Similar; consistent with same signer | **Uncertain** |
| < 0.80 | Different; normal variation | **Likely genuine** |
2. Cite forensic document examination literature on signature variability
**Pros:**
- Intuitive and explainable
- Based on established forensic principles
- Does not require labeled data
**Cons:**
- Thresholds are somewhat arbitrary
- May not account for digital signature pads (lower variation)
- Requires supporting citations
**Academic Support:**
> "Signature verification presents several unique difficulties: high intra-class variability (an individual's signature may vary greatly day-to-day), large temporal variation (signature may change completely over time), and high inter-class similarity (forgeries attempt to be indistinguishable)."
> — Stanford CS231n Report, 2016 [[3]](#ref3)
> "A genuine signer's signature is naturally unstable even at short time-intervals, presenting inherent variation that digital copies lack."
> — Consensus-Threshold Criterion, arXiv:2401.03085, 2024 [[4]](#ref4)
---
### Option 4: Pixel-Level Copy Detection (Technical Verification)
**Description:**
Detect exact copies through pixel-level analysis, independent of feature similarity.
**Methodology:**
1. For high-similarity pairs (>0.95), perform additional checks:
```python
# Check 1: Exact pixel match
if np.array_equal(image1, image2):
return "DEFINITE_COPY"
# Check 2: Structural Similarity Index (SSIM)
ssim_score = structural_similarity(image1, image2)
if ssim_score > 0.999:
return "DEFINITE_COPY"
# Check 3: Histogram correlation
hist_corr = cv2.compareHist(hist1, hist2, cv2.HISTCMP_CORREL)
if hist_corr > 0.999:
return "LIKELY_COPY"
```
2. Use copy-move forgery detection (CMFD) techniques from image forensics
**Pros:**
- Technical proof of copying
- Not dependent on threshold selection
- Provides definitive evidence for exact copies
**Cons:**
- Only detects exact copies (not scaled/rotated)
- Requires additional processing
- May miss high-quality forgeries
**Academic Support:**
> "Block-based methods segment an image into overlapping blocks and extract features. The forgery regions are determined by computing the similarity between block features using DCT (Discrete Cosine Transform) or SIFT (Scale-Invariant Feature Transform)."
> — Copy-Move Forgery Detection Survey, 2024 [[2]](#ref2)
---
### Option 5: Siamese Network with Learned Threshold (Advanced)
**Description:**
Train a Siamese neural network on signature pairs to learn optimal decision boundaries.
**Methodology:**
1. Collect training data:
- Positive pairs: Same accountant, different documents
- Negative pairs: Different accountants
2. Train Siamese network with contrastive or triplet loss
3. Network learns embedding space where:
- Same-person signatures cluster together
- Different-person signatures separate
4. Threshold is learned during training, not manually set
**Architecture:**
```
┌──────────────┐ ┌──────────────┐
│ Signature 1 │ │ Signature 2 │
└──────┬───────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ CNN │ │ CNN │ (Shared weights)
│ Encoder │ │ Encoder │
└──────┬───────┘ └──────┬───────┘
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Embedding │ │ Embedding │
│ Vector │ │ Vector │
└──────┬───────┘ └──────┬───────┘
│ │
└────────┬───────────┘
│
▼
┌───────────────┐
│ Distance │
│ Metric │
└───────┬───────┘
│
▼
┌───────────────┐
│ Same/Different│
└───────────────┘
```
**Pros:**
- Learns optimal threshold from data
- State-of-the-art performance
- Handles complex variations
**Cons:**
- Requires substantial training data
- Computationally expensive
- May overfit to specific accountant styles
**Academic Support:**
> "SigNet provided better results than the state-of-the-art results on most of the benchmark signature datasets by learning a feature space where similar observations are placed in proximity."
> — SigNet, arXiv:1707.02131, 2017 [[5]](#ref5)
> "Among various distance measures employed in the t-Siamese similarity network, the Manhattan distance technique emerged as the most effective."
> — Triplet Siamese Similarity Networks, Mathematics, 2024 [[6]](#ref6)
---
## 5. Academic References
<a name="ref1"></a>
### [1] Single Known Sample Verification (MDPI 2020)
**Title:** An Offline Signature Verification and Forgery Detection Method Based on a Single Known Sample and an Explainable Deep Learning Approach
**Authors:** Hadjadj, I. et al.
**Journal:** Applied Sciences, 10(11), 3716
**Year:** 2020
**URL:** https://www.mdpi.com/2076-3417/10/11/3716
**Key Findings:**
- Accuracy: 94.37% - 99.96%
- FRR: 0% - 5.88%
- FAR: 0.22% - 5.34%
- Voting method with adjustable thresholds
<a name="ref2"></a>
### [2] Copy-Move Forgery Detection Survey (Springer 2024)
**Title:** Copy-move forgery detection in digital image forensics: A survey
**Journal:** Multimedia Tools and Applications
**Year:** 2024
**URL:** https://link.springer.com/article/10.1007/s11042-024-18399-2
**Key Findings:**
- Block-based, keypoint-based, and deep learning methods reviewed
- DCT and SIFT for feature extraction
- Statistical thresholds for anomaly detection
<a name="ref3"></a>
### [3] Stanford CS231n Signature Verification Report
**Title:** Offline Signature Verification with Convolutional Neural Networks
**Institution:** Stanford University
**Year:** 2016
**URL:** https://cs231n.stanford.edu/reports/2016/pdfs/276_Report.pdf
**Key Findings:**
- High intra-class variability challenge
- Low inter-class similarity for skilled forgeries
- CNN-based feature extraction
<a name="ref4"></a>
### [4] Consensus-Threshold Criterion (arXiv 2024)
**Title:** Consensus-Threshold Criterion for Offline Signature Verification using Convolutional Neural Network Learned Representations
**Year:** 2024
**URL:** https://arxiv.org/abs/2401.03085
**Key Findings:**
- Achieved 1.27% FAR (vs 8.73% and 17.31% in prior work)
- Consensus-threshold distance-based classifier
- Uses SigNet and SigNet-F features
<a name="ref5"></a>
### [5] SigNet: Siamese Network for Signature Verification (arXiv 2017)
**Title:** SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification
**Authors:** Dey, S. et al.
**Year:** 2017
**URL:** https://arxiv.org/abs/1707.02131
**Key Findings:**
- Siamese architecture with shared weights
- Euclidean distance minimization for genuine pairs
- State-of-the-art on GPDS, CEDAR, MCYT datasets
<a name="ref6"></a>
### [6] Triplet Siamese Similarity Networks (MDPI 2024)
**Title:** Enhancing Signature Verification Using Triplet Siamese Similarity Networks in Digital Documents
**Journal:** Mathematics, 12(17), 2757
**Year:** 2024
**URL:** https://www.mdpi.com/2227-7390/12/17/2757
**Key Findings:**
- Manhattan distance outperforms Euclidean and Minkowski
- Triplet loss for inter-class/intra-class optimization
- Tested on 4NSigComp2012, SigComp2011, BHSig260
<a name="ref7"></a>
### [7] Original Siamese Network Paper (NeurIPS 1993)
**Title:** Signature Verification using a "Siamese" Time Delay Neural Network
**Authors:** Bromley, J. et al.
**Conference:** NeurIPS 1993
**URL:** https://papers.neurips.cc/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf
**Key Findings:**
- Introduced Siamese architecture for signature verification
- Cosine similarity = 1.0 for genuine pairs
- Foundational work for modern approaches
<a name="ref8"></a>
### [8] Australian Journal of Forensic Sciences (2024)
**Title:** Handling high level of uncertainty in forensic signature examination
**Journal:** Australian Journal of Forensic Sciences, 57(5)
**Year:** 2024
**URL:** https://www.tandfonline.com/doi/full/10.1080/00450618.2024.2410044
**Key Findings:**
- Type-2 Neutrosophic similarity measure
- 98% accuracy (vs 95% for Type-1)
- Addresses ambiguity in forensic analysis
<a name="ref9"></a>
### [9] Benchmark Datasets
**CEDAR Dataset:**
- 55 signers × 24 genuine + 24 forged signatures
- URL: https://paperswithcode.com/dataset/cedar-signature
**GPDS-960 Corpus:**
- 960 writers × 24 genuine + 30 forgeries
- 600 dpi grayscale scans
- URL: https://www.researchgate.net/publication/220860371
---
## 6. Recommendations
### For Academic Publication
| Priority | Option | Effort | Rigor | Recommendation |
|----------|--------|--------|-------|----------------|
| 1 | **Option 1 + Option 2** | High | Very High | Create small labeled dataset + validate statistical threshold |
| 2 | **Option 2 + Option 3** | Low | Medium | Statistical threshold + physical impossibility argument |
| 3 | **Option 4** | Medium | High | Add pixel-level verification for definitive cases |
### Suggested Approach
1. **Primary method:** Use statistical threshold (Option 2)
- Report threshold as μ + 2σ ≈ 0.944 (close to your current 0.95)
- Statistically defensible without ground truth
2. **Supporting evidence:** Physical impossibility argument (Option 3)
- Cite forensic literature on signature variability
- Emphasize that identical signatures are physically impossible
3. **Validation (if time permits):** Small labeled subset (Option 1)
- Manually verify 100-200 samples
- Calculate EER to validate threshold choice
4. **Technical proof:** Pixel-level analysis (Option 4)
- Add SSIM analysis for high-similarity pairs
- Report exact copy counts separately
### Suggested Report Language
> "We adopt a similarity threshold of 0.95 (approximately μ + 2σ, representing the 96th percentile of our similarity distribution) to classify signatures as potential copy-paste instances. This threshold is supported by: (1) statistical outlier detection principles, (2) the physical impossibility of pixel-identical handwritten signatures, and (3) alignment with forensic document examination literature [cite: Hadjadj 2020, arXiv:2401.03085]."
---
## 7. Next Steps for Discussion
### Questions for Research Partners
1. **Data availability:** Do we have access to any documents with known authentic signatures for validation?
2. **Expert resources:** Can we involve a forensic document examiner for ground truth labeling?
3. **Scope decision:** Should we focus on statistical validation (faster) or pursue full EER analysis (more rigorous)?
4. **Publication target:** What level of rigor does the target journal require?
5. **Time constraints:** How much time can we allocate to validation before submission?
### Proposed Action Items
| Task | Owner | Deadline | Notes |
|------|-------|----------|-------|
| Review this document | All partners | TBD | Discuss options |
| Select validation approach | Team decision | TBD | Based on resources |
| Implement selected approach | TBD | TBD | After decision |
| Update threshold if needed | TBD | TBD | Based on validation |
| Draft methodology section | TBD | TBD | For paper |
---
## Appendix: Code for Statistical Threshold Calculation
```python
import numpy as np
from scipy import stats
# Your similarity data
similarities = [...] # Load from your analysis
# Calculate statistics
mean_sim = np.mean(similarities)
std_sim = np.std(similarities)
percentiles = np.percentile(similarities, [90, 95, 99, 99.7])
print(f"Mean (μ): {mean_sim:.4f}")
print(f"Std (σ): {std_sim:.4f}")
print(f"μ + 2σ: {mean_sim + 2*std_sim:.4f}")
print(f"μ + 3σ: {mean_sim + 3*std_sim:.4f}")
print(f"Percentiles: 90%={percentiles[0]:.4f}, 95%={percentiles[1]:.4f}, "
f"99%={percentiles[2]:.4f}, 99.7%={percentiles[3]:.4f}")
# Threshold recommendations
thresholds = {
"Conservative (μ+3σ)": min(1.0, mean_sim + 3*std_sim),
"Standard (μ+2σ)": mean_sim + 2*std_sim,
"Liberal (95th percentile)": percentiles[1],
}
for name, thresh in thresholds.items():
count_above = np.sum(similarities > thresh)
pct_above = 100 * count_above / len(similarities)
print(f"{name}: {thresh:.4f} → {count_above} pairs ({pct_above:.2f}%)")
```
---
*Document prepared for research discussion. Please share feedback and questions with the team.*