gbanyan
ac3372d2d2
Paper A v4.3: restructure §III baseline-first + deep trim (audit #3/#4)
...
#3 Reorder §III around "establish normative baseline → show who deviates":
new order A–O with I=Normative Baseline + inter-CPA coincidence floor
(old L.0–L.3), J=Firm-Level Deviation (old L.4+L.6), K=Why the
distribution gives no threshold (old §I distributional + L.5), L=K=3
partition (old §J), M=Convergent checks (old §K), N=limits (old §M),
O=data source (old §N). ~170 cross-refs remapped (two-pass tokenized),
incl. 5 spelled-out "Section III-X" refs.
#4 Deep trim: §III 10,960→8,461w (−23%). Removed §III↔§IV-M and
§III↔§IV-F table duplication (Results keeps canonical tables; §III keeps
method+headline+pointer); condensed distributional diagnostics;
consolidated repeated caveat. No locked number changed.
Also: §V-C header "Case Study"→"Out-of-Sample Target"; abstract 251→250w;
housekeeping (rm superseded draft_section_L_bcd.md + v4.0 pandoc docx,
remove stale OCR/handoff docs, gitignore .serena/).
codex gpt-5.5 review: 0 BLOCKER / 3 MAJOR / 3 MINOR; 3 MAJOR fixed
(§III-J.2 observed-vs-counterfactual transition, §III-M table pointers
κ→XI/pixel→XIV, §III-N stale tightening figures); 2nd pass clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-05 01:49:09 +08:00
gbanyan
a261a22bd2
Add Deloitte distribution & independent dHash analysis scripts
...
- Script 13: Firm A normality/multimodality analysis (Shapiro-Wilk, Anderson-Darling, KDE, per-accountant ANOVA, Beta/Gamma fitting)
- Script 14: Independent min-dHash computation across all pairs per accountant (not just cosine-nearest pair)
- THRESHOLD_VALIDATION_OPTIONS: 2026-01 discussion doc on threshold validation approaches
- .gitignore: exclude model weights, node artifacts, and xlsx data
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-20 21:34:24 +08:00
gbanyan
52612e14ba
Add hybrid signature extraction with name-based verification
...
Implement VLM name extraction + CV detection hybrid approach to
replace unreliable VLM coordinate system with name-based verification.
Key Features:
- VLM extracts signature names (周寶蓮, 魏興海, etc.)
- CV or PDF text layer detects regions
- VLM verifies each region against expected names
- Signatures saved with person names: signature_周寶蓮.png
- Duplicate prevention and rejection handling
Test Results:
- 5 PDF pages tested
- 7/10 signatures extracted (70% recall)
- 100% precision (no false positives)
- No blank regions extracted (previous issue resolved)
Files:
- extract_pages_from_csv.py: Extract pages from CSV (tested: 100 files)
- extract_signatures_hybrid.py: Hybrid extraction (current working solution)
- extract_handwriting.py: CV-only approach (component)
- extract_signatures_vlm.py: Deprecated VLM coordinate approach
- PROJECT_DOCUMENTATION.md: Complete project history and results
- SESSION_INIT.md: Session handoff documentation
- SESSION_CHECKLIST.md: Status checklist
- NEW_SESSION_PROMPT.txt: Template for next session
- HOW_TO_CONTINUE.txt: Visual handoff guide
- COMMIT_SUMMARY.md: Commit preparation guide
- README.md: Quick start guide
- README_page_extraction.md: Page extraction docs
- README_hybrid_extraction.md: Hybrid approach docs
- .gitignore: Exclude diagnostic scripts and outputs
Known Limitations:
- 30% of signatures missed due to conservative CV parameters
- Text layer method untested (all test PDFs are scanned images)
- Performance: ~24 seconds per PDF
Next Steps:
- Tune CV parameters for higher recall
- Test with larger dataset (100+ files)
- Process full dataset (86,073 files)
🤖 Generated with Claude Code
2025-10-26 23:39:52 +08:00
gbanyan