gbanyan/pdf_signature_extraction
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
6 Commits 3 Branches 1 Tag
a261a22bd224c70fd75897ff7d7ef6678ed3ca7a
Commit Graph

6 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
gbanyan a261a22bd2 Add Deloitte distribution & independent dHash analysis scripts 
- Script 13: Firm A normality/multimodality analysis (Shapiro-Wilk, Anderson-Darling, KDE, per-accountant ANOVA, Beta/Gamma fitting)
- Script 14: Independent min-dHash computation across all pairs per accountant (not just cosine-nearest pair)
- THRESHOLD_VALIDATION_OPTIONS: 2026-01 discussion doc on threshold validation approaches
- .gitignore: exclude model weights, node artifacts, and xlsx data

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-20 21:34:24 +08:00
gbanyan 939a348da4 Add Paper A (IEEE TAI) complete draft with Firm A-calibrated dual-method classification 
Paper draft includes all sections (Abstract through Conclusion), 36 references,
and supporting scripts. Key methodology: Cosine similarity + dHash dual-method
verification with thresholds calibrated against known-replication firm (Firm A).

Includes:
- 8 section markdown files (paper_a_*.md)
- Ablation study script (ResNet-50 vs VGG-16 vs EfficientNet-B0)
- Recalibrated classification script (84,386 PDFs, 5-tier system)
- Figure generation and Word export scripts
- Citation renumbering script ([1]-[36])
- Signature analysis pipeline (12 steps)
- YOLO extraction scripts

Three rounds of AI review completed (GPT-5.4, Claude Opus 4.6, Gemini 3 Pro).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-06 23:05:33 +08:00
gbanyan 21df0ff387 Complete PP-OCRv5 research and v4 vs v5 comparison 
## 研究成果

### PP-OCRv5 API 測試
- 成功升級到 PaddleOCR 3.3.2 (PP-OCRv5)
- 理解新 API 結構和調用方式
- 驗證基礎檢測功能

### 關鍵發現
❌ PP-OCRv5 **沒有內建手寫分類功能**
- text_type 字段是語言類型，不是手寫/印刷分類
- 仍需要 OpenCV Method 3 來分離手寫和印刷文字

### 完整 Pipeline 對比測試
- v4 (2.7.3): 檢測 14 個文字 → 4 個候選區域
- v5 (3.3.2): 檢測 50 個文字 → 7 個候選區域
- 主簽名區域：兩個版本幾乎相同 (1150x511 vs 1144x511)

### 性能分析
優點：
- v5 手寫識別準確率 +13.7% (文檔承諾)
- 可能減少漏檢

缺點：
- 過度檢測（印章小字等）
- API 完全重寫，不兼容
- 仍無法替代 OpenCV Method 3

### 文件
- PP_OCRV5_RESEARCH_FINDINGS.md: 完整研究報告
- signature-comparison/: v4 vs v5 對比結果
- test_results/: v5 測試輸出
- test_*_pipeline.py: 完整測試腳本

### 建議
當前方案（v2.7.3 + OpenCV Method 3）已足夠穩定，
除非遇到大量漏檢，否則暫不升級到 v5。

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-27 11:21:55 +08:00
gbanyan 8f231da3bc Complete OpenCV Method 3 implementation with 86.5% handwriting retention 
- Implemented comprehensive feature analysis based on size, stroke length, and regularity
- Size-based scoring: height >50px indicates handwriting
- Stroke length ratio: >0.4 indicates handwriting
- Irregularity metrics: low compactness/solidity indicates handwriting
- Successfully tested on sample PDF with 2 signatures (楊智惠, 張志銘)
- Created detailed documentation: CURRENT_STATUS.md and NEW_SESSION_HANDOFF.md
- Stable PaddleOCR 2.7.3 configuration documented (numpy 1.26.4, opencv 4.6.0.66)
- Prepared research plan for PP-OCRv5 upgrade investigation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
 2025-11-27 10:35:46 +08:00
gbanyan 479d4e0019 Add PaddleOCR masking and region detection pipeline 
- Created PaddleOCR client for remote server communication
- Implemented text masking + region detection pipeline
- Test results: 100% recall on sample PDF (found both signatures)
- Identified issues: split regions, printed text not fully masked
- Documented 5 solution options in PADDLEOCR_STATUS.md
- Next: Implement region merging and two-stage cleaning
 2025-10-28 22:28:18 +08:00
gbanyan 52612e14ba Add hybrid signature extraction with name-based verification 
Implement VLM name extraction + CV detection hybrid approach to
replace unreliable VLM coordinate system with name-based verification.

Key Features:
- VLM extracts signature names (周寶蓮, 魏興海, etc.)
- CV or PDF text layer detects regions
- VLM verifies each region against expected names
- Signatures saved with person names: signature_周寶蓮.png
- Duplicate prevention and rejection handling

Test Results:
- 5 PDF pages tested
- 7/10 signatures extracted (70% recall)
- 100% precision (no false positives)
- No blank regions extracted (previous issue resolved)

Files:
- extract_pages_from_csv.py: Extract pages from CSV (tested: 100 files)
- extract_signatures_hybrid.py: Hybrid extraction (current working solution)
- extract_handwriting.py: CV-only approach (component)
- extract_signatures_vlm.py: Deprecated VLM coordinate approach
- PROJECT_DOCUMENTATION.md: Complete project history and results
- SESSION_INIT.md: Session handoff documentation
- SESSION_CHECKLIST.md: Status checklist
- NEW_SESSION_PROMPT.txt: Template for next session
- HOW_TO_CONTINUE.txt: Visual handoff guide
- COMMIT_SUMMARY.md: Commit preparation guide
- README.md: Quick start guide
- README_page_extraction.md: Page extraction docs
- README_hybrid_extraction.md: Hybrid approach docs
- .gitignore: Exclude diagnostic scripts and outputs

Known Limitations:
- 30% of signatures missed due to conservative CV parameters
- Text layer method untested (all test PDFs are scanned images)
- Performance: ~24 seconds per PDF

Next Steps:
- Tune CV parameters for higher recall
- Test with larger dataset (100+ files)
- Process full dataset (86,073 files)

🤖 Generated with Claude Code
v1.0-hybrid-70percent 		2025-10-26 23:39:52 +08:00