# Signature Verification Threshold Validation Options

**Report Date:** 2026-01-14
**Purpose:** Discussion document for research partners on threshold selection methodology
**Context:** Validating copy-paste detection thresholds for accountant signature analysis

---

## Table of Contents

1. [Current Findings Summary](#1-current-findings-summary)
2. [The Core Problem](#2-the-core-problem)
3. [Key Metrics Explained](#3-key-metrics-explained)
4. [Validation Options](#4-validation-options)
5. [Academic References](#5-academic-references)
6. [Recommendations](#6-recommendations)
7. [Next Steps for Discussion](#7-next-steps-for-discussion)

---

## 1. Current Findings Summary

Our YOLO-based signature extraction and similarity analysis produced the following results:

| Metric | Value |
|--------|-------|
| Total PDFs analyzed | 84,386 |
| Total signatures extracted | 168,755 |
| High similarity pairs (>0.95) | 659,111 |
| Classified as "copy-paste" | 71,656 PDFs (84.9%) |
| Classified as "authentic" | 76 PDFs (0.1%) |
| Uncertain | 12,651 PDFs (15.0%) |

**Current threshold used:**
- Copy-paste: similarity ≥ 0.95
- Authentic: similarity ≤ 0.85
- Uncertain: 0.85 < similarity < 0.95

---

## 2. The Core Problem

### 2.1 What is Ground Truth?

**Ground truth labels** are pre-verified classifications that serve as the "correct answer" for machine learning evaluation. For signature verification:

| Label | Meaning | How to Obtain |
|-------|---------|---------------|
| **Genuine** | Physically hand-signed by the accountant | Expert forensic examination |
| **Copy-paste/Forged** | Digitally copied from another document | Pixel-level analysis or expert verification |

### 2.2 Why We Need Ground Truth

To calculate rigorous metrics like EER (Equal Error Rate), we need labeled data:

```
EER Calculation requires:
├── Known genuine signatures → Calculate FRR at each threshold
├── Known forged signatures  → Calculate FAR at each threshold
└── Find threshold where FAR = FRR → This is EER
```

### 2.3 Our Current Limitation

We do not have pre-labeled ground truth data. Our current classification is based on:
- **Domain assumption**: Identical handwritten signatures are physically impossible
- **Similarity threshold**: Arbitrarily selected at 0.95

This approach is reasonable but may be challenged in academic peer review without additional validation.

---

## 3. Key Metrics Explained

### 3.1 Error Rate Metrics

| Metric | Full Name | Formula | Interpretation |
|--------|-----------|---------|----------------|
| **FAR** | False Acceptance Rate | Forgeries Accepted / Total Forgeries | Security risk |
| **FRR** | False Rejection Rate | Genuine Rejected / Total Genuine | Usability risk |
| **EER** | Equal Error Rate | Point where FAR = FRR | Overall performance |
| **AER** | Average Error Rate | (FAR + FRR) / 2 | Combined error |

### 3.2 Visual Representation of EER

```
        100% ┌─────────────────────────────────────┐
             │ FRR                                 │
             │  \                                  │
             │   \                                 │
        Rate │    \        ╳ ← EER point           │
             │     \      /                        │
             │      \    /                         │
             │       \  /   FAR                    │
          0% │────────\/──────────────────────────│
             └─────────────────────────────────────┘
             Low ←──── Threshold ────→ High
```

### 3.3 Benchmark Performance (from Literature)

| System | Dataset | EER | Reference |
|--------|---------|-----|-----------|
| SigNet (Siamese CNN) | GPDS-300 | 3.92% | Dey et al., 2017 |
| Consensus-Threshold | GPDS-300 | 1.27% FAR | arXiv:2401.03085 |
| Type-2 Neutrosophic | Custom | 98% accuracy | IASC 2024 |
| InceptionV3 Transfer | CEDAR | 99.10% accuracy | Springer 2024 |

---

## 4. Validation Options

### Option 1: Manual Ground Truth Creation (Most Rigorous)

**Description:**
Manually verify a subset of signatures with human expert examination.

**Methodology:**
1. Randomly sample ~100-200 signature pairs from different similarity ranges
2. Expert examines original PDF documents for:
   - Scan artifact variations (genuine scans have unique noise)
   - Pixel-perfect alignment (copy-paste is exact)
   - Ink pressure and stroke variations
   - Document metadata (creation dates, software used)
3. Label each pair as "genuine" or "copy-paste"
4. Calculate EER, FAR, FRR at various thresholds
5. Select optimal threshold based on EER

**Pros:**
- Academically rigorous
- Enables standard metric calculation (EER, FAR, FRR)
- Defensible in peer review

**Cons:**
- Time-consuming (estimated 20-40 hours for 200 samples)
- Requires forensic document expertise
- Subjective in edge cases

**Academic Support:**
> "The final verification results can be obtained by the voting method with different thresholds and can be adjusted according to different types of application requirements."
> — Hadjadj et al., Applied Sciences, 2020 [[1]](#ref1)

---

### Option 2: Statistical Distribution-Based Threshold (No Labels Needed)

**Description:**
Use the statistical distribution of similarity scores to define outliers.

**Methodology:**
1. Calculate mean (μ) and standard deviation (σ) of all similarity scores
2. Define thresholds based on standard deviations:

| Threshold | Formula | Percentile | Classification |
|-----------|---------|------------|----------------|
| Very High | > μ + 3σ | 99.7% | Definite copy-paste |
| High | > μ + 2σ | 95% | Likely copy-paste |
| Normal | μ ± 2σ | 5-95% | Uncertain |
| Low | < μ - 2σ | <5% | Likely genuine |

**Your Data:**
```
Mean similarity (μ) = 0.7608
Std deviation (σ)   = 0.0916

Thresholds:
- μ + 2σ = 0.944 (95th percentile)
- μ + 3σ = 1.035 (99.7th percentile, capped at 1.0)

Your current 0.95 threshold ≈ μ + 2.07σ (96th percentile)
```

**Pros:**
- No manual labeling required
- Statistically defensible
- Based on actual data distribution

**Cons:**
- Assumes normal distribution (may not hold)
- Does not provide FAR/FRR metrics
- Less intuitive for non-statistical audiences

**Academic Support:**
> "Keypoint-based detection methods employ statistical thresholds derived from feature distributions to identify anomalous similarity patterns."
> — Copy-Move Forgery Detection Survey, Multimedia Tools & Applications, 2024 [[2]](#ref2)

---

### Option 3: Physical Impossibility Argument (Domain Knowledge)

**Description:**
Use the physical impossibility of identical handwritten signatures as justification.

**Methodology:**
1. Define threshold based on handwriting science:

| Similarity | Physical Interpretation | Classification |
|------------|------------------------|----------------|
| = 1.0 | Pixel-identical; physically impossible for handwriting | **Definite copy** |
| > 0.98 | Near-identical; extremely improbable naturally | **Very likely copy** |
| 0.90 - 0.98 | Highly similar; unusual but possible | **Suspicious** |
| 0.80 - 0.90 | Similar; consistent with same signer | **Uncertain** |
| < 0.80 | Different; normal variation | **Likely genuine** |

2. Cite forensic document examination literature on signature variability

**Pros:**
- Intuitive and explainable
- Based on established forensic principles
- Does not require labeled data

**Cons:**
- Thresholds are somewhat arbitrary
- May not account for digital signature pads (lower variation)
- Requires supporting citations

**Academic Support:**
> "Signature verification presents several unique difficulties: high intra-class variability (an individual's signature may vary greatly day-to-day), large temporal variation (signature may change completely over time), and high inter-class similarity (forgeries attempt to be indistinguishable)."
> — Stanford CS231n Report, 2016 [[3]](#ref3)

> "A genuine signer's signature is naturally unstable even at short time-intervals, presenting inherent variation that digital copies lack."
> — Consensus-Threshold Criterion, arXiv:2401.03085, 2024 [[4]](#ref4)

---

### Option 4: Pixel-Level Copy Detection (Technical Verification)

**Description:**
Detect exact copies through pixel-level analysis, independent of feature similarity.

**Methodology:**
1. For high-similarity pairs (>0.95), perform additional checks:

```python
# Check 1: Exact pixel match
if np.array_equal(image1, image2):
    return "DEFINITE_COPY"

# Check 2: Structural Similarity Index (SSIM)
ssim_score = structural_similarity(image1, image2)
if ssim_score > 0.999:
    return "DEFINITE_COPY"

# Check 3: Histogram correlation
hist_corr = cv2.compareHist(hist1, hist2, cv2.HISTCMP_CORREL)
if hist_corr > 0.999:
    return "LIKELY_COPY"
```

2. Use copy-move forgery detection (CMFD) techniques from image forensics

**Pros:**
- Technical proof of copying
- Not dependent on threshold selection
- Provides definitive evidence for exact copies

**Cons:**
- Only detects exact copies (not scaled/rotated)
- Requires additional processing
- May miss high-quality forgeries

**Academic Support:**
> "Block-based methods segment an image into overlapping blocks and extract features. The forgery regions are determined by computing the similarity between block features using DCT (Discrete Cosine Transform) or SIFT (Scale-Invariant Feature Transform)."
> — Copy-Move Forgery Detection Survey, 2024 [[2]](#ref2)

---

### Option 5: Siamese Network with Learned Threshold (Advanced)

**Description:**
Train a Siamese neural network on signature pairs to learn optimal decision boundaries.

**Methodology:**
1. Collect training data:
   - Positive pairs: Same accountant, different documents
   - Negative pairs: Different accountants
2. Train Siamese network with contrastive or triplet loss
3. Network learns embedding space where:
   - Same-person signatures cluster together
   - Different-person signatures separate
4. Threshold is learned during training, not manually set

**Architecture:**
```
┌──────────────┐     ┌──────────────┐
│  Signature 1 │     │  Signature 2 │
└──────┬───────┘     └──────┬───────┘
       │                    │
       ▼                    ▼
┌──────────────┐     ┌──────────────┐
│   CNN        │     │   CNN        │  (Shared weights)
│   Encoder    │     │   Encoder    │
└──────┬───────┘     └──────┬───────┘
       │                    │
       ▼                    ▼
┌──────────────┐     ┌──────────────┐
│  Embedding   │     │  Embedding   │
│  Vector      │     │  Vector      │
└──────┬───────┘     └──────┬───────┘
       │                    │
       └────────┬───────────┘
                │
                ▼
        ┌───────────────┐
        │   Distance    │
        │   Metric      │
        └───────┬───────┘
                │
                ▼
        ┌───────────────┐
        │  Same/Different│
        └───────────────┘
```

**Pros:**
- Learns optimal threshold from data
- State-of-the-art performance
- Handles complex variations

**Cons:**
- Requires substantial training data
- Computationally expensive
- May overfit to specific accountant styles

**Academic Support:**
> "SigNet provided better results than the state-of-the-art results on most of the benchmark signature datasets by learning a feature space where similar observations are placed in proximity."
> — SigNet, arXiv:1707.02131, 2017 [[5]](#ref5)

> "Among various distance measures employed in the t-Siamese similarity network, the Manhattan distance technique emerged as the most effective."
> — Triplet Siamese Similarity Networks, Mathematics, 2024 [[6]](#ref6)

---

## 5. Academic References

<a name="ref1"></a>
### [1] Single Known Sample Verification (MDPI 2020)
**Title:** An Offline Signature Verification and Forgery Detection Method Based on a Single Known Sample and an Explainable Deep Learning Approach
**Authors:** Hadjadj, I. et al.
**Journal:** Applied Sciences, 10(11), 3716
**Year:** 2020
**URL:** https://www.mdpi.com/2076-3417/10/11/3716
**Key Findings:**
- Accuracy: 94.37% - 99.96%
- FRR: 0% - 5.88%
- FAR: 0.22% - 5.34%
- Voting method with adjustable thresholds

<a name="ref2"></a>
### [2] Copy-Move Forgery Detection Survey (Springer 2024)
**Title:** Copy-move forgery detection in digital image forensics: A survey
**Journal:** Multimedia Tools and Applications
**Year:** 2024
**URL:** https://link.springer.com/article/10.1007/s11042-024-18399-2
**Key Findings:**
- Block-based, keypoint-based, and deep learning methods reviewed
- DCT and SIFT for feature extraction
- Statistical thresholds for anomaly detection

<a name="ref3"></a>
### [3] Stanford CS231n Signature Verification Report
**Title:** Offline Signature Verification with Convolutional Neural Networks
**Institution:** Stanford University
**Year:** 2016
**URL:** https://cs231n.stanford.edu/reports/2016/pdfs/276_Report.pdf
**Key Findings:**
- High intra-class variability challenge
- Low inter-class similarity for skilled forgeries
- CNN-based feature extraction

<a name="ref4"></a>
### [4] Consensus-Threshold Criterion (arXiv 2024)
**Title:** Consensus-Threshold Criterion for Offline Signature Verification using Convolutional Neural Network Learned Representations
**Year:** 2024
**URL:** https://arxiv.org/abs/2401.03085
**Key Findings:**
- Achieved 1.27% FAR (vs 8.73% and 17.31% in prior work)
- Consensus-threshold distance-based classifier
- Uses SigNet and SigNet-F features

<a name="ref5"></a>
### [5] SigNet: Siamese Network for Signature Verification (arXiv 2017)
**Title:** SigNet: Convolutional Siamese Network for Writer Independent Offline Signature Verification
**Authors:** Dey, S. et al.
**Year:** 2017
**URL:** https://arxiv.org/abs/1707.02131
**Key Findings:**
- Siamese architecture with shared weights
- Euclidean distance minimization for genuine pairs
- State-of-the-art on GPDS, CEDAR, MCYT datasets

<a name="ref6"></a>
### [6] Triplet Siamese Similarity Networks (MDPI 2024)
**Title:** Enhancing Signature Verification Using Triplet Siamese Similarity Networks in Digital Documents
**Journal:** Mathematics, 12(17), 2757
**Year:** 2024
**URL:** https://www.mdpi.com/2227-7390/12/17/2757
**Key Findings:**
- Manhattan distance outperforms Euclidean and Minkowski
- Triplet loss for inter-class/intra-class optimization
- Tested on 4NSigComp2012, SigComp2011, BHSig260

<a name="ref7"></a>
### [7] Original Siamese Network Paper (NeurIPS 1993)
**Title:** Signature Verification using a "Siamese" Time Delay Neural Network
**Authors:** Bromley, J. et al.
**Conference:** NeurIPS 1993
**URL:** https://papers.neurips.cc/paper/1993/file/288cc0ff022877bd3df94bc9360b9c5d-Paper.pdf
**Key Findings:**
- Introduced Siamese architecture for signature verification
- Cosine similarity = 1.0 for genuine pairs
- Foundational work for modern approaches

<a name="ref8"></a>
### [8] Australian Journal of Forensic Sciences (2024)
**Title:** Handling high level of uncertainty in forensic signature examination
**Journal:** Australian Journal of Forensic Sciences, 57(5)
**Year:** 2024
**URL:** https://www.tandfonline.com/doi/full/10.1080/00450618.2024.2410044
**Key Findings:**
- Type-2 Neutrosophic similarity measure
- 98% accuracy (vs 95% for Type-1)
- Addresses ambiguity in forensic analysis

<a name="ref9"></a>
### [9] Benchmark Datasets
**CEDAR Dataset:**
- 55 signers × 24 genuine + 24 forged signatures
- URL: https://paperswithcode.com/dataset/cedar-signature

**GPDS-960 Corpus:**
- 960 writers × 24 genuine + 30 forgeries
- 600 dpi grayscale scans
- URL: https://www.researchgate.net/publication/220860371

---

## 6. Recommendations

### For Academic Publication

| Priority | Option | Effort | Rigor | Recommendation |
|----------|--------|--------|-------|----------------|
| 1 | **Option 1 + Option 2** | High | Very High | Create small labeled dataset + validate statistical threshold |
| 2 | **Option 2 + Option 3** | Low | Medium | Statistical threshold + physical impossibility argument |
| 3 | **Option 4** | Medium | High | Add pixel-level verification for definitive cases |

### Suggested Approach

1. **Primary method:** Use statistical threshold (Option 2)
   - Report threshold as μ + 2σ ≈ 0.944 (close to your current 0.95)
   - Statistically defensible without ground truth

2. **Supporting evidence:** Physical impossibility argument (Option 3)
   - Cite forensic literature on signature variability
   - Emphasize that identical signatures are physically impossible

3. **Validation (if time permits):** Small labeled subset (Option 1)
   - Manually verify 100-200 samples
   - Calculate EER to validate threshold choice

4. **Technical proof:** Pixel-level analysis (Option 4)
   - Add SSIM analysis for high-similarity pairs
   - Report exact copy counts separately

### Suggested Report Language

> "We adopt a similarity threshold of 0.95 (approximately μ + 2σ, representing the 96th percentile of our similarity distribution) to classify signatures as potential copy-paste instances. This threshold is supported by: (1) statistical outlier detection principles, (2) the physical impossibility of pixel-identical handwritten signatures, and (3) alignment with forensic document examination literature [cite: Hadjadj 2020, arXiv:2401.03085]."

---

## 7. Next Steps for Discussion

### Questions for Research Partners

1. **Data availability:** Do we have access to any documents with known authentic signatures for validation?

2. **Expert resources:** Can we involve a forensic document examiner for ground truth labeling?

3. **Scope decision:** Should we focus on statistical validation (faster) or pursue full EER analysis (more rigorous)?

4. **Publication target:** What level of rigor does the target journal require?

5. **Time constraints:** How much time can we allocate to validation before submission?

### Proposed Action Items

| Task | Owner | Deadline | Notes |
|------|-------|----------|-------|
| Review this document | All partners | TBD | Discuss options |
| Select validation approach | Team decision | TBD | Based on resources |
| Implement selected approach | TBD | TBD | After decision |
| Update threshold if needed | TBD | TBD | Based on validation |
| Draft methodology section | TBD | TBD | For paper |

---

## Appendix: Code for Statistical Threshold Calculation

```python
import numpy as np
from scipy import stats

# Your similarity data
similarities = [...]  # Load from your analysis

# Calculate statistics
mean_sim = np.mean(similarities)
std_sim = np.std(similarities)
percentiles = np.percentile(similarities, [90, 95, 99, 99.7])

print(f"Mean (μ): {mean_sim:.4f}")
print(f"Std (σ): {std_sim:.4f}")
print(f"μ + 2σ: {mean_sim + 2*std_sim:.4f}")
print(f"μ + 3σ: {mean_sim + 3*std_sim:.4f}")
print(f"Percentiles: 90%={percentiles[0]:.4f}, 95%={percentiles[1]:.4f}, "
      f"99%={percentiles[2]:.4f}, 99.7%={percentiles[3]:.4f}")

# Threshold recommendations
thresholds = {
    "Conservative (μ+3σ)": min(1.0, mean_sim + 3*std_sim),
    "Standard (μ+2σ)": mean_sim + 2*std_sim,
    "Liberal (95th percentile)": percentiles[1],
}

for name, thresh in thresholds.items():
    count_above = np.sum(similarities > thresh)
    pct_above = 100 * count_above / len(similarities)
    print(f"{name}: {thresh:.4f} → {count_above} pairs ({pct_above:.2f}%)")
```

---

*Document prepared for research discussion. Please share feedback and questions with the team.*
