9.8 KiB
phase, plan, type, wave, depends_on, files_modified, autonomous, must_haves
| phase | plan | type | wave | depends_on | files_modified | autonomous | must_haves | ||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 06-validation | 02 | execute | 1 |
|
true |
|
Purpose: Demonstrates that top candidate rankings are robust to reasonable weight perturbations (+-5-10%), satisfying success criterion 3 (rank stability). This is the core of the "are our results defensible?" validation.
Output: sensitivity.py module with weight perturbation, Spearman correlation analysis, and stability classification.
<execution_context> @/Users/gbanyan/.claude/get-shit-done/workflows/execute-plan.md @/Users/gbanyan/.claude/get-shit-done/templates/summary.md </execution_context>
@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/06-validation/06-RESEARCH.md@src/usher_pipeline/scoring/integration.py @src/usher_pipeline/config/schema.py @src/usher_pipeline/persistence/duckdb_store.py
Task 1: Create sensitivity analysis module with weight perturbation and rank correlation src/usher_pipeline/scoring/sensitivity.py Create `src/usher_pipeline/scoring/sensitivity.py` with:-
EVIDENCE_LAYERS list constant: ["gnomad", "expression", "annotation", "localization", "animal_model", "literature"]
-
DEFAULT_DELTAS list constant: [-0.10, -0.05, 0.05, 0.10]
-
STABILITY_THRESHOLD float constant: 0.85 (Spearman rho threshold for "stable")
-
perturb_weight(baseline: ScoringWeights, layer: str, delta: float) -> ScoringWeights function:
- Get baseline weights as dict via baseline.model_dump()
- Apply perturbation: w_dict[layer] = max(0.0, min(1.0, w_dict[layer] + delta))
- Renormalize ALL weights so they sum to 1.0: divide each by total
- Return new ScoringWeights instance
- Raise ValueError if layer not in EVIDENCE_LAYERS
-
run_sensitivity_analysis(store: PipelineStore, baseline_weights: ScoringWeights, deltas: list[float] | None = None, top_n: int = 100) -> dict function:
- Default deltas to DEFAULT_DELTAS if None
- Compute baseline scores via compute_composite_scores(store, baseline_weights)
- Sort by composite_score DESC, take top_n genes as baseline ranking
- For each layer in EVIDENCE_LAYERS, for each delta in deltas:
- Create perturbed weights via perturb_weight()
- Compute perturbed scores via compute_composite_scores(store, perturbed_weights)
- Sort by composite_score DESC, take top_n genes
- Inner join baseline and perturbed on gene_symbol to get paired scores
- If fewer than 10 overlapping genes, log warning and record rho=None
- Otherwise compute spearmanr() on paired composite_score columns
- Record: layer, delta, perturbed_weights (as dict), spearman_rho, spearman_pval, overlap_count (how many of top_n genes appear in both), top_n
- Return dict with keys: baseline_weights (dict), results (list of per-perturbation dicts), top_n, total_perturbations
- Use structlog for progress logging (log each perturbation result)
IMPORTANT: The compute_composite_scores function re-queries the DB each time. This is by design -- different weights produce different composite_score values from the same underlying evidence layer scores.
For the Spearman correlation, join baseline_top_n and perturbed_top_n DataFrames on gene_symbol (inner join). Use the composite_score from each as the paired values. This measures whether the relative ordering of shared top genes is preserved.
-
summarize_sensitivity(analysis_result: dict) -> dict function:
- From the results list, compute:
- min_rho, max_rho, mean_rho across all perturbations (excluding None values)
- count of stable perturbations (rho >= STABILITY_THRESHOLD)
- count of unstable perturbations (rho < STABILITY_THRESHOLD)
- most_sensitive_layer: layer with lowest mean rho across its perturbations
- most_robust_layer: layer with highest mean rho across its perturbations
- overall_stable: bool = all non-None rhos >= STABILITY_THRESHOLD
- Return dict with: min_rho, max_rho, mean_rho, stable_count, unstable_count, total_perturbations, overall_stable, most_sensitive_layer, most_robust_layer
- From the results list, compute:
-
generate_sensitivity_report(analysis_result: dict, summary: dict) -> str function:
- Follow the formatting pattern from generate_validation_report() in validation.py
- Show table: Layer | Delta | Spearman rho | p-value | Stable?
- Show summary: overall stability verdict, most/least sensitive layers
- Include interpretation text
Use structlog, polars, scipy.stats.spearmanr imports. Import compute_composite_scores from usher_pipeline.scoring.integration, ScoringWeights from usher_pipeline.config.schema, PipelineStore from usher_pipeline.persistence.duckdb_store. Run: `cd /Users/gbanyan/Project/usher-exploring && python -c " from usher_pipeline.scoring.sensitivity import perturb_weight, EVIDENCE_LAYERS, STABILITY_THRESHOLD, DEFAULT_DELTAS from usher_pipeline.config.schema import ScoringWeights
Test weight perturbation
w = ScoringWeights() p = perturb_weight(w, 'gnomad', 0.10) p.validate_sum() # Must not raise print(f'Original gnomad: {w.gnomad}, Perturbed: {p.gnomad:.4f}') assert p.gnomad > w.gnomad, 'Perturbed weight should be higher'
Test renormalization
total = p.gnomad + p.expression + p.annotation + p.localization + p.animal_model + p.literature assert abs(total - 1.0) < 1e-6, f'Weights must sum to 1.0, got {total}'
Test edge: perturb to near-zero
p_low = perturb_weight(w, 'gnomad', -0.25) p_low.validate_sum() assert p_low.gnomad >= 0.0, 'Weight must not go negative'
print('All perturb_weight tests passed') "` exits 0 sensitivity.py exists with perturb_weight (renormalizing), run_sensitivity_analysis (computing Spearman rho for top-N genes across all layer/delta combinations), summarize_sensitivity (stability classification), and generate_sensitivity_report (formatted output). Weights always renormalize to sum=1.0 after perturbation.
Task 2: Export sensitivity module from scoring package src/usher_pipeline/scoring/__init__.py Update `src/usher_pipeline/scoring/__init__.py` to add imports and exports for the sensitivity module:Add imports:
from usher_pipeline.scoring.sensitivity import (
perturb_weight,
run_sensitivity_analysis,
summarize_sensitivity,
generate_sensitivity_report,
EVIDENCE_LAYERS,
STABILITY_THRESHOLD,
)
Add to all list: "perturb_weight", "run_sensitivity_analysis", "summarize_sensitivity", "generate_sensitivity_report", "EVIDENCE_LAYERS", "STABILITY_THRESHOLD"
NOTE: Plan 01 may have already updated init.py to add negative_controls exports. If so, ADD the sensitivity imports alongside those -- do not remove them. Read the file first to check current state.
Run: cd /Users/gbanyan/Project/usher-exploring && python -c "from usher_pipeline.scoring import perturb_weight, run_sensitivity_analysis, summarize_sensitivity, generate_sensitivity_report, EVIDENCE_LAYERS, STABILITY_THRESHOLD; print(f'EVIDENCE_LAYERS: {EVIDENCE_LAYERS}'); print(f'STABILITY_THRESHOLD: {STABILITY_THRESHOLD}'); print('All sensitivity exports OK')" exits 0
All sensitivity analysis functions and constants are importable from usher_pipeline.scoring. Existing exports from negative_controls (Plan 01) are preserved.
<success_criteria>
- perturb_weight correctly perturbs one layer and renormalizes to sum=1.0
- run_sensitivity_analysis computes Spearman rho for all layer x delta combinations
- summarize_sensitivity classifies perturbations as stable/unstable
- generate_sensitivity_report produces human-readable output
- All functions exported from scoring package </success_criteria>