usher-exploring/.planning/phases/06-validation/06-01-PLAN.md

phase, plan, type, wave, depends_on, files_modified, autonomous, must_haves

phase	plan	type	wave	depends_on	files_modified	autonomous	must_haves
06-validation	01	execute	1		src/usher_pipeline/scoring/negative_controls.py src/usher_pipeline/scoring/validation.py src/usher_pipeline/scoring/__init__.py	true	truths artifacts key_links Housekeeping genes are compiled as a curated negative control set with source provenance Negative control validation shows housekeeping genes rank below 50th percentile median Positive control validation reports recall@k metrics at k=10%, 20%, top-100 Known genes achieve >70% recall in top 10% of scored candidates path provides exports src/usher_pipeline/scoring/negative_controls.py Housekeeping gene compilation and negative control validation HOUSEKEEPING_GENES_CORE compile_housekeeping_genes validate_negative_controls path provides exports src/usher_pipeline/scoring/validation.py Enhanced positive control validation with recall@k and per-source breakdown validate_known_gene_ranking compute_recall_at_k generate_validation_report path provides contains src/usher_pipeline/scoring/__init__.py Updated exports including negative control functions validate_negative_controls from to via pattern src/usher_pipeline/scoring/negative_controls.py DuckDB scored_genes table PERCENT_RANK window function query PERCENT_RANK.*ORDER BY composite_score from to via pattern src/usher_pipeline/scoring/validation.py src/usher_pipeline/scoring/known_genes.py compile_known_genes import from usher_pipeline.scoring.known_genes import

Implement negative control validation with housekeeping genes and enhance positive control validation with recall@k metrics.

Purpose: Negative controls ensure the scoring system does not indiscriminately rank all genes high (complementing the existing positive control validation). Enhanced positive control metrics (recall@k) provide the specific ">70% in top 10%" measurement required by success criteria.

Output: Two modules -- negative_controls.py (new) and enhanced validation.py (updated) -- ready for integration into the comprehensive validation report (Plan 03).

<execution_context> @/Users/gbanyan/.claude/get-shit-done/workflows/execute-plan.md @/Users/gbanyan/.claude/get-shit-done/templates/summary.md </execution_context>

@.planning/PROJECT.md @.planning/ROADMAP.md @.planning/STATE.md @.planning/phases/06-validation/06-RESEARCH.md

@src/usher_pipeline/scoring/validation.py @src/usher_pipeline/scoring/known_genes.py @src/usher_pipeline/scoring/quality_control.py @src/usher_pipeline/scoring/init.py @src/usher_pipeline/persistence/duckdb_store.py

Task 1: Create negative control validation module with housekeeping genes src/usher_pipeline/scoring/negative_controls.py Create `src/usher_pipeline/scoring/negative_controls.py` with:

1. HOUSEKEEPING_GENES_CORE frozenset constant containing 13 curated housekeeping genes: RPL13A, RPL32, RPLP0, GAPDH, ACTB, B2M, HPRT1, TBP, SDHA, PGK1, PPIA, UBC, YWHAZ. Include inline comments grouping by function (ribosomal, metabolic, transcription/reference).

2. compile_housekeeping_genes() -> pl.DataFrame function returning DataFrame with columns:

   • gene_symbol (str)
   • source (str): "literature_validated" for all
   • confidence (str): "HIGH" for all Follow the exact same pattern as compile_known_genes() in known_genes.py.

3. validate_negative_controls(store: PipelineStore, percentile_threshold: float = 0.50) -> dict function:

   • Register housekeeping genes as temporary DuckDB table _housekeeping_genes
   • Use the same PERCENT_RANK window function pattern as validate_known_gene_ranking() in validation.py
   • Query: join ranked_genes CTE with _housekeeping_genes on gene_symbol
   • INVERTED validation logic: validation_passed = median_percentile < percentile_threshold
   • Return dict with keys: total_expected, total_in_dataset, median_percentile, top_quartile_count, in_high_tier_count, validation_passed, housekeeping_gene_details (top 20 by percentile ASC)
   • Clean up temp table after query
   • Use structlog logger with info/warning levels matching validation.py patterns

4. generate_negative_control_report(metrics: dict) -> str function:

   • Follow the exact formatting pattern from generate_validation_report() in validation.py
   • Show gene table with Score, Percentile, headers
   • Include interpretation text for pass/fail

Use structlog, polars, duckdb imports matching existing scoring module patterns. Import PipelineStore from usher_pipeline.persistence.duckdb_store. Run: cd /Users/gbanyan/Project/usher-exploring && python -c "from usher_pipeline.scoring.negative_controls import HOUSEKEEPING_GENES_CORE, compile_housekeeping_genes, validate_negative_controls, generate_negative_control_report; df = compile_housekeeping_genes(); print(f'Housekeeping genes: {df.height}'); assert df.height == 13; assert set(df.columns) == {'gene_symbol', 'source', 'confidence'}; print('OK')" exits 0 negative_controls.py exists with 13 curated housekeeping genes, compile function returns correct DataFrame structure, validate function uses PERCENT_RANK with inverted threshold logic, report function generates human-readable output.

Task 2: Enhance positive control validation with recall@k metrics src/usher_pipeline/scoring/validation.py, src/usher_pipeline/scoring/__init__.py **In validation.py**, add the following functions (do NOT modify existing functions, only ADD):

1. compute_recall_at_k(store: PipelineStore, k_values: list[int] | None = None) -> dict function:

   • Default k_values: [100, 500, 1000, 2000] (absolute counts)
   • Also compute recall at percentage thresholds: top 5%, 10%, 20% of scored genes
   • Query scored_genes ordered by composite_score DESC (WHERE NOT NULL)
   • For each k: count how many known genes (from compile_known_genes, deduplicated on gene_symbol) appear in top-k
   • Recall@k = found_in_top_k / total_known_unique
   • Return dict with: recalls_absolute (dict mapping k -> recall float), recalls_percentage (dict mapping pct_string -> recall float), total_known_unique (int), total_scored (int)
   • Use structlog for logging results

2. validate_positive_controls_extended(store: PipelineStore, percentile_threshold: float = 0.75) -> dict function:

   • Call existing validate_known_gene_ranking(store, percentile_threshold) to get base metrics
   • Call compute_recall_at_k(store) to get recall metrics
   • Add per-source breakdown: compute median percentile separately for "omim_usher" and "syscilia_scgs_v2" genes
   • Per-source query: same PERCENT_RANK CTE but filter JOIN by source
   • Return dict combining base metrics + recall_at_k + per_source_breakdown (dict mapping source -> {median_percentile, count, top_quartile_count})
   • This is the "full" positive control validation for Phase 6

In init.py, add exports for: compute_recall_at_k, validate_positive_controls_extended, and also add imports/exports for negative_controls module: HOUSEKEEPING_GENES_CORE, compile_housekeeping_genes, validate_negative_controls, generate_negative_control_report. Run: cd /Users/gbanyan/Project/usher-exploring && python -c "from usher_pipeline.scoring import compute_recall_at_k, validate_positive_controls_extended, HOUSEKEEPING_GENES_CORE, compile_housekeeping_genes, validate_negative_controls, generate_negative_control_report; print('All imports OK')" exits 0 validation.py has compute_recall_at_k and validate_positive_controls_extended functions. init.py exports all new functions from both negative_controls.py and updated validation.py. Recall@k computes at both absolute and percentage thresholds. Per-source breakdown separates OMIM from SYSCILIA metrics.

- `python -c "from usher_pipeline.scoring.negative_controls import HOUSEKEEPING_GENES_CORE; assert len(HOUSEKEEPING_GENES_CORE) == 13"` -- housekeeping genes compiled - `python -c "from usher_pipeline.scoring import validate_negative_controls, compute_recall_at_k, validate_positive_controls_extended"` -- all functions importable - `python -c "from usher_pipeline.scoring.negative_controls import compile_housekeeping_genes; df = compile_housekeeping_genes(); assert 'gene_symbol' in df.columns and 'source' in df.columns"` -- DataFrame structure correct

<success_criteria>

• negative_controls.py creates housekeeping gene set and validates they rank low (inverted threshold)
• validation.py compute_recall_at_k measures recall at multiple k values including percentage-based thresholds
• validate_positive_controls_extended combines percentile + recall + per-source metrics
• All new functions exported from scoring.init </success_criteria>

After completion, create `.planning/phases/06-validation/06-01-SUMMARY.md`