fix: rescue gnomAD genes with non-Ensembl IDs via gene_symbol fallback

gnomAD v4.1 gene_id column is mixed: ~101K Ensembl IDs + ~111K NCBI numeric.
Now falls back to gene_symbol → gene_universe lookup for non-ENSG entries.

Coverage: 78.5% → 90.3% (17,875 → 20,555 genes with gnomAD scores).
Sufficient evidence (≥4 layers): 19,946 → 20,683 genes.
Validation median percentile: 79.9% → 81.1%.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

data/report/candidates.provenance.yaml

@@ -1,12 +1,12 @@
-generated_at: '2026-02-15T21:13:11.954116+00:00'
+generated_at: '2026-02-15T21:49:14.608460+00:00'
 output_files:
 - candidates.tsv
 - candidates.parquet
 statistics:
-  total_candidates: 21103
+  total_candidates: 21177
-  high_count: 18
+  high_count: 82
-  medium_count: 9577
+  medium_count: 9626
-  low_count: 11508
+  low_count: 11469
 column_count: 22
 column_names:
 - gene_id

data/report/reproducibility.json

@@ -1,6 +1,6 @@
 {
-  "run_id": "e7486ff1-f9be-403b-a68d-115fc845f4a1",
+  "run_id": "093b405b-5a20-4f35-b5b8-04f02451789d",
-  "timestamp": "2026-02-15T21:13:12.288563+00:00",
+  "timestamp": "2026-02-15T21:49:14.940206+00:00",
   "pipeline_version": "0.1.0",
   "parameters": {
     "gnomad": 0.2,
@@ -49,9 +49,9 @@
   ],
   "validation_metrics": {},
   "tier_statistics": {
-    "total": 21103,
+    "total": 21177,
-    "high": 18,
+    "high": 82,
-    "medium": 9577,
+    "medium": 9626,
-    "low": 11508
+    "low": 11469
   }
 }

data/report/reproducibility.md

@@ -1,7 +1,7 @@
 # Pipeline Reproducibility Report
 
-**Run ID:** `e7486ff1-f9be-403b-a68d-115fc845f4a1`
+**Run ID:** `093b405b-5a20-4f35-b5b8-04f02451789d`
-**Timestamp:** 2026-02-15T21:13:12.288563+00:00
+**Timestamp:** 2026-02-15T21:49:14.940206+00:00
 **Pipeline Version:** 0.1.0
 
 ## Parameters
@@ -39,7 +39,7 @@
 
 ## Tier Statistics
 
-- **Total Candidates:** 21103
+- **Total Candidates:** 21177
-- **HIGH:** 18
+- **HIGH:** 82
-- **MEDIUM:** 9577
+- **MEDIUM:** 9626
-- **LOW:** 11508
+- **LOW:** 11469

src/usher_pipeline/evidence/gnomad/load.py

@@ -29,18 +29,40 @@ def load_to_duckdb(
     """
     logger.info("gnomad_load_start", row_count=len(df))
 
-    # Enrich with Ensembl gene_id from gene_universe if missing
+    # Enrich with Ensembl gene_id from gene_universe
-    # gnomAD data only has gene_symbol (HGNC); we need Ensembl gene_id for scoring JOINs
+    # gnomAD gene_id is mixed: some Ensembl (ENSG...), some NCBI numeric (4622).
-    if "gene_id" not in df.columns or df["gene_id"].null_count() == len(df):
+    # For rows without a valid Ensembl ID, fall back to gene_symbol lookup.
-        logger.info("gnomad_enriching_gene_ids", msg="Mapping gene_symbol to Ensembl gene_id via gene_universe")
+    gene_map = store.conn.execute(
-        gene_map = store.conn.execute(
+        "SELECT gene_id AS ensembl_id, gene_symbol FROM gene_universe"
-            "SELECT gene_id, gene_symbol FROM gene_universe"
+    ).pl()
-        ).pl()
+
-        if "gene_id" in df.columns:
+    if "gene_id" not in df.columns:
-            df = df.drop("gene_id")
+        # No gene_id column at all — join entirely via gene_symbol
+        logger.info("gnomad_enriching_gene_ids", msg="No gene_id column; mapping via gene_symbol")
+        df = df.join(gene_map.rename({"ensembl_id": "gene_id"}), on="gene_symbol", how="left")
+    else:
+        # gene_id exists but may contain non-Ensembl IDs — patch those via gene_symbol
+        is_ensembl = pl.col("gene_id").str.starts_with("ENSG")
+        before_ensembl = df.filter(is_ensembl).height
+
+        # Join gene_symbol → ensembl_id for fallback
         df = df.join(gene_map, on="gene_symbol", how="left")
-        matched = df.filter(pl.col("gene_id").is_not_null()).height
+        # Use original gene_id if it's Ensembl, otherwise use ensembl_id from gene_universe
-        logger.info("gnomad_gene_id_enrichment", matched=matched, total=len(df))
+        df = df.with_columns(
+            pl.when(is_ensembl)
+            .then(pl.col("gene_id"))
+            .otherwise(pl.col("ensembl_id"))
+            .alias("gene_id")
+        ).drop("ensembl_id")
+
+        after_ensembl = df.filter(pl.col("gene_id").str.starts_with("ENSG")).height
+        logger.info(
+            "gnomad_gene_id_enrichment",
+            before_ensembl=before_ensembl,
+            after_ensembl=after_ensembl,
+            rescued=after_ensembl - before_ensembl,
+            total=len(df),
+        )
 
     # Calculate summary statistics for provenance
     measured_count = df.filter(pl.col("quality_flag") == "measured").height