---
phase: 02-prototype-evidence-layer
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - src/usher_pipeline/evidence/__init__.py
  - src/usher_pipeline/evidence/gnomad/__init__.py
  - src/usher_pipeline/evidence/gnomad/models.py
  - src/usher_pipeline/evidence/gnomad/fetch.py
  - src/usher_pipeline/evidence/gnomad/transform.py
  - tests/test_gnomad.py
  - pyproject.toml
autonomous: true

must_haves:
  truths:
    - "gnomAD constraint TSV downloads with retry and streams to disk without loading entirely into memory"
    - "Coverage quality filter removes genes with mean depth <30x or <90% CDS covered"
    - "Missing constraint data (pLI, LOEUF) is preserved as NULL, not zero"
    - "Quality flag column distinguishes 'measured' from 'incomplete_coverage' genes"
    - "LOEUF scores are normalized to 0-1 range with inversion (high score = more constrained)"
  artifacts:
    - path: "src/usher_pipeline/evidence/gnomad/models.py"
      provides: "Pydantic model for gnomAD constraint record"
      contains: "class ConstraintRecord"
    - path: "src/usher_pipeline/evidence/gnomad/fetch.py"
      provides: "gnomAD constraint file download with retry"
      contains: "def download_constraint_metrics"
    - path: "src/usher_pipeline/evidence/gnomad/transform.py"
      provides: "Coverage filter, NULL handling, normalization"
      contains: "def filter_by_coverage"
    - path: "tests/test_gnomad.py"
      provides: "Unit tests for gnomAD fetch and transform"
      contains: "test_"
  key_links:
    - from: "src/usher_pipeline/evidence/gnomad/fetch.py"
      to: "httpx"
      via: "streaming download with tenacity retry"
      pattern: "httpx\\.stream|@retry"
    - from: "src/usher_pipeline/evidence/gnomad/transform.py"
      to: "polars"
      via: "lazy scan with null handling and coverage filter"
      pattern: "pl\\.scan_csv|pl\\.col.*is_null|quality_flag"
    - from: "src/usher_pipeline/evidence/gnomad/transform.py"
      to: "src/usher_pipeline/evidence/gnomad/models.py"
      via: "uses ConstraintRecord or column names for validation"
      pattern: "ConstraintRecord|loeuf_normalized"
---

<objective>
Implement gnomAD constraint data retrieval, quality filtering, and normalization -- the core data pipeline for the prototype evidence layer.

Purpose: This plan establishes the evidence layer pattern (fetch -> filter -> normalize) that all future evidence layers in Phase 3 will follow. Getting this right with gnomAD sets the template.
Output: Downloadable gnomAD constraint data, coverage-filtered and normalized, ready for DuckDB storage.
</objective>

<execution_context>
@/Users/gbanyan/.claude/get-shit-done/workflows/execute-plan.md
@/Users/gbanyan/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-prototype-evidence-layer/02-RESEARCH.md
@src/usher_pipeline/config/schema.py
@src/usher_pipeline/api_clients/base.py
@src/usher_pipeline/persistence/duckdb_store.py
@pyproject.toml
@config/default.yaml
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create gnomAD data model and download module</name>
  <files>
    src/usher_pipeline/evidence/__init__.py
    src/usher_pipeline/evidence/gnomad/__init__.py
    src/usher_pipeline/evidence/gnomad/models.py
    src/usher_pipeline/evidence/gnomad/fetch.py
    pyproject.toml
  </files>
  <action>
Create the evidence layer package structure and gnomAD-specific modules.

1. Create `src/usher_pipeline/evidence/__init__.py` -- empty package init for evidence layers.

2. Create `src/usher_pipeline/evidence/gnomad/__init__.py` -- exports from models, fetch, transform.

3. Create `src/usher_pipeline/evidence/gnomad/models.py`:
   - Define `ConstraintRecord` as a Pydantic BaseModel with fields:
     - `gene_id: str` (Ensembl gene ID, e.g. ENSG00000...)
     - `gene_symbol: str`
     - `transcript: str` (canonical transcript ID)
     - `pli: float | None` (NULL if no estimate)
     - `loeuf: float | None` (NULL if no estimate -- CRITICAL: not 0.0)
     - `loeuf_upper: float | None` (upper bound of LOEUF CI)
     - `mean_depth: float | None` (mean exome depth)
     - `cds_covered_pct: float | None` (fraction of CDS bases with adequate coverage)
     - `quality_flag: str` -- "measured", "incomplete_coverage", or "no_data"
     - `loeuf_normalized: float | None` -- normalized score (filled by transform)
   - Define `GNOMAD_CONSTRAINT_URL` constant -- use gnomAD v4.1 constraint metrics download URL: `https://storage.googleapis.com/gcp-public-data--gnomad/release/4.1/constraint/gnomad.v4.1.constraint_metrics.tsv`
     If that exact URL is wrong, the code should accept a configurable URL parameter with this as default. The file may be .tsv.bgz (bgzip compressed) -- handle both .tsv and .tsv.bgz extensions.
   - Define column name constants mapping gnomAD TSV columns to our field names. The gnomAD v4.0+ constraint file uses columns like: `gene`, `transcript`, `mane_select`, `lof.oe_ci.upper` (LOEUF), `lof.pLI`, `mean_proportion_covered` (CDS coverage). Inspect actual header at download time and log column names if they differ from expected. Research note: column names may vary between v2.1.1 and v4.x -- code MUST handle this by looking for known column name variants.

4. Create `src/usher_pipeline/evidence/gnomad/fetch.py`:
   - `download_constraint_metrics(output_path: Path, url: str = GNOMAD_CONSTRAINT_URL, force: bool = False) -> Path`:
     - If `output_path` exists and `force=False`, return early (checkpoint pattern).
     - Use `httpx` with streaming to download to disk (NOT into memory -- file can be ~50-100MB).
     - Wrap with `@retry` from tenacity: 5 attempts, exponential backoff (min=4s, max=60s), retry on httpx.HTTPStatusError / httpx.ConnectError / httpx.TimeoutException.
     - Log download progress with structlog (use `structlog.get_logger()`).
     - If file is .bgz compressed, decompress after download using gzip (bgzip is gzip-compatible).
     - Return path to final TSV file.
   - `parse_constraint_tsv(tsv_path: Path) -> pl.LazyFrame`:
     - Use `pl.scan_csv(tsv_path, separator="\t", null_values=["NA", "", "."])` for lazy evaluation.
     - Select and rename relevant columns to match our ConstraintRecord field names.
     - Handle column name variants between gnomAD versions (v2.1.1 uses `oe_lof_upper`, v4.x might use `lof.oe_ci.upper` -- check actual file and map accordingly).
     - Log detected column names and gnomAD version info.
     - Return LazyFrame (do NOT call .collect() -- leave lazy for downstream filtering).

5. Update `pyproject.toml` -- add `httpx>=0.28` and `structlog>=25.0` to dependencies. Keep all existing dependencies unchanged.
  </action>
  <verify>
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/pip install -e ".[dev]"` -- succeeds with httpx and structlog installed.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/python -c "from usher_pipeline.evidence.gnomad.models import ConstraintRecord, GNOMAD_CONSTRAINT_URL; print(GNOMAD_CONSTRAINT_URL)"` -- prints URL.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/python -c "from usher_pipeline.evidence.gnomad.fetch import download_constraint_metrics, parse_constraint_tsv; print('imports OK')"` -- prints imports OK.
  </verify>
  <done>
    gnomAD evidence module exists with ConstraintRecord model, streaming download function with retry, and lazy TSV parser. httpx and structlog are installed as dependencies.
  </done>
</task>

<task type="auto">
  <name>Task 2: Create coverage filter, normalization, and unit tests</name>
  <files>
    src/usher_pipeline/evidence/gnomad/transform.py
    tests/test_gnomad.py
  </files>
  <action>
Create the transform module and comprehensive tests for the entire gnomAD evidence layer.

1. Create `src/usher_pipeline/evidence/gnomad/transform.py`:
   - `filter_by_coverage(lf: pl.LazyFrame, min_depth: float = 30.0, min_cds_pct: float = 0.9) -> pl.LazyFrame`:
     - Does NOT drop rows -- adds `quality_flag` column:
       - "measured" if mean_depth >= min_depth AND cds_covered_pct >= min_cds_pct AND loeuf is not null
       - "incomplete_coverage" if coverage thresholds not met but gene exists
       - "no_data" if loeuf AND pli are both null
     - Preserves ALL genes (important for downstream: "unknown" != "zero").
     - Returns LazyFrame with quality_flag column added.

   - `normalize_scores(lf: pl.LazyFrame) -> pl.LazyFrame`:
     - Compute min-max normalization on LOEUF scores for genes with quality_flag == "measured".
     - INVERT the scale: lower LOEUF = more constrained = HIGHER normalized score.
     - Formula: `loeuf_normalized = (loeuf_max - loeuf) / (loeuf_max - loeuf_min)` for measured genes.
     - Genes with quality_flag != "measured" get `loeuf_normalized = NULL` (not 0.0).
     - Returns LazyFrame with loeuf_normalized column added.

   - `process_gnomad_constraint(tsv_path: Path, min_depth: float = 30.0, min_cds_pct: float = 0.9) -> pl.DataFrame`:
     - Convenience function composing: parse_constraint_tsv -> filter_by_coverage -> normalize_scores -> .collect()
     - Returns materialized DataFrame ready for DuckDB storage.
     - Logs summary stats: total genes, measured count, incomplete count, no_data count.

2. Create `tests/test_gnomad.py` with tests using synthetic data (NO real gnomAD download in tests):

   Create a pytest fixture `sample_constraint_tsv(tmp_path)` that writes a small TSV file with ~10-15 rows covering edge cases:
   - Normal genes with good coverage (measured)
   - Genes with low depth (<30x) (incomplete_coverage)
   - Genes with low CDS coverage (<90%) (incomplete_coverage)
   - Genes with NULL loeuf/pli (no_data)
   - Genes with extreme LOEUF values (0.0, 2.5) for normalization bounds
   - Gene with LOEUF = 0.0 (most constrained -- should normalize to 1.0)

   Tests (at least 10):
   - `test_parse_constraint_tsv_returns_lazyframe`: Verify parse returns LazyFrame with expected columns
   - `test_parse_constraint_tsv_null_handling`: NA/empty values become polars null, not zero
   - `test_filter_by_coverage_measured`: Good coverage genes get quality_flag="measured"
   - `test_filter_by_coverage_incomplete`: Low depth/CDS genes get quality_flag="incomplete_coverage"
   - `test_filter_by_coverage_no_data`: NULL loeuf+pli genes get quality_flag="no_data"
   - `test_filter_preserves_all_genes`: Row count before == row count after (no genes dropped)
   - `test_normalize_scores_range`: All non-null normalized scores are in [0, 1]
   - `test_normalize_scores_inversion`: Lower LOEUF -> higher normalized score
   - `test_normalize_scores_null_preserved`: NULL loeuf stays NULL after normalization
   - `test_normalize_scores_incomplete_stays_null`: incomplete_coverage genes get NULL normalized score
   - `test_process_gnomad_constraint_end_to_end`: Full pipeline returns DataFrame with all expected columns
   - `test_constraint_record_model_validation`: ConstraintRecord validates correctly, rejects bad types
   - `test_download_skips_if_exists`: download_constraint_metrics returns early if file exists and force=False (mock httpx)

   Use `polars.testing.assert_frame_equal` where appropriate.
   Mock httpx in download tests (do NOT make real HTTP requests).
  </action>
  <verify>
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/pytest tests/test_gnomad.py -v` -- all tests pass.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/pytest tests/ -v` -- full test suite passes (existing + new).
  </verify>
  <done>
    Transform module implements coverage filtering (preserving all genes with quality flags), LOEUF normalization with inversion, and NULL preservation. 10+ tests pass covering all edge cases including null handling, normalization bounds, and download skip behavior.
  </done>
</task>

</tasks>

<verification>
1. All new modules import without error
2. All tests pass: `pytest tests/test_gnomad.py -v`
3. Full test suite still passes: `pytest tests/ -v`
4. NULL values are preserved through the pipeline (not converted to 0)
5. Normalized LOEUF scores are in [0, 1] range with correct inversion
6. Quality flags correctly categorize genes by coverage status
</verification>

<success_criteria>
- gnomAD evidence module exists at src/usher_pipeline/evidence/gnomad/ with models, fetch, transform
- Download function uses httpx streaming with tenacity retry (not requests library)
- Coverage filter adds quality_flag without dropping any genes
- Normalization inverts LOEUF (lower LOEUF = higher score) and preserves NULLs
- 10+ unit tests pass covering edge cases
- httpx and structlog added to pyproject.toml dependencies
</success_criteria>

<output>
After completion, create `.planning/phases/02-prototype-evidence-layer/02-01-SUMMARY.md`
</output>
