---
phase: 02-prototype-evidence-layer
plan: 02
type: execute
wave: 2
depends_on: ["02-01"]
files_modified:
  - src/usher_pipeline/evidence/gnomad/load.py
  - src/usher_pipeline/cli/evidence_cmd.py
  - src/usher_pipeline/cli/main.py
  - tests/test_gnomad_integration.py
autonomous: true

must_haves:
  truths:
    - "gnomAD constraint data is written to DuckDB gnomad_constraint table with all columns including quality_flag and loeuf_normalized"
    - "Checkpoint-restart works: re-running the command skips download and processing if DuckDB table exists"
    - "Provenance sidecar JSON is saved alongside the gnomAD data recording source URL, version, processing steps"
    - "CLI command 'usher-pipeline evidence gnomad' runs the full fetch-transform-load pipeline"
    - "GCON-03 is addressed: constraint evidence is stored as weak signal, not direct cilia involvement"
  artifacts:
    - path: "src/usher_pipeline/evidence/gnomad/load.py"
      provides: "DuckDB persistence for gnomAD constraint data"
      contains: "def load_to_duckdb"
    - path: "src/usher_pipeline/cli/evidence_cmd.py"
      provides: "CLI evidence subcommand group with gnomad command"
      contains: "def gnomad"
    - path: "tests/test_gnomad_integration.py"
      provides: "Integration tests for gnomAD evidence layer"
      contains: "test_"
  key_links:
    - from: "src/usher_pipeline/evidence/gnomad/load.py"
      to: "src/usher_pipeline/persistence/duckdb_store.py"
      via: "saves constraint DataFrame to DuckDB via PipelineStore"
      pattern: "PipelineStore|save_dataframe"
    - from: "src/usher_pipeline/evidence/gnomad/load.py"
      to: "src/usher_pipeline/persistence/provenance.py"
      via: "records provenance metadata for gnomAD processing"
      pattern: "ProvenanceTracker|record_step"
    - from: "src/usher_pipeline/cli/evidence_cmd.py"
      to: "src/usher_pipeline/evidence/gnomad/fetch.py"
      via: "orchestrates download-transform-load pipeline"
      pattern: "download_constraint_metrics|process_gnomad_constraint|load_to_duckdb"
    - from: "src/usher_pipeline/cli/main.py"
      to: "src/usher_pipeline/cli/evidence_cmd.py"
      via: "registers evidence command group"
      pattern: "cli\\.add_command|evidence"
---

<objective>
Wire gnomAD evidence layer into DuckDB persistence and CLI, completing the full fetch-transform-load pipeline with checkpoint-restart and provenance tracking.

Purpose: This plan completes the prototype evidence layer by connecting the data retrieval/transformation (Plan 01) to storage and CLI, proving the end-to-end pattern works for all future evidence layers.
Output: Working CLI command that downloads, filters, normalizes, and persists gnomAD constraint data with full provenance.
</objective>

<execution_context>
@/Users/gbanyan/.claude/get-shit-done/workflows/execute-plan.md
@/Users/gbanyan/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-prototype-evidence-layer/02-RESEARCH.md
@.planning/phases/02-prototype-evidence-layer/02-01-SUMMARY.md
@src/usher_pipeline/persistence/duckdb_store.py
@src/usher_pipeline/persistence/provenance.py
@src/usher_pipeline/cli/main.py
@src/usher_pipeline/cli/setup_cmd.py
@config/default.yaml
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create DuckDB loader and CLI evidence command</name>
  <files>
    src/usher_pipeline/evidence/gnomad/load.py
    src/usher_pipeline/evidence/gnomad/__init__.py
    src/usher_pipeline/cli/evidence_cmd.py
    src/usher_pipeline/cli/main.py
  </files>
  <action>
1. Create `src/usher_pipeline/evidence/gnomad/load.py`:
   - `load_to_duckdb(df: pl.DataFrame, store: PipelineStore, provenance: ProvenanceTracker, description: str = "") -> None`:
     - Save DataFrame to DuckDB table named `gnomad_constraint` using `store.save_dataframe()` with `replace=True` (idempotent -- CREATE OR REPLACE).
     - Record provenance step: "load_gnomad_constraint" with details including row_count, measured_count (quality_flag == "measured"), incomplete_count, no_data_count, null_loeuf_count.
     - Log summary stats with structlog.

   - `query_constrained_genes(store: PipelineStore, loeuf_threshold: float = 0.6) -> pl.DataFrame`:
     - Convenience query: SELECT genes from gnomad_constraint WHERE loeuf < threshold AND quality_flag = 'measured'.
     - Returns polars DataFrame sorted by loeuf ascending (most constrained first).
     - This demonstrates DuckDB query capability and validates GCON-03 interpretation: constrained genes are "important but under-studied" signals, not direct cilia evidence.

2. Update `src/usher_pipeline/evidence/gnomad/__init__.py`:
   - Add exports for load_to_duckdb, query_constrained_genes, and all models/fetch/transform exports.

3. Create `src/usher_pipeline/cli/evidence_cmd.py`:
   - Create a Click command group `evidence` with help: "Fetch and process evidence layer data."
   - Create subcommand `gnomad`:
     - Options: `--force` (re-download and reprocess), `--url` (override download URL), `--min-depth` (default 30.0), `--min-cds-pct` (default 0.9)
     - Full pipeline orchestration:
       a. Load config from context (same pattern as setup_cmd.py)
       b. Create PipelineStore and ProvenanceTracker from config
       c. Check checkpoint: `store.has_checkpoint('gnomad_constraint')` -- if exists and no --force, print summary and return
       d. Download gnomAD constraint TSV to `config.data_dir / "gnomad"` directory
       e. Process with `process_gnomad_constraint()` from transform module
       f. Load to DuckDB with `load_to_duckdb()`
       g. Save provenance sidecar to `config.data_dir / "gnomad" / "constraint.provenance.json"`
       h. Print summary: total genes, measured/incomplete/no_data counts, DuckDB path
     - Use colored Click output (green for success, yellow for checkpoint skip, red for errors)
     - Wrap in try/finally for store.close() cleanup
     - Log all steps with structlog

4. Update `src/usher_pipeline/cli/main.py`:
   - Import evidence command group: `from usher_pipeline.cli.evidence_cmd import evidence`
   - Register with: `cli.add_command(evidence)`
   - This enables: `usher-pipeline evidence gnomad [--force] [--url URL]`
  </action>
  <verify>
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/python -c "from usher_pipeline.evidence.gnomad.load import load_to_duckdb, query_constrained_genes; print('load imports OK')"` -- prints OK.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/usher-pipeline evidence --help` -- shows gnomad subcommand.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/usher-pipeline evidence gnomad --help` -- shows --force, --url, --min-depth, --min-cds-pct options.
  </verify>
  <done>
    DuckDB loader persists gnomAD data with provenance. CLI `evidence gnomad` command is available with checkpoint-restart, force-rerun, and configurable options.
  </done>
</task>

<task type="auto">
  <name>Task 2: Create integration tests for full gnomAD pipeline</name>
  <files>
    tests/test_gnomad_integration.py
  </files>
  <action>
Create integration tests verifying the full gnomAD evidence layer pipeline end-to-end.

Create `tests/test_gnomad_integration.py` with the following:

1. Fixtures:
   - `test_config(tmp_path)`: Creates a temp config YAML pointing to tmp_path for data_dir, cache_dir, duckdb_path. Uses the same pattern as tests/test_integration.py.
   - `sample_tsv(tmp_path)`: Creates a synthetic gnomAD constraint TSV with ~15 rows covering:
     - 5 well-covered genes with measured LOEUF/pLI (varying values 0.1 to 2.0)
     - 3 genes with low depth (<30x)
     - 3 genes with low CDS coverage (<90%)
     - 2 genes with NULL LOEUF and pLI
     - 2 genes at normalization bounds (LOEUF=0.0, LOEUF=3.0)

2. Integration tests (at least 8):
   - `test_full_pipeline_to_duckdb`: process_gnomad_constraint -> load_to_duckdb -> verify DuckDB table has all rows, correct columns, quality_flags
   - `test_checkpoint_restart_skips_processing`: Load data, verify has_checkpoint returns True, simulate re-run check
   - `test_provenance_recorded`: After load_to_duckdb, verify provenance has step "load_gnomad_constraint" with expected details
   - `test_provenance_sidecar_created`: Verify .provenance.json file is written with correct metadata
   - `test_query_constrained_genes_filters_correctly`: Load data, then query_constrained_genes with threshold=0.6 returns only measured genes below threshold
   - `test_null_loeuf_not_in_constrained_results`: Genes with NULL LOEUF are excluded from constrained gene queries
   - `test_duckdb_schema_has_quality_flag`: Verify gnomad_constraint table has quality_flag column with non-null values
   - `test_normalized_scores_in_duckdb`: Load and verify loeuf_normalized values are in [0,1] for measured genes and NULL for others
   - `test_cli_evidence_gnomad_help`: Use Click CliRunner to invoke `evidence gnomad --help`, verify output
   - `test_cli_evidence_gnomad_with_mock`: Use Click CliRunner + mock download to test CLI runs without error (mock the actual download, provide synthetic TSV)

   Use `polars.testing.assert_frame_equal` where appropriate.
   All tests use tmp_path -- no real gnomAD downloads.
   Mock httpx downloads in CLI test -- provide synthetic TSV file instead.

3. Run full test suite to ensure no regressions:
   `pytest tests/ -v` -- all tests pass (existing 49-50 + new ~18-20).
  </action>
  <verify>
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/pytest tests/test_gnomad_integration.py -v` -- all integration tests pass.
    Run: `cd /Users/gbanyan/Project/usher-exploring && .venv/bin/pytest tests/ -v` -- full suite passes with no regressions.
  </verify>
  <done>
    Integration tests prove: gnomAD data flows from TSV through transform to DuckDB with correct quality flags, normalized scores, NULL handling, checkpoint-restart, and provenance. CLI command works end-to-end with mocked downloads. Full test suite passes.
  </done>
</task>

</tasks>

<verification>
1. CLI command `usher-pipeline evidence gnomad --help` works
2. Integration tests verify full pipeline: fetch -> transform -> DuckDB with provenance
3. Checkpoint-restart: re-running skips if gnomad_constraint table exists
4. DuckDB table has correct schema: gene_id, gene_symbol, pli, loeuf, quality_flag, loeuf_normalized
5. Provenance sidecar JSON captures source URL, version, processing steps
6. Full test suite passes: `pytest tests/ -v`
7. Requirements satisfied:
   - GCON-01: pLI and LOEUF retrieved and stored per gene
   - GCON-02: Coverage quality filter with quality flags
   - GCON-03: Constraint treated as weak signal (query_constrained_genes is informational, not cilia-direct)
</verification>

<success_criteria>
- DuckDB gnomad_constraint table stores all genes with quality_flag and loeuf_normalized columns
- NULL loeuf values remain NULL in DuckDB (not converted to 0)
- Checkpoint-restart works: second run detects existing table and skips
- Provenance JSON records source URL, gnomAD version, and processing step details
- CLI `usher-pipeline evidence gnomad` orchestrates the full pipeline
- 8+ integration tests pass covering end-to-end pipeline, checkpoint, provenance, CLI
- Full test suite passes with no regressions from Phase 1
</success_criteria>

<output>
After completion, create `.planning/phases/02-prototype-evidence-layer/02-02-SUMMARY.md`
</output>
