gbanyan/usher-exploring
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
95 Commits 1 Branch 0 Tags
master
Commit Graph
95 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
gbanyanandClaude Opus 4.6 bc40704323 docs: finalize top20 analysis with convergent signal observations 
v5 of top20 report adds key biological insights: dynein system dominance
(#1 + #2), Na+/K+ pump convergence (3 subunits in top 20), ARL3 as best
novel ciliopathy candidate (52% cilia literature), MAPRE3 as most
under-studied high-scoring gene (81 papers).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 10:21:37 +08:00
gbanyanandClaude Opus 4.6 a228a56984 fix: deduplicate gene_symbol in scoring to prevent non-canonical ID inflation 
gene_universe contains 1,539 gene_symbols mapping to multiple Ensembl IDs
(3,033 excess). Non-canonical IDs lack data in some evidence tables, causing
weighted_sum/available_weight to inflate their composite scores.

Fix: after scoring SQL, keep one row per gene_symbol with the most
evidence_count (tiebreak by composite_score DESC). 22,604 → 19,555 genes.

Results: HIGH 82→4, all top 20 now have 5-6 layers with expression data.
Validation PASSED (CDH23 98.3rd percentile, median known gene 83.3%).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 10:02:32 +08:00
gbanyanandClaude Opus 4.6 b151faa80a docs: update top20 analysis with 93.8% gnomAD coverage results 
Reflects improved gnomAD gene_symbol fallback (+3,471 genes).
Adds dual ranking (≥3 vs ≥4 layers) to address normalization inflation.
New high-priority candidates: ARL3 (52% cilia literature), HDAC6 (known
ciliogenesis regulator). HIGH tier: 82 genes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 05:58:36 +08:00
gbanyanandClaude Opus 4.6 e2a3dc2bc8 fix: rescue gnomAD genes with non-Ensembl IDs via gene_symbol fallback 
gnomAD v4.1 gene_id column is mixed: ~101K Ensembl IDs + ~111K NCBI numeric.
Now falls back to gene_symbol → gene_universe lookup for non-ENSG entries.

Coverage: 78.5% → 90.3% (17,875 → 20,555 genes with gnomAD scores).
Sufficient evidence (≥4 layers): 19,946 → 20,683 genes.
Validation median percentile: 79.9% → 81.1%.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 05:49:27 +08:00
gbanyanandClaude Opus 4.6 13bba55ac7 docs: add top 20 candidate gene analysis with full 6-layer evidence 
Detailed analysis of top candidates from complete pipeline run with gnomAD
constraint, tissue expression, annotation, localization, animal model, and
literature evidence. Includes per-gene evidence breakdown, biological
interpretation, and prioritized recommendations for follow-up.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 05:34:13 +08:00
gbanyanandClaude Opus 4.6 fe8e13c1a1 fix: restore gnomAD and expression evidence layers for complete 6-layer scoring 
Three bugs prevented gnomAD and expression data from contributing to scores:
1. gnomAD COLUMN_VARIANTS mapped "gene" (HGNC symbol) to gene_id instead of
   gene_symbol, causing JOIN miss with gene_universe (Ensembl IDs)
2. Expression HPA data was fetched but never merged (lf_hpa unused)
3. GTEx versioned Ensembl IDs (ENSG*.5) didn't match gene_universe

Results: gnomAD 78.5% coverage, expression 87.4%, 19946 genes with ≥4 layers.
HIGH tier refined from 44 → 18 candidates. Validation PASSED (CDH23 96.5th pctl).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 05:25:37 +08:00
gbanyanandClaude Opus 4.6 b63251a996 fix: resolve JOIN explosion from duplicate gene_ids across evidence tables 
- scoring/integration.py: Use CTEs with GROUP BY gene_id to deduplicate
  all evidence tables before LEFT JOIN (gnomAD had 211K rows for 18K
  genes due to per-transcript data; annotation/localization also had dupes)
- literature/transform.py: Deduplicate gene_symbols before PubMed queries
  to avoid querying the same symbol multiple times
- evidence_cmd.py: Fix Polars .alias() error in literature summary display
- Updated report results with full 6-layer scoring (44 HIGH, 7268 MEDIUM)
- Validation PASSED: known Usher genes median percentile 86.9%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-16 04:48:13 +08:00
gbanyanandClaude Opus 4.6 63e3cccd3c fix: implement real checkpoint persistence for literature layer 
Previously checkpoints only logged a message but never wrote to DuckDB,
causing all progress to be lost on process kill. Now every batch_size
genes (default 500) are persisted to a literature_partial table.
On restart the CLI loads partial results and resumes from where it
left off. The partial table is cleaned up after successful completion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-13 21:14:16 +08:00
gbanyanandClaude Opus 4.6 dc36730cb4 docs: update gitignore to track report results, add README cross-links 
- Revised .gitignore to ignore raw data/cache but track data/report/
  (candidates TSV/Parquet, plots, reproducibility metadata)
- Added zh↔en cross-links between README.md and README.en.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-13 05:57:27 +08:00
gbanyanandClaude Opus 4.6 674a9ae845 docs: add project README (zh/en) and CLAUDE.md 
Researcher-facing README explaining pipeline rationale, six evidence
layers with scientific basis, scoring methodology, and step-by-step
execution guide for CLI newcomers. CLAUDE.md provides development
context for future Claude Code sessions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-13 04:50:02 +08:00
gbanyanandClaude Opus 4.6 6605ff0f2b fix: resolve runtime bugs for pipeline execution on Python 3.14 + latest deps 
- gene_mapping: wrap mygene fetch_all generator in list() to fix len() error
- gene_mapping: raise MAX_EXPECTED_GENES to 23000 (mygene DB growth)
- setup_cmd: rename gene_universe columns to gene_id/gene_symbol for
  consistency with all downstream evidence layer code
- gnomad: handle missing coverage columns in v4.1 constraint TSV
- expression: fix HPA URL (v23.proteinatlas.org) and GTEx URL (v8 path)
- expression: fix Polars pivot() API change (columns -> on), collect first
- expression: handle missing GTEx tissues (Eye - Retina not in v8)
- expression: ensure all expected columns exist even when sources unavailable
- expression/load: safely check column existence before filtering
- localization: fix HPA subcellular URL to v23
- animal_models: fix httpx stream response.read() before .text access
- animal_models: increase infer_schema_length for HCOP and MGI TSV parsing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-13 03:44:01 +08:00
gbanyanandClaude Opus 4.6 a2ef2125ba chore: complete v1.0 MVP milestone 
Archive v1.0 milestone: 6 phases, 21 plans, 40/40 requirements.
Reorganize ROADMAP.md, evolve PROJECT.md, archive requirements.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-12 21:31:43 +08:00
gbanyan c284804493 docs: milestone v1.0 audit — passed (40/40 requirements) 2026-02-12 10:28:53 +08:00
gbanyan 294b4defbb docs(phase-06): complete phase execution 2026-02-12 04:57:56 +08:00
gbanyanandClaude Opus 4.6 c97592d629 docs(06-03): complete comprehensive validation report & CLI plan 
- Created 06-03-SUMMARY.md documenting plan execution
- Updated STATE.md:
  - Current Position: Phase 6 COMPLETE (21/21 plans)
  - Performance Metrics: Phase 06 3/3 plans, 10 min total, 3.3 min/plan avg
  - Added decisions for comprehensive validation report and weight tuning recommendations
  - Session Continuity: Stopped at 06-03-PLAN.md completion

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-12 04:53:05 +08:00
gbanyan 5879ae9768 test(06-03): unit tests for all validation modules 
- Created test_validation.py with 13 comprehensive tests
- create_synthetic_scored_db helper: designs scores for known genes high, housekeeping low
- Tests for negative controls:
  - test_compile_housekeeping_genes_structure
  - test_compile_housekeeping_genes_known_genes_present
  - test_validate_negative_controls_with_synthetic_data
  - test_validate_negative_controls_inverted_logic
- Tests for recall@k:
  - test_compute_recall_at_k
- Tests for sensitivity/perturbation:
  - test_perturb_weight_renormalizes
  - test_perturb_weight_large_negative
  - test_perturb_weight_invalid_layer
- Tests for validation report:
  - test_generate_comprehensive_validation_report_format
  - test_recommend_weight_tuning_all_pass
  - test_recommend_weight_tuning_positive_fail
  - test_recommend_weight_tuning_negative_fail
  - test_recommend_weight_tuning_sensitivity_fail
- All tests pass using synthetic DuckDB with tmp_path isolation
 2026-02-12 04:50:17 +08:00
gbanyan 10f19f89f4 feat(06-03): comprehensive validation report and CLI validate command 
- Created validation_report.py with comprehensive report generation
  - generate_comprehensive_validation_report: combines positive, negative, sensitivity
  - recommend_weight_tuning: provides targeted weight adjustment recommendations
  - save_validation_report: persists report to file
- Created validate_cmd.py CLI command following score_cmd.py pattern
  - Orchestrates positive controls, negative controls, sensitivity analysis
  - Options: --force, --skip-sensitivity, --output-dir, --top-n
  - Styled output with click.echo patterns
  - Provenance tracking for all validation steps
- Updated main.py to register validate command
- Updated scoring.__init__.py to export validation_report functions
 2026-02-12 04:48:25 +08:00
gbanyan 2d29f43848 docs(06-02): complete sensitivity analysis plan 
- Create SUMMARY.md with implementation details and verification results
- Update STATE.md: progress 100% (20/20 plans), plan 06-02 complete
- Record decisions: perturbation deltas, stability threshold, renormalization
- All tasks completed with 2 commits in 3 minutes
 2026-02-12 04:44:13 +08:00
gbanyan a2d6e97acf docs(06-01): complete negative controls and recall@k validation plan 
Summary:
- Created negative_controls.py with 13 housekeeping genes
- Added recall@k metrics (absolute and percentage thresholds)
- Added per-source breakdown for OMIM vs SYSCILIA
- Updated STATE.md with Phase 6 progress and decisions

Plan Summary: .planning/phases/06-validation/06-01-SUMMARY.md
 2026-02-12 04:43:59 +08:00
gbanyan 0084a67fba feat(06-02): export sensitivity analysis module from scoring package 
- Add sensitivity module imports and exports
- Preserve existing negative_controls exports from Plan 01
- All sensitivity functions and constants now importable from usher_pipeline.scoring
 2026-02-12 04:41:27 +08:00
gbanyan 0f615c0d53 feat(06-01): add recall@k metrics and extended positive control validation 
- compute_recall_at_k() measures recall at absolute (100, 500, 1000, 2000) and percentage (5%, 10%, 20%) thresholds
- validate_positive_controls_extended() combines percentile + recall + per-source metrics
- Per-source breakdown separates OMIM Usher from SYSCILIA SCGS v2 validation
- Export all new functions in __init__.py including negative control imports
 2026-02-12 04:41:00 +08:00
gbanyan a7589d9bf1 feat(06-02): implement sensitivity analysis module with weight perturbation and Spearman correlation 
- Add perturb_weight() function with renormalization to maintain sum=1.0
- Add run_sensitivity_analysis() for parameter sweep across all layers and deltas
- Add summarize_sensitivity() for stability classification
- Add generate_sensitivity_report() for human-readable output
- Default perturbations: ±5% and ±10% with stability threshold 0.85
 2026-02-12 04:40:21 +08:00
gbanyan e488ff2d7a feat(06-01): add negative control validation with housekeeping genes 
- Create negative_controls.py with 13 curated housekeeping genes
- HOUSEKEEPING_GENES_CORE frozenset with source provenance
- compile_housekeeping_genes() returns DataFrame matching known_genes pattern
- validate_negative_controls() uses PERCENT_RANK with inverted threshold logic
- generate_negative_control_report() provides human-readable output
 2026-02-12 04:39:48 +08:00
gbanyan 844295c681 docs(06): create phase plan 2026-02-12 04:33:17 +08:00
gbanyan ca2b715d8e docs(phase-06): research validation domain with controls and sensitivity analysis 2026-02-12 04:27:26 +08:00
gbanyan 964c1cebe8 docs(phase-05): complete phase execution 2026-02-12 04:15:50 +08:00
gbanyan 00e2836eb9 docs(05-03): complete CLI report command plan 
- Create 05-03-SUMMARY.md with comprehensive execution report
- Update STATE.md: Phase 5 complete (3/3 plans), progress 90% (18/20)
- Record decisions: CLI pattern, configurable thresholds, skip flags, graceful degradation
- Document deviations: 2 auto-fixed bugs (tier threshold format, parameter name)
- Update metrics: Phase 05 total 12 min (4.0 min/plan avg)
- All tests pass (9/9 CliRunner integration tests)
 2026-02-12 04:09:26 +08:00
gbanyan c10d59548f test(05-03): add CliRunner integration tests for report command 
- Create test_report_cmd.py with 9 comprehensive tests
- Test fixtures: test_config (minimal YAML), populated_db (synthetic scored_genes data)
- Test coverage: help output, file generation, tier counts, visualizations, skip flags, custom thresholds, error handling, custom output directory
- Synthetic data design: 3 HIGH, 5 MEDIUM, 5 LOW, 4 EXCLUDED, 3 NULL composite_score
- All tests pass with isolated tmp_path DuckDB instances
- Fixed report_cmd.py tier threshold format (uppercase keys: HIGH/MEDIUM/LOW, composite_score field)
- Fixed write_candidate_output parameter name (filename_base not base_filename)
 2026-02-12 04:07:34 +08:00
gbanyan 2ab25ef5c2 feat(05-03): implement CLI report command 
- Create report_cmd.py following established CLI pattern
- Orchestrate full output pipeline: tiering, evidence summary, dual-format output, visualizations, reproducibility reports
- Support --output-dir, --force, --skip-viz, --skip-report flags
- Configurable tier thresholds (--high-threshold, --medium-threshold, --low-threshold, --min-evidence-high, --min-evidence-medium)
- Register report command in main.py CLI entry point
- Follow score_cmd.py pattern: config load, store init, checkpoint check, pipeline steps, summary display, cleanup
- CLI now has 5 commands: setup, evidence, score, report, info
 2026-02-12 04:05:52 +08:00
gbanyan 5f14dc2e64 docs(05-02): complete visualization and reproducibility report plan 
- Plan 05-02 executed successfully
- 2 tasks completed with 2 commits
- 13 tests passing (6 visualization + 7 reproducibility)
- 4 files created, 2 files modified
- Duration: 5 minutes
- Updated STATE.md with progress (17/20 plans complete, 85%)
 2026-02-12 04:03:08 +08:00
gbanyanandClaude Opus 4.6 434c79c0a8 docs(05-01): complete output generation core plan 
- Add 05-01-SUMMARY.md with performance metrics and decisions
- Update STATE.md to Phase 5, Plan 1 of 3 (80% overall progress)
- Record key decisions: configurable tiers, dual-format output, YAML provenance
- Document deviation: pl.count() -> pl.len() deprecation fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-12 04:01:24 +08:00
gbanyan 5af63eab46 feat(05-02): implement reproducibility report module with JSON and Markdown output 
- Create ReproducibilityReport dataclass with all metadata fields
- Implement generate_reproducibility_report function
- Extract parameters from PipelineConfig (scoring weights, data versions)
- Capture software environment (Python, polars, duckdb versions)
- Build filtering steps from ProvenanceTracker
- Compute tier statistics from tiered DataFrame
- Support optional validation metrics
- to_json: write as indented JSON for machine-readable format
- to_markdown: write with tables and headers for human-readable format
- 7 tests covering all report fields, formats, and edge cases
 2026-02-12 04:00:21 +08:00
gbanyan 4e46b488f1 feat(05-01): add dual-format writer with provenance and tests 
- Implement TSV+Parquet writer with deterministic output sorting
- Generate YAML provenance sidecar with statistics and metadata
- Add comprehensive unit tests (9 tests covering all functionality)
- Fix deprecated pl.count() -> pl.len() usage
 2026-02-12 03:59:26 +08:00
gbanyan 150417ffcc feat(05-02): implement visualization module with matplotlib/seaborn plots 
- Add matplotlib>=3.8.0 and seaborn>=0.13.0 to dependencies
- Create visualizations.py with 3 plot functions and orchestrator
- plot_score_distribution: histogram colored by confidence tier
- plot_layer_contributions: bar chart of evidence layer coverage
- plot_tier_breakdown: pie chart of tier distribution
- Use Agg backend for headless/CLI safety
- All plots saved at 300 DPI with proper figure cleanup
- 6 tests covering file creation, edge cases, and return values
 2026-02-12 03:57:50 +08:00
gbanyan d2ef3a2b84 feat(05-01): implement tiering logic and evidence summary module 
- Add confidence tier classification (HIGH/MEDIUM/LOW) based on composite_score and evidence_count
- Add supporting_layers and evidence_gaps columns per gene
- Use vectorized polars expressions for performance
- Configurable thresholds for tier assignment
 2026-02-12 03:56:42 +08:00
gbanyan 6ab7fd1378 docs(05-output-cli): create phase plan 2026-02-11 21:14:37 +08:00
gbanyan 1799906138 docs(05): research phase output-cli domain 2026-02-11 21:07:56 +08:00
gbanyan de678858cd docs(phase-04): complete phase execution 2026-02-11 21:01:50 +08:00
gbanyan 386fbf51b2 docs(04-03): complete CLI score command and tests plan 
- SUMMARY.md: CLI orchestration with checkpoint-restart + 10 comprehensive tests
- STATE.md: Updated position (Phase 4 complete), progress (75%), velocity, decisions
- Duration: 3 minutes, 2 tasks, 4 files (3 created, 1 modified)
 2026-02-11 20:56:31 +08:00
gbanyan a6ad6c6d19 test(04-03): add unit and integration tests for scoring module 
- test_scoring.py: 7 unit tests for known genes, weight validation, NULL preservation
- test_scoring_integration.py: 3 integration tests for end-to-end pipeline with synthetic data
- Tests verify NULL handling (genes with no evidence get NULL composite score)
- Tests verify known genes rank highly when given high scores
- Tests verify QC detects missing data above thresholds
- All tests use synthetic data (no external API calls, fast, reproducible)
 2026-02-11 20:54:39 +08:00
gbanyan d57a5f2826 feat(04-03): add CLI score command with checkpoint-restart 
- Created score_cmd.py following evidence_cmd.py pattern
- Orchestrates full scoring pipeline: known genes -> composite scores -> QC -> validation
- Options: --force, --skip-qc, --skip-validation for flexible iteration
- Registered score command in main CLI group
- Displays comprehensive summary with quality flag distribution
 2026-02-11 20:52:37 +08:00
gbanyan c501951b0f docs(04-02): complete QC and validation plan 
- Add SUMMARY for quality control and positive control validation
- Update STATE.md: Plan 2 of 3 in Phase 04 complete
- Progress: 70% (14/20 plans complete)
- Decisions: scipy MAD outlier detection, PERCENT_RANK validation
 2026-02-11 20:50:00 +08:00
gbanyan 70a5d6eff8 feat(04-02): implement positive control validation 
- Create validation.py with known gene ranking validation
- validate_known_gene_ranking: PERCENT_RANK window function over all genes
- Computes median percentile, top quartile count/fraction for known genes
- generate_validation_report: human-readable text output with formatted table
- Update __init__.py to export run_qc_checks, validate_known_gene_ranking, generate_validation_report
 2026-02-11 20:47:59 +08:00
gbanyan ba2f97ac55 feat(04-02): implement QC checks for scoring results 
- Add scipy>=1.14 dependency for MAD-based outlier detection
- Create quality_control.py with 4 QC functions
- compute_missing_data_rates: NULL rate detection with warn/error thresholds
- compute_distribution_stats: mean/median/std per layer with anomaly detection
- detect_outliers: MAD-based robust outlier detection (>3 MAD)
- run_qc_checks: orchestrator with composite score percentiles
 2026-02-11 20:46:57 +08:00
gbanyan 71c4e8f736 docs(04-01): complete known gene compilation and weighted scoring plan 
- Known genes: 38 (10 OMIM Usher + 28 SYSCILIA SCGS v2 core)
- ScoringWeights.validate_sum() enforcing weight sum = 1.0
- NULL-preserving weighted average (weighted_sum / available_weight)
- Quality flags based on evidence_count thresholds
- Per-layer contributions for explainability
- 2 tasks, 4 files, 4 min duration
 2026-02-11 20:44:09 +08:00
gbanyan f441e8c1ad feat(04-01): implement multi-evidence weighted scoring integration 
- Create join_evidence_layers() with LEFT JOIN preserving NULLs from all 6 evidence tables
- Implement compute_composite_scores() with NULL-preserving weighted average (weighted_sum / available_weight)
- Add quality_flag classification based on evidence_count (sufficient/moderate/sparse/no_evidence)
- Include per-layer contribution columns for explainability
- Add persist_scored_genes() to save scored_genes table to DuckDB
- Log summary stats: coverage, mean/median scores, quality distribution, NULL rates
 2026-02-11 20:41:44 +08:00
gbanyan 0cd2f7c9dd feat(04-01): implement known gene compilation and ScoringWeights validation 
- Create scoring module with OMIM_USHER_GENES (10 genes) and SYSCILIA_SCGS_V2_CORE (28 genes)
- Implement compile_known_genes() returning DataFrame with gene_symbol, source, confidence
- Add load_known_genes_to_duckdb() to persist known genes table
- Add ScoringWeights.validate_sum() method enforcing weight sum constraint (1.0 ± 1e-6)
 2026-02-11 20:41:31 +08:00
gbanyan ed21f18a98 fix(03-05): handle NULL columns and deprecated polars API in animal models 
- Add NULL/empty column checks in fetch_ortholog_mapping
- Fix NULL handling in filter_sensory_phenotypes with is_not_null guard
- Replace deprecated str.concat with str.join
- Add explicit schema to empty DataFrames for consistency
 2026-02-11 20:38:36 +08:00
gbanyan a52724aff4 docs(04): create phase plan for scoring and integration 2026-02-11 20:31:55 +08:00
gbanyan 32988c631f docs(04): research multi-evidence weighted scoring with NULL preservation 2026-02-11 20:24:42 +08:00
gbanyan 190bedaa80 docs(phase-03): complete phase execution 2026-02-11 19:18:12 +08:00
gbanyan e72c516669 docs(03-06): complete literature evidence layer 
- Created SUMMARY.md with full implementation details
- Updated STATE.md: progress 60%, 12/20 plans complete, Phase 3 complete
- Documented 4 key decisions (tier priority, bias mitigation, context weights, rate limiting)
- All verification criteria met: 17/17 tests pass, CLI functional, bias mitigation validated
- Self-check PASSED: all files and commits verified

Key accomplishments:
- PubMed evidence layer queries per gene across cilia/sensory/cytoskeleton/polarity contexts
- Quality tier classification: direct_experimental > hts_hit > functional_mention > incidental
- Bias mitigation via log2(total_pubmed_count) prevents well-studied gene dominance
- Novel genes with 10 total/5 cilia publications score higher than TP53-like genes with 100K total/5 cilia
- Biopython Entrez integration with rate limiting (3/sec default, 10/sec with API key)
 2026-02-11 19:13:26 +08:00
gbanyan 0e89bf0dd6 docs(03-02): complete expression evidence layer plan 
- Create 03-02-SUMMARY.md with performance metrics, decisions, and deviations
- Update STATE.md: 5 of 6 plans complete in Phase 03 (03-06 remaining)
- Update progress: 55% complete (11/20 plans across all phases)
- Add key decisions: Tau calculation, expression scoring, CellxGene optional
- Record duration: 12 min for 2 tasks (9 files modified)
- Self-check passed: all files and commits verified

Expression layer provides:
- HPA/GTEx tissue expression with Tau specificity index
- Usher-tissue enrichment scoring (retina, inner ear, cilia)
- Optional CellxGene single-cell integration
- CLI command with checkpoint-restart
- 11 passing unit and integration tests
 2026-02-11 19:12:18 +08:00
gbanyan cfe4b830e6 docs(03-03): complete protein features plan with SUMMARY and STATE updates 2026-02-11 19:10:03 +08:00
gbanyan 053f0d926b docs(03-05): complete animal model phenotype evidence layer plan 
- SUMMARY.md: Ortholog-mapped animal evidence from MGI/ZFIN/IMPC
- Confidence-weighted scoring (mouse +0.4, zebrafish +0.3, IMPC +0.3)
- 14/14 tests passing: ortholog confidence, keyword filtering, NULL preservation
- Deviations: Schema mismatches, NULL handling, polars deprecations auto-fixed
- Duration: 10 minutes, 2 tasks, 8 files, 2 commits
 2026-02-11 19:08:45 +08:00
gbanyan d8009f1236 docs(03-04): complete subcellular localization evidence layer 
- Created SUMMARY.md with full implementation details
- Updated STATE.md: progress 40%, 8/20 plans complete
- Documented 4 key decisions (evidence terminology, NULL semantics, embedded proteomics, evidence weighting)
- All verification criteria met: 17/17 tests pass, CLI functional, DuckDB integration complete
 2026-02-11 19:08:01 +08:00
gbanyan 46059874f2 feat(03-03): implement protein evidence layer with UniProt/InterPro integration 
- Create protein features data model with domain, coiled-coil, TM, cilia motifs
- Implement fetch.py with UniProt REST API and InterPro API queries
- Implement transform.py with feature extraction, motif detection, normalization
- Implement load.py with DuckDB persistence and provenance tracking
- Add CLI protein command following evidence layer pattern
- Add comprehensive unit and integration tests (all passing)
- Handle NULL preservation and List(Null) edge case
- Add get_steps() method to ProvenanceTracker for test compatibility
 2026-02-11 19:07:30 +08:00
gbanyan bcd3c4ffbe feat(03-05): add animal model DuckDB loader, CLI, and comprehensive tests 
- load.py: DuckDB persistence with provenance tracking, ortholog confidence distribution stats
- CLI animal-models command: checkpoint-restart pattern, top scoring genes display
- 10 unit tests: ortholog confidence scoring, keyword filtering, multi-organism bonus, NULL preservation
- 4 integration tests: full pipeline, checkpoint-restart, provenance tracking, empty phenotype handling
- All tests pass (14/14): validates fetch->transform->load->CLI flow
- Fixed polars deprecations: str.join replaces str.concat, pl.len replaces pl.count
 2026-02-11 19:06:49 +08:00
gbanyan 99bc975a2c docs(03-01): complete annotation completeness plan 2026-02-11 19:05:56 +08:00
gbanyan 942aaf2ec3 feat(03-04): add localization CLI command and comprehensive tests 
- Add localization subcommand to evidence command group
- Implement checkpoint-restart pattern for HPA download
- Display summary with evidence type distribution
- Create 17 unit and integration tests (all pass)
- Test HPA parsing, evidence classification, scoring, and DuckDB persistence
- Fix evidence type terminology (computational vs predicted) for consistency
- Mock HTTP calls in integration tests for reproducibility
 2026-02-11 19:05:22 +08:00
gbanyan d70239c4ce feat(03-01): add annotation DuckDB loader, CLI command, and tests 
- Create load_to_duckdb with provenance tracking and tier distribution stats
- Add query_poorly_annotated helper to find under-studied genes
- Register `evidence annotation` CLI command with checkpoint-restart pattern
- Add comprehensive unit tests (9 tests) covering GO extraction, NULL handling, tier classification, score normalization, weighting
- Add integration tests (6 tests) for pipeline, idempotency, checkpoint-restart, provenance, queries
- All 15 tests pass with proper NULL preservation and schema validation
 2026-02-11 19:03:10 +08:00
gbanyan 0e389c7e41 feat(03-05): implement animal model evidence fetch and transform 
- models.py: AnimalModelRecord with ortholog confidence, phenotype flags, and normalized scoring
- fetch.py: Retrieve orthologs from HCOP, phenotypes from MGI/ZFIN/IMPC with retry
- transform.py: Filter sensory/cilia-relevant phenotypes, score with confidence weighting
- Ortholog confidence: HIGH (8+ sources), MEDIUM (4-7), LOW (1-3)
- Scoring: mouse +0.4, zebrafish +0.3, IMPC +0.3, weighted by confidence
- NULL preservation: no ortholog = NULL score (not zero)
 2026-02-11 19:00:24 +08:00
gbanyan 8aa66987f8 feat(03-06): implement literature evidence models, PubMed fetch, and scoring 
- Create LiteratureRecord pydantic model with context-specific counts
- Implement PubMed query via Biopython Entrez with rate limiting (3/sec default, 10/sec with API key)
- Define SEARCH_CONTEXTS for cilia, sensory, cytoskeleton, cell_polarity queries
- Implement evidence tier classification: direct_experimental > functional_mention > hts_hit > incidental > none
- Implement quality-weighted scoring with bias mitigation via log2(total_pubmed_count) normalization
- Add biopython>=1.84 dependency to pyproject.toml
- Support checkpoint-restart for long-running PubMed queries (estimated 3-11 hours for 20K genes)
 2026-02-11 19:00:20 +08:00
gbanyan 6645c59b0b feat(03-04): create localization evidence data model and processing 
- Define LocalizationRecord model with HPA and proteomics fields
- Implement fetch_hpa_subcellular to download HPA bulk data
- Implement fetch_cilia_proteomics with curated reference gene sets
- Implement classify_evidence_type (experimental vs computational)
- Implement score_localization with cilia proximity scoring
- Implement process_localization_evidence end-to-end pipeline
- Create load_to_duckdb for persistence with provenance
 2026-02-11 19:00:09 +08:00
gbanyan adbb74b965 feat(03-01): implement annotation evidence fetch and transform modules 
- Create AnnotationRecord model with GO counts, UniProt scores, tier classification
- Implement fetch_go_annotations using mygene.info batch queries
- Implement fetch_uniprot_scores using UniProt REST API
- Add classify_annotation_tier with 3-tier system (well/partial/poor)
- Add normalize_annotation_score with weighted composite (GO 50%, UniProt 30%, Pathway 20%)
- Implement process_annotation_evidence end-to-end pipeline
- Follow NULL preservation pattern from gnomAD (unknown != zero)
- Use lazy polars evaluation where applicable
 2026-02-11 18:58:45 +08:00
gbanyan 0d252da348 docs(03): create phase plan 2026-02-11 18:46:28 +08:00
gbanyan 3354cfe006 docs(phase-03): research core evidence layers domain 2026-02-11 18:37:14 +08:00
gbanyan ffb4963d2b docs(phase-02): complete phase execution 2026-02-11 18:28:13 +08:00
gbanyan a0388cf4e1 docs(02-02): complete gnomAD evidence layer integration plan 
- DuckDB persistence: gnomad_constraint table with CREATE OR REPLACE (idempotent)
- CLI evidence command: usher-pipeline evidence gnomad with checkpoint-restart
- Provenance tracking: records processing steps, saves sidecar JSON
- Query helpers: query_constrained_genes validates GCON-03 interpretation
- 12 integration tests: end-to-end pipeline, checkpoint, provenance, CLI
- Phase 2 complete: Evidence layer pattern established for future sources
- Duration: 4 min, 2 tasks, 5 files, 70 tests passing

Phase 2 (Prototype Evidence Layer) complete.
 2026-02-11 18:23:32 +08:00
gbanyan 56e04e68c2 test(02-02): add comprehensive gnomAD integration tests 
- 12 integration tests covering full pipeline: fetch->transform->load->query
- test_full_pipeline_to_duckdb: End-to-end pipeline verification with DuckDB storage
- test_checkpoint_restart_skips_processing: Checkpoint detection works correctly
- test_provenance_recorded: Provenance step records expected details
- test_provenance_sidecar_created: JSON sidecar file creation and structure
- test_query_constrained_genes_filters_correctly: Query returns only measured genes below threshold
- test_null_loeuf_not_in_constrained_results: NULL LOEUF genes excluded from queries
- test_duckdb_schema_has_quality_flag: Schema includes quality_flag with valid values
- test_normalized_scores_in_duckdb: Normalized scores in [0,1] for measured genes, NULL for others
- test_cli_evidence_gnomad_help: CLI help text displays correctly
- test_cli_evidence_gnomad_with_mock: CLI command runs end-to-end with mocked download
- test_idempotent_load_replaces_table: Loading twice replaces table (not appends)
- test_quality_flag_categorization: Quality flags correctly categorize genes

All 70 tests pass (58 existing + 12 new), no regressions
 2026-02-11 18:20:59 +08:00
gbanyan ee27f3ad2f feat(02-02): add DuckDB loader and CLI evidence command for gnomAD 
- load_to_duckdb: Saves constraint DataFrame to gnomad_constraint table with provenance tracking
- query_constrained_genes: Queries constrained genes by LOEUF threshold (validates GCON-03 interpretation)
- evidence_cmd.py: CLI command group with gnomad subcommand (fetch->transform->load orchestration)
- Checkpoint-restart: Skips processing if gnomad_constraint table exists (--force to override)
- Full CLI: usher-pipeline evidence gnomad [--force] [--url URL] [--min-depth N] [--min-cds-pct N]
 2026-02-11 18:19:07 +08:00
gbanyan c6198122ac docs(02-01): complete gnomAD constraint data pipeline plan 2026-02-11 18:16:35 +08:00
gbanyan 174c4af02d feat(02-01): add gnomAD transform pipeline and comprehensive tests 
- Implement filter_by_coverage with quality_flag categorization (measured/incomplete_coverage/no_data)
- Add normalize_scores with LOEUF inversion (lower LOEUF = higher score)
- NULL preservation throughout pipeline (unknown != zero constraint)
- process_gnomad_constraint end-to-end pipeline function
- 15 comprehensive unit tests covering edge cases:
  - NULL handling and preservation
  - Coverage filtering without dropping genes
  - Normalization bounds and inversion
  - Mixed type handling for robust parsing
- Fix column mapping to handle gnomAD v4.x loeuf/loeuf_upper duplication
- All existing tests continue to pass
 2026-02-11 18:14:41 +08:00
gbanyan a88b0eea60 feat(02-01): add gnomAD constraint data models and download module 
- Create evidence layer package structure
- Define ConstraintRecord Pydantic model with NULL preservation
- Implement streaming download with httpx and tenacity retry
- Add lazy TSV parser with column name variant handling
- Add httpx and structlog dependencies
 2026-02-11 18:11:49 +08:00
gbanyan c7753e7b1c docs(02): create phase plan 2026-02-11 17:47:23 +08:00
gbanyan d328467737 docs(phase-02): research prototype evidence layer 2026-02-11 17:41:35 +08:00
gbanyanandClaude Opus 4.6 34437fdf0a docs(phase-01): complete phase execution 
Phase 1 (Data Infrastructure) verified: 5/5 must-haves, 12/12 artifacts,
9/9 key links, 7/7 requirements satisfied. All 4 plans executed across
3 waves with 49 tests passing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-11 16:50:30 +08:00
gbanyan 102dcdbe84 docs(01-04): complete CLI integration and end-to-end testing plan 
- CLI entry point with setup and info commands
- Full infrastructure integration verified
- 6 integration tests with mocked APIs
- Phase 01 Data Infrastructure complete
 2026-02-11 16:45:12 +08:00
gbanyan e4d71d0790 test(01-04): add integration tests verifying module wiring 
- test_config_to_store_roundtrip: config -> PipelineStore -> save/load
- test_config_to_provenance: config -> ProvenanceTracker -> sidecar
- test_full_setup_flow_mocked: full setup with mocked mygene (fetch, map, validate, save, provenance)
- test_checkpoint_skip_flow: verify checkpoint-restart skips re-fetch
- test_setup_cli_help: CLI help output verification
- test_info_cli: info command with config display

All tests pass with mocked API calls (no external dependencies).
 2026-02-11 16:42:13 +08:00
gbanyan f33b048635 feat(01-04): add CLI entry point with setup and info commands 
- Create click-based CLI with command group (--config, --verbose options)
- Add 'info' command displaying pipeline version, config hash, data source versions
- Add 'setup' command orchestrating full infrastructure flow:
  - Load config -> create store/provenance
  - Fetch gene universe (with checkpoint-restart)
  - Map Ensembl IDs to HGNC + UniProt
  - Validate mapping quality gates
  - Save to DuckDB with provenance sidecar
- Update pyproject.toml entry point to usher_pipeline.cli.main:cli
- Add .gitignore for data/, *.duckdb, build artifacts, provenance files
 2026-02-11 16:39:50 +08:00
gbanyan e29d39d1dc docs(01-02): complete gene ID mapping and validation plan 
- Gene universe definition with mygene protein-coding gene retrieval
- Batch Ensembl->HGNC+UniProt mapping with edge case handling
- Validation gates with configurable success rate thresholds
- 15 comprehensive tests with mocked API responses
 2026-02-11 16:35:57 +08:00
gbanyanandClaude Opus 4.6 92322b1d7c docs(01-03): complete DuckDB persistence and provenance tracking plan 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-11 16:34:00 +08:00
gbanyan 0200395d9e feat(01-02): create mapping validation gates with tests 
- Add MappingValidator with configurable success rate thresholds (min_success_rate, warn_threshold)
- Add validate_gene_universe for gene count, format, and duplicate checks
- Add save_unmapped_report for manual review output
- Implement 15 comprehensive tests with mocked mygene responses (no real API calls)
- Tests cover: successful mapping, notfound handling, uniprot list parsing, batching, validation gates, universe validation
 2026-02-11 16:33:36 +08:00
gbanyan 98a1a750dd feat(01-03): create provenance tracker with comprehensive tests 
- ProvenanceTracker class for metadata tracking
- Records pipeline version, data source versions, config hash, timestamps
- Sidecar JSON export alongside outputs
- DuckDB _provenance table support
- 13 comprehensive tests (8 DuckDB + 5 provenance)
- All tests pass (12 passed, 1 skipped - pandas)
 2026-02-11 16:31:51 +08:00
gbanyan d51141f7d5 feat(01-03): create DuckDB persistence layer with checkpoint-restart 
- PipelineStore class for DuckDB-based storage
- save_dataframe/load_dataframe for polars and pandas
- Checkpoint system with has_checkpoint and metadata tracking
- Parquet export capability
- Context manager support
 2026-02-11 16:30:25 +08:00
gbanyan 9ee3ec2e84 docs(01-01): complete project scaffold and config system plan 
- Created comprehensive SUMMARY.md with all execution details
- Updated STATE.md: 1/4 plans in phase 1 complete, 16.7% overall progress
- Documented deviation (venv creation) and decisions
- Verified all files and commits exist (self-check passed)
 2026-02-11 16:28:03 +08:00
gbanyan 4204116772 feat(01-01): create base API client with retry and caching 
- CachedAPIClient with SQLite persistent cache
- Exponential backoff retry on 429/5xx/network errors (tenacity)
- Rate limiting with skip for cached responses
- from_config classmethod for pipeline integration
- 5 passing tests for cache creation, rate limiting, and config integration
 2026-02-11 16:25:46 +08:00
gbanyan 4a80a0398e feat(01-01): create Python package scaffold with config system 
- pyproject.toml: installable package with bioinformatics dependencies
- Pydantic config schema with validation (ensembl_release >= 100, directory creation)
- YAML config loader with override support
- Default config with Ensembl 113, gnomAD v4.1
- 5 passing tests for config validation and hashing
 2026-02-11 16:24:35 +08:00
gbanyan cab2f5fc66 docs(01-data-infrastructure): create phase plan 2026-02-11 16:04:42 +08:00
gbanyan 982f7f5a9b docs(01-data-infrastructure): research phase domain 2026-02-11 15:56:40 +08:00
gbanyan f80f384a61 docs: create roadmap (6 phases) 2026-02-11 15:47:36 +08:00
gbanyan 0fb1a9581f docs: define v1 requirements 2026-02-11 15:31:05 +08:00
gbanyan bb7bfaedab docs: complete project research 2026-02-11 14:52:06 +08:00
gbanyan c0abe8bc6c chore: add project config 2026-02-11 14:41:35 +08:00
gbanyan e2c202d689 docs: initialize project 2026-02-11 14:40:36 +08:00