Data-quality roadmap: R-A–R-F + DQ-5 fixes (citations 590→1075, section signal 10%→95%) #16
Reference in New Issue
Block a user
No description provided.
Delete Branch "integration/roadmap"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Consolidates the data-quality roadmap (developed off
audit/loop-1on five feature branches) and reconciles it with the audit-remediation work already onmain(#12–15).What's included
.texingest (\input/\includeflattening + per-\sectionchunking) + legacy bare-gzip source handling.graph reportwarns on low citing-paper coverage, not just paper count.\sectiontitle (hybrid with the controlled vocab); section signal 10% → 95% non-body.openalex._canonical_id).Reconciliation with main (#12–15)
Merged
origin/mainin and resolved 4 conflicts:cli.py— R-D citing-coverage warning and main's small-corpus JSON / hub-title changes (both kept).ingest.py— C-7 (paper.id= caller's bare id) pinned first, then DQ-2 abstract recovery.test_ingest.py/test_openalex.py— both branches' tests kept; fixed an auto-merged contradictorypaper.idassertion.papers.id): kept both layers —openalex._canonical_idnormalisesfetch_paper's id (DQ-5); ingest pins to the caller's bare id (C-7); they converge on the bare canonical id.Corpus impact (live Jetson DB)
body.Validation
main.Full record:
docs/audit/DATA-QUALITY-2026-06-15.md(DQ-1..DQ-5 + R-A..R-F resolution blocks) anddocs/SESSION-2026-06-17-open-items.md.🤖 Generated with Claude Code
search paper(DQ-4) 553ae3995d12/29 papers had zero citations: OpenAlex indexes arXiv preprints and theses without a parsed reference list (verified referenced_works_count=0 server-side for all 11 with an openalex_id). Not an ingest bug — a source-coverage limit. - ingest.py: when OpenAlex returns no references, fall back to a Semantic Scholar reference supplement (_s2_reference_supplement); citing_id rewritten to canonical papers.id, DOI cited-ids lowercased. Removed a now-redundant discarded S2 probe in the OpenAlex-404 arXiv branch. - semanticscholar.py: fix TypeError on S2's HTTP-200 {"data": null} no-refs responses (.get("data", []) returns None when the key is present-but-null). - tests: regression test for the OpenAlex-empty -> S2 path. - Live DB backfilled idempotently: +330 citations (590->920), zero-out-edge papers 12->2, citing coverage 17->27/29. Only the 2 TU-Berlin depositonce theses remain (no references in any source). - docs: DATA-QUALITY-2026-06-15.md — DQ-1 resolution, DQ-4 result, and a free-source acquisition roadmap (R-A..R-E). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>