diff --git a/docs/audit/DATA-QUALITY-2026-06-15.md b/docs/audit/DATA-QUALITY-2026-06-15.md index 5719674..e15ec46 100644 --- a/docs/audit/DATA-QUALITY-2026-06-15.md +++ b/docs/audit/DATA-QUALITY-2026-06-15.md @@ -396,7 +396,33 @@ is self-contained so a cold session can pick it up. - **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where OpenAlex+S2 are empty. -### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **MED lever, MED-HIGH effort** +### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **DONE 2026-06-17** + +**Resolution (what was done) — all 13 arXiv papers re-ingested from `.tex`:** +- **Multi-file flatten:** `arxiv.fetch_source` now inlines `\input`/`\include` + (`tex.flatten_inputs`) so multi-file projects yield the full body, not the + primary file's include skeleton (e.g. math/0603097 = 15 files / 12 includes; + 2305.10988 = 6 / 5). Also handles legacy single-file **bare-gzipped** `.tex` + (no tar wrapper) for old math/* papers (commit 121b582). +- **Section-aware chunking:** new `tex.chunk_sections` returns `(title, chunk)` + pairs (chunks never span a `\section`); the ingest `.tex` path labels the stored + `section` from the real heading instead of running `classify_section` on a + header-less word-window. Quality-gated as pairs so labels stay aligned. +- **Unblocked by the arXiv half of DQ-5** (commit f504ca6): `_canonical_id` + strips the `10.48550/arxiv.` DOI prefix so an arXiv paper's id is the bare + arXiv id — without it, re-ingest tripped `papers_openalex_id_key` (confirmed + live, then fixed; idempotent re-ingest verified). +- **Outcome:** `section` column gained signal — ~34/329 arXiv chunks now + intro/theorem/proof (was ~all `body`); math/0603097 `{body:31}`→`{body:29, + proof:9}`, 2310.17529 gained proof+intro, etc. Math papers with descriptively + titled sections stay `body` (classify_section's fixed vocab) — modest but real + (R-12). No paper gutted (chunk counts stable/▲); F-16 gate unchanged. +- **Files:** `codex/parsing/tex.py`, `codex/sources/arxiv.py`, `codex/ingest.py`, + `codex/sources/openalex.py`, `scripts/rc_tex_reingest.py` (driver, dry-run + default), + tests. Full suite **370 passed**; ruff/mypy clean. Branch + `feat/tex-ingest` (commits 1da81c7, 121b582, f504ca6). + +**Original plan (for context):** - **What:** re-ingest from arXiv LaTeX source through the existing `codex.parsing.tex.latex_to_text` path instead of flattened `.txt`. - **Why:** highest-fidelity input — preserves math, structure, and section