docs(audit): mark R-C done (section-aware .tex ingest + DQ-5 arXiv half)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-18 07:47:43 +02:00
parent f504ca62db
commit c22c0db3d0

View File

@@ -396,7 +396,33 @@ is self-contained so a cold session can pick it up.
- **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where - **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where
OpenAlex+S2 are empty. OpenAlex+S2 are empty.
### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **MED lever, MED-HIGH effort** ### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **DONE 2026-06-17**
**Resolution (what was done) — all 13 arXiv papers re-ingested from `.tex`:**
- **Multi-file flatten:** `arxiv.fetch_source` now inlines `\input`/`\include`
(`tex.flatten_inputs`) so multi-file projects yield the full body, not the
primary file's include skeleton (e.g. math/0603097 = 15 files / 12 includes;
2305.10988 = 6 / 5). Also handles legacy single-file **bare-gzipped** `.tex`
(no tar wrapper) for old math/* papers (commit 121b582).
- **Section-aware chunking:** new `tex.chunk_sections` returns `(title, chunk)`
pairs (chunks never span a `\section`); the ingest `.tex` path labels the stored
`section` from the real heading instead of running `classify_section` on a
header-less word-window. Quality-gated as pairs so labels stay aligned.
- **Unblocked by the arXiv half of DQ-5** (commit f504ca6): `_canonical_id`
strips the `10.48550/arxiv.` DOI prefix so an arXiv paper's id is the bare
arXiv id — without it, re-ingest tripped `papers_openalex_id_key` (confirmed
live, then fixed; idempotent re-ingest verified).
- **Outcome:** `section` column gained signal — ~34/329 arXiv chunks now
intro/theorem/proof (was ~all `body`); math/0603097 `{body:31}`→`{body:29,
proof:9}`, 2310.17529 gained proof+intro, etc. Math papers with descriptively
titled sections stay `body` (classify_section's fixed vocab) — modest but real
(R-12). No paper gutted (chunk counts stable/▲); F-16 gate unchanged.
- **Files:** `codex/parsing/tex.py`, `codex/sources/arxiv.py`, `codex/ingest.py`,
`codex/sources/openalex.py`, `scripts/rc_tex_reingest.py` (driver, dry-run
default), + tests. Full suite **370 passed**; ruff/mypy clean. Branch
`feat/tex-ingest` (commits 1da81c7, 121b582, f504ca6).
**Original plan (for context):**
- **What:** re-ingest from arXiv LaTeX source through the existing - **What:** re-ingest from arXiv LaTeX source through the existing
`codex.parsing.tex.latex_to_text` path instead of flattened `.txt`. `codex.parsing.tex.latex_to_text` path instead of flattened `.txt`.
- **Why:** highest-fidelity input — preserves math, structure, and section - **Why:** highest-fidelity input — preserves math, structure, and section