Data-quality roadmap: R-A–R-F + DQ-5 fixes (citations 590→1075, section signal 10%→95%) #16
@@ -396,7 +396,33 @@ is self-contained so a cold session can pick it up.
|
||||
- **Acceptance:** Crossref recovers refs and/or an abstract for ≥1 paper where
|
||||
OpenAlex+S2 are empty.
|
||||
|
||||
### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **MED lever, MED-HIGH effort**
|
||||
### R-C — Prefer arXiv `.tex` source over `.txt` for chunk fidelity · **DONE 2026-06-17**
|
||||
|
||||
**Resolution (what was done) — all 13 arXiv papers re-ingested from `.tex`:**
|
||||
- **Multi-file flatten:** `arxiv.fetch_source` now inlines `\input`/`\include`
|
||||
(`tex.flatten_inputs`) so multi-file projects yield the full body, not the
|
||||
primary file's include skeleton (e.g. math/0603097 = 15 files / 12 includes;
|
||||
2305.10988 = 6 / 5). Also handles legacy single-file **bare-gzipped** `.tex`
|
||||
(no tar wrapper) for old math/* papers (commit 121b582).
|
||||
- **Section-aware chunking:** new `tex.chunk_sections` returns `(title, chunk)`
|
||||
pairs (chunks never span a `\section`); the ingest `.tex` path labels the stored
|
||||
`section` from the real heading instead of running `classify_section` on a
|
||||
header-less word-window. Quality-gated as pairs so labels stay aligned.
|
||||
- **Unblocked by the arXiv half of DQ-5** (commit f504ca6): `_canonical_id`
|
||||
strips the `10.48550/arxiv.` DOI prefix so an arXiv paper's id is the bare
|
||||
arXiv id — without it, re-ingest tripped `papers_openalex_id_key` (confirmed
|
||||
live, then fixed; idempotent re-ingest verified).
|
||||
- **Outcome:** `section` column gained signal — ~34/329 arXiv chunks now
|
||||
intro/theorem/proof (was ~all `body`); math/0603097 `{body:31}`→`{body:29,
|
||||
proof:9}`, 2310.17529 gained proof+intro, etc. Math papers with descriptively
|
||||
titled sections stay `body` (classify_section's fixed vocab) — modest but real
|
||||
(R-12). No paper gutted (chunk counts stable/▲); F-16 gate unchanged.
|
||||
- **Files:** `codex/parsing/tex.py`, `codex/sources/arxiv.py`, `codex/ingest.py`,
|
||||
`codex/sources/openalex.py`, `scripts/rc_tex_reingest.py` (driver, dry-run
|
||||
default), + tests. Full suite **370 passed**; ruff/mypy clean. Branch
|
||||
`feat/tex-ingest` (commits 1da81c7, 121b582, f504ca6).
|
||||
|
||||
**Original plan (for context):**
|
||||
- **What:** re-ingest from arXiv LaTeX source through the existing
|
||||
`codex.parsing.tex.latex_to_text` path instead of flattened `.txt`.
|
||||
- **Why:** highest-fidelity input — preserves math, structure, and section
|
||||
|
||||
Reference in New Issue
Block a user