feat(tex): section-aware multi-file .tex ingest (R-C)

Flatten multi-file arXiv LaTeX (\input/\include) in arxiv.fetch_source so the full body is assembled rather than just the primary file's include skeleton, and add section-aware chunking (tex.chunk_sections) so .tex chunks never span a \section boundary and are labelled by their real heading. The ingest .tex path now produces (section, chunk) pairs, quality-gated together so labels stay aligned; the stored 'section' column gains genuine signal instead of mostly 'body' (serves DQ-3 fidelity + audit R-12).

Adds scripts/rc_tex_reingest.py to re-ingest arXiv papers from .tex (dry-run by default; replaces chunks). Tests: flatten_inputs, chunk_sections, multi-file fetch_source, section-label ingest, is_arxiv_id. Full suite 365 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-18 04:15:20 +02:00
parent 1516684bbb
commit 1da81c7b28
8 changed files with 356 additions and 42 deletions

View File

@@ -0,0 +1,19 @@
"""Tests for scripts.rc_tex_reingest."""
from __future__ import annotations
from scripts.rc_tex_reingest import is_arxiv_id
def test_is_arxiv_id_accepts_modern_and_legacy() -> None:
assert is_arxiv_id("2305.10988")
assert is_arxiv_id("math/0603097")
assert is_arxiv_id("1005.2698")
def test_is_arxiv_id_rejects_dois_and_wids() -> None:
assert not is_arxiv_id("10.1007/s00454-019-00132-8")
assert not is_arxiv_id("10.14279/depositonce-20357")
assert not is_arxiv_id("W2971636899")
assert not is_arxiv_id("https://openalex.org/W2971636899")
assert not is_arxiv_id("")