feat(quality): store real section titles for .tex chunks (R-F)

Add quality.section_label(title, content): for section-aware .tex ingest, keep the controlled bucket (intro/theorem/proof/abstract/bibliography) when the real \section heading maps to one, else store the cleaned title verbatim (e.g. 'preliminaries', 'rigidity') instead of collapsing to 'body'. Math papers title most sections descriptively, so R-C's vocab-only mapping left ~90% as 'body'; this lifts the section signal toward near-full. .pdf/.txt/run_quality_pass keep content-based classify_section unchanged. Safe because the section column is write-only (no consumer filters on the vocab).

New _clean_title (strip LaTeX/numbering, lower-case, truncate). Tests: section_label bucket/verbatim/fallback paths + _clean_title; .tex ingest stores a descriptive heading as its title. Full suite 377 passed; ruff + mypy clean.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-18 09:03:46 +02:00
parent c22c0db3d0
commit c70ff9aa1e
4 changed files with 82 additions and 7 deletions

View File

@@ -110,6 +110,42 @@ def classify_section(text: str) -> str:
return "body"
def _clean_title(title: str) -> str:
"""Normalize a LaTeX ``\\section`` title for use as a section label.
Strips LaTeX commands (``\\emph`` …) and ``{}$``, drops leading section
numbering (``3.2 ``), collapses whitespace, lower-cases, and truncates to
60 chars so the stored ``section`` value is a clean, comparable string.
"""
t = re.sub(r"\\[a-zA-Z]+\*?", " ", title) # LaTeX commands
t = re.sub(r"[{}$]", "", t) # braces / math delimiters
t = re.sub(r"^\s*\d+(?:\.\d+)*\.?\s*", "", t) # leading numbering "3.2 "
return " ".join(t.split()).strip().lower()[:60]
def section_label(title: str | None, content: str) -> str:
"""Label a chunk's section, preferring the real ``\\section`` heading (R-F).
For section-aware ``.tex`` ingest *title* is the real heading: if it maps to a
controlled bucket via :func:`classify_section` (intro / theorem / proof /
abstract / bibliography) that bucket is kept (cross-source consistency);
otherwise the cleaned title itself is stored (e.g. ``"preliminaries"``,
``"rigidity"``) — far more signal than collapsing everything to ``body``.
When *title* is None / blank (the ``.pdf`` / ``.txt`` / ``run_quality_pass``
paths) this falls back to content-based :func:`classify_section` — unchanged
behaviour. Safe to store free-text titles because the ``section`` column is
write-only (no consumer filters on the controlled vocabulary).
"""
if not title or not title.strip():
return classify_section(content)
cleaned = _clean_title(title)
if not cleaned:
return classify_section(content)
bucket = classify_section(cleaned)
return bucket if bucket != "body" else cleaned
# ---------------------------------------------------------------------------
# Batch filter
# ---------------------------------------------------------------------------