feat(quality): store real section titles for .tex chunks (R-F)
Add quality.section_label(title, content): for section-aware .tex ingest, keep the controlled bucket (intro/theorem/proof/abstract/bibliography) when the real \section heading maps to one, else store the cleaned title verbatim (e.g. 'preliminaries', 'rigidity') instead of collapsing to 'body'. Math papers title most sections descriptively, so R-C's vocab-only mapping left ~90% as 'body'; this lifts the section signal toward near-full. .pdf/.txt/run_quality_pass keep content-based classify_section unchanged. Safe because the section column is write-only (no consumer filters on the vocab). New _clean_title (strip LaTeX/numbering, lower-case, truncate). Tests: section_label bucket/verbatim/fallback paths + _clean_title; .tex ingest stores a descriptive heading as its title. Full suite 377 passed; ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -110,6 +110,42 @@ def classify_section(text: str) -> str:
|
||||
return "body"
|
||||
|
||||
|
||||
def _clean_title(title: str) -> str:
|
||||
"""Normalize a LaTeX ``\\section`` title for use as a section label.
|
||||
|
||||
Strips LaTeX commands (``\\emph`` …) and ``{}$``, drops leading section
|
||||
numbering (``3.2 ``), collapses whitespace, lower-cases, and truncates to
|
||||
60 chars so the stored ``section`` value is a clean, comparable string.
|
||||
"""
|
||||
t = re.sub(r"\\[a-zA-Z]+\*?", " ", title) # LaTeX commands
|
||||
t = re.sub(r"[{}$]", "", t) # braces / math delimiters
|
||||
t = re.sub(r"^\s*\d+(?:\.\d+)*\.?\s*", "", t) # leading numbering "3.2 "
|
||||
return " ".join(t.split()).strip().lower()[:60]
|
||||
|
||||
|
||||
def section_label(title: str | None, content: str) -> str:
|
||||
"""Label a chunk's section, preferring the real ``\\section`` heading (R-F).
|
||||
|
||||
For section-aware ``.tex`` ingest *title* is the real heading: if it maps to a
|
||||
controlled bucket via :func:`classify_section` (intro / theorem / proof /
|
||||
abstract / bibliography) that bucket is kept (cross-source consistency);
|
||||
otherwise the cleaned title itself is stored (e.g. ``"preliminaries"``,
|
||||
``"rigidity"``) — far more signal than collapsing everything to ``body``.
|
||||
|
||||
When *title* is None / blank (the ``.pdf`` / ``.txt`` / ``run_quality_pass``
|
||||
paths) this falls back to content-based :func:`classify_section` — unchanged
|
||||
behaviour. Safe to store free-text titles because the ``section`` column is
|
||||
write-only (no consumer filters on the controlled vocabulary).
|
||||
"""
|
||||
if not title or not title.strip():
|
||||
return classify_section(content)
|
||||
cleaned = _clean_title(title)
|
||||
if not cleaned:
|
||||
return classify_section(content)
|
||||
bucket = classify_section(cleaned)
|
||||
return bucket if bucket != "body" else cleaned
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Batch filter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user