Data-quality roadmap: R-A–R-F + DQ-5 fixes (citations 590→1075, section signal 10%→95%) #16
@@ -454,6 +454,31 @@ is self-contained so a cold session can pick it up.
|
||||
violates most publisher terms and can get the whole institution's access
|
||||
revoked.** Do not wire a publisher login into the ingest pipeline.
|
||||
|
||||
### R-F — Richer section labels: store real `\section` titles · **DONE 2026-06-17**
|
||||
|
||||
**Resolution (what was done):**
|
||||
- **Why:** R-C labelled `.tex` chunks by mapping the real `\section` heading through
|
||||
`classify_section`'s fixed vocab, so descriptively-titled Math sections collapsed
|
||||
to `body` (only **34/329 = 10 %** non-`body`). A research pass confirmed the
|
||||
`section` column is **write-only** (nothing filters on the vocab — `mcp_server.py`
|
||||
and `wiki.py` don't even SELECT it), so storing free-text titles is safe.
|
||||
- **Change:** new `quality.section_label(title, content)` (+ `_clean_title`) — keep the
|
||||
controlled bucket (intro/theorem/proof/abstract/bibliography) when the heading maps
|
||||
to one, else store the cleaned real title ("the flip algorithm", "main results", …).
|
||||
The `.tex` ingest path uses it; `.pdf`/`.txt`/`run_quality_pass` keep
|
||||
`classify_section` unchanged. `_clean_title` strips LaTeX commands/accents/ties +
|
||||
leading numbering, lower-cases, truncates at a word boundary. ~1 helper + 1 ingest
|
||||
line; no schema change (column already free `TEXT`).
|
||||
- **Result (live):** re-applied to the 13 arXiv papers → non-`body` **34/329 (10 %) →
|
||||
314/329 (95 %)**. Only `math/0306167` (old AMS-TeX, no `\section{}`) stays `body`.
|
||||
- **Tests:** `section_label` / `_clean_title` units + `.tex` ingest stores a
|
||||
descriptive heading verbatim. Full suite 377 passed; ruff/mypy clean. Branch
|
||||
`feat/section-labels` (off `feat/tex-ingest`).
|
||||
- **Pending (infra, not code):** an accent-cleanup polish (`m\"obius`→`mobius`) landed
|
||||
in `_clean_title`, and a quick in-place `UPDATE` re-clean (no re-embed) was started,
|
||||
but the Jetson went offline (ping 100 % loss) before it finished — a few live labels
|
||||
still show LaTeX accents/ties. Re-run the in-place re-clean when the Jetson is back.
|
||||
|
||||
---
|
||||
|
||||
## Priority for the next session
|
||||
|
||||
Reference in New Issue
Block a user