Data-quality roadmap: R-A–R-F + DQ-5 fixes (citations 590→1075, section signal 10%→95%) #16

Merged
user2595 merged 31 commits from integration/roadmap into main 2026-06-27 07:12:31 +00:00
Showing only changes of commit 5c60a7eda4 - Show all commits

View File

@@ -454,6 +454,31 @@ is self-contained so a cold session can pick it up.
violates most publisher terms and can get the whole institution's access violates most publisher terms and can get the whole institution's access
revoked.** Do not wire a publisher login into the ingest pipeline. revoked.** Do not wire a publisher login into the ingest pipeline.
### R-F — Richer section labels: store real `\section` titles · **DONE 2026-06-17**
**Resolution (what was done):**
- **Why:** R-C labelled `.tex` chunks by mapping the real `\section` heading through
`classify_section`'s fixed vocab, so descriptively-titled Math sections collapsed
to `body` (only **34/329 = 10 %** non-`body`). A research pass confirmed the
`section` column is **write-only** (nothing filters on the vocab — `mcp_server.py`
and `wiki.py` don't even SELECT it), so storing free-text titles is safe.
- **Change:** new `quality.section_label(title, content)` (+ `_clean_title`) — keep the
controlled bucket (intro/theorem/proof/abstract/bibliography) when the heading maps
to one, else store the cleaned real title ("the flip algorithm", "main results", …).
The `.tex` ingest path uses it; `.pdf`/`.txt`/`run_quality_pass` keep
`classify_section` unchanged. `_clean_title` strips LaTeX commands/accents/ties +
leading numbering, lower-cases, truncates at a word boundary. ~1 helper + 1 ingest
line; no schema change (column already free `TEXT`).
- **Result (live):** re-applied to the 13 arXiv papers → non-`body` **34/329 (10 %) →
314/329 (95 %)**. Only `math/0306167` (old AMS-TeX, no `\section{}`) stays `body`.
- **Tests:** `section_label` / `_clean_title` units + `.tex` ingest stores a
descriptive heading verbatim. Full suite 377 passed; ruff/mypy clean. Branch
`feat/section-labels` (off `feat/tex-ingest`).
- **Pending (infra, not code):** an accent-cleanup polish (`m\"obius`→`mobius`) landed
in `_clean_title`, and a quick in-place `UPDATE` re-clean (no re-embed) was started,
but the Jetson went offline (ping 100 % loss) before it finished — a few live labels
still show LaTeX accents/ties. Re-run the in-place re-clean when the Jetson is back.
--- ---
## Priority for the next session ## Priority for the next session