From 5c60a7eda4ad3e5fdb87c45b5a6a83099d679e33 Mon Sep 17 00:00:00 2001 From: Tarik Moussa Date: Thu, 18 Jun 2026 09:25:03 +0200 Subject: [PATCH] =?UTF-8?q?docs(audit):=20add=20R-F=20(richer=20section=20?= =?UTF-8?q?labels)=20resolution=20=E2=80=94=2010%=20to=2095%=20non-body?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- docs/audit/DATA-QUALITY-2026-06-15.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/audit/DATA-QUALITY-2026-06-15.md b/docs/audit/DATA-QUALITY-2026-06-15.md index e15ec46..8058482 100644 --- a/docs/audit/DATA-QUALITY-2026-06-15.md +++ b/docs/audit/DATA-QUALITY-2026-06-15.md @@ -454,6 +454,31 @@ is self-contained so a cold session can pick it up. violates most publisher terms and can get the whole institution's access revoked.** Do not wire a publisher login into the ingest pipeline. +### R-F — Richer section labels: store real `\section` titles · **DONE 2026-06-17** + +**Resolution (what was done):** +- **Why:** R-C labelled `.tex` chunks by mapping the real `\section` heading through + `classify_section`'s fixed vocab, so descriptively-titled Math sections collapsed + to `body` (only **34/329 = 10 %** non-`body`). A research pass confirmed the + `section` column is **write-only** (nothing filters on the vocab — `mcp_server.py` + and `wiki.py` don't even SELECT it), so storing free-text titles is safe. +- **Change:** new `quality.section_label(title, content)` (+ `_clean_title`) — keep the + controlled bucket (intro/theorem/proof/abstract/bibliography) when the heading maps + to one, else store the cleaned real title ("the flip algorithm", "main results", …). + The `.tex` ingest path uses it; `.pdf`/`.txt`/`run_quality_pass` keep + `classify_section` unchanged. `_clean_title` strips LaTeX commands/accents/ties + + leading numbering, lower-cases, truncates at a word boundary. ~1 helper + 1 ingest + line; no schema change (column already free `TEXT`). +- **Result (live):** re-applied to the 13 arXiv papers → non-`body` **34/329 (10 %) → + 314/329 (95 %)**. Only `math/0306167` (old AMS-TeX, no `\section{}`) stays `body`. +- **Tests:** `section_label` / `_clean_title` units + `.tex` ingest stores a + descriptive heading verbatim. Full suite 377 passed; ruff/mypy clean. Branch + `feat/section-labels` (off `feat/tex-ingest`). +- **Pending (infra, not code):** an accent-cleanup polish (`m\"obius`→`mobius`) landed + in `_clean_title`, and a quick in-place `UPDATE` re-clean (no re-embed) was started, + but the Jetson went offline (ping 100 % loss) before it finished — a few live labels + still show LaTeX accents/ties. Re-run the in-place re-clean when the Jetson is back. + --- ## Priority for the next session