From fa43f69dfdbe3c54eef8c6ab1b3e2fd569eaeaf4 Mon Sep 17 00:00:00 2001 From: Tarik Moussa Date: Wed, 17 Jun 2026 00:23:16 +0200 Subject: [PATCH] =?UTF-8?q?docs(audit):=20DQ-3=20content=20fidelity=20veri?= =?UTF-8?q?fied=20clean=20=E2=80=94=20assessment=20axis=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-ran chunk_text + is_quality_chunk on all 29 source .txt files vs stored chunks. Coverage 109-114% everywhere (the >100% is the 64/512-word overlap, i.e. full retention); stored==kept for every paper. Only 18 chunks dropped corpus-wide, all low_alpha = garbled OCR tables in 2 scanned-book sources (0-387-29555-0_13, depositonce-5415) — no real math content lost. Head/tail aligned (no truncation). Closes the data-quality assessment axis: DQ-1 (citations), DQ-2 (metadata), DQ-3 (fidelity), DQ-4 (retrieval) all resolved. DQ-5 (re-ingest idempotency) remains, tracked separately. Co-Authored-By: Claude Opus 4.8 --- docs/audit/DATA-QUALITY-2026-06-15.md | 38 ++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 6 deletions(-) diff --git a/docs/audit/DATA-QUALITY-2026-06-15.md b/docs/audit/DATA-QUALITY-2026-06-15.md index 81b8a4c..1d46ea4 100644 --- a/docs/audit/DATA-QUALITY-2026-06-15.md +++ b/docs/audit/DATA-QUALITY-2026-06-15.md @@ -4,8 +4,10 @@ applied to live DB + wired into ingest). **DQ-4 MEASURED 2026-06-16** — retrieval is strong after fixing a P0 `codex search paper` crash. **DQ-2 RESOLVED 2026-06-16** (1911.00966 fully recovered from arXiv; s00454 abstract from S2; book chapter is a -documented limit; recovery wired into ingest). **DQ-3 still open. New: DQ-5** -(`ingest_paper` not idempotent for DOI papers — blocks R-A/R-C). +documented limit; recovery wired into ingest). **DQ-3 RESOLVED 2026-06-16** +(content fidelity clean — full coverage, only OCR junk dropped). **All four +assessment items (DQ-1..DQ-4) DONE. New: DQ-5** (`ingest_paper` not idempotent +for DOI papers — blocks R-A/R-C, tracked separately). **Roadmap R-A..R-E** added 2026-06-16 (free-source acquisition levers — see Roadmap section). **Author:** Audit-Loop (Opus). **Intended reader:** a *cold* session with no memory of the audit conversation — everything needed to continue is in this file. @@ -157,7 +159,29 @@ incomplete, or unrepresentative data. This document is that second axis — a - **Acceptance:** every paper has at least a bibkey + non-zero abstract embedding, or the exceptions are documented with rationale. -### DQ-3 — Content fidelity (chunks vs source `.txt`): NOT YET VERIFIED · **MED** +### DQ-3 — Content fidelity (chunks vs source `.txt`): VERIFIED CLEAN · **RESOLVED 2026-06-16** + +**Resolution (read-only; re-ran `chunk_text` + `is_quality_chunk` on all 29 +source files and compared to stored chunks):** +- **No paper is silently gutted.** Coverage (stored chunk chars / source chars) + is **109–114% for every paper** — the >100% is exactly the 64/512-word overlap + inflation, i.e. full retention. `stored == kept` for all 29, so the live DB + matches what the current pipeline produces. +- **Only 18 chunks dropped corpus-wide, all `low_alpha`** (0 short, 0 bib), + concentrated in `10.1007/0-387-29555-0_13` (6) and `depositonce-5415` (10). + Inspected: they are **garbled OCR tables** from scanned-book sources (alpha + 0.30–0.36, e.g. `"(u: 1) 7 < u D:= ud3424 — else H®: out.v. Family 1…"`) — + exactly the artefacts the alpha-ratio gate targets. **No real math content lost** + (the handoff's feared failure mode did not occur; math prose stays > 0.40 alpha). +- **Head/tail alignment exact** on sampled papers: chunk 0 begins at the source + head (title/authors/abstract captured), last chunk ends at the source tail → no + front/back truncation. +- **Note:** 0 bibliography drops across the corpus → the upstream `.txt` extraction + had already stripped reference lists, which is why DQ-1's citation graph relies + on the API/GROBID sources rather than text-mined refs. +- **Acceptance met** for all 29 papers (not just a sample). No action needed. + +**Original open question (for context):** - **Open question:** does the stored chunk set faithfully reconstruct each source file, or did the chunker / `filter_chunks` silently drop material (e.g. an abstract, a section, math-heavy passages)? The whole corpus was ingested from @@ -324,9 +348,11 @@ is self-contained so a cold session can pick it up. verified strong; surfaced the zero-vector pollution that motivates DQ-2). 3. ~~**DQ-2**~~ — **DONE 2026-06-16** (1911.00966 fully recovered; s00454 abstract from S2; book chapter documented; recovery wired into ingest). -4. **DQ-5** (NEXT — `ingest_paper` not idempotent for DOI papers; **blocks R-A/R-C**; - careful — M-1 canonical-id territory), then **DQ-3** (chunk-vs-source fidelity). -5. **Roadmap levers** (see section above): **R-A** (GROBID on arXiv PDFs — closes +4. ~~**DQ-3**~~ — **DONE 2026-06-16** (content fidelity clean; full coverage, only + 18 OCR-junk chunks dropped, no math lost). **Assessment axis DQ-1..DQ-4 complete.** +5. **DQ-5** (NEXT — `ingest_paper` not idempotent for DOI papers; **blocks R-A/R-C**; + careful — M-1 canonical-id territory). +6. **Roadmap levers** (see section above): **R-A** (GROBID on arXiv PDFs — closes the last 2 zero-citation theses; needs DQ-5 first), **R-B** (Crossref refs — abstract half already shipped in DQ-2), **R-C** (`.tex` ingest — serves DQ-3; needs DQ-5 first), **R-D** (F-15 coverage warning, quick). **R-E** (paywall)