From b371119264093c4ab0a9e11794003e95ce016dcd Mon Sep 17 00:00:00 2001 From: Tarik Moussa Date: Wed, 17 Jun 2026 00:33:45 +0200 Subject: [PATCH] =?UTF-8?q?docs(audit):=20scope=20DQ-3=20to=20txt=E2=86=92?= =?UTF-8?q?chunks=20+=20open=20DQ-6=20(PDF=E2=86=92txt=20extraction=20fide?= =?UTF-8?q?lity)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clarify that DQ-3's 'VERIFIED CLEAN' covers the ingest pipeline (txt -> chunks) only — it treats the .txt as ground truth. The upstream PDF/source -> .txt extraction (done outside codex-py) is unmeasured and shown to be lossy (DQ-3's dropped chunks were garbled OCR tables already present in the .txt). Add DQ-6 to measure that layer; 36 source PDFs are available. Note R-A/R-C (re-ingest from PDF/.tex) would largely sidestep it. Co-Authored-By: Claude Opus 4.8 --- docs/audit/DATA-QUALITY-2026-06-15.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/audit/DATA-QUALITY-2026-06-15.md b/docs/audit/DATA-QUALITY-2026-06-15.md index 88d62ee..c942325 100644 --- a/docs/audit/DATA-QUALITY-2026-06-15.md +++ b/docs/audit/DATA-QUALITY-2026-06-15.md @@ -181,6 +181,32 @@ source files and compared to stored chunks):** had already stripped reference lists, which is why DQ-1's citation graph relies on the API/GROBID sources rather than text-mined refs. - **Acceptance met** for all 29 papers (not just a sample). No action needed. +- **⚠ SCOPE LIMIT — DQ-3 covers `.txt → chunks` ONLY.** It treats each source + `.txt` as ground truth and verifies the *ingest pipeline* preserves it. It does + **not** measure the layer above — **PDF/source → `.txt`** — which was done by an + upstream tool in `ConformalLabpp` (outside codex-py) and is unexamined. There is + already evidence that layer is lossy: the 18 dropped chunks were garbled OCR + tables *already present in the `.txt`* (i.e. the PDF→txt step garbled the + table/formula regions), and the bibliographies were stripped upstream. So + "VERIFIED CLEAN" means **the loader faithfully preserves whatever the `.txt` + contains, not that no information was lost from the source paper.** See DQ-6. + +### DQ-6 — PDF/source → `.txt` extraction fidelity: NOT MEASURED · **MED, open** +- **Open question:** how much did the upstream PDF→`.txt` extraction (done in + `ConformalLabpp`, outside codex-py) drop or garble vs the original papers? + Math/equations, tables, multi-column layouts, and figure captions are the usual + casualties of PDF text extraction — and we have direct evidence of garble + (DQ-3's low_alpha drops were already-garbled tables in the `.txt`). +- **Checkable:** the 36 original PDFs are present in `…/ConformalLabpp/papers/` + (map each paper's `source_path` `.txt` to its sibling `.pdf` by filename stem). +- **To investigate:** for a sample (incl. the known-bad scanned book + `0-387-29555-0_13` and a born-digital arXiv PDF), extract PDF text independently + and compare length / page-coverage to the `.txt`; spot-check whether specific + theorem statements & equations survive. Flag papers where the `.txt` is missing + large spans. +- **Acceptance:** a per-sample source→txt coverage estimate, or a list of papers + whose `.txt` is materially degraded (candidates for re-extraction — note roadmap + **R-A/R-C** would re-ingest from PDF/`.tex` directly and largely sidestep this). **Original open question (for context):** - **Open question:** does the stored chunk set faithfully reconstruct each source