docs(audit): scope DQ-3 to txt→chunks + open DQ-6 (PDF→txt extraction fidelity)

Clarify that DQ-3's 'VERIFIED CLEAN' covers the ingest pipeline (txt -> chunks)
only — it treats the .txt as ground truth. The upstream PDF/source -> .txt
extraction (done outside codex-py) is unmeasured and shown to be lossy (DQ-3's
dropped chunks were garbled OCR tables already present in the .txt). Add DQ-6 to
measure that layer; 36 source PDFs are available. Note R-A/R-C (re-ingest from
PDF/.tex) would largely sidestep it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Tarik Moussa
2026-06-17 00:33:45 +02:00
parent ea73b1475c
commit b371119264

View File

@@ -181,6 +181,32 @@ source files and compared to stored chunks):**
had already stripped reference lists, which is why DQ-1's citation graph relies
on the API/GROBID sources rather than text-mined refs.
- **Acceptance met** for all 29 papers (not just a sample). No action needed.
- **⚠ SCOPE LIMIT — DQ-3 covers `.txt chunks` ONLY.** It treats each source
`.txt` as ground truth and verifies the *ingest pipeline* preserves it. It does
**not** measure the layer above — **PDF/source → `.txt`** — which was done by an
upstream tool in `ConformalLabpp` (outside codex-py) and is unexamined. There is
already evidence that layer is lossy: the 18 dropped chunks were garbled OCR
tables *already present in the `.txt`* (i.e. the PDF→txt step garbled the
table/formula regions), and the bibliographies were stripped upstream. So
"VERIFIED CLEAN" means **the loader faithfully preserves whatever the `.txt`
contains, not that no information was lost from the source paper.** See DQ-6.
### DQ-6 — PDF/source → `.txt` extraction fidelity: NOT MEASURED · **MED, open**
- **Open question:** how much did the upstream PDF→`.txt` extraction (done in
`ConformalLabpp`, outside codex-py) drop or garble vs the original papers?
Math/equations, tables, multi-column layouts, and figure captions are the usual
casualties of PDF text extraction — and we have direct evidence of garble
(DQ-3's low_alpha drops were already-garbled tables in the `.txt`).
- **Checkable:** the 36 original PDFs are present in `…/ConformalLabpp/papers/`
(map each paper's `source_path` `.txt` to its sibling `.pdf` by filename stem).
- **To investigate:** for a sample (incl. the known-bad scanned book
`0-387-29555-0_13` and a born-digital arXiv PDF), extract PDF text independently
and compare length / page-coverage to the `.txt`; spot-check whether specific
theorem statements & equations survive. Flag papers where the `.txt` is missing
large spans.
- **Acceptance:** a per-sample source→txt coverage estimate, or a list of papers
whose `.txt` is materially degraded (candidates for re-extraction — note roadmap
**R-A/R-C** would re-ingest from PDF/`.tex` directly and largely sidestep this).
**Original open question (for context):**
- **Open question:** does the stored chunk set faithfully reconstruct each source