From 8af9f9ad0a663a85795d6f0dce99b424a7218536 Mon Sep 17 00:00:00 2001 From: Tarik Moussa Date: Mon, 15 Jun 2026 10:57:24 +0200 Subject: [PATCH] docs(audit): runtime-validate C-4 & C-11, close nougat.py review Exercised the two runtime-confirmable HIGHs against real code paths (injected fake LLM, no Ollama/DB): - C-11 CONFIRMED: 4 distinct leads -> 2 files; connection+gap L-0001 silently overwritten by improvement L-0001 (2/4 leads lost). - C-4 CONFIRMED: zero-claim (uncited) page published to wiki/ + marked compiled, not quarantined (grounding_rate=0.0 < 0.5 but total_claims>0 guard skips it). - nougat.py now fully reviewed -> folds into R-6 (only retries ConnectError, wait_fixed(0), unwrapped in ingest). No new finding. Audit/finding-identification pass complete; remediation planning still open. Co-Authored-By: Claude Fable 5 --- docs/audit/AUDIT-2026-06-15-full-repo.md | 40 +++++++++++++++++++++--- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/docs/audit/AUDIT-2026-06-15-full-repo.md b/docs/audit/AUDIT-2026-06-15-full-repo.md index cc7035d..ffd54f1 100644 --- a/docs/audit/AUDIT-2026-06-15-full-repo.md +++ b/docs/audit/AUDIT-2026-06-15-full-repo.md @@ -483,10 +483,11 @@ cluster into five root themes, not random scatter. - **Then:** MED batch (R-1, R-8, R-9, C-15, C-10) and the LOW polish. ### Coverage honesty -Every `codex/` module was read. Not deep-reviewed: `nougat.py` (54-line `.pdf` -wrapper, skimmed), and the exact runtime DB state (URL-form ids in C-7, and the -`paper_identifiers` existence in M-1) is **inferred from code** and flagged -"confirm on the Jetson DB" — I could not query the live database from here. +Every `codex/` module was read. After the follow-up pass **all are deep-reviewed** +(`nougat.py` included — see Runtime Validation). Runtime state that was initially +*inferred from code* (C-7 id-form, M-1 table existence, and the C-4/C-11 +behaviours) has since been **empirically confirmed** — see the *Live DB +Verification* and *Runtime Validation* sections below. --- @@ -535,3 +536,34 @@ HIGH **8 → 7** (M-1 → MED). C-7 and C-1 are now **confirmed**, not inferred should lead remediation. The dominant root theme (**un-normalized identity**) is empirically validated: 35/36 ids in the "wrong" form, 13 papers mis-classified by the older code path, 57 edges rescued by the newer one. + +--- + +# Runtime Validation — 2026-06-15 (C-4 and C-11 reproduced end-to-end) + +The two HIGH findings flagged "code-certain, runtime-confirmable" were exercised +against the **real** code paths (injected fake LLM; no Ollama, no DB needed). + +### C-11 — CONFIRMED ✅ (lead-ID collision → silent loss; stays HIGH) +Reproduced the exact CLI assembly (`cli.py:421` `connections + gaps + +improvements`, each stage numbered from `L-0001`): **4 distinct leads** (2 +connection, 1 gap, 1 improvement) → `write_leads` produced **2 files**. +`grounded/L-0001.md` survived as the **improvement** ("IMPROVEMENT-one"); the +connection and gap `L-0001` were silently overwritten. **2 of 4 leads lost.** + +### C-4 — CONFIRMED ✅ (zero-claim page bypasses quarantine; stays HIGH) +Drove the real `compile_all` with a fake LLM returning confident **uncited** prose +(→ 0 parseable claims). With `wiki_min_grounding_rate = 0.5` and an actual +grounding rate of `0.0`, the page was **published to `wiki/` and marked +compiled** (`report.compiled=['test-concept']`, `report.quarantined=[]`) — not +quarantined to `wiki/draft/`. The `total_claims > 0` guard (`wiki.py:730`) is the +exact bypass; the bug fires precisely when grounding evidence is weakest. + +### `nougat.py` — now fully reviewed (no new finding) +Folds into **R-6** (PDF-path resilience). Detail: `pdf_to_markdown` retries **only** +`httpx.ConnectError` with `wait_fixed(0)` (no 5xx retry, no back-off) and is +unwrapped at `ingest.py:183` — a Nougat 5xx aborts PDF ingest. + +**Closure:** every `codex/` module is deep-reviewed; the two runtime-confirmable +HIGHs are empirically reproduced. The **audit / finding-identification pass is +complete**. Remediation planning is the open next phase.