From 967e6baa5908f850a790fd1ebc716819658f25a5 Mon Sep 17 00:00:00 2001 From: Tarik Moussa Date: Sun, 14 Jun 2026 03:54:42 +0200 Subject: [PATCH] =?UTF-8?q?feat(wiki):=20ADR-F12-wiki-compile.md=20?= =?UTF-8?q?=E2=80=94=20GO=20decision=20+=20grounding=20measurement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document Spike result (grounding ~0.95, hallucination ≤5% on 3 concepts), content-5-gram guard design, quarantine mechanism, LLM graceful degradation, cross-ref injection rules, and conflict detection MVP. R-27 satisfied. Co-Authored-By: Claude Sonnet 4.6 --- docs/adr/ADR-F12-wiki-compile.md | 127 +++++++++++++++++++++++++++++++ 1 file changed, 127 insertions(+) create mode 100644 docs/adr/ADR-F12-wiki-compile.md diff --git a/docs/adr/ADR-F12-wiki-compile.md b/docs/adr/ADR-F12-wiki-compile.md new file mode 100644 index 0000000..05fff11 --- /dev/null +++ b/docs/adr/ADR-F12-wiki-compile.md @@ -0,0 +1,127 @@ +# ADR-F12 — Wiki-Compile: Grounded Concept Pages + +**Status:** IMPL (GO after Spike-Session 2026-06-08; Gate-Fixes 2026-06-14) +**Owners:** F-12 Worker (Sonnet) +**Last updated:** 2026-06-14 + +--- + +## Context + +F-12 adds a wiki-compile layer on top of the RAG substrate: for each concept +in `wiki/concepts.yaml`, retrieve Top-K chunks via hybrid dense+FTS search, +synthesise a concept page via local LLM (Ollama / qwen2.5:7b), and write +the result to `wiki/.md`. + +Primary risk: **hallucination**. An LLM can invent theorems, formulas, and +citations that are not in the source corpus. If left unchecked, the wiki +becomes an authoritative-looking source of misinformation. + +Secondary risk: **undetected quarantine escape**. A page with many ungrounded +claims must not appear in the main `wiki/` directory alongside verified pages. + +--- + +## Spike Result (2026-06-08) + +Manual evaluation on 3 concepts (discrete-conformal-map, circle-packing, +lobachevsky-function), corpus = 3 papers (Springborn2008, BPS2015, Luo2004), +95 chunks: + +| Metric | Result | +|---|---| +| Grounding rate | ~0.95 (38/40 sampled claims grounded) | +| Hallucination rate | ≤ 5 % (no invented theorem, no false formula in sample) | +| False ungrounded | ~5 % (legitimate claims not matched due to paraphrase) | + +Conclusion: **GO** — grounded synthesis is viable on this corpus. + +--- + +## Decision + +### Grounding Guard (implemented in `codex/wiki.py: _run_grounding_guard`) + +**Approach:** Content-5-gram match in content-word space. + +1. **Sentence granularity:** Extract the *last sentence* of each claim text + (split on `.!?`, take the last non-empty fragment). Citations annotate the + immediately preceding sentence; checking the whole paragraph would allow + a hallucinated block to be grounded by a single phrase at its end. + +2. **Content-5-gram:** Remove stopwords from both claim sentence and chunk + text. Require 5 consecutive non-stopword tokens from the claim sentence + to appear as 5 consecutive non-stopword tokens in the chunk's content-word + stream (same bibkey). This prevents trivial bypass via common scientific + phrases ("the discrete conformal map" = only 2 content words → ungrounded). + +3. **Short-sentence fallback:** Claims with < 5 content words in the last + sentence are marked **ungrounded** by default (insufficient signal). + +**Adversarial test (Review-Gate Finding):** +> "The Riemann hypothesis is proven by combining Hodge theory with quantum +> entanglement. Furthermore P=NP holds for hyperbolic tetrahedra. +> The discrete conformal map [Springborn2008 #chunk 0]" + +→ Last sentence: "The discrete conformal map" → 3 content words < 5 → +`grounded = False`. Bypass closed. + +### Quarantine Mechanism (implemented in `compile_all`) + +Pages with `grounding_rate < wiki_min_grounding_rate` (default 0.5) are: +- Written to `wiki/draft/.md` instead of `wiki/.md` +- Tracked in `CompileReport.quarantined` +- Logged as `QUARANTINED` in `wiki/log.md` + +`codex wiki check` exits with: +- `0` — all claims grounded, `wiki/draft/` empty +- `1` — at least one `⚠` in `wiki/*.md` +- `2` — `wiki/draft/` non-empty (quarantined pages present) + +Exit 2 takes priority over Exit 1. + +Config: `WIKI_MIN_GROUNDING_RATE` (env/`.env`), default `0.5`. + +### LLM Graceful Degradation + +`compile_concept` wraps `llm.generate()` in a `try/except`. On any exception +(httpx.ConnectError, timeout, …), it logs a warning and returns an empty +`ConceptPage` (no crash). An empty page has 0 claims → grounding rate 0.0 → +quarantine path. + +### Conflict Detection (MVP) + +`_detect_conflicts` uses an adversative keyword heuristic ("but", "however", +"in contrast", "contradicts", …) across pairs of chunks from different bibkeys. +Results in `CompileReport.conflicts`. This is a signal, not a blocker — +conflicts are logged but do not block page compilation. + +### Cross-Reference Injection + +- Inline-code spans (`` `…` ``) are protected by null-byte placeholders + before `_inject_cross_refs` runs, then restored. This prevents concept + titles inside code from being rewritten. +- `all_concepts` passed to `compile_concept` is always the **full** concept + list from `concepts.yaml`, regardless of `--concept` filter. The filter + controls which pages are regenerated, not which cross-refs are resolved. + +--- + +## Fallback + +If grounding rate drops below 50% (config threshold), pages land in +`wiki/draft/` (extraktiver Fallback). Human review required before promotion +to `wiki/`. + +For `wiki_min_grounding_rate = 0.0`, quarantine is disabled (all pages written +to `wiki/` regardless of grounding). Not recommended for production. + +--- + +## Consequences + +- R-27 met: Grounding ≥ 90% on spike corpus; Halluzinations-Rate ≤ 5%. +- R-24 partially met: conflict detection is keyword-heuristic (MVP), not + semantic. Full semantic conflict detection deferred to F-13. +- No DB schema changes (F-12 constraint). All state in `wiki/` directory. +- Ollama endpoint required at compile time; graceful degradation if unavailable.