967e6baa59
Document Spike result (grounding ~0.95, hallucination ≤5% on 3 concepts), content-5-gram guard design, quarantine mechanism, LLM graceful degradation, cross-ref injection rules, and conflict detection MVP. R-27 satisfied. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
128 lines
4.9 KiB
Markdown
128 lines
4.9 KiB
Markdown
# ADR-F12 — Wiki-Compile: Grounded Concept Pages
|
|
|
|
**Status:** IMPL (GO after Spike-Session 2026-06-08; Gate-Fixes 2026-06-14)
|
|
**Owners:** F-12 Worker (Sonnet)
|
|
**Last updated:** 2026-06-14
|
|
|
|
---
|
|
|
|
## Context
|
|
|
|
F-12 adds a wiki-compile layer on top of the RAG substrate: for each concept
|
|
in `wiki/concepts.yaml`, retrieve Top-K chunks via hybrid dense+FTS search,
|
|
synthesise a concept page via local LLM (Ollama / qwen2.5:7b), and write
|
|
the result to `wiki/<slug>.md`.
|
|
|
|
Primary risk: **hallucination**. An LLM can invent theorems, formulas, and
|
|
citations that are not in the source corpus. If left unchecked, the wiki
|
|
becomes an authoritative-looking source of misinformation.
|
|
|
|
Secondary risk: **undetected quarantine escape**. A page with many ungrounded
|
|
claims must not appear in the main `wiki/` directory alongside verified pages.
|
|
|
|
---
|
|
|
|
## Spike Result (2026-06-08)
|
|
|
|
Manual evaluation on 3 concepts (discrete-conformal-map, circle-packing,
|
|
lobachevsky-function), corpus = 3 papers (Springborn2008, BPS2015, Luo2004),
|
|
95 chunks:
|
|
|
|
| Metric | Result |
|
|
|---|---|
|
|
| Grounding rate | ~0.95 (38/40 sampled claims grounded) |
|
|
| Hallucination rate | ≤ 5 % (no invented theorem, no false formula in sample) |
|
|
| False ungrounded | ~5 % (legitimate claims not matched due to paraphrase) |
|
|
|
|
Conclusion: **GO** — grounded synthesis is viable on this corpus.
|
|
|
|
---
|
|
|
|
## Decision
|
|
|
|
### Grounding Guard (implemented in `codex/wiki.py: _run_grounding_guard`)
|
|
|
|
**Approach:** Content-5-gram match in content-word space.
|
|
|
|
1. **Sentence granularity:** Extract the *last sentence* of each claim text
|
|
(split on `.!?`, take the last non-empty fragment). Citations annotate the
|
|
immediately preceding sentence; checking the whole paragraph would allow
|
|
a hallucinated block to be grounded by a single phrase at its end.
|
|
|
|
2. **Content-5-gram:** Remove stopwords from both claim sentence and chunk
|
|
text. Require 5 consecutive non-stopword tokens from the claim sentence
|
|
to appear as 5 consecutive non-stopword tokens in the chunk's content-word
|
|
stream (same bibkey). This prevents trivial bypass via common scientific
|
|
phrases ("the discrete conformal map" = only 2 content words → ungrounded).
|
|
|
|
3. **Short-sentence fallback:** Claims with < 5 content words in the last
|
|
sentence are marked **ungrounded** by default (insufficient signal).
|
|
|
|
**Adversarial test (Review-Gate Finding):**
|
|
> "The Riemann hypothesis is proven by combining Hodge theory with quantum
|
|
> entanglement. Furthermore P=NP holds for hyperbolic tetrahedra.
|
|
> The discrete conformal map [Springborn2008 #chunk 0]"
|
|
|
|
→ Last sentence: "The discrete conformal map" → 3 content words < 5 →
|
|
`grounded = False`. Bypass closed.
|
|
|
|
### Quarantine Mechanism (implemented in `compile_all`)
|
|
|
|
Pages with `grounding_rate < wiki_min_grounding_rate` (default 0.5) are:
|
|
- Written to `wiki/draft/<slug>.md` instead of `wiki/<slug>.md`
|
|
- Tracked in `CompileReport.quarantined`
|
|
- Logged as `QUARANTINED` in `wiki/log.md`
|
|
|
|
`codex wiki check` exits with:
|
|
- `0` — all claims grounded, `wiki/draft/` empty
|
|
- `1` — at least one `⚠` in `wiki/*.md`
|
|
- `2` — `wiki/draft/` non-empty (quarantined pages present)
|
|
|
|
Exit 2 takes priority over Exit 1.
|
|
|
|
Config: `WIKI_MIN_GROUNDING_RATE` (env/`.env`), default `0.5`.
|
|
|
|
### LLM Graceful Degradation
|
|
|
|
`compile_concept` wraps `llm.generate()` in a `try/except`. On any exception
|
|
(httpx.ConnectError, timeout, …), it logs a warning and returns an empty
|
|
`ConceptPage` (no crash). An empty page has 0 claims → grounding rate 0.0 →
|
|
quarantine path.
|
|
|
|
### Conflict Detection (MVP)
|
|
|
|
`_detect_conflicts` uses an adversative keyword heuristic ("but", "however",
|
|
"in contrast", "contradicts", …) across pairs of chunks from different bibkeys.
|
|
Results in `CompileReport.conflicts`. This is a signal, not a blocker —
|
|
conflicts are logged but do not block page compilation.
|
|
|
|
### Cross-Reference Injection
|
|
|
|
- Inline-code spans (`` `…` ``) are protected by null-byte placeholders
|
|
before `_inject_cross_refs` runs, then restored. This prevents concept
|
|
titles inside code from being rewritten.
|
|
- `all_concepts` passed to `compile_concept` is always the **full** concept
|
|
list from `concepts.yaml`, regardless of `--concept` filter. The filter
|
|
controls which pages are regenerated, not which cross-refs are resolved.
|
|
|
|
---
|
|
|
|
## Fallback
|
|
|
|
If grounding rate drops below 50% (config threshold), pages land in
|
|
`wiki/draft/` (extraktiver Fallback). Human review required before promotion
|
|
to `wiki/`.
|
|
|
|
For `wiki_min_grounding_rate = 0.0`, quarantine is disabled (all pages written
|
|
to `wiki/` regardless of grounding). Not recommended for production.
|
|
|
|
---
|
|
|
|
## Consequences
|
|
|
|
- R-27 met: Grounding ≥ 90% on spike corpus; Halluzinations-Rate ≤ 5%.
|
|
- R-24 partially met: conflict detection is keyword-heuristic (MVP), not
|
|
semantic. Full semantic conflict detection deferred to F-13.
|
|
- No DB schema changes (F-12 constraint). All state in `wiki/` directory.
|
|
- Ollama endpoint required at compile time; graceful degradation if unavailable.
|