user2595/codex-py
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
chore/scrub-private-ip
codex-py/docs/adr/ADR-F12-wiki-compile.md
Tarik Moussa 967e6baa59 feat(wiki): ADR-F12-wiki-compile.md — GO decision + grounding measurement 
Document Spike result (grounding ~0.95, hallucination ≤5% on 3 concepts),
content-5-gram guard design, quarantine mechanism, LLM graceful degradation,
cross-ref injection rules, and conflict detection MVP.
R-27 satisfied.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-14 05:11:06 +02:00

4.9 KiB
Raw Permalink Blame History

ADR-F12 — Wiki-Compile: Grounded Concept Pages

Status: IMPL (GO after Spike-Session 2026-06-08; Gate-Fixes 2026-06-14) Owners: F-12 Worker (Sonnet) Last updated: 2026-06-14

Context

F-12 adds a wiki-compile layer on top of the RAG substrate: for each concept in wiki/concepts.yaml, retrieve Top-K chunks via hybrid dense+FTS search, synthesise a concept page via local LLM (Ollama / qwen2.5:7b), and write the result to wiki/<slug>.md.

Primary risk: hallucination. An LLM can invent theorems, formulas, and citations that are not in the source corpus. If left unchecked, the wiki becomes an authoritative-looking source of misinformation.

Secondary risk: undetected quarantine escape. A page with many ungrounded claims must not appear in the main wiki/ directory alongside verified pages.

Spike Result (2026-06-08)

Manual evaluation on 3 concepts (discrete-conformal-map, circle-packing, lobachevsky-function), corpus = 3 papers (Springborn2008, BPS2015, Luo2004), 95 chunks:

Metric Result
Grounding rate ~0.95 (38/40 sampled claims grounded)
Hallucination rate ≤ 5 % (no invented theorem, no false formula in sample)
False ungrounded ~5 % (legitimate claims not matched due to paraphrase)

Conclusion: GO — grounded synthesis is viable on this corpus.

Decision

Grounding Guard (implemented in codex/wiki.py: _run_grounding_guard)

Approach: Content-5-gram match in content-word space.

  1. Sentence granularity: Extract the last sentence of each claim text (split on .!?, take the last non-empty fragment). Citations annotate the immediately preceding sentence; checking the whole paragraph would allow a hallucinated block to be grounded by a single phrase at its end.

  2. Content-5-gram: Remove stopwords from both claim sentence and chunk text. Require 5 consecutive non-stopword tokens from the claim sentence to appear as 5 consecutive non-stopword tokens in the chunk's content-word stream (same bibkey). This prevents trivial bypass via common scientific phrases ("the discrete conformal map" = only 2 content words → ungrounded).

  3. Short-sentence fallback: Claims with < 5 content words in the last sentence are marked ungrounded by default (insufficient signal).

Adversarial test (Review-Gate Finding):

"The Riemann hypothesis is proven by combining Hodge theory with quantum entanglement. Furthermore P=NP holds for hyperbolic tetrahedra. The discrete conformal map [Springborn2008 #chunk 0]"

→ Last sentence: "The discrete conformal map" → 3 content words < 5 → grounded = False. Bypass closed.

Quarantine Mechanism (implemented in compile_all)

Pages with grounding_rate < wiki_min_grounding_rate (default 0.5) are:

  • Written to wiki/draft/<slug>.md instead of wiki/<slug>.md
  • Tracked in CompileReport.quarantined
  • Logged as QUARANTINED in wiki/log.md

codex wiki check exits with:

  • 0 — all claims grounded, wiki/draft/ empty
  • 1 — at least one ⚠ in wiki/*.md
  • 2 — wiki/draft/ non-empty (quarantined pages present)

Exit 2 takes priority over Exit 1.

Config: WIKI_MIN_GROUNDING_RATE (env/.env), default 0.5.

LLM Graceful Degradation

compile_concept wraps llm.generate() in a try/except. On any exception (httpx.ConnectError, timeout, …), it logs a warning and returns an empty ConceptPage (no crash). An empty page has 0 claims → grounding rate 0.0 → quarantine path.

Conflict Detection (MVP)

_detect_conflicts uses an adversative keyword heuristic ("but", "however", "in contrast", "contradicts", …) across pairs of chunks from different bibkeys. Results in CompileReport.conflicts. This is a signal, not a blocker — conflicts are logged but do not block page compilation.

Cross-Reference Injection

  • Inline-code spans (`…`) are protected by null-byte placeholders before _inject_cross_refs runs, then restored. This prevents concept titles inside code from being rewritten.
  • all_concepts passed to compile_concept is always the full concept list from concepts.yaml, regardless of --concept filter. The filter controls which pages are regenerated, not which cross-refs are resolved.

Fallback

If grounding rate drops below 50% (config threshold), pages land in wiki/draft/ (extraktiver Fallback). Human review required before promotion to wiki/.

For wiki_min_grounding_rate = 0.0, quarantine is disabled (all pages written to wiki/ regardless of grounding). Not recommended for production.

Consequences

  • R-27 met: Grounding ≥ 90% on spike corpus; Halluzinations-Rate ≤ 5%.
  • R-24 partially met: conflict detection is keyword-heuristic (MVP), not semantic. Full semantic conflict detection deferred to F-13.
  • No DB schema changes (F-12 constraint). All state in wiki/ directory.
  • Ollama endpoint required at compile time; graceful degradation if unavailable.