# AUDIT — Full Repository (iterative)

**Auditor:** Audit-Loop (Opus)
**Started:** 2026-06-15
**Scope:** entire `main` tree, audited iteratively module-by-module.
**Companion doc:** [`AUDIT-2026-06-15-loop-1.md`](AUDIT-2026-06-15-loop-1.md) —
the F-15 / dev-loop-1 findings (S-1…U-4). This document continues the **same
finding-ID register** (so `C-4` here follows `C-3` there) and covers the rest of
the repo.

> Severity: **HIGH** fix before relying · **MED** fix soon · **LOW** polish ·
> **INFO** accepted/no-action. Findings only — remediation planned separately.

## Coverage tracker

| Part | Area | Status |
|------|------|--------|
| (loop-1) | F-15 graph, db/schema, ingest, discover, mcp, provenance-scan, cli(search+graph), config(graph) | ✅ documented |
| **A** | Grounding guard (`wiki.py` `_parse_claims`/`_run_grounding_guard`, quarantine) — shared by `synthesis.py` | ✅ documented |
| **B** | `sources/` (arxiv, openalex, semanticscholar) | ✅ documented |
| **C** | `parsing/` (grobid, nougat, mathpix, figures, tex) | ✅ documented |
| **D** | `synthesis.py` orchestration (find_connections/gaps/improvements, conjectures, write_leads) | ✅ documented |
| **E** | `wiki.py` rest (retrieve, compile_concept, cross-refs, index/log, embed) | ✅ documented |
| **F** | `embed.py`, `quality.py`, `models.py` | ✅ documented |
| **G** | `cli.py` rest (ingest/wiki/synthesis/provenance commands), `config.py` rest | ✅ documented |

---

## Part A — Grounding guard (the "zero-fact-leak" core)

The guard lives in `wiki.py` (`_parse_claims:216`, `_run_grounding_guard:299`)
and is imported by `synthesis.py:57-59`, so **one implementation gates both**
the wiki pages and the synthesis leads. A claim is "grounded" iff the **last
sentence** before its `[BibKey #loc]` citation contains a run of 5 consecutive
non-stopword tokens that appears as a substring of any chunk of the same bibkey.

### C-4 — Zero-claim pages bypass quarantine and publish as "compiled" · **HIGH**
- **Evidence:** `wiki.py:730` — `if total_claims > 0 and grounding_rate < threshold:`
  quarantines; the `else` (`:737-742`) **publishes to `wiki/` and records the
  hash** as successfully compiled.
- **Bug:** when the LLM output has **no parseable citations** (`total_claims == 0`,
  so `grounding_rate == 0.0`), the `total_claims > 0` guard is false → it takes
  the *publish* branch. The quarantine gate is skipped exactly when grounding
  evidence is **weakest** (zero citations).
- **Trigger paths:** (a) the LLM emits uncited prose; (b) **LLM outage** — the
  module header guarantees "LLM unavailable → empty ConceptPage, no crash"
  (`wiki.py:13-14`), so a transient Ollama outage writes near-empty stub pages to
  `wiki/`, marks them compiled, and stores their hash. On the next
  `changed_only` run the hash matches → the stub is **skipped forever** until a
  forced recompile. A research wiki silently fills with empty/uncited pages.

### C-5 — "Every claim is grounded" holds only for the last cited sentence · **MED**
- **Evidence:** `_CLAIM_RE` (`wiki.py:211-213`) captures `text = [^\[]+?` (the
  span back to the previous `]`/start); `_last_sentence` (`:279-291`) then keeps
  only the final sentence; `_run_grounding_guard` checks that sentence only.
- **Two holes in the guarantee:**
  1. **Paragraph passengers** — any sentence that is *not* the last one before a
     citation is never grounding-checked and never `⚠`-marked. "Hallucination one.
     Hallucination two. Grounded tail [Bib]." passes with only the tail verified.
  2. **Uncited text is invisible** — a sentence with no `[BibKey]` is not a Claim
     at all (`_parse_claims` only yields cited spans), so it is never grounded
     nor marked. The LLM can emit fully uncited assertions into a published page.
- **Impact:** the effective invariant is "*the last sentence immediately before
  each citation has lexical overlap with the cited paper*" — materially weaker
  than the N-02 "zero-fact-leak enforcement" framing. The module **header is
  honest** ("Substring-Match MVP", `wiki.py:4,13`); the commit/ADR language is
  not. Recommend aligning the claim or strengthening the guard (check every
  sentence; flag uncited sentences).

### C-6 — Substring match does not enforce the documented "5 consecutive tokens" · **LOW**
- **Evidence:** `wiki.py:347-349` — `gram = " ".join(content[i:i+5])`;
  `if any(gram in src for src in sources)` where `src` is the space-joined
  content-word string of the chunk.
- **Issue:** a *substring* test on the joined string lets the first/last gram
  token match as a **prefix/suffix of a longer chunk word** (e.g. gram
  `"cat dog …"` substring-matches inside `"scat dogma …"`). The docstring claims
  matching "as 5 consecutive non-stopword **tokens**" — true sequence matching
  would compare token lists with a sliding window (or add boundary sentinels).
  Low real-world impact at 5-gram length, but the claimed invariant is not what
  the code enforces.

### R-1 — Tokenizer brittleness causes false-negative over-quarantine · **MED**
- **Evidence:** `_content_words` (`wiki.py:294-296`) lowercases and `.split()`s on
  whitespace only — **no punctuation stripping**. `_last_sentence` splits on
  `.!?` (`:276`).
- **Impact:** punctuation stays glued to tokens, so a faithful paraphrase grounds
  only on *exact* surface form — `"volume,"` ≠ `"volume"`, `"L(gamma1)"` must
  match character-for-character, and a trailing decimal like `"V = 1.5"` makes
  `_last_sentence` return the fragment `"5"` (split at the `.`), which then has
  `< 5` content words → **forced ungrounded** (`:339-344`). Math-heavy and terse
  claims are systematically marked ungrounded even when verbatim, inflating the
  ungrounded ratio and pushing real pages into quarantine (and into C-4's
  blind spot when it drives `total_claims` interactions). The passing test
  `test_grounding_guard_…` only succeeds because its fixture reproduces the chunk
  text *exactly*.

### R-2 — Reference-list filter can drop theorem-dense chunks · **LOW**
- **Evidence:** `_retrieve_chunks` (`wiki.py:188-199`) skips a chunk when
  `>60 %` of its lines match `^\s*(\[\d+\]|[A-Z][a-z]+,?\s+[A-Z]\.)`.
- **Issue:** the second alternative (`[A-Z][a-z]+,?\s+[A-Z]\.`) also matches prose
  like `"Theorem A."`, `"Lemma B."`, `"Section C."`. A chunk formatted as a list
  of theorems/lemmas can exceed 60 % and be discarded as a "reference list" —
  silently dropping exactly the mathematical content the wiki wants.

### R-3 — Conflict detection has near-zero precision · **LOW**
- **Evidence:** `_detect_conflicts` (`wiki.py:367-…`) flags a pair `(bk1, bk2)` if
  **each** has *any* chunk containing a hedging keyword
  (`but|however|in contrast|…`, `_CONFLICT_KEYWORDS:361-364`).
- **Issue:** it never checks the keywords relate to a shared topic or to each
  other. In a corpus where most papers contain "however"/"but" somewhere, this
  flags ~O(n²) pairs as "conflicting" — noise surfaced in `CompileReport`.
  Labeled "heuristic signal only", but as-is it is not actionable.

**Part A net:** 1 HIGH (C-4), 2 MED (C-5, R-1), 3 LOW (C-6, R-2, R-3). The guard
is a reasonable lexical MVP, but (a) it has a concrete publish-bypass bug for
zero-citation output, and (b) the "zero-fact-leak" framing overstates what a
last-sentence substring check can guarantee.

---

## Part B — `sources/` (external API clients)

Solid foundations: all three clients set httpx timeouts; OpenAlex and S2 retry
429/5xx with exponential back-off (tenacity); S2 enforces a ≥1 req/s floor with a
monotonic clock under a lock. The findings are about **ID format consistency**
and a couple of dead/inert paths.

### C-7 — `papers.id` is stored in OpenAlex *URL* form, breaking the bare-ID contract · **HIGH**
- **Evidence:** `openalex.py:94` — `_map_paper` sets
  `id = data.get("doi") or data.get("id") or ""`. Real OpenAlex returns `doi` as
  a **full URL** (`https://doi.org/10.x`) and `id` as `https://openalex.org/W…`
  (the repo's own test confirms the URL form for `openalex_id`,
  `test_openalex.py:92`). `ingest.py:82` takes `paper.id` from this unchanged and
  upserts it as `papers.id` (`ingest.py:138`).
- **Why it's currently masked:** the test fixture `_SAMPLE_WORK["doi"]` is a
  **bare** DOI `"10.1145/3592430"` (`test_openalex.py:18`) — not the URL form real
  OpenAlex emits — and **no test asserts `paper.id`** at all. So the suite is
  green while production stores `papers.id = "https://doi.org/10.1145/3592430"`.
- **Impact:**
  1. **Bare-ID lookups fail.** `ingest_all.sh` ingests `"10.1145/1964921.1964997"`,
     but `papers.id` becomes the URL; `codex graph related 10.1145/…`,
     `discover citing 10.1145/…`, search-by-id, etc. find nothing.
  2. **Second ID-mismatch axis (compounds C-1).** `cited_id` from the S2/GROBID
     paths is a **bare** DOI/arXiv (`semanticscholar.py:98-99`,
     `ingest.py:187`), so a bare-DOI reference can never equal a URL-form
     `papers.id` — bare-DOI cross-citations stay "dangling" regardless of the
     F-15 W-ID resolution (which only matches `papers.openalex_id`, the URL form).
- **Action:** confirm with `SELECT id FROM papers LIMIT 5` on the Jetson DB; if
  URL-form, normalize `papers.id` to a canonical bare form at ingest (and the
  `cited_id` writers to match), then add a representative fixture + `paper.id`
  assertion (see T-3).

### T-3 — OpenAlex test fixture is unrepresentative and skips `paper.id` · **MED**
- **Evidence:** `_SAMPLE_WORK["doi"] = "10.1145/3592430"` (bare) vs the real
  `https://doi.org/…` URL form; `test_fetch_paper_*` assert title/year/authors/
  abstract/`openalex_id` but **never** `paper.id`.
- **Impact:** the id-mapping — the exact surface of C-7 — is untested, and the
  fixture actively hides the URL-form behaviour. Any normalization fix needs this
  closed first or it will regress invisibly.

### R-5 — S2 `fetch_references` breaks on legacy arXiv IDs containing `/` · **LOW**
- **Evidence:** `semanticscholar.py:80` interpolates `paper_id` into the path
  `…/paper/{paper_id}/references`. The ingest fallback passes `arXiv:math/0603097`
  (`ingest.py:91`) — the embedded `/` yields a malformed path → 404 →
  `fetch_references` silently returns `[]`. Legacy-arXiv papers that miss the
  OpenAlex path get **zero references** with no error.

### R-4 — `arxiv.fetch_source` no-op try/except + no retry · **LOW**
- **Evidence:** `arxiv.py:41-44` — `try: httpx.get(...) except httpx.RequestError: raise`
  catches and immediately re-raises, doing nothing. Unlike OpenAlex/S2, the arXiv
  client has **no** retry/back-off, so a transient network blip aborts ingest.
  Minor resilience asymmetry.

### C-9 — `semanticscholar.fetch_recommendations` is unreachable app code · **LOW**
- **Evidence:** referenced only by `tests/sources/test_semanticscholar.py`; no
  `codex/` module or CLI calls it (`grep fetch_recommendations`). Tested but
  never wired — dead feature surface (decide: wire a `discover recommend`
  command, or drop).

**Part B net:** 1 HIGH (C-7, ID-form contract — the big one; confirm on live DB),
1 MED (T-3, fixture masks it), 3 LOW (R-4, R-5, C-9). The retry/timeout/rate-limit
hygiene is genuinely good; the weakness is the **un-normalized, heterogeneous ID
space** (URL vs bare, W-ID vs DOI vs arXiv vs S2-hash) threaded through `papers.id`
and `citations.cited_id` — the same root cause behind C-1 and the F-15 ID dance.

---

## Part C — `parsing/` (tex, grobid, nougat, mathpix, figures)

> **Scope reality:** the live corpus is ingested entirely as **`.txt`**
> (`ingest_all.sh` maps every paper to a `*.txt` file), so the `.tex`-clean,
> GROBID, Nougat, Mathpix, and Figures paths are **latent** — only
> `tex.chunk_text` + `quality.*` run on the live data. The findings below are
> real but mostly fire only once `.tex`/`.pdf --rich` ingest is used.
>
> **Not a finding:** the broad `except Exception` in `mathpix.py:109,151` and
> `figures.py:125,152` are *justified* per-item graceful degradation
> (`fitz.open` fails → return empty; pix2tex/image fails → skip that item), not
> swallowed bugs.

### C-10 — Formulas/figures key on the PDF *filename*, not `paper.id` · **MED** (latent, `.pdf --rich`)
- **Evidence:** `figures.py:119` and the mathpix extractor set
  `paper_id = Path(pdf_path).stem`. `ingest.py:289,303` insert
  `f.paper_id` / `fig.paper_id` **unchanged** into `formulas`/`figures`, whose
  schema declares `paper_id TEXT REFERENCES papers(id)` (`schema.sql:97,115`).
- **Impact:** the filename stem (e.g. `springborn-2008-weighted-delaunay…`) will
  not equal `papers.id` (an OpenAlex URL/bare id — see C-7), so the FK insert
  **fails and aborts rich ingest**, or (without FK enforcement) writes orphaned
  rows. Identity is derived from the *file*, not the *paper*. Fix: pass `paper.id`
  into the extractors, or have ingest overwrite `f.paper_id = paper.id` before insert.

### R-7 — `_clean_latex` misses `\(…\)` / `\[…\]` math + strips all math from chunks · **LOW** (`.tex` path)
- **Evidence:** `tex.py:21-56` strips `%comment`, `$…$`, `$$…$$`,
  `equation`/`align` envs, `\cite/\label/\ref` — but **not** the `\(…\)` / `\[…\]`
  math delimiters. Papers using those leave raw `\( … \)` markup in the chunk text.
- **Impact:** (a) leftover math markup pollutes embeddings; (b) *all* math is
  removed from chunk text, so chunk-level semantic search can never match formula
  content — it relies entirely on the separate formula FTS (F-09), which is itself
  latent here. Acceptable as a separation-of-concerns choice, but worth stating.

### R-6 — PDF-path external calls are unwrapped / un-retried · **LOW** (`.pdf` path)
- **Evidence:** `ingest.py:185` calls `extract_references()` outside any
  `try`; GROBID (`grobid.py:54,133`) and the mathpix HTTP calls `raise_for_status`
  with **no** retry/back-off (unlike OpenAlex/S2). A transient GROBID/Mathpix
  5xx aborts the whole PDF ingest.
- **INFO:** `grobid.py:56` uses stdlib `ET.fromstring` on the server's XML.
  Fine for a trusted self-hosted GROBID; switch to `defusedxml` if it is ever
  pointed at an untrusted endpoint.

**Part C net:** 1 MED (C-10), 2 LOW (R-6, R-7), all **latent** for the current
`.txt` corpus. `nougat.py` was skimmed only (54 lines, `.pdf`-only wrapper) — no
deep review yet; flagged in the ledger. Parsing hygiene is otherwise good.

> **Request (a) — second audit pass — is now complete:** §8 of the loop-1 doc
> (grounding guard ✓ Part A, `sources/` ✓ Part B, `parsing/` ✓ Part C) is closed.
> Remaining whole-repo parts D–G are the broader sweep.

---

## Part D — `synthesis.py` orchestration

The lead pipeline (connections → gaps → improvements → conjectures) reuses the
Part-A grounding guard for the three grounded kinds and degrades gracefully on
LLM/DB failure (`_safe_llm_generate`/`_list_paper_titles` swallow + log). The
conjecture invariants are genuinely solid (status hard-set `"unverified"`,
mandatory falsification path or the lead is dropped, written to `conjectures/`
only). Two real defects and a doc contradiction:

### C-11 — Lead-ID collision silently overwrites synthesis output · **HIGH**
- **Evidence:** every stage restarts its counter at `seq = 1`
  (`synthesis.py:341` connections, `:415` gaps, `:530` improvements,
  `:634` conjectures) and ids are `L-0001, L-0002, …` (`_make_lead_id`). The CLI
  concatenates the grounded kinds — `all_leads = connections + gaps + improvements`
  (`cli.py:421`) — and `write_leads` routes **all** grounded kinds to the *same*
  `grounded/<id>.md` (`synthesis.py:783-796`), writing in list order.
- **Bug:** connection `L-0001`, gap `L-0001`, improvement `L-0001` all target
  `grounded/L-0001.md`; later writes overwrite earlier → **improvements clobber
  gaps clobber connections** at each number. Surviving files =
  `max(#connections, #gaps, #improvements)`, **not** the sum. With 5/3/4 leads you
  get 5 files, not 12; connections are lost first. Silent data loss of the
  synthesis engine's core output, masked because "files do get written".
- **Fix direction:** namespace ids by kind (`L-C-0001` / `L-G-0001` / `L-I-0001`)
  or use a single shared counter across stages.

### R-8 — Default-topic gap detection is a false-positive generator · **MED**
- **Evidence:** `find_gaps` defaults `probe_topics` to paper **titles** when
  `topics is None` (`synthesis.py:418-420`); a title query retrieves mostly its
  *own* paper's chunks → `bibkeys_for_topic` ≈ 1, which is `< synthesis_gap_min_coverage`
  (`:450-453`) → the topic is flagged as a gap and the LLM is asked to articulate
  one.
- **Impact:** nearly every ingested paper becomes a spurious "Gap: '<title>'
  covered by only 1 bibkey(s)" candidate — the LLM invents a gap for material that
  is actually well covered. The grounding guard catches some, but the default
  signal is mostly noise. (User-supplied `--topics` avoid this.)

### D-4 — `write_leads` vs `_update_index` docstrings contradict each other · **LOW**
- **Evidence:** `write_leads` (`synthesis.py:762-765`) claims it "maintains an
  **append-only** `INDEX.md`: a previously-recorded id is kept on subsequent runs",
  but `_update_index` (`:802-803`) states — and implements — a "**full rebuild**,
  not append-only" by globbing the on-disk dirs.
- **Impact:** there is no append-only / id-retention logic; the index purely
  reflects current disk state. The guarantee in the `write_leads` docstring is
  false (and interacts with C-11: overwritten ids simply vanish from the index).

**Part D net:** 1 HIGH (C-11, silent lead overwrite), 1 MED (R-8, gap noise),
1 LOW (D-4, doc contradiction). The conjecture path is exemplary; the grounded
path's persistence layer is where it breaks.

---

## Part E — `wiki.py` rest (compile, cross-refs, state, index/log)

Good: `yaml.safe_load`, deterministic `_chunk_hash` (sorted by chunk id) driving a
sound incremental-skip, append-only `log.md`, inline-code protection during
cross-ref injection. The issues are math-corruption, a dead step, and a marking
order bug.

### R-9 — Cross-ref injection corrupts LaTeX math · **MED**
- **Evidence:** `_inject_cross_refs` (`wiki.py:426`) protects only inline-code
  spans (`_INLINE_CODE_RE`); it then runs whole-word, case-insensitive title/alias
  → `[[slug]]` replacement over everything else (`:429-438`). **Math spans
  (`$…$`, `$$…$$`, `\(…\)`) are not protected.**
- **Impact:** the synthesis prompt explicitly asks for LaTeX `$ … $`
  (`wiki.py:495`), so bodies contain math. A concept title/alias that appears
  inside a formula (e.g. a concept named "Energy" inside `$E_{\text{Energy}}$`,
  or a single-word alias) is rewritten to `[[slug]]`, corrupting the LaTeX. The
  risk scales with how common the concept titles/aliases are as math tokens.

### C-12 — `_try_embed_formulas` is a no-op that still costs a DB round-trip per concept · **LOW**
- **Evidence:** `wiki.py:623-637` opens a connection, queries
  `information_schema.tables` for `formulas`, and returns — the actual embed is
  `"(Future: …)"`. Called unconditionally per concept (`compile_concept:609`).
- **Impact:** step 4 of `compile_concept`'s docstring ("Embed formula chunks")
  does nothing, and each concept compile pays an extra DB connection + query for
  no effect. Placeholder masquerading as a feature.

### C-13 — Ungrounded-claim marking can miss claims mutated by cross-ref injection · **LOW**
- **Evidence:** order in `compile_concept` — claims parsed from `raw_output`
  (`:602-603`), then `raw_output` is rewritten by `_inject_cross_refs` (`:606`),
  then `_render_page_markdown` (`:612`) re-finds the *original* `claim.text` by
  regex in the **injected** body (`:544-552`).
- **Impact:** if an ungrounded claim's text contained a term that got replaced
  with `[[slug]]`, the original text no longer matches → the `⚠` mark is silently
  not applied. (Also note: wiki's marking lacks the `(?<!⚠ )` guard that
  synthesis's `_mark_ungrounded` has — two divergent mark implementations.)

### R-10 — Non-atomic compile-state write · **LOW**
- **Evidence:** `_save_compile_state` (`wiki.py:475-477`) writes
  `.compile-state.json` in place. A crash mid-write corrupts it; `_load`
  then catches `JSONDecodeError` → `{}` → every concept recompiles. Graceful but
  loses incrementality. Write to a temp file + `os.replace` for atomicity.

### R-11 — `load_concepts` raises `KeyError` on malformed YAML · **LOW**
- **Evidence:** `entry["slug"]` / `entry["title"]` (`wiki.py:135-136`) are
  unguarded; a concepts.yaml entry missing a key crashes the whole compile with a
  bare `KeyError` rather than a actionable message.

> **Minor:** `compile_all` builds the index from `state.keys()` (`:745`), which
> accumulates slugs across runs — a concept removed from `concepts.yaml` lingers
> in `index.md`/state until manually pruned. Low.

**Part E net:** 1 MED (R-9, math corruption), 4 LOW (C-12, C-13, R-10, R-11). The
incremental-compile machinery is sound; the rendering/cross-ref layer has the rough
edges. `OllamaClient.generate` (no retry, 120 s timeout, `raise` on error) feeds
directly into **C-4** (its exception → empty page → published), so C-4's blast
radius includes every transient Ollama hiccup.

---

## Part F — `embed.py`, `quality.py`, `models.py`

The best-built corner of the repo. `embed.py` L2-normalises correctly (cosine ⇒
dot product), guards empty input and zero-norm rows, and caches a process
singleton. `quality.py` thresholds are all config-driven, `run_quality_pass` is
CLI-wired (`cli.py:513-517`) and atomic (single commit). `models.py` dataclasses
mirror the schema cleanly. Two dead/low-signal items and one corroboration:

### C-14 — The sparse-embedding ("hybrid") path is dead code · **LOW**
- **Evidence:** `Embedder.encode_sparse` / `encode` / `_coerce_sparse` are called
  nowhere in `codex/` (grep for `encode_sparse` / `.encode(` finds only
  `str.encode` in `wiki.py:455`). Only `encode_dense` is wired (ingest, search,
  wiki). There is no sparse column in the schema.
- **Impact:** the `embed.py` header and ADR-0002 advertise "**Hybrid dense +
  sparse** embeddings via BGE-M3" as a headline, but the sparse half is
  unreachable — the actual "hybrid" elsewhere is dense + Postgres FTS. Either wire
  sparse retrieval or drop the capability + correct the docs.

### R-12 — `classify_section` is low-signal as fed · **LOW**
- **Evidence:** `classify_section` matches section headers in the **first 200
  chars** (`quality.py:103-106`), but `tex.chunk_text` produces overlapping
  *word-window* chunks that almost never begin at a header. So most chunks fall
  through to `"body"` (or the DOI-density `"bibliography"` fallback).
- **Impact:** the `chunks.section` column is mostly `"body"` — limited value for
  any section-aware retrieval that depends on it. Not wrong, just weak signal
  given the chunker.

### Corroboration of C-7 (not a new finding)
- `models.Paper.id` docstring (`models.py:20-21`) states the canonical id is "an
  arXiv ID … **or a DOI (e.g. `10.1145/3592430`)**" — the **bare** form. Production
  fills it from OpenAlex in **URL** form (C-7), so the code violates its own model
  contract. This makes C-7 a documented-contract breach, not a judgement call.

**Part F net:** 2 LOW (C-14, R-12) + a C-7 corroboration. These three modules are
clean; no correctness defects.

---

## Part G — `cli.py` rest + `config.py`

Most CLI commands are thin Typer wrappers delegating to already-audited modules
(`ingest`, `search`, `discover`, `wiki`, `synthesis`, `provenance`, `quality`,
`graph`). Settings are pydantic-validated with sensible bounds (`gt=0`, `0..1`).
Findings are config-coupling and output hygiene.

### C-15 — `embedding_dim` is a footgun decoupled from the hardcoded schema · **MED**
- **Evidence:** `embedding_dim` is configurable (`config.py:67-74`, default 1024)
  and `.env.example`/`schema.sql:7` invite other models (Qwen3 1024, **Jina v4
  2048**). But `schema.sql` hardcodes `vector(1024)` on every embedding column
  (`:25,38,102,119`), and `Embedder` derives the real output dim from the *model*,
  using the `dim` setting only for empty-input zero-vectors (`embed.py:50,89`).
- **Impact:** set `EMBEDDING_DIM=2048` (or pick a non-1024 model) and inserts
  fail on a pgvector dimension mismatch — silently for empty abstracts
  (`(1,2048)` zero-vector into `vector(1024)`), structurally for real vectors.
  Nothing couples the knob to the DDL; the setting reads as supported but is not.

### R-13 — `export_bib` does no BibTeX escaping and forces `@article` · **LOW**
- **Evidence:** `provenance.py:174-181` interpolates `title`/`author`/`year` raw
  into `@article{…}`; no escaping/brace-balancing of BibTeX specials
  (`{ } \ % # $ _ &`).
- **Impact:** a title with an unbalanced brace or a stray special char produces a
  malformed `.bib` entry that breaks Doxygen/BibTeX. Also every paper is emitted
  as `@article` although the corpus includes theses/books/chapters
  (`ingest_all.sh`) — bibliographically wrong (functional for `@cite` keys though).

### S-4 — Secrets stored as plain `str`, not `SecretStr` · **LOW**
- **Evidence:** `database_url`, `mathpix_app_key`, `mcp_auth_token`
  (`config.py:29,148,249`) are plain `str`. Any `repr(Settings())` or debug log of
  the settings object prints the DB password / API key / bearer token in clear.
- **Impact:** log/exception-dump leakage surface (complements S-1). Use
  `pydantic.SecretStr` for credential fields.

### R-14 — `mcp_transport` enum not enforced · **LOW**
- **Evidence:** `config.py:229-236` documents "stdio or http" but no validator;
  `main()` (`mcp_server.py:323`) treats anything non-`http` as stdio. A typo
  (`htttp`) silently starts stdio instead of failing loudly.

### D-5 — Inconsistent `validation_alias` on `nougat_url` only · **LOW**
- **Evidence:** `nougat_url` adds `AliasChoices("NOUGAT_URL", "nougat_url")`
  (`config.py:47`) that no other field has — redundant under
  `case_sensitive=False`, and the inconsistency invites confusion.

> **Corroboration of C-7:** `cli.py:45` ingest does **no** id normalization and
> echoes `result.paper_id` (the URL-form id) straight back — no normalization
> layer exists at any tier (source → ingest → CLI).

**Part G net:** 1 MED (C-15, dim/schema footgun), 4 LOW (R-13, S-4, R-14, D-5).
CLI/config are otherwise solid and well-documented.

---

# Executive Summary — whole-repo audit complete (loop-1 + Parts A–G)

**Tally: 45 findings — 8 HIGH, 11 MED, 26 LOW/INFO** (after live-DB verification:
**7 HIGH, 12 MED** — M-1 reclassified, see the verification section at the bottom).
Test suite 330 green, `ruff`/`mypy` clean. The codebase is well-structured and idiomatic; the defects
cluster into five root themes, not random scatter.

### The 8 HIGH findings
| ID | Theme | One-liner |
|----|-------|-----------|
| S-1 | secrets | live DB password in un-ignored `.env.jetson-ingest` (mitigated this session) |
| M-1 | migration | `paper_identifiers` has no migration path; `schema.sql` not re-applyable, `apply_schema` never called |
| C-1 | identity | `discover.py` keeps the ID-mismatch bug F-15 fixed in `graph.py` (over-reports ingested papers as leads) |
| C-7 | identity | `papers.id` stored in OpenAlex **URL** form, violating the bare-ID contract (breaks lookups + cross-citation match) |
| C-4 | silent failure | zero-citation/LLM-outage wiki pages **bypass quarantine** and publish as "compiled" |
| C-11 | silent data loss | synthesis leads collide on `L-000N` across stages → improvements overwrite gaps overwrite connections |
| D-1 | doc integrity | ADR-F15 GO rests on spike numbers measured **before** the shipped resolution fix |
| T-1 | verification | the F-15 resolution SQL (the substance of the last 2 commits) has **zero** real-DB coverage |

### Five root-cause themes
1. **Un-normalized identity (the dominant theme).** C-1, C-7, C-10, M-1, T-3,
   and most of the F-15 ID dance are one problem: a paper's identity exists in
   many un-reconciled forms (bare vs URL · DOI vs arXiv vs OpenAlex-W vs S2-hash
   vs PDF-filename) with **no canonical normalization layer**. One normalization
   boundary at ingest would retire ~5 HIGH/MED findings at once.
2. **"Green but unverified."** T-1/T-2/T-3 — the most error-prone code (resolution
   SQL, migrations, id-mapping) is mocked or tested with unrepresentative fixtures,
   so real bugs (C-4, C-7, C-11) sit behind 330 passing tests.
3. **Silent failure / data loss under degradation.** C-4 (publish empty), C-11
   (overwrite leads), R-1 (over-quarantine) corrupt *outputs* instead of erroring
   — the worst failure mode for a research tool whose value is trust.
4. **Overstated guarantees.** D-1 (stale ADR), D-2/D-4 (doc↔code drift), C-5
   ("zero-fact-leak" ≫ a last-sentence substring check), C-14 ("hybrid" sparse is
   dead) — the docs/commit-language claim more than the code delivers.
5. **Latent rich-path debt.** C-10, R-6, R-7 are real but dormant because the live
   corpus is `.txt`; they detonate on the first `.pdf --rich` ingest.

### Suggested remediation sequencing (themes, not a plan — planning deferred)
- **First:** S-1 (rotate credential — secret already protected), M-1 (migration
  path), C-7+C-1 (one identity-normalization layer), T-1 (real-DB test) — these
  unblock trust in the live corpus.
- **Then:** C-4 + C-11 (stop silent output corruption), D-1 (re-validate ADR-F15).
- **Then:** MED batch (R-1, R-8, R-9, C-15, C-10) and the LOW polish.

### Coverage honesty
Every `codex/` module was read. After the follow-up pass **all are deep-reviewed**
(`nougat.py` included — see Runtime Validation). Runtime state that was initially
*inferred from code* (C-7 id-form, M-1 table existence, and the C-4/C-11
behaviours) has since been **empirically confirmed** — see the *Live DB
Verification* and *Runtime Validation* sections below.

---

# Live DB Verification — 2026-06-15 (read-only, via existing SSH tunnel)

Ran read-only queries against the live Jetson corpus (`papers=36`,
`citations=811`). This **confirms the two identity HIGHs with hard numbers** and
**corrects an over-statement in M-1**.

### C-7 — CONFIRMED ✅ (severity stands: HIGH)
`papers.id` form distribution: **`doi_url=35`, `bare_arxiv=1`, everything else 0**
(of 36). Samples: `https://doi.org/10.48550/arxiv.math/0603097`,
`https://doi.org/10.48550/arxiv.2310.17529`. Even arXiv papers are stored as their
arXiv-DOI **URL**, never the bare `math/0603097` / `2301.x` the schema, `models.Paper`,
and `ingest_all.sh` assume. The single bare id is the one paper OpenAlex 404'd
(fell to `ingest.py:94`). → bare-ID lookups fail for 35/36 papers. Confirmed.

### C-1 — CONFIRMED ✅ with impact (severity stands: HIGH)
- `citations.cited_id` is uniformly **`oa_url=811`** (all OpenAlex URLs) — the live
  corpus was ingested via the OpenAlex path, so the heterogeneous bare-DOI/S2 mix
  theorised in Part B is **not present here** (it would appear only via S2/GROBID,
  i.e. `.pdf`/fallback ingest).
- **`discover.py` wrongly flags 13 already-ingested papers as "dangling / not
  ingested"** (their `cited_id` is an OpenAlex URL whose paper *is* in the corpus
  via `openalex_id`). Meanwhile **`graph.py` correctly resolves 57 cross-citation
  edges** via the `openalex_id` JOIN. The two "dangling" implementations
  measurably disagree — exactly the C-1 divergence, now quantified.

### M-1 — REFINED ⤵ (downgrade acute severity; structural finding stands)
`paper_identifiers` **exists on the live DB with 39 rows** — the catastrophic
"graph JOIN crashes with relation-does-not-exist" scenario **does not occur**;
the table was applied (manually or via DB re-create). The *structural* gap stands:
`apply_schema` is still non-idempotent and never invoked, so the **next** schema
change has no migration path — but the acute "live graph is broken" framing was
**over-stated**. Re-rank M-1 from HIGH → **MED** (latent migration debt, not an
active outage).

### C-2 — PARTIALLY REFUTED ⤵
`paper_identifiers` has **39 rows vs ~35 papers-with-openalex_id**, so ~4 papers
carry a diverging alias → the `pi` JOIN is **not** fully inert (it resolves a
handful of cases the `p` JOIN can't). Keep as LOW/INFO: the table earns its keep
for a few papers, though the cost/benefit at this corpus size is still marginal.

### Net adjustment to the tally
HIGH **8 → 7** (M-1 → MED). C-7 and C-1 are now **confirmed**, not inferred — they
should lead remediation. The dominant root theme (**un-normalized identity**) is
empirically validated: 35/36 ids in the "wrong" form, 13 papers mis-classified by
the older code path, 57 edges rescued by the newer one.

---

# Runtime Validation — 2026-06-15 (C-4 and C-11 reproduced end-to-end)

The two HIGH findings flagged "code-certain, runtime-confirmable" were exercised
against the **real** code paths (injected fake LLM; no Ollama, no DB needed).

### C-11 — CONFIRMED ✅ (lead-ID collision → silent loss; stays HIGH)
Reproduced the exact CLI assembly (`cli.py:421` `connections + gaps +
improvements`, each stage numbered from `L-0001`): **4 distinct leads** (2
connection, 1 gap, 1 improvement) → `write_leads` produced **2 files**.
`grounded/L-0001.md` survived as the **improvement** ("IMPROVEMENT-one"); the
connection and gap `L-0001` were silently overwritten. **2 of 4 leads lost.**

### C-4 — CONFIRMED ✅ (zero-claim page bypasses quarantine; stays HIGH)
Drove the real `compile_all` with a fake LLM returning confident **uncited** prose
(→ 0 parseable claims). With `wiki_min_grounding_rate = 0.5` and an actual
grounding rate of `0.0`, the page was **published to `wiki/` and marked
compiled** (`report.compiled=['test-concept']`, `report.quarantined=[]`) — not
quarantined to `wiki/draft/`. The `total_claims > 0` guard (`wiki.py:730`) is the
exact bypass; the bug fires precisely when grounding evidence is weakest.

### `nougat.py` — now fully reviewed (no new finding)
Folds into **R-6** (PDF-path resilience). Detail: `pdf_to_markdown` retries **only**
`httpx.ConnectError` with `wait_fixed(0)` (no 5xx retry, no back-off) and is
unwrapped at `ingest.py:183` — a Nougat 5xx aborts PDF ingest.

**Closure:** every `codex/` module is deep-reviewed; the two runtime-confirmable
HIGHs are empirically reproduced. The **audit / finding-identification pass is
complete**. Remediation planning is the open next phase.

---

# Migration Incident — 2026-06-15 (M-1 confirmed live, new privilege dimension)

While running the C-7 re-ingest migration (`infra/reingest_canonical_ids.sh`),
M-1 manifested exactly as predicted — and exposed a dimension the static audit
did **not** capture.

**What happened.** The helper ran `TRUNCATE papers CASCADE` (corpus wiped to 0),
then `ingest_all.sh` failed on the first chunk insert:
`UndefinedColumn: column "section" of relation "chunks" does not exist`. The live
`chunks` table predates the F-16 `section` column, and nothing had ever applied
that DDL to the live DB (M-1: `apply_schema` is never called; `schema.sql` is not
re-applyable). The ingest code was ahead of the live schema.

**New finding — privilege separation (extends M-1).** The fix
`ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT` then failed with
`InsufficientPrivilege: must be owner of table chunks`. The live ownership:

| table | owner |
|-------|-------|
| papers, chunks, citations, code_links, formulas, figures | `postgres` |
| paper_identifiers | `researcher` (app user) |

The app role `researcher` has DML + `TRUNCATE` but **not** DDL on the
postgres-owned core tables. So a `codex migrate` / `apply_schema` invoked with the
application's `DATABASE_URL` would fail on every `ALTER` / `CREATE INDEX` against
those tables. **Implication for the M-1 fix:** making `schema.sql` idempotent is
necessary but *not sufficient* — the migration must run under a privileged role
(owner/superuser), via a separate admin connection, not the least-privilege app
user. This is now part of M-1's remediation scope.

**Consequences observed.**
- Corpus left wiped (papers = 0) until the owner adds the column and re-ingest is
  re-run — a destructive-then-fail sequence.
- Helper hardened (`fix(infra)` on `fix/audit-wave-2`): a **preflight** verifies
  `chunks.section` exists *before* the `TRUNCATE`, aborting early with the
  owner-level `ALTER` to run, so a schema gap can no longer cost the corpus.

**Unblock (owner action):**
`sudo -u postgres psql -d papers -c "ALTER TABLE chunks ADD COLUMN IF NOT EXISTS section TEXT;"`
then re-run `bash ingest_all.sh`.

This incident is the strongest possible validation of M-1 and should anchor its
Wave-3 fix: **idempotent `schema.sql` + a `codex migrate` command that connects as
a privileged role**, plus a documented owner/grant model for the app user.